Difference between revisions of "DAQ Troubleshooting"
From PREX Wiki
Jump to navigationJump to search (Created page with " == For shift workers == thumb|600px|right|Figure 7 - CODA window showing EB1 disconnected. File:Kcoda_while_running.png|thumb|400px|right|Figu...") |
|||
(8 intermediate revisions by 3 users not shown) | |||
Line 1: | Line 1: | ||
+ | <span style="color:red">In the process of editing and updating this page. Please contact Juliette Mammei <B>[mailto:crowder@jlab.org?Subject=PREXwiki crowder@jlab.org]</B> with questions or suggestions.</span> | ||
+ | <br><br> | ||
+ | [[Main_Page|<B>PREX Main</B>]]<< [[CODA]] | ||
== For shift workers == | == For shift workers == | ||
− | [[File:EB_disconnected.png|thumb|600px|right|Figure | + | [[File:EB_disconnected.png|thumb|600px|right|Figure 1 - CODA window showing EB1 disconnected. ]] |
− | [[File:Kcoda_while_running.png|thumb|400px|right|Figure | + | [[File:Kcoda_while_running.png|thumb|400px|right|Figure 2 - If you attempt to kill CODA while is it running, this window pops up. Type Y if you really want to kill it.]] |
− | [[File:ROC_disconnected.png|thumb|600px|right|Figure | + | [[File:ROC_disconnected.png|thumb|600px|right|Figure 3 - If a ROC becomes disconnected, you will see warning messages in the message window.]] |
− | [[File:ROC_disconnected_2.png|thumb|600px|right|Figure | + | [[File:ROC_disconnected_2.png|thumb|600px|right|Figure 4 - There may be a report that a particular ROC is disconnected.]] |
− | [[File:Reboot_ROC_from_terminal.png|thumb|400px|right|Figure | + | [[File:Reboot_ROC_from_terminal.png|thumb|400px|right|Figure 5 - If the ROC's console is responding, you can type ''reboot''. ]] |
− | * Either the Event Builder (EB1) or Event Recorder (ER1) are disconnected (see Figure | + | * Either the Event Builder (EB1) or Event Recorder (ER1) are disconnected (see Figure 1) |
− | ** | + | ** Type '''kcoda''' in a terminal, say '''Y'''(see Figure 2) |
** '''startcoda''' (follow instructions for [[CODA#Starting the GUI| starting CODA]]) | ** '''startcoda''' (follow instructions for [[CODA#Starting the GUI| starting CODA]]) | ||
− | * If a ROC is disconnected (see Figures | + | ** If that fails try again at least once before moving on to other fixes |
− | *# | + | ** It is common for the ER1 to fail when going from Configure to Download in the Run Control GUI too quickly, so patience is advised |
− | *# | + | * If a ROC is disconnected (see Figures 3 and 4) |
− | *## | + | *# Try to reset; if successful you should get Client is back message and everything should report ''Configured'' |
− | *## | + | *# If a ROC is still reporting a problem - note which ROC is reporting the problem |
+ | *## If the xterm console for that ROC is responding type '''reboot''' and enter (see Figure 5) | ||
+ | *### If the problem persists after reboot try rebooting the other ROCs too, try to restart CODA twice, and call the RC | ||
+ | *### Also make sure that the Red "platform" GUI has connected before doing ROC reboots | ||
+ | *### Sometimes it takes several reboots of CODA and ROCs to work - going slow and having patience is key | ||
+ | *### If the ROCs don't even connect and are unresponsive to the terminal/telnet session then remotely power cycling is necessary | ||
+ | *## If the xterm console does not respond to input then try to ping the hostname | ||
+ | *### Hostnames are CH: halladaq6.jlab.org, Injector: qweak1.jlab.org, RHRS: hallavme14.jlab.org, LHRS: happex7.jlab.org | ||
+ | *### If the ping fails to see the hostname then the ROC is turned off or the network switch and console servers are off too and something needs to be rebooted remotely or manually | ||
+ | *## Check with an available expert in the Counting House or call the RC | ||
== For Experts == | == For Experts == | ||
+ | |||
+ | *[[DAQ NFS]] - exists but empty | ||
+ | *[[DAQ Network Ethernet]] - exists but empty | ||
+ | *[[DAQ Network Portserver]] - exists but empty | ||
+ | *<font color="magenta">adaq1:/tmp fills up</font> : see https://logbooks.jlab.org/entry/3724325#comment-23569 | ||
+ | ** DAQ Documentation [[DAQ_Doc_Portal | Portal]] |
Latest revision as of 11:13, 26 August 2019
In the process of editing and updating this page. Please contact Juliette Mammei crowder@jlab.org with questions or suggestions.
PREX Main<< CODA
For shift workers
- Either the Event Builder (EB1) or Event Recorder (ER1) are disconnected (see Figure 1)
- Type kcoda in a terminal, say Y(see Figure 2)
- startcoda (follow instructions for starting CODA)
- If that fails try again at least once before moving on to other fixes
- It is common for the ER1 to fail when going from Configure to Download in the Run Control GUI too quickly, so patience is advised
- If a ROC is disconnected (see Figures 3 and 4)
- Try to reset; if successful you should get Client is back message and everything should report Configured
- If a ROC is still reporting a problem - note which ROC is reporting the problem
- If the xterm console for that ROC is responding type reboot and enter (see Figure 5)
- If the problem persists after reboot try rebooting the other ROCs too, try to restart CODA twice, and call the RC
- Also make sure that the Red "platform" GUI has connected before doing ROC reboots
- Sometimes it takes several reboots of CODA and ROCs to work - going slow and having patience is key
- If the ROCs don't even connect and are unresponsive to the terminal/telnet session then remotely power cycling is necessary
- If the xterm console does not respond to input then try to ping the hostname
- Hostnames are CH: halladaq6.jlab.org, Injector: qweak1.jlab.org, RHRS: hallavme14.jlab.org, LHRS: happex7.jlab.org
- If the ping fails to see the hostname then the ROC is turned off or the network switch and console servers are off too and something needs to be rebooted remotely or manually
- Check with an available expert in the Counting House or call the RC
- If the xterm console for that ROC is responding type reboot and enter (see Figure 5)
For Experts
- DAQ NFS - exists but empty
- DAQ Network Ethernet - exists but empty
- DAQ Network Portserver - exists but empty
- adaq1:/tmp fills up : see https://logbooks.jlab.org/entry/3724325#comment-23569
- DAQ Documentation Portal