Difference between revisions of "ET Bridge"
| (9 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
[[DAQ_Doc_Portal|DAQ Documentation Portal]] << [[DAQ_Testing|DAQ Testing Portal]] << [[DAQ_Software|DAQ Software]] | [[DAQ_Doc_Portal|DAQ Documentation Portal]] << [[DAQ_Testing|DAQ Testing Portal]] << [[DAQ_Software|DAQ Software]] | ||
| − | Two ET Bridge solution - see also - [[DAQ_Software|DAQ Software]] | + | Two ET Bridge solution - see also - [[DAQ_Software|DAQ Software]] and [[:Media:ET_Users_Guide.pdf|ET Users Manual]] |
[[File:Two_et.png|frame|Two ET Bridging logical diagram]] | [[File:Two_et.png|frame|Two ET Bridging logical diagram]] | ||
| Line 10: | Line 10: | ||
== Solution == | == Solution == | ||
| − | Carl Timmer has proposed an ET bridge | + | Carl Timmer has proposed an ET bridge between two separate ET systems |
=== Bob's Note - 9/21/2018 === | === Bob's Note - 9/21/2018 === | ||
| Line 16: | Line 16: | ||
R. Michaels Sept 21, 2018 | R. Michaels Sept 21, 2018 | ||
| − | This describes the scheme, as it exists now, for the 2-ET system. | + | This describes the scheme, as it exists now, for the 2-ET system. See the [[:media:Two_et.png|design diagram]]. |
| − | See the [[:media:Two_et.png|design diagram]]. | ||
==== CODA now employs two new scripts ==== | ==== CODA now employs two new scripts ==== | ||
| Line 33: | Line 32: | ||
These processes are killed by "kcoda" | These processes are killed by "kcoda" | ||
| − | There is a potential timing issue with ETbridge. If it starts too early | + | There is a potential timing issue with ETbridge. If it starts too early it times out because a CODA run takes awhile to start putting data into the ET system. For now, I tie the script to the "go" transition of the event builder (EB) and sleep for 20 sec to let events start to come. Perhaps a more elegant solution can be found. |
| − | it times out because a CODA run takes awhile to start putting data into | ||
| − | the ET system. | ||
| − | the event builder (EB) and sleep for 20 sec to let events start to come. | ||
| − | Perhaps a more elegant solution can be found. | ||
==== A client may connect to the 2nd ET ==== | ==== A client may connect to the 2nd ET ==== | ||
| Line 48: | Line 43: | ||
==== Some work needs to be done to optimize the performance. ==== | ==== Some work needs to be done to optimize the performance. ==== | ||
| − | The following two observations are "good" | + | The following two observations are "good" |
| − | + | * No matter how slow the client is, it does not cause deadtime on the DAQ. (Curiously, though, if it is run as a BLOCKING station it can cause deadtime, so these ET systems are not fully decoupled.) | |
| − | the DAQ. (Curiously, though, if it is run as a BLOCKING station it | + | * If the client is "infinitely fast" it receives 100% of data for a DAQ rate of 960 Hz. |
| − | can cause deadtime, so these ET systems are not fully decoupled.) | ||
| − | |||
| − | for a DAQ rate of 960 Hz. | ||
| − | Where I need to do some work is to see what the ultimate speed | + | Where I need to do some work is to see what the ultimate speed of the client can be for a given DAQ rate. I think a 1 kHz client can work at 240 Hz DAQ rate, but I need to test this some more. |
| − | of the client can be for a given DAQ rate. I think a 1 kHz client | + | |
| − | can work at 240 Hz DAQ rate, but I need to test this some more. | + | === Testing ET Bridge === |
| + | |||
| + | Because the Two ET system is no longer directly connecting the CODA DAQ into the client, with the ET Bridge serving as intermediary, it is not guaranteed that the data that is being transferred to the client is coming as quickly as it should be (due to buffer sizes or the event rate being too fast relative to the speed of the analyzer). | ||
| + | |||
| + | Therefore Cameron and Bob have devised a series of tests to see how much deadtime is seen in the analyzer or in the DAQ based on event blocking or non-blocking modes of ET and ET bridge use. | ||
| + | |||
| + | For notes on testing see DAQ testing notes: | ||
| + | * [[DAQ_Testing/20181102]] | ||
| + | * [[DAQ_Testing/20181105]] | ||
| + | * [[DAQ_Testing/20181203]] | ||
| + | * [[DAQ_Testing/20181205]] | ||
| + | * [[DAQ_Testing/20181217]] - Conclusive results | ||
| + | * [[DAQ_Testing/20190322]] - Speed test | ||
| + | |||
| + | ==== Conclusions ==== | ||
| + | |||
| + | I have written up my notes from the most recent ET deadtime test ([[DAQ_Testing/20181217]]) where I stress tested the ET buffers, online analyzer efficiency, and the ET bridge transfer rate at 30 Hz and 1 kHz and I'm confident that I understand how the system works enough (as in [[DAQ_Testing/20181205]] description) to prescribe default setup that is safe for the DAQ and writing to tape, but that can potentially have missed events in the analyzer if it is too slow, using a non-blocking ET bridge with a queue size of around 500, and a blocking online analyzer client station in the 64 bit ET system. | ||
| + | |||
| + | When running JAPAN on single ROC/crate configurations (i.e. 1/4 of the data we expect in the experiment) it will analyze faster than 1/4 of 1 millisecond (I don't remember the exact number right now), meaning that we are safe using a JAPAN based decoder online even at 1 kHz. We should though be careful to multi-thread and separate the online analyzer processes apart that are doing decoding vs. further online analyses in ROOT which may be substantially slower/event. | ||
| + | |||
| + | CODA and Japan can both run fast enough as long as the circular buffer is turned off - see [[DAQ_Testing/20190322]] | ||
| + | |||
| + | See [https://ace.phys.virginia.edu:80/HAPPEX/3590 Haplog 3590] for a weekly meeting summary of this topic. | ||
| + | |||
| + | ==== ET Bridge Source Code ==== | ||
| + | The ET Bridge source code that was used to compile the apar@adaq1:~/bin/etbridge program lives inside a specific folder on apar@adaq1:~/devices/et-12.0_Bob_Version | ||
| + | |||
| + | It was also being developed by Cameron, in apar@adaq1:~/cameronc/et-12.0_updated/et-12.0/src/etbridge/ | ||
| + | |||
| + | https://github.com/JeffersonLab/HallA-Parity-DAQ/blob/develop/et-12.0_Bob_Version/et32_2_et64.c | ||
[[Category:DAQ]] | [[Category:DAQ]] | ||
Latest revision as of 14:42, 20 May 2021
DAQ Documentation Portal << DAQ Testing Portal << DAQ Software Two ET Bridge solution - see also - DAQ Software and ET Users Manual
Problem
Because CODA 2.6.2 is 32 bit compiled while our analysis software is 64 bit compiled, in order to have an online analyzer that can run feedback on live data we need to generate a new bridging solution to be able to have a CODA ET system capable of communicating both with the 32 CODA DAQ and with the 64 bit online analyzer.
Solution
Carl Timmer has proposed an ET bridge between two separate ET systems
Bob's Note - 9/21/2018
2-ET system and example client code R. Michaels Sept 21, 2018
This describes the scheme, as it exists now, for the 2-ET system. See the design diagram.
CODA now employs two new scripts
- start2ndET .... This starts the second, 64-bit ET system It is started with the usual "startcoda" script.
- ETbridge .... this starts the "bridge" program that transfers data from the 32-bit ET (which is a part of CODA 2.6.2) to the second, 64-bit ET. This program came from Carl Timmer with minor modifications by me.
When running successfully you should see these process The first process is launched, as usual, by the startcoda script. The 2nd process is launched by start2ndET, which is launched by startcoda And the 3rd is launched by ETbridge, which is tied to the EB "go" transition.
11915 pts/8 Sl 0:00 et_start .......... CODA 2.6.2 32-bit 12028 pts/8 Sl 0:00 et_start_64 -p 4444 .... 2nd ET 64-bit 12032 pts/8 Sl 0:00 etbridge -fin /tmp/et_sys_par1 -fout /tmp/et_sys_par2
These processes are killed by "kcoda"
There is a potential timing issue with ETbridge. If it starts too early it times out because a CODA run takes awhile to start putting data into the ET system. For now, I tie the script to the "go" transition of the event builder (EB) and sleep for 20 sec to let events start to come. Perhaps a more elegant solution can be found.
A client may connect to the 2nd ET
A "client" would be JAPAN, for example. The client should use
- port 4444 (this is the default in THaEtClient)
- setenv SESSION par2 (do not the default par1 !) on the apar account
- run as ET_HOST_AS_LOCAL and ET_STATION_NONBLOCKING.
Some work needs to be done to optimize the performance.
The following two observations are "good"
- No matter how slow the client is, it does not cause deadtime on the DAQ. (Curiously, though, if it is run as a BLOCKING station it can cause deadtime, so these ET systems are not fully decoupled.)
- If the client is "infinitely fast" it receives 100% of data for a DAQ rate of 960 Hz.
Where I need to do some work is to see what the ultimate speed of the client can be for a given DAQ rate. I think a 1 kHz client can work at 240 Hz DAQ rate, but I need to test this some more.
Testing ET Bridge
Because the Two ET system is no longer directly connecting the CODA DAQ into the client, with the ET Bridge serving as intermediary, it is not guaranteed that the data that is being transferred to the client is coming as quickly as it should be (due to buffer sizes or the event rate being too fast relative to the speed of the analyzer).
Therefore Cameron and Bob have devised a series of tests to see how much deadtime is seen in the analyzer or in the DAQ based on event blocking or non-blocking modes of ET and ET bridge use.
For notes on testing see DAQ testing notes:
- DAQ_Testing/20181102
- DAQ_Testing/20181105
- DAQ_Testing/20181203
- DAQ_Testing/20181205
- DAQ_Testing/20181217 - Conclusive results
- DAQ_Testing/20190322 - Speed test
Conclusions
I have written up my notes from the most recent ET deadtime test (DAQ_Testing/20181217) where I stress tested the ET buffers, online analyzer efficiency, and the ET bridge transfer rate at 30 Hz and 1 kHz and I'm confident that I understand how the system works enough (as in DAQ_Testing/20181205 description) to prescribe default setup that is safe for the DAQ and writing to tape, but that can potentially have missed events in the analyzer if it is too slow, using a non-blocking ET bridge with a queue size of around 500, and a blocking online analyzer client station in the 64 bit ET system.
When running JAPAN on single ROC/crate configurations (i.e. 1/4 of the data we expect in the experiment) it will analyze faster than 1/4 of 1 millisecond (I don't remember the exact number right now), meaning that we are safe using a JAPAN based decoder online even at 1 kHz. We should though be careful to multi-thread and separate the online analyzer processes apart that are doing decoding vs. further online analyses in ROOT which may be substantially slower/event.
CODA and Japan can both run fast enough as long as the circular buffer is turned off - see DAQ_Testing/20190322
See Haplog 3590 for a weekly meeting summary of this topic.
ET Bridge Source Code
The ET Bridge source code that was used to compile the apar@adaq1:~/bin/etbridge program lives inside a specific folder on apar@adaq1:~/devices/et-12.0_Bob_Version
It was also being developed by Cameron, in apar@adaq1:~/cameronc/et-12.0_updated/et-12.0/src/etbridge/
https://github.com/JeffersonLab/HallA-Parity-DAQ/blob/develop/et-12.0_Bob_Version/et32_2_et64.c
