ET Bridge

DAQ Documentation Portal << DAQ Testing Portal << DAQ Software Two ET Bridge solution - see also - DAQ Software and ET Users Manual

Two ET Bridging logical diagram

Problem

Because CODA 2.6.2 is 32 bit compiled while our analysis software is 64 bit compiled, in order to have an online analyzer that can run feedback on live data we need to generate a new bridging solution to be able to have a CODA ET system capable of communicating both with the 32 CODA DAQ and with the 64 bit online analyzer.

Solution

Carl Timmer has proposed an ET bridge between two separate ET systems

Bob's Note - 9/21/2018

2-ET system and example client code
R. Michaels   Sept 21, 2018

This describes the scheme, as it exists now, for the 2-ET system. See the design diagram.

CODA now employs two new scripts

• start2ndET .... This starts the second, 64-bit ET system It is started with the usual "startcoda" script.
• ETbridge .... this starts the "bridge" program that transfers data from the 32-bit ET (which is a part of CODA 2.6.2) to the second, 64-bit ET. This program came from Carl Timmer with minor modifications by me.

When running successfully you should see these process The first process is launched, as usual, by the startcoda script. The 2nd process is launched by start2ndET, which is launched by startcoda And the 3rd is launched by ETbridge, which is tied to the EB "go" transition.

11915 pts/8    Sl     0:00 et_start .......... CODA 2.6.2 32-bit
12028 pts/8    Sl     0:00 et_start_64 -p 4444 .... 2nd ET 64-bit
12032 pts/8    Sl     0:00 etbridge -fin /tmp/et_sys_par1 -fout /tmp/et_sys_par2

These processes are killed by "kcoda"

There is a potential timing issue with ETbridge. If it starts too early it times out because a CODA run takes awhile to start putting data into the ET system. For now, I tie the script to the "go" transition of the event builder (EB) and sleep for 20 sec to let events start to come. Perhaps a more elegant solution can be found.

A client may connect to the 2nd ET

A "client" would be JAPAN, for example. The client should use

• port 4444 (this is the default in THaEtClient)
• setenv SESSION par2 (do not the default par1 !) on the apar account
• run as ET_HOST_AS_LOCAL and ET_STATION_NONBLOCKING.

Some work needs to be done to optimize the performance.

The following two observations are "good"

• No matter how slow the client is, it does not cause deadtime on the DAQ. (Curiously, though, if it is run as a BLOCKING station it can cause deadtime, so these ET systems are not fully decoupled.)
• If the client is "infinitely fast" it receives 100% of data for a DAQ rate of 960 Hz.

Where I need to do some work is to see what the ultimate speed of the client can be for a given DAQ rate. I think a 1 kHz client can work at 240 Hz DAQ rate, but I need to test this some more.

Testing ET Bridge

Because the Two ET system is no longer directly connecting the CODA DAQ into the client, with the ET Bridge serving as intermediary, it is not guaranteed that the data that is being transferred to the client is coming as quickly as it should be (due to buffer sizes or the event rate being too fast relative to the speed of the analyzer).

Therefore Cameron and Bob have devised a series of tests to see how much deadtime is seen in the analyzer or in the DAQ based on event blocking or non-blocking modes of ET and ET bridge use.

For notes on testing see DAQ testing notes:

• DAQ_Testing/20181102
• DAQ_Testing/20181105
• DAQ_Testing/20181203
• DAQ_Testing/20181205
• DAQ_Testing/20181217 - Conclusive results
• DAQ_Testing/20190322 - Speed test

Conclusions

I have written up my notes from the most recent ET deadtime test (DAQ_Testing/20181217) where I stress tested the ET buffers, online analyzer efficiency, and the ET bridge transfer rate at 30 Hz and 1 kHz and I'm confident that I understand how the system works enough (as in DAQ_Testing/20181205 description) to prescribe default setup that is safe for the DAQ and writing to tape, but that can potentially have missed events in the analyzer if it is too slow, using a non-blocking ET bridge with a queue size of around 500, and a blocking online analyzer client station in the 64 bit ET system.

When running JAPAN on single ROC/crate configurations (i.e. 1/4 of the data we expect in the experiment) it will analyze faster than 1/4 of 1 millisecond (I don't remember the exact number right now), meaning that we are safe using a JAPAN based decoder online even at 1 kHz. We should though be careful to multi-thread and separate the online analyzer processes apart that are doing decoding vs. further online analyses in ROOT which may be substantially slower/event.

CODA and Japan can both run fast enough as long as the circular buffer is turned off - see DAQ_Testing/20190322

See Haplog 3590 for a weekly meeting summary of this topic.

ET Bridge Source Code

The ET Bridge source code that was used to compile the apar@adaq1:~/bin/etbridge program lives inside a specific folder on apar@adaq1:~/devices/et-12.0_Bob_Version

It was also being developed by Cameron, in apar@adaq1:~/cameronc/et-12.0_updated/et-12.0/src/etbridge/

https://github.com/JeffersonLab/HallA-Parity-DAQ/blob/develop/et-12.0_Bob_Version/et32_2_et64.c