DAQ Testing/20181205

Back to Main Page >> DAQ Documentation Portal >> DAQ Testing >> DAQ Commissioning Notes

Previous Day of Testing << >> Next Day of Testing

December 5th, 2018 Testers: Cameron

Goals

• Test ET deadtime with et_monitor C++ executable

ET Deadtime

See also ET Bridge overview

et_monitor program

Program: In the bin or in the CODA install directory of choice you should find an already compiled program called "et_monitor" which accomplishes the same goals as the Java "monitorGui" except without using a gui - it prints the information directly to your monitor (meaning you will need to either pipe it to another readable output, pause the output using ctrl+S and +Q, or use a large enough screen to see the whole dump of information for each refresh).

Running: Execute the program with ```et_monitor -f /path/to/et_sys_par_temp_file``` and note that the temporary ET file is likely called /tmp/et_sys_par1 or 2. On the Hall A adaq computers I have placed the 32 and 64 bit versions (which attach to the 32 and 64 bit ets when they are running respectively) of et_monitor in the ~apar/bin/ folder.

Outputs: The et_monitor will reveal the network of stations and buffers operating within the et system utilizing the /tmp/et_sys_par* file you pass it. The output is simple and easy to interpret (and significantly easier to interpret if you read the ET users manual first, and see ET Users Guide). Some things that would be normal to see in standard Prex/Crex DAQ running are as follows:

• 32 Bit ET: connected directly to CODA and event builder, recorder, and rc-gui. Has several ET "stations" running:
  • "Grand Central Station": This is where event memory is allocated and cleared for reuse. The grand central station is the only mandatory part of the ET system and is configured automatically by the primary ET process. The user can edit properties, such as total number and size of events to be stored in memory and shuttled around. The C based ET system utilizes a disk resident file to keep track of all the components of the system and carries descriptors of each chunk (array of events sent around between stations) of events so that more complicated parallelized ET station configurations can operate unambiguously and avoid race conditions (not relevant for simple PREX/CREX uses). It is absolutely necessary that there be enough events and long enough buffers such that the Grand Central Station never runs out of available units of memory to assign new events for filling with data and writing to "Tape".
  • "Tape": This is the "event recorder" which is where the data that has been filled by CODA is written to disk (and eventually sent to the MSS tape storage system by the user in their own time). It is absolutely imperative that this Tape station receives every single event generated and filled with data by the DAQ so that no events are lost or corrupted, and so this station is definitely to be run in blocking mode to ensure this.
  • "ETBridge": In the 32 bit-to-64 bit paradigm of Prex/Crex where we will continue using a 32 bit CODA (2.6.2) but also want to analyze the data live with an online analysis program (stripped down Japan running feedback software) it is necessary to utilize a 64 bit ET system so that Japan can interface directly with CODA to allow for data processing as quickly as possible. Because it is impossible to recompile old CODA and ET to utilize 64 bit architecture in the 2.6.2 framework we instead rely on the readily available ETBridge ET program which is used to interface separated ET systems as a client program attached to its own station. This station can be run in either blocking or non-blocking mode, depending on if the user demands all events be processed by the second ET system attached on the other end of the bridge (blocking mode), or if the user prefers that the primary (32 bit) ET system operate unperturbed by any lag/delay/failure of the second ET system and its stations and client analyzers (non-blocking mode).
• 64 Bit ET:
  • "Grand Central Station": Same idea as before, but this one receives already generated events from the ETBridge and puts them into he 64 bit ET cycle.
  • "Client": This is a user defined client that can have a client program attached to do online analysis - note that there will be a lag of however large the "to chunk buffer" in the ETBridge is (and setting that chunk size to 1 will potentially take more than 1/1000Hz = 1 ms, so be careful when running at high rates, but low rates like 240 Hz should be manageable - I will test this explicitly later).

Concerns

Deadtime: The concept of DAQ deadtime is a bit difficult to impose into the ET system, as the use of long buffers and chunks of events allows for a substantial separation of processes between different stations and bridged ET systems. One very important distinction to make is that whenever a station or client possesses an event it will hold onto that event until it is done with it or the client or station is closed (fatally or intentionally, both will return the claimed events back into the ET cycle flow). The difference between blocking and non-blocking treatment of events is that a blocking system will act like a dam, preventing any events from passing by it until more space in the input-buffer is available (as the result of successfully processing those events and placing them into the output buffer/immediately into the next station's input buffer if it is not also full), and a non-blocking system will act like a plunge pool (in the events as water in a stream analogy) where the waterfall of incoming events will dump enough into the hole at the bottom of the waterfall and once that hole is full the water will continue to flow over and past the hole, thus causing a mis-ordering of events (the initial water/events are now stuck or slowly moving out of the hole and the rest of the water/events continue to flow past it indifferently).

Lag: Note that in both of these cases (blocking and non-blocking) the process of filling up the hole until the events overflow (non-blocking) or stopping up the dam until the dam is opened and more events can come in and do it again (blocking) that however large the buffer (hole or dam volume) is will determine how much lag there is of the events being produced until they are passed into the client of the station (the station is the hole or dam) and onto the next station's input buffer. Consequently, the act of adding a ETBridge (some sort of lock system - ETBridge - to let a water treatment plant - the online analyzer - see all the events) will necessarily introduce a delay by however much its input buffer is filled by. This is avoidable if the buffer simply never gets backed up, meaning that the analyzer in the 64 bit ET is at least as fast as the data production rate with no hiccups so that the ETBridge never has to store anything in its buffer (it is fast enough at transferring that its buffer does not fill before it passes the data along as long as the chunk size is small but not excessively small such that the chunk sending rate isn't slower than the event rate - this is only potentially a problem if you use a high rate and a small chunk size in non-blocking mode ETBridge).

Possible Aborting: It is feasible that we could introduce a third mode of ETBridge that instead of blocking (and stopping new events from being filled with data at Grand Central Station once the available long total array of events is stopped up in the ETBridge dam) or non-blocking (in which case the events stuck in the plunge pool/input buffer of the ETBridge are mixed up and returned - if ever - to the main stream of events out of order), you can have a non-blocking-abort mode in which once the input buffer of the bridge is full (determined by querying the input buffer on its occupancy) it "gets" the full input buffer and "puts" it into the output buffer, bypassing actually analyzing any of the events, unloading everything fully into the stream and then beginning on a new buffer (so this would act exactly like a dam that fills to the brim and then naturally releases its load without breaking or overflowing or permanently stopping the flow), and it is also possible to just recycle those events directly into the Grand Central Station with a "dump" so that those events don't pollute any downstream stations that may be added (i.e. for a future experiment with multiple online analyzers). Although the ETBridge station is number 3, after the "Tape" process which is number 2, so that it doesn't matter if the ETBridge jumbles up the order of events, because the only thing these events have left to do is get recycled by the Grand Central Station, it may still be nice to have such a non-blocking-abort mode enabled, or at least to think about it for future experiments with higher rates and potential for failure.

Notes on Testing, Abridged (Cameron's Notebook)

There are 4 cases of blocking/non-blocking mode in 2 ET bridged operations, mixing blocking and non-blocking ETBridge and the online analyzer client program in the 64 bit ET system.