Difference between revisions of "WAC Collection"

From PREX Wiki
Jump to: navigation, search
(Collector Code)
Line 55: Line 55:
The collector code lives in [https://github.com/leafybillow/collector Tao's collector repository]
The collector code lives in [https://github.com/leafybillow/collector Tao's collector repository]
==== One pass ====
~/PREX/prompt/collector/do_agg_slug.sh ~/PREX/prompt/collector/run_list/slug##.list
==== Details ====
To Collect:
To Collect:
# login as apar@adaq3
# login as apar@adaq3

Revision as of 16:23, 16 August 2019

HOW TOs for shift crew
Expert Tools
All Expert Contacts

PREX Main << Weekly Analysis Coordinator << WAC Notes


Data Monitoring

The WAC is primarily tasked with verifying the quality of the data taken during each shift - the WAC Notes pages are the ideal place for this information.

Also see PVDB and Online analysis for more useful tools

Set the RCND environment variables on an aonl computer with:

setenv RCDB_CONNECTION mysql://rcdb@hallcdb.jlab.org:3306/a-rcdb

Analysis Minding

It is necessary that the WAC keep track of the various runs, conditions, cuts, channel maps, and pedestal changes as a function of time and file system.

Reanalyzing with updated cuts as needed

The WAC is responsible for updating the "official" rootfiles for everyone else to use (and everyone else is not permitted to change these root files without WAC permission).

Utilizing the "Prompt" WAC version of the "operations" branch of Japan is the ideal way to accomplish this.

Prompt Code

The prompt code lives in the master branch of Prex-Prompt

An easy script to launch a new terminal, log into a new processor, and run a copy of prompt in the right environment is a cascade of shell scripts living in the ~/scripts/ folder Disclaimer: this technique is computationally intensive, though convenient, also the aggregator shell script appears to hate it...

[apar@aonl1 ~/scripts]$ ./terminal_prompt_1.sh adaq3 3602

Tracking Data over time

Helpful link for all steps of the process: https://docs.google.com/spreadsheets/d/1WVee1TgdSmqHRLoPxF7-2ty0YYNyqH3qhMulZXQItNQ/edit?usp=sharing

The run-wise plots are uploaded to the folder /u/group/halla/www/hallaweb/html/parity/prex/onlinePlots, which is visible from Haweb

The WAC is also responsible for keeping track of the data over the course of "slugs" (generally defined as in/out insertions of the IHWP or flips of the Wein).

  • There are several codebases, initially we used Tao's postpan collection tool, the "Collector".
  • This was then enhanced with a more general drawing and textfile printing macro.
  • For general data analysis and daily aggregation, this has since been replaced with the root-file reading and histogram filling/parsing set of methods referred to as the "Aggregator"
  • The Aggregator is designed for pulling the data out of rootfiles directly and calculating histogram based quantities off of that (rather than trusting intermediary programs to have the extensibility needed for arbitrary data manipulations)

Collector Code

The collector code lives in Tao's collector repository

One pass

~/PREX/prompt/collector/do_agg_slug.sh ~/PREX/prompt/collector/run_list/slug##.list


To Collect:

  1. login as apar@adaq3
  2. cd into ~/PREX/prompt/collector
  3. source ../../setup_japan.tcsh

WAC List Creation

  1. create a list of run number you want to (see for example test.list)
  2. use PVDB/RCND/RCDB commands/website
  3. make a slug#.list in ~/PREX/prompt/collector/run_list/


  1. run the command: ./collector -d ../results/ -l run_list/test.list (Note:replace test with your file name)
  2. a root file named prexPrompt_test.root will be stored in the rootfiles directory
  3. to produce plots stay in the collector directory and run ./aggregate
  4. you will be asked to enter a test part of the rootfile, enter it and hit Enter
  5. a aggregate_plots_test.pdf file will be created in the plots directory.

Draw Post Pan:

  1. cd into drawPostpan
  2. [apar@adaq1 ~/PREX/prompt/collector/drawPostpan]$ root -l -b -q drawPostPan.C'("../rootfiles/prexPrompt_slug10.root","slug10-Left","list.txt")'


To reanalyze and make new run and minirun files for each run

  • Optional: Make your slug list, then if the run/minirun individual stub files haven't been made yet do:
  • Once the aggregator is done running on all runs in a given slug you can hadd them together and add the units branch to them with
~/PREX/prompt/Aggregator/drawPostpan/accumulate_mini_aggFiles_list.sh slug11
  • Then to make plots/csv file do
[apar@adaq2 ~/PREX/prompt/Aggregator/drawPostpan]$ setenv CAM_OUTPUTDIR /chafs2/work1/apar/aggRootfiles/slugRootfiles/grandRootfile 
[apar@adaq2 ~/PREX/prompt/Aggregator/drawPostpan]$ root -l -b -q plotAgg.C'("aggRootfiles/slugRootfiles/minirun_slug","plots/summary_minirun_slug", slug, ihwp, wein, hrs)'
The first parameter points to location of accumulated minirun files.
The second parameter points to location of output plots and text file.
The third parameter is the slug number and needs to be entered by user.
Enter ihwp = 1 for in, 2 for out
Enter wein = 1 for right, 2 for left
Enter hrs = 0 for both, 1 for right only, 2 for left only
  • To look at the data by hand, find the output slug aggregated files and do:
root /chafs2/work1/apar/aggRootfiles/slugRootfiles/run_slug11.root
root [1] agg->Draw("reg_asym_usl_mean:run_number","","*")
  • To make the grand aggregator plots do
root -l -b -q grandAgg.C'("/chafs2/work1/apar/aggRootfiles/slugRootfiles/grandRootfile/grand_aggregator.root","outLocation...")'

Proposed Data Frame update

A Data frame is a useful ROOT 6.12 construct for managing data. It marries some of the Draw() data managing functionality into a MultiThread safe and TTree data traversal optimized framework:

ROOT::RDataFrame d("mul",_file0); or ROOT::RDataFrame d1(*tchainName)
auto cutData = d.Filter("cuts");
auto hist1 = cutData.Define("defined1","math*branch1.leaf1").Histo1D("defined1");
auto hist2 = cutData.Define("defined2","math*branch2.leaf2").Histo1D("defined2");

This executes all of the actual histo filling and math, traversing the tree, all at once (they are "Lazy"). See documentation for more.

Caveat: it needs the exact branch.leaf name, it won't search for the first leaf for you.


  • Loop over a config input text file to get the draw commands, cuts and output names (if blank or NULL then just use the draw command as output name)
  • Define all of those using .Define and .Filter and .Histo1D
  • GetMean, RMS, Errors, and nEntries for each
  • Make a totally fresh agg tree and entry with the same traditional structure/or add a line to be appended into agg file