WAC Collection PREXII

From PREX Wiki
Jump to navigationJump to search
Documentation
HOW TOs for shift crew
Expert Tools
All Expert Contacts


PREX Main << Weekly Analysis Coordinator << WAC Notes << WAC Collection CREX

Daily

Data Monitoring

The WAC is primarily tasked with verifying the quality of the data taken during each shift - the WAC Notes pages are the ideal place for this information.

Also see PVDB and Online analysis for more useful tools

Set the RCND environment variables on an aonl computer with: 

setenv RCDB_CONNECTION mysql://rcdb@hallcdb.jlab.org:3306/a-rcdb

Analysis Minding

It is necessary that the WAC keep track of the various runs, conditions, cuts, channel maps, and pedestal changes as a function of time and file system.

Reanalyzing with updated cuts as needed

The WAC is responsible for updating the "official" rootfiles for everyone else to use (and everyone else is not permitted to change these root files without WAC permission).

Utilizing the "Prompt" WAC version of the "operations" branch of Japan is the ideal way to accomplish this.

Prompt Code

The prompt code lives in the master branch of Prex-Prompt

An easy script to launch a new terminal, log into a new processor, and run a copy of prompt in the right environment is a cascade of shell scripts living in the ~/scripts/ folder Disclaimer: this technique is computationally intensive, though convenient, also the aggregator shell script appears to hate it... 

[apar@aonl1 ~/scripts]$ ./terminal_prompt_1.sh adaq3 3602

Tracking Data over time

Helpful link for all steps of the process: https://docs.google.com/spreadsheets/d/1WVee1TgdSmqHRLoPxF7-2ty0YYNyqH3qhMulZXQItNQ/edit?usp=sharing

The run-wise plots are uploaded to the folder /u/group/halla/www/hallaweb/html/parity/prex/onlinePlots, which is visible from Haweb

The WAC is also responsible for keeping track of the data over the course of "slugs" (generally defined as in/out insertions of the IHWP or flips of the Wein).

  • There are several codebases, initially we used Tao's postpan collection tool, the "Collector". The collector code lives in Tao's collector repository
  • This was then enhanced with a more general drawing and textfile printing macro.
  • For general data analysis and daily aggregation, this has since been replaced with the root-file reading and histogram filling/parsing set of methods referred to as the "Aggregator"
  • The Aggregator is designed for pulling the data out of rootfiles directly and calculating histogram based quantities off of that (rather than trusting intermediary programs to have the extensibility needed for arbitrary data manipulations)

Do all the Plots (Except grand_prototype)

Create a list of good production runs formatted one run number per line. Save the list in the run_list directory with a file name in the form: 

  ~/PREX/prompt/collector/run_list/slug##.list

Run this script, using the full file path to your newly created run list, and the second argument corresponding to the spectrometers; 0 for both, 1 for RHRS, 2 for LHRS: 

  bash ~/PREX/prompt/EZ_WAC.sh ~/PREX/prompt/collector/run_list/slug##.list #

It will pause momentarily to ask you to enter the string associated with your slug list. If you've matched the formatting above it should be "slug##" This script will create the files listed below and copy them to the online slug directories Slug Plots: 

  o_slug##.txt
  slug##.list
  y_postPanPlots_slug##.pdf
  aggregate_maindet_slug##.pdf
  slug_charge_mon.pdf
  summary_minirun_slug##.pdf
  summary_minirun_slug##.txt
  summary_minirun_slug_linear##.txt
  total_charge_mon.pdf

To create the grand-aggregator plots you will need to run: 

  cd ~/PREX/prompt/Aggregator/drawPostpan
  setenv CAM_OUTPUTDIR /chafs2/work1/apar/aggRootfiles/slugRootfiles/grandRootfile 
  root -l -b -q plotAgg.C'("aggRootfiles/slugRootfiles/minirun_slug","plots/summary_minirun_slug", slug, ihwp, wein, hrs)'
  
  The first parameter points to location of accumulated minirun files.
  The second parameter points to location of output plots and text file.
  The third parameter is the slug number and needs to be entered by user.
  Enter ihwp = 1 for in, 2 for out
  Enter wein = 1 for right, 2 for left
  Enter hrs = 0 for both, 1 for right only, 2 for left only

You will need to run the above command for *each* slug one at a time. Trying to run more than one simultaneously will not work.

After you have finished running the above commands, you can create the pdf with: 

  cd ~/PREX/prompt/Aggregator/drawPostpan
  root -l -b -q grandAgg.C'("/chafs2/work1/apar/aggRootfiles/slugRootfiles/grandRootfile/grand_aggregator.root","~/PREX/prompt/hallaweb_online/slug/slug_list/slug##/OUTFILE")'

Where OUTFILE is the string you want to use as your pdf file name without the ".pdf" Since this only uses slugs 26 onward, usually these a good choice is "slug_26-##"

Old Step-by-step Instructions

To Collect:

  1. login as apar@adaq3
  2. cd into ~/PREX/prompt/collector
  3. source ../../setup_japan.tcsh
  4. OR
  5. gojapan
  6. cd ~/PREX/prompt/collector

WAC List Creation

  1. create a list of run number you want to (see for example test.list)
  2. use PVDB/RCND/RCDB commands/website
  3. make a slug##.list in ~/PREX/prompt/collector/run_list/

Collector:

  1. run the command: ./collector -d ../results/ -l run_list/slug##.list (Note:replace slug##.list with your file name)
  2. a root file named prexPrompt_slug##.root will be stored in the rootfiles directory
  3. to produce plots stay in the collector directory and run ./aggregate
  4. you will be asked to enter slug##, enter it and hit Enter
  5. a aggregate_plots_slug##.pdf file will be created in the plots directory.

Draw Post Pan:

  1. cd into drawPostpan
  2. [apar@adaq1 ~/PREX/prompt/collector/drawPostpan]$ root -l -b -q drawPostPan.C'("~/PREX/prompt/collector/rootfiles/prexPrompt_slug##.root","slug##","list.txt")'

Aggregator

https://github.com/JeffersonLab/japan/blob/feature-aggregatorAddRegressed/rootScripts/aggregator/drawPostpan/README.md

To reanalyze and make new run and minirun files for each run

  • Optional: Make your slug list, then if the run/minirun individual stub files haven't been made yet do.
  • Once the aggregator is done running on all runs in a given slug you can hadd them together and add the units branch to them with
cd ~/PREX/prompt/Aggregator/drawPostpan/ ; ~/PREX/prompt/Aggregator/drawPostpan/accumulate_mini_aggFiles_list.sh slug##
  • Then to make plots/csv file do
[apar@adaq2 ~/PREX/prompt/Aggregator/drawPostpan]$ setenv CAM_OUTPUTDIR /chafs2/work1/apar/aggRootfiles/slugRootfiles/grandRootfile 
[apar@adaq2 ~/PREX/prompt/Aggregator/drawPostpan]$ root -l -b -q plotAgg.C'("aggRootfiles/slugRootfiles/minirun_slug","plots/summary_minirun_slug", slug, ihwp, wein, hrs)'
The first parameter points to location of accumulated minirun files.
The second parameter points to location of output plots and text file.
The third parameter is the slug number and needs to be entered by user.
Enter ihwp = 1 for in, 2 for out
Enter wein = 1 for right, 2 for left
Enter hrs = 0 for both, 1 for right only, 2 for left only
  • To look at the data by hand, find the output slug aggregated files and do:
root /chafs2/work1/apar/aggRootfiles/slugRootfiles/run_slug11.root
root [1] agg->Draw("reg_asym_usl_mean:run_number","","*")
  • To make the grand aggregator plots do
root -l -b -q grandAgg.C'("/chafs2/work1/apar/aggRootfiles/slugRootfiles/grandRootfile/grand_aggregator.root","outLocation...")'

Auto-Slug and Grand plot script

If you want a Grand Plot from slug N->now, and you know that all of N->now-1 has already been analyzed/has a grand root file in the right place, then check out the agg-scripts/make_grand_plots.sh.

You pass it as argument the slug run list file you want to use, then the HRS status (0 for both working), and lastly the slug number to start the set of grand plots from (currently defaults to 60 if no input given, as 60 is the start of the current wien at time of writing)


Proposed Data Frame update

A Data frame is a useful ROOT 6.12 construct for managing data. It marries some of the Draw() data managing functionality into a MultiThread safe and TTree data traversal optimized framework: 

ROOT::RDataFrame d("mul",_file0); or ROOT::RDataFrame d1(*tchainName)
auto cutData = d.Filter("cuts");
auto hist1 = cutData.Define("defined1","math*branch1.leaf1").Histo1D("defined1");
auto hist2 = cutData.Define("defined2","math*branch2.leaf2").Histo1D("defined2");
hist1->Draw()

This executes all of the actual histo filling and math, traversing the tree, all at once (they are "Lazy"). See documentation for more.

Caveat: it needs the exact branch.leaf name, it won't search for the first leaf for you.

Plan:

  • Loop over a config input text file to get the draw commands, cuts and output names (if blank or NULL then just use the draw command as output name)
  • Define all of those using .Define and .Filter and .Histo1D
  • GetMean, RMS, Errors, and nEntries for each
  • Make a totally fresh agg tree and entry with the same traditional structure/or add a line to be appended into agg file
Retrieved from "https://prex.jlab.org/wiki/index.php?title=WAC_Collection_PREXII&oldid=9016"