Difference between revisions of "Setting up Postpan"

From PREX Wiki
Jump to: navigation, search
(Getting Started)
 
(13 intermediate revisions by the same user not shown)
Line 1: Line 1:
 +
Following these steps will allow you to use existing 'pass1' japan output root files to perform regression using postpan. If you want to [[Running prompt|run postpan's ./prompt]], you will need your own working local version of japan, and some of the instructions below will change as well.
 +
 
== Getting Started ==
 
== Getting Started ==
# Login to apar account on aonl1: <code>ssh apar@aonl1 </code>
+
# Login to the apar account on aonl2: <code>ssh apar@aonl2 </code>
 
# Create a directory for output file storage
 
# Create a directory for output file storage
 +
#* Making your own folder is important, otherwise whatever you change in your directory will be changed in main prex-prompt.
 
#* Right now, aonl2 has space.
 
#* Right now, aonl2 has space.
#* <code> mkdir /aonl2/work1/my_name </code>
+
#* <code> mkdir /aonl2/work1/my_directory </code>
 
# Create a directory for the code
 
# Create a directory for the code
 
## Go to your own personal apar directory. If you don't have one, make one.
 
## Go to your own personal apar directory. If you don't have one, make one.
##* e.g. if your directory name is my_directory, then <code> cd my_directory </code>
+
##* e.g. if your directory name is my_directory, then <code> cd ~/my_directory </code>
 
## Make a new folder inside your directory
 
## Make a new folder inside your directory
 
##* <code> mkdir prompt </code>
 
##* <code> mkdir prompt </code>
 
# Clone the code from github
 
# Clone the code from github
##* <code> cd prompt </code>
+
#* <code> cd prompt </code>
##* <code> git clone https://github.com/JeffersonLab/prex-prompt.git </code>
+
#* <code> git clone https://github.com/JeffersonLab/prex-prompt.git </code>
 
# Set the environment to use ROOT version 6
 
# Set the environment to use ROOT version 6
#* Create a file called reset_env.csh in the new prex-prompt folder. Copy and paste the following code into it:
+
#* Create a file called reset_env_postpan.csh in the new prex-prompt folder. Copy and paste the following code into it:
 
  setenv PATH `echo $PATH | sed 's/:/\n/g' | grep -v "4.34.36" |grep -v ROOT | awk 'NR==1{printf"%s",$1}; NR>1{printf":%s",$1}'`
 
  setenv PATH `echo $PATH | sed 's/:/\n/g' | grep -v "4.34.36" |grep -v ROOT | awk 'NR==1{printf"%s",$1}; NR>1{printf":%s",$1}'`
 
  setenv LD_LIBRARY_PATH `echo $LD_LIBRARY_PATH | sed 's/:/\n/g' | grep -v "5.34.36" | grep -v ROOT | awk 'NR==1{printf"%s",$1}; NR>1{printf":%s",$1}'`
 
  setenv LD_LIBRARY_PATH `echo $LD_LIBRARY_PATH | sed 's/:/\n/g' | grep -v "5.34.36" | grep -v ROOT | awk 'NR==1{printf"%s",$1}; NR>1{printf":%s",$1}'`
Line 19: Line 22:
  
 
== Setting up for analysis ==
 
== Setting up for analysis ==
Currently prex-prompt that you pull from github is missing files.  
+
Currently, ther version of prex-prompt that you pull from github is missing files. There are also many symbolic links that need to be changed. Right now, the symbolic links point to the current working version of japan, and running postpan with these links unchanged could <span style="color:red"> delete or replace files, upload unnecessary plots to the web, and cause a lot of headaches for the WAC. '''So don't skip these steps!''' </span>
 
+
Make sure you are in your new <code> prex-prompt </code> directory
+
cd prex-prompt
+
 
+
You will see several files inside the prex-prompt directory. Remove all the links (the one in cyan color. e.g Aggregator, BMODextractor, hallaweb_online
+
etc). Don't directly delete the link file, it will not only be deleted from your directory but also from original prex-prompt. Use the following command to remove the link files
+
rm <folder_name> -i
+
e.g. rm Aggregator -i,
+
It will ask you yes or no to remove the file, type
+
y
+
After you remove all the linked files from your directory, make a directories of
+
the same names that you just removed from your prex-prompt in different location.
+
mkdir /aonl2/work1/<my_directory>/<folder_name>
+
e.g mkdir /aonl2/work1/<my_directory>/Aggregator
+
 
+
Now change the directories in to a link. Use the following command to create a link
+
ln -s /aonl2/work1/my_directory/<file_name> <file_name>
+
e.g. ln -s /aonl2/work1/<my_directory>/Aggregator Aggregator.
+
 
+
Do similarly for all the folders. Making your own folder is important otherwise whatever you will
+
change in your directory will be changed in main prex-prompt.
+
 
+
Go inside the postpan  directory and remove the link file called “results” similarly as you
+
removed other links and make a new directory called results and convert it into link using above
+
method. Basically you need to remove all the link folders that you see and make your own
+
folder.
+
 
+
* To run prompt-analysis
+
Go to the prex-prompt directory and do,
+
./prompt.sh <run_number>
+
 
+
* Making summary plots for single run
+
./summary.sh <run_number>
+
  
* For list of runs, make a list of runs in a file and do,
+
=== Getting the missing files ===
./prompt_runlist.sh <runlist_filename>
+
Copy the following missing files and directories to your <code> prex-prompt/postpan </code> directory from the <code> ~/PREX/prompt/postpan </code> directory. Do '''NOT''' try to save time by deleting and replacing your entire postpan directory. Git tracking will be all messed up!
  
This will analyse and plot the run number that you gave. Your plots will be saved in
+
'''Example:''' <code> cp ~/PREX/prompt/postpan/redana ~/my_dir/prompt/prex-prompt/postpan </code>
"SummaryPlots" folder and text file will be saved at "SummaryText" folder. The root file will
+
# conf/
be saved in "japanOutput" folder.
+
# scripts/
 +
# src/
 +
# redana
 +
# libRedAna.so
 +
# RedDict_rdict.pcm
 +
# rootlogon.C
 +
# Also copy device_list.h from the <code> ~/PREX/prompt/postpan/rootmacros </code> directory to your <code> prex-prompt/postpan/rootmacros </code> directory
  
* summary.sh script uploads plots and text summary to the webpage. So be mindful
+
=== Fixing the symbolic links ===
when pushing this button.
+
You will see several symbolic links (the ones in cyan color) inside the <code> prex-prompt </code> directory.
 +
# Remove the current symbolic links
 +
#* '''Example:''' <code> rm Aggregator -i </code>
 +
#* <span style="color:red"> '''PROCEED WITH CAUTION''': Typing <code> rm Aggregator/ -i </code> (with a slash '/' ) will delete EVERYTHING in the actual Aggregator folder. Please be careful, because autocomplete will put a slash - no slashes allowed! <span style="color:red">
 +
## Aggregator
 +
## BMODextractor
 +
## hallaweb_online
 +
## japanOutput
 +
## LogFiles
 +
## LRBOutput
 +
## results
 +
## SummaryPlots
 +
## SummaryText
 +
## results in the <code> postpan </code> directory
 +
#* You will be asked if you want to delete the symbolic link. Type <code> y </code> for yes.
 +
# Create new directories
 +
#* This is what your new output file directory <code> /aonl2/work1/my_directory </code> is for
 +
#* Create directories in this folder with the same names as the links that you just removed from your prex-prompt, '''EXCEPT japanOutput'''
 +
#* '''Example:''' <code> mkdir /aonl2/work1/<my_directory>/Aggregator </code>
 +
# Create new symbolic links pointing to the new directories you just made
 +
## Go into the directory you want to create the link in
 +
##* <code> cd ~/my_directory/prompt/prex-postpan/ </code>
 +
## Create a new symbolic link that points to your own personal folder
 +
##* '''Example:''' <code> ln -s /aonl2/work1/<my_directory>/Aggregator Aggregator </code>
 +
## We want to use the already-made japan 'pass1' root files so we don't have to re-run them. Point <code> japanOutput </code> to:
 +
##* <code> ln -s /chafs2/work1/apar/japanOutput japanOutput </code>
  
 +
== Running postpan ==
 +
At last, you are ready to run!
  
* To change the dependent and independent variables (dv and iv) for the regression analysis,
+
=== Running with regression variables already in the root file ===
and run postpan, go to postpan/cong directory and add a desired variables in combo_reg.conf
+
# Go into your prex-prompt directory: <code> cd ~/my_directory/prompt/prex-prompt/ </code>
file. If the run has already analyzed in japanOutput file you can use the following command to
+
# Source ROOT Version 6: <code> source reset_env_postpan.csh </code>
get a postpan results.
+
# Change the dependent and independent variables for the regression analysis, and add any additional cuts
 +
#* These can be found in <code> prex-prompt/postpan/conf/combo_reg.conf </code>
 +
#* Change variables: dv are the dependent varibles and iv are the independent variables
 +
#* Add your own custom cuts
 +
# Run postpan
 +
#* <code> ./auto_postpan.sh <run_number> </code>
  
* To run the postpan
+
=== Running with regression variables not in the root file ===
./auto_postpan.sh <run_number>
+
Sometimes, you'll want to do regression with a linear combination of variables. This is a bit more complicated. You'll need to re-run japan on the raw data so the root tree has the variables you want to look at.
 +
# First, [[Setting up japan|set up japan]]
 +
# Source your <code> reset_env_japan.csh file </code>
 +
# Open the japan/Parity/prminput/prex_datahandler.<run#>.map file corresponding to the run you are going to analyse
 +
# See what file is used for  [ [ QwCombiner ] ]
 +
# Open that file and add your new variables
 +
#* There are examples in the file
 +
# Next, follow the instructions for [[Running prompt|setting up prompt for analysis]]
 +
#* Do NOT run prompt!
 +
# Run <code> ./qwparity -r <run_number> -c prex_prompt.conf </code>
 +
#* This will generate a pass1 japan output file in your japanOutput directory.
 +
# Change the dependent and independent variables for the regression analysis, and add any additional cuts
 +
#* These can be found in <code> prex-prompt/postpan/conf/combo_reg.conf </code>
 +
#* Change variables: dv are the dependent varibles and iv are the independent variables
 +
#* Add your own custom cuts
 +
# Run postpan
 +
#* <code> ./auto_postpan.sh <run_number> </code>
  
Postpan rootfiles will be saved in “results” directory.
+
== Where is the output? ==
 +
Running postpan will generate 2 ouput files that will be saved in the <code> results </code> directory. Running <code> ./auto_postpan.sh </code> again on the same run number will create files with the same name. So rename your output files if you are going to do this, so you can keep track of them.
 +
# '''prexPrompt_<run_number>_postpan_summary.txt'''
 +
#* Raw and corrected means and RMSs of your dependent variables
 +
#* Raw mean and RMS of your independent variables
 +
#* Lists these statistics for the entire run as well as separate mini-runs
 +
# '''prexPrompt_<run_number>_regress_postpan.root'''
 +
#* Any cuts you put in the configuration file are applied by the cut <code> ok_cut == 1 </code>
 +
#* The reg tree contains dependent variables before and after correction, and the independent variables
 +
#* The mini tree contains data that you can sort based on mini run number, <code> minirun </code>

Latest revision as of 15:26, 26 July 2019

Following these steps will allow you to use existing 'pass1' japan output root files to perform regression using postpan. If you want to run postpan's ./prompt, you will need your own working local version of japan, and some of the instructions below will change as well.

Getting Started

  1. Login to the apar account on aonl2: ssh apar@aonl2
  2. Create a directory for output file storage
    • Making your own folder is important, otherwise whatever you change in your directory will be changed in main prex-prompt.
    • Right now, aonl2 has space.
    • mkdir /aonl2/work1/my_directory
  3. Create a directory for the code
    1. Go to your own personal apar directory. If you don't have one, make one.
      • e.g. if your directory name is my_directory, then cd ~/my_directory
    2. Make a new folder inside your directory
      • mkdir prompt
  4. Clone the code from github
  5. Set the environment to use ROOT version 6
    • Create a file called reset_env_postpan.csh in the new prex-prompt folder. Copy and paste the following code into it:
setenv PATH `echo $PATH | sed 's/:/\n/g' | grep -v "4.34.36" |grep -v ROOT | awk 'NR==1{printf"%s",$1}; NR>1{printf":%s",$1}'`
setenv LD_LIBRARY_PATH `echo $LD_LIBRARY_PATH | sed 's/:/\n/g' | grep -v "5.34.36" | grep -v ROOT | awk 'NR==1{printf"%s",$1}; NR>1{printf":%s",$1}'`
setenv ROOTSYS /adaqfs/apps/ROOT/6.14-04 <\code>

Setting up for analysis

Currently, ther version of prex-prompt that you pull from github is missing files. There are also many symbolic links that need to be changed. Right now, the symbolic links point to the current working version of japan, and running postpan with these links unchanged could delete or replace files, upload unnecessary plots to the web, and cause a lot of headaches for the WAC. So don't skip these steps!

Getting the missing files

Copy the following missing files and directories to your prex-prompt/postpan directory from the ~/PREX/prompt/postpan directory. Do NOT try to save time by deleting and replacing your entire postpan directory. Git tracking will be all messed up!

Example: cp ~/PREX/prompt/postpan/redana ~/my_dir/prompt/prex-prompt/postpan

  1. conf/
  2. scripts/
  3. src/
  4. redana
  5. libRedAna.so
  6. RedDict_rdict.pcm
  7. rootlogon.C
  8. Also copy device_list.h from the ~/PREX/prompt/postpan/rootmacros directory to your prex-prompt/postpan/rootmacros directory

Fixing the symbolic links

You will see several symbolic links (the ones in cyan color) inside the prex-prompt directory.

  1. Remove the current symbolic links
    • Example: rm Aggregator -i
    • PROCEED WITH CAUTION: Typing rm Aggregator/ -i (with a slash '/' ) will delete EVERYTHING in the actual Aggregator folder. Please be careful, because autocomplete will put a slash - no slashes allowed!
    1. Aggregator
    2. BMODextractor
    3. hallaweb_online
    4. japanOutput
    5. LogFiles
    6. LRBOutput
    7. results
    8. SummaryPlots
    9. SummaryText
    10. results in the postpan directory
    • You will be asked if you want to delete the symbolic link. Type y for yes.
  2. Create new directories
    • This is what your new output file directory /aonl2/work1/my_directory is for
    • Create directories in this folder with the same names as the links that you just removed from your prex-prompt, EXCEPT japanOutput
    • Example: mkdir /aonl2/work1/<my_directory>/Aggregator
  3. Create new symbolic links pointing to the new directories you just made
    1. Go into the directory you want to create the link in
      • cd ~/my_directory/prompt/prex-postpan/
    2. Create a new symbolic link that points to your own personal folder
      • Example: ln -s /aonl2/work1/<my_directory>/Aggregator Aggregator
    3. We want to use the already-made japan 'pass1' root files so we don't have to re-run them. Point japanOutput to:
      • ln -s /chafs2/work1/apar/japanOutput japanOutput

Running postpan

At last, you are ready to run!

Running with regression variables already in the root file

  1. Go into your prex-prompt directory: cd ~/my_directory/prompt/prex-prompt/
  2. Source ROOT Version 6: source reset_env_postpan.csh
  3. Change the dependent and independent variables for the regression analysis, and add any additional cuts
    • These can be found in prex-prompt/postpan/conf/combo_reg.conf
    • Change variables: dv are the dependent varibles and iv are the independent variables
    • Add your own custom cuts
  4. Run postpan
    • ./auto_postpan.sh <run_number>

Running with regression variables not in the root file

Sometimes, you'll want to do regression with a linear combination of variables. This is a bit more complicated. You'll need to re-run japan on the raw data so the root tree has the variables you want to look at.

  1. First, set up japan
  2. Source your reset_env_japan.csh file
  3. Open the japan/Parity/prminput/prex_datahandler.<run#>.map file corresponding to the run you are going to analyse
  4. See what file is used for [ [ QwCombiner ] ]
  5. Open that file and add your new variables
    • There are examples in the file
  6. Next, follow the instructions for setting up prompt for analysis
    • Do NOT run prompt!
  7. Run ./qwparity -r <run_number> -c prex_prompt.conf
    • This will generate a pass1 japan output file in your japanOutput directory.
  8. Change the dependent and independent variables for the regression analysis, and add any additional cuts
    • These can be found in prex-prompt/postpan/conf/combo_reg.conf
    • Change variables: dv are the dependent varibles and iv are the independent variables
    • Add your own custom cuts
  9. Run postpan
    • ./auto_postpan.sh <run_number>

Where is the output?

Running postpan will generate 2 ouput files that will be saved in the results directory. Running ./auto_postpan.sh again on the same run number will create files with the same name. So rename your output files if you are going to do this, so you can keep track of them.

  1. prexPrompt_<run_number>_postpan_summary.txt
    • Raw and corrected means and RMSs of your dependent variables
    • Raw mean and RMS of your independent variables
    • Lists these statistics for the entire run as well as separate mini-runs
  2. prexPrompt_<run_number>_regress_postpan.root
    • Any cuts you put in the configuration file are applied by the cut ok_cut == 1
    • The reg tree contains dependent variables before and after correction, and the independent variables
    • The mini tree contains data that you can sort based on mini run number, minirun