Difference between revisions of "PVDB"

From PREX Wiki
Jump to: navigation, search
(Instruction for users)
 
(20 intermediate revisions by 4 users not shown)
Line 1: Line 1:
[https://prex.jlab.org/wiki/index.php/Main_Page <B>PREX Main</B>] >> [[Weekly Analysis Coordinator]]
+
[[Main_Page|<B>PREX Main</B>]] << [[Weekly Analysis Coordinator]]
  
 
__TOC__
 
__TOC__
Line 12: Line 12:
 
* RCDB git repository: https://github.com/JeffersonLab/rcdb
 
* RCDB git repository: https://github.com/JeffersonLab/rcdb
 
<br>
 
<br>
Run information is filled at the Start/End of each run as well as after the prompt analysis. For adaq machines, the connection string is set to:<br>
+
Run information is filled at the Start/End of each run as well as after the prompt analysis. <br>
RCDB_CONNECTION: mysql://apcoda@cdaqdb1.jlab.org:3306/a-rcdb
+
  
 
A wrapper script is located at:<br>
 
A wrapper script is located at:<br>
Line 19: Line 18:
 
~adaq/scripts/addRunEnd2pvdb
 
~adaq/scripts/addRunEnd2pvdb
  
 +
* Important Note:
 +
<b>When one needs to modify codes in pvdb directory, please let [mailto:sanghwa@jlab.org Sanghwa] know so that things can be tracked properly. Some of the codes are used for online update, monitoring and shift crew, not just by experts.<br>
 +
Everyone is encouraged to fork the repository and do pull request if one would like to contribute.</b>
  
 
== Instruction for users ==
 
== Instruction for users ==
 
* Connection:
 
* Connection:
RCDB_CONNECTION is already set on adaq machines to the master DB.  
+
One can access the DB copy from any jlab network computer as rcdb. No password required.<br>
To read DB entries from other machines, set it to the read-only copy:
+
In your script, set a hard coded connection string, for example:
  > setenv RCDB_CONNECTION mysql://apcoda@hallcdb.jlab.org:3306/a-rcdb
+
con_str="mysql://rcdb@hallcdb.jlab.org:3306/a-rcdb"
 +
db = rcdb.RCDBProvider(con_str)
 +
 
 +
Or, from non adaq machine (aonl, farm, ..)
 +
  > setenv RCDB_CONNECTION mysql://rcdb@hallcdb.jlab.org:3306/a-rcdb
 +
 
 +
* Note for connection from adaq machine:<br>
 +
RCDB_CONNECTION is set on adaq machines to the master DB by default in order to fill run conditions into the DB.<br>
 +
Please do NOT connect to the master DB. Editing the DB contents should be done through [mailto:sanghwa@jlab.org Sanghwa Park] except for run QA conditions done by the WAC (see below for WAC DB Tools).<br>
 +
If one needs to access DB from adaq machines, set the connection string separately through a hard coded string in a script.
  
 
=== How to read from the DB ===
 
=== How to read from the DB ===
 
* using CLI:
 
* using CLI:
 
To see available options:<br>
 
To see available options:<br>
<b>PLEASE DO NOT USE "write" or "replace" unless you really need!</b>
+
<b>PLEASE DO NOT USE "write" or "replace" without talking to Sanghwa</b>
 
  > rcnd -h
 
  > rcnd -h
  
Line 46: Line 57:
  
  
=== How to add/modify SL's comments for the DB===
+
=== Simple update script for many runs/conditions===
 +
One can update a list of runs for various conditions using an example script<br>
 +
* Copy the script to your working directory:
 +
On the adaq machine as apar:
 +
pvdb/prex/script/update_conditions.py
 +
 
 +
* The script requires input file with the following format:
 +
  run# <condition name> <value>
 +
 
 +
For example:
 +
  3000 respin_comment this run has no charge feed back running
 +
  3000 slug 111
 +
  3001 arm_flag 1
 +
  3005 respin_comment write your comment here 
 +
 
 +
* Set TESTMODE to True to test the script. If set True, it won't update the DB but will print out the input contents
 +
 
 +
* To udpate DB, set TESTMODE to False and type
 +
  > python udpate_conditions.py <input file name>
 +
 
 +
 
 +
=== How to add/modify comments for the DB===
 
on the adaq machines, type:
 
on the adaq machines, type:
 
  > slcomment
 
  > slcomment
  
[[File:Comment gui.png|700px]]
+
[[File:New_slcomment.png]]
 +
 
 +
- Type Run Number and click "OK" <br>
 +
- Existing comment will appear on the box below. If there is nothing, it means there was no comment before. <br>
 +
- Add your comment and click "SAVE AND EXIT"
 +
 
 +
=== WAC DB TOOL ===
 +
on the adaq machines, type:
 +
> pvdb_edit
 +
 
 +
[[File:WACDBTOOL.png]]
 +
 
 +
- Type Run Number and click "Connect" <br>
 +
- Run type and Run flag currently stored in the DB will be shown. You may not see nothing for "Run flag" if this is the first time you are updating the condition.<br>
 +
- Select new "Run type" and "Run flag". <br>
 +
- Click "SAVE" to store the new conditions into the DB
 +
 
 +
=== Make a run list from DB ===
 +
* Example script: https://github.com/JeffersonLab/pvdb/blob/master/prex/examples/make_run_list.py
 +
 
 +
You can run it from any jlab subnet as far as environment variables are set properly. Things are set in aonl machines already. <br>
 +
 
 +
* From aonl as apar:
 +
1) log in
 +
> ssh apar@aonl#
 +
 
 +
2) Copy the script to your working directory
 +
> cp ~/pvdb/prex/examples/make_run_list.py <your_directory>
 +
 
 +
3) How to use, available options
 +
> python make_run_list --help
 +
 
 +
* Start from scratch (e.g. from ifarm):
 +
1) clone the repository
 +
> git clone https://github.com/JeffersonLab/pvdb
 +
> cd pvdb
 +
> git submodule init
 +
> git submodule update
 +
 
 +
2) set environment variables
 +
> source environment.csh
 +
 
 +
3) test if everything is ok (if you see the list of condition types, you are good to go)
 +
> rcnd
 +
 
 +
=== Update Run End Information ===
 +
If a run did not end properly (e.g., coda crash, ..), one can update the run information using coda file and prompt analysis output. Please make sure you give some time for the prompt analysis to finish.
 +
 
 +
0) log in to adaq
 +
> ssh apar@adaq#
 +
 
 +
1) Go to pvdb script directory
 +
> cd ~pvdb/prex/scripts
 +
 
 +
2) run the following script
 +
> python fixer_run_end.py run#
 +
> e.g) python fixer_run_end.py 3411
 +
 
 +
=== Accumulated charge monitor ===
 +
 
 +
* By default, it shows the accumulated charge vs run number for production runs. <br>
 +
Default selection criteria: Production run, 208Pb target <br>
 +
The bash script runs: <br>
 +
- acc_charge.py: to get the run by run total charge and good charge. Also it prints out the total good charge.<br>
 +
- DrawChargeMon.C: to plot accumulated charge vs run <br>
 +
 
 +
How to use: <br>
 +
Go to
 +
> cd ~/PREX/prompt
 +
> ./get_charge.sh
 +
 
 +
* For WAC:
 +
To get the charge for specific range, run list <br>
 +
1) Copy the python script to your working directory
 +
> cp ~/pvdb/prex/examples/acc_charge.py <where you want to save>
  
To replace the existing comment, select yes for "Replace Comment?". To combine the existing comment with the new comment, select "no".<br>
+
2) To print out usage:
 +
> python acc_charge.py -h
 +
usage:
 +
    Get accumulated charge
 +
    Returns text output file
 +
    :run#, start_time, charge_all, good_charge
 +
    :charge_all: for all events, using the average current from epics and run length
 +
    :good_charge: for events in good multiplicity patterns
 +
    How to use
 +
    1) specific run range
 +
    > python acc_charge.py --run=3000-3100
 +
    2) run list (input list file required)
 +
    > python acc_charge.py --list=list.txt
 +
optional arguments:
 +
  -h, --help  show this help message and exit
 +
  --run RUN    run range
 +
  --list LIST  use run list file
  
If you don't want to submit the comment, just close the window (click X on the top right) and don't click the bottom button.
+
2-1) For example, to specify run range:
 +
> python acc_charge.py --run=3400-3450
 +
2-2) To use a run list
 +
> python acc_charge.py --list listfilename
 +
2-3) Select only runs marked as Good:
 +
> python acc_charge.py --run=3400-3450 --goodrun=True
  
 
= Expert tools =  
 
= Expert tools =  
Line 91: Line 218:
  
 
* June 11, 2019<br>
 
* June 11, 2019<br>
- Added conditions "bad_run_flag" and "bad_run_reason" so that users can mark a doubtful run and add the reason for that.<br>
+
- Added "sl_comment" condition and created gui script for WAC. Alias set as "slcomment" and available from any adaq machines.<br>
- Added "sl_comment" condition and created gui script for common use. Alias set as "slcomment" and available from any adaq machines.
+
- script: ~/scripts/guis/pvdblog.tcl (use edit_comment.py)
 +
 
 +
* June 13, 2019<br>
 +
- Added "good_charge" and "run_flag" conditions <br>
 +
- replace comment gui script with a new one: ~/scripts/gui/wac_comment.py
  
 
== Other info ==
 
== Other info ==
Line 104: Line 235:
 
** apar (access from adaq subnet, pw required), granted all privileges on a-rcdb, pvdb
 
** apar (access from adaq subnet, pw required), granted all privileges on a-rcdb, pvdb
 
** apcoda (access from adaq subnet, no pw required), granted all privileges on a-rcdb, pvdb. Used to push run information from adaq.
 
** apcoda (access from adaq subnet, no pw required), granted all privileges on a-rcdb, pvdb. Used to push run information from adaq.
 +
 +
* To modify rcdb_home:
 +
location: /groups/halla/www/hallweb/html/rcdb_home <br>
 +
First, set the primary group to harcdb
 +
> sg harcdb
 +
  
 
----
 
----
 
back to [[Weekly Analysis Coordinator | WAC]]
 
back to [[Weekly Analysis Coordinator | WAC]]

Latest revision as of 10:14, 25 November 2019

PREX Main << Weekly Analysis Coordinator


Parity Run Database (also called PVDB) stores run related information using mysql. DB is running on cdaqdb1 (alias of cdaqfs1) and mirrored over to hallcdb (read-only slave of the cdaqdb1 master). There are a-rcdb and c-rcdb for Hall A and Hall C, respectively. It utilizes existing rcdb framework that has been developed and used for Hall D. While we keep the same schema from the rcdb, we added some parity conditions such as helicity and spin rotator information. For questions, please contact sanghwa@jlab.org.

General information


Run information is filled at the Start/End of each run as well as after the prompt analysis.

A wrapper script is located at:
~adaq/scripts/addRunStart2pvdb
~adaq/scripts/addRunEnd2pvdb

  • Important Note:

When one needs to modify codes in pvdb directory, please let Sanghwa know so that things can be tracked properly. Some of the codes are used for online update, monitoring and shift crew, not just by experts.
Everyone is encouraged to fork the repository and do pull request if one would like to contribute.

Instruction for users

  • Connection:

One can access the DB copy from any jlab network computer as rcdb. No password required.
In your script, set a hard coded connection string, for example:

con_str="mysql://rcdb@hallcdb.jlab.org:3306/a-rcdb"
db = rcdb.RCDBProvider(con_str)

Or, from non adaq machine (aonl, farm, ..)

> setenv RCDB_CONNECTION mysql://rcdb@hallcdb.jlab.org:3306/a-rcdb
  • Note for connection from adaq machine:

RCDB_CONNECTION is set on adaq machines to the master DB by default in order to fill run conditions into the DB.
Please do NOT connect to the master DB. Editing the DB contents should be done through Sanghwa Park except for run QA conditions done by the WAC (see below for WAC DB Tools).
If one needs to access DB from adaq machines, set the connection string separately through a hard coded string in a script.

How to read from the DB

  • using CLI:

To see available options:
PLEASE DO NOT USE "write" or "replace" without talking to Sanghwa

> rcnd -h

Print total number of runs and last run number as well as condition names

> rcnd 

List condition names (with description)

> rcnd --list

Get a condition value for a given run:

> rcnd run# condition_name
ex) > rcnd 2461 run_type


Simple update script for many runs/conditions

One can update a list of runs for various conditions using an example script

  • Copy the script to your working directory:

On the adaq machine as apar: pvdb/prex/script/update_conditions.py

  • The script requires input file with the following format:
 run# <condition name> <value>

For example:

 3000 respin_comment this run has no charge feed back running
 3000 slug 111
 3001 arm_flag 1
 3005 respin_comment write your comment here  
  • Set TESTMODE to True to test the script. If set True, it won't update the DB but will print out the input contents
  • To udpate DB, set TESTMODE to False and type
 > python udpate_conditions.py <input file name>


How to add/modify comments for the DB

on the adaq machines, type:

> slcomment

New slcomment.png

- Type Run Number and click "OK"
- Existing comment will appear on the box below. If there is nothing, it means there was no comment before.
- Add your comment and click "SAVE AND EXIT"

WAC DB TOOL

on the adaq machines, type:

> pvdb_edit

WACDBTOOL.png

- Type Run Number and click "Connect"
- Run type and Run flag currently stored in the DB will be shown. You may not see nothing for "Run flag" if this is the first time you are updating the condition.
- Select new "Run type" and "Run flag".
- Click "SAVE" to store the new conditions into the DB

Make a run list from DB

You can run it from any jlab subnet as far as environment variables are set properly. Things are set in aonl machines already.

  • From aonl as apar:

1) log in

> ssh apar@aonl#

2) Copy the script to your working directory

> cp ~/pvdb/prex/examples/make_run_list.py <your_directory>

3) How to use, available options

> python make_run_list --help
  • Start from scratch (e.g. from ifarm):

1) clone the repository

> git clone https://github.com/JeffersonLab/pvdb
> cd pvdb
> git submodule init
> git submodule update

2) set environment variables

> source environment.csh

3) test if everything is ok (if you see the list of condition types, you are good to go)

> rcnd

Update Run End Information

If a run did not end properly (e.g., coda crash, ..), one can update the run information using coda file and prompt analysis output. Please make sure you give some time for the prompt analysis to finish.

0) log in to adaq

> ssh apar@adaq#

1) Go to pvdb script directory

> cd ~pvdb/prex/scripts

2) run the following script

> python fixer_run_end.py run#
> e.g) python fixer_run_end.py 3411

Accumulated charge monitor

  • By default, it shows the accumulated charge vs run number for production runs.

Default selection criteria: Production run, 208Pb target
The bash script runs:
- acc_charge.py: to get the run by run total charge and good charge. Also it prints out the total good charge.
- DrawChargeMon.C: to plot accumulated charge vs run

How to use:
Go to

> cd ~/PREX/prompt
> ./get_charge.sh
  • For WAC:

To get the charge for specific range, run list
1) Copy the python script to your working directory

> cp ~/pvdb/prex/examples/acc_charge.py <where you want to save>

2) To print out usage:

> python acc_charge.py -h
usage: 
   Get accumulated charge
   Returns text output file
   :run#, start_time, charge_all, good_charge
   :charge_all: for all events, using the average current from epics and run length
   :good_charge: for events in good multiplicity patterns
   How to use
   1) specific run range
   > python acc_charge.py --run=3000-3100
   2) run list (input list file required)
   > python acc_charge.py --list=list.txt
optional arguments:
 -h, --help   show this help message and exit
 --run RUN    run range
 --list LIST  use run list file

2-1) For example, to specify run range:

> python acc_charge.py --run=3400-3450

2-2) To use a run list

> python acc_charge.py --list listfilename

2-3) Select only runs marked as Good:

> python acc_charge.py --run=3400-3450 --goodrun=True

Expert tools

Deployment log

  • May 2019:

- pvdb successfully tested with the parity DAQ. The wrapper scripts are integrated into halogRunStart_parity and halogRunEnd_parity.

- Modified login script to set the environment variables for the connection string and pvdb home on adaq machines.

  • June 10, 2019

- After the transition from hcrcdb to cdaqdb1, we set up the database again. Now the database name is a-rcdb.

Login as root: (if you don’t know the password and you think you have a reason why you need it, contact Sanghwa Park, Brad Sawatzky.

DB user ‘pvdb’ created with access granted from jlab subnet. Granted all privileges on a-rcdb:

> GRANT ALL PRIVILEGES ON `a-rcdb`.* TO ‘pvdb’@’%.jlab.org’;
> SHOW GRANTS for pvdb@’%.jlab.org’;

- adaq user account already exists (apcoda - no pw, apar - pw required). Granted ALL privileges on a-rcdb.

- DB structure created:

 1) Use get_started.py
 To create both default rcdb condition types and parity default types (defined in parity_rcdb)
 > python get_started.py
 Check with rcnd command:
 > rcnd (should print conditions)
 2) To add new conditon types:
 Use add_condition_type.py
 Using CLI (rcnd):
 >rcnd --create my_value --type string --description "This is my value"
 ConditionType created with name='my_value', type='string', is_many_per_run='False

- Added condition type "experiment"

  • June 11, 2019

- Added "sl_comment" condition and created gui script for WAC. Alias set as "slcomment" and available from any adaq machines.
- script: ~/scripts/guis/pvdblog.tcl (use edit_comment.py)

  • June 13, 2019

- Added "good_charge" and "run_flag" conditions
- replace comment gui script with a new one: ~/scripts/gui/wac_comment.py

Other info

  • connect to the master DB (recommended only for authorized ppl):
> mysql -h cdaqdb1 -u <user_name> -p
  • use pvdb database for test and development. Do it by setting the connection string accordingly.
  • a-rcdb accounts:
    • pvdb (access from jlab subnet, pw required), granted all privileges on a-rcdb, pvdb
    • apar (access from adaq subnet, pw required), granted all privileges on a-rcdb, pvdb
    • apcoda (access from adaq subnet, no pw required), granted all privileges on a-rcdb, pvdb. Used to push run information from adaq.
  • To modify rcdb_home:

location: /groups/halla/www/hallweb/html/rcdb_home
First, set the primary group to harcdb

> sg harcdb



back to WAC