Useful Notes and Links

Reynier Cruz-Torres, PhD

DD4HEP

See more details here and here.

Getting DD4HEP working on Cori

First time
ssh into Cori, go to a convenient directory, and run: 
shifterimg pull eicweb/jug_xl:nightly
shifter --image=eicweb/jug_xl:nightly /bin/bash
eic-shell
source /opt/detector/setup.sh
cd /global/project/projectdirs/m3763/reynier/
mkdir DD4HEP # ONLY THE FIRST TIME
cd DD4HEP
git clone https://eicweb.phy.anl.gov/EIC/benchmarks/reconstruction_benchmarks.git

Subsequent times:

shifter --image=eicweb/jug_xl:nightly /bin/bash
eic-shell
source /opt/detector/setup.sh
cd /global/project/projectdirs/m3763/reynier/DD4HEP

shifterimg pull eicweb/jug_xl:nightly

Testing:

Try the following commands, which run 100 pions:

cd reconstruction_benchmarks/
bash benchmarks/tracking/central_pions.sh

This produces several root files, e.g.:

In the events tree we can find some of the useful variables:

Generated:

MCParticles.momentum.x = 0.000000, 0.000000, -2.105457
MCParticles.momentum.y = 0.000000, 0.000000, 1.102192
MCParticles.momentum.z = 10.000000, 0.000000, -0.241078

The first two entries are the electron and proton kinematics. Here are the reconstructed ones:

ReconstructedParticles.p.x = -2.093476
ReconstructedParticles.p.y = 1.124782
ReconstructedParticles.p.z = -0.241075

Running a batch job

cd /global/project/projectdirs/m3763/reynier/DD4HEP/reconstruction_benchmarks/slurm_scripts
sbatch 1_slurm_test.sh

This script collects all the necessary slurm information and runs 2_run_test_pions.sh. This runs benchmarks/tracking/central_pions.sh. This script collects a series of arguments (e.g. number of events) and then:

Passing background hepmc file to DD4HEP GEANT simulation

npsim --runType batch --numberOfEvents -1 --compactFile ${DETECTOR_PATH}/epic.xml --inputFiles ./out_int_window_100.0ns_nevents_100000.hepmc --outputFile out_0um_new.edm4hep.root --random.seed 42

Compiling EICrecon

First time
- If you want to contribute (push) code: 
git clone git@github.com:eic/EICrecon
- you may need to be added to some list. Alternatively, do: 
git clone https://github.com/eic/EICrecon
cd EICrecon
cmake -S . -B build
cmake --build build --target install -- -j8
source ./bin/eicrecon-this.sh

which eicrecon

Using Barak’s plug-in

Step 1:
git branch -a

We may see by default that we are using the main branch, but we want to use track-qa-barak. To use that branch instead do:

git checkout track-qa-barak

then recompile EICrecon:

cmake -S . -B build; cmake --build build --target install -- -j8
source ./bin/eicrecon-this.sh
Step 2:
cd src/tests/track_qa/

Use this command to run the simulation and reconstruction:

EICrecon/blob/track-qa-barak/src/tests/track_qa/run_sim.sh

You’ll need to adjust the steering file to generate the particle eta, momentum range you want.

Step 3:

To switch between truth seeded and real seeded tracks, toggle this line in EICrecon/blob/track-qa-barak/src/tests/track_qa/trackqa_processor.cc

Truth seeding for track reconstruction:

auto trajectories = event->Get<eicrecon::TrackingResultTrajectory>("CentralCKFTrajectories");

Realistic seeding for track reconstruction:

auto trajectories = event->Get<eicrecon::TrackingResultTrajectory>("CentralCKFSeededTrajectories");
Step 4:
eicrecon -Pplugins=dump_flags,track_qa -Ppodio:output_file=eicrecon_out.root -Ppodio:output_include_collections=MCParticles,CentralTrackSeedingResults -Ptrack_qa:LogLevel=trace -Pjana:nevents=100 ../../../../reconstruction_benchmarks/sim_pions_p_0_10_GeV_eta_-4_4_10000.edm4hep.root | tee eicrecon_out.dat

Software stability

download a specific image version e.g.

shifterimg pull eicweb/jug_xl:nightly-2023-02-16

You can check all available container version on docker at https://hub.docker.com/r/eicweb/jug_xl/tags and check all downloaded version on Cori by:

shifterimg images | grep "jug"

Once you download (pull) a tagged version, you just need to load the same image every time by:

shifter --image=eicweb/jug_xl:nightly-2023-02-16 /bin/bash

Changing beampipe thickness

First time 
cd /project/projectdirs/alice/reynier/eic/
mkdir repos
cd /project/projectdirs/alice/reynier/eic/repos/
- From the same directory, make an install directory and link properly: 
mkdir install
export EIC_SHELL_PREFIX=/project/projectdirs/alice/reynier/eic/repos/install
export LD_LIBRARY_PATH=$EIC_SHELL_PREFIX/lib:$LD_LIBRARY_PATH
source $EIC_SHELL_PREFIX/setup.sh
- From the same directory clone the epic repo and edit beampipe gold coating thickness: 
git clone https://github.com/eic/epic
cd epic
vim compact/central_beampipe.xml
- For example, set: ```gold_thickness="1e-10*um"``` - Then compile this change: 
mkdir build; cd build
cmake .. -DCMAKE_INSTALL_PREFIX=$EIC_SHELL_PREFIX; make; make install; source setup.sh
- Test these changes with a material scan: 
cd /global/project/projectdirs/m3763/reynier/DD4HEP/
materialScan ${DETECTOR_PATH}/epic.xml 0 0 0 0 50 0

Subsequent times

export EIC_SHELL_PREFIX=/project/projectdirs/alice/reynier/eic/repos/install
export LD_LIBRARY_PATH=$EIC_SHELL_PREFIX/lib:$LD_LIBRARY_PATH
source $EIC_SHELL_PREFIX/setup.sh
cd /project/projectdirs/alice/reynier/eic/repos/epic/build/

cmake .. -DCMAKE_INSTALL_PREFIX=$EIC_SHELL_PREFIX; make; make install; source setup.sh

source setup.sh

cd /global/project/projectdirs/m3763/reynier/DD4HEP/
materialScan ${DETECTOR_PATH}/epic.xml 0 0 0 0 50 0

Old method (deprecated)

cd /project/projectdirs/alice/reynier/eic
mkdir repos; cd repos
git clone https://github.com/eic/ip6.git
cd /project/projectdirs/alice/reynier/eic/repos/ip6
vim ip6/central_beampipe.xml

change the gold coating. For example, set:

gold_thickness="1e-10*um"

Then compile this change:

cmake -B build -S . -DCMAKE_INSTALL_PREFIX=install -DEPIC_ECCE_LEGACY_COMPAT=OFF;cmake --build build;cmake --install build

These commands are explained here. Then source:

source install/setup.sh

And finally run a material scan to see if the change shows up:

materialScan ${DETECTOR_PATH}/epic.xml 0 0 0 0 50 0

Synchrotron radiation files

input hepmc files stored on Cori in /global/project/projectdirs/m3763/reynier/DD4HEP/input_hepmc under the filename: out_int_window_100.0ns_nevents_100000_pid_22_Escale_x1.0_status_4_1_seed_1.hepmc where the last number before the .hepmc extension. These files are about 9 Gb each. The output Geant hit files are stored in /global/project/projectdirs/m3763/reynier/DD4HEP/generated_SR_background.

Admixing signal and backgrounds:

Copied hepmc file from central data storage:

mc cp -r --insecure S3/eictest/EPIC/EVGEN/DIS/NC/10x275/minQ2=1/pythia8NCDIS_10x275_minQ2=1_beamEffects_xAngle=-0.025_hiDiv_1.hepmc .

Passed these events through Geant to generate hits:

npsim --runType batch --numberOfEvents 10000 --compactFile ${DETECTOR_PATH}/epic.xml --inputFiles ./dis_events/pythia8NCDIS_10x275_minQ2\=1_beamEffects_xAngle\=-0.025_hiDiv_1.hepmc --outputFile dis_events/dis_for_background_test_10000.edm4hep.root

eicrecon dis_for_background_test.edm4hep.root -Ppodio:background_filename=generated_SR_background/geant_out_int_window_100.0ns_nevents_100000_pid_22_Escale_x1.0_status_4_1_seed_1.edm4hep.root -Ppodio:num_background_events=1

Modify and compile juggler

Yue Shi sent us two codes for the updated realistic seeding: TrackParamACTSSeeding.cpp and track_reconstruction.py. We need to do

  1. Get a copy of Juggler: cd /project/projectdirs/alice/reynier/eic
  2. Update TrackParamACTSSeeding.cpp with the file from Yue Shi: cp ~/TrackParamACTSSeeding.cpp juggler/JugTrack/src/components/
  3. Compile this new juggler version: 
    mkdir local
ATHENA_PREFIX=/project/projectdirs/alice/reynier/eic/
mkdir build; cd build
cmake ../juggler/ -DCMAKE_INSTALL_PREFIX=$ATHENA_PREFIX
make
make install
export JUGGLER_INSTALL_PREFIX=$ATHENA_PREFIX
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${ATHENA_PREFIX}/lib
  4. Also copy track_reconstruction.py from Yue Shi: 
    mv ../track_reconstruction.py reconstruction_benchmarks/benchmarks/tracking/options/track_reconstruction.py

DD4HEP tutorials

All links can be found here