First Run
With the installation complete, you should be able to cd to a test directory and run examples there with the run.py scripts. Run with:
./run.pyor:
python run.pyAdd the --help or -h option to see a list of command line options and inputs defined for this specific experiment. The options shown by --help consist of four distinct sections, optional arguments, execution options and experimental input arguments, output arguments and debugging and profiling options.
usage: run.py [-h] [--np NP] [--tpp TPP] [--runtime HH:MM:SS]
[--build {CPU,GPU}]
[--flavor {petsc,boomeramg,parasails,pt,direct}]
[--platform {desktop,archer,archer_intel,archer_camel,curie,medtronic,mephisto,smuc_f,smuc_t,smuc_i,vsc2,vsc3,wopr}]
[--queue QUEUE] [--vectorized-fe VECTORIZED_FE] [--dry-run]
[--CARP-opts CARP_OPTS] [--gdb [PROC [PROC ...]]]
[--ddd [PROC [PROC ...]]] [--ddt] [--valgrind [OUTFILE]]
[--valgrind-options [OPT=VAL [OPT=VAL ...]]] [--map]
[--scalasca] [--ID ID] [--suffix SUFFIX]
[--overwrite-behaviour {prompt,overwrite,delete,error}]
[--silent] [--visualize] [--mech-element {P1-P1,P1-P0}]
[--postprocess {phie,optic,activation,axial,filament,efield,mechanics}
[{phie,optic,activation,axial,filament,efield,mechanics} ...]]
[--tend TEND] [--clean]optional arguments:
-h, --help show this help message and exit
--CARP-opts CARP_OPTS
arbitrary CARP options to append to command
--mech-element {P1-P1,P1-P0}
CARP default mechanics finite element (default: P1-P0)
--postprocess {phie,optic,activation,axial,filament,efield,mechanics} [{phie,optic,activation,axial,filament,efield,mechanics} ...]
postprocessing mode(s) to execute
--tend TEND Duration of simulation (ms)
--clean clean generated output files and directoriesexecution options:
--np NP number of processes
--tpp TPP threads per process
--runtime HH:MM:SS max job runtime
--build {CPU,GPU} CARP build to use (default: CPU)
--flavor {petsc,boomeramg,parasails,pt,direct}
CARP flavor
--platform {desktop,archer,archer_intel,archer_camel,curie,medtronic,mephisto,smuc_f,smuc_t,smuc_i,vsc2,vsc3,wopr}
pick a hardware profile from available platforms
--queue QUEUE select a queue to submit job to (batch systems only)
--vectorized-fe VECTORIZED_FE
vectorized FE assembly (default: on with FEMLIB_CUDA,
off otherwise)
--dry-run show command line without running the testdebugging and profiling:
--gdb [PROC [PROC ...]]
start (optionally specified) processes in gdb
--ddd [PROC [PROC ...]]
start (optionally specified) processes in ddd
--ddt start in Allinea ddt debugger
--valgrind [OUTFILE] start in valgrind mode, use in conjunction with --gdb
for interactive mode
--valgrind-options [OPT=VAL [OPT=VAL ...]]
specify valgrind CLI options, without preceding '--'
(default: leak-check=full track-origins=yes)
--map start using Allinea map profiler
--scalasca start in scalasca profiling modeoutput options:
--ID ID manually specify the job ID (output directory)
--suffix SUFFIX add a suffix to the job ID (output directory)
--overwrite-behaviour {prompt,overwrite,delete,error}
behaviour when output directory already exists
--silent toggle silent output
--visualize toggle test results visualisationTo understand how the command line of individual experiments is built it may be insightful to inspect the command line assembled and executed by the script. This is achieved by launching the run script with an additional --dry option. For instance, running a simple experiment like
./run.py --visualize --dryyields the following output (marked up with additional explanatory comments)
-------------------------------------------------------------------------------------------
Launching CARP Simulation 2017-06-13_simple_20.0_pt_np8
-------------------------------------------------------------------------------------------
# pick launcher, core number and executable
mpiexec -n 8 /home/plank/src/carp-dcse-pt/bin/carp.debug.petsc.pt \
\
# feed in static parameter sets compiled in the par-file
+F simple.par \
\
# provide petsc and pt solver settings for elliptic/parabolic PDE, mechanics PDE and
# Purkinje system
+F /home/plank/src/carputils/carputils/resources/options/pt_ell_amg \
+F /home/plank/src/carputils/carputils/resources/options/pt_para_amg \
+F /home/plank/src/carputils/carputils/resources/options/pt_mech_amg \
-ellip_options_file /home/plank/src/carputils/carputils/resources/options/pt_ell_amg \
-parab_options_file /home/plank/src/carputils/carputils/resources/options/pt_para_amg \
-purk_options_file /home/plank/src/carputils/carputils/resources/petsc_options/pastix_opts \
-mechanics_options_file /home/plank/src/carputils/carputils/resources/options/pt_mech_amg \
\
# pick numerical settings, toggle between pt or petsc, Purkinje always uses petsc
-ellip_use_pt 1 \
-parab_use_pt 1 \
-purk_use_pt 0 \
-mech_use_pt 1 \
\
# simulation ID is the name of the output directory
-simID 2017-06-13_simple_20.0_pt_np1 \
\
# pick the mesh we use and define simulation duration
-meshname meshes/2016-02-20_aTlJAwvpWB/block \
-tend 20.0 \
\
# define electrical stimulus
-stimulus[0].x0 450.0 \
-stimulus[0].xd 100.0 \
-stimulus[0].y0 -300.0 \
-stimulus[0].yd 600.0 \
-stimulus[0].z0 -300.0 \
-stimulus[0].zd 600.0 \
\
# some extra output needed for visualization purposes
-gridout_i 3 \
-gridout_e 3
-----------------------------------------------------------------------------------------
Launching Meshalyzer
-----------------------------------------------------------------------------------------
/home/plank/src/meshalyzer/meshalyzer 2017-06-13_simple_20.0_pt_np1/block_i \
2017-06-13_simple_20.0_pt_np1/vm.igb.gz simple.mshzIt is always possible to change general global settings such as the solvers to be used or the target computing platform for which the command line is being assembled.
For instance, switching from the default flavor pt set in settings.yaml to flavor petsc is achieved by
./run.py --visualize --flavor petsc --dryCompared to above, the numerical settings selection has changed
...
# whenever possible, we use petsc solvers now
-ellip_use_pt 0 \
-parab_use_pt 0 \
-purk_use_pt 0 \
-mech_use_pt 0 \
...Setting a different default platform in the settings.yaml file or at the command line through the --platform input parameter changes launcher and command line, also generating a submission script if run on a cluster. For instance, we can easily generate a submission script for the UK national supercomputer ARCHER by adding the appropriate platform string
./run.py --np 532 --platform archer --runtime 00:30:00 --drywhich writes a submission script and, without the --dry option, would directly submit to the queuing system
Requested mesh already exists, skipping generation.
---------------------------------------------------------------------------------------------
Batch Job Submission
---------------------------------------------------------------------------------------------
qsub 2017-06-14_simple_20.0_pt_np528.pbswith the generated submission script
#!/bin/bash --login
#PBS -N 2017-06-14_simp
#PBS -l select=22
#PBS -l walltime=00:30:00
#PBS -A e348
# Make sure any symbolic links are resolved to absolute path
export PBS_O_WORKDIR=$(readlink -f $PBS_O_WORKDIR)
# Change to the directory that the job was submitted from
# (remember this should be on the /work filesystem)
cd $PBS_O_WORKDIR
# Set the number of threads to 1
# This prevents any system libraries from automatically
# using threading.
export OMP_NUM_THREADS=1
################################################################################
# Summary
# Run script executed with options:
# --np 528 --platform archer --runtime 00:30:00 --dry
################################################################################
# Execute Simulation
mkdir -p 2017-06-14_simple_20.0_pt_np528
aprun -n 528 /compute/src/carp-dcse-pt/bin/carp.debug.petsc.pt \
+F simple.par \
+F /compute/src/carputils/carputils/resources/options/pt_ell_amg_large \
+F /compute/src/carputils/carputils/resources/options/pt_para_amg_large \
+F /compute/src/carputils/carputils/resources/options/pt_mech_amg_large \
-ellip_use_pt 1 \
-parab_use_pt 1 \
-purk_use_pt 0 \
-mech_use_pt 1 \
-ellip_options_file /compute/src/carputils/carputils/resources/options/pt_ell_amg_large \
-parab_options_file /compute/src/carputils/carputils/resources/options/pt_para_amg_large \
-purk_options_file /compute/src/carputils/carputils/resources/petsc_options/pastix_opts \
-mechanics_options_file /compute/src/carputils/carputils/resources/options/pt_mech_amg_large \
-vectorized_fe 0 \
-mech_finite_element 0 \
-simID 2017-06-14_simple_20.0_pt_np528 \
-meshname meshes/2015-12-04_qdHibkmous/block \
-tend 20.0 \
-stimulus[0].x0 450.0 \
-stimulus[0].xd 100.0 \
-stimulus[0].y0 -300.0 \
-stimulus[0].yd 600.0 \
-stimulus[0].z0 -300.0 \
-stimulus[0].zd 600.0