# iprbench ## install iprbench ```sh graffy@alambix-frontal:/opt/ipr/cluster/work.local/graffy/bug3372$ python3 -m venv iprbench.venv graffy@alambix-frontal:/opt/ipr/cluster/work.local/graffy/bug3372$ source ./iprbench.venv/bin/activate (iprbench.venv) graffy@alambix-frontal:/opt/ipr/cluster/work.local/graffy/bug3372$ pip install ./iprbench.git ``` ## run unit tests ```sh 20241018-16:56:05 graffy@graffy-ws2:~/work/starbench/iprbench.git$ python3 -m unittest ``` ## launch a benchmark on the current system Here's a simple example to run the benchmark `mamul1` on the current system: ```sh (iprbench.venv) graffy@alambix50:/opt/ipr/cluster/work.local/graffy/bug3958/iprbench.git$ iprbench-run --benchmark-id 'mamul1' --config '{"fortran_compiler": "gfortran:", "blas_library": ":", "matrix_size": 1024, "num_loops":10, "num_cores":2, "launcher": "manual"}' --results-dir /tmp/mamul1_out --target-system-type-id 'debian' --resultsdb-params '{"type": "tsv-files", "tsv_results_dir": "/tmp/mamul1_out/tsv"}' DEBUG:root:extracting package iprbench.resources.mamul1 resource CMakeLists.txt to /tmp/mamul1_out/mamul1 DEBUG:root:extracting package iprbench.resources.mamul1 resource mamul1.F90 to /tmp/mamul1_out/mamul1 DEBUG:root:shell_command = "starbench --source-tree-provider '{"type": "existing-dir", "dir-path": "/tmp/mamul1_out/mamul1"}' --num-cores 2 --output-dir=/tmp/mamul1_out/output --cmake-path=/usr/bin/cmake --cmake-option=-DCMAKE_BUILD_TYPE=Release --cmake-option=-DCMAKE_Fortran_COMPILER=gfortran --cmake-option=-DBLA_VENDOR=OpenBLAS --benchmark-command='./mamul1 1024 10' --output-measurements=/tmp/mamul1_out/output/measurements.tsv" creating build directory /tmp/mamul1_out/output/worker executing the following command in parallel (2 parallel runs) : '['mkdir', '-p', '/tmp/mamul1_out/output/worker/build']' mean duration : 0.002 s (2 runs) configuring /tmp/mamul1_out/mamul1 into /tmp/mamul1_out/output/worker/build ... executing the following command in parallel (2 parallel runs) : '['/usr/bin/cmake', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_Fortran_COMPILER=gfortran', '-DBLA_VENDOR=OpenBLAS', '/tmp/mamul1_out/mamul1']' mean duration : 0.057 s (2 runs) building /tmp/mamul1_out/output/worker/build ... executing the following command in parallel (2 parallel runs) : '['make']' mean duration : 0.368 s (2 runs) benchmarking /tmp/mamul1_out/output/worker/build ... executing the following command in parallel (2 parallel runs) : '['./mamul1', '1024', '10']' mean duration : 1.933 s (2 runs) duration : 1.933 s DEBUG:root:output_measurements_file_path = /tmp/mamul1_out/output/measurements.tsv DEBUG:root:row = Unnamed: 0 0.000000 run_id 0.000000 duration 1.932536 Name: 0, dtype: float64 DEBUG:root:row = Unnamed: 0 1.000000 run_id 1.000000 duration 1.933324 Name: 1, dtype: float64 total number of cores (including virtual cores) on this host : 24 DEBUG:root:table_file_path=/tmp/mamul1_out/tsv/mamul1.tsv measurement_time ipr_bench_version host_id ... duration_stddev duration_min duration_max 0 2024-11-27 10:51:02.551947 0.0.13 ... 0.000557 1.932536 1.933324 [1 rows x 18 columns] ``` Now that the run has completed succesfully, the results can be found in the folder `/tmp/mamul1_out/tsv` that we chose: ```sh (iprbench.venv) graffy@alambix50:/opt/ipr/cluster/work.local/graffy/bug3958/iprbench.git$ cat /tmp/mamul1_out/tsv/mamul1.tsv measurement_time ipr_bench_version host_id host_fqdn user num_cpus cpu_model launcher fortran_compiler blas_library num_cores matrix_size num_loops duration_avg duration_med duration_stddev duration_min duration_max 2024-11-27 10:51:02.551947 0.0.13 alambix50.ipr.univ-rennes.fr graffy 2 intel_xeon_x5650 manual gfortran:12.2.0 libopenblas-pthread:0.3.21 2 1024 10 1.93293 1.93293 0.0005572001435750071 1.932536 1.933324 ``` ## launch benchmark jobs on alambix cluster The following example command submits jobs (one job per cpu architecture) that execute the benchmark `hibridon` on the cluster `alambix` (only for the architecture `intel_xeon_x5650` in this example). ```sh (iprbench.venv) graffy@alambix50:/opt/ipr/cluster/work.local/graffy/bug3958/iprbench.git$ results_dir="$GLOBAL_WORK_DIR/graffy/iprbenchs/test_results/clusterbench_submit/$(date --iso-=seconds)"; clusterbench-submit --cluster-id 'alambix' --arch-regexp "intel_xeon_x5650.*" --benchmark-id 'hibridon' --config '{"fortran_compiler": "ifort:", "blas_library": "intelmkl:", "test_id": "arch4_quick", "hibridon_version": "a3bed1c3ccfbca572003020d3e3d3b1ff3934fad", "cmake_path": "cmake", "num_cores": 1, "launcher": "graffy.manual"}' --results-dir "${results_dir}" --resultsdb-params '{"type": "tsv-files", "tsv_results_dir": "'"$results_dir"'"}' --target-system-type-id "fr.univ-rennes.ipr.cluster-node" INFO:root:available host groups: dict_keys(['intel_core_i5_8350u', 'intel_xeon_x5550', 'intel_xeon_x5650', 'intel_xeon_e5-2660', 'intel_xeon_e5-2660v2', 'intel_xeon_e5-2660v4', 'intel_xeon_gold_6140', 'intel_xeon_gold_6154', 'intel_xeon_gold_5220', 'intel_xeon_gold_6226r', 'intel_xeon_gold_6248r', 'intel_xeon_gold_6348', 'amd_epyc_7282', 'amd_epyc_7452']) INFO:root:requested host groups: ['intel_xeon_x5650'] DEBUG:root:iprbench_venv_hardcoded_path = /tmp/user/59825/iprbench.venv INFO:root:creating /opt/ipr/cluster/work.global/graffy/iprbenchs/test_results/clusterbench_submit/2024-11-27T11:39:42+01:00/iprbench.venv.tgz (the virtual environment that will be used in this bench by all its jobs at some point) Collecting virtualenv-clone Using cached virtualenv_clone-0.5.7-py3-none-any.whl (6.6 kB) Installing collected packages: virtualenv-clone Successfully installed virtualenv-clone-0.5.7 DEBUG:root:type of resultsdb_params = DEBUG:root:resultsdb_params = {'type': 'tsv-files', 'tsv_results_dir': '/opt/ipr/cluster/work.global/graffy/iprbenchs/test_results/clusterbench_submit/2024-11-27T11:39:42+01:00'} DEBUG:root:resultsdb_params = {"type": "tsv-files", "tsv_results_dir": "/opt/ipr/cluster/work.global/graffy/iprbenchs/test_results/clusterbench_submit/2024-11-27T11:39:42+01:00"} DEBUG:root:tags_dict = {'': 'hibridon', '': '/opt/ipr/cluster/work.global/graffy/iprbenchs/test_results/clusterbench_submit/2024-11-27T11:39:42+01:00/intel_xeon_x5650/starbench.job', '': '/tmp/user/59825/iprbench.venv', '': '/opt/ipr/cluster/work.global/graffy/iprbenchs/test_results/clusterbench_submit/2024-11-27T11:39:42+01:00/iprbench.venv.tgz', '': '{\\"fortran_compiler\\": \\"ifort:\\", \\"blas_library\\": \\"intelmkl:\\", \\"test_id\\": \\"arch4_quick\\", \\"hibridon_version\\": \\"a3bed1c3ccfbca572003020d3e3d3b1ff3934fad\\", \\"cmake_path\\": \\"cmake\\", \\"num_cores\\": 12, \\"launcher\\": \\"graffy.manual.alambix.job\'${JOB_ID}\'\\"}', '': '/opt/ipr/cluster/work.global/graffy/iprbenchs/test_results/clusterbench_submit/2024-11-27T11:39:42+01:00', '': '{\\"type\\": \\"tsv-files\\", \\"tsv_results_dir\\": \\"/opt/ipr/cluster/work.global/graffy/iprbenchs/test_results/clusterbench_submit/2024-11-27T11:39:42+01:00\\"}', '': '12', '': 'fr.univ-rennes.ipr.cluster-node'} DEBUG:root:ram_per_core = 1.073741824G DEBUG:root:qsub_args = ['-pe', 'smp', '12', '-l', '"hostname=alambix50.ipr.univ-rennes.fr"', '-S', '/bin/bash', '-cwd', '-m', 'ae', '-l', 'mem_available=1.073741824G', '-j', 'y', '-N', 'hibridon_intel_xeon_x5650'] DEBUG:root:qsub_command = qsub -pe smp 12 -l "hostname=alambix50.ipr.univ-rennes.fr" -S /bin/bash -cwd -m ae -l mem_available=1.073741824G -j y -N hibridon_intel_xeon_x5650 /opt/ipr/cluster/work.global/graffy/iprbenchs/test_results/clusterbench_submit/2024-11-27T11:39:42+01:00/intel_xeon_x5650/starbench.job , working_dir=/opt/ipr/cluster/work.global/graffy/iprbenchs/test_results/clusterbench_submit/2024-11-27T11:39:42+01:00/intel_xeon_x5650 Your job 18886 ("hibridon_intel_xeon_x5650") has been submitted ``` The following command shows that the job is running ```sh (iprbench.venv) graffy@alambix50:/opt/ipr/cluster/work.local/graffy/bug3958/iprbench.git$ qstat job-ID prior name user state submit/start at queue slots ja-task-ID ----------------------------------------------------------------------------------------------------------------- 18886 0.65000 hibridon_i graffy r 11/26/2024 18:15:32 short.q@alambix50.ipr.univ-ren 12 ``` the configuration of the benchmark (`--config`) is defined to run the test `arch4_quick` using the latest versions of ifort and mkl: ```json { "fortran_compiler": "ifort:", "blas_library": "intelmkl:", "test_id": "arch4_quick", "hibridon_version": "a3bed1c3ccfbca572003020d3e3d3b1ff3934fad", "cmake_path": "cmake", "num_cores": 1, "launcher": "graffy.manual" } ``` This will cause the benchmark to use the latest versions on ifort and mkl available on the cluster node that run the benchmark. note: the value given to `num_cores` is not important as `clusterbench-submit` overwrites it with the number of cores of the cluster node that runs the benchmark. the results database backend used in the benchmark (`--resultsdb-params`) is: ```json { "type": "tsv-files", "tsv_results_dir": "/opt/ipr/cluster/work.global/graffy/iprbenchs/test_results/clusterbench_submit/2024-11-27T11:39:42+01:00" } ``` This means that we want to register the results of the benchmark in the tsv (tab separated values) file `/opt/ipr/cluster/work.global/graffy/iprbenchs/test_results/clusterbench_submit/2024-11-27T11:39:42+01:00/hibridon.tsv`. Please note that this result database backend is not really appropriate for `clusterbench-submit`, as it suffers from racing conditions (`sqlserver-viassh-database` would be a better alternative, but it requires a more complicate setup). When the jobs successfully complete, they put their results of the benchmark in `$results_dir` (eg `/opt/ipr/cluster/work.global/graffy/iprbenchs/test_results/clusterbench_submit/2024-11-27T11:39:42+01:00`) ```sh (iprbench.venv) graffy@alambix50:/opt/ipr/cluster/work.local/graffy/bug3958/iprbench.git$ cat /opt/ipr/cluster/work.global/graffy/iprbenchs/test_results/clusterbench_submit/2024-11-27T11\:39\:42+01\:00/hibridon.tsv measurement_time ipr_bench_version host_id host_fqdn user num_cpus cpu_model launcher num_cores hibridon_version fortran_compiler blas_library test_id cmake_path duration_avg duration_med duration_stddev duration_min duration_max num_threads_per_run 2024-11-27 11:42:49.511113 0.0.14 alambix50.ipr.univ-rennes.fr graffy 2 intel_xeon_x5650 graffy.manual.alambix.job18886 12 a3bed1c3ccfbca572003020d3e3d3b1ff3934fad ifort:2021.13.1 intelmkl:2024.2.1 arch4_quick cmake 3.8646755 3.8377410000000003 0.2467767843388766 3.569571 4.220125 1 ``` ## launch benchmark jobs on alambix cluster ### run hibridon benchmark on alambix (production version where results are stored on `iprbenchs` database): ```sh (iprbench.venv) graffy@alambix50:/opt/ipr/cluster/work.local/graffy/bug3958/iprbench.git$ results_dir="$GLOBAL_WORK_DIR/graffy/iprbenchs/test_results/clusterbench_submit/$(date --iso-=seconds)"; clusterbench-submit --cluster-id 'alambix' --arch-regexp ".*" --benchmark-id 'hibridon' --config '{"fortran_compiler": "ifort:", "blas_library": "intelmkl:", "test_id": "nh3h2_qma_long", "hibridon_version": "a3bed1c3ccfbca572003020d3e3d3b1ff3934fad", "cmake_path": "cmake", "num_cores": 1, "launcher": "graffy.manual"}' --results-dir "${results_dir}" --resultsdb-params '{ "type": "sqlserver-viassh-database", "db_server_fqdn": "iprbenchsdb.ipr.univ-rennes1.fr", "db_user": "iprbenchw", "db_name": "iprbenchs", "ssh_user": "iprbenchw" }' --target-system-type-id "fr.univ-rennes.ipr.cluster-node" ``` note: for these runs to succeed, graffy@alambix is expected have have the privilieges to write in `iprbenchs` database. This is ensured by allowing `graffy@alambix` to ssh to `iprbenchw@iprbenchsdb.ipr.univ-rennes1.fr` with the ssh key `perf-bencher` (which has been added to `iprbenchw@iprbenchsdb.ipr.univ-rennes1.fr:~iprbenchw/.ssh/authorized_keys` by maco), using the following setup: ```sh raffy@alambix-frontal:~$ ls -la ./.ssh/ total 92 drwx------ 2 graffy spm 4096 27 nov. 15:46 . drwxr-xr-x 171 graffy spm 12288 4 déc. 19:22 .. -rw------- 1 graffy spm 607 22 mai 2024 authorized_keys -rw-r----- 1 graffy spm 296 27 nov. 15:46 config -rw------- 1 graffy spm 16658 18 nov. 22:48 known_hosts ... -rw------- 1 graffy spm 411 27 nov. 15:29 perf-bencher -rw-r--r-- 1 graffy spm 98 27 nov. 15:29 perf-bencher.pub ... ``` ```sh graffy@alambix-frontal:~$ cat .ssh/config ... Host iprbenchsdb.ipr.univ-rennes1.fr hostname iprbenchsdb.ipr.univ-rennes1.fr user test_iprbenchw IdentityFile ~/.ssh/perf-bencher ``` ### run hibridon benchmark on alambix (test version where results are stored on `test_iprbenchs` database): This example is the same as the previous one except: - use `test_iprbenchs` database instead of `iprbenchs` - use `test_iprbenchw` user instead of `iprbenchw` ## graph the results of benchmarks `showresults` is a command line tool that graphs the results after they've been downloaded from the results directory (for example `/opt/ipr/cluster/work.global/graffy/hibridon/benchmarks/starbench/hibridon/2024-10-10T12:11:44+02:00`) to the hardcoded (at the moment) path `/home/graffy/work/starbench/starbench.git/usecases/ipr/hibridon/results` ```sh 20241010-16:30:54 graffy@graffy-ws2:~/work/starbench/iprbench.git$ showresults ```