starbench

a tool to benchmark a git cmake application using embarassingly parallel runs

starbench is a tool designed to build and test the performance of an application versioned in a git repository and using the cmake build system.

In order to measure the performance of the code in hpc (high performance computing) environment, starbench is designed to make all the cores busy. For this, it uses the same technique as in hpl's stardgemm test (that's where the 'star' prefix comes from): the same code is run on each CPU core. This way, we performances measures are expected to be more realistic, as the cores won't benefit from the unrealistic boost provided by the memory cache of unued cores.

If the user provides:

• the url of the repository
• the commit number of the version to test
• the number of cores the benchmark should use (usually the number of cores of the machine that executes the benchmark)
• the benchmark command to use

then starbench will do the rest:

1. clone the repository to a temporary location
2. checkout the requested version
3. configure the build
4. build the code
5. run the becnhmark command for each core
6. output the average duration of the benchmark

example

starbench.py --git-repos-url https://github.com/hibridon/hibridon --code-version a3bed1c3ccfbca572003020d3e3d3b1ff3934fad --git-user g-raffy --git-pass-file "$HOME/.github/personal_access_tokens/bench.hibridon.cluster.ipr.univ-rennes1.fr.pat" --num-cores 2 --output-dir=/tmp/hibench --cmake-path=/opt/cmake/cmake-3.23.0/bin/cmake --cmake-option=-DCMAKE_BUILD_TYPE=Release --cmake-option=-DBUILD_TESTING=ON --benchmark-command='ctest --output-on-failure -L ^arch4_quick$'