tightloop

This is a competition! Can your language or your superior assembly language skills beat everyone else?

You implement one or more programs in a suite of tests. You run your code in the harness provided and push back the benchmark report. tightloop operates by running your code alongside a reference implementation. As well as a reference implementation, you will have competitors' code that you run against on your system. You tell the harness which competitors you wish to run against and the harness will obey and generate a report with the timings and memory usage of all the runs. This report is what you push back to the repo. The dynamic content on the site will update and you will see your code ranked alongside others on the leaderboard.

Clearly, there is room for deception in this scheme. The test harness produces a report in clear text which can easily be edited or completely falsified. Be warned: unreproducible results can lead to a ban.

Benchmark inputs will be randomized to avoid predictable results. All input is via stdin.

You don't have to implement all of the benchmarks.

The test harness will gather your system information so to make it easy on us, we're requiring that you run linux at this stage.

We encourage closed source submissions. Your programs may be binaries or bytecode for your proprietary VM. Programs that attempt a rm -rf / will not be received well.

Suite

The tests that can be run are

Some tests are borrowed from the computer language benchmarks game.

What you must provide

You must add a directory to tightloop/contrib and a text file within that directory called config that has a description and groups that you want your benchmarks to be compared with.

For example, our intrepid Pauly wants to contribute some tests that he compiles with the gcc on his system. He forks the tightloop repo to his pauly-star codeberg account and then he runs

$ git clone git@codeberg.org:pauly-star/tightloop.git
$ git checkout -b funbranch
$ mkdir contrib/pauly
$ cd contrib/pauly
$ cat >config
Pauly's best benchmarks
c
native
x86_64
gcc
^D

Here, Pauly has put his entry in the c, native, x86_64 and gcc groups. A group is any identifier you like - groups are defined by the crowd. Pauly will be able to see his results alongside others in these groups as well as in the global summary.

Pauly can add a makefile in contrib/pauly/ and his source. He could simply add his binaries to contrib/pauly/bin that are named nbody, knucleotide... and the test harness will just use these. But Pauly wants to share his algorithms and his mighty threading work so he edits some files and does this:

$ pwd
/home/pauly/tightloop/contrib/pauly
$ ls
Makefile config knucleotide.c nbody.c
$ git add Makefile config nbody.c knucleotide.c
$ git commit -m 'first cut'
$ cd ../..
$ bin/run contrib/pauly
running Pauly's best benchmarks
on "AMD A6-6310 APU with AMD Radeon R4 Graphics"
make... complete
nbody 120ms,10MB
knucleotide 230ms,81MB
checking agreement with reference implementation
make... complete
nbody OK
knucleotide OK
report at "contrib/pauly/AMD A6-6310 APU with AMD Radeon R4 Graphics.report"

Notice Pauly uses the test harness bin/run in bin. See Pauly's work here

Getting on the Leaderboard

In order for tightloop to rank your code, you need to run alongside your competitors code. The harness will do this for you, for example, Pauly wants to beat Jack and kelas:

$ bin/run contrib/pauly contrib/jack contrib/kelas
running Pauly's best benchmarks
make... complete
nbody 120ms,10MB
knucleotide 230ms,81MB
running Jack's benchmarks
make... complete
nbody 125ms,9MB
knucleotide 259ms,20MB
running kelas bcc code
make... complete
nbody 110ms,3MB
knucleotide 203ms,7MB
checking agreement with reference implementation
make... complete
nbody OK
knucleotide OK
report at "contrib/pauly/AMD A6-6310 APU with AMD Radeon R4 Graphics.report"

Damn, kelas beat Pauly again! Oh well, Pauly beat Jack, at least. Now he reports back by pushing the report to his repo and submits a pull request.

$ git add 'contrib/pauly/AMD A6-6310 APU with AMD Radeon R4 Graphics.report'
$ git commit -m 'added report'
$ git push

Then he goes to his codeberg.org fork of tightloop and submits a pull request of his branch

Cheating

You must not falsify a report. Anything else is permitted.

If you submit a closed source program, people can still trace your system calls to uncover cheating.

Harness

System information is gathered by

$ uname -s -n -r -m
$ awk -F": " \
 '/^model name/ { N=2ドル } \
 /^cpu MHz/ { H=2ドル } \
 /^cpu core/ { C=2ドル } \
 /^Mem:/ {split(2,ドルa," ");M=a[1];F=a[2]} \
 END { printf "%s\nH%d C%s M%s F%s\n",N,H,C,M,F }' /proc/cpuinfo <(free -mh)

The harness draws a baseline of your system by running the reference implementations with increasing n to find the largest n that runs in less than 0.5 seconds.

 n=1
 while true
 do 
 bin/gen nbody $n >nbody.in_
 if timeout 0.5 bin/nbody nbody.in_ >/dev/null
 then
 mv nbody.in_ nbody.in
 n=$((n + 1))
 else break
 fi
 done

and then runs your test as

$ timeout 10 /usr/bin/time -o nbody.time -f $FORMAT contrib/pauly/bin/nbody nbody.in >nbody.out

FORMAT is 'C%C\nD%D E%E F%F I%I K%K M%M O%O P%P R%R S%S U%U W%W X%X Z%Z c%c e%e k%k p%p r%r s%s t%t w%w x%x'

No tests are accepted that run more than 10 seconds (or 20 times the reference implementation).