Profiling Launchers via Hello Worlds

I'm doing everything on an HP Elitebook G11 with a Ryzen 7 PRO 8840U (downclocked for pwoer) on CachyOS. I will compare with what's already installed on my computer. hyperfine -N's arguments are shown after "Benchmark".

(By default, hyperfine runs commands like sh -c "./ex param", starting a shell process whose overhead it subtracts. -N makes hyperfine execute directly.

On average, -N adds 100 μs to headline time, 40 μs to user time but decreases system time by ~250 μs - because of overcompensation.)

Profiling Launchers on compiled programs

First I'll show the upper limit of CLI performance, by profiling minimalistic helloworlds simply printing their input.

Janet

(defnmain[_&args](print"Hello";args))

Compiling the above with jpm --no-core quickbin hw.janet hw:

Benchmark 1: ./hw world
 Time (mean ± σ): 558.0 μs ± 101.5 μs [User: 311.1 μs, System: 186.3 μs]
 Range (min ... max): 385.9 μs ... 2145.4 μs 5602 runs

Without --no-core:

Benchmark 1: ./hw2 world
 Time (mean ± σ): 2.2 ms ± 0.2 ms [User: 1.4 ms, System: 0.7 ms]
 Range (min ... max): 1.7 ms ... 4.1 ms 1314 runs

So going forward, I'll use --no-core.

Interpreting as a script:

Benchmark 1: janet hw.janet 'world'
 Time (mean ± σ): 2.2 ms ± 0.2 ms [User: 1.4 ms, System: 0.7 ms]
 Range (min ... max): 1.6 ms ... 4.1 ms 1537 runs

Fennel (Lua)

(fn main[_&args](print (.. "Hello "(table.concat args))))(main((or table.unpack _G.unpack)arg 0))

To compile a Fennel binary with LuaJit:

git clone https://luajit.org/git/luajit.git
cd luajit
make # gitcloned and made to avoid issues with library versions
cd ..
fennel --compile-binary hw-fennel.fnl build/hw-fennel-luajit luajit/src/libluajit.a luajit/src

git clone https://github.com/lua/lua.git
cd lua
make
cd ..
fennel --compile-binary hw-fennel.fnl build/hw-fennel-5.5 lua/liblua.a lua/src

Go

packagemainimport("fmt""os""strings")funcmain(){fmt.Println("Hello "+strings.Join(os.Args[1:],""))}

compiled with go build -o build/hw-go hw-go.go

C

gcc -O3 -o build/hw-c hw-c.c compiles:

#include <stdio.h>
int main(int argc, char **argv) {
 fputs("Hello ", stdout);
 for (int i = 1; i < argc; i++)
 fputs(argv[i], stdout);
 putchar('\n');
 return 0;
}

Rust

rustc -O -o build/hw-rust hw-rust.rs compiles:

fn main(){letmutout=String::from("Hello ");forarginstd::env::args().skip(1){out.push_str(&arg);}println!("{out}");}

Assembly

section .data
 msg db "Hello "
 msglen equ $ - msg
 nl db 10
section .text
 global _start
_start:
 mov rax, 1
 mov rdi, 1
 mov rsi, msg
 mov rdx, msglen
 syscall
 mov r12, [rsp] ; argc
 lea r13, [rsp+8] ; &argv[0]
 mov rbx, 1 ; skip argv[0]
.loop:
 cmp rbx, r12
 jge .done
 mov rsi, [r13 + rbx*8]
 mov rcx, rsi
.strlen:
 cmp byte [rcx], 0
 je .gotlen
 inc rcx
 jmp .strlen
.gotlen:
 sub rcx, rsi
 mov rdx, rcx
 mov rax, 1
 mov rdi, 1
 syscall
 inc rbx
 jmp .loop
.done:
 mov rax, 1
 mov rdi, 1
 mov rsi, nl
 mov rdx, 1
 syscall
 mov rax, 60
 xor rdi, rdi
 syscall

which we compile:

nasm -felf64 hw-assembly.asm -o build/hw-assembly.o
ld build/hw-assembly.o -o build/hw-assembly

Javascript

deno compile -o build/hw-js hw-js.js compiles:

console.log("hello", ...Deno.args);

BQN

•Out"Hello "∾∾•args

TCL

puts "Hello, [lindex $argv 0]!"

Awk

BEGIN {
 s = "Hello "
 for (i = 1; i < ARGC; i++) s = s ARGV[i]
 print s
}

Sed

s/^/Hello /

N.b. sed relies on | (or a file) for input so it'll be run differently.

Bash

#!/usr/bin/env bash
IFS=
echo "Hello $*"

Python3

import sys
def main(_, *args):
 print("Hello " + "".join(args))
main(*sys.argv)

fish

#!/usr/bin/env fish
echo "Hello "(string join '' -- $argv)

Running and Results:

hyperfine --warmup 5 -N \
 'build/hw-assembly World' \
 'build/hw-rust world' \
 'build/hw-go World' \
 'build/hw-c World' \
 'build/hw-fennel-luajit World' \
 'build/hw-fennel-5.5 World' \
 'build/hw-janet world' \
 'build/hw-js world' \
 'janet hw-janet.janet world' \
 'gawk -f hw-awk.awk World' \
 'echo World | sed -f hw-sed.sed' \
 'bash hw-bash.sh World' \
 'bqn hw-bqn.bqn World' \
 'python3 hw-python.py World' \
 'fennel hw-fennel.fnl World' \
 'fish hw-fish.fish World' \
 'deno run hw-js.js world' \
 'tclsh8.6 hw-tcl.tcl World'

Summarized:

1.8 ms ± 0.2 ms	1.0 ms	0.7 ms	1.3 ms ... 3.5 ms	15.98x

What More...

What languages does it make sense to have more benchmarks with? JS will always lose, Rust, C etc. will always win. Certainly Go and Python3. Maybe Fennel/Lua. Maybe Racket.

What to benchmark? Naively I think of:

• randomly generates strings into a hashmap
• splits them on a to make more with parents from the originals?
• places them into alphabetical order in an array (ignoring copies)
• tells you how many distinct strings you split on

Trying to specify it:

• inputs: count, string length, alphabet size, seed, delimiter letter
• generate n strings via a splitmix64 PRNG
• split each on the delimiter and add the new strings to the collection of strings
• alphabetize into an array
• return: number of unique strings, how many original strings contained one or more delimiters,

But actually I like forcing specific datastructures, to exercise normal operations (e.g. hashmaps) and prevents e.g. bitmungling for performance

• Read a log file
• Parse each line
• Extract stuff (timestamp, severity, msg?)
• count occurances of... severity?
• output JS of the most frequent msgs?

SSG

• read .md files with e.g. tables (just needs a looot of .md?)
• maybe some sort of middleware?
• transform into html

or

• read csv/json
• handle malformed lines (how?) (maybe logic to invalidate some too?)
• somehow clean/reformat (make emails lower case, format dates?, derive fields)
• maybe group in a few dimensions?
• sum or average something
• sort
• make an html report?