mitata

Install

Recommendations

Quick Start

configure your experience

import { run } from 'mitata';
await run({ format: 'json' }) // output json
await run({ filter: /newArray.*/ }) // only run benchmarks that match regex filter
await run({ throw: true }); // will immediately throw instead of handling error quietly
await run({ format: { mitata: { name: 'fixed' } } }); // benchmarks name column is fixed length
// c++
auto stats = runner.run({ .colors = true, .format = "json", .filter = std::regex(".*") });

garbage collection

bench('lots of allocations', () => {
 Array.from({ length: 1024 }, () => Array.from({ length: 1024 }, () => new Array(1024)));
})
 // mode: false | 'once' (default) | 'inner'
 // once mode runs gc after warmup
 // inner mode runs gc after warmup and before each (batch-)iteration
 .gc('inner');

gc impact and memory usage

------------------------------------------- -------------------------------
new Array(512) 509.42 ns/iter 536.53 ns ▅▃█ ▂ 
 (449.52 ns ... 632.54 ns) 609.34 ns ███ ▃▅▆█▇ 
 ( 0.00 b ... 24.00 kb) 1.61 kb ▆████▅▄██████▅▅▅█▅▄▂▂
Array.from(512) 1.29 μs/iter 1.30 μs ▂▆█ 
 (1.27 μs ... 1.48 μs) 1.40 μs ▂███▇▆▃▃▂▁▁▂▁▁▁▁▁▁▁▁▁
 gc(457.25 μs ... 760.54 μs) 512.32 b ( 0.00 b... 84.00 kb)

universal compatibility

how to use mitata with engine CLIs like d8, jsc, graaljs, spidermonkey

$ xs bench.mjs
$ quickjs bench.mjs
$ d8 --expose-gc bench.mjs
$ spidermonkey -m bench.mjs
$ graaljs --js.timer-resolution=1 bench.mjs
$ /System/Library/Frameworks/JavaScriptCore.framework/Versions/Current/Helpers/jsc bench.mjs

// bench.mjs
import { print } from './src/lib.mjs';
import { run, bench } from './src/main.mjs'; // git clone
import { run, bench } from './node_modules/mitata/src/main.mjs'; // npm install
print('hello world'); // works on every engine

adding arguments and parameters to your benchmarks has never been so easy

import { bench } from 'mitata';
bench(function* look_mom_no_spaghetti(state) {
 const len = state.get('len');
 const len2 = state.get('len2');
 yield () => new Array(len * len2);
})
.args('len', [1, 2, 3])
.range('len', 1, 1024) // 1, 8, 64, 512...
.dense_range('len', 1, 100) // 1, 2, 3 ... 99, 100
.args({ len: [1, 2, 3], len2: ['4', '5', '6'] }) // every possible combination

computed parameters

bench('deleting $keys from object', function* (state) {
 const keys = state.get('keys');
 const obj = {};
 for (let i = 0; i < keys; i++) obj[i] = i;
 yield {
 [0]() {
 return { ...obj };
 },
 bench(p0) {
 for (let i = 0; i < keys; i++) delete p0[i];
 },
 };
}).args('keys', [1, 10, 100]);

concurrency

bench('sleepAsync(1000) x $concurrency', function* () {
 // concurrency inherited from arguments
 yield async () => await sleepAsync(1000);
}).args('concurrency', [1, 5, 10]);
bench('sleepAsync(1000) x 5', function* () {
 yield {
 // concurrency is set manually
 concurrency: 5,
 async bench() {
 await sleepAsync(1000);
 },
 };
});

hardware counters

------------------------------------------- -------------------------------
new Array(1024) 332.67 ns/iter 337.90 ns █ 
 (295.63 ns ... 507.93 ns) 455.66 ns ▂██▇▄▂▂▂▁▂▁▃▃▃▂▂▁▁▁▁▁
 2.41 ipc ( 48.66% stalls) 37.89% L1 data cache
 1.11k cycles 2.69k instructions 33.09% retired LD/ST ( 888.96)
new URL(google.com) 246.40 ns/iter 245.10 ns █▃ 
 (206.01 ns ... 841.23 ns) 302.39 ns ▁▁▁▁▂███▇▃▂▂▂▂▂▂▂▁▁▁▁
 4.12 ipc ( 1.05% stalls) 98.88% L1 data cache
 856.49 cycles 3.53k instructions 28.65% retired LD/ST ( 1.01k)

helpful warnings

-------------------------------------- -------------------------------
1 + 1 318.63 ps/iter 325.37 ps ▇ █ !
 (267.92 ps ... 14.28 ns) 382.81 ps ▁▁▁▁▁▁▁█▁▁█▁▁▁▁▁▁▁▁▁▁
empty function 319.36 ps/iter 325.37 ps █ ▅ !
 (248.62 ps ... 46.61 ns) 382.81 ps ▁▁▁▁▁▁▃▁▁█▁█▇▁▁▁▁▁▁▁▁
! = benchmark was likely optimized out (dead code elimination)

powerful visualizations right in your terminal

import { summary, barplot, boxplot, lineplot } from 'mitata';
// wrap bench() calls in visualization scope
barplot(() => {
 bench(...)
});
 ┌ ┐
 1 + 1 ┤■しかく 318.11 ps 
 Date.now() ┤■しかく■しかく■しかく■しかく■しかく■しかく■しかく■しかく■しかく■しかく■しかく■しかく■しかく■しかく■しかく■しかく■しかく■しかく■しかく■しかく■しかく■しかく■しかく■しかく■しかく■しかく■しかく■しかく■しかく■しかく■しかく■しかく■しかく■しかく 27.69 ns 
 └ ┘
// scopes can be async
await boxplot(async () => {
 // ...
});
 ┌ ┐
 ╷┌─┬─┐ ╷
 Bubble Sort ├┤ │ ├───────────────────────┤
 ╵└─┴─┘ ╵
 ┬ ╷
 Quick Sort │───┤
 ┴ ╵
 ┬
 Native Sort │
 ┴
 └ ┘
 90.88 μs 2.43 ms 4.77 ms
// can combine multiple visualizations
lineplot(() => {
 summary(() => {
 // ...
 });
 // bench() calls here wont be part of summary
});
summary
 new Array($len)
 5.42...8.33x faster than Array.from($len)
 ┌ ┐
 Array.from($size) ⢠⠊
 new Array($size) ⢀⠔⠁ 
 ⡠⠃ 
 ⢀⠎ 
 ⡔⠁ 
 ⡠⠊ 
 ⢀⠜ 
 ⡠⠃ 
 ⡔⠁ 
 ⢀⠎ 
 ⡠⠃ 
 ⢀⠜ 
 ⢠⠊ ⣀⣀⠤⠤⠒
 ⡰⠁ ⣀⡠⠤⠔⠒⠊⠉ 
 ⣀⣀⣀⠤⠜ ⣀⡠⠤⠒⠊⠉ 
 ⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣤⣔⣒⣒⣊⣉⠭⠤⠤⠤⠤⠤⠒⠊⠉ 
 └ ┘

give your own code power of mitata

#include "src/mitata.hpp"
int main() {
 auto stats = mitata::lib::fn([]() { /***/ })
}

import { B, measure } from 'mitata';
// lowest level for power users
const stats = await measure(function* (state) {
 const size = state.get('x');
 yield {
 [0]() {
 return size;
 },
 bench(size) {
 return new Array(size);
 },
 };
}, {
 args: { x: 1 },
 batch_samples: 5 * 1024,
 min_cpu_time: 1000 * 1e6,
});
// explore how magic happens
console.log(stats.debug) // -> jit optimized source code of benchmark
// higher level api that includes mitata's argument and range features
const b = new B('new Array($x)', function* (state) {
 const size = state.get('x');
 yield () => new Array(size);
}).args('x', [1, 5, 10]);
const trial = await b.run();

accuracy down to picoseconds

// node --expose-gc --allow-natives-syntax tools/compare.mjs
clk: ~2.71 GHz
cpu: Apple M2 Pro
runtime: node 23.3.0 (arm64-darwin)
benchmark avg (min ... max) p75 p99 (min ... top 1%)
------------------------------------------- -------------------------------
a / b 4.59 ns/iter 4.44 ns █ 
 (4.33 ns ... 25.86 ns) 6.91 ns ██▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
 6.70 ipc ( 2.17% stalls) NaN% L1 data cache
 16.80 cycles 112.52 instructions 0.00% retired LD/ST ( 0.00)
a / b (computed) 4.23 ns/iter 4.10 ns ▇█ 
 (3.88 ns ... 30.03 ns) 7.26 ns ██▅▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
 6.40 ipc ( 2.10% stalls) NaN% L1 data cache
 15.70 cycles 100.53 instructions 0.00% retired LD/ST ( 0.00)
4.59 ns/iter - https://npmjs.com/mitata
// vs other libraries
a / b x 90,954,882 ops/sec ±2.13% (92 runs sampled)
10.99 ns/iter - https://npmjs.com/benchmark
┌─────────┬───────────┬──────────────────────┬─────────────────────┬────────────────────────────┬───────────────────────────┬──────────┐
│ (index) │ Task name │ Latency average (ns) │ Latency median (ns) │ Throughput average (ops/s) │ Throughput median (ops/s) │ Samples │
├─────────┼───────────┼──────────────────────┼─────────────────────┼────────────────────────────┼───────────────────────────┼──────────┤
│ 0 │ 'a / b' │ '27.71 ± 0.09%' │ '41.00' │ '28239766 ± 0.01%' │ '24390243' │ 36092096 │
└─────────┴───────────┴──────────────────────┴─────────────────────┴────────────────────────────┴───────────────────────────┴──────────┘
27.71 ns/iter - vitest bench / https://npmjs.com/tinybench
a / b x 86,937,932 ops/sec (11 runs sampled) v8-never-optimize=true min..max=(11.32ns...11.62ns)
11.51 ns/iter - https://npmjs.com/bench-node
╔══════════════╤═════════╤════════════════════╤═══════════╗
║ Slower tests │ Samples │ Result │ Tolerance ║
╟──────────────┼─────────┼────────────────────┼───────────╢
║ Fastest test │ Samples │ Result │ Tolerance ║
╟──────────────┼─────────┼────────────────────┼───────────╢
║ a / b │ 10000 │ 14449822.99 op/sec │ ± 4.04 % ║
╚══════════════╧═════════╧════════════════════╧═══════════╝
69.20 ns/iter - https://npmjs.com/cronometro

writing good benchmarks

examples

dead code elimination

import { do_not_optimize } from 'mitata';
bench(function* () {
 // ❌ Bad: jit can see that function has zero side-effects
 yield () => new Array(0);
 // will get optimized to:
 /*
 yield () => {};
 */
 // ✅ Good: do_not_optimize(value) emits code that causes side-effects
 yield () => do_not_optimize(new Array(0));
});

garbage collection pressure

// ❌ Bad: unpredictable gc pauses
bench(() => {
 const bigArray = new Array(1000000);
});
// ✅ Good: gc before each (batch-)iteration
bench(() => {
 const bigArray = new Array(1000000);
}).gc('inner'); // run gc before each iteration

loop invariant code motion optimization

bench(function* (ctx) {
 const str = 'abc';
 // ❌ Bad: JIT sees that both str and 'c' search value are constants/comptime-known
 yield () => str.includes('c');
 // will get optimized to:
 /*
 yield () => true;
 */
 // ❌ Bad: JIT sees that computation doesn't depend on anything inside loop
 const substr = ctx.get('substr');
 yield () => str.includes(substr);
 // will get optimized to:
 /*
 const 0ドル = str.includes(substr);
 yield () => 0ドル;
 */
 // ✅ Good: using computed parameters prevents jit from performing any loop optimizations
 yield {
 [0]() {
 return str;
 },
 [1]() {
 return substr;
 },
 bench(str, substr) {
 return do_not_optimize(str.includes(substr));
 },
 };
}).args('substr', ['c']);

License