Benchmark🔗 i

1Example: Measuring the Impact of Racket’s JIT🔗 i

In the following example we evaluate the use of Racket’s JIT on two simple programs, running each with and without JIT, and then plot the results. The plot renderer groups together the results of running the same program under different options, distinguishing the different options using colors.

> (require benchmarkplot/pictracketracket/runtime-pathcompiler/find-exe)

> (define-runtime-path fib-path

"examples/macro-examples/fib.rkt")

> (define-runtime-path collatz-path

"examples/macro-examples/collatz.rkt")

> (define-runtime-path compiled-dir

"examples/macro-examples/compiled")

> (define results

(run-benchmarks

;files to run (whats)

(list fib-pathcollatz-path)

;list of options (hows)

(list (list 'jit'no-jit))

;how to run each benchmark

(lambda (file jit)

(if (equal? jit'jit)

(system* (find-exe)file )

(system* (find-exe)"-j"file )))

#:build

(lambda (file jit)

(system* (find-exe)"-l""raco""make"file ))

#:clean

(lambda (file jit)

(system* (find-executable-path "rm")"-r""-f"compiled-dir))

#:num-trials30

#:make-name(lambda (path)

(let-values

([(1file-name2)(split-path path)])

(path->string file-name)))))

> (parameterize ([plot-x-ticks no-ticks ])

(plot-pict

#:title"jit vs no jit"

#:x-label#f

#:y-label"normalized time"

(render-benchmark-alts

;default options

(list 'jit)

results)))

[画像:image]

"fib.rkt"

#lang racket

(define (fibn)(if (<= n1)n(+ (fib(- n1))(fib(- n2)))))

(time (fib26))

"collatz.rkt"

#lang racket

(define (collatzn)

(if (even? n)

(/ n2)

(+ (* 3n)1)))

(define (collatz-rangem)

(for-each (lambda (n)(collatzn))

(stream->list (in-range 0m))))

(time (collatz-range10000))

It’s important to note that both examples explicitly call time. This is because, by default, the time of a benchmark is not the time to evaluate (run... ), but rather the time reported to stdout by (run... ). Had we merely wanted to time how long it took to evaluate (run... ) we would have called run-benchmarks with #:extract-time'delta-time. Such an example will be given in the next section.

2Example: Vectors vs Lists🔗 i

Here we use 'delta-time to demonstrate measuring the time to evaluate (run... ). At a high level we will be evaluating lists vs vectors for map and append. Specifically, map and append (or the appropriate variant) will correspond to our whats, and the list/vector size and whether we are doing list or vector operations will corespond to our hows. That is, for each input size, for each of list and vector, we will evaluate map and append.

> (require benchmarkplot/pictracket/matchracket/vectorracket/list)

;list/vector sizes

> (define sizes(list 50000100000))

> (define lists(map (lambda (i)(range i))sizes))

> (define vectors(map list->vector lists))

> (define results

(run-benchmarks

;operations (whats)

(list 'square-map'self-append)

;list of options (hows)

(list

;sizes (and their indices) in the sizes list

(map cons (range (length sizes))sizes)

;implementations of operations

(list 'vector'list))

;to run each benchmark

(lambda (opindex/sizeimpl)

(let ([fn

(match (cons opimpl)

[(cons 'square-map'vector)

(lambda (i)(vector-map (lambda (x)(* xx))i))]

[(cons 'square-map'list)

(lambda (i)(map (lambda (x)(* xx))i))]

[(cons 'self-append'vector)

(lambda (i)(vector-append ii))]

[(cons 'self-append'list)

(lambda (i)(append ii))])]

[input(list-ref (match impl

['vectorvectors]

['listlists])

(car index/size))])

(fninput)))

;don't extract time, instead time (run ...)

#:extract-time'delta-time

#:num-trials30))

> (parameterize ([plot-x-ticks no-ticks ])

(plot-pict

#:title"vectors vs lists"

#:x-label#f

#:y-label"normalized time"

(render-benchmark-alts

;default options

(list (cons 050000)'list)

results

;format options so we can omit the index in the size list

#:format-opts(lambda (opts)

(let ([index/size(car opts)]

[impl(cadr opts)])

(format "size: ~a, ~a"(cdr index/size)impl))))))

[画像:image]

3Running Benchmarks🔗 i

build and clean are functions with the same argument types as run and are run before and after each num-trials runs of run.

When not 'delta-time, extract-time is applied to the contents of stdout after each run of run, and must produce a real? that is the run time of the test. In this case it is expected that each of the whats report their time via stdout when run. When 'delta-time, the time of the benchmark is considered to be the time it takes to evaluate (runargs... ). In this case, each of the whats do not need to report their own time to stdout.

make-name takes an element of whats and produces a human-readable representation of the benchmark’s name.

skip has the same argument types as run, and when it evaluates to true the associated benchmark/options combination is skipped.

If results-file is non-false, results will be automatically persisted to results-file-<n>. Incremental results are recorded after all runs with a given set of options are finished. For more information about persisting results, see Persisting Results.

Benchmarks are logically specified as an (n+1)-dimensional matrix where n of the dimensions are options (how to run) and one dimension is the names/files (what to run). The names/files dimension is specified in the whats list. The options dimensions are specified as lists in the hows list.

In Example: Measuring the Impact of Racket’s JIT, (list "fib.rkt""collatz.rkt") is the whats and (list (list 'jit'no-jit)) is the hows. That is, there is only one dimension of options for running (list 'jit'no-jit), and two things to run "fib.rkt" and "collatz.rkt".

4Benchmark results🔗 i

5Plotting🔗 i

benchmark exports a renderer2d? for plotting results of benchmarks using the plot library.

norm-opts is the list of options (from hows) of run that are the standard options.

For example, (list 'jit) is the list of standard options in Example: Measuring the Impact of Racket’s JIT.

benchmark-results are the results produced by run-benchmarks .

format-opts formats the list of options into strings for displaying on the legend of the plot.

If normalize? is #f, then the results will not be normalized.

6Persisting Results🔗 i

Results can be persisted with record-results and retrieved with get-past-results . Results can also be automatically persisted by run-benchmarks when passed a #:results-file argument.