Guide: run benchmarks locally

中文版：local-benchmarking_CN.md

Run runtime benchmarks on a single machine and inspect results in the terminal TUI. This is the local, single-machine workflow (SQLite). For the distributed multi-collector flow see the deploying-collector guide.

Prerequisites

Linux required — the benchmark framework uses Linux perf counters.
A Rust toolchain to benchmark: either a path to a rustc executable (e.g. path/to/stage1/bin/rustc) or a +-prefixed toolchain name (e.g. +nightly).
Allow userspace perf counters:
```
sudo bash -c 'echo -1 > /proc/sys/kernel/perf_event_paranoid'
```
Without this, the first benchmark fails with a message pointing at /proc/sys/kernel/perf_event_paranoid.
/usr/bin/time installed (used for max-rss measurement) and setarch (used to disable ASLR per process).

Step 1 — Build the collector

cargo build --release

The binary is target/release/collector.

Step 2 — Run all runtime benchmarks

# Using a toolchain name
./target/release/collector bench_runtime_local +nightly

# Using a rustc path
./target/release/collector bench_runtime_local path/to/stage1/bin/rustc

By default results go to results.db (SQLite) in the current directory; the artifact is identified by the toolchain name (or "Id" for a path — override with --id). You will see:

Using database `results.db`
Executing 124 benchmarks

Finished std/std_bench::hash::map::find_existing (1/124)
        Instructions: min:        1,234,567    mean:    1,240,000    stddev:        3,210
            Cycles: min:        2,000,000    mean:    2,010,000    stddev:        5,400
      Wall time [ns]: min:      500,000,000    mean:    503,000,000    stddev:      800,000
      Branch misses: min:           12,000    mean:       12,500    stddev:          150
        Cache misses: min:           45,000    mean:       46,000    stddev:          320
    Cache references: min:          120,000    mean:    121,000    stddev:          410
        Memory [kb]: min:           38,000    mean:       38,200    stddev:           90

The output prints per-benchmark min/mean/stddev for the well-known metrics in this order: Instructions, Cycles, Wall time [ns], Branch misses, Cache misses, Cache references, Memory [kb]. Custom numeric metrics and JSON reports print afterwards.

Benchmark names are module_path!()::fn_name (e.g. std_bench::hash::map::find_existing), registered by the #[bench] macro. Prefix-match filters operate on these full names.

Step 3 — Filter which benchmarks run

Filters are prefix matches on the full benchmark name (except --exact_match, which is exact equality). They are mutually exclusive — --exact_match conflicts with the others.

# Only one group
./target/release/collector bench_runtime_local +nightly --group std

# Include/exclude by name prefix
./target/release/collector bench_runtime_local +nightly --include std_bench::hash
./target/release/collector bench_runtime_local +nightly --exclude std_bench::io

# Exclude by suffix
./target/release/collector bench_runtime_local +nightly --exclude_suffix _slow

# Run only exact matches
./target/release/collector bench_runtime_local +nightly --exact_match std_bench::hash::map::find_existing

# Pass extra args to benchmark binaries (after --)
./target/release/collector bench_runtime_local +nightly -- --tpch-scale-factor 100

Step 4 — Control iterations, isolation, and the DB

Option	Default	Effect
`--iterations <N>`	`5`	Measured iterations per benchmark (3 warm-up iterations always precede).
`--no-isolate`	off	Compile benchmark crates in-place (reuse `target/`), instead of an isolated tempdir. Faster for repeat runs.
`--id <ID>`	toolchain name / `"Id"`	Identifier stored as the artifact's commit SHA.
`--tag <TAG>`	`default`	The collector tag stored on results. Use distinct tags to compare runs.
`--db <PATH>`	`results.db` (env `DATABASE_URL`)	SQLite (or Postgres) DB path/URL.
`--purge`	off	Remove existing data for the artifact before benchmarking.
`--cargo <PATH>`	`cargo`	Path to a `cargo` binary.
`--cargo_config <ARGS>`	—	Extra `cargo --config` args.

Step 5 — Compare two runs with `bench_cmp`

./target/release/collector bench_cmp --db results.db

This opens an interactive TUI. If you don't pass base/modified on the command line you pick them interactively.

TUI keys (lowercase)

Key	Action
`m`	Switch between Compile and Runtime comparison modes. The default is Compile; press `m` after a runtime-only run to see your data.
`a` / `s`	Cycle the displayed metric backward / forward.
`f`	Toggle showing only significant changes.
`1` / `!`	Cycle the base collector tag forward / backward.
`2` / `@`	Cycle the modified collector tag forward / backward.
`Enter`	Open a scrollable/tabular detail view for a JSON row (e.g. `perf-record` data) — Runtime mode only.
`↑` / `↓`	Navigate the table.
`q` / `Esc`	Quit (at the root screen); on a pushed detail screen, pops back instead.

Keys are matched lowercase — Shift+M will not switch modes. 1/2 switch tags (collector/machine identifiers), not target-platform triples.

Non-interactive comparison

./target/release/collector bench_cmp --db results.db <base> <modified> --metric wall-time

Troubleshooting

Cannot create perf_event group ... perf_event_paranoid — set perf_event_paranoid = -1 (see Prerequisites).
Empty Compile table in bench_cmp — you ran bench_runtime_local, so press m to switch to Runtime mode.
--include find_existing matches nothing — #[bench] names are module_path::fn_name; use the full prefix, e.g. --include std_bench::hash.
Artifacts collide between runs on the same machine — give each run a distinct --tag (e.g. --tag before / --tag after).
Re-running overwrites previous data — pass --purge, or use purge_artifact <name> --db results.db (see manual-modifications).

Next steps

Write your own benchmarks → writing-benchmarks guide
Profile a regression → profiling guide
Distributed execution → deploying-collector guide