Rust Bench — Rust Performance Benchmarking & Profiling Framework
A comprehensive toolkit for runtime performance benchmarking, data collection, terminal visualization, and profiling of Rust programs. Built on top of rustc-perf, extended with enhanced runtime benchmark capabilities, custom metric support, and a TUI comparison interface.
Core Features
- ✅ Runtime Benchmark — Full Rust standard library benchmark suite and custom benchmarks, powered by hardware performance counters (
perf_event) - ✅ Terminal Data Display — Real-time min/mean/stddev statistics output during benchmarking, plus an interactive TUI (
bench_cmp) for comparing results between two artifact versions - ✅ Profile — Runtime profiling with Cachegrind (instruction-level) and perf-record (sampling-based)
Table of Contents
Quick Start
Prerequisites
- Rust stable toolchain
- Linux (required for
perf_eventhardware counters) - Set perf event permissions:
sudo bash -c 'echo -1 > /proc/sys/kernel/perf_event_paranoid'
Build
cargo build --release
Run Runtime Benchmarks
# Run all runtime benchmarks
./target/release/collector bench_runtime_local <RUSTC>
# Run a specific benchmark group only
./target/release/collector bench_runtime_local <RUSTC> --group std
# Filter benchmarks by name prefix
./target/release/collector bench_runtime_local <RUSTC> --include find_existing
./target/release/collector bench_runtime_local <RUSTC> --exclude hashmap
# Reuse compiled artifacts for faster repeated runs
./target/release/collector bench_runtime_local <RUSTC> --no-isolate
# Specify database and iterations
./target/release/collector bench_runtime_local <RUSTC> --db my_results.db --iterations 10
Where <RUSTC> is a path to a rustc executable (e.g. path/to/stage1/bin/rustc) or a +-prefixed toolchain specifier (e.g. +nightly).
Common Options (bench_runtime_local)
| Option | Description |
|---|---|
--id <ID> |
Identifier for the benchmark results |
--db <DATABASE> |
Database file path (default: results.db) |
--iterations <N> |
Number of iterations per benchmark (default: 5) |
--include <PREFIX> |
Include only benchmarks matching this prefix |
--exclude <PREFIX> |
Exclude benchmarks matching this prefix |
--group <GROUP> |
Only compile and run the specified benchmark group |
--no-isolate |
Reuse previously compiled benchmark artifacts |
--purge |
Remove old data for the artifact before benchmarking |
Compare Results
# Interactive TUI comparison (select artifacts interactively)
./target/release/collector bench_cmp --db results.db
Run Profiling
# Cachegrind profiling (all benchmarks in a group)
./target/release/collector profile_runtime <RUSTC> cachegrind
# Profile a specific group
./target/release/collector profile_runtime <RUSTC> cachegrind --group std
# perf-record profiling
./target/release/collector profile_runtime <RUSTC> perf-record
# Compare two rustc versions (cachegrind diff)
./target/release/collector profile_runtime <RUSTC> cachegrind --rustc2 <RUSTC2>
# Precise Cachegrind mode (requires Valgrind ≥ 3.22)
DEP_VALGRIND=<valgrind-install>/include cargo run --release --bin collector \
--features precise-cachegrind profile_runtime <RUSTC> cachegrind
Features
3.1 Runtime Benchmark
The runtime benchmark suite measures how fast Rust programs execute when compiled with a specific version of rustc. Benchmarks are organized into groups — each group is a separate crate in collector/runtime-benchmarks/.
Standard Library Benchmark Groups
| Group | Content |
|---|---|
std |
HashMap, HashSet, IO (BufRead/Copy/Cursor/Impls), Path, Time |
core |
Core library benchmarks |
alloc |
Memory allocator benchmarks |
Custom Benchmark Groups
| Group | Content |
|---|---|
fmt |
Formatting benchmarks |
hashmap |
Hash map performance |
compression |
Compression algorithms |
css |
CSS parsing |
nbody |
N-body simulation |
nes |
NES emulator |
parsing |
Text parsing |
raytracer |
Ray tracing |
svg |
SVG processing |
text-search |
Text search |
bufreader |
Buffered reading |
daft-vllm |
Custom benchmark example (with custom metrics, JSON reports, extra CLI args) |
Collected Metrics
The benchmarking framework uses Linux perf_event_open system calls to gather hardware performance counters:
| Metric | Description |
|---|---|
instructions:u |
CPU instruction count |
cycles:u |
CPU cycle count |
wall-time |
Wall-clock time |
branch-misses |
Branch prediction misses |
cache-misses |
Cache misses |
cache-references |
Cache references |
max-rss |
Maximum resident set size (memory) |
| Custom metrics | Any user-defined numeric metrics |
Each benchmark is executed in two passes: one for hardware performance counters and one for wall-clock time measurement, ensuring accurate and isolated results.
3.2 Terminal Data Display
Real-time Benchmark Output
After each benchmark completes, statistics are printed to the terminal:
Finished std/find_existing (1/42)
[Instructions]: min: 1,234,567 mean: 1,234,890 stddev: 234
[Cycles]: min: 456,789 mean: 457,012 stddev: 156
[Wall time [ns]]: min: 234,000 mean: 235,000 stddev: 2,000
[Branch misses]: min: 1,234 mean: 1,245 stddev: 12
[Cache misses]: min: 56 mean: 58 stddev: 3
[Memory [kb]]: min: 2,048 mean: 2,048 stddev: 0
- Custom numeric metrics (e.g.
throughput_tok/s,latency_p99_ms) are also displayed automatically - JSON reports are shown as
<JSON data> (view in bench_cmp)markers - Progress tracking:
Finished <group>/<benchmark> (<current>/<total>)
bench_cmp — Interactive TUI Comparison
The bench_cmp command provides a terminal-based interactive UI (powered by ratatui) for comparing benchmark results between two artifact versions:
./target/release/collector bench_cmp --db results.db
Features:
- Mode switching — Press
Mto switch between Compile and Runtime comparison modes - Summary panel — Regression/improvement summary statistics at the top
- Detailed table — Benchmark name, Before value, After value, Change with 95% confidence interval
- Metric cycling — Press
A/Sto cycle through available metrics (instructions, cycles, wall-time, etc.) - Significance filter — Press
Fto toggle between showing all results or significant changes only - Target switching — Press
1/2to switch base/modified target platforms - JSON detail view — Press
Enteron a JSON row (e.g.perf-recorddata) to open a detail view with scrollable/tabular display - Navigation —
↑/↓to navigate,q/Escto quit
3.3 Profile
The profile_runtime subcommand profiles runtime benchmarks with one of two profilers:
Cachegrind
- Purpose: Instruction-level profiling with nearly deterministic results
- Output: Raw data (
cgoutprefix) + human-readable annotated output (cgannprefix) - Diff support: When using
--rustc2, generates diff files comparing two versions - Precise mode: With
--features precise-cachegrind, only the actual benchmark code is instrumented (requires Valgrind ≥ 3.22)
perf-record
- Purpose: Sampling-based profiling, great for finding hot functions
- Output: Raw data (
perfprefix) + annotated report (perfreportprefix) - Slowdown: Negligible
Profile Options
| Option | Description |
|---|---|
<PROFILER> |
cachegrind or perf-record |
<RUSTC> |
Path to rustc executable or +toolchain specifier |
--group <GROUP> |
Profile only a specific benchmark group |
--iterations <N> |
Number of profiling iterations (default: 5) |
--include <PREFIX> |
Include benchmarks by prefix |
--exclude <PREFIX> |
Exclude benchmarks by prefix |
--rustc2 <RUSTC> |
Second rustc for comparison profiling (cachegrind supports diff) |
Writing Benchmarks
4.1 Using the #[bench] Macro
The recommended way to write benchmarks for testing Rust function performance. The #[bench] macro automatically registers your function with the global benchmark group.
Step 1: Create or choose a benchmark group
Each benchmark group is a binary crate in collector/runtime-benchmarks/. By convention, if the group directory is called foo, the crate name should be foo-bench.
Step 2: Add benchlib dependency
[dependencies]
benchlib = { path = "../../benchlib" }
Step 3: Write benchmarks with #[bench]
use std::collections::HashMap;
use benchlib::benchmark::{bench, Bencher};
#[bench]
fn find_existing(b: &mut Bencher) {
let mut m = HashMap::new();
for i in 1..1001 {
m.insert(i, i);
}
b.iter(
|| {}, // constructor: prepare per-iteration data (NOT measured)
|_| { // bench: the code that IS measured
for i in 1..1001 {
m.contains_key(&i);
}
},
);
}
#[bench]
fn grow_by_insertion(b: &mut Bencher) {
let mut m = HashMap::new();
for i in 1..1001 {
m.insert(i, i);
}
let mut k = 1001;
b.iter(
|| {},
|_| {
m.insert(k, k);
k += 1;
},
);
}
Step 4: Set up main.rs
fn main() {
benchlib::benchmark::run_global_benchmark_group();
}
How Bencher::iter(constructor, bench) Works
constructor: Called before each measurement to prepare input data. Its return value is passed tobench. This code is NOT measured.bench: Receives the output ofconstructor. This is the code that IS measured with hardware performance counters and wall-clock time.- The system automatically performs 3 warm-up iterations followed by N measured iterations (configurable via
--iterations).
4.2 Custom Benchmark (daft-vllm Example)
For scenarios where you need full control over measurement logic, custom metrics, JSON reports, or custom CLI arguments, use the benchlib::custom API.
Core APIs
| API | Purpose |
|---|---|
benchlib::custom::init(&[names]) |
Initialize CLI, parse args, return BenchAction |
benchlib::custom::parse_extra_args(&extra) |
Parse custom args passed after -- |
BenchmarkSample::new(duration) |
Create a sample with wall-clock time |
sample.set("metric", value) |
Set a custom numeric metric |
BenchmarkResult::with_samples(name, samples) |
Create a result from samples |
result.add_report("name", &data) |
Add a JSON report (any Serialize type) |
result.add_report(METRIC_PERF_RECORD, &entries) |
Add perf-record profiling data |
output_message(&mut stdout, msg) |
Output the result as JSON to stdout |
Complete Example (from daft-vllm/src/main.rs)
use benchlib::clap;
use benchlib::comm::messages::{
BenchmarkMessage, BenchmarkResult, BenchmarkSample,
PerfRecordEntry, METRIC_PERF_RECORD,
};
use benchlib::comm::output_message;
use benchlib::custom::{init, parse_extra_args, BenchAction};
use std::time::Duration;
/// Custom arguments parsed from `-- ...` on the command line.
#[derive(clap::Parser, Debug)]
struct VllmArgs {
#[arg(long, default_value = "default-model")]
model: String,
#[arg(long, default_value_t = 16)]
batch_size: u32,
}
fn main() -> benchlib::anyhow::Result<()> {
let action = init(&["daft-vllm-example"])?;
match action {
BenchAction::Run { iterations, benchmarks, extra } => {
let vllm_args: VllmArgs = parse_extra_args(&extra)?;
let mut stdout = std::io::stdout().lock();
for name in &benchmarks {
let mut samples = Vec::with_capacity(iterations as usize);
for _ in 0..iterations {
// Your benchmark logic here
let mut sample = BenchmarkSample::new(Duration::from_millis(0));
// Add custom numeric metrics
sample.set("throughput_tok/s", 1234.5);
sample.set("latency_p99_ms", 42.0);
samples.push(sample);
}
let mut result = BenchmarkResult::with_samples(name.clone(), samples);
// Add a structured JSON report
#[derive(serde::Serialize)]
struct LatencyDist { p50: f64, p90: f64, p99: f64 }
result.add_report("latency_distribution", &LatencyDist {
p50: 12.5, p90: 35.2, p99: 42.0,
});
// Add perf-record profiling data
let perf_entries = vec![
PerfRecordEntry {
function: "engine::execute_plan".to_string(),
delay: 15.3,
time_rate: 0.35,
sample_count: 350,
is_rust: true,
category: "compute".to_string(),
shared_object: "libdaft.so".to_string(),
},
];
result.add_report(METRIC_PERF_RECORD, &perf_entries);
output_message(&mut stdout, BenchmarkMessage::Result(result))?;
}
}
BenchAction::Profile(profile_args) => {
let vllm_args: VllmArgs = parse_extra_args(&profile_args.extra)?;
for _ in 0..profile_args.iterations {
// Your profiling logic here
}
}
}
Ok(())
}
Running Custom Benchmarks
# Run with default arguments
./target/release/collector bench_runtime_local <RUSTC> --group daft-vllm
# Pass custom arguments after --
./target/release/collector bench_runtime_local <RUSTC> --group daft-vllm \
-- --model gpt2 --batch-size 32
Project Structure
rust-bench/
├── collector/ # Data collector
│ ├── benchlib/ # Core benchmarking library
│ │ ├── src/benchmark.rs # #[bench] macro + BenchmarkGroup + Bencher
│ │ ├── src/custom.rs # Custom benchmark framework (init/BenchAction)
│ │ ├── src/comm/messages.rs # Message protocol + metric definitions
│ │ └── src/measure/ # Performance counter measurement (perf_event)
│ ├── benchlib-macros/ # #[bench] proc-macro implementation
│ ├── runtime-benchmarks/ # Runtime benchmark suite
│ │ ├── std/ # Standard library benchmarks (HashMap, IO, Path, Time)
│ │ ├── core/ # Core library benchmarks
│ │ ├── alloc/ # Allocator benchmarks
│ │ ├── fmt/ # Formatting benchmarks
│ │ ├── hashmap/ # Hash map benchmarks
│ │ ├── daft-vllm/ # Custom benchmark example
│ │ └── ... # More benchmark groups
│ ├── compile-benchmarks/ # Compile-time benchmark suite
│ └── src/
│ ├── bin/collector.rs # CLI entry point
│ ├── runtime/mod.rs # Runtime benchmark execution engine
│ ├── runtime/profile.rs # Runtime profiling (cachegrind, perf-record)
│ └── compare/screen.rs # TUI comparison interface (bench_cmp)
├── database/ # Database layer (SQLite/Postgres)
├── site/ # Web frontend (optional)
├── Cargo.toml # Workspace configuration
└── README.md
License
The original rustc-perf code is licensed under the MIT license, managed by the Reuse Specification.
The compile-time benchmarks have their own separate licenses, check the collector/compile-benchmarks/REUSE.toml file.
Modifications and extensions by the Xuanwu Team are also licensed under the MIT License.
Copyright (c) Xuanwu Team. All rights reserved.