Glossary

中文版：glossary_CN.md

A glossary of domain-specific terminology used throughout rust-bench. Benchmarks appear simple on the surface but carry surprising complexity, so a consistent vocabulary avoids confusion.

This glossary covers runtime benchmarks and the distributed benchmarking system only. Compile-time benchmark concepts (profile/scenario/codegen-backend inherited from upstream rustc-perf) are out of scope for this documentation set.

Common terms

• metric: the name of a quantifiable quantity being measured (e.g. instructions:u, wall-time, max-rss).
• artifact: a specific build of the software under test, labelled by an identifier. For Rust toolchain artifacts the name is usually a commit SHA or a tag like 1.80.0; for non-Rust repos it is the commit SHA or release tag built by the repo's build_cmd. An artifact is uniquely identified by the triple (name, tag, repo) — the same commit benchmarked on different machines is a different artifact.
• benchmark suite: the entire collection of benchmarks, here the runtime benchmark suite under collector/runtime-benchmarks/.

Runtime benchmark terms

• benchmark: a single function whose execution is measured by rustc-compiled code. Reported by a benchmark binary built from a benchmark group crate via the benchlib message protocol.
• benchmark group: a crate under collector/runtime-benchmarks/<group>/ that contains a set of runtime benchmarks. Each group is compiled into one binary that communicates with the collector over stdout (line-delimited JSON BenchmarkMessage::Result frames). See collector/src/runtime/benchmark.rs.
• benchmark name: the string identity of a benchmark. For #[bench]-annotated functions the name is module_path!()::fn_name (e.g. std_bench::hash::map::find_existing), registered automatically at startup via the ctor attribute. Custom benchmarks report arbitrary names. The collector stores this name verbatim in the database.

Platforms and repositories

• RepoPlatform: the forge hosting a repository. Serialized lowercase: github or gitcode (collector/src/lib.rs:36). Determines webhook headers, API base URLs, auth strategy, and PR/MR URL paths.
• repo identifier: the full <platform>/<owner>/<repo> string (e.g. gitcode/xuanwu/rust, github/rust-lang/rust). Used for platform detection and webhook routing. Parsed from [repos.<key>].repo in site-config.toml.
• repo key: the [repos.<key>] table name in site-config.toml (e.g. rust, daft). This is what the database stores to identify a repository; it never contains /. Webhook payloads carrying a repo identifier are mapped back to a repo key via Config::repo_key_for_repo (site/src/load.rs:268).
• default repo key: the repo key used as fallback when none is specified. Defaults to rust; if absent, the first configured repo key (site/src/load.rs:240).

Testing

• test case: a combination of parameters identifying the measurement of a single benchmark. For runtime benchmarks a test case is the pair (benchmark, target) where target is the compilation target triple (e.g. x86_64-unknown-linux-gnu, aarch64-unknown-linux-gnu). See database::RuntimeTestCase.
• test: the act of running an artifact under a test case. Each test is composed of several iterations.
• test iteration: a single execution that makes up a test. The default is 5 measured iterations (preceded by 3 warm-up iterations) per benchmark (collector/src/runtime/mod.rs:38, benchlib/src/benchmark.rs:58-83).
• test result: the set of all gathered statistics from running a test. The minimum statistic value across iterations is used for analysis and the website.
• statistic: a single measured value of a metric in a test iteration.
• statistic description: the combination of a metric and a test case — it describes a statistic.
• statistic series: statistics for the same statistic description over time.
• run: a set of tests for all currently available test cases measured on a given artifact.

Analysis

• artifact comparisons: the comparison of two artifacts, composed of many test result comparisons. Shown on the comparison page (/compare.html) and in PR/MR completion comments.
• test result comparison: the relative change between two test results for the same test case but different artifacts, computed as 100 * (b - a) / a. See comparison-analysis.md.
• significance threshold: the value above which a test result comparison is considered an outlier against historical data — i.e. a real change rather than noise. Computed as Q3 + (Q3 - Q1) * 3.0 (the upper IQR fence; IQR_MULTIPLIER = 3.0), from the sorted percent-changes of deltas over the previous 30 commits (site/src/comparison.rs:1137-1175).
• significant test result comparison: a comparison whose absolute relative change is at or above the significance threshold.
• relevant test result comparison: a comparison that is significant and whose magnitude is Small or above. Relevance filters out statistically-significant but tiny changes (site/src/comparison.rs:1280-1283).
• test result comparison magnitude: how "large" a change is, bucketed VerySmall/Small/Medium/Large/VeryLarge. It is the average of two factors: how far the change exceeds the significance threshold, and the absolute percentage size of the change (metric-dependent; noisy metrics use a 2× factor) (site/src/comparison.rs:1289-1334).

Job queue

These terms relate to the distributed job queue that distributes benchmarking work across collectors.

• benchmark request: a request to benchmark a run on a given artifact. Comes in three types — Master (a merged commit on the default branch), Try (a PR/MR head, optionally with a parent), Release (a published tag). Stored in the benchmark_request table; created from webhooks or the seed_master_commits startup pass.
• collector: a machine that performs benchmarks. Each collector has a unique name (its collector tag) registered in the collector_config table.
• collector tag: the name identifying a collector (e.g. Kunpeng 920B, AMD EPYC 9654). Listed under [collectors].tags in site-config.toml and must match a collector_config.name. Drives job fan-out: the site creates one job per (benchmark_group, collector_tag).
• benchmark group (request): a comma-separated subset name selected for a benchmark request (e.g. alloc, daft). When a request specifies no groups, the repo's configured benchmark_groups are used. Each job targets exactly one benchmark group.
• job: a high-level work item defining a set of test cases to benchmark on a specific collector tag. Stored in the job_queue table; uniquely keyed by (request_tag, benchmark_group, tag).
• job queue: the queue of jobs awaiting execution, polled by collectors every 30 s.
• runtime config (QueueRuntimeConfig): the serialized payload attached to each job, carrying the benchmark group, repo key, build command/dir, per-collector overrides (iterations, no_isolate, filters), and extra_args. See database/src/lib.rs:1627.

Other

• self-profile: per-query/per-function timing data gathered with rustc's -Zself-profile. Stored out-of-database (local directory or S3) and served through a site endpoint. Out of scope for the runtime-benchmark guides.