Architecture

中文版：architecture_CN.md

A high-level view of the rust-bench system: its components, how they communicate, the multi-platform/multi-repository model, and the lifecycle of a benchmark request. This is the reference layer (factual/structural). For step-by-step runbooks see the guides.

1. Overview

rust-bench is a runtime performance benchmarking and profiling framework for Rust programs, built on rustc-perf and extended for runtime benchmarks, custom metrics, a TUI comparison interface, and multi-platform/multi-repository support.

The system measures how fast Rust programs execute when compiled with a given rustc, using hardware performance counters (perf_event_open), wall-clock timing, and memory high-water-mark measurement. It supports both local benchmarking (single machine, results in a SQLite DB) and an automated, distributed flow in which a central Site receives webhooks, a set of Collectors pull jobs from a queue and execute them, and the Site posts comparison comments back to the originating PR/MR.

Compile-time benchmark machinery inherited from upstream rustc-perf exists in the codebase but is out of scope for this documentation set.

2. Components

                 ┌──────────────────────────────────────────────┐
  GitCode/GitHub │                     Site                      │
   webhooks ───► │  ┌──────────┐ ┌──────────┐ ┌───────────────┐  │
                 │  │ HTTP API │ │ Job queue│ │ Platform API  │  │
                 │  │ /server │ │ /tick    │ │ GitHub/GitCode│  │
                 │  └──────────┘ └────┬─────┘ └───────────────┘  │
                 └────────────────────┼──────────────────────────┘
                                      │ Postgres job_queue
                                      ▼
   ┌──────────────────────────────────────────────────────────────┐
   │                        Collector(s)                          │
   │  collect-job-queue.sh loop → benchmark_job_queue             │
   │  ┌────────────────┐  ┌────────────────────────────────────┐│
   │  │ toolchain build│  │ benchlib message protocol (stdout)  ││
   │  │ /artifact fetch│  │ ← BenchmarkMessage::Result (JSON)   ││
   │  └────────────────┘  └────────────────────────────────────┘│
   └──────────────────────────────┬──────────────────────────────┘
                                  │ results
                                  ▼
                          ┌────────────────┐
                          │  Database      │  Postgres (distributed)
                          │  (Postgres /   │  SQLite (local only)
                          │   SQLite)      │
                          └────────────────┘

Site (site/)

The web server and job-queue orchestrator. Entry point site/src/main.rs:16:

• Loads config from site-config.toml (or env vars) and the DB into a SiteCtxt (site/src/load.rs:383).
• Starts an HTTP server (site/src/server.rs) on port 2346 (env PORT) serving the API, frontend, and the webhook endpoint at /perf/webhook (server.rs:343).
• Spawns a periodic job-queue tick every QUEUE_UPDATE_INTERVAL_SECONDS (default 30 s) that creates requests, enqueues jobs, and posts completion comments (site/src/job_queue/mod.rs:419, main.rs:80).
• Holds one PlatformApi client per configured repo key (load.rs:401) for outbound GitHub/GitCode calls (posting comments, fetching PR/commit data, collaborator checks).
• On startup, seeds benchmark_request rows for the last 30 days of merge commits per repo (load.rs:541-613, invoked at main.rs:72).

Collector (collector/)

The benchmark executor. Binary collector (collector/src/bin/collector.rs), library collector/src/lib.rs:

• Two main modes: bench_runtime_local (single-shot local run into a DB) and benchmark_job_queue (daemon loop pulling from Postgres).
• Discovers and compiles runtime benchmark group crates (collector/src/runtime/benchmark.rs), runs each benchmark binary, and reads line-delimited JSON BenchmarkMessage::Result frames from stdout (collector/src/runtime/mod.rs:124-157).
• Writes per-iteration numeric samples to runtime_pstat and per-benchmark JSON reports to runtime_pstat.json_value (mod.rs:232-274).
• Built via collect-job-queue.sh (production loop) with --features s3-sdk for self-profile S3 storage.

Database (database/)

Two interchangeable backends behind a Connection/Pool trait (database/src/lib.rs, database/src/pool/):

• Postgres (pool/postgres.rs) — required for the distributed job queue (multi-collector).
• SQLite (pool/sqlite.rs) — local single-machine use only; does not support benchmark_job_queue.

The schema is duplicated across both backends via parallel migrations. See database/schema.md.

benchlib (collector/benchlib/)

The measurement library linked into every runtime benchmark binary (collector/benchlib/src/lib.rs):

• Defines the BenchmarkMessage/BenchmarkResult/BenchmarkSample protocol (benchlib/src/comm/messages.rs) and well-known METRIC_* constants.
• Measures hardware counters via the perf-event crate (perf_event_open(2) syscall), wall time via Instant, and max-rss via /usr/bin/time. Linux-only.
• Provides the #[bench] proc-macro (collector/benchlib-macros/) and a custom framework (benchlib/src/custom.rs) for benchmarks needing full control over measurement/CLI args.

3. Repositories and platforms

rust-bench benchmarks more than one repository, hosted on more than one forge.

Platforms

RepoPlatform (collector/src/lib.rs:36) is Github or Gitcode, serialized lowercase and used as the prefix of repo identifiers like github/rust-lang/rust or gitcode/xuanwu/rust. The platform determines:

• Webhook headers: X-GitHub-Event vs X-GitCode-Event (event names differ too: GitHub issue_comment/push; GitCode Note Hook/Push Hook/Tag Push Hook) — site/src/api.rs:922-945.
• Signature header: X-Hub-Signature-256 (GitHub) vs X-GitCode-Signature-256 (GitCode) — api.rs:947-952.
• API base / web base / PR-MR URL path — site/src/load.rs:93-122.
• Auth strategy: GitHub uses HTTP Basic (rust-bench + token); GitCode uses Bearer token — site/src/github/platform_client.rs:151-164.

Repositories

Each repo is a [repos.<key>] table in site-config.toml (site/src/load.rs:55). A repo carries:

• repo — the platform-prefixed identifier (<platform>/<owner>/<repo>).
• default_branch, benchmark_groups (default groups to run), build_cmd/build_dir (for non-Rust repos), is_rust, post_build.

The database stores the repo key (e.g. rust, daft), never the full identifier. An incoming webhook's repository path is mapped back to a repo key via Config::repo_key_for_repo (load.rs:268); an unknown repo returns 403 FORBIDDEN (site/src/request_handlers/webhook.rs:64-76).

A repo may be non-Rust (is_rust = false, e.g. the Daft dataframe library): the collector runs build_cmd to produce artifacts, then post_build to install them, before benchmarking. The QueueRuntimeConfig payload on each job carries the per-repo build instructions to the collector (database/src/lib.rs:1627).

4. Request lifecycle

A benchmark request moves through the following states (database/src/lib.rs:1134; DB strings are snake_case waiting_for_artifacts / artifacts_ready / in_progress / completed):

flowchart LR
    W[waiting_for_artifacts] -->|try build artifacts appear| A
    A[artifacts_ready] -->|cron enqueues jobs| I
    I[in_progress] -->|all jobs done, no failures| C[completed]
    I -->|job failures| C

1. Creation — a webhook arrives at /perf/webhook (server.rs:343). After HMAC-SHA256 verification against [keys.<platform>].secret (webhook.rs:5-23), the platform and event type are detected (api.rs:922-945):
   • push to the default branch → one Master request per merge commit (github.rs:206-273).
   • Tag Push Hook (GitCode) or a push whose ref is refs/tags/... (GitHub) → one Release request (github.rs:165-204, platform_client.rs:244-254).
   • issue_comment / Note Hook containing /rust-bench try|master|tag → a Try/Master/Release request (github.rs:596-863). The author must be a collaborator of the upstream repo (github.rs:551-582).
2. Queueing — the periodic tick (job_queue/mod.rs:419) orders artifacts_ready/in_progress requests topologically by parent completion (mod.rs:80-168), keeps at most one request in_progress, and enqueues one job per (benchmark_group, collector_tag) for the chosen request and its parent (the latter is backfilling) — mod.rs:222-362.
3. Execution — a collector dequeues a matching job (job_queue poll every 30 s), builds/fetches the artifact per the QueueRuntimeConfig, compiles and runs the benchmark group, and writes results to the DB. See job-queue.md and the deploying-collector guide.
4. Completion — the tick detects all (non-optional) jobs of the in_progress request finished, marks it completed, and for Try/Master requests posts a comparison comment to the PR/MR with per-collector-tag and cross-machine comparison links (job_queue/mod.rs:538-639, site/src/github/comparison_summary.rs:26-134). Release requests get no comment.

5. Data model

A condensed view; full column definitions live in database/schema.md.

• artifact — uniquely keyed by (name, tag, repo). name is a SHA or tag; tag is the collector tag (default default); repo is the repo key. The same commit on two machines is two rows.
• runtime_pstat_series — a (benchmark, target, metric) triple; runtime_pstat holds the per-(series, artifact, collection) measured value (numeric) and optional json_value (structured reports, e.g. perf-record entries).
• collector_config — (name unique, is_active, last_heartbeat_at, date_added, commit_sha). name is the collector tag.
• job_queue — (request_tag, benchmark_group, tag) unique; carries collector_name, status, retry, is_optional, and the serialized runtime_config. Status strings: queued/in_progress/success/failure.
• benchmark_request — (tag, commit_type, pr, repo, status, benchmark_groups, ...). commit_type is master/try/release.

Cross-references

• How to run benchmarks locally → local-benchmarking guide
• How to wire up GitCode/GitHub → platform-integration guide
• Config file reference → configuration.md
• Queue mechanics, retries, backfilling → job-queue.md
• Deployment model → deployment.md
• Comparison algorithm → comparison-analysis.md