Job queue

中文版：job-queue_CN.md

Read the glossary first, especially the job-queue terms.

Beyond single-machine local benchmarking, rust-bench can run as a distributed system: a single compiler/software artifact is benchmarked in parallel across several collectors (machines, possibly different architectures), coordinated through a Postgres database. This document describes that distributed system. For the step-by-step runbook see the deploying-collector guide.

High-level overview

Two entities cooperate (site/src/job_queue/mod.rs, site/src/main.rs:80):

The Site receives requests to benchmark a specific artifact, splits each into per-(benchmark_group, collector_tag) jobs, waits until all (non-optional) jobs complete, then posts a comparison comment on the originating PR/MR.
A set of collectors repeatedly poll the job_queue table for jobs matching their name, dequeue one, benchmark it, and store results in the database.

The Site and collectors communicate only through the Postgres database — there is no separate broker. The Site runs a periodic tick (default every 30 s, env QUEUE_UPDATE_INTERVAL_SECONDS) that advances request state.

Site

The Site's job in the distributed system is to create benchmark requests, split them into jobs, and mark requests complete.

Creating benchmark requests

A benchmark request is a row in benchmark_request (database/src/lib.rs:1227). Three types (BenchmarkRequestType, database/src/lib.rs:1192):

Master — a merged commit on a repo's default_branch. Created from push webhooks (one request per merge commit) and by the seed_master_commits startup pass (last 30 days). Created in artifacts_ready status.
Try — a PR/MR head. Created by /rust-bench try [sha] [force=true] comments. May start in waiting_for_artifacts (no SHA yet) or artifacts_ready (SHA known).
Release — a published tag. Created from Tag Push Hook (GitCode) or tag-ref push (GitHub), or /rust-bench tag <name>. Created in artifacts_ready.

Every master/try request has a parent (the commit it will be compared against once finished); the parent should generally be benchmarked first. A request's tag column is the artifact identifier (commit SHA or release tag; try requests without a SHA use try-<pr>).

Request state machine

flowchart LR
    W[waiting_for_artifacts] -->|try build artifacts appear| A
    A[artifacts_ready] -->|cron enqueues jobs| I
    I[in_progress] -->|all non-optional jobs done| C[completed]

DB status strings are snake_case: waiting_for_artifacts, artifacts_ready, in_progress, completed (database/src/lib.rs:1134-1147).
waiting_for_artifacts is only used by try requests whose compiler artifacts are not yet built. Transition to artifacts_ready happens when the SHA becomes known.
Once completed, a request's status never changes — though new jobs may still be generated for it via backfilling.

Request queue ordering

Multiple requests may be ready simultaneously, so the Site orders them (job_queue/mod.rs:80-168). Ordering (most to least priority):

in_progress requests.
artifacts_ready requests, sorted topologically by parent completion — requests whose parent is already done have priority. Within a topological level, ordered by PR number then creation time.

The Site maintains the invariant that at most one request is in_progress at a time. Even if a collector finishes early, it waits for all jobs of the in-progress request to complete. This serialises runs for predictability; it could be relaxed in future.

Enqueuing jobs

On each tick, if no request is in_progress and at least one is artifacts_ready, the Site atomically transitions the chosen request to in_progress and enqueues jobs into job_queue (mod.rs:222-362).

For each collector tag in [collectors].tags and each benchmark group resolved for the request (the request's benchmark_groups if set, else the repo's configured groups), one job is enqueued:

enqueue_benchmark_job(request_tag, benchmark_group, runtime_config, is_optional=false, collector_tag) — mod.rs:269-307.
The runtime_config is a serialized QueueRuntimeConfig (database/src/lib.rs:1627) carrying the repo key, build command/dir, is_rust, post_build, per-collector overrides (iterations, no_isolate, filters), and extra_args.

A job is uniquely keyed by (request_tag, benchmark_group, tag) — the tag here is the collector tag used when storing results. Duplicate enqueue attempts are no-ops (JobExistedOrParentNotFound).

Job state machine (database/src/lib.rs:1462):

flowchart LR
    Q[queued] -->|dequeued first time| I
    I[in_progress] -->|success| S[success]
    I -->|failure retry<MAX| I
    I -->|failure retry=MAX| F[failure]

Once enqueued, the tick repeatedly checks whether all non-optional jobs of the in-progress request (and its parent — see backfilling) are complete. When they are, the request transitions to completed and a comparison comment is posted (mod.rs:538-639). Release requests get no comment.

Backfilling

When enqueuing jobs for a request, the Site also enqueues the same jobs for its parent (mod.rs:337-349). Parents should already have results, so in the common case the parent jobs already exist and the insert is a no-op.

Backfilling matters when a request uses non-default parameters that the parent was never benchmarked with — e.g. a try request benchmarking a benchmark group the parent's master run didn't include. New jobs are created for the parent with those parameters; collectors run them and "backfill" results into the already-completed parent request. The parent's status stays completed.

Collectors

Each collector's job is to dequeue jobs, run them, and store results. See deploying-collector guide for the runbook.

Registration

Collectors are registered in the collector_config table (name unique). Register with:

cargo run --bin collector add_collector \
  --db "postgres://user:pass@host/db" \
  --collector_name "Kunpeng 920B" \
  --is_active

add_collector inserts name, date_added, last_heartbeat_at, is_active (collector/src/bin/collector.rs:689). commit_sha is updated at runtime. Only is_active = true collectors receive jobs. The collector_name must match an entry in [collectors].tags for the Site to enqueue jobs for it.

Postgres required. benchmark_job_queue does not work with SQLite.

Dequeuing jobs

cargo run --bin collector benchmark_job_queue \
  --db "postgres://..." --collector_name "Kunpeng 920B" \
  --check_git_sha --git_sha <sha>

After start, the collector polls job_queue (every 30 s) for a job whose collector_name matches and status is queued or already-in_progress-by-itself. On dequeue it marks the job in_progress, increments retry (the dequeue counter), then builds/fetches the artifact per the job's runtime_config and runs all its test cases.

Failure handling and retries

Three failure kinds (collector/src/bin/collector.rs benchmark_job_queue handler):

Handled transient (network/DB error): error recorded in the errors table; job stays in_progress, dequeued again after a wait.
Handled permanent (e.g. artifacts unavailable for the SHA): error recorded; job marked failure; collector moves on.
Unhandled (panic): the collector systemd/script restarts the process; the in-progress job is dequeued again on next loop (its retry counter was already incremented).

If a collector dequeues a job whose retry has reached MAX_RETRIES, it marks the job failure. In-progress (own) jobs are prioritised over new queued jobs.

Automatic git update

collector/collect-job-queue.sh runs the collector in a loop. Before each iteration it does git pull && git reset --hard @{upstream}, rustup update stable, and cargo build --release -p collector --features s3-sdk. With --check_git_sha, the running collector compares its SHA against the repo HEAD; if a newer version exists, it shuts down (delayed until the in-progress request finishes) so the script can rebuild it.

Heartbeat

The collector periodically updates collector_config.last_heartbeat_at. The status page shows heartbeats; a too-old heartbeat marks the collector Offline. A stopped collector that owns in-progress jobs will halt the whole system until manual intervention, since there is no work-stealing between collector tags.