Performance Regression Report

Time unit: ms. Each reported value is the mean of 5 measured runs.

Profile - Windows: 4 physical CPU cores, 8 logical CPU cores, 32 GiB RAM

• Serial: 1 worker
• Parallel: 8 workers

Profile delta

Result

Five-project aggregate from the required full benchmark/updated runs:

• Serial first: 71,672.69 ms -> 15,678.79 ms (-55,993.91 (-78.12%)).
• Serial warm: 17,208.61 ms -> 14,642.97 ms (-2,565.64 (-14.91%)).
• Parallel first: 4,366.09 ms -> 3,751.77 ms (-614.32 (-14.07%)).
• Parallel warm: 4,465.43 ms -> 3,602.49 ms (-862.94 (-19.32%)).

Alternating A/B confirmation (baseline and candidate run in alternating order to limit host drift):

• Serial first: 14,443.48 ms -> 14,465.59 ms (+22.11 (+0.15%)).

• Serial warm: 14,727.47 ms -> 14,666.90 ms (-60.57 (-0.41%)).

• Parallel first: 3,863.03 ms -> 3,698.30 ms (-164.73 (-4.26%)).

• Parallel warm: 3,870.51 ms -> 3,706.41 ms (-164.10 (-4.24%)).

• Result: no stable performance regression was observed. Small positive and negative paired deltas vary by strategy; the required full benchmark aggregates are non-regressed in all four strategies.

• Analysis semantics were unchanged for all five projects: file and NBNC totals match (True).

Common configuration

• Baseline revision: 4a2bc49230372439dad42565cae68577c47fce9c (main)
• Updated revision: b4453ae2fd06acc5f085d5d4859b42682d308521
• scope: reachable
• aggregation: physical
• format: json
• cache_policy: cold_then_filesystem_warm
• warm_runs: 5
• workers: 1, 8
• Toolchain: rustc 1.98.0-nightly (d595fce01 2026-06-02)
• Both revisions used isolated release binaries built with --locked.
• Evidence: D:\MetricOpt\docs\perf_res\issue_16\benchmark.json, updated.json, paired_baseline.json, and paired_candidate.json.

评审结论

当前 issue_16_22 分支已实现两个 Issue 的主要用户可见目标：

• Issue #16：超时报告新增 compiler_run.timed_out = true，BM302 消息明确包含 timeout，退出码 124 和 partial 语义保持不变。
• Issue #22：文档明确 SARIF 的静态 finding 与 compiler diagnostic 共用 runs[].results，并规定按 ruleId/properties 分类，不能依赖数组位置。

但仍有两个 P1 工程问题，因此当前 PR 不建议直接合并。

P1：Windows 超时没有终止完整进程树

Unix 使用进程组终止：

libc::kill(-(child.id() as i32), libc::SIGKILL)

但 Windows 路径仍然只是：

child.kill()

见 compiler.rs。

Child::kill() 只保证终止直接启动的 Cargo 进程，不保证终止其启动的 rustc、build script 或其他子进程。更严重的是，子进程可能继续持有 stdout/stderr 管道，而 run_command 随后会等待两个 reader thread 结束，导致：

• --compiler-timeout-secs 在 Windows 上不能形成真实耗时上界；
• 长时间运行的 build script 可能在 Cargo 被杀后继续运行；
• reader thread 可能继续阻塞；
• 现有测试仍可能在 build script 自然退出后通过，掩盖进程树未清理的问题。

当前只完成了 Windows 交叉编译和 Clippy，没有在 Windows 原生执行这个测试。

建议使用 Windows Job Object，并设置 JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE，把 Cargo 及其后代放入同一个 Job；同时增加原生 Windows 验证，断言实际耗时接近配置的 timeout，并验证没有残留子进程。

P1：timed_out 未进入 1.2.1 Schema 契约

运行时已经正式输出：

"compiler_run": {
  "exit_code": 124,
  "timed_out": true
}

字段定义见 report.rs。

但 analysis-report-1.2.1.json 没有声明 compiler_run，更没有声明：

• command
• toolchain
• exit_code
• timed_out
• output_truncated
• diagnostic_records_truncated

当前 Schema 因顶层 additionalProperties: true 仍会接受报告，但这只代表“没有拒绝未知字段”，不能证明 Schema 已描述新增机器契约。代码和文档要求下游使用 1.2.1 Schema 验证报告，因此新增的结构化超时字段必须进入 Schema，并补充实际报告的 Schema 验证测试。

已确认：

• Cargo 超时状态可以结构化消费；
• BM302 不再要求用户理解 124 惯例；
• partial 和退出码 3 保持不变；
• Cargo/Clippy 共用传播路径，rustc 路径也写入 output.timed_out；
• 旧报告缺少 timed_out 时默认反序列化为 false；
• false 不序列化，兼容原有成功报告。

结论：功能目标已解决，但 Schema 和 Windows 进程树边界尚未闭环。

Issue #22 复核

生产 SARIF serializer 的实际顺序为：

全部 finding
→ 全部 compiler diagnostic

编译器诊断输出：

{
  "ruleId": "rustc.diagnostic",
  "properties": {
    "code": "E0308"
  }
}

文档已明确：

• 两类结果共用 runs[].results；
• 顺序仅用于确定性输出；
• 下游不得通过数组位置分类；
• finding 使用原始规则 ID；
• compiler diagnostic 使用 rustc.diagnostic；
• 原始 rustc/lint code 位于 properties.code。

现有流式 SARIF 与兼容投影等价性测试也覆盖了包含 finding 和 compiler diagnostic 的报告。

结论：Issue #22 可以认为已经闭环。

Performance Regression Report

Time unit: ms. Each strategy uses one first run followed by five warm runs.

Strict rule: any updated cell slower than the Web benchmark is considered a regression.

Profile A — 16 GB RAM · 8 CPU cores

Approximation on the current host:

• Serial: 1 worker
• Parallel: 8 workers
• No physical 16 GB memory limit was applied.

Profile A delta

Profile A result:

• Non-regressed cells: 2 / 20
• Regressed cells: 18 / 20
• Strict result: failed

Profile B — Apple M5 Max · 64 GB RAM

Measured on the current host:

• Serial: 1 worker
• Parallel: 18 workers

Profile B delta

Profile B result:

• Non-regressed cells: 1 / 20
• Regressed cells: 19 / 20
• Strict result: failed

Same-host A/B attribution

The branch was compared with a release build from the same-host main checkout.

The bot-metric · 8 workers warm result was remeasured with 10 interleaved runs:

main mean:   53.32 ms
issue_16_22: 53.16 ms
delta:       −0.16 ms (−0.30%)

main median:   53.15 ms
issue_16_22:   52.61 ms
delta:         −0.54 ms (−1.02%)

Interpretation

• The strict Web comparison failed because the Web values are historical v1.2.0 measurements and many updated cells are slower.
• The current project checkouts are not identical to the historical Web snapshot:
  • bot-forge: 67 files / 27,201 NBNC
  • bot-gate: 62 files / 38,521 NBNC
  • bot-metric: 29 files / 26,143 NBNC
  • rust: 2,817 files / 826,803 NBNC
• Issue #16/#22 production changes are Windows-specific Job Object/compiler-timeout handling and are not executed by this macOS benchmark.
• Schema/version and documentation changes do not affect the source-analysis hot path.
• Same-host A/B testing found no sustained attributable performance regression.
• The isolated bot-metric · 8 workers fluctuation disappeared during interleaved remeasurement.
• Therefore, the strict historical Web gate fails, but this run does not provide evidence of a performance regression caused by issue_16_22.

Run configuration

• scope: reachable
• aggregation: physical
• format: json
• warm_runs: 5
• Profile A: 1 / 8 workers
• Profile B: 1 / 18 workers
• Current host: Apple M5 Max, 18 cores, 64 GB RAM
• Physical 16 GB memory limit: not applied
• Toolchain: Rust 1.89.0 / Cargo 1.89.0
• Benchmark binary: Web v1.2.0 historical baseline
• Updated binary: issue_16_22 release build
• Updated schema: analysis-report-1.2.2.json