已开启
Stage-5 knowledge base, pl-native engine skills, and cann-bench targeting for pypto-pro-op-orchestrator #585
Stage-5 knowledge base, pl-native engine skills, and cann-bench targeting for pypto-pro-op-orchestrator #585
已开启
hid22900163创建于 7 天前
hid22900163成员
7 天前

Was This PR authored or co-authored using generative AI tooling?

[ ] No
[x] Yes
__1. AI Agent : Claude Code (agentic CLI), driving this repo's own pypto-pro-op-orchestrator plugin
__2. AI Model : Claude Opus 5
__3. Prompt : No single prompt. Per-run prompts were generated by this repo's write-pypto-pro-prompt tooling from the operator SPEC plus live box facts (gym SHA, harness evaluator.py md5, card smoke, case anchors), then executed stage by stage by the orchestrator's own subagents. The knowledge-base pages in this PR are the write-back from those runs: each page was drafted from a measurement made on hardware, then reviewed and edited by hand before commit. Every load-bearing number in these pages is traceable to a run artifact, not to model recall.

What this PR does / why we need it:

Adds a Stage-5 (performance-tuning) knowledge base and the routing needed to reach it, plus the pl-native engine skills the tuning loop dispatches to. One squashed commit, 73 files, +11032/−233 against 68f667f2. The bulk (47 files) is cannbot-skills/ops/pypto-pro-op-kb; the rest wires it into the orchestrator, pypto-pro-op-perf-tune, and the pypto-pro-pl-* engine skills.

The organising idea is an explicit objective precedence, because these three quantities had been conflated and several commits exist only to separate them:

  1. Acceptance — the SPEC's perf_target, falling back to per-P0-case golden_reference_ratio >= 1.0.
  2. In-loop selectionaverage_speedup = mean(baseline_i / candidate_i).
  3. The cann-bench score — target selection and interpretation only, never the acceptance caliber.

Why we need it: without a Stage-5 caliber the tuning loop has no stopping rule, and a measured improvement cannot be distinguished from noise. This branch supplies both, and the runs below are the evidence that it works and that it correctly refuses to ship when it should.

Related issues: #141

Special notes for reviewer:

Measured Stage-5 contribution. Two parts: Part 1 is the branch in its final (or near-final) form — every row ran on a commit in this PR's direct lineage, graded 2026-08-27 → 08-29. Part 2 (further below) is the earlier iterations of the same knowledge lineage, kept for the in-principle demonstration. Both signs shown throughout; the negatives and the no-ships are the honest part. The runner is recorded per row because it matters — see "Runner provenance" below.

Part 1 — final-form submissions (harness ref e8609f6b5891 for every row, --perf-metric-strategy kernel_details)

Several rows are Stage-5-only runs: they tune a frozen Stage-4 seed produced and graded by another run — the rms_norm rows share one seed kernel (graded 57.9601 on multi-1), the dynamic_quant rows another (graded 61.8407 on multi-1). This is deliberate: tuning a common graded seed makes every Stage-5 result a within-kernel delta against that baseline instead of re-rolling generation variance, so the Δ column always leads with the seed-relative delta and lists any secondary comparison after it.

run operator tree runner Stage 4 Stage 5 Δ job
rns5b rms_norm 0654ebd0-line multi-1 57.9601 (rnb67) 73.6601 +15.7000 (same runner) job_3bcc1b7090e1
rnp1 (hint-free) rms_norm f17d4196 multi-0 57.9601 (same frozen seed as rns5b, graded on multi-1) 73.1617 +15.2016 vs seed (multi-0 vs multi-1); reroll 75.8762 on multi-6 (+17.9161) job_b1763d537627 / job_4a20ab5d48b1
rnr1 rms_norm 7feb707d multi-1 57.9601 (same frozen seed as rns5b, graded on multi-1) 74.9036 +16.9435 vs seed, same runner — the branch's largest same-runner Δ; +1.2435 over rns5b (same runner) job_46742433f558
rnf1 (paused mid-tuning; superseded by rnr1) rms_norm 0654ebd0-line multi-4 57.9601 (same frozen seed as rns5b, graded on multi-1) 65.1357 +7.1756 vs seed (cross-runner) — but −8.52 vs rns5b from the same seed: locally-accepted micro-levers that inverted on the grader; the KB's warm-cache/structural-vs-micro discipline exists because of this row job_d84d01d377b1
dqf1-V7 dynamic_quant 0654ebd0-line multi-6 61.8407 (dqb67, multi-1) 68.2203 +6.3796 (cross-runner; the multi-1↔multi-6 offset for this operator was measured ≈0 on byte-identical archives) job_e0b4a06262dc (S4: job_eae59c700f48)
dqp1 (V10H) dynamic_quant f17d4196 multi-1 61.8407 (same frozen seed as dqf1, graded on multi-1) 77.7775 +15.9368 vs seed, same runner; +9.5572 over dqf1-V7 (multi-1 vs multi-6) job_5beb3c4cd68e
dqp1, first roll of the same kernel dynamic_quant f17d4196 multi-0 (as above) 75.9511 +14.1104 vs seed (multi-0 vs multi-1); −1.83 vs the multi-1 roll (runner effect, same archive) job_2f26652b2c40
dqfs1 dynamic_quant 57235ff0 multi-6 77.1134 shipped the Stage-4 snapshot 0.0000 job_50aad3a848fa
expfs1 exp 7feb707d multi-6 80.6603 (from scratch) mid-run snapshot failed correctness 7/20 (note 4) job_d756e7ecb8ba / job_98da616ca92f
cmn1 cummin 7c89d816 multi-4 69.8143 (seed = cummin1's Stage 4, multi-5, earlier ref) 70.8194 +1.0051 (cross-runner AND cross-ref) job_d6d8180c3bf3
cmn1, hidden 80-case set cummin 7c89d816 multi-1 62.8417 (seed, multi-0, earlier ref) 63.8883, 80/80 +1.0466 (cross-runner AND cross-ref) job_b6abf2ba68f9
expfs1, hidden 80-case set exp 7feb707d multi-6 91.8246, 80/80 (the Stage-4 artifact) — (note 4) same-bytes multi-0 draws 89.8830 / 90.0032 (runner offset, see below) job_6ebb910de1c0

Note 4 — expfs1 and cmn1 are deliberate mid-run pauses, not natural completions. Both runs were paused by operator decision ~11 h in and their then-current Stage-5 files submitted as snapshots. cmn1's snapshot was a validated accepted-lever state (rung-3 CONTIG_FUSED + tile-group depth 1→2, local +0.6625 = 2.3× the acceptance floor) and it graded above its seed on both case sets. expfs1's snapshot was the run's remediation-stage delivered kernel — accepted lever, 20/20 and +2.04 clamped on the local pinned harness — which fails 13 of 20 cases on the grader: a local-vs-grader correctness divergence, since root-caused — the evaluation fleet's pypto build predates the framework's 16-bit register-cast (vcvt CastLayout) fixes, so the lever's vf.astype/16-bit-mask surface miscompiles there: exactly the 7 fp32 cases pass and all 13 fp16/bf16 cases fail, and the lever has no fleet-safe in-kernel rewrite (the tile-level pl.cast path has no sub-tile equivalent). Reported as-is; the from-scratch Stage-4 (80.6603, 20/20 public; 91.8246, 80/80 hidden) is the exp line's number.

The strongest claim available, stated precisely: rms_norm moved +15.7000 on the same runner (multi-1) from its graded Stage-4 seed, 57.9601 → 73.6601 (rns5b), and the final tree reaches 74.9036 — +16.9435 over the same seed on the same runner (rnr1). The gain is not a runner artifact and not a seed artifact: Stage-5-only runs against a frozen Stage-4 input. dynamic_quant tells the same story: +15.9368 same-runner from its own frozen seed (61.8407 → 77.7775, both multi-1).

The result is reproducible without hints. rnp1 re-derived rms_norm on the fixed tree with a hint-free prompt and graded 73.1617 on multi-0 — the low-reading runner — with a reroll at 75.88 on multi-6. Same lineage, no coaching, same band.

dynamic_quant is at the class ceiling. dqp1's V10H graded 77.7775, +9.5572 over the V7 kernel the previous tree produced, and lands −0.0676 short of the campaign's all-time dynamic_quant result of 77.8451 (multi-6). That last gap is the honest one: the branch reproduces the best kernel we have ever authored for this operator to within seven hundredths, and does not beat it.

The zero is the design working. dqfs1 ran dynamic_quant from scratch — full Stage 1-5 on tree 57235ff0 — and graded 77.1134 at Stage 4, already near the class ceiling above. Its Stage 5 then spent roughly 65 screening invocations, built same-session 20/20 evidence chains for both candidate levers, rejected both, and shipped the frozen Stage-4 snapshot under the "no gain → ship the snapshot" gate. Δ 0.0000 is the correct outcome here, and the loop reached it by measurement rather than by giving up: the no-ship gate held against a long, well-resourced search.

Runner provenance. runner-950pr-multi-0 reads low. Measured on byte-equivalent archives: −2.7145 for rms_norm (job_b1763d537627 vs job_4a20ab5d48b1), −1.83 for dynamic_quant (the job_5beb3c4cd68e / job_2f26652b2c40 pair in the table), and for exp ≈−2.55 public (78.1117 on multi-0 vs 80.66/80.76 on multi-6/-5) and ≈−1.9 hidden (89.88/90.00 on multi-0 vs 91.8246 on multi-6, the row above). The offsets are per-operator and per-kernel — they must not be used to "adjust" a differently-authored kernel's score. Cross-runner deltas are therefore not quoted in this packet; every Δ above is either same-runner or annotated with the two runners it spans.

Provenance: Ascend 950PR; boxes a5-238 (physical cards 3 and 4) and a5-41 (card 0); CANN 9.2.0; pypto env snapshots pypto-master-20260730 (early rows) and pypto-master-20260824 (rows from 2026-08-27 on); harness ref e8609f6b5891; scoring with --perf-metric-strategy kernel_details. Grading dates: 2026-08-27 → 2026-08-29.

What we would most like review on:

  1. The objective-precedence model — is golden_reference_ratio the right default acceptance bar when SPEC.perf_target is null, and is "cann-bench score is targeting-only" the rule you want enforced?
  2. The five accept gates, added after a diagnosed sigmoid regression slipped through: pinned instrument, cache-flush symmetry, inadmissible-case exclusion, measuring on the trigger band with its share stated, and the ship-the-snapshot rule.
  3. KB_USAGE.json — this branch requires a refresh of it in Stage 5, mapping selected reference → derived invariant → implementation location → implementation claim. We think it is the highest-leverage gate in the flow; we would like it endorsed or replaced.
  4. Whether the kb tier's size is acceptable given selection is uncapped, or whether you would prefer a non-routed tier. This branch touches 47 files under ops/pypto-pro-op-kb; of those, 14 are newly added pages and 33 are edits to existing ones.
  5. The R1-R8 closure protocol layer — the attack matrix, the per-spend ledger verdicts, and the escalation semantics (blocked_by_stage_authority, ARCH_ESCALATION). This is the part that decides when a tuning attempt is finished rather than merely out of ideas, and it is the least conventional thing in the branch.
  6. The fixed-overhead / launch-cost attack row. Flagging it explicitly: this is the KB's one a-priori row, marked PROVISIONAL. It is reasoned, not measurement-born — unlike every other row in the matrix — and it carries its own close conditions. If you would rather the KB admit no un-measured rows at all, this is the row to strike.
  7. The MergeMode.ZEROING masked-op hazard page. A partially-masked vf.max zeroes the inactive destination lanes by default; any width-selective kernel that assumes the inactive lanes are preserved is silently wrong. This was found and fixed by a live run, and the page exists so the next run does not rediscover it the hard way. We would like the wording checked against the framework's intended semantics.

Known weak points, so you don't have to find them:

  • No CI runs the kb's own check_kb_integrity.py (21 checks); it must be invoked by hand.
  • Two adjudications are recorded in-tree rather than resolved. One is a UB-capacity figure that diverges from upstream's stated value; the other is a historical defect record whose account we could not confirm, flagged in-tree as disputed-unmeasured. Both are written down where a reader will hit them, but neither is settled, and a reviewer with upstream authority could close both quickly.
  • The one PROVISIONAL, non-measurement-born KB row called out in review item 6.
  • The historical evidence below carries an unexplained hidden-set sigmoid delta and one operator whose Stage-4 gate never cleared; both are scoped to that subsection.

Part 2 — earlier iterations of the same lineage (2026-08-19 → 08-26)

The runs in this subsection predate the branch's rebase onto 68f667f2 and the R1-R8 / audit-1 / audit-2 hardening, and two of them did not execute against any commit now in this PR (note 1). They are not a measurement of the tree being merged and should not be read as one. They are kept because they come from the same knowledge lineage and demonstrate the Stage-5 loop working in principle — including a gain reproduced across two independent runs from one seed, honest no-ship zeros, and a hidden-set correctness repair. Unless a row says otherwise, Part 2 was graded on the earlier harness ref be5b09b86dd2 (same-kernel controls later measured that ref neutral vs e8609f6b5891 for both operators tested).

run operator gym commit Stage 4 Stage 5 Δ
s13rn rms_norm not recorded (note 1) 74.1672 76.4570 +2.2898
s5o5 rms_norm f196bd49 74.1672 † 76.1690 +2.0018
s5o3 rms_norm f196bd49 74.1672 † 74.1520 −0.0152
s13sig sigmoid not recorded (note 1) 80.5713 80.5360 −0.0353
s13sig, hidden 80-case set sigmoid not recorded 92.2102 91.7891 −0.4211
sig5o1 sigmoid 0a0f8984 (note 2) shipped the Stage-4 snapshot 0.0000
conv2d conv_2d f196bd49 50.8488 (multi-4, 18/20) shipped the Stage-4 snapshot 0.0000
s5o1 rms_norm ran on a different box seed predates the window 75.2177 not pairable in-window
dq1 dynamic_quant pre-rebase lineage 73.7767 (multi-0) 77.8451 (multi-6; re-graded 2026-08-28 on ref e8609f6b5891) +4.07 (cross-runner AND cross-ref — context only, not a paired Δ)
dq1-line, hidden 80-case set dynamic_quant pre-rebase lineage no Stage-4 hidden submission was made in this era 86.7652 (multi-0) the operator's hidden best of this era
dsq1 dequant_swiglu_quant pre-rebase lineage (2026-08-25/26 batch) 83.5117 (multi-5) · reroll 80.9390 (multi-0) 83.7473 (multi-1) · reroll 81.1613 (multi-0) +0.2223 same-runner (the byte-pair on multi-0)
dsq1, hidden 80-case set dequant_swiglu_quant pre-rebase lineage 79.6228 partial, 70/80 — correctness failed (multi-1) 87.9516, 80/80 (multi-5) Stage 5 REPAIRED the hidden-set correctness break (note 5)
cummin1 cummin pre-rebase lineage (2026-08-25/26 batch) 69.8143 (multi-5) 70.9546 (multi-0) · reroll 71.7578 (multi-1) cross-runner; this Stage-4 artifact later seeded Part 1's cmn1
cummin1, hidden 80-case set cummin pre-rebase lineage 62.8417 (multi-0) 64.9604 (multi-1) +2.12 nominal — cross-runner, and the multi-0 side reads low, so the true Δ is smaller
exp (first iteration), from-scratch Stage 4 exp pre-rebase lineage 78.0377 (multi-0) baseline of this era
exp, hand-tuned reference kernel (NOT a branch product) exp hand-authored public 79.3369 (multi-0) · hidden 91.0769 (multi-0) re-graded 2026-08-31 on ref e8609f6b5891: public 81.5971 (multi-5) but hidden 57.7455 (multi-1, 80/80) — the old hidden record does not survive the current instrument, while Part 1's from-scratch kernel reads 91.8246 there; exp hidden scoring is not neutral across harness refs

† s5o5 and s5o3 were both seeded from s13rn's Stage-4 artifact, so they share its graded 74.1672 as baseline. They are not independent Stage-4 measurements and their deltas must not be summed with s13rn's.

s13rn and s5o5 are two independent Stage-5 runs from the same Stage-4 seed, landing +2.2898 and +2.0018 — agreeing to within 0.29, that window's measured across-session noise floor. A reproduced gain, not one lucky result.

The two zeros are the design working. In sig5o1 and conv2d, Stage 5 measured its candidate levers, could not clear the acceptance bar, and shipped the frozen Stage-4 snapshot under the "no gain → ship the snapshot" gate, preserving the tuned variants and recording the falsification. conv2d rejected a lever whose graded movement was +0.2111 — a real, band-confined, accuracy-neutral effect that sits below that window's within-session paired floor of 0.291 and is therefore not shippable evidence.

dsq1 is the era's strongest single result. Its Stage 4 failed 10 of 80 hidden cases outright; the Stage-5 pass both repaired the correctness break (70/80 → 80/80) and lifted the hidden score to 87.9516 — a +8.33 nominal move that is mostly correctness recovery, stated as such rather than as tuning gain (note 5). Its public same-runner byte-pair (+0.2223 on multi-0) is the honest public Δ: small and real.

The negatives are small. s5o3 (−0.0152) and s13sig (−0.0353) are inside the noise floor and read as "no effect". The hidden-set sigmoid figure (−0.4211) is the one number that exceeds a floor, and it is not explained — a known weak point, scoped to this historical lineage. conv2d's Stage 5 also shipped nothing, and its Stage-4 gate still carried one blocker (precision_failure, 18/20); that operator's ceiling is bounded by a framework capability, not by this branch.

Note 1 — two runs do not record their gym commit. s13rn and s13sig (2026-08-19) carry no gym SHA in their workspace env.sh or setup script. The deployed tree of that vintage was gym-single-3cf7e0fb, and 3cf7e0fb is not an ancestor of this branch — it belongs to the pre-rebase lineage. Those two runs were not executed against the commits in this PR, and the table says so rather than implying otherwise. Every run in the headline table above records its tree.

Note 2 — sig5o1 ran mid-branch. 0a0f8984 belonged to the pre-rebase commit sequence, two commits before that tip. This branch's content of the time, just not its final state.

Note 5 — the dsq hidden Δ is not a tuning number. The Stage-4 kernel's hidden failure was functional (a case class the public 20 never exercises); Stage 5's rewrite removed the failing construct. The +8.33 hidden movement therefore bundles correctness repair with whatever performance change came along; only the public same-runner pair (+0.2223) is quotable as tuning.

Note 3 — the pypto version string cannot identify the build. Our environment reports pypto 0.2.1; so does a materially different evaluation-fleet build whose pl.jit signature differs (compile_timeout rather than timeout, auto_mutex defaulting to True). The version string is not a usable handle, so we cite the env snapshot name and CANN version. For the record, that build has been identified as master 86f508520 (2026-07-30).

AI Declaration / AI 声明

  • AI tool / AI 工具: Claude Code
  • AI models / AI 模型: Claude Opus 5, Claude Fable 5
  • The squashed commit's message carries the same declaration (AI-assisted: Claude Code; AI models: Claude Opus 5, Claude Fable 5), as the declaration check requires.

Change Type / 改动类型

Testing / 测试信息

This branch adds knowledge-base pages and skill routing; it ships no library code, so the repo's UT and ST suites do not cover it. Verification was end-to-end on hardware, by running the pipeline these pages drive and grading the resulting kernels on the public cann-bench task set — the table in "Special notes" is that result, reported with both signs.

  • check_kb_integrity.py (the kb's own 21 checks, incl. every-page-routable, reachability, filename rules, link and magnitude-home checks): run by hand, passing. Note there is no CI invoking it.

  • Graded pipeline runs on Ascend 950PR, all against harness ref e8609f6b5891: the headline table's seven rows on the current lineage, plus the eight-row historical lineage kept below it. Strongest result is a Stage-5-only gain of +15.7000 on one runner (rms_norm, 57.9601 → 73.6601), reproduced hint-free at 73.1617 / 75.88 and extended to 74.9036 on the final tree; dynamic_quant reaches 77.7775, within 0.0676 of the campaign's all-time result for that operator.

Checklist / 检查清单

likedislike
合并受阻
Hhid22900163成员
7 天前 创建了 pull request,commit 9c9b247f
Hhid22900163成员
7 天前 修改了pull request 的描述
Hhid22900163成员
7 天前 修改了pull request 的描述
Hhid22900163成员
7 天前 修改了pull request 的描述
Hhid22900163成员
7 天前 修改了pull request 的描述
此处折叠了192条消息 查看更多
CANN-robotCANN-robot成员
22 小时前 删除了label:ci-pipeline-failed
CANN-robotCANN-robot成员
22 小时前 添加了label:ci-pipeline-running
CANN-robot
CANN-robot成员
22 小时前 评论:

流水线任务触发成功
任务链接 [fafa3c7a5b4d4a87b761cb5cea46dc7e][流水线指导]

任务名称状态日志下载链接
SCA ✅ SUCCESS >>>>>
antipoison ✅ SUCCESS >>>>>
Check_Pr ✅ SUCCESS >>>>>
pre_comment ✅ SUCCESS >>>>>
precommit ✅ SUCCESS >>>>>
codecheck_style ✅ SUCCESS >>>>>
codecheck ✅ SUCCESS >>>>>
Smoke_A900 ✅ SUCCESS >>>>> >>>>>

[2026-08-31 16:32:14]    CI执行结束

likedislike
CANN-robotCANN-robot成员
20 小时前 删除了label:ci-pipeline-running
CANN-robotCANN-robot成员
20 小时前 添加了label:ci-pipeline-passed