Star106
92
代码
代码
Issues32
Pull Requests8
Actions
项目讨论
Wiki
分析
Star106
92
cann-robotcann-robotfeat(report): 评测报告支持 AGENT/HARNESS/MODEL 提交标签渲染
d1d5cc47创建于 14 天前历史提交
<!DOCTYPE html>
<html lang="zh-CN">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width,initial-scale=1.0">
<title>CANN-Bench Evaluation Report — 2026-05-25</title>
<style>
/* =============================================================
   CANN-Bench Report Style v4 — Academic-Technical
   Design: unified sans-serif, WCAG AA contrast, continuous
   score gradient (no ad-hoc tiers), semantic HTML, zebra tables.
   ============================================================= */

/* --- Reset & Variables --- */
*,*::before,*::after{box-sizing:border-box;margin:0;padding:0}
:root{
  --ink:#1a1a2e;
  --ink-mid:#4a4a5e;
  --ink-light:#7a7a8e;
  --rule:#c8c8d0;
  --rule-light:#e8e8ee;
  --bg-page:#f0f2f5;
  --bg-paper:#fff;
  --bg-stripe:#f8f9fb;
  --bg-callout:#f4f6fa;
  --score-high:#1a6b3c;
  --score-mid:#b8860b;
  --score-low:#c0392b;
  --accent:#1e4d7b;
  --font-stack:-apple-system,BlinkMacSystemFont,"Segoe UI","Noto Sans SC","PingFang SC","Microsoft YaHei",sans-serif;
  --font-mono:"SF Mono","Cascadia Code","Consolas","Noto Sans Mono SC",monospace;
  --content-width:860px;
  --pad-h:40px;
}
html{font-size:16px}
body{
  background:var(--bg-page);
  display:flex;justify-content:center;
  padding:24px 8px 48px;
  font-family:var(--font-stack);
  color:var(--ink);
  line-height:1.6;
}
.report{
  width:100%;max-width:var(--content-width);
  background:var(--bg-paper);
  box-shadow:0 0 0 1px rgba(0,0,0,.06),0 2px 16px rgba(0,0,0,.08);
}

/* --- Header Bar --- */
.report-header{
  background:var(--ink);
  color:#fff;
  padding:10px var(--pad-h);
  font-size:.6875rem;
  letter-spacing:.18em;
  text-align:center;
  text-transform:uppercase;
}
.report-header span{opacity:.55;font-weight:400}

/* --- Title Area --- */
.title-area{
  padding:36px var(--pad-h) 28px;
  text-align:center;
  border-bottom:3px double var(--ink);
}
.title-area .journal-line{
  font-size:.75rem;color:var(--ink-light);
  letter-spacing:.06em;margin-bottom:12px;
}
.title-area h1{
  font-size:1.5rem;font-weight:700;color:var(--ink);
  line-height:1.4;margin-bottom:8px;
}
.title-area h2{
  font-size:.875rem;font-weight:400;color:var(--ink-mid);
  font-style:italic;
}
.title-area .meta-line{
  font-size:.75rem;color:var(--ink-mid);margin-top:14px;
}
.title-area .meta-line+.meta-line{margin-top:2px}

/* --- Abstract --- */
.abstract{
  margin:20px var(--pad-h) 0;
  padding:20px 24px;
  background:var(--bg-callout);
  border:1px solid var(--rule-light);
  border-radius:4px;
}
.abstract h3{
  font-size:.8125rem;font-weight:700;color:var(--ink);
  margin-bottom:8px;text-transform:uppercase;letter-spacing:.04em;
}
.abstract p{
  font-size:.8125rem;color:var(--ink-mid);
  line-height:1.75;text-align:justify;
}

/* --- Sections --- */
.section{padding:0 var(--pad-h);margin-top:28px}
.section h3{
  font-size:1rem;font-weight:700;color:var(--ink);
  padding-bottom:7px;
  border-bottom:1px solid var(--rule);
  margin-bottom:16px;
}
.section h3 .sec-num{font-weight:700;margin-right:8px}
.section h4{
  font-size:.8125rem;font-weight:700;color:var(--ink-mid);
  margin:16px 0 8px;
}
.section p,.section li{
  font-size:.8125rem;color:var(--ink-mid);line-height:1.75;
}

/* --- KPI Strip --- */
.kpi-strip{
  display:flex;gap:0;margin:16px 0;
  border:1px solid var(--rule-light);border-radius:4px;
  overflow:hidden;
}
.kpi-item{
  flex:1;text-align:center;
  padding:18px 12px;
  background:var(--bg-paper);
  border-right:1px solid var(--rule-light);
}
.kpi-item:last-child{border-right:none}
.kpi-item .kpi-value{
  font-size:1.5rem;font-weight:700;color:var(--ink);
  line-height:1.2;
}
.kpi-item .kpi-label{
  font-size:.6875rem;color:var(--ink-light);
  margin-top:4px;text-transform:uppercase;letter-spacing:.03em;
}
.kpi-item .kpi-sub{
  font-size:.6875rem;color:var(--ink-light);margin-top:2px;
}

/* --- Tables (three-line LaTeX style) --- */
.table-wrap{margin:14px 0}
.table-wrap caption{
  font-size:.75rem;font-weight:600;color:var(--ink-mid);
  text-align:center;margin-bottom:8px;
  caption-side:top;
}
.table-wrap table{
  width:100%;border-collapse:collapse;
  font-size:.75rem;
}
.table-wrap thead th{
  padding:9px 10px;text-align:center;
  font-weight:700;color:var(--ink);
  border-top:2px solid var(--ink);
  border-bottom:1.5px solid var(--ink);
  font-size:.6875rem;text-transform:uppercase;letter-spacing:.03em;
  background:var(--bg-paper);
}
.table-wrap tbody td{
  padding:8px 10px;text-align:center;
  color:var(--ink-mid);
  border-bottom:1px solid var(--rule-light);
}
.table-wrap tbody tr:nth-child(even){background:var(--bg-stripe)}
.table-wrap tbody tr:last-child{border-bottom:1.5px solid var(--ink)}
.table-wrap .col-name{text-align:left;font-weight:600;color:var(--ink)}

/* --- Score coloring --- */
.score-cell{font-weight:600}
.score-high{color:var(--score-high)}
.score-mid{color:var(--score-mid)}
.score-low{color:var(--score-low)}

/* --- Bar Chart --- */
.bar-chart{display:flex;flex-direction:column;gap:7px;margin:12px 0}
.bar-row{display:flex;align-items:center;gap:10px}
.bar-label{font-size:.6875rem;color:var(--ink-mid);width:130px;text-align:right;white-space:nowrap}
.bar-track{flex:1;height:16px;background:var(--rule-light);border-radius:3px;overflow:hidden}
.bar-fill{height:100%;border-radius:3px}
.bar-val{font-size:.6875rem;color:var(--ink-mid);width:80px;text-align:left;font-weight:600}

/* --- Setup / Metadata Grid --- */
.setup-grid{display:grid;grid-template-columns:1fr 1fr;gap:14px;margin-top:14px}
.setup-block{
  border:1px solid var(--rule-light);
  border-radius:4px;padding:16px 18px;
}
.setup-block h4{
  font-size:.6875rem;font-weight:700;color:var(--ink);
  margin:0 0 10px;text-transform:uppercase;letter-spacing:.06em;
}
.setup-block table{width:100%;border-collapse:collapse;font-size:.6875rem}
.setup-block td{padding:4px 6px;color:var(--ink-mid);border-bottom:1px solid var(--rule-light);line-height:1.55}
.setup-block .kv-k{font-weight:600;color:var(--ink);white-space:nowrap;width:38%}

/* --- Score Distribution Histogram --- */
.score-dist{display:flex;align-items:end;gap:4px;height:80px;margin:16px 0;padding:0 4px}
.score-dist .hist-bar{flex:1;background:var(--accent);border-radius:2px 2px 0 0;min-height:2px;opacity:.85}
.score-dist .hist-label{text-align:center;font-size:.5625rem;color:var(--ink-light);margin-top:4px}

/* --- Three-axis Breakdown --- */
.axis-bar-wrap{margin:6px 0}
.axis-bar-label{font-size:.625rem;color:var(--ink-mid);margin-bottom:2px;display:flex;justify-content:space-between}
.axis-bar-track{height:8px;background:var(--rule-light);border-radius:2px;overflow:hidden}
.axis-bar-fill{height:100%;border-radius:2px}

/* --- Three-axis scoring visual --- */
.axis-viz{
  display:grid;grid-template-columns:1fr 1fr 1fr;
  gap:12px;margin:16px 0;
}
.axis-card{
  text-align:center;padding:16px 12px;
  border:1px solid var(--rule-light);border-radius:6px;
}
.axis-card .axis-icon{
  font-size:1.5rem;margin-bottom:6px;
}
.axis-card .axis-name{
  font-size:.6875rem;font-weight:700;color:var(--ink);
  text-transform:uppercase;letter-spacing:.03em;
}
.axis-card .axis-weight{
  font-size:.625rem;color:var(--ink-light);margin-top:2px;
}
.axis-card .axis-bar{
  height:6px;background:var(--rule-light);
  border-radius:3px;margin-top:8px;overflow:hidden;
}
.axis-card .axis-fill{height:100%;border-radius:3px}
.axis-card .axis-detail{
  font-size:.625rem;color:var(--ink-mid);margin-top:6px;line-height:1.5;
}

/* --- Detail lists --- */
.detail-list{list-style:disc;padding-left:20px;margin:8px 0}
.detail-list li{font-size:.8125rem;color:var(--ink-mid);line-height:1.7}

/* --- Note / Callout --- */
.note{
  font-size:.6875rem;color:var(--ink-light);
  line-height:1.7;margin-top:6px;font-style:italic;
}

/* --- References --- */
.references{
  padding:22px var(--pad-h);margin-top:24px;
  border-top:1px solid var(--rule);
}
.references h4{
  font-size:.8125rem;font-weight:700;color:var(--ink);margin-bottom:10px;
}
.references ol{
  font-size:.6875rem;color:var(--ink-mid);
  padding-left:20px;line-height:2;
}

/* --- Footnote --- */
.footnote{
  padding:14px var(--pad-h);margin-top:20px;
  border-top:1px solid var(--rule);
  font-size:.625rem;color:var(--ink-light);line-height:1.8;
}

/* --- Formula Blocks --- */
.formula-block{
  background:#fafbfc;
  border:1px solid var(--rule-light);
  border-left:3px solid var(--accent);
  border-radius:4px;
  padding:16px 20px;
  margin:14px 0;
  font-family:var(--font-mono);
  font-size:.75rem;
  line-height:1.9;
  color:var(--ink);
}
.formula-block .eq-num{
  float:right;font-weight:700;color:var(--ink-light);
  font-size:.6875rem;margin-left:16px;
}
.formula-block .eq-body{
  display:block;text-align:center;padding:6px 0;
  letter-spacing:.02em;
}
.formula-block .eq-note{
  display:block;font-size:.625rem;color:var(--ink-light);
  font-family:var(--font-stack);margin-top:6px;line-height:1.6;
}

/* --- Flow Diagrams (pure CSS) --- */
.flow-h{
  display:flex;align-items:stretch;justify-content:center;
  gap:0;margin:18px 0;flex-wrap:wrap;
}
.flow-node{
  flex:0 0 auto;min-width:100px;max-width:180px;
  border:1.5px solid var(--rule);
  border-radius:6px;padding:14px 16px;
  text-align:center;background:var(--bg-paper);
  display:flex;flex-direction:column;justify-content:center;
}
.flow-node .flow-title{
  font-size:.6875rem;font-weight:700;color:var(--ink);
  margin-bottom:4px;text-transform:uppercase;letter-spacing:.03em;
}
.flow-node .flow-desc{
  font-size:.625rem;color:var(--ink-light);line-height:1.5;
}
.flow-node.accent{border-color:var(--accent);background:#f5f8fc}
.flow-arrow{
  flex:0 0 auto;display:flex;align-items:center;
  padding:0 8px;font-size:1.25rem;color:var(--ink-light);
  font-family:var(--font-stack);
}

/* --- Pipeline (vertical) --- */
.flow-v{
  display:flex;flex-direction:column;align-items:stretch;
  gap:0;margin:16px 0;
}
.flow-v .flow-row-item{
  display:flex;align-items:center;gap:14px;
  padding:10px 14px;
  border-left:3px solid var(--accent);
  margin-bottom:10px;
  background:var(--bg-stripe);
  border-radius:0 4px 4px 0;
}
.flow-v .flow-step{
  width:36px;height:36px;border-radius:50%;
  background:var(--accent);color:#fff;
  display:flex;align-items:center;justify-content:center;
  font-size:.6875rem;font-weight:700;flex-shrink:0;
}
.flow-v .flow-body{flex:1}
.flow-v .flow-body .flow-title{
  font-size:.75rem;font-weight:700;color:var(--ink);
}
.flow-v .flow-body .flow-desc{
  font-size:.6875rem;color:var(--ink-mid);margin-top:2px;
}

/* --- Utility --- */
.mt-12{margin-top:12px}
.mt-16{margin-top:16px}

/* --- Responsive --- */
@media(max-width:680px){
  :root{--pad-h:20px}
  .setup-grid{grid-template-columns:1fr}
  .kpi-strip{flex-wrap:wrap}
  .kpi-item{flex:1 1 50%}
}

/* --- Seal / Stamp --- */
.seal-section{
  padding:0 var(--pad-h);margin-top:28px;text-align:center;
}
.seal-section h3{
  font-size:1rem;font-weight:700;color:var(--ink);
  padding-bottom:7px;margin-bottom:16px;
  border-bottom:1px solid var(--rule);
}
.seal-wrapper{
  display:flex;justify-content:center;align-items:center;
  margin:28px 0 20px;
}
.seal{
  position:relative;width:180px;height:180px;
  border:4px solid var(--score-high);border-radius:50%;
  display:flex;align-items:center;justify-content:center;
  background:rgba(26,107,60,.06);
  box-shadow:inset 0 0 0 3px rgba(26,107,60,.12);
}
.seal::after{
  content:'';position:absolute;inset:8px;
  border:2px solid var(--score-high);border-radius:50%;
  opacity:.65;
}
.seal-inner{
  text-align:center;color:var(--score-high);
}
.seal-inner .seal-title{
  font-size:.625rem;font-weight:700;
  letter-spacing:.15em;text-transform:uppercase;
  margin-bottom:4px;
}
.seal-inner .seal-star{
  font-size:2.4rem;font-weight:700;line-height:1;
}
.seal-inner .seal-date{
  font-size:.5rem;font-weight:500;margin-top:6px;
  opacity:.7;letter-spacing:.06em;
}
.seal-arc{
  position:absolute;top:22px;left:50%;
  transform:translateX(-50%);
  font-size:.5rem;font-weight:700;color:var(--score-high);
  letter-spacing:.22em;opacity:.85;
  white-space:nowrap;
}
.seal-arc-bottom{
  position:absolute;bottom:28px;left:50%;
  transform:translateX(-50%);
  font-size:.4375rem;font-weight:600;color:var(--score-high);
  letter-spacing:.18em;opacity:.7;
  white-space:nowrap;
}
.seal-cert{
  padding:10px 0;
}
.seal-cert p{
  font-size:.75rem;color:var(--ink-mid);
  line-height:1.8;margin:0;
}
.seal-cert .cert-sign{
  font-size:.6875rem;color:var(--ink-light);
  margin-top:14px;font-style:italic;
}
</style>
</head>
<body>
<main class="report">

  <!-- ============================================================ -->
  <!-- HEADER BAR                                                    -->
  <!-- ============================================================ -->
  <div class="report-header">
    CANN-Bench Evaluation Report
    &ensp;&middot;&ensp;
    <span>V0.1.0 &middot; 2026-05-25</span>
  </div>

  <!-- ============================================================ -->
  <!-- TITLE                                                         -->
  <!-- ============================================================ -->
  <div class="title-area">
    <h1>CANN-Bench 评测报告</h1>
    <h2>Ascend C Operator Benchmark — AI Code Generation Evaluation</h2>
    <div class="meta-line">
      53 Operators &ensp;&middot;&ensp;
      1,060 Cases &ensp;&middot;&ensp;
      4 Difficulty Levels
    </div>
    <div class="meta-line">CANN-Bench V0.1.0 &ensp;&middot;&ensp; 2026-05-25</div>
  </div>

  <!-- ============================================================ -->
  <!-- ABSTRACT                                                      -->
  <!-- ============================================================ -->
  <div class="abstract">
    <h3>Abstract / 摘要</h3>
    <p>
      本报告基于 CANN-Bench 评测框架，Agent/Skill为CANNBot-skill，Harness为CANNBot-harness，BaseModel为DeepSeek V4 Pro。使用
      CANN-Bench全量基准测试数据集，针对 53 个算子进行了系统性评测，覆盖 Level 1 至 Level 4 共 4 个难度等级、
      1,060 个评测用例。涵盖编译正确性、功能精度和性能优化三个维度。
      整体通过率为 48.9%（518/1,060），总得分为 131（满分 5,300）。
    </p>
  </div>

  <!-- ============================================================ -->
  <!-- 1. OVERVIEW                                                   -->
  <!-- ============================================================ -->
  <div class="section">
    <h3><span class="sec-num">1.</span> Evaluation Dataset Overview / 评测集概述</h3>

    <p>
      CANN-Bench评测集是一个面向昇腾算子开发的全量基准测试数据集，用于系统性验证算子代码的
      编译正确性、功能精度和运行性能。评测集覆盖 53 个算子，分为 4 个难度等级，每个算子配套完整的
      规格描述（desc.md）、原型定义（proto.yaml）、PyTorch 参考实现（golden.py）和多场景测试用例
      （cases.yaml / cases.csv），为算子开发提供标准化的评测输入。
    </p>

    <h4>1.1 Difficulty Level Classification / 难度分级</h4>
    <div class="table-wrap">
      <table style="text-align:center">
        <caption>Table 1. Difficulty Levels and Operator Distribution</caption>
        <thead><tr>
          <th>Level</th><th>Operators</th><th>Cases</th><th>Typical Examples</th>
        </tr></thead>
        <tbody>
          <tr>
            <td class="col-name" style="text-align:center">Level 1 — 基础算子</td>
            <td>8</td>
            <td>160</td>
            <td>Exp, Gelu, Sigmoid, Mish, SwiGlu</td>
          </tr>
          <tr>
            <td class="col-name" style="text-align:center">Level 2 — 中级算子</td>
            <td>16</td>
            <td>320</td>
            <td>Softmax, Gather, ApplyAdamW, RmsNorm</td>
          </tr>
          <tr>
            <td class="col-name" style="text-align:center">Level 3 — 高级算子</td>
            <td>21</td>
            <td>420</td>
            <td>Conv2D, TopK, NMS, GroupedMatmul</td>
          </tr>
          <tr>
            <td class="col-name" style="text-align:center">Level 4 — 复杂算子</td>
            <td>8</td>
            <td>160</td>
            <td>GQA, LSTM, MHA, MLA</td>
          </tr>
        </tbody>
      </table>
    </div>

    <h4>1.2 Scoring System / 评分体系</h4>
    <p>
      评测采用三维度独立加权评估，权重 w_c=0.2（编译）、w_f=0.3（功能精度）、w_p=0.5（性能）。
      性能得分采用硬件锚点公式（HW-anchored），以 baseline_perf_us 和 t_hw_us 为参考锚点，
      评估候选 kernel 时间在校准基线时间与硬件理论上限之间的相对位置。
    </p>

    <div class="axis-viz">
      <div class="axis-card">
        <div class="axis-icon">&#9881;</div>
        <div class="axis-name">Compilation / 编译</div>
        <div class="axis-weight">w_c = 0.2 &ensp; max 20</div>
        <div class="axis-bar"><div class="axis-fill" style="width:20%;background:var(--accent)"></div></div>
        <div class="axis-detail"><strong>Contribution:</strong> w_c &middot; &delta;_pass &middot; 100</div>
        <div class="axis-detail">&delta;_pass &isin; {0, 1} — 一票否决制</div>
      </div>
      <div class="axis-card">
        <div class="axis-icon">&#9878;</div>
        <div class="axis-name">Function / 功能精度</div>
        <div class="axis-weight">w_f = 0.3 &ensp; max 30</div>
        <div class="axis-bar"><div class="axis-fill" style="width:30%;background:var(--score-high)"></div></div>
        <div class="axis-detail"><strong>Contribution:</strong> &Sigma;&delta;_acc,i &middot; w_f / N &middot; 100</div>
        <div class="axis-detail">MERE &lt; threshold 且 MARE &lt; 10&times;threshold（双指标判定）</div>
      </div>
      <div class="axis-card">
        <div class="axis-icon">&#9889;</div>
        <div class="axis-name">Performance / 性能</div>
        <div class="axis-weight">w_p = 0.5 &ensp; max 50</div>
        <div class="axis-bar"><div class="axis-fill" style="width:50%;background:var(--score-mid)"></div></div>
        <div class="axis-detail"><strong>Contribution:</strong> &Sigma;&delta;_acc,i &middot; w_p &middot; score_i / N &middot; 100</div>
      </div>
    </div>

    <div class="formula-block">
      <span class="eq-num">Eq. 1</span>
      <span class="eq-body">EachOperatorScore = [ w_c&middot;&delta;_pass + &Sigma;<sub>i</sub> &delta;_acc,i&middot;(w_f + w_p&middot;score_i) / N ] &middot; 100</span>
      <span class="eq-note">N = max(declared_cases, run_cases, 1)。编译失败时整算子计 0。</span>
    </div>
  </div>
  <!-- ============================================================ -->
  <!-- INSERTION POINT: Section 2+ will be appended dynamically -->
  <!-- ============================================================ -->