0
代码
代码
Issues
Pull Requests
讨论
Wiki
分析
0
chenjiayan2025chenjiayan2025Add verifier data cache remote and bench docs
2be1d67e创建于 25 天前历史提交
文件最后提交记录最后更新时间
cpu
!1797 merge sol into br_agents feat(op): support SOL-ExecBench format in OpTaskBuilder and tests Created-by: yiyanzhi_akane1 Commit-by: yiyanzhi_akane1 Merged-by: yiyanzhi_akane1 Description: **What type of PR is this?** /kind feature **What does this PR do / why do we need it**: 本 PR 在 br_agents 之上引入 **SOL-ExecBench** 相关能力，并补充 **可复现/备份用的数据集与下载脚本**，便于在网络连接存在问题时的评测数据下载。 主要内容包括： 1. **备份数据集与文档**：在 benchmark/sol-execbench/ 增加 README、download.sh（及 .gitignore 调整），用于拉取或备份 SOL-ExecBench 数据；与 benchmark/SPEC.md 索引同步。 2. **OpTaskBuilder 与 SOL 格式**：支持将用户需求生成为 SOL-ExecBench 所需的 definition.json / workload.jsonl / reference.py 等结构；保留 op_task_builder_workflow 单轮/多轮入口；更新 build_op_task.j2 与相关 state/task 衔接。 3. **SOL 正确性与验证**：新增 sol_verifier.py、verify_sol_template.j2、sol_correctness.py；修正 SOL 校验基础设施，并覆盖 Ascend 等后端；kernel_verifier 小改动以接入 SOL 路径；放宽对 get_inputs 等硬编码要求以符合真实 SOL 参考实现。 4. **示例与复现**：examples/kernel_related/ 下增加 CPU/GPU 批跑与单 case 脚本；mock_sol_relu 最小样例；reproduce/wip/run_sol_bench_batch.py 批跑入口。 5. **测试**：test_bench_sol_cpu.py、test_bench_sol_triton_cuda.py、tests/op/st/sol_bench/test_op_builder_sol.py 覆盖 bench 与 OpTaskBuilder SOL 流程。 **Which issue(s) this PR fixes**: Fix issues: [#366](https://gitcode.com/mindspore/akg/issues/366) **Special notes for your reviewers**: - **数据与脚本**：sol-execbench/download.sh 与 README 的路径、依赖（如 wget/curl）请在目标环境确认；大文件通常仍由 .gitignore 排除，仅脚本与说明入库。 - **工作流**：op_task_builder_workflow.py 为上游曾删除、本分支为 SOL 测试与工具链保留的封装，若与全局 workflow 策略冲突请指出合并方式。 - **验证**：SOL 验证依赖模板与 sol_correctness；Ascend/CUDA/CPU 行为差异请在有设备的环境回归 test_bench_sol_*。 - **提交历史**：包含从「后端 SOL 初始化」到「备份数据集」的多条 commit，建议按模块审阅 op/verifier、benchmark/sol-execbench、examples/kernel_related。 --- ### 变更概览 ``` akg_agents/.gitignore | 5 +- akg_agents/benchmark/SPEC.md | 1 + akg_agents/benchmark/sol-execbench/README.md | 61 ++++ akg_agents/benchmark/sol-execbench/download.sh | 60 ++++ .../kernel_related/cpu/run_sol_bench_cpu_cpp.py | 75 +++++ .../kernel_related/gpu/run_single_sol_case.py | 61 ++++ .../gpu/run_sol_bench_gpu_triton_batch.py | 182 +++++++++++ .../kernel_related/mock_sol_relu/definition.json | 35 +++ .../kernel_related/mock_sol_relu/reference.py | 5 + .../kernel_related/mock_sol_relu/workload.jsonl | 2 + .../python/akg_agents/op/agents/op_task_builder.py | 57 ++-- .../op/langgraph_op/op_task_builder_state.py | 7 +- .../python/akg_agents/op/langgraph_op/task.py | 9 +- .../prompts/op_task_builder/build_op_task.j2 | 16 +- .../op/resources/templates/verify_sol_template.j2 | 176 +++++++++++ .../op/resources/utils/sol_correctness.py | 110 +++++++ akg_agents/python/akg_agents/op/verifier/SPEC.md | 1 + .../akg_agents/op/verifier/kernel_verifier.py | 8 +- .../python/akg_agents/op/verifier/sol_verifier.py | 112 +++++++ .../op/workflows/op_task_builder_workflow.py | 348 +++++++++++++++++++++ akg_agents/reproduce/wip/run_sol_bench_batch.py | 107 +++++++ akg_agents/tests/op/bench/test_bench_sol_cpu.py | 66 ++++ .../tests/op/bench/test_bench_sol_triton_cuda.py | 67 ++++ .../tests/op/st/sol_bench/test_op_builder_sol.py | 74 +++++ 24 files changed, 1611 insertions(+), 34 deletions(-) ``` ### 关键变更说明 | 区域 | 说明 | |------|------| | benchmark/sol-execbench/ | SOL-ExecBench 备份/下载说明与脚本 | | op/agents, op/langgraph_op, op/workflows | OpTaskBuilder SOL 输出与工作流 | | op/verifier, op/resources | SOL 验证模板、正确性工具与 sol_verifier | | examples/kernel_related, reproduce/wip | 运行示例与批跑脚本 | | tests/op | SOL bench 与 OpTaskBuilder ST 测试 | ### Commit 列表（origin_gitcode/br_agents..HEAD） - a50041d8 add backup sol execbench dataset - 15b07167 fix(op/sol): remove hardcoded get_inputs requirement for SOL format - 704446d4 fix(op/sol): fix sol verification infrastructure and add ascend support - 12b10832 add tests/op/bench/test_bench_sol_triton_cuda.py - d4a8f96c feat(op): support SOL-ExecBench format in OpTaskBuilder and tests - 91dcc7c0 init support sol execbench for backends ### 遗留问题 当前缺少相关的性能测试工具与验证，待后续补充。 See merge request: mindspore/akg!17972 个月前
gpu
refactor: unify benchmark downloads and remove submodules 1 个月前
mock_sol_relu
fix(op/sol): remove hardcoded get_inputs requirement for SOL format - Update OpTaskBuilder prompt to make get_inputs optional based on custom_inputs_entrypoint - Update test_op_builder_sol.py to conditionally check for get_inputs 2 个月前
skill_evolution
update logs path 1 个月前
bench_lite_common.py
feat: bench_lite multi-mode runner 增强与安全加固 - 新增 performance / full 模式：支持性能评测、评分、排行榜、submission 提取 - 新增 --team-name 路径穿越防御（validate_team_name） - 新增 --backend all 与 --arch/--dsl/--backend-name 互斥校验 - 新增 --backends 参数（仅 --backend all 模式，支持子集选择与去重） - 统一 payload schema：config 包含 team_name/workflow/backend_name/backends - 修复 box-drawing 字符乱码（替换为 ASCII 分隔线） - 子进程隔离性能评测（multiprocessing spawn 模式，超时 terminate/kill） - 更新 RUNNER.md 文档 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> 2 个月前
run_cuda_to_ascend_conversion.py
update docs for br_agents update akg_agents docs mv old docs add v2 docs 3 个月前
run_cuda_to_ascend_evolve.py
update configs 3 个月前
run_kernel_agent.py
Enhance logging in wrap_agent_with_logging to include both run_llm and run_llm_with_tools. Introduce a helper function for context details retrieval, improving structured logging of LLM interactions. 3 个月前
run_kernel_profile.py
refactor: rm old yaml and examples 3 个月前
run_torch_adaptive_search_triton_ascend.py
feat: enhance adaptive search and evolution scripts for SOL benchmarks - Added support for SOL-ExecBench input mode in run_torch_adaptive_search_triton_ascend.py and run_torch_evolve_triton_ascend.py. - Introduced functions to extract operator names and task descriptions from SOL dataset directories. - Updated configuration handling to include bench_type for adaptive search and evolution processes. - Enhanced task generation to accommodate SOL-specific requirements, improving flexibility for benchmarking. - Updated documentation and usage examples for clarity on new input modes. 1 个月前
run_torch_ascend_triton_batch_attention.py
update run attention examples Made-with: Cursor 2 个月前
run_torch_bench_lite.py
feat: bench_lite multi-mode runner 增强与安全加固 - 新增 performance / full 模式：支持性能评测、评分、排行榜、submission 提取 - 新增 --team-name 路径穿越防御（validate_team_name） - 新增 --backend all 与 --arch/--dsl/--backend-name 互斥校验 - 新增 --backends 参数（仅 --backend all 模式，支持子集选择与去重） - 统一 payload schema：config 包含 team_name/workflow/backend_name/backends - 修复 box-drawing 字符乱码（替换为 ASCII 分隔线） - 子进程隔离性能评测（multiprocessing spawn 模式，超时 terminate/kill） - 更新 RUNNER.md 文档 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> 2 个月前
run_torch_evolve_triton_ascend.py
feat: enhance adaptive search and evolution scripts for SOL benchmarks - Added support for SOL-ExecBench input mode in run_torch_adaptive_search_triton_ascend.py and run_torch_evolve_triton_ascend.py. - Introduced functions to extract operator names and task descriptions from SOL dataset directories. - Updated configuration handling to include bench_type for adaptive search and evolution processes. - Enhanced task generation to accommodate SOL-specific requirements, improving flexibility for benchmarking. - Updated documentation and usage examples for clarity on new input modes. 1 个月前
run_torch_npu_triton_single.py
feat: 完善attention kernel生成框架并添加测试工具 - Attention kernel生成与优化：添加AI生成代码框架、昇腾平台测试、float16数据类型优化及CPU兼容性改进 - 测试工具：添加AKG Kernels Bench Lite批量测试脚本和torch CUDA Triton批量attention测试 - 文档更新：重构triton-cuda-attention SKILL文档至v2.0，添加OpenMP并行编程约束文档 - 架构重构：将verifier模块从core层迁移至op层 - 修复与清理：修复A100 flash attention数据库引用、API 403问题，清理临时测试文件 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> 2 个月前
run_torch_npu_triton_single_with_cache.py
Add verifier data cache remote and bench docs 25 天前