0
代码
代码
Issues
Pull Requests
讨论
Wiki
分析
0
ZYQ5333ZYQ5333feat(evaluation): introduce full evaluation system for agents & workflows (suites, tasks, graders, metrics, benchmarks, UI)
a4862177创建于 28 天前历史提交
文件最后提交记录最后更新时间
feat_v0_1_2_init_sqlite_ops_b4f4c6589bc5.py
feat: idempotent migrate script modification and fix agent model error Co-authored-by: cs123abc<chensheng63@huawei.com> # message auto-generated for no-merge-commit merge: !653 merge develop-20260210 into develop feat: idempotent migrate script modification and fix agent model error Created-by: chensheng12345 Commit-by: cs123abc Merged-by: ZYQ5333 Description: <!-- Thanks for sending a pull request! Here are some tips for you: 1) If this is your first time, please read our contributor guidelines: https://gitcode.com/openJiuwen/openJiuwen/blob/master/CONTRIBUTING.md 2) If you want to contribute your code but don't know who will review and merge, please add label openJiuwen-assistant to the pull request, we will find and do it as soon as possible. --> **What type of PR is this?** <!-- Choose one label from bug, task, feature and refactor, and replace <label> below the comment block. If this pr is not only bugfix/task/feature and also a refactor, you can append /kind refactor label after /kind bug, /kind task and /kind feature. --> /kind refactor **What does this PR do / why do we need it**: * Need to describe clearly **Which issue(s) this PR fixes**: <!-- *Automatically closes linked issue when PR is merged. Usage: Fixes #<issue number>, or Fixes (paste link of issue). --> Fixes # **What scenarios were tested, and what were the verification results（Function, performance, reliability, etc.）**： * Need to describe clearly **Self-checklist**:（**Please check carefully,and mark an x in the [] brackets. We will review your completion status.**） + - [x] **Design**: Has the solution corresponding to the PR been reviewed by the Maintainer, and have all review comments been replied to and revised + - [x] **Test**: Has the code in the PR been fully covered by UT/ST test cases, and have the newly added test cases been uploaded to the repository along with this PR or already uploaded. + - [x] **Verification**: Does the PR description contains a detailed description of the verification results regarding the achievement of the expected goals for the Feature, Refactor, and Bugfix to this PR. + - [x] **Interface**: Does it involve changes to external interfaces? The corresponding changes have been approved by the interface review organization, and the annotation information for the API has been correctly refreshed. + - [x] **Document**: Does it involve modifications to the official website documentation? If so, please submit the materials to the Doc repository in a timely manner. <!-- **Special notes for your reviewers**: --> <!-- + - [ ] Whether it causes forward compatibility failure --> <!-- + - [ ] Whether the dependent third-party library change is involved --> See merge request: openJiuwen/agent-studio!6533 个月前
feat_v0_1_3_update_sqlite_ops_20260126_f6e49cd8c97d.py
feat: idempotent migrate script modification and fix agent model error Co-authored-by: cs123abc<chensheng63@huawei.com> # message auto-generated for no-merge-commit merge: !653 merge develop-20260210 into develop feat: idempotent migrate script modification and fix agent model error Created-by: chensheng12345 Commit-by: cs123abc Merged-by: ZYQ5333 Description: <!-- Thanks for sending a pull request! Here are some tips for you: 1) If this is your first time, please read our contributor guidelines: https://gitcode.com/openJiuwen/openJiuwen/blob/master/CONTRIBUTING.md 2) If you want to contribute your code but don't know who will review and merge, please add label openJiuwen-assistant to the pull request, we will find and do it as soon as possible. --> **What type of PR is this?** <!-- Choose one label from bug, task, feature and refactor, and replace <label> below the comment block. If this pr is not only bugfix/task/feature and also a refactor, you can append /kind refactor label after /kind bug, /kind task and /kind feature. --> /kind refactor **What does this PR do / why do we need it**: * Need to describe clearly **Which issue(s) this PR fixes**: <!-- *Automatically closes linked issue when PR is merged. Usage: Fixes #<issue number>, or Fixes (paste link of issue). --> Fixes # **What scenarios were tested, and what were the verification results（Function, performance, reliability, etc.）**： * Need to describe clearly **Self-checklist**:（**Please check carefully,and mark an x in the [] brackets. We will review your completion status.**） + - [x] **Design**: Has the solution corresponding to the PR been reviewed by the Maintainer, and have all review comments been replied to and revised + - [x] **Test**: Has the code in the PR been fully covered by UT/ST test cases, and have the newly added test cases been uploaded to the repository along with this PR or already uploaded. + - [x] **Verification**: Does the PR description contains a detailed description of the verification results regarding the achievement of the expected goals for the Feature, Refactor, and Bugfix to this PR. + - [x] **Interface**: Does it involve changes to external interfaces? The corresponding changes have been approved by the interface review organization, and the annotation information for the API has been correctly refreshed. + - [x] **Document**: Does it involve modifications to the official website documentation? If so, please submit the materials to the Doc repository in a timely manner. <!-- **Special notes for your reviewers**: --> <!-- + - [ ] Whether it causes forward compatibility failure --> <!-- + - [ ] Whether the dependent third-party library change is involved --> See merge request: openJiuwen/agent-studio!6533 个月前
feat_v0_1_4_add_evaluation_tables_20260314_a8c3f2b9e1d5.py
feat(evaluation): introduce full evaluation system for agents & workflows (suites, tasks, graders, metrics, benchmarks, UI) Co-authored-by: Michael<michael.atamuk@huawei.com> Co-authored-by: adi_amir<adi.amir1@huawei.com> Co-authored-by: nizzan<nizzan.kimhi@huawei.com> Co-authored-by: @aharonamir1<amir.aharon@huawei.com> # message auto-generated for no-merge-commit merge: !1023 merge evaluation into develop feat(evaluation): introduce full evaluation system for agents & workflows (suites, tasks, graders, metrics, benchmarks, UI) Created-by: michaelhuawei Commit-by: Michael;michaelhuawei;aharonamir1;@aharonamir1;nikita-mee;nizzan;adi_amir Merged-by: ZYQ5333 Description: <!-- Thanks for sending a pull request! Here are some tips for you: 1) If this is your first time, please read our contributor guidelines: [https://gitcode.com/openJiuwen/openJiuwen/blob/master/CONTRIBUTING.md](https://gitcode.com/openJiuwen/openJiuwen/blob/master/CONTRIBUTING.md) 2) If you want to contribute your code but don't know who will review and merge, please add label openJiuwen-assistant to the pull request. --> **What type of PR is this?** /kind feature /kind refactor --- ## **What does this PR do / why do we need it** This PR introduces the **Evaluation System for Agents and Workflows**, a major new module that provides first‑class, systematic evaluation capabilities across all OpenJiuwen workflow patterns. It enables teams to measure correctness, reliability, semantic quality, latency, token usage, and regression behavior for any workflow or agent. The system solves three long‑standing gaps: 1. **No regression detection** — previously no structured way to verify that workflow changes preserved correctness. 2. **No comparative measurement** — no shared metrics to compare versions of agents/workflows. 3. **No sampling support** — LLM nondeterminism required multi‑trial evaluation, which did not exist. This feature adds a complete backend + frontend evaluation pipeline, including suites, tasks, graders, metrics, benchmark loading, and a full results UI. --- ## **Which issue(s) this PR fixes** Fixes #<issue-number> --- ## **What scenarios were tested, and what were the verification results** **Functional verification** - Created evaluation suites, added tasks, updated tasks, deleted tasks. - Loaded all seven benchmark YAML files; validated correct task creation. - Ran evaluation against workflows and agents with deterministic, model-based, and code-based graders. - Verified pattern detection across all six structural patterns (Routing, Chaining, Parallelization, Orchestrator‑Worker, Evaluator‑Optimizer, Memory Usage). - Confirmed correct grader behavior: deterministic checks, LLM judge calls, code-based execution, weight aggregation. - Confirmed metrics engine correctness: pass/fail, pass@k, pass^k, score distribution, latency stats, token usage, reliability, per-grader breakdown. - Verified custom aggregate metrics execution, including error handling. - Confirmed run lifecycle: RUNNING → COMPLETED/FAILED, immutability of completed runs. - Verified large-suite behavior (50 tasks × 5 trials) and UI rendering of large trace sets. **Performance verification** - Execution engine calls scale linearly with tasks × trials. - Model-based graders correctly issue LLM judge calls per trial. - No regressions to existing workflow/agent execution performance. **Reliability verification** - Flakiness metric validated using multi-trial runs. - Pattern detection validated with synthetic traces and real workflows. - Code-based grader error paths tested (exceptions, invalid returns). **Frontend verification** - Full CRUD for suites and tasks. - Run dialog correctly configures workflow/agent target and trial count. - Results UI renders Overview, Metrics, Graders, and Traces tabs with correct visibility rules. - Zustand store state transitions validated. --- ## **Self-checklist** + - [x] **Design**: Reviewed with maintainers; all comments addressed. + - [x] **Test**: Full UT/ST coverage for harness, graders, metrics, pattern validator, API, and frontend store. + - [x] **Verification**: PR description includes detailed verification results for feature, refactor, and bugfix aspects. + - [x] **Interface**: Adds new external API endpoints under /evaluation; no breaking changes to existing interfaces. + - [x] **Document**: Benchmark usage, suite/task schema, and evaluation workflow documented; docs PR prepared separately. --- ## **Special notes for reviewers** - This module is **fully additive** — no existing endpoints or execution logic are modified. - Code-based graders and custom metrics use exec(); this is an accepted constraint for v1 and will be hardened later. - Large evaluation runs can produce multi‑MB result payloads; pagination is planned for a future release. - Pattern detection is heuristic; tasks may override pattern_type explicitly. See merge request: openJiuwen/agent-studio!102328 天前
feat_v0_1_5_add_pattern_types_20260427_c3d5e8f2a1b7.py
feat(evaluation): introduce full evaluation system for agents & workflows (suites, tasks, graders, metrics, benchmarks, UI) Co-authored-by: Michael<michael.atamuk@huawei.com> Co-authored-by: adi_amir<adi.amir1@huawei.com> Co-authored-by: nizzan<nizzan.kimhi@huawei.com> Co-authored-by: @aharonamir1<amir.aharon@huawei.com> # message auto-generated for no-merge-commit merge: !1023 merge evaluation into develop feat(evaluation): introduce full evaluation system for agents & workflows (suites, tasks, graders, metrics, benchmarks, UI) Created-by: michaelhuawei Commit-by: Michael;michaelhuawei;aharonamir1;@aharonamir1;nikita-mee;nizzan;adi_amir Merged-by: ZYQ5333 Description: <!-- Thanks for sending a pull request! Here are some tips for you: 1) If this is your first time, please read our contributor guidelines: [https://gitcode.com/openJiuwen/openJiuwen/blob/master/CONTRIBUTING.md](https://gitcode.com/openJiuwen/openJiuwen/blob/master/CONTRIBUTING.md) 2) If you want to contribute your code but don't know who will review and merge, please add label openJiuwen-assistant to the pull request. --> **What type of PR is this?** /kind feature /kind refactor --- ## **What does this PR do / why do we need it** This PR introduces the **Evaluation System for Agents and Workflows**, a major new module that provides first‑class, systematic evaluation capabilities across all OpenJiuwen workflow patterns. It enables teams to measure correctness, reliability, semantic quality, latency, token usage, and regression behavior for any workflow or agent. The system solves three long‑standing gaps: 1. **No regression detection** — previously no structured way to verify that workflow changes preserved correctness. 2. **No comparative measurement** — no shared metrics to compare versions of agents/workflows. 3. **No sampling support** — LLM nondeterminism required multi‑trial evaluation, which did not exist. This feature adds a complete backend + frontend evaluation pipeline, including suites, tasks, graders, metrics, benchmark loading, and a full results UI. --- ## **Which issue(s) this PR fixes** Fixes #<issue-number> --- ## **What scenarios were tested, and what were the verification results** **Functional verification** - Created evaluation suites, added tasks, updated tasks, deleted tasks. - Loaded all seven benchmark YAML files; validated correct task creation. - Ran evaluation against workflows and agents with deterministic, model-based, and code-based graders. - Verified pattern detection across all six structural patterns (Routing, Chaining, Parallelization, Orchestrator‑Worker, Evaluator‑Optimizer, Memory Usage). - Confirmed correct grader behavior: deterministic checks, LLM judge calls, code-based execution, weight aggregation. - Confirmed metrics engine correctness: pass/fail, pass@k, pass^k, score distribution, latency stats, token usage, reliability, per-grader breakdown. - Verified custom aggregate metrics execution, including error handling. - Confirmed run lifecycle: RUNNING → COMPLETED/FAILED, immutability of completed runs. - Verified large-suite behavior (50 tasks × 5 trials) and UI rendering of large trace sets. **Performance verification** - Execution engine calls scale linearly with tasks × trials. - Model-based graders correctly issue LLM judge calls per trial. - No regressions to existing workflow/agent execution performance. **Reliability verification** - Flakiness metric validated using multi-trial runs. - Pattern detection validated with synthetic traces and real workflows. - Code-based grader error paths tested (exceptions, invalid returns). **Frontend verification** - Full CRUD for suites and tasks. - Run dialog correctly configures workflow/agent target and trial count. - Results UI renders Overview, Metrics, Graders, and Traces tabs with correct visibility rules. - Zustand store state transitions validated. --- ## **Self-checklist** + - [x] **Design**: Reviewed with maintainers; all comments addressed. + - [x] **Test**: Full UT/ST coverage for harness, graders, metrics, pattern validator, API, and frontend store. + - [x] **Verification**: PR description includes detailed verification results for feature, refactor, and bugfix aspects. + - [x] **Interface**: Adds new external API endpoints under /evaluation; no breaking changes to existing interfaces. + - [x] **Document**: Benchmark usage, suite/task schema, and evaluation workflow documented; docs PR prepared separately. --- ## **Special notes for reviewers** - This module is **fully additive** — no existing endpoints or execution logic are modified. - Code-based graders and custom metrics use exec(); this is an accepted constraint for v1 and will be hardened later. - Large evaluation runs can produce multi‑MB result payloads; pagination is planned for a future release. - Pattern detection is heuristic; tasks may override pattern_type explicitly. See merge request: openJiuwen/agent-studio!102328 天前