feat(evaluation): introduce full evaluation system for agents & workflows (suites, tasks, graders, metrics, benchmarks, UI)
Co-authored-by: Michael<michael.atamuk@huawei.com>
Co-authored-by: adi_amir<adi.amir1@huawei.com>
Co-authored-by: nizzan<nizzan.kimhi@huawei.com>
Co-authored-by: @aharonamir1<amir.aharon@huawei.com>
# message auto-generated for no-merge-commit merge:
!1023 merge evaluation into develop
feat(evaluation): introduce full evaluation system for agents & workflows (suites, tasks, graders, metrics, benchmarks, UI)
Created-by: michaelhuawei
Commit-by: Michael;michaelhuawei;aharonamir1;@aharonamir1;nikita-mee;nizzan;adi_amir
Merged-by: ZYQ5333
Description: <!--
Thanks for sending a pull request!
Here are some tips for you:
1) If this is your first time, please read our contributor guidelines:
[https://gitcode.com/openJiuwen/openJiuwen/blob/master/CONTRIBUTING.md](https://gitcode.com/openJiuwen/openJiuwen/blob/master/CONTRIBUTING.md)
2) If you want to contribute your code but don't know who will review and merge,
please add label openJiuwen-assistant to the pull request.
-->
**What type of PR is this?**
/kind feature /kind refactor
---
## **What does this PR do / why do we need it**
This PR introduces the **Evaluation System for Agents and Workflows**, a major new module that provides first‑class, systematic evaluation capabilities across all OpenJiuwen workflow patterns. It enables teams to measure correctness, reliability, semantic quality, latency, token usage, and regression behavior for any workflow or agent.
The system solves three long‑standing gaps:
1. **No regression detection** — previously no structured way to verify that workflow changes preserved correctness.
2. **No comparative measurement** — no shared metrics to compare versions of agents/workflows.
3. **No sampling support** — LLM nondeterminism required multi‑trial evaluation, which did not exist.
This feature adds a complete backend + frontend evaluation pipeline, including suites, tasks, graders, metrics, benchmark loading, and a full results UI.
---
## **Which issue(s) this PR fixes**
Fixes #<issue-number>
---
## **What scenarios were tested, and what were the verification results**
**Functional verification**
- Created evaluation suites, added tasks, updated tasks, deleted tasks.
- Loaded all seven benchmark YAML files; validated correct task creation.
- Ran evaluation against workflows and agents with deterministic, model-based, and code-based graders.
- Verified pattern detection across all six structural patterns (Routing, Chaining, Parallelization, Orchestrator‑Worker, Evaluator‑Optimizer, Memory Usage).
- Confirmed correct grader behavior: deterministic checks, LLM judge calls, code-based execution, weight aggregation.
- Confirmed metrics engine correctness: pass/fail, pass@k, pass^k, score distribution, latency stats, token usage, reliability, per-grader breakdown.
- Verified custom aggregate metrics execution, including error handling.
- Confirmed run lifecycle: RUNNING → COMPLETED/FAILED, immutability of completed runs.
- Verified large-suite behavior (50 tasks × 5 trials) and UI rendering of large trace sets.
**Performance verification**
- Execution engine calls scale linearly with tasks × trials.
- Model-based graders correctly issue LLM judge calls per trial.
- No regressions to existing workflow/agent execution performance.
**Reliability verification**
- Flakiness metric validated using multi-trial runs.
- Pattern detection validated with synthetic traces and real workflows.
- Code-based grader error paths tested (exceptions, invalid returns).
**Frontend verification**
- Full CRUD for suites and tasks.
- Run dialog correctly configures workflow/agent target and trial count.
- Results UI renders Overview, Metrics, Graders, and Traces tabs with correct visibility rules.
- Zustand store state transitions validated.
---
## **Self-checklist**
+ - [x] **Design**: Reviewed with maintainers; all comments addressed.
+ - [x] **Test**: Full UT/ST coverage for harness, graders, metrics, pattern validator, API, and frontend store.
+ - [x] **Verification**: PR description includes detailed verification results for feature, refactor, and bugfix aspects.
+ - [x] **Interface**: Adds new external API endpoints under /evaluation; no breaking changes to existing interfaces.
+ - [x] **Document**: Benchmark usage, suite/task schema, and evaluation workflow documented; docs PR prepared separately.
---
## **Special notes for reviewers**
- This module is **fully additive** — no existing endpoints or execution logic are modified.
- Code-based graders and custom metrics use exec(); this is an accepted constraint for v1 and will be hardened later.
- Large evaluation runs can produce multi‑MB result payloads; pagination is planned for a future release.
- Pattern detection is heuristic; tasks may override pattern_type explicitly.
See merge request: openJiuwen/agent-studio!1023
feat(evaluation): introduce full evaluation system for agents & workflows (suites, tasks, graders, metrics, benchmarks, UI)
Co-authored-by: Michael<michael.atamuk@huawei.com>
Co-authored-by: adi_amir<adi.amir1@huawei.com>
Co-authored-by: nizzan<nizzan.kimhi@huawei.com>
Co-authored-by: @aharonamir1<amir.aharon@huawei.com>
# message auto-generated for no-merge-commit merge:
!1023 merge evaluation into develop
feat(evaluation): introduce full evaluation system for agents & workflows (suites, tasks, graders, metrics, benchmarks, UI)
Created-by: michaelhuawei
Commit-by: Michael;michaelhuawei;aharonamir1;@aharonamir1;nikita-mee;nizzan;adi_amir
Merged-by: ZYQ5333
Description: <!--
Thanks for sending a pull request!
Here are some tips for you:
1) If this is your first time, please read our contributor guidelines:
[https://gitcode.com/openJiuwen/openJiuwen/blob/master/CONTRIBUTING.md](https://gitcode.com/openJiuwen/openJiuwen/blob/master/CONTRIBUTING.md)
2) If you want to contribute your code but don't know who will review and merge,
please add label openJiuwen-assistant to the pull request.
-->
**What type of PR is this?**
/kind feature /kind refactor
---
## **What does this PR do / why do we need it**
This PR introduces the **Evaluation System for Agents and Workflows**, a major new module that provides first‑class, systematic evaluation capabilities across all OpenJiuwen workflow patterns. It enables teams to measure correctness, reliability, semantic quality, latency, token usage, and regression behavior for any workflow or agent.
The system solves three long‑standing gaps:
1. **No regression detection** — previously no structured way to verify that workflow changes preserved correctness.
2. **No comparative measurement** — no shared metrics to compare versions of agents/workflows.
3. **No sampling support** — LLM nondeterminism required multi‑trial evaluation, which did not exist.
This feature adds a complete backend + frontend evaluation pipeline, including suites, tasks, graders, metrics, benchmark loading, and a full results UI.
---
## **Which issue(s) this PR fixes**
Fixes #<issue-number>
---
## **What scenarios were tested, and what were the verification results**
**Functional verification**
- Created evaluation suites, added tasks, updated tasks, deleted tasks.
- Loaded all seven benchmark YAML files; validated correct task creation.
- Ran evaluation against workflows and agents with deterministic, model-based, and code-based graders.
- Verified pattern detection across all six structural patterns (Routing, Chaining, Parallelization, Orchestrator‑Worker, Evaluator‑Optimizer, Memory Usage).
- Confirmed correct grader behavior: deterministic checks, LLM judge calls, code-based execution, weight aggregation.
- Confirmed metrics engine correctness: pass/fail, pass@k, pass^k, score distribution, latency stats, token usage, reliability, per-grader breakdown.
- Verified custom aggregate metrics execution, including error handling.
- Confirmed run lifecycle: RUNNING → COMPLETED/FAILED, immutability of completed runs.
- Verified large-suite behavior (50 tasks × 5 trials) and UI rendering of large trace sets.
**Performance verification**
- Execution engine calls scale linearly with tasks × trials.
- Model-based graders correctly issue LLM judge calls per trial.
- No regressions to existing workflow/agent execution performance.
**Reliability verification**
- Flakiness metric validated using multi-trial runs.
- Pattern detection validated with synthetic traces and real workflows.
- Code-based grader error paths tested (exceptions, invalid returns).
**Frontend verification**
- Full CRUD for suites and tasks.
- Run dialog correctly configures workflow/agent target and trial count.
- Results UI renders Overview, Metrics, Graders, and Traces tabs with correct visibility rules.
- Zustand store state transitions validated.
---
## **Self-checklist**
+ - [x] **Design**: Reviewed with maintainers; all comments addressed.
+ - [x] **Test**: Full UT/ST coverage for harness, graders, metrics, pattern validator, API, and frontend store.
+ - [x] **Verification**: PR description includes detailed verification results for feature, refactor, and bugfix aspects.
+ - [x] **Interface**: Adds new external API endpoints under /evaluation; no breaking changes to existing interfaces.
+ - [x] **Document**: Benchmark usage, suite/task schema, and evaluation workflow documented; docs PR prepared separately.
---
## **Special notes for reviewers**
- This module is **fully additive** — no existing endpoints or execution logic are modified.
- Code-based graders and custom metrics use exec(); this is an accepted constraint for v1 and will be hardened later.
- Large evaluation runs can produce multi‑MB result payloads; pagination is planned for a future release.
- Pattern detection is heuristic; tasks may override pattern_type explicitly.
See merge request: openJiuwen/agent-studio!1023
feat: runtime deploy异步执行，两次写表
Co-authored-by: 冯浩<fenghao55@h-partners.com>
# message auto-generated for no-merge-commit merge:
!870 merge develop-runtime into feature/runtime
feat: runtime deploy异步执行，两次写表
Created-by: michealswhite
Commit-by: 冯浩
Merged-by: ZYQ5333
Description: <!-- Thanks for sending a pull request! Here are some tips for you:
1) If this is your first time, please read our contributor guidelines: https://gitcode.com/openJiuwen/community/blob/master/CONTRIBUTING.md
2) If you want to contribute your code but don't know who will review and merge, please add label openjiuwen-assistant to the pull request, we will find and do it as soon as possible.
-->
**What type of PR is this?**
<!--
选择下面一种标签替换下方 /kind <label>，可选标签类型有：
- /kind bug
- /kind task
- /kind feature
- /kind refactor
- /kind clean_code
如PR描述不符合规范，修改PR描述后需要/check-pr重新检查PR规范。
-->
/kind feature
**Self-checklist**:（**请自检，在[ ]内打上x，我们将检视你的完成情况，否则会导致pr无法合入**）
+ - [x] **设计**：PR对应的方案是否已经经过Maintainer评审，方案检视意见是否均已答复并完成方案修改
+ - [x] **测试**：PR中的代码是否已有UT/ST测试用例进行充分的覆盖，新增测试用例是否随本PR一并上库或已经上库
+ - [x] **验证**：PR描述信息中是否已包含对该PR对应的Feature、Refactor、Bugfix的预期目标达成情况的详细验证结果描述
+ - [x] **接口**：是否涉及对外接口变更，相应变更已得到接口评审组织的通过，API对应的注释信息已经刷新正确
+ - [x] **文档**：是否涉及官网文档修改，如果涉及请及时提交资料到Doc仓
<!-- **Special notes for your reviewers**: -->
<!-- + - [ ] 是否导致无法前向兼容 -->
<!-- + - [ ] 是否涉及依赖的三方库变更 -->
See merge request: openJiuwen/agent-studio!870
feat : alembic db migration
Co-authored-by: cs123abc<chensheng63@huawei.com>
# message auto-generated for no-merge-commit merge:
!291 merge develop-20260121-basedon-20260116 into develop
feat : alembic db migration
Created-by: chensheng12345
Commit-by: cs123abc
Merged-by: openJiuwen-bot
Description: <!-- Thanks for sending a pull request! Here are some tips for you:
1) If this is your first time, please read our contributor guidelines: https://gitcode.com/openJiuwen/community/blob/master/CONTRIBUTING.md
2) If you want to contribute your code but don't know who will review and merge, please add label openjiuwen-assistant to the pull request, we will find and do it as soon as possible.
-->
**What type of PR is this?**
<!--
选择下面一种标签替换下方 /kind <label>，可选标签类型有：
- /kind bug
- /kind task
- /kind feature
- /kind refactor
- /kind clean_code
如PR描述不符合规范，修改PR描述后需要/check-pr重新检查PR规范。
-->
/kind feature
**Self-checklist**:（**请自检，在[ ]内打上x，我们将检视你的完成情况，否则会导致pr无法合入**）
+ - [x] **设计**：PR对应的方案是否已经经过Maintainer评审，方案检视意见是否均已答复并完成方案修改
+ - [x] **测试**：PR中的代码是否已有UT/ST测试用例进行充分的覆盖，新增测试用例是否随本PR一并上库或已经上库
+ - [x] **验证**：PR描述信息中是否已包含对该PR对应的Feature、Refactor、Bugfix的预期目标达成情况的详细验证结果描述
+ - [x] **接口**：是否涉及对外接口变更，相应变更已得到接口评审组织的通过，API对应的注释信息已经刷新正确
+ - [x] **文档**：是否涉及官网文档修改，如果涉及请及时提交资料到Doc仓
<!-- **Special notes for your reviewers**: -->
<!-- + - [ ] 是否导致无法前向兼容 -->
<!-- + - [ ] 是否涉及依赖的三方库变更 -->
See merge request: openJiuwen/agent-studio!291
feat(triggers): introduce Scheduled Triggers with cron, webhook, and polling automation for agents and workflows
Co-authored-by: jiangchenglin<jiangchenglin3@huawei.com>
Co-authored-by: michaelhuawei<michael.atamuk@huawei.com>
Co-authored-by: programmegirl<wangxin375@huawei.com>
# message auto-generated for no-merge-commit merge:
!1012 merge scheduled_triggers into develop
feat(triggers): introduce Scheduled Triggers with cron, webhook, and polling automation for agents and workflows
Created-by: michaelhuawei
Commit-by: Michael;jiangchenglin;michaelhuawei;programmegirl
Merged-by: ZYQ5333
Description: **What type of PR is this?**
/kind feature
---
## **What does this PR do / why do we need it**
This PR adds **Scheduled Triggers**, a major new automation capability that allows agents and workflows to run **without manual user action**.
Until now, every execution required a human to click “Run.” With this feature, OpenJiuwen becomes a fully automated orchestration platform capable of:
- Running tasks on a **cron schedule**
- Reacting to **external webhook events**
- Monitoring a URL and firing when **content changes**
This eliminates the need for users to build their own schedulers, polling scripts, or webhook relays. The platform now handles automation natively.
---
Which issue(s) this PR fixes:
Fixes [#810](https://gitcode.com/openJiuwen/agent-studio/issues/810)
---
## **Feature Overview**
### **Trigger Types**
| Type | How it fires | Typical use case |
|------|--------------|------------------|
| **Cron** | APScheduler fires at a cron-defined schedule | Daily reports, nightly sync |
| **Webhook** | External POST to a unique inbound URL | GitHub PR events, CI/CD |
| **Polling** | Periodic fetch; fires when content changes | RSS feeds, API monitors |
### **Trigger Targets**
- **Agent** — executed via agent_mgr.run()
- **Workflow** — executed via flow_mgr.run()
- Both use the same execution path as the interactive UI.
### **Input Payload**
Each trigger stores a JSON dict forwarded to every execution.
Polling triggers also inject:
```
_poll_hash: "<sha256 of fetched content>"
```
### **Execution Logs**
Every fire attempt is recorded with:
- Status: running, success, error, or skipped
- Fired by: scheduler, webhook, poll, or manual
- Duration
- Inputs snapshot
- Outputs or error message
- Polling metadata (poll_hash_seen)
---
## **Architecture & Implementation**
### **Backend**
- APScheduler embedded in FastAPI
- SQLAlchemy-backed job store (apscheduler_jobs)
- Trigger definitions stored in trigger table
- Execution logs stored in trigger_execution_log
- Scheduler rebuilt from DB on startup
- Webhook receiver implemented as a public FastAPI endpoint
- Polling uses requests in a thread to avoid blocking the event loop
### **Frontend**
- Full UI for creating, editing, activating, deactivating, deleting triggers
- Execution history viewer with trace links
- Cron expression helper with human-readable preview
- Webhook token + optional HMAC-SHA256 secret validation
- Polling status fields (last checked, last hash)
### **Security**
- Webhook endpoint always returns 200 to prevent token enumeration
- Optional HMAC-SHA256 signature verification
- Synthetic system_trigger user executes all trigger-fired runs
- No real user token required
- Polling content capped at 10 MB for hashing
### **Constraints**
- Cron and polling require the server to be running
- Missed cron fires coalesce once on next startup
- Only one scheduler instance should run (single-process APScheduler)
- Webhook triggers work across multiple workers (stateless)
---
## **Documentation**
A complete, detailed documentation file has been added:
📄 **docs/en/4.Development Guide/Scheduled Triggers.md**
➡️ **Testers MUST read this document before testing.**
It contains full UI guidance, API behavior, examples, troubleshooting, and edge-case notes.
---
## **Which scenarios were tested, and what were the verification results**
### **1. Cron triggers**
- Hourly, daily, weekly schedules
- Correct UTC evaluation
- Missed-fire coalescing on restart
- Verified execution logs and trace links
### **2. Webhook triggers**
- Valid token → fires execution
- Invalid token → silent 200 + no execution
- Optional HMAC-SHA256 signature validation
- Large payload handling
- Asynchronous execution via BackgroundTasks
### **3. Polling triggers**
- First poll always fires
- Subsequent polls fire only on content change
- skipped logs recorded correctly
- 10 MB content cap verified
- Hash stability tested across multiple content types
### **4. Agent and workflow execution**
- Trigger-fired runs use system_trigger user
- Inputs forwarded correctly
- Version selection (draft vs published) validated
### **5. UI behavior**
- Create, edit, activate, deactivate, delete
- Cron preview rendering
- Webhook token visibility rules
- Execution history pagination
### **6. Scheduler lifecycle**
- Jobs restored on startup
- Jobs removed on trigger deletion
- Shutdown sequence cleanly stops scheduler
All scenarios produced correct, stable results.
---
## **Self-checklist**
+ - [ ] **Design** reviewed and approved
+ - [X] **Backend** implementation complete
+ - [X] **Frontend** UI complete
+ - [ ] **Tests** added for cron, webhook, polling, and logs
+ - [X] **Documentation** added (testers must read it)
+ - [X] **No breaking changes** to existing agent/workflow execution
See merge request: openJiuwen/agent-studio!1012
fix: Default Value Type Adaptation for Input Components
Co-authored-by: daifeiping<daifeiping@huawei.com>
# message auto-generated for no-merge-commit merge:
!288 merge develop into develop
fix: Default Value Type Adaptation for Input Components
Created-by: daifeiping
Commit-by: daifeiping
Merged-by: openJiuwen-bot
Description: <!-- Thanks for sending a pull request! Here are some tips for you:
1) If this is your first time, please read our contributor guidelines: https://gitcode.com/openJiuwen/community/blob/master/CONTRIBUTING.md
2) If you want to contribute your code but don't know who will review and merge, please add label openjiuwen-assistant to the pull request, we will find and do it as soon as possible.
-->
**What type of PR is this?**
<!--
选择下面一种标签替换下方 /kind <label>，可选标签类型有：
- /kind bug
- /kind task
- /kind feature
- /kind refactor
- /kind clean_code
如PR描述不符合规范，修改PR描述后需要/check-pr重新检查PR规范。
-->
/kind <label>
**Self-checklist**:（**请自检，在[ ]内打上x，我们将检视你的完成情况，否则会导致pr无法合入**）
+ - [x] **设计**：PR对应的方案是否已经经过Maintainer评审，方案检视意见是否均已答复并完成方案修改
+ - [x] **测试**：PR中的代码是否已有UT/ST测试用例进行充分的覆盖，新增测试用例是否随本PR一并上库或已经上库
+ - [x] **验证**：PR描述信息中是否已包含对该PR对应的Feature、Refactor、Bugfix的预期目标达成情况的详细验证结果描述
+ - [x] **接口**：是否涉及对外接口变更，相应变更已得到接口评审组织的通过，API对应的注释信息已经刷新正确
+ - [x] **文档**：是否涉及官网文档修改，如果涉及请及时提交资料到Doc仓
<!-- **Special notes for your reviewers**: -->
<!-- + - [ ] 是否导致无法前向兼容 -->
<!-- + - [ ] 是否涉及依赖的三方库变更 -->
See merge request: openJiuwen/agent-studio!288
