| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
feat: whiteboard layout quality eval harness (#425) * feat(eval): add state manager bridging ActionEngine for eval Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(eval): add shared types for whiteboard layout eval harness * feat(eval): add SSE chat client for whiteboard eval * feat(eval): add Playwright capture module for whiteboard screenshots * feat(eval): add VLM scorer for whiteboard layout evaluation * feat(eval): add report generator for whiteboard eval results * feat(eval): add 8 constructed scenarios for whiteboard layout eval * feat(eval): add minimal whiteboard render page for Playwright screenshots Creates app/eval/whiteboard/page.tsx — a headless client page that seeds the stageStore with a synthetic slide scene, exposes window.__setElements() for Playwright to inject PPTElement[], and renders them via ScreenElement inside a 1000×562.5px white canvas. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(eval): add main runner for whiteboard layout eval Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(eval): add eval:whiteboard script, install tsx, gitignore results * fix(eval): fix TS errors, lint, and prettier formatting * fix(eval): address code review — add cue_user/empty turn guards, validate VLM output, fix empty array crash * refactor(eval): replace synthetic scenarios with realistic ones - Replace 8 generic scenarios with 6 that match real usage patterns - Multi-agent discussion with short user replies (嗯, 明白了, 继续) - Include real slide scene data as initialStoreState - Generated agent configs with Chinese names and proper roles - Cover: physics, math, finance, primary school, economics, medical Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: extract shared agent loop from use-chat-sessions Extract the core agent loop logic into lib/chat/agent-loop.ts as a pure async function with callback injection. Both the frontend React hook and the eval harness now share the same loop — SSE parsing, exit conditions (END/cue_user/empty turns/max turns), and director state accumulation. The frontend wires StreamBuffer callbacks for UI pacing; the eval wires ActionEngine + message accumulation for headless execution. If loop logic changes in the shared module, both consumers automatically stay in sync. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(eval): use project LLM infrastructure, fix eval page and model config - Rewrite scorer to use resolveModel() + generateText() from AI SDK instead of raw fetch — supports all providers (OpenAI, Google, Anthropic) - Model config via env vars (EVAL_CHAT_MODEL, EVAL_SCORER_MODEL), matching the pattern from outline-language eval - Fix eval page: bootstrap store before SceneProvider mounts - Fix __dirname for tsx CJS mode - Remove --api-key/--scorer-model CLI args (use env vars instead) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(eval): organize results by model/timestamp * fix: remove double turnCount increment in shared agent loop The extracted agent-loop.ts had both `turnCount = directorState?.turnCount ?? turnCount + 1` (line 190) and a redundant `turnCount++` (line 215), causing multi-agent scenarios to hit maxTurns at half the expected number of iterations. Also removes unused processSSEStream import from use-chat-sessions.ts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(eval): organize screenshots by scenario subdirectory Results structure: results/<model>/<timestamp>/<scenario>/run0_turn1.png Report files stay at the timestamp level. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(eval): revise scorer rubric and add rescore mode - Replace space_utilization with rendering_correctness and content_completeness - Rubric now evaluates from a teacher's perspective (empty space is normal) - readability emphasizes font size consistency - Add --rescore flag to re-score existing screenshots without re-running chat - Increase maxOutputTokens to 2000, add JSON parse error recovery - Score errors no longer abort the entire scenario Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(eval): sharpen scorer rubric with teacher-perspective examples The rubric now catches specific classroom whiteboard failure modes: - overlap now explicitly penalizes writing over existing content when empty space is available (spatial planning failure) - rendering_correctness calls out diagram accuracy (e.g., parabola drawn as V-shape), raw subscripts (G_x), Chinese inside LaTeX math mode - content_completeness emphasizes canvas edge clipping and bare unlabeled diagrams - readability penalizes text styled as UI components (gray card backgrounds) - overall instructed to weight overlap and rendering_correctness more heavily - explicit note to ignore the "N" page UI element Also increases maxOutputTokens to 3000 since longer rubric produces longer justifications. Reporter now guards against null scores (scorer failures no longer crash report generation). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address code review findings Critical fixes: - use-chat-sessions: restore agent_end handling, currentMessageId fallback for text_delta/action events with missing messageId, and re-throw on SSE error events (previously silently pushed to buffer only). - eval runner: serialize ActionEngine executions via promise chain. void-fire-and-forget raced with ensureWhiteboardOpen's 2s delay and could insert elements out of order or before the first element was committed to the store. Important fixes: - CHAT_MODEL default: 'openai/gpt-4o-mini' -> 'openai:gpt-4o-mini' (parseModelString splits on ':', not '/'). - CheckpointResult.score is now VlmScore | null; removed the 'as unknown as' cast that hid the null contract from consumers. - Delete dead code: eval/whiteboard-layout/chat-client.ts and components/chat/process-sse-stream.ts (both unused after the shared agent loop refactor). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: 杨慎 <117187635+cosarah@users.noreply.github.com> | 4 个月前 | |
chore(packages): publish the @openmaic/* SDK family to npm (#778) (#780) * chore(packages): publish the @openmaic/* SDK family to npm (#778) Prepares the @openmaic/{dsl,renderer,importer} family for its first npm publish, and moves the SDK packages onto the @openmaic scope. Why the scope move: the @maic org name is unavailable on npm (an unscoped `maic` package already holds the name), so @maic/* is not claimable. @openmaic matches the project name, the scope is free, and the repo already ships an @openmaic/docs package — so the SDK family now lines up with that convention. - rename @maic/{dsl,renderer,importer} -> @openmaic/* across packages, the workspace glob, the package dir, and all import sites; lockfile regenerated - renderer: add publishConfig (public, registry.npmjs.org) — was missing, so a scoped publish would default to the wrong registry / restricted access - importer: add a files allowlist (dist, README, LICENSE) and drop the fragile .npmignore blacklist that shipped src; add an exports map so ESM consumers resolve dist/index.js instead of falling back to the .cjs main - all three: add a prepublishOnly build (+ test/typecheck) guard so a publish can never ship a stale or empty dist - add a tag-triggered publish workflow with npm provenance, pinned by name to the three @openmaic packages so the vendored forks (mathml2omml, pptxgenjs) are never published Refs #778, #720 (Phase 1). * fix(packages): address cross-review on the @openmaic publish prep Cross-review (Claude /code-review + codex) on this PR surfaced: - renderer's advertised CJS entry was broken: it keeps @openmaic/dsl external and imports a runtime enum from it, but dsl is ESM-only (no `require` condition), so `require('@openmaic/renderer')` would throw ERR_PACKAGE_PATH_NOT_EXPORTED. Make renderer ESM-only: drop the `.cjs` rollup output, `main` now points at the ESM build, and the `require` conditions are removed from `exports`. (importer is unaffected — it bundles dsl, so its CJS build still works.) - prepublishOnly re-ran the test suite during `pnpm -r publish`, so a flaky test after dsl had already published gave a non-atomic partial release. Reduce prepublishOnly to a build-only guard (never ship stale/empty dist) and move the real test/typecheck gate into the workflow, before any publish. - document that an @openmaic/* tag publishes the whole family via `pnpm -r` (pnpm skips already-published versions); the tag is a release marker, not a per-package gate. Verified: dsl + renderer + importer build; renderer emits ESM only (0 .cjs), all exports entries resolve; `npm pack` ships dist + README + LICENSE with no src leak; frozen-lockfile passes. Refs #778. * style: reflow @openmaic/dsl type imports past print-width after rename The @maic -> @openmaic rename lengthened two single-line type imports past prettier's 100-col width; prettier --check flagged them. Pure formatting. Refs #778. * docs(importer): mark @openmaic/importer browser-only (cr-loop accepted limitation) codex cross-review flagged that the published @openmaic/importer throws `XMLHttpRequest is not a constructor` when loaded in a pure Node process — its rollup build is browser-targeted (`nodeResolve({browser:true})` + a browser pdf.js build). The app only consumes it client-side ('use client'), so this is by design. Document it as an accepted limitation: prominent browser-only note in the README and a `browser` field in the manifest. Refs #778. | 2 个月前 | |
refactor(whiteboard): file-based prompts + geometry conflict detection (#485) * feat(prompts): declare whiteboard role prompt IDs + whiteboard-reference snippet * feat(prompts): add whiteboard role template skeletons + reference snippet stub * refactor(orchestration): move buildWhiteboardGuidelines to role-specific markdown templates * docs(prompts): whiteboard guidelines now in markdown templates * feat(prompts): whiteboard-reference canvas + JSON output sections * feat(prompts): whiteboard-reference action schemas for text/shape/line * feat(prompts): whiteboard-reference action schemas for latex/chart/table * feat(prompts): whiteboard-reference action schemas for code/delete/clear/close * feat(prompts): whiteboard-reference LaTeX JSON escape section * feat(prompts): whiteboard-reference bounds + overlap section * feat(prompts): whiteboard-reference font + latex height tables * feat(prompts): whiteboard-reference pre-output checklist * feat(prompts): expand teacher whiteboard template * feat(prompts): expand assistant whiteboard template * feat(prompts): expand student whiteboard template * test(prompts): assert whiteboard-reference sections reach every role prompt * tune(prompts): cap teacher element budget at 4/turn, encourage wb_clear on crowded board * tune(prompts): simplify teacher template — single conservative rule + state-awareness emphasis * tune(prompts): add LaTeX-height × text-fontSize pairing table for visual weight consistency * feat(orchestration): preserve image parts from UIMessage to AI SDK ModelMessage * feat(eval): attach prior-turn whiteboard screenshot as user message image part Gated behind EVAL_ATTACH_PRIOR_SCREENSHOT=1 env. When on, captures a screenshot after every turn (not just checkpoints) and attaches it as a file part with mediaType:'image/png' on the next user message. Teacher prompt gets a new 'Prior-state image' section telling the agent to treat it as direct visual feedback. * fix(orchestration): strip data: prefix before passing image to AI SDK Vercel AI SDK's streamText/generateText treats ImagePart.image as a URL to fetch when it's a string. data: URLs fail the http/https scheme check and throw AI_DownloadError. Strip the data URL prefix and pass raw base64 with mediaType separately — AI SDK treats base64 strings as data content. * refactor(whiteboard): add conflict summarizer, drop prior-screenshot feature Eval across flash (image-on vs off × repeat-3) and pro (same) showed the prior-screenshot image-input feature gives ~0 net benefit: +0.4 overall on flash and −0.6 on pro. The image feedback loop's theoretical upside (agent self-correcting from visual) didn't materialize — weak models couldn't act on the image and strong models "found things to fix" and over-corrected. Meanwhile programmatic geometry detection on the raw JSON cleanly lifted overlap from 6.3 → 8.1 on flash. So: remove the image-input plumbing, keep a programmatic equivalent. ## Removed (reverts 0e07e9e, b5675bc, image half of 097ec2f) - lib/orchestration/ai-sdk-adapter.ts: revert multimodal content passthrough - lib/orchestration/summarizers/message-converter.ts: revert image part extraction; user content is back to string-only - lib/orchestration/summarizers/conversation-summary.ts: revert multimodal content flattening - lib/orchestration/director-graph.ts: revert HumanMessage multimodal cast - tests/orchestration/message-converter.test.ts: deleted (was purely image-conversion tests) - eval/whiteboard-layout/runner.ts: drop image attach, vision gate, per-turn screenshot capture; revert to checkpoint-only capture - Role templates (wb-teacher / wb-assistant / wb-student): strip "Prior-state image" sections ## Added - lib/orchestration/summarizers/whiteboard-conflicts.ts: pure geometry detector. Computes bbox IoU, line segment vs bbox intersection, and canvas edge clipping on the current whiteboard state. Renders a text block for inclusion in the system prompt. - lib/orchestration/summarizers/state-context.ts: wires the conflict block in after the whiteboard element listing. - Role templates: replace image-reading guidance with concise "Layout conflicts" sections pointing at the computed list, with a strong anti-action anchor so agents don't self-trigger wb_clear when the list is empty. ## Eval instrumentation (orthogonal) - EVAL_ENABLE_THINKING=1 opt-in to enable model thinking per-request. app/api/chat/route.ts reads body.thinking; runner.ts forwards it. Default chat path stays on enabled:false for latency. - Per-turn wall-clock timing: eval/whiteboard-layout/{runner,reporter, types}.ts capture and report mean/p50/p95/total turn latency. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(whiteboard): address code-review feedback W1: drop stale "prior-state image" reference from conflict-block prose (lib/orchestration/summarizers/whiteboard-conflicts.ts) — the image feature was removed; the text now says "real visible problem on the current board". W3: type `thinking` on StatelessChatRequest instead of reading it through a cast. `app/api/chat/route.ts` now accesses `body.thinking` directly; frontend callers can discover the field via TS completion. W4: unify canvas height constant to 563 (matching the actual rendered pixel count from capture.ts). The geometry detector's CANVAS_HEIGHT, the snippet's Dimensions / coordinate system / layout-guide text, and examples all now agree. N6: add tests/orchestration/whiteboard-conflicts.test.ts — 17 cases covering empty input, bbox overlap threshold (30%, 50%, 100%, plus 10% sub-threshold), line-crosses-bbox (through, endpoint-inside, path-above), canvas clipping on all 4 edges, exact-edge placement (not reported), malformed elements (skipped not crashed), and output format. All 43 tests pass; tsc --noEmit clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(eval): finish canvas-height 563 unification Second-pass code review caught one residual 562.5 in app/eval/whiteboard/page.tsx that the first cleanup (commit 7a006bb) missed. The 0.5px disagreement was harmless in practice (Playwright captures rounded to 563), but inconsistent with the geometry detector and the prompt snippet now agreeing on 563. All 43 tests pass; tsc clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: prettier format whiteboard-conflicts.test + eval/runner CI's `pnpm check` (prettier --check) caught two unformatted files: - eval/whiteboard-layout/runner.ts - tests/orchestration/whiteboard-conflicts.test.ts Auto-fixed via `pnpm prettier --write`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: 杨慎 <117187635+cosarah@users.noreply.github.com> | 4 个月前 | |
test+fix(orchestration): rip maxTurns; restore answer-the-user's-question behavior (#598 follow-up) (#599) * test(orchestration): add premature-END regression eval (#598 follow-up) Adds eval/orchestration/ following the outline-language pattern: a runner that A/B-tests the director against the same scenario with two prompt variants — current main (post-#554) and a synthesised pre-#554 baseline that strips the role-aware summary labels AND the new system.md rules 10/11/12 together. Pass criterion is now framed as a regression guard rather than a fixture discrimination test: every scenario's post-fix END rate must stay below EVAL_END_THRESHOLD (default 20%). The pre-vs-post Δ is reported as informational data, since #554's reviewer feedback was that earlier fixtures didn't discriminate. Empirical finding across 7 shipped model configs (gpt-4.1-mini, gpt-4o-mini, gpt-5.4-nano, qwen-plus, qwen3.5-flash, deepseek-chat, deepseek-v4-flash, gemini-2.5-flash, claude-haiku-4-5) with 5 scenarios modelled on #511 (incl. the exact Tiananmen-3D objection trace, soft pushback after a long resolved-looking discussion, topic pivot, brief acknowledgement, and explicit teacher-signals-end-then-user-objects): no scenario produced a non-zero END rate in either variant. Useful data for the #554/#598 discussion — the prompt-layer rules don't measurably change behavior on these shipped models with these prompts, but the eval is now wired so future regressions show up. * test(orchestration): add director question-answering eval (#598 follow-up) Adds an A/B eval probing whether the director routes correctly when the conversation contains an unanswered user question — both in the first-turn drift case (agents started answering but drifted onto adjacent topics, no frustration yet) and in the escalated frustration case (user has already complained multiple times). The prod symptom this targets ("答非所问"): agents reply on adjacent topics rather than answering the literal question, and the director keeps picking peer agents for "variety" instead of routing back to the teacher to actually answer. Scenarios: 5 synthesized cases across math, biology, English grammar, physics, and calculus. 3 are first-turn drift (no frustration yet), 2 include frustration signals. Each captures: user asks specific question, agents reply on adjacent topics, director is asked to pick the next move. Decision rule-judged into USER | TEACHER | OTHER_AGENT | END: - USER → ✓ valid (cue user to clarify) - TEACHER → ✓ valid (re-route to teacher to answer) - OTHER_AGENT → ✗ wrong (peer-agent "variety" routing — the bug) - END → ✗ wrong A/B: - baseline : current director template with rule 13 stripped - with_rule : current director template as-shipped Pass = with_rule.correctRate ≥ EVAL_PASS_THRESHOLD (default 0.7). Pre-vs-post Δ is reported as informational only. Note this commit only adds the eval; the rule it strips/restores in the A/B is injected in the next commit. * fix(orchestration): inject director rule for answering user questions Adds rule 13 to lib/prompts/templates/director/system.md so the director recognizes when the conversation contains an unanswered user question and routes appropriately — to the teacher (literal role match) by default, or to USER cue when the question is ambiguous or the user has already complained multiple times. This addresses the "答非所问" symptom: when agents drift onto adjacent topics, the director keeps picking peer agents for "variety" rather than routing back to the teacher to actually answer. The rule is preventive (applies whenever the question is unanswered, including the first turn after drift) rather than reactive-only (waiting for the user to express frustration). Rule wording uses constrained-output framing: literal `role: teacher` match (not "best fit" / "highest priority" / "most knowledgeable"), and an explicit list of forbidden choices (peer agents, END). Earlier softer wordings let the model pick the assistant when it seemed more topically relevant; the literal-match wording prevents that. Validated by the question-answering eval added in the previous commit. On google:gemini-3-flash-preview, 5 samples per variant across 5 synthesized scenarios: | Scenario | baseline | with_rule | |----------------------------------------------------|----------|-----------| | math_quadratic_axis_drift_no_frustration | 100% | 100% | | bio_dark_reaction_drift_no_frustration | 80% | 100% | | english_team_isare_drift_no_frustration | 20% | 100% | | physics_inertial_mass_escalated_frustration | 0% | 100% | | calculus_product_rule_drift_no_frustration | 60% | 100% | | **mean** | **52%** | **100%** | 3 of 5 scenarios reproduce the bug at baseline (correct rate ≤ 60%); the rule lifts every scenario to 100%. The remaining 2 baselines are already high — the rule doesn't regress them. * fix(orchestration): teach agents to answer the user's literal question Adds an "Answering the User's Question" section to the agent system prompt. Before this, the agent SP had no rule for direct-answering — the persona, role guideline, current-state context, and length guidelines all pushed the agent to advance the lesson and inspire thought, with no explicit directive to first answer the user's literal question. The result observed in prod ("答非所问"): when the user asks a specific question (a formula, a yes/no, a term), agents reply with adjacent concepts, examples, or follow-up questions — but never the literal answer. The new section directs the agent, on every response where the user message contains a question, to: - lead with the concrete answer to the literal question, - not pivot to an adjacent topic even when it seems pedagogically richer, - treat "inspire thought" and peer-differentiation guidance as applying only AFTER the literal question is answered, - say "I don't know" directly when uncertain rather than answering a different question, - on frustration signals from prior turns, look back at the original question (before the frustration) and answer THAT specifically. This is the agent-side counterpart to the director rule landed in the previous commit. Director routes to the teacher when the question is unanswered; the teacher now has explicit guidance for what to do when dispatched. * refactor+fix(orchestration): rip maxTurns end-to-end; distinguish errors from completion Two coupled cleanups in one commit since the outcome-handler rewrite naturally falls out of removing the client-side max-turns loop. The director graph used `maxTurns: turnCount + 1` to force "one director→agent cycle per request" — a hack expressing the single-round contract as a turn limit. This commit removes the hack and the surrounding user-facing setting, replacing implicit intent with explicit graph topology. Server: - `director-graph.ts`: drop `maxTurns` from state, drop the hard-cap check at the top of `directorNode`, change `agent_generate → director` edge to `agent_generate → END`. Single-round contract is now expressed by topology. - `lib/types/chat.ts`: drop `maxTurns` and `currentTurn` from `SessionConfig`; drop `maxTurns?` from `CreateSessionRequest.trigger`. Client: - `lib/store/settings.ts`: drop `maxTurns` state, `setMaxTurns` action, the `'10'` default, and the localStorage migration path. - `components/agent/agent-bar.tsx`: drop the entire compact-stepper UI block plus the unused MessageSquare/Minus/Plus lucide imports. - `components/settings/agent-settings.tsx`: drop the AgentSettings prop and the multi-agent-only Input block. - `lib/chat/agent-loop.ts`: drop `maxTurns` parameter, change `while (turnCount < maxTurns)` to `while (true)` controlled by inner exits, drop `'max_turns'` reason from `AgentLoopOutcome`. - `components/chat/use-chat-sessions.ts`: drop the maxTurns lookup, drop the trailing arg to runAgentLoop, drop vestigial `maxTurns: 0` / `currentTurn: 0` from session creation. - `eval/whiteboard-layout/runner.ts`: drop unused MAX_AGENT_TURNS plus the trailing arg. i18n: drop `settings.maxTurns` and `settings.maxTurnsDesc` from six locales (en-US, zh-CN, zh-TW, ja-JP, ar-SA, ru-RU). Motivation for removing the user-facing setting: the LLM director controls round length via cue_user / END decisions, so a numeric "max discussion turns" knob is redundant with rule-driven routing. When the director prompt was buggy, the cap masked the bug by force- ending; once director routing is fixed, the cap just removes user agency. Before, every non-`cue_user` outcome of the agent loop mapped to `status: 'completed'` in `use-chat-sessions.ts`. That conflated four distinct paths: - `end`: director chose END (legitimate completion) - `empty_turns`: two consecutive agents returned no text or actions (parser failure, model timeout, etc.) - `no_done`: SSE stream ended without a `done` event (infrastructure failure) - `aborted`: user pressed stop (handled elsewhere) The new switch routes `end` to completed, `empty_turns` and `no_done` through `clearLiveSessionAfterError` so the user sees an inline System message describing what went wrong instead of a quiet "completed". - `lib/types/chat.ts`: add `'error'` to `SessionStatus`. - `components/chat/use-chat-sessions.ts`: rewrite the outcome handler as a switch; have `clearLiveSessionAfterError` set `status: 'error'` so all entry points (the runAgentLoop outcome AND existing throw paths from fetchChat / SSE error events) converge on the same visible state. - `components/chat/session-list.tsx`: add `AlertCircle` red icon for the new error status. - i18n: add `chat.error.emptyAgentResponses` and `chat.error.streamInterrupted` strings to all six locales. This is the in-product fix that the prod investigation surfaced as an independent root cause of the #511 "discussion ended unexpectedly" symptom — separate from the director and agent prompt fixes in the earlier commits of this PR. * fix(chat): drop dead settingsState binding; wire eval:orchestration:answering script Two minor CR follow-ups: - components/chat/use-chat-sessions.ts: the `settingsState` lookup at the top of runAgentLoopFn was the entry point for the maxTurns lookup ripped in the previous commit; with maxTurns gone, the binding is unused and ESLint flags it. Drop the line. (Other uses of useSettingsStore.getState() elsewhere in this file are unaffected.) - package.json: the answering eval runner had no convenience script, while the premature-end runner did. Add eval:orchestration:answering for symmetry and discoverability. * fix(orchestration): route ambiguous/unaddressed user turns to teacher, never USER cue Addresses reviewer issues 2 & 3 (ambiguous "帮我看一下这个" / "Can you take a look at this?"): the director was cueing USER for vague input, which makes no agent speak — the user faces dead air with nothing to react to ("nobody responds"); and when an agent did speak it guessed a topic instead of asking what the user meant. Root cause was the previous rule 13, which offered `{"next_agent":"USER"}` as a valid response to an ambiguous user turn. A USER cue is the wrong tool here: it returns control to the user without any agent speaking. The right behavior is to dispatch the teacher, who either answers or — if the message is too vague — asks a clarifying question out loud. Director (`director/system.md` rule 13): - When the most recent user turn is unaddressed (clear question, vague request, OR frustration signal), the only acceptable output is the literal `role: teacher` agent id. - USER cue is now explicitly forbidden as a response to a user turn (reserved for rule 7, where an agent has just asked the user something). - Tightened the END prohibition: a high turn count or a thoroughly-discussed topic does NOT mean the specific question was answered. Added rule 5 (don't-drag-on) to the override list, since that pressure was driving the director to END on well-discussed-but-unanswered turns. Agent (`agent-system/system.md`): - Added a clause to "Answering the User's Question": if the user's message is too vague to answer (no clear referent), ask ONE short clarifying question with a concrete option or two — do not guess a topic and lecture, do not stay silent. Eval (`answering-runner.ts` + `scenarios/answering.json`): - Reclassified USER as wrong (dead air); only TEACHER counts as correct now (teacher answers, or asks a clarifying question for vague input). - Added two ambiguity scenarios: a bare "帮我看一下这个" opener, and a vague "再帮我看看这个吧" follow-up after a real Q&A. - Made the rule-13 strip regex match by rule number rather than heading text, so it survives heading rewrites. On google:gemini-3-flash-preview, 10 samples per variant, all 7 scenarios PASS (with_rule mean ~97%, baseline ~57%). The two ambiguity scenarios: ambiguous_look_at_this 100%/100%, ambiguous_vague_followup 20%→100%. Manually verified the agent side: for "帮我看一下这个" the teacher now replies "…你想讨论的是具体的代码逻辑，还是 Agent 的决策流程图呢?" (clarifying question + concrete options) instead of guessing. Note: reviewer issue 1 (follow-up not answered) could not be reproduced on this branch across 7 configurations; it is consistent with the deployed environment running older code and/or provider API flakiness, which the C7 error-vs-completion distinction in this PR already surfaces instead of a silent 已结束. * docs(eval): fix stale physics scenario description after USER→wrong reclassification The physics_inertial_mass scenario description still said "USER-cue signal — but TEACHER re-answer is also valid", which contradicts the new classifier (USER is dead air = wrong; only TEACHER is correct). Cosmetic only — the description field is printed into the report, never used in classification. | 3 个月前 | |
refactor(eval): unify outline-language and whiteboard-layout harness (#453) * feat(eval): add resolveEvalModel helper with fail-fast Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval): add createRunDir helper with path sanitization Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval): add markdown report helpers * refactor(eval): move language test cases under eval/outline-language * feat(eval): add outline-language types * feat(eval): add outline-language LLM judge * feat(eval): add outline-language reporter * feat(eval): add outline-language runner entry * chore(eval): add eval:outline-language pnpm script * refactor(eval): adopt shared createRunDir and drop gpt-4o fallbacks in whiteboard runner * refactor(eval): drop redundant non-null assertions after narrowing Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(eval): remove SCORER_MODEL_DEFAULT hardcoded gpt-4o fallback * chore(eval): delete tests/generation and clean up vitest/gitignore config * docs(eval): explain why outline-language runner pre-validates env vars Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(eval): escape pipe chars in markdown summary table cells LLM judge output may contain | which breaks GFM table rendering. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: 杨慎 <117187635+cosarah@users.noreply.github.com> | 4 个月前 | |
chore(packages): publish the @openmaic/* SDK family to npm (#778) (#780) * chore(packages): publish the @openmaic/* SDK family to npm (#778) Prepares the @openmaic/{dsl,renderer,importer} family for its first npm publish, and moves the SDK packages onto the @openmaic scope. Why the scope move: the @maic org name is unavailable on npm (an unscoped `maic` package already holds the name), so @maic/* is not claimable. @openmaic matches the project name, the scope is free, and the repo already ships an @openmaic/docs package — so the SDK family now lines up with that convention. - rename @maic/{dsl,renderer,importer} -> @openmaic/* across packages, the workspace glob, the package dir, and all import sites; lockfile regenerated - renderer: add publishConfig (public, registry.npmjs.org) — was missing, so a scoped publish would default to the wrong registry / restricted access - importer: add a files allowlist (dist, README, LICENSE) and drop the fragile .npmignore blacklist that shipped src; add an exports map so ESM consumers resolve dist/index.js instead of falling back to the .cjs main - all three: add a prepublishOnly build (+ test/typecheck) guard so a publish can never ship a stale or empty dist - add a tag-triggered publish workflow with npm provenance, pinned by name to the three @openmaic packages so the vendored forks (mathml2omml, pptxgenjs) are never published Refs #778, #720 (Phase 1). * fix(packages): address cross-review on the @openmaic publish prep Cross-review (Claude /code-review + codex) on this PR surfaced: - renderer's advertised CJS entry was broken: it keeps @openmaic/dsl external and imports a runtime enum from it, but dsl is ESM-only (no `require` condition), so `require('@openmaic/renderer')` would throw ERR_PACKAGE_PATH_NOT_EXPORTED. Make renderer ESM-only: drop the `.cjs` rollup output, `main` now points at the ESM build, and the `require` conditions are removed from `exports`. (importer is unaffected — it bundles dsl, so its CJS build still works.) - prepublishOnly re-ran the test suite during `pnpm -r publish`, so a flaky test after dsl had already published gave a non-atomic partial release. Reduce prepublishOnly to a build-only guard (never ship stale/empty dist) and move the real test/typecheck gate into the workflow, before any publish. - document that an @openmaic/* tag publishes the whole family via `pnpm -r` (pnpm skips already-published versions); the tag is a release marker, not a per-package gate. Verified: dsl + renderer + importer build; renderer emits ESM only (0 .cjs), all exports entries resolve; `npm pack` ships dist + README + LICENSE with no src leak; frozen-lockfile passes. Refs #778. * style: reflow @openmaic/dsl type imports past print-width after rename The @maic -> @openmaic rename lengthened two single-line type imports past prettier's 100-col width; prettier --check flagged them. Pure formatting. Refs #778. * docs(importer): mark @openmaic/importer browser-only (cr-loop accepted limitation) codex cross-review flagged that the published @openmaic/importer throws `XMLHttpRequest is not a constructor` when loaded in a pure Node process — its rollup build is browser-targeted (`nodeResolve({browser:true})` + a browser pdf.js build). The app only consumes it client-side ('use client'), so this is by design. Document it as an accepted limitation: prominent browser-only note in the README and a `browser` field in the manifest. Refs #778. | 2 个月前 | |
chore(packages): publish the @openmaic/* SDK family to npm (#778) (#780) * chore(packages): publish the @openmaic/* SDK family to npm (#778) Prepares the @openmaic/{dsl,renderer,importer} family for its first npm publish, and moves the SDK packages onto the @openmaic scope. Why the scope move: the @maic org name is unavailable on npm (an unscoped `maic` package already holds the name), so @maic/* is not claimable. @openmaic matches the project name, the scope is free, and the repo already ships an @openmaic/docs package — so the SDK family now lines up with that convention. - rename @maic/{dsl,renderer,importer} -> @openmaic/* across packages, the workspace glob, the package dir, and all import sites; lockfile regenerated - renderer: add publishConfig (public, registry.npmjs.org) — was missing, so a scoped publish would default to the wrong registry / restricted access - importer: add a files allowlist (dist, README, LICENSE) and drop the fragile .npmignore blacklist that shipped src; add an exports map so ESM consumers resolve dist/index.js instead of falling back to the .cjs main - all three: add a prepublishOnly build (+ test/typecheck) guard so a publish can never ship a stale or empty dist - add a tag-triggered publish workflow with npm provenance, pinned by name to the three @openmaic packages so the vendored forks (mathml2omml, pptxgenjs) are never published Refs #778, #720 (Phase 1). * fix(packages): address cross-review on the @openmaic publish prep Cross-review (Claude /code-review + codex) on this PR surfaced: - renderer's advertised CJS entry was broken: it keeps @openmaic/dsl external and imports a runtime enum from it, but dsl is ESM-only (no `require` condition), so `require('@openmaic/renderer')` would throw ERR_PACKAGE_PATH_NOT_EXPORTED. Make renderer ESM-only: drop the `.cjs` rollup output, `main` now points at the ESM build, and the `require` conditions are removed from `exports`. (importer is unaffected — it bundles dsl, so its CJS build still works.) - prepublishOnly re-ran the test suite during `pnpm -r publish`, so a flaky test after dsl had already published gave a non-atomic partial release. Reduce prepublishOnly to a build-only guard (never ship stale/empty dist) and move the real test/typecheck gate into the workflow, before any publish. - document that an @openmaic/* tag publishes the whole family via `pnpm -r` (pnpm skips already-published versions); the tag is a release marker, not a per-package gate. Verified: dsl + renderer + importer build; renderer emits ESM only (0 .cjs), all exports entries resolve; `npm pack` ships dist + README + LICENSE with no src leak; frozen-lockfile passes. Refs #778. * style: reflow @openmaic/dsl type imports past print-width after rename The @maic -> @openmaic rename lengthened two single-line type imports past prettier's 100-col width; prettier --check flagged them. Pure formatting. Refs #778. * docs(importer): mark @openmaic/importer browser-only (cr-loop accepted limitation) codex cross-review flagged that the published @openmaic/importer throws `XMLHttpRequest is not a constructor` when loaded in a pure Node process — its rollup build is browser-targeted (`nodeResolve({browser:true})` + a browser pdf.js build). The app only consumes it client-side ('use client'), so this is by design. Document it as an accepted limitation: prominent browser-only note in the README and a `browser` field in the manifest. Refs #778. | 2 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 4 个月前 | ||
| 2 个月前 | ||
| 4 个月前 | ||
| 3 个月前 | ||
| 4 个月前 | ||
| 2 个月前 | ||
| 2 个月前 |