| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
fix(agent): respond to the user's turn before lecturing (+ answer-content eval) (#699) * test(orchestration): add answer-content eval for agent first-reply answering The existing director question-answering eval only checks that the director ROUTES to the teacher; it never generates the dispatched agent's reply, so it cannot catch the case where routing is correct but the agent's first sentence drifts (greets / opens a lecture / pivots) and only a later turn answers. This adds an answer-content eval that runs the real agent-generate inputs (buildStructuredPrompt + convertMessagesToOpenAI), parses the structured output with the app's runtime parser, and uses an LLM judge for two signals: leads_with_answer (first sentence addresses the literal ask) and answered_anywhere (addressed at all). Their gap quantifies "drift-then-answer". A/B mirrors the answering-runner rule-13 strip: baseline removes the "Answering the User's Question" section from agent-system, with_rule is as-shipped. Scenarios are synthetic/anonymized (opening-lecture override, ignored format/capability/navigation request, ignored correction, frustration re-ask, adjacent pivot, vague-clarify) plus clean controls; no real user data. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(orchestration): give answer-content scenarios live-shaped classroom context The minimal store state (empty scenes, null stage, no profile) made the assembled agent prompt far thinner than the live agent-generate path and suppressed the slide-narration "lecture reflex" that drives opener drift. Each scenario now carries a topical slide scene (title + body + latex elements with ids), a stage (name/description/languageDirective), and a student profile, so buildStructuredPrompt emits a "# Current State", "# Student Profile" and "# Language" of the same shape and bulk as production. The runner reads this context from the scenario instead of a hardcoded empty state. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(agent): respond to the user's turn before continuing the lecture Generalize the agent-system "Answering the User's Question" section into "Responding to the User's Turn": the user's latest message always takes priority over continuing the planned lecture. Beyond factual questions it now explicitly covers the interaction requests agents were ignoring — navigation/pacing ("skip to next page"), format/language ("explain in Chinese"), capability requests it cannot fulfill ("make a video" -> say so + offer an alternative), corrections, frustration re-asks, and vague asks (ask one clarifying question). Strengthen the answer-content eval that measures this: - the navigation case now uses a 3-slide deck so "skip to next page" is well-posed; runner supports multi-scene state and strips the renamed section for the A/B; - judge is fairer to legitimate behavior: explaining primarily in the requested language while keeping technical terms standard counts; a verbal transition to the next slide counts as honoring navigation (no page-turn action exists). Result (gemini-3-flash-preview, n=10): mean leads-with-answer rises from 42% (section stripped) to 93% (as-shipped); all 12 scenarios pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(agent): don't over-respond to bare acknowledgements; guard empty eval set Code-review follow-ups on the "Responding to the User's Turn" change: - Prompt: the absolute "every user message needs a direct response / treat continuing the lecture as a failure" wording had no carve-out for bare acknowledgements ("ok" / "嗯" / "good" / "thanks"), risking the agent halting the lesson to manufacture a reply, and conflicted with agent-initiated discussions where no user turn exists. Add a bare-acknowledgement carve-out, allow a brief "好的"/"Sure" before the answer, and scope the override to actual requests (and to "if there is a user turn to respond to"). - Eval: answer-content-runner exited 0 with a PASS verdict and NaN% aggregates when the scenario set was empty (e.g. an EVAL_SCENARIO typo). Guard it: error and exit 1 when no scenario matches. Re-run unchanged otherwise (gemini-3-flash-preview, n=10): mean leads-with-answer 42% (stripped) -> 94% (as-shipped), 11/12 scenarios pass; the chem opener is variance-prone around the 70% bar and not over-tuned. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(eval): cr-loop cleanups on answer-content-runner - ScenarioAgentSpec extends the shared ScenarioAgent (./types) instead of re-declaring id/name/role/priority. - Drop the dead expectedPreFix plumbing from ScenarioResult (it was copied in but never read; it stays as an informational annotation in the scenarios). - extractTexts: replace the hand-rolled JSON fallback with the app's finalizeParser, which recovers plain-prose / unclosed-array output the same way the runtime does. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: tighten acknowledgement carve-out; don't surface action-JSON as text (cr round 2) Round-2 review follow-ups. Prompt: the bare-acknowledgement carve-out was too loose — it could swallow ack+request hybrids ("嗯，那为什么X?") as acks (re-introducing drift), double-listed "好的" as both an allowed lead-in and a no-response ack, and listed pacing words ("继续"/"go on") that collide with the navigation bullet. Restrict it to a STANDALONE acknowledgement with nothing else attached; route an embedded question/request to that request; route pacing words to the navigation bullet. Eval: extractTexts gated finalizeParser recovery on !state.jsonStarted, so an actions-only / unclosed array yields "no text" (the judge's intended empty case) instead of finalizeParser's raw-buffer fallback surfacing action JSON as fake speech. Re-run (gemini-3-flash-preview, n=5): no regression — 11/12 pass, meta-requests 100/100/80/100; the chem opener remains variance-prone around the 70% bar. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: address review — honest slide-nav, strict judge booleans, errors fail (cr) Review follow-ups from @cosarah: - Navigation (P2): agents have no slide-navigation action, so "do it" + a verbal transition let "skip to next page" pass the eval while the UI stayed on the same slide. Reframe prompt, judge, and scenario so a correct reply ACKNOWLEDGES the request and is HONEST that it cannot change the slide itself (continue verbally / tell the user how to navigate); pretending it flipped the slide no longer counts. - Judge booleans (P2): answer-content-judge coerced fields with Boolean(...), so a string "false" became true. Parse strictly (real bool or "true"/"false"); malformed output is flagged as an error. - Error accounting (P3): aggregate excluded errored samples from the denominator, letting a scenario pass on one good sample. Errors now count as failures (denominator = all requested samples). Re-run (gemini-3-flash-preview, n=10): navigation 0% -> 100% under the honest framing; aggregate leads-with-answer ~93%, no regression. Opener scenarios (math/chem) remain variance-prone near the 70% bar. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(eval): judge first-sentence lead; use canonical teacher actions (cr round 2) Second review round from @cosarah: - leads_with_answer was judged against texts[0] (the whole first JSON text block), so "Today we'll discuss parabolas. The formula is x=-b/(2a)." counted as leading even though the first sentence drifted. Now extract the first ~2 sentences as the "opening" and have the judge make the call: a brief acknowledgement of the user before the answer counts as leading, but a greeting / self-intro / topic preamble before the answer does not. (Two sentences + semantic judge handles "Sorry, let me clarify. m is inertial mass." vs "Welcome! …" without a brittle keyword list.) - The runner hardcoded TEACHER_ACTIONS, omitting canonical teacher actions (play_video, wb_draw_chart). Use getActionsForRole('teacher') so the available- actions section matches production. Re-run (gemini-3-flash-preview agent, n=10): 11/12 pass, aggregate leads ~90%. The chem opener now correctly fails (~10%): its "Welcome! Today we…" persona greets first, so at sentence granularity it is genuine first-sentence drift (answered-anywhere ~90%) — exactly what the stricter lead measurement should catch. Note: the judge model is deepseek-v4-pro in this run because the anthropic gateway is currently returning Bad Request in our environment; the eval is judge-model-agnostic via EVAL_JUDGE_MODEL. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(eval): judge the opening across all text blocks, not just texts[0] (cr round 3) Review follow-up from @cosarah: leadFromTexts only looked inside the first text block, so a valid multi-block reply like [{"type":"text","content":"Sure."},{"type":"text","content":"The derivative is 2x."}] was judged on "Sure." alone and failed leads_with_answer despite following the allowed acknowledgement-then-answer pattern. Build the opening from the first ~2 speech sentences across ALL ordered text chunks. Re-run (gemini-3-flash-preview, n=10, deepseek judge): 11/12 pass, aggregate leads ~93%. The chem opener stays sub-threshold (~50%) — its "Welcome! Today we…" persona greets before answering about half the time, a genuine first-sentence drift. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: 杨慎 <117187635+cosarah@users.noreply.github.com> | 2 个月前 | |
refactor(generation): consume the package everywhere and delete lib/generation (#1090) Part D of #1057 — the switch. All consumers import @openmaic/generation; lib/generation deleted (net -11,900 lines). PBL wiring preserved (single-call -> classified fallback -> injected loop), streaming outline route deduplicated onto buildOutlinePrompt, serverExternalPackages wired, and the change verified by a real-model four-kind end-to-end run against the production build with a rendered-classroom screenshot check. Closes #1057. | 29 天前 | |
refactor(generation): consume the package everywhere and delete lib/generation (#1090) Part D of #1057 — the switch. All consumers import @openmaic/generation; lib/generation deleted (net -11,900 lines). PBL wiring preserved (single-call -> classified fallback -> injected loop), streaming outline route deduplicated onto buildOutlinePrompt, serverExternalPackages wired, and the change verified by a real-model four-kind end-to-end run against the production build with a rendered-classroom screenshot check. Closes #1057. | 29 天前 | |
refactor(eval): unify outline-language and whiteboard-layout harness (#453) * feat(eval): add resolveEvalModel helper with fail-fast Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval): add createRunDir helper with path sanitization Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval): add markdown report helpers * refactor(eval): move language test cases under eval/outline-language * feat(eval): add outline-language types * feat(eval): add outline-language LLM judge * feat(eval): add outline-language reporter * feat(eval): add outline-language runner entry * chore(eval): add eval:outline-language pnpm script * refactor(eval): adopt shared createRunDir and drop gpt-4o fallbacks in whiteboard runner * refactor(eval): drop redundant non-null assertions after narrowing Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(eval): remove SCORER_MODEL_DEFAULT hardcoded gpt-4o fallback * chore(eval): delete tests/generation and clean up vitest/gitignore config * docs(eval): explain why outline-language runner pre-validates env vars Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(eval): escape pipe chars in markdown summary table cells LLM judge output may contain | which breaks GFM table rendering. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: 杨慎 <117187635+cosarah@users.noreply.github.com> | 4 个月前 | |
chore(packages): publish the @openmaic/* SDK family to npm (#778) (#780) * chore(packages): publish the @openmaic/* SDK family to npm (#778) Prepares the @openmaic/{dsl,renderer,importer} family for its first npm publish, and moves the SDK packages onto the @openmaic scope. Why the scope move: the @maic org name is unavailable on npm (an unscoped `maic` package already holds the name), so @maic/* is not claimable. @openmaic matches the project name, the scope is free, and the repo already ships an @openmaic/docs package — so the SDK family now lines up with that convention. - rename @maic/{dsl,renderer,importer} -> @openmaic/* across packages, the workspace glob, the package dir, and all import sites; lockfile regenerated - renderer: add publishConfig (public, registry.npmjs.org) — was missing, so a scoped publish would default to the wrong registry / restricted access - importer: add a files allowlist (dist, README, LICENSE) and drop the fragile .npmignore blacklist that shipped src; add an exports map so ESM consumers resolve dist/index.js instead of falling back to the .cjs main - all three: add a prepublishOnly build (+ test/typecheck) guard so a publish can never ship a stale or empty dist - add a tag-triggered publish workflow with npm provenance, pinned by name to the three @openmaic packages so the vendored forks (mathml2omml, pptxgenjs) are never published Refs #778, #720 (Phase 1). * fix(packages): address cross-review on the @openmaic publish prep Cross-review (Claude /code-review + codex) on this PR surfaced: - renderer's advertised CJS entry was broken: it keeps @openmaic/dsl external and imports a runtime enum from it, but dsl is ESM-only (no `require` condition), so `require('@openmaic/renderer')` would throw ERR_PACKAGE_PATH_NOT_EXPORTED. Make renderer ESM-only: drop the `.cjs` rollup output, `main` now points at the ESM build, and the `require` conditions are removed from `exports`. (importer is unaffected — it bundles dsl, so its CJS build still works.) - prepublishOnly re-ran the test suite during `pnpm -r publish`, so a flaky test after dsl had already published gave a non-atomic partial release. Reduce prepublishOnly to a build-only guard (never ship stale/empty dist) and move the real test/typecheck gate into the workflow, before any publish. - document that an @openmaic/* tag publishes the whole family via `pnpm -r` (pnpm skips already-published versions); the tag is a release marker, not a per-package gate. Verified: dsl + renderer + importer build; renderer emits ESM only (0 .cjs), all exports entries resolve; `npm pack` ships dist + README + LICENSE with no src leak; frozen-lockfile passes. Refs #778. * style: reflow @openmaic/dsl type imports past print-width after rename The @maic -> @openmaic rename lengthened two single-line type imports past prettier's 100-col width; prettier --check flagged them. Pure formatting. Refs #778. * docs(importer): mark @openmaic/importer browser-only (cr-loop accepted limitation) codex cross-review flagged that the published @openmaic/importer throws `XMLHttpRequest is not a constructor` when loaded in a pure Node process — its rollup build is browser-targeted (`nodeResolve({browser:true})` + a browser pdf.js build). The app only consumes it client-side ('use client'), so this is by design. Document it as an accepted limitation: prominent browser-only note in the README and a `browser` field in the manifest. Refs #778. | 2 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 2 个月前 | ||
| 29 天前 | ||
| 29 天前 | ||
| 4 个月前 | ||
| 2 个月前 |