| Files | Last commit | Last update |
|---|---|---|
docs: eval 23b at 1.25.0 — karaoke-style word + faststart-fix coverage (#280) Continues eval 23 (closed partial at 1.19.3, evals/results/iteration-23-partial.json) with real-execution prompts for the two features shipped since: caption.py --karaoke-style word (#276) and the +faststart fixes (#275/#277/#279). - evals/agent_prompts_24.json: two new prompts, kw1 (karaoke-style word, phrased after the #276 reporter's own complaint that the highlight "just changed colour") and ff1 (a YouTube template delivery through the loudness-normalize path, probing for a faststart regression). - Both run for real (actual ffmpeg encodes, not simulated) and graded with the repo's own evals/grade_runs_24.py: routing 100%, report format 2/2, disclosure 2/2, real execution 2/2. kw1's generated ASS carries \fscx112\fscy112 on the active word (genuinely bigger, not just recoloured); ff1's output has moov before mdat after the loudnorm re-encode and check.py's loudness/true-peak rows PASS. No regression found. - docs/roadmap.md: corrects the stale "released version today" line (1.24.1 -> 1.25.0, which is only a chore: faststart-path bump, #279), adds table rows for 1.24.1/1.25.0, and a narrative section for eval 23b. - The Opus-focused-grader and 55-prompt trigger-judge passes are NOT run here: this sandbox has no independent/non-Claude model access, the same blocker iteration-23-partial already documented (see also docs/design-decisions.md's "The evaluation is Claude-only"). Documented as an open gap, not faked. Claude-Session: https://claude.ai/code/session_013tPrsGXLuaGU7t643QNUZs Co-authored-by: Claude <noreply@anthropic.com> | 1 month ago | |
docs: eval 23b at 1.25.0 — karaoke-style word + faststart-fix coverage (#280) Continues eval 23 (closed partial at 1.19.3, evals/results/iteration-23-partial.json) with real-execution prompts for the two features shipped since: caption.py --karaoke-style word (#276) and the +faststart fixes (#275/#277/#279). - evals/agent_prompts_24.json: two new prompts, kw1 (karaoke-style word, phrased after the #276 reporter's own complaint that the highlight "just changed colour") and ff1 (a YouTube template delivery through the loudness-normalize path, probing for a faststart regression). - Both run for real (actual ffmpeg encodes, not simulated) and graded with the repo's own evals/grade_runs_24.py: routing 100%, report format 2/2, disclosure 2/2, real execution 2/2. kw1's generated ASS carries \fscx112\fscy112 on the active word (genuinely bigger, not just recoloured); ff1's output has moov before mdat after the loudnorm re-encode and check.py's loudness/true-peak rows PASS. No regression found. - docs/roadmap.md: corrects the stale "released version today" line (1.24.1 -> 1.25.0, which is only a chore: faststart-path bump, #279), adds table rows for 1.24.1/1.25.0, and a narrative section for eval 23b. - The Opus-focused-grader and 55-prompt trigger-judge passes are NOT run here: this sandbox has no independent/non-Claude model access, the same blocker iteration-23-partial already documented (see also docs/design-decisions.md's "The evaluation is Claude-only"). Documented as an open gap, not faked. Claude-Session: https://claude.ai/code/session_013tPrsGXLuaGU7t643QNUZs Co-authored-by: Claude <noreply@anthropic.com> | 1 month ago |