AAkitaOnRailsfeat(consolidation): add bounded verbatim residue
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
feat(evals): score literal fidelity of the consolidation prompt (#1215) First implementation step of docs/design-verbatim-residue.md: the measurement harness, with no change under crates/. - evals/src/fidelity.rs: zero-LLM scoring that separates a fact the observation projection dropped (projection_loss) from one the model dropped after seeing it (llm_loss), plus keep-token recall, invented tokens and wikilinks, and must_not_mention hits. The primary measure is the per-fixture must_mention list, independent of any token miner. - ab: those scores for every run of both sides, --runs N (the noise band), --generated-seed, the exact request per fixture, report.json with commit, seed, sha256, providers, models and N. - projection: new provider-free subcommand, with --dump. - evals/src/generate.rs: a seeded 300-observation session in production capture shapes with fifteen planted facts; its own SplitMix64 keeps the published sha256 reproducible. - fixtures 06 (pt-BR tail and error forms) and 07 (injection and traps). - design doc: the maintainer's Q1-Q3 answers, post-#1221 prompt sizes, the R2 line, the harness-first PR order and the Q1 criterion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> | 17 天前 | |
feat(consolidation): add bounded verbatim residue | 14 天前 | |
feat(evals): R2 retrieval A/B harness with accuracy+latency+context-tokens triple Extends the in-repo retrieval eval into an A/B evaluation that runs the same question set through a baseline and candidate config and reports a delta over the triple: - accuracy: hit@k / recall@k (existing); - latency: per-query timing, p50/p95; - context-tokens: sum of returned hit title+snippet, chars/4 estimate, mean/median. Config knobs: --embeddings none|local, --reranker (now configurable, was force-removed), --server-env, and --query-arg passthrough into memory_query (ready for pin_first/include_superseded/etc.), each with a --candidate-* twin. Delta report in markdown+JSON preserving provenance (commit, dataset sha, hw). Single-config path unchanged. Also fixes a latent harness deadlock (child stderr pipe now drained on a background thread) that surfaced with two servers. Dev tooling only (evals is publish=false); numbers land in docs/benchmarks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MDbhmszrjG9s5MrPrTuNtm | 1 个月前 | |
feat(evals): add retrieval decomposition instruments The retrieval harness could only report an end-to-end hit@k, which mixes evidence the index never sees with ranking quality. This adds opt-in instruments that separate the two, with defaults that keep the published baselines reproducible: - --reachability reads each evidence session back and reports how many has_answer turns survive into the session page, the observation bodies and the local embedder's 512-token window (exact tokenizer); - --engine grep, a server-free whole-session term-overlap baseline that reproduces a published grep baseline adapter per question; - --ingest whole-session|captured-session, one page per session through /admin/write-page, for ranking-only and capture-only upper bounds; - --date-prefix turn|session-start|none, --stratify N, and the pinned cleaned LongMemEval-S variant; - ingest throughput, raw-fallback usage, and paired hit flips in A/B deltas. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MDbhmszrjG9s5MrPrTuNtm | 17 天前 | |
feat(consolidation): add bounded verbatim residue | 14 天前 |