| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
feat(extraction): gate Reddit + Amazon site_data behind anti-bot block detection Audit C5: Reddit and Amazon URLs that returned anti-bot challenges or "Page Not Found" landings were silently treated as successful empty site_data responses — callers had no way to tell the difference between a real empty payload and a blocked page. - detectAntiBotBlock() exported from reddit + amazon site extractors, scans the first 10KB for canonical block / not-found phrases ("blocked by network security", "Page Not Found", robot-check prompts, etc.). - extract() short-circuits to null on a blocked body, refusing to emit fake site_data. - routedExtract surfaces the block reason via ExtractionResult.site_data_blocked. - handleFetch promotes it to FetchOutput.fetch_failed="blocked" so callers branch honestly while the fallback markdown body still ships. - Positive fixtures unchanged: real product / thread bodies still emit site_data with no false-positive blocked envelopes. | 3 个月前 | |
fix(extraction/brand): broaden brand fonts heuristic to cover inline-style + CSS rules + Google Fonts 0.1.20 cross-codebase acceptance found brand fonts populated on 1 of 6 tested sites (Figma — via inline style). Stripe, Linear, Vercel, HackerNews, example.com all returned provenance.fonts: "unknown" because the heuristic only recognized --brand-font-* / --font-family-* custom properties. Broaden the heuristic to a priority chain that picks the first source yielding ≥1 brand font: 1. CSS custom properties — provenance "css-vars" (existing) 2. <style>-block rules (body / h1, h2 selectors) — provenance "css-rule" (new) 3. Inline style="font-family:..." on <h1>/<h2>/<body> — provenance "inline-style" 4. Google Fonts <link href="...css?family=..."> — provenance "google-fonts-link" (new) Generic families (sans-serif, system-ui, helvetica, etc.) continue to be stripped from every source. Sources are not merged — first hit wins so callers know which signal is authoritative. Add 6 representative font-shape fixtures (Stripe, Linear, Vercel, HackerNews, Figma, example.com) and pin per-source expectations in a new acceptance suite. Aggregate target: ≥4 of 5 brand sites produce a non-empty fonts.body or fonts.headings. example.com stays honest at fonts: undefined + provenance.fonts: "unknown". | 3 个月前 | |
feat(fetch): add CDP client for Chrome DevTools Protocol session discovery | 5 个月前 | |
test(cache): add integration tests for content change detection with versioned HTML fixtures | 5 个月前 | |
test(extract,crawl): integration fixtures for markdown post-process | 4 个月前 | |
test(extraction): lock the sparsity gate against dense ordinal leaderboards The committed negative fixtures all lacked rank-ordinal rows, so nothing exercised the sparsity gate (filled/gridCells < 0.5) — the last line of defense that stops a dense ordinal leaderboard from over-firing the per-story segmentation. Behavior was correct, but a future relaxation of the 0.5 threshold would silently over-fire with zero failing test. Add a fixture-based negative: a header-less dense leaderboard where most rows are rank-led but two are non-ordinal, so the record-start rows are a minority (the starts<bodyRows and >=3-records checks both pass) and every cell is populated (filled/gridCells ≈ 1.0). The sparsity gate is the ONLY gate left to reject segmentation, so the test isolates it. Mutation-verified: loosening the threshold to <= 1.0 folds the leaderboard to rank/title/meta and fails the test. | 2 个月前 | |
test(integration): add multi-browser rendering and distribution tests | 5 个月前 | |
feat(plugins): add pluginsDir config and shape validation module | 5 个月前 | |
feat(repl): add output formatters with golden fixture tests | 5 个月前 | |
Parity attack 1 engines (#117) * fix(search): point marginalia adapter at api2 endpoint * fix(search): drop fmt=html and harden mojeek request shape Mojeek 403s are IP-reputation/rate-limit driven; fmt=html (which SearXNG never sends for web search) plus a thin header set triggered blocking through an entire benchmark run. Adopt the SearXNG-proven shape: safe=0, no fmt, no explicit s offset on page 1, browser-like Accept and Accept-Language headers. Parser and selector fallbacks unchanged. * feat(search): add wiby long-tail engine adapter * feat(search): replace startpage with wiby in general vertical * feat(search): breaker half-open probe with exponential backoff * feat(search): expose breaker_open reason in engine telemetry * feat(doctor): per-engine breaker health and --probe-engines * fix(search): reclaim stuck half-open probe after cooldown deadline A plugin engine whose search() never settles would hold the breaker half-open forever — probing stayed true and every later caller was rejected with no recovery path. When a caller finds the probe in flight longer than cooldownMs, treat it as failed: reopen with exponential backoff and clear probing so a future caller re-probes. * fix(search): sanitize upstream error strings before storing lastError Strip control characters and cap length so doctor/telemetry can't print terminal escape sequences from hostile upstream bodies. * fix(search): guard wiby results to http(s) URLs only Wiby payloads are untrusted — skip javascript:/data:/ftp:/protocol- relative entries in parseResults. Also pin non-2xx throw behavior with a test so the breaker keeps counting upstream failures. * test(search): cover telemetry merge branch for breaker_open skips Multi-query aggregation: a breaker-skipped dispatch merging into an existing engine entry must flip outcome to skipped and attach reason/cooldown, with first-seen-wins semantics matching error. | 3 个月前 | |
fix(extraction): extract modern GitHub blob from embedded React payload Modern github.com/<owner>/<repo>/blob/<ref>/<path> pages render file content client-side from a react-app.embeddedData JSON payload; the legacy .blob-code-content / .highlight selectors match only a ~500-char stub. Parse rawLines from payload["codeViewBlobLayoutRoute.StyledBlob"] and blob metadata (displayName/language/truncated) from payload.codeViewBlobLayoutRoute.blob into a fenced code block. Add extractGithubBlobWithRawFallback with an injected fetcher that transforms to raw.githubusercontent.com only when the payload is absent or truncated (large files) — a complete payload never double-fetches. Legacy selectors kept as the final fallback for old cached / enterprise pages. | 2 个月前 | |
feat: SPA shell and content emptiness detection | 5 个月前 | |
test(extract): add structured-data fixtures (json-ld/microdata/rdfa) | 4 个月前 | |
test(search): exercise URL-safe base64 path in Bing decoder fixture | 5 个月前 | |
test(fetch): add integration tests for browser action execution with HTML fixtures | 5 个月前 | |
test+docs: prune vacuous spawn assertion + race-safe xenova-compat write Removes the 'does NOT spawn a python subprocess during rerank' block from tests/integration/onnx-rerank-e2e.test.ts — that assertion was vacuously true because the test mocks onnxRerank itself. Spawn contract is now enforced by tests/integration/rerank-spawns-python.test.ts (CI-gated, unmocked). README wording: ML reranker (capability) instead of in-process ONNX reranker (implementation). CHANGELOG entry under 1.2.0. Race fix: dump_tokens.py and reranker_server.py now write the xenova-compat patched tokenizer.json to a process-unique temp file (PID-suffixed) instead of the shared model dir, removing the concurrent-write race that surfaced in Task 7 integration tests. Both scripts also clean up any legacy tokenizer_xenova_compat.json files from prior runs. | 4 个月前 | |
test(fetch): add integration tests for browser action execution with HTML fixtures | 5 个月前 | |
test(fetch): add integration tests for browser action execution with HTML fixtures | 5 个月前 | |
feat(reranker): add reranker_server.py + equivalence corpus + dump_tokens.py JSON-line protocol matches embedding_server.py. Uses tokenizers (Rust) + onnxruntime. Refuses spawn if tokenizer.json post_processor is not TemplateProcessing. Truncation strategy fixed to 'only_second' (matches @xenova/transformers default). Corpus is 60 (query, doc) pairs across 6 buckets including multilingual, emoji+ZWJ, literal special tokens, and edge cases — used by Task 4 equivalence test. | 4 个月前 | |
feat(reranker): add reranker_server.py + equivalence corpus + dump_tokens.py JSON-line protocol matches embedding_server.py. Uses tokenizers (Rust) + onnxruntime. Refuses spawn if tokenizer.json post_processor is not TemplateProcessing. Truncation strategy fixed to 'only_second' (matches @xenova/transformers default). Corpus is 60 (query, doc) pairs across 6 buckets including multilingual, emoji+ZWJ, literal special tokens, and edge cases — used by Task 4 equivalence test. | 4 个月前 | |
chore(deps): drop @xenova/transformers, onnxruntime-node, protobufjs override Removes: - @xenova/transformers (Node JS tokenizer) - onnxruntime-node (in-process ONNX runtime) - onnxruntime-node TypeScript shim - protobufjs override (was needed only for the xenova dep chain) - src/search/reranker/tokenizer.ts (now dead) Equivalence test now compares Python tokenizers output against a frozen xenova snapshot captured pre-deletion (tests/fixtures/reranker-tokenizer-snapshot.json). The corpus + snapshot are effectively frozen after this commit. EXPECTED_MISMATCH_BUCKETS (emoji + long-doc-truncation) skip logic preserved from Task 4.5. Adds tests/unit/package-deps.test.ts to enforce in CI that the dropped deps never reappear. src/index.ts comment updated - no behavior change. Version bumped 1.1.0 -> 1.2.0. | 4 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 3 个月前 | ||
| 3 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 4 个月前 | ||
| 2 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 3 个月前 | ||
| 2 个月前 | ||
| 5 个月前 | ||
| 4 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 4 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 4 个月前 | ||
| 4 个月前 | ||
| 4 个月前 |