| Files | Last commit | Last update |
|---|---|---|
Merge branch 'main' into fix/mimalloc-3.5.3-preinit-free | 11 days ago | |
Adopt the Developer Certificate of Origin for all commits Every commit must now carry a Signed-off-by trailer matching its author, certifying the right to submit the change under the project's MIT license (DCO 1.1, the Linux kernel mechanism). Enforcement is strict at three layers: a commit-msg hook rejects unsigned commits locally (scripts/install-git-hooks.sh), and the new DCO workflow rejects every push and pull request containing one. Merge commits and bot authors are exempt, matching standard DCO checks. Signed-off-by: Martin Vogel <martin.vogel@datadice.io> | 4 months ago | |
fix(hooks): clear repository git environment for tests Signed-off-by: Brian McNaboe <bmcnaboe@gmail.com> | 2 months ago | |
ci(codeql): PR scan per language inside PR CI, full scan on main Bring CodeQL back to pull requests as jobs of PR CI itself (needed by ci-ok, no polling), narrowed to first-party code, while push to main, the weekly run, the dry run and the release keep the full scan. - codeql.yml: one leg per language (c-cpp, javascript-typescript, python, actions), scope in .github/codeql/codeql-config.yml. The full scan traces the whole build, vendored C included. The PR scan (light: true) analyses each selected language's whole first-party scope; c-cpp compiles the third-party objects before tracing (scripts/build.sh --vendored-only) and then traces the first-party build plus the vendored files the PR changes (--reuse-vendored --retrace-from, make -W). Diff-informed analysis (codeql-action v4.38.1, forced on for PR calls) only restricts reported alerts. The alert check moves into each leg and reads the analysed ref. Push and PR runs cancel superseded runs per language; dispatched runs never cancel. Compiler cache off so every compile is traced. - pr.yml: a codeql job runs the legs of the selected codeql-* lanes; ci-ok needs it. - select-lanes.sh: codeql-cpp/-js/-py/-actions lanes, selected by a change in their scope (any C/C++ incl. vendored; JS/TS, Python and workflows inside the config's paths minus paths-ignore) and by every full run (undeterminable diff, workflows, build, CodeQL machinery). --codeql-scope LANGUAGE prints a list's in-scope paths. - codeql-retrace-vendored.sh: the vendored C files a PR changes (empty, never narrowing, when the diff cannot be determined). - Makefile.cbm: cbm-third-party builds the vendored prod objects only. - tests: the gate contract pins the new layout; the selector contract pins that a C change always selects the C scan, an undeterminable diff selects every language, and a vendored-only change retraces exactly the changed vendored files. Selector fixtures, ci-ok's job map, the replay's codeql-action failures (lane codeql-cpp) and the interface probes follow. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 18 days ago | |
test(harness): show sanitizer reports in the suite failure summary A Windows sanitizer job went red on one suite that aborted under AddressSanitizer, and the job log showed an empty "every failure site" section followed by the shadow-byte legend. The summary in scripts/run-tests-parallel.sh searched the suite log for "FAIL" only. A sanitizer report never contains that word, and its useful part (error kind, access size, top frames) is at the start of the report, far above the last 15 lines. The summary now lives in scripts/suite-failure-report.sh, which the harness sources. Between the unchanged failure-site and last-15-lines sections it prints, for a failing suite, the test that was running and the first sanitizer report. The framework flushes each test name before the test body runs, so the last test line ahead of a report is the test that was running; a report that comes after the last test finished is labelled as such instead of blaming a test. The report is shown from its header (AddressSanitizer, LeakSanitizer, ThreadSanitizer, MemorySanitizer, UndefinedBehaviorSanitizer) to its own SUMMARY line, at most 40 lines, without the shadow dump and the legend; a recoverable "runtime error:" line is shown with its stack only. Further sanitizer headers, "runtime error:" lines and SUMMARY lines follow (at most 10), then signal and timeout lines (at most 5). A suite that dies with no report and no completion summary gets the running test named. A trap-mode UBSan build stops on an illegal instruction and prints nothing, so that name is the only evidence its log holds. A log with an assertion failure and no report prints exactly what it printed before. The same file keeps the logs a red run needs. An EXIT trap in the harness copies the log of every suite that did not record rc=0, plus results.txt, into test-logs/failed/. The trap covers the scheduler giving up and the union guard as well as the normal end, and it never changes the exit status. scripts/run-test-wave.py reported pass=0 for a suite that died without its summary line. In that case, and only then, it now counts the PASS markers in the log. A suite with a summary line is never recounted, so the totals of a green run are unchanged. tests/test_failure_report_contract.sh drives the real functions with fabricated suite logs and runs as Step 0i2 of scripts/test.sh. It fails on the previous behaviour in every sanitizer, running-test and result-line check and passes on it in the unchanged-output checks. The awk code uses no interval expressions and no character classes; the test passes with BSD awk and grep on macOS and with mawk and GNU grep on Ubuntu. scripts/README.md names the new internal file. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 26 days ago | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 6 months ago | |
fix(grammar): accept non-ASCII letters in Pascal identifiers (#2211) Delphi code that calls 1C's OLE API through late-bound Variants uses Cyrillic member names (v8_Base.<Cyrillic>.<Cyrillic>('1', Now), NM.<Cyrillic>()). Every such name produced a tree-sitter ERROR node, so the unit was reported as parse_partial and a procedure whose own name is Cyrillic was dropped from the graph. Root cause is upstream tree-sitter-pascal (Isopod, the pin we vendor): its identifier token is ASCII-only, /[&]?[a-zA-Z_]+[0-9_a-zA-Z]*/, and there is no other token a non-ASCII letter can lex as. Fix: pascal becomes a self-maintained fork of the pin we already vendored (Isopod/tree-sitter-pascal @ 042119eca2e1, v0.10.2 = upstream master), following the arkts model. One patch (tools/tree-sitter-pascal/patches/unicode-identifiers.patch) changes only that token to /[&]?[\p{L}_][\p{L}\p{M}\p{Nd}_]*/. Keywords are still extracted from this `word` token, so begin/end/procedure lex as before (`begin` glued to Cyrillic letters is one identifier; `&begin` stays an escaped identifier). Node types are unchanged. The \p{} classes compile to numeric code-point ranges: the vendored parser.c has 0 non-ASCII bytes. Regenerated with tree-sitter-cli 0.25.10 --abi 14 (ABI unchanged, no external scanner; tree_sitter/parser.h is byte-identical to chialisp's). Fidelity: the UNPATCHED grammar regenerated with 0.25.10 gives the same symbol/token/field/production counts as upstream's shipped parser and byte-identical parse trees on upstream's 88 corpus cases and its three examples/*.pas (37,773 tree lines for codetoolmanager.pas); the patched parser gives the same identical trees on all of those ASCII inputs and passes all 88 corpus cases. Proof (7 Pascal files: the report's snippet, a Cyrillic procedure with keyword-boundary cases, an ASCII control, a broken unit, upstream's three examples; fresh caches): parse_partial 3 -> 1 (only the genuinely broken unit remains), nodes 581 -> 582 (the Cyrillic-named procedure), edges 1480 -> 1482 (+1 DEFINES, +1 SEMANTICALLY_RELATED); per-file function counts for the ASCII files are unchanged. Registered like arkts: MANIFEST.md fork row + note, audit FORKS map, THIRD_PARTY.md, vendored checksums. LICENSE stays upstream's MIT, byte-identical. Tests: parse_coverage gains the report's snippet and a Cyrillic-named procedure (not partial, definitions extracted), an ASCII control, and a guard that broken Pascal is still flagged; the fork carries test/corpus/unicode_identifiers.txt. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 26 days ago | |
fix(test): hand over a kept benchmark runtime only after its daemon stopped With CBM_BENCH_KEEP_RUNTIME=1, bench_finish ran `daemon stop` with its output and exit status discarded and wrote runtime-root.txt regardless. `daemon stop` refuses while a committed client still uses the daemon and fails when the daemon does not answer; the handoff then promised a stopped daemon that was still running. A refused stop now prints the explanation of the product, writes no handoff, keeps the root without cleaning it up (the daemon may still be using it) and exits non-zero. A stop with nothing to stop still exits 0, so a clean run is unaffected. Section 7 step 8 of the evaluation plan follows the same rule: the kept root is removed only if its daemon stopped, and is kept with a message otherwise. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 6 days ago | |
fix(test): surface why a benchmark daemon did not start Both harnesses discarded the output of `daemon start`, so a refused start was reported only as "private daemon did not start", the symptom without its cause. The output now goes to a run-private file next to the index stderr, the shape both scripts already use for the index call, and is printed under "--- daemon start output ---" when the start fails. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 6 days ago | |
ci(freebsd): add a dispatch-only, informational FreeBSD build workflow Add .github/workflows/freebsd.yml, a standalone workflow triggered only by workflow_dispatch. It builds the UI-embedded binary inside FreeBSD 15.1 and 14.4 x86_64 VMs (vmactions/freebsd-vm, full-SHA pin to v1.5.9) with the canonical scripts/build.sh and runs the binary with --version. It declares read-only permissions, uploads and publishes nothing, is not part of ci-ok and is not called by release.yml or dry-run.yml, so a FreeBSD red never gates a pull request or a release. scripts/env.sh now detects FreeBSD and selects gmake there (BSD make cannot read the GNU make syntax of Makefile.cbm) and make everywhere else; the environment summary line reports the choice. scripts/build.sh runs $MAKE at all three build call sites. macOS, Linux and Windows still run make, so their builds are unchanged. Distilled from #1422 by PR9000: the release matrix, the x86_64-host choice, the aarch64 exclusion, the vmactions VM approach with a SHA pin, fail-fast: false and the gmake detection all come from that pull request, which built on the FreeBSD groundwork of #1093, #1423 and #1467. Co-authored-by: Pedro Ramos <131530838+pr9000@users.noreply.github.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 3 days ago | |
Adopt the Developer Certificate of Origin for all commits Every commit must now carry a Signed-off-by trailer matching its author, certifying the right to submit the change under the project's MIT license (DCO 1.1, the Linux kernel mechanism). Enforcement is strict at three layers: a commit-msg hook rejects unsigned commits locally (scripts/install-git-hooks.sh), and the new DCO workflow rejects every push and pull request containing one. Merge commits and bot authors are exempt, matching standard DCO checks. Signed-off-by: Martin Vogel <martin.vogel@datadice.io> | 4 months ago | |
Merge branch 'main' into feat/ruby-lsp-per-file | 19 days ago | |
Forbid test skips and convert existing skips to hard failures Add scripts/check-no-test-skips.sh (run from lint) which fails the lint phase on any plain SKIP() or direct tf_skip_count manipulation; only SKIP_PLATFORM() (for genuinely platform-specific tests) is tolerated. Add FAIL() and SKIP_PLATFORM() helpers to the test framework and convert the remaining SKIP()/perf-gated skips across the suite into pass-or-fail assertions, so a suite that cannot meet its preconditions reports a red failure instead of a silent skip. | 5 months ago | |
fix(scripts): use /usr/bin/env bash shebangs (NixOS has no /bin/bash) On NixOS (and other non-FHS systems) /bin/bash does not exist, so scripts with an absolute shebang fail to run. Switch the remaining holdouts to /usr/bin/env bash: eleven scripts/*.sh, test-infrastructure/run.sh, and the three Claude Code hook scripts emitted by src/cli/cli.c (gate, session reminder, subagent reminder). Distilled from PR #674, with parser-test coverage preserved: the infra_parse_shell* fixtures in tests/test_pipeline.c intentionally keep #!/bin/bash so absolute-path shebang extraction stays covered, and tests/repro fixtures are untouched. Also replace the GitHub-PAT-shaped fixture string flagged in the #674 thread with an obviously fake placeholder (ghp_FAKE...) that still matches the ghp_ + 36-alnum secret detector. Co-authored-by: Sandro Jäckel <sandro.jaeckel@gmail.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 4 months ago | |
revert(release): re-embed runtime assets into the single shipped binary Externalizing the integration templates (#1492/#1493) and the UI bundle (#1501/#1503) was done to reduce the Microsoft `Wacatac.B!ml` surface. It did not work: across dry runs the flagged artifact count stayed at ~3 and the detections merely moved between artifacts. Dissection of run 31286803592 shows there is no structural cause to fix. The verdicts split across every axis at once — linux-amd64 (dynamic) flagged while linux-amd64-portable (static) is clean, but linux-arm64 (dynamic) clean while linux-arm64-portable (static) is flagged. The two macOS binaries have identical segment structure and split clean/flagged. Siblings from one build landed in different variant buckets (.B vs .C). Entropy is low everywhere (code_vectors.bin 4.166, grammar tables 3.464 bits/byte, against 7.5-8.0 for packed payloads), so the packed-payload hypothesis is excluded too. So the complexity bought nothing, and installation goes back to being self-contained: one binary that carries its own UI and agent integration templates, with no adjacent data file that has to resolve before `install` works. Only the UI-capable composition ships from now on, under the historical unsuffixed archive name. Removed: src/ui/asset_pack.{c,h}, asset_pack_stub.c, asset_manifest_stub.c, scripts/pack-ui-assets.mjs, src/cli/integration_assets.{c,h}, assets/cbm-integrations.json, scripts/gen-integrations-hash.sh, the --verify-runtime-assets probe (nothing adjacent left to verify), and the composition gates A6/A7 whose property is now deliberately inverted. Restored: scripts/embed-frontend.sh, src/ui/embedded_{assets.h,stub.c}, the compiled-in hook/adapter template bodies, and the embed/EMBED_OBJS build path. Kept from the reverted commits, re-applied by hand where a wholesale file restore would have dropped them: - cbm_module_path_utf8() in both self-path sites. GetModuleFileNameA renders through the ANSI code page and mangles non-ASCII install paths. - the /__cbm/ui-readiness HMAC proof, secure_random and cbm_hmac_sha256, so `daemon start --open` still waits for a genuine CBM listener. - X-Content-Type-Options: nosniff on served assets. - the MinGW noexecstack gate, -lbcrypt, and the cppcheck/zip CI fixes. Archives are now codebase-memory-mcp-<os>-<arch>[-portable] with exactly four members (binary, LICENSE, installer, THIRD_PARTY_NOTICES.md). That restores the names every static package manifest already points at — aur, chocolatey, homebrew, scoop, winget and glama were all broken by the -ui- rename. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 2 months ago | |
Cross-service communication discovery + RAM-first incremental indexing AST-based detection of HTTP calls, async dispatch (Pub/Sub, Cloud Tasks, Kafka, SQS, etc.), and config accesses via resolved qualified names. Route nodes as cross-service rendezvous points with infra→handler matching. Constant propagation for module-level string assignments. YAML infrastructure URL extraction from Cloud Scheduler configs. RAM-first incremental pipeline: load DB into graph buffer, purge changed file nodes, extract directly into existing buffer (resolver sees all nodes), dump back to disk. Zero edge gap on kubernetes/django/meilisearch/neovim. - service_patterns.c: ~170 library patterns (90 HTTP, 50 async, 30 config) - pass_route_nodes.c: Route node creation + infra URL matching - extract_unified.c: string constant collection + string ref classification - extract_calls.c: first_string_arg + keyword argument extraction - pipeline_incremental.c: RAM-first load→purge→extract→resolve→dump - graph_buffer.c: load_from_db, delete_by_file, foreach visitors - C++ LSP crash fix: NULL guard in cbm_type_substitute | 7 months ago | |
fix: prefer toolchain-provided linker to embed web-ui assets Signed-off-by: Attila Oláh <attila@dorn.haus> | 17 days ago | |
ci(freebsd): add a dispatch-only, informational FreeBSD build workflow Add .github/workflows/freebsd.yml, a standalone workflow triggered only by workflow_dispatch. It builds the UI-embedded binary inside FreeBSD 15.1 and 14.4 x86_64 VMs (vmactions/freebsd-vm, full-SHA pin to v1.5.9) with the canonical scripts/build.sh and runs the binary with --version. It declares read-only permissions, uploads and publishes nothing, is not part of ci-ok and is not called by release.yml or dry-run.yml, so a FreeBSD red never gates a pull request or a release. scripts/env.sh now detects FreeBSD and selects gmake there (BSD make cannot read the GNU make syntax of Makefile.cbm) and make everywhere else; the environment summary line reports the choice. scripts/build.sh runs $MAKE at all three build call sites. macOS, Linux and Windows still run make, so their builds are unchanged. Distilled from #1422 by PR9000: the release matrix, the x86_64-host choice, the aarch64 exclusion, the vmactions VM approach with a SHA pin, fail-fast: false and the gmake detection all come from that pull request, which built on the FreeBSD groundwork of #1093, #1423 and #1467. Co-authored-by: Pedro Ramos <131530838+pr9000@users.noreply.github.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 3 days ago | |
fix(build): stop blob regeneration from reverting the ELF stack hardening vendored/nomic/code_vectors_blob.S is a GENERATED file that was hand-edited. 1f674c80 added the ELF .note.GNU-stack section to it after finding that every Linux binary we had ever shipped requested an executable stack: an object carrying no such note tells GNU ld nothing about its stack requirement, and ld then assumes the worst for the entire link. That fix went into the artifact only. scripts/extract_nomic_vectors.py still emitted the Mach-O branch alone, so the next regeneration would have overwritten the tracked .S, dropped both the ELF and COFF branches, and put GNU_STACK RWE back into every release -- silently, because nothing compared the generator against its own output. write_blob_s() now emits the tracked wrapper verbatim, and the new contract test fails if the two ever drift again. The template is compared rather than executed so the test does not need the torch/transformers import the extraction script pulls in. The write is pinned to UTF-8 because the WHY comment carries an em dash and the default encoding is locale-dependent. check-binary-composition.sh already fails a release whose binary has an executable stack; this closes the same hole at edit time, where the diff is still small enough to read. Reported and originally fixed by Anand Aiyer in #1151, which also carried the COFF branch. That PR sat unreviewed for a month while the .S half was rediscovered and landed independently without credit. Co-authored-by: Anand Aiyer <anand@aiyer.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 2 months ago | |
fix: keep large thread-locals off the static TLS block Indexing broke on x86-64 Linux: the index worker refused to start and SIGKILLed its own process group, so every venue reported nothing but "index worker ended with killed (exit=-1, signal=9)". glibc places a thread's static TLS block inside the stack allocation it makes for that thread. This branch's new thread-local caches -- a 512-slot field cache (~24 KB), per-depth tree cursors (~4 KB) and two parked CBMArenas (~8 KB; an arena carries blocks[256] + block_sizes[256]) -- took static TLS from 47,905 to 93,513 bytes. The 64 KB parent-death watchdog thread then could not be created at all, and a worker without process-tree containment correctly refuses to index. Four layers, because there were four things wrong: - The big thread-locals move to the heap behind a thread-local POINTER, allocated once per thread and reused, each with a correct fallback if allocation fails (the field cache and cursor pool are caches; a miss is slower, never wrong). Static TLS is back to 52,233 bytes. - PARENT_WATCHDOG_STACK_SIZE 64 KB -> 256 KB. "The watchdog only polls, a tiny stack suffices" was true when written and false once TLS reached ~50 KB, because the TLS block comes out of that same allocation. - cbm_thread_create retries with the default stack on EINVAL. A stack size is a hint about what a thread needs; the platform refusing the hint is not a reason to fail to create the thread. - tests/test_thread_stack_tls_contract.sh (Step 5a of scripts/test.sh) keeps static TLS a quarter clear of the smallest requested stack, reading that smallest size from the sources so a new small stack tightens the gate automatically. Two diagnostics that made this findable at all, and that stay: smoke-test.sh prints the worker log its failure message names (CI deletes that sandbox with the job, so the one artifact naming the cause was unreadable), and worker_containment_unavailable() now reports WHICH of its conditions failed, with pid/pgrp/ppid. Verified in the amd64 container, the venue nothing on the local ladder covers: baseline main indexes 9039 nodes/15067 edges, this branch before the fix refuses to start the worker, and after it indexes 9039/15067. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 1 month ago | |
doc-links(pdf): notices for the PDF tables; the Unicode table generator The PDF text layer carries data tables (D3). Their sources and notices are now in THIRD_PARTY.md (shipped in every archive through THIRD_PARTY_NOTICES.md) and in the SBOM: - the Adobe Glyph List subset (BSD-3-Clause, Copyright 2002-2019 Adobe), - the printable-ASCII advance widths of Helvetica and Times-Roman from Adobe's core-14 font metrics (the AFM files are not distributed), - the Unicode Character Database data of pdf_unicode.c (Unicode License v3, full text included). scripts/gen-pdf-unicode.py (stdlib CPython only) regenerates pdf_unicode.c byte for byte with CPython 3.14.6 (Unicode 16.0.0); pdf_tables.c is maintained data from here on. License texts were taken from their sources (adobe-type-tools/agl-aglfn, unicode.org). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 30 days ago | |
feat(py_lsp): Round 1 parity push (parent-modules, cast, Self, fwd-refs) Parity push #1 — six gap fixes on top of the 11-phase plan: - Parent-module bindings: `import a.b.c` now binds `a`, `a.b`, `a.b.c` via py_bind_dotted_prefixes. Walks the dotted prefix chain on every import, regardless of import_is_from_style. Plus an attribute-access path on a MODULE that detects submodules: when the registry has any function whose qn starts with `<mod>.<attr>.`, evaluating `mod.attr` yields MODULE("<mod>.<attr>"). This makes `os.path.join(...)` and similar resolve correctly. - typing.cast(T, x) returns NAMED(T) (re-evaluating T as an annotation, so generic subscripts and forward references both resolve). Detects bare `cast(...)` and qualified `typing.cast(...)` forms. - typing.assert_type(x, T) is a no-op at runtime; we type the result as type-of(x). - Forward references as quoted strings: `def f(x: "Foo")` strips quotes and re-resolves the inner annotation. Also handles double-quoted and single-quoted forms. - Self return type substitution: methods declared returning `Self` / `typing.Self` / `typing_extensions.Self` substitute to NAMED(receiver) at call resolution time. Enables fluent / builder pattern chaining to keep resolving methods after each step. - Generic subscript stripping: annotations like `list[Foo]` strip to `list` for v1; `Optional[Foo]` likewise. Container element-type substitution still deferred (Phase 8+). Generator additions: - Allowlist now includes urllib, http, concurrent (top stdlib usage). - `from X import *` re-export following at generation time. Modules like os.path (a star-import shim of posixpath / ntpath / genericpath) now have all forwarded definitions registered under their own QN. Iterates to a fixed point with an 8-step ceiling. typeshed tree expansion is bounded — only modules transitively reachable from the allowlist are pulled in. Stats: 137 modules, 904 classes (2797 methods), 886 free functions (was 114 / 753 / 794). Generated file grows from 20K to ~22K lines. 6 new test_py_lsp.c cases. All 2874 prior tests stay green. | 5 months ago | |
fix(release): externalize runtime assets and harden VT verification Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 2 months ago | |
chore(ui): drop sharp and its LGPL libvips; the license gate allows no exception The v0.12.0 release run's license gate blocked @img/sharp-wasm32@0.35.5 (Apache-2.0 AND LGPL-3.0-or-later AND MIT). sharp sat in graph-ui's tree only because @huggingface/transformers lists it for its Node image path. CBM Lens runs Transformers in the browser (a tokenizer and a causal LM on WebGPU), never processes images, and the Transformers web build already replaces sharp with an empty object. Yet sharp installed 14 LGPL packages (10 @img/sharp-libvips-*, 3 @img/sharp-win32-*, @img/sharp-wasm32), and the gate let them through two ways: - a reviewed exception for exactly @img/sharp-wasm32@0.35.0 ("installed but not shipped"), which the 0.35.5 bump in #2632 tripped; - collect() skipped every os/cpu-restricted package as "build tooling", which hid all ten LGPL libvips binaries. A non-permissive license has no place in the tree, shipped or not: - sharp now resolves to graph-ui/stubs/sharp, our own MIT stand-in that throws if anything ever calls it (direct dependency file:./stubs/sharp plus override "sharp": "$sharp"). The whole sharp family leaves the lockfile: 441 -> 411 packages, zero LGPL/GPL left (audited). - ui_browser_licenses.py loses the exception and keeps only the browser build evidence check; license-gate-check-npm.py validates that evidence unconditionally, checks platform-restricted packages like any other, and has no exclusion path; gen-ui-licenses.py drops the exception section. - tests: the exception tests become guards: an unshipped LGPL package blocks, a platform-restricted LGPL package blocks, a platform-restricted MIT package passes. Against the old scripts both LGPL guards fail. The shipped browser bundle is unchanged: rebuilt, it differs from main's only by the build's own -dirty version marker. License gate stage 3 on this tree: 211 production packages, all allow-listed. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 25 days ago | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 6 months ago | |
Adopt the Developer Certificate of Origin for all commits Every commit must now carry a Signed-off-by trailer matching its author, certifying the right to submit the change under the project's MIT license (DCO 1.1, the Linux kernel mechanism). Enforcement is strict at three layers: a commit-msg hook rejects unsigned commits locally (scripts/install-git-hooks.sh), and the new DCO workflow rejects every push and pull request containing one. Merge commits and bot authors are exempt, matching standard DCO checks. Signed-off-by: Martin Vogel <martin.vogel@datadice.io> | 4 months ago | |
chore(ui): drop sharp and its LGPL libvips; the license gate allows no exception The v0.12.0 release run's license gate blocked @img/sharp-wasm32@0.35.5 (Apache-2.0 AND LGPL-3.0-or-later AND MIT). sharp sat in graph-ui's tree only because @huggingface/transformers lists it for its Node image path. CBM Lens runs Transformers in the browser (a tokenizer and a causal LM on WebGPU), never processes images, and the Transformers web build already replaces sharp with an empty object. Yet sharp installed 14 LGPL packages (10 @img/sharp-libvips-*, 3 @img/sharp-win32-*, @img/sharp-wasm32), and the gate let them through two ways: - a reviewed exception for exactly @img/sharp-wasm32@0.35.0 ("installed but not shipped"), which the 0.35.5 bump in #2632 tripped; - collect() skipped every os/cpu-restricted package as "build tooling", which hid all ten LGPL libvips binaries. A non-permissive license has no place in the tree, shipped or not: - sharp now resolves to graph-ui/stubs/sharp, our own MIT stand-in that throws if anything ever calls it (direct dependency file:./stubs/sharp plus override "sharp": "$sharp"). The whole sharp family leaves the lockfile: 441 -> 411 packages, zero LGPL/GPL left (audited). - ui_browser_licenses.py loses the exception and keeps only the browser build evidence check; license-gate-check-npm.py validates that evidence unconditionally, checks platform-restricted packages like any other, and has no exclusion path; gen-ui-licenses.py drops the exception section. - tests: the exception tests become guards: an unshipped LGPL package blocks, a platform-restricted LGPL package blocks, a platform-restricted MIT package passes. Against the old scripts both LGPL guards fail. The shipped browser bundle is unchanged: rebuilt, it differs from main's only by the build's own -dirty version marker. License gate stage 3 on this tree: 211 production packages, all allow-listed. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 25 days ago | |
Bundle third-party notices into release archives; extend release tooling Release archives now carry THIRD_PARTY_NOTICES.md, generated by scripts/gen-third-party-notices.sh from THIRD_PARTY.md, the grammar manifest, and the per-component license texts; the Homebrew formula and AUR PKGBUILD install it alongside the binary. The SBOM gains per-component license metadata, corrected versions, and the previously missing vendored libraries. The security workflow gains a vendored-license scan with an explicit allow-list policy, and the release workflow exposes a skip_perf input for releases that do not touch pipeline logic. | 4 months ago | |
ci(security): pin the ScanCode install; pluggy 1.7.0 breaks it The license gate installed `scancode-toolkit` unpinned. pluggy 1.7.0 (uploaded to PyPI 2026-10-15 09:50 UTC) no longer finds the hookspecs that plugincode 32.0.0 declares on a class (`@pre_scan_spec class PreScanPlugin`), so scancode crashes at import with "ValueError: did not find any 'pre_scan' hooks" and `security / license-gate` fails on every run, whatever the tree. The last green gate started at 09:50:14 UTC; every run since is red. `ci-ok` and the release `verify` job both need the security workflow, so this blocks every PR and the release. Install exactly scancode-toolkit 32.5.0 (the version every green run used) with pluggy constrained to 1.6.0 through PIP_CONSTRAINT, which the pip that pipx runs reads. The local hint in license-gate.sh names the same command. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 23 days ago | |
revert(release): re-embed runtime assets into the single shipped binary Externalizing the integration templates (#1492/#1493) and the UI bundle (#1501/#1503) was done to reduce the Microsoft `Wacatac.B!ml` surface. It did not work: across dry runs the flagged artifact count stayed at ~3 and the detections merely moved between artifacts. Dissection of run 31286803592 shows there is no structural cause to fix. The verdicts split across every axis at once — linux-amd64 (dynamic) flagged while linux-amd64-portable (static) is clean, but linux-arm64 (dynamic) clean while linux-arm64-portable (static) is flagged. The two macOS binaries have identical segment structure and split clean/flagged. Siblings from one build landed in different variant buckets (.B vs .C). Entropy is low everywhere (code_vectors.bin 4.166, grammar tables 3.464 bits/byte, against 7.5-8.0 for packed payloads), so the packed-payload hypothesis is excluded too. So the complexity bought nothing, and installation goes back to being self-contained: one binary that carries its own UI and agent integration templates, with no adjacent data file that has to resolve before `install` works. Only the UI-capable composition ships from now on, under the historical unsuffixed archive name. Removed: src/ui/asset_pack.{c,h}, asset_pack_stub.c, asset_manifest_stub.c, scripts/pack-ui-assets.mjs, src/cli/integration_assets.{c,h}, assets/cbm-integrations.json, scripts/gen-integrations-hash.sh, the --verify-runtime-assets probe (nothing adjacent left to verify), and the composition gates A6/A7 whose property is now deliberately inverted. Restored: scripts/embed-frontend.sh, src/ui/embedded_{assets.h,stub.c}, the compiled-in hook/adapter template bodies, and the embed/EMBED_OBJS build path. Kept from the reverted commits, re-applied by hand where a wholesale file restore would have dropped them: - cbm_module_path_utf8() in both self-path sites. GetModuleFileNameA renders through the ANSI code page and mangles non-ASCII install paths. - the /__cbm/ui-readiness HMAC proof, secure_random and cbm_hmac_sha256, so `daemon start --open` still waits for a genuine CBM listener. - X-Content-Type-Options: nosniff on served assets. - the MinGW noexecstack gate, -lbcrypt, and the cppcheck/zip CI fixes. Archives are now codebase-memory-mcp-<os>-<arch>[-portable] with exactly four members (binary, LICENSE, installer, THIRD_PARTY_NOTICES.md). That restores the names every static package manifest already points at — aur, chocolatey, homebrew, scoop, winget and glama were all broken by the -ui- rename. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 2 months ago | |
fix(ci): parse clang-tidy analyzer warning suffix Signed-off-by: Your GitHub Username <your_email@example.com> | 1 month ago | |
build(lint): pin analyzer suppressions to the code they argue about The memory gate stays gating, and it now has a way to record a genuine false positive that cannot quietly outlive its own reasoning. A whitelist entry names one (file, function, check) and carries two things: why the analyzer is wrong, argued from the code, and what was tried before concluding that. The entry is pinned to the sha256 of that function's text. Edit the function and the entry stops counting -- the finding comes back and has to be argued again against the code as it now is. This is the part that matters. A suppression that survives the code it was written about reads as "reviewed" while being nothing of the kind, which is worse than no suppression at all. The gate fails on: a finding with no entry, a finding whose entry has gone stale, and an entry that asserts rather than argues (there is a floor on how much reasoning an entry must actually contain -- the mechanical half of "argued, not asserted"; whether the argument is correct stays a review question). An entry that matches no finding is reported but does not fail, because analyzer versions differ across platforms. NOLINT is still not honoured here and the gate does not read it. The whitelist ships empty: the analyzer is currently clean across LINT_SRCS, so nothing is being suppressed today. The mechanism exists for the first finding that genuinely warrants it. Verified by exercising each path rather than only the clean one: an unaccounted finding fails and names its enclosing function; an argued entry with a matching hash passes; the same entry fails as STALE once the hash no longer matches the function; and an entry that says only "false positive" fails for not arguing its case. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 3 months ago | |
feat(mem): memory on the core, waste cut, results spilled under budget Second increment on the memory core. The first landed the core, the linter and phase attribution with zero production call sites; this one migrates the memory that matters, proves where it goes, removes the waste the proof exposed, and makes the memory budget a promise: over budget, extraction results go to disk and the run completes at the floor instead of aborting. Every number below is from a real index on this machine (M5 Pro, 48 GB) with CBM_MEM_PHASES=1 and the release build. WHAT THE PROOF SAID (Go corpus: 21,875 files, 345 MB of source) The graph buffer, blamed for the peak since the 2026-07 research, is 3% of the memory: 0.55 GB peak. The per-file extraction results are the memory: 14.8 GB written into their arenas, of which 3.4 GB is reachable. The rest is temporaries -- cbm_node_text copies at 496 call sites, per-node QN strings, abandoned generations of every growable array -- that an arena cannot free and that the result kept alive until resolve ended, because the result owned the arena. The arenas were charged 28.8 GB of capacity for those 14.8 GB. On the kernel the same shape was 8.9 GB used by 89,731 results, and resolve grew it to 13.3 GB used / 17.2 GB capacity because the C walk allocated its working set in the result arena and stored arena pointers into the shared registry. After that, 7 GB of the kernel's post-resolve memory had no class at all. WHAT CHANGED Attribution. The graph buffer (87 raw sites -> 0), arena blocks, the tree-sitter slab allocator, the bound tree-sitter/SQLite allocators, the semantic path (pass_semantic_edges, pass_similarity, minhash: 80 sites), the hash table (Verstable, a class per table via CTX_TY: cbm_ht_create_in) and CBM_DYN_ARRAY allocate through the core. Classes arena, ts_tree, hash_table and dyn_array join the table. Two cross-boundary frees of buffer-owned strings (pass_importance, pass_complexity) go through a new setter, cbm_gbuf_node_set_properties_json. The linter baseline drops 4086 -> 3916. Result compaction (internal/cbm/result_compact.c). At the end of a file's extraction, everything reachable from its result is measured, copied into one exact-size arena (strings interned by content within the file) and the working arena is handed back. Later appends restart growth at the default block (cbm_arena_init_exact, arena grow_size). The stale duplicate internal/cbm/arena.h, which shared the foundation header's include guard and won or lost by include order, is a shim now. Record slimming. CBMCall 320 -> 72 bytes: the eight inline argument slots (256 bytes, mostly empty) are allocated on first capture. CBMUsage 48 -> 40. The per-file overlay contract (src/pipeline/pass_lsp_cross.c). One lifecycle for every language's cross-file resolve: a scratch arena, a per-file overlay registry chained to the sealed shared base, the walk against the overlay, resolved calls appended into the result, scratch destroyed. type_registry gains chain-aware iterators (every yielded index belongs to it->reg) and copy-on-write refinement (cbm_registry_func/type_for_update), and the C, C#, Go and Python walks use them. This is what an ASan heap-use-after-free in the lsp_resolution_probe suite demanded: the C walk stored per-call arena objects into the shared registry, and a kernel run with a naive scratch arena lost 1,045 edges to the resulting garbage reads. It also ends unsynchronized writes into the shared registry from parallel resolve workers. The 7 GB with no class, found and cleaned. gbuf_index tables and keys were 2.5 GB live and 7.1 GB at peak: the 18 worker buffers built the five secondary indexes they never query (cbm_gbuf_new_worker builds none), and 18M per-key index arrays started at 8 slots (cbm_da_push_min starts them at 2); edge property JSON is 81% "{}" (interned). The semantic pass allocated a 512-token stride per function up front (packed tokens with offsets: cbm_sem_corpus_add_docs_batch). Edge dedup keys are 128-bit hashes instead of sprintf'd strings (edge_key_map_t). Each worker keeps one reusable working arena between files (cbm_work_arena_take/give: rewind, not free) and every phase mark returns freed pages to the OS. Budget metric. The over-budget check compares the charged footprint -- max(phys_footprint, mimalloc commit) on macOS -- not resident_size: after extraction the kernel worker sat at 17.4 GB RSS with 5.5 GB charged, because pages mimalloc has purged stay resident until the OS wants them, and phys_footprint under-reports after MADV_FREE_REUSABLE cycles. Spill / admission control (internal/cbm/result_spill.c, the extract gate in pass_parallel.c). At 15/16 of the budget (or over it, or with CBM_MEM_SPILL=1) the gate latches spill mode -- early, because the gate sees the crossing per file pull and the workers' in-flight files carry the charge past the line before the first sweep lands: every compacted result is parked on disk as header + block in one append-only file per writer under <cache>/spill/, the results already cached are swept out by every worker, and registry build, def collection, surface rows, resolve and the infra-route passes load a result only for the moment they read it (relocated by base delta). The abort path stays closed while a sweep runs or a cached result remains; it fires only at the floor -- graph + registries + in-flight files. A result that owns sub-results stays in memory; a retained parse tree is dropped with the parked result and re-parsed on load. Every consumer of the result cache reads a slot through cbm_pipeline_result_acquire()/release() (pipeline_internal.h): a parked slot is NULL in the array, and the passes that indexed it directly lost every __route__infra__ node and its INFRA_MAPS edges before the contract. The other things the transient loads found: a def label that was the result's own pointer (SIGBUS in the Go cross registry builder), two JVM helpers that passed their input through, a dedup table whose keys borrowed sr->value (the insert spun on freed memory), and a latch that was visible before the store was open (a peer found nothing to park and failed the run while its neighbour wrote 9 GB). The arena rewind reuse branch had defeated the nblocks = CBM_ARENA_MAX_BLOCKS OOM seam the suites use; it reuses only blocks that exist. The incremental and probe routes still index the cache array themselves and therefore never spill (ctx->spill_allowed stays false there; follow-up). The semantic pass under a budget (src/pipeline/pass_semantic_edges.c). The pass held every function's tokens, the corpus's per-document token ids and the vectors being stored at once: 4.9 GB on the kernel, on top of the floor, and nothing sized it to the headroom. Now, when the charged footprint plus that transient would cross the budget, phases 2-4 run per batch of functions: tokenize -> count into the corpus -> free; finalize once; tokenize again -> vectorize -> store -> free, with pages returned to the OS after every batch. Tokenization is deterministic, so both passes see the same tokens and the graph does not change; the run pays with a second tokenize. CBM_SEM_BATCH=<n> forces the batch size (the batched-equals-unbatched test). The corpus frees its per-document token ids at the end of finalize (the co-occurrence pass is their only reader), and mem.semantic.step lines (CBM_MEM_PHASES=1) put the class's live/peak bytes and the charged footprint at every sub-phase. The semantic corpus and tokenizer allocate through the core (54 raw sites -> 0). This is not attribution only: cbm_sem_tokenize handed out libc strdup blocks that pass_semantic_edges freed with cbm_free -- mi_free in the production build -- since the pass moved onto the core. No libc-backed test build can see a cross-allocator free, so the core now refuses a block it does not own (mi_is_in_heap_region) when CBM_MEM_PHASES=1 is set, fatally: every proof run in this PR ran with it armed, and the Go and kernel indexes report zero foreign blocks. Accounting without contention. The core charged every allocation and free to shared per-class atomics; with 18 workers on 18 cores every block bounced the same three cache lines, and the bench against the shipped v0.10.8 showed it as CPU time rising faster than wall time for the same graph: Kotlin 38 -> 121 s CPU (+58% wall), TypeScript 210 -> 596 s (+67%), Go 177 -> 312 s. The deltas are thread-local now and reach the shared counters once per 256 KB or 512 blocks of change per thread, when a parallel-for worker ends, and whenever a reader looks (its own thread first). Phase marks read after the join and are exact; a class peak is a diagnostic and can be low by threads x 256 KB. Result: Kotlin 10.9 s / 43 s CPU (the residual +22% wall against v0.10.8 tracks its +34% nodes), TypeScript 33.9 s (v0.10.8: 33.4), Go 48.4 s (v0.10.8: 52.6), Java 81.1 s (v0.10.8: 79.8); graphs unchanged. SQLite on a heap of its own. The same bench put the kernel's write step at 132 s where v0.10.8 needed 9.7 s, all of it in one coverage publish statement. A stack sample named the cost: mimalloc's free-page search and its periodic heap collect, walking the graph's pages -- millions of them once the graph lives on the core -- for every statement-journal chunk SQLite allocates. SQLite now allocates from a mimalloc heap of its own per thread (mi_heap_new; mi_free is heap-agnostic, so nothing else changes), and the periodic collect is set to the maximum mimalloc allows since the explicit release at phase marks and spill sweeps is where memory goes back. Coverage publish 132 -> 9.4 s; profiled kernel 376 -> 248 s (v0.10.8: 296 s). Formatting without a process lock. On macOS every vsnprintf takes the process locale under an unfair lock (localeconv_l inside __vfprintf); eighteen workers formatting type names on the C# corpus collapsed into it -- the 10 MB JIT test files took 68 s each instead of under a second -- and cbm_arena_sprintf paid it twice per string (size pass, write pass). Each thread now formats with a C locale object of its own (vsnprintf_l), and the common short string is formatted once into a stack buffer. Per-file budgets that cover the whole file. The 5 s per-file budget covered the parse only; the LSP walks after it had none. C# JIT stress files (a 23 MB single expression among them) parsed inside the budget -- the old wall-clock parse timeout used to drop them, which hid the rest -- and then held a worker for 346 s each in the usage walk (tree-sitter's ts_node_parent descends from the root: quadratic on a deep tree), and the cross-file resolve ran on the same trees. Three rules now, one site for every language (cbm_extract_file_ex, honoured by cbm_pxc_dispatch_file): a parse that used more than half the budget disqualifies the file from the LSP walks (lsp_skipped); the unified walk checks its thread CPU time every 1024 nodes against six budgets and stops there, keeping what it found (walk_truncated, implies lsp_skipped); a file whose parse plus walk spent the budget is skipped by the LSP walks as well. Each decision is logged with the path (extract.lsp.skipped, extract.walk.truncated). C# extraction 355 -> 39 s; the whole index 463 -> 128 s wall and 27 -> 11 GB peak against v0.10.8 with 299 more nodes. The walk cuts exactly the five JIT stress files (hugeexpr1, hugeSimpleExpr1, HugeArray1, HugeField1/2); the 15 MB System.Runtime.Intrinsics reference file walks to the end and, with four generic-nesting JIT regression tests, only skips the LSP walk under the parse-plus-walk rule. The crash behind the C# bench. dotnet/runtime is 42,555 files, and its resolve phase died with SIGBUS on every run of this branch. A libc-backed ASan build of the server named it: heap-use-after-free in c_adl_resolve, reading a type name that another worker had allocated in its per-file scratch arena and freed at the end of its file. The C++ class walk refines a method's return type when the declaration is more specific than the pre-registered one (NAMED -> pointer/reference/template), and it did that by casting the chained lookup result to non-const and writing a scratch-arena signature into it -- into the sealed shared base when the entry lived there. The overlay contract had exactly one bypass, and it was this cast; every lookup returns const, so a grep for the cast is the audit. The refinement now goes through cbm_registry_func_for_update: copy-on-write into the overlay, the base untouched. Test: clsp_method_return_refinement_is_copy_on_write (a sealed base with a NAMED method return, an in-class pointer declaration; the base signature pointer is unchanged and the overlay copy carries the refinement plus a marker only a copy has) -- RED on the cast, GREEN on the accessor. The fixed ASan build then indexed the whole corpus clean: 42,555 files, 1,224,981 nodes, 5,794,320 edges, worker exit 0, no report. Production build backs the core with mimalloc explicitly (never the global override, which macOS's two-level namespace forbids); the test build keeps libc so the sanitizers see every block -- the same split the tree-sitter and SQLite bindings already use. PROOF Go peak RSS 16.7 -> 4.0-5.2 GB; result arenas 14,804/28,817 -> 856/856 MB; wall 55 -> 52-60 s; graph inside the pre-existing run-to-run jitter (gRPC Route nodes: two identical runs differ by 21). Go CBM_MEM_SPILL=1: peak RSS 1.7-2.3 GB, 21,875 results parked (950 MB on disk), graph identical: 39 infra routes, 158 INFRA_MAPS, 21,875 surface rows, nodes/edges inside the Route jitter. Go CBM_MEM_BUDGET_MB=1200: spill latched at 1,204 MB, the first sweep took the charge to 768 MB, the run completed at 1.99 GB peak RSS. Kernel DEFAULT budget (24,576 MB): aborted before, completes now without spilling. Extraction 35.2 -> 16.5 GB RSS (5.6 GB footprint, 15.3 GB committed); results 8.9 GB = capacity; resolve tracked 24.6 -> 19.1 GB; peak RSS 27.3 -> 20.1 GB. Nodes identical (8,529,729); edges inside the +-400 run-to-run band the kernel shows between identical runs. Kernel CBM_MEM_BUDGET_MB=16000: aborted before, completes now through the spill path. Latched at 17.2 GB charged; 89,731 results parked (9.3 GB on disk, 370,188 loads); extraction end 6.2 GB RSS / 5.1 GB committed; resolve 8.6 GB RSS / 11.5 GB committed; peak RSS 16.6 GB (the semantic pass plateau, 13.3 GB RSS, is graph-derived and outside what spill can move). Nodes identical; edges 15,773,598. Kernel CBM_MEM_BUDGET_MB=15000, before the semantic batching: completes, every phase mark under 15,000 MB by every metric, but the process RSS high-water mark reached 16.9 GB inside the semantic pass (class peak 4.9 GB against 0.8 GB live). Kernel CBM_MEM_BUDGET_MB=15000, with it: completes; the charged footprint stays under 15,000 MB through the whole semantic pass (14,018 -> 14,381 MB across its sub-phases, 6 batches of 125,644 functions); every phase mark is under budget by every metric (max footprint 12.6 GB, max commit 11.5 GB); the process RSS high-water mark is 15.65 GB and is set in extraction, at the moment the gate latches (15,002 MB charged) and before the first sweep lands -- a 4% overshoot of in-flight work, no longer the semantic pass. Nodes identical, edges 15,772,913, zero foreign blocks. Kernel CBM_MEM_BUDGET_MB=15000, with the anticipatory latch: spill mode enters at 14,062 MB (near_budget); the charged footprint never exceeds 11.8 GB at any mark; the process RSS high-water mark is 15.1 GB in extraction and 15.3 GB during the final write-out -- 1-2% over, the in-flight window and the dump transient. Nodes identical, edges 15,772,543, zero foreign blocks. Go CBM_MEM_BUDGET_MB=1200 with it: latched at 1,131 MB, extraction peak 998 MB RSS, graph identical to the baseline. BENCH AGAINST THE SHIPPED v0.10.8 (same machine, same driver, 2026-09-14) corpus wall s peak RSS GB nodes / edges / CALLS perl 5.7 -> 5.3 0.09 -> 0.05 -3 / -130 / -3 php 6.3 -> 5.8 0.66 -> 0.20 +21.9% / +1.4% / +23.8% rust 6.8 -> 6.3 1.15 -> 0.67 +1 / +56 / -369 c 5.9 -> 5.7 0.16 -> 0.19 identical kotlin 8.9 -> 9.4 1.55 -> 0.96 +33.5% / -22.8% / +12.1% django 10.9 -> 10.2 3.13 -> 1.12 +6 / -14.5% / -28.4% go 52.6 -> 42.9 17.2 -> 3.9 +14.2% / -10.3% / +2.2% java 79.8 -> 73.8 27.6 -> 11.0 -121 / -1.9% / -0.6% csharp 463.3 -> 128.1 27.0 -> 11.2 +299 / -0.9% / -0.6% typescript 33.4 -> 33.6 12.0 -> 2.5 -186 / -3.5% / +130 kernel 294.8 -> 233.3 27.5 -> 25.1 -171 / -1.3% / -0.1% No corpus is slower beyond noise; CPU time follows wall (kernel 2,442 -> 1,403 s). Every graph delta against v0.10.8 is main between the release and the branch base (339b3f40), not this PR: a build of the base commit run through the same driver reproduces the final graphs exactly on django and kotlin, within the pre-existing gRPC Route jitter on go (30 nodes, 27 edges, CALLS identical), and on C# with the same node count. This PR's own C# difference against the base is 5 CALLS (the five truncated JIT stress files) plus one ijwhost swap and 563 edges of 5.79 M; the base takes 448 s / 29.7 GB for that corpus. Diagnostics that made this measurable stay in: the extract.arenas and extract.census lines (behind CBM_MEM_PHASES=1), the CBM_MEM_RELEASE=1 release-to-OS probe, and peak_charged_mb on every mem.phase line -- the high-water mark of the budget metric itself (cbm_mem_peak_charged), next to the RSS peak that counts pages already purged but not yet reclaimed. Tests: extract_compact_* and extract_spill_round_trip_keeps_every_field (extraction), parallel_spill_mode_builds_the_same_graph (parallel: every file parked, node count and every edge type equal to the in-memory run), pipeline_semantic_batched_matches_unbatched (pipeline: one repo indexed unbatched and with CBM_SEM_BATCH=5 -- byte-identical node vectors, token vectors and SEMANTICALLY_RELATED edges), extract_lsp_skipped_when_parse_used_ its_budget_share and extract_walk_truncated_at_its_cpu_budget (extraction: the seams mark a file / stop the walk, the LSP walk and the dispatcher skip it, the defs stay), registry_overlay_chain_iterates_ and_copies_on_write (c_lsp), arena exact-init/growth, mem charged/footprint/ pressure. Not in this increment: run-to-run edge jitter (+-400 on the kernel, Route class on Go) predates this work and has its own item; the incremental route spilling; a tighter result encoding (def ids instead of QN strings in resolved calls) is the next lever on the 9 GB the kernel's results still take. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 1 month ago | |
fix(hooks): scope pre-commit clang-tidy to staged changes (#1264) make -j3 -f Makefile.cbm lint runs the full lint-tidy sweep, which currently surfaces ~5,100 pre-existing clang-tidy findings across 113 files. Since the pre-commit hook (scripts/hooks/pre-commit) runs that target unconditionally, it blocks every commit for every contributor who has clang-tidy on PATH, regardless of what the commit touches. Add lint-tidy-diff (scripts/lint-tidy-diff.sh), which runs the same clang-tidy binary/config but passes clang-tidy's own -line-filter so only lines the commit actually added or modified can produce a diagnostic. Pre-existing findings on untouched lines are not reported, whether they are in an untouched file or on an untouched line of a file the commit does touch. The hook now runs lint-ci (unchanged: cppcheck + clang-format + no-suppress) plus lint-tidy-diff instead of the full lint target; make lint / make lint-tidy / scripts/lint.sh are unchanged and remain the full-tree audit. Signed-off-by: Yyunozor <yyunozor@icloud.com> | 3 months ago | |
ci: unify smoke/soak/test venues onto canonical leg entries The staged venue-unification bundle: _smoke.yml onto the shared wrappers with an extracted-artifact input (CBM_SMOKE_ARTIFACT_DIR), soak-legs.sh as the one canonical soak entry (quick + query-leak legs across all venues), protected per-user TEMP roots shared across venues, clean-disk preflights before every VM/Docker run, win.sh and run.sh routed through the canonical scripts, soak.yml retired, and the venue-parity contract (whitelist walker: workflows may provision or call canonical entries, nothing else). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 3 months ago | |
feat(qa): memory + CPU waste sanitizer over the memory core Answers a question the existing tooling could not: of everything we allocate, how much is never used? Lanes, all driven by scripts/memwaste.sh: event every allocation observed through the memory core — size, age, churn, and whether the bytes were ever written access SanitizerCoverage load/store instrumentation, so "allocated and never touched" is measured rather than inferred scaling k vs 2k replicas through clang profile counters, to catch super-linear work per file padding allocator slack per size class ratchet the numbers recorded per venue so a regression is visible The event lane needed a foreign-free counter: on Windows the override sees frees for blocks it never allocated (they come from the CRT), and counting those as untracked tripped the soundness gate. mem_events now separates them (14 daemon / 5 worker on the Go corpus). Two portability fixes the lanes surfaced, both real: - the enable decision must not be cached before `environ` exists. Under -fsanitize-coverage the UBSan preinit runs dlsym -> malloc -> our wrapper before glibc sets up environ, so the lane silently disabled itself and produced no dump at all on Linux. - the access map needs 48 bits: arm64 Linux hands out addresses above 2^47, which a 47-bit map dropped on the floor. mem_profile is retired into this: it measured a strict subset and had its own sampling story. Local-only by decision — not a PR-CI gate. _memwaste.yml runs the lanes on all three platforms report-only, and pr.yml/release.yml carry the wiring. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 1 month ago | |
feat(qa): memory + CPU waste sanitizer over the memory core Answers a question the existing tooling could not: of everything we allocate, how much is never used? Lanes, all driven by scripts/memwaste.sh: event every allocation observed through the memory core — size, age, churn, and whether the bytes were ever written access SanitizerCoverage load/store instrumentation, so "allocated and never touched" is measured rather than inferred scaling k vs 2k replicas through clang profile counters, to catch super-linear work per file padding allocator slack per size class ratchet the numbers recorded per venue so a regression is visible The event lane needed a foreign-free counter: on Windows the override sees frees for blocks it never allocated (they come from the CRT), and counting those as untracked tripped the soundness gate. mem_events now separates them (14 daemon / 5 worker on the Go corpus). Two portability fixes the lanes surfaced, both real: - the enable decision must not be cached before `environ` exists. Under -fsanitize-coverage the UBSan preinit runs dlsym -> malloc -> our wrapper before glibc sets up environ, so the lane silently disabled itself and produced no dump at all on Linux. - the access map needs 48 bits: arm64 Linux hands out addresses above 2^47, which a 47-bit map dropped on the floor. mem_profile is retired into this: it measured a strict subset and had its own sampling story. Local-only by decision — not a PR-CI gate. _memwaste.yml runs the lanes on all three platforms report-only, and pr.yml/release.yml carry the wiring. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 1 month ago | |
Merge pull request #2197 from astandrik/fix/1696-memlab-runtime-isolation fix(test): isolate memlab harness daemon runtime | 1 month ago | |
Lower the memory-core baseline for src/mcp/mcp.c Signed-off-by: tahodev <85001874+tahodev@users.noreply.github.com> | 6 days ago | |
feat(qa): memory + CPU waste sanitizer over the memory core Answers a question the existing tooling could not: of everything we allocate, how much is never used? Lanes, all driven by scripts/memwaste.sh: event every allocation observed through the memory core — size, age, churn, and whether the bytes were ever written access SanitizerCoverage load/store instrumentation, so "allocated and never touched" is measured rather than inferred scaling k vs 2k replicas through clang profile counters, to catch super-linear work per file padding allocator slack per size class ratchet the numbers recorded per venue so a regression is visible The event lane needed a foreign-free counter: on Windows the override sees frees for blocks it never allocated (they come from the CRT), and counting those as untracked tripped the soundness gate. mem_events now separates them (14 daemon / 5 worker on the Go corpus). Two portability fixes the lanes surfaced, both real: - the enable decision must not be cached before `environ` exists. Under -fsanitize-coverage the UBSan preinit runs dlsym -> malloc -> our wrapper before glibc sets up environ, so the lane silently disabled itself and produced no dump at all on Linux. - the access map needs 48 bits: arm64 Linux hands out addresses above 2^47, which a 47-bit map dropped on the floor. mem_profile is retired into this: it measured a strict subset and had its own sampling story. Local-only by decision — not a PR-CI gate. _memwaste.yml runs the lanes on all three platforms report-only, and pr.yml/release.yml carry the wiring. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 1 month ago | |
feat(qa): memory + CPU waste sanitizer over the memory core Answers a question the existing tooling could not: of everything we allocate, how much is never used? Lanes, all driven by scripts/memwaste.sh: event every allocation observed through the memory core — size, age, churn, and whether the bytes were ever written access SanitizerCoverage load/store instrumentation, so "allocated and never touched" is measured rather than inferred scaling k vs 2k replicas through clang profile counters, to catch super-linear work per file padding allocator slack per size class ratchet the numbers recorded per venue so a regression is visible The event lane needed a foreign-free counter: on Windows the override sees frees for blocks it never allocated (they come from the CRT), and counting those as untracked tripped the soundness gate. mem_events now separates them (14 daemon / 5 worker on the Go corpus). Two portability fixes the lanes surfaced, both real: - the enable decision must not be cached before `environ` exists. Under -fsanitize-coverage the UBSan preinit runs dlsym -> malloc -> our wrapper before glibc sets up environ, so the lane silently disabled itself and produced no dump at all on Linux. - the access map needs 48 bits: arm64 Linux hands out addresses above 2^47, which a 47-bit map dropped on the floor. mem_profile is retired into this: it measured a strict subset and had its own sampling story. Local-only by decision — not a PR-CI gate. _memwaste.yml runs the lanes on all three platforms report-only, and pr.yml/release.yml carry the wiring. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 1 month ago | |
feat(qa): memory + CPU waste sanitizer over the memory core Answers a question the existing tooling could not: of everything we allocate, how much is never used? Lanes, all driven by scripts/memwaste.sh: event every allocation observed through the memory core — size, age, churn, and whether the bytes were ever written access SanitizerCoverage load/store instrumentation, so "allocated and never touched" is measured rather than inferred scaling k vs 2k replicas through clang profile counters, to catch super-linear work per file padding allocator slack per size class ratchet the numbers recorded per venue so a regression is visible The event lane needed a foreign-free counter: on Windows the override sees frees for blocks it never allocated (they come from the CRT), and counting those as untracked tripped the soundness gate. mem_events now separates them (14 daemon / 5 worker on the Go corpus). Two portability fixes the lanes surfaced, both real: - the enable decision must not be cached before `environ` exists. Under -fsanitize-coverage the UBSan preinit runs dlsym -> malloc -> our wrapper before glibc sets up environ, so the lane silently disabled itself and produced no dump at all on Linux. - the access map needs 48 bits: arm64 Linux hands out addresses above 2^47, which a 47-bit map dropped on the floor. mem_profile is retired into this: it measured a strict subset and had its own sampling story. Local-only by decision — not a PR-CI gate. _memwaste.yml runs the lanes on all three platforms report-only, and pr.yml/release.yml carry the wiring. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 1 month ago | |
feat(qa): memory + CPU waste sanitizer over the memory core Answers a question the existing tooling could not: of everything we allocate, how much is never used? Lanes, all driven by scripts/memwaste.sh: event every allocation observed through the memory core — size, age, churn, and whether the bytes were ever written access SanitizerCoverage load/store instrumentation, so "allocated and never touched" is measured rather than inferred scaling k vs 2k replicas through clang profile counters, to catch super-linear work per file padding allocator slack per size class ratchet the numbers recorded per venue so a regression is visible The event lane needed a foreign-free counter: on Windows the override sees frees for blocks it never allocated (they come from the CRT), and counting those as untracked tripped the soundness gate. mem_events now separates them (14 daemon / 5 worker on the Go corpus). Two portability fixes the lanes surfaced, both real: - the enable decision must not be cached before `environ` exists. Under -fsanitize-coverage the UBSan preinit runs dlsym -> malloc -> our wrapper before glibc sets up environ, so the lane silently disabled itself and produced no dump at all on Linux. - the access map needs 48 bits: arm64 Linux hands out addresses above 2^47, which a 47-bit map dropped on the floor. mem_profile is retired into this: it measured a strict subset and had its own sampling story. Local-only by decision — not a PR-CI gate. _memwaste.yml runs the lanes on all three platforms report-only, and pr.yml/release.yml carry the wiring. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 1 month ago | |
fix(memwaste): index under one explicit budget on every venue scripts/memwaste.sh set no memory budget, so the daemon derived one from the machine's RAM: 1792 MB on the 7 GB macos-14 runner, about 4 GB on the 16 GB Linux and Windows runners. The event layer itself holds 0.5-1 GB, and on macOS the index stopped with over_memory_budget before it measured anything, in both full-mode runs since the lane was added (dry runs 37212102899 and 38186491924, peaks 3449 and 3111 MB at the stop). The script now passes CBM_MEM_BUDGET_MB=4096, what the larger runners already derived, unless the caller sets one. Checked on macOS arm64 with CBM_WORKERS=3, the macos-14 runner's core count: - budget 1792: over_memory_budget in parallel_extract (rss 3904 MB), the CI failure; - budget 4096: the index completes and the report is written; - the edited script with no budget in the environment: 4096 is applied, rc 0. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 27 days ago | |
fix(msan): reports print with fast unwinding; five suites run again Five suites (grammar_regression grammar_labels pipeline lang_contract grammar_probe_e) were excluded from the MSan lane for "stack-overflow under instrumentation". That was never deep recursion. MSan aborted while REPORTING real uninitialized reads: - every abort was on the main thread. MSan's "T<n>" is the OS thread id and equalled the pid in all five x86-64 aborts (CI job 91868259558); the old block read it as a worker-thread ordinal, so the worker-stack knob, MSAN_ORIGINS and CBM_WORKERS=1 could never change anything; - with the default slow unwinder (the image's MSan-instrumented libunwind) a report exhausted the main thread's 256 MiB stack, and the overflow handler then failed with "nested bug in the same thread"; - with fast_unwind_on_fatal=1 the same test prints the use-of-uninitialized-value report instead (measured on one tree and test: slow unwind aborts at the stack limit, fast unwind reports wit/scanner.c:13). The two reads are fixed in the previous commits of this series, so: - MSAN_OPTIONS defaults to fast_unwind_on_fatal=1 (the build already keeps frame pointers); - MSAN_EXCLUDE is down to cli, whose install-path failure is a separate, still-undiagnosed cause; - the known-red block, the header (it still claimed CI runs without exclusions) and the MSAN_ORIGINS and stack comments say what is true; - the partial-lane warning and the exclude-name typo check lived in a branch behind `elif true` that never ran. They now run in the per-suite path, the check as a case match instead of a grep -q pipe that pipefail can misreport; - the comments in scripts/ci/msan-lane.sh and above the test-msan job in _test.yml no longer say the five suites overflow. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 27 days ago | |
Drop the nim grammar and refresh language counts Removes the heaviest vendored grammar (66 MB) along with its language wiring and test fixtures; the supported-language count moves to 158 across README, site, and manifest. The grammar manifest additionally records the canonical-source decisions for the five registry-disagreement grammars and the per-directory license files restored earlier this week. | 4 months ago | |
docs: refresh project messaging and Lens website Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 18 days ago | |
daemon: permanent lifecycle, daemon-backed CLI/hooks, real-Windows hardening, long-path launcher transactions Daemon lifecycle and Windows correctness, verified on a real Windows 11 ARM64 VM through the maintained test-infrastructure/vm drivers, plus the macOS and Linux arm64 suites and the container lint gate. Daemon lifecycle: - daemon start/stop/status subcommands. `daemon start` launches a PERMANENT daemon (spawn shape is byte-exact argv; survives idle periods and session ends) and reports an already-active daemon instead of failing. Permanence is honored at every stop latch: last-committed-client disconnect, host initial-client window, coordinator release, and application final-session close — a permanent daemon also keeps admitting new sessions after its last one closes. - daemon stop refuses while sessions are active and lists the blocking peers (pid/role) that must finish first; an idle daemon drains through the activation-shutdown machinery with the ACK ordered after connection interrupts. A second stop is idempotent. The wire ops are no-cohort first-frame requests with peer fingerprint authentication, so stop/status never conflict with an exact-build admission gate. - One-shot CLI commands now execute through the daemon (index workers keep their local supervised path). A cold CLI run that had to spawn a temporary daemon prints a hint that `daemon start` removes the per-command startup tax; a warm daemon is recycled silently. - Hooks are connect-only fail-open: with no daemon present the hook emits a visible, rate-limited notice (Claude-dialect systemMessage plus stderr for other dialects) and always exits 0 — augmentation is never allowed to block the caller's tool use. - Version skew: a newer-build client automatically drains an older-build permanent daemon (strict semantic-version triples only; dev builds never auto-drain) and the build-conflict message names `cbm daemon stop` as the manual escape hatch. Windows IPC/runtime (real-VM verified): - ipc(win): persistent pending overlapped ConnectNamedPipe. The accept path used to destroy its listening pipe instance on every 20 ms poll timeout; a client attaching in the teardown window was severed or left on an orphaned pipe object whose HELLO no server handle could ever read, absorbing the connect until the client's own timeout expired. The pending connect now survives poll timeouts and nothing is destroyed while a client could be attaching. - ipc(win): drain-before-close for final responses. Closing a named-pipe server handle can discard a just-sent response before the peer reads it (POSIX stream sockets never lose buffered data on close). A bounded cbm_daemon_ipc_connection_drain (read-until-peer-EOF; no-op on POSIX, immediate on interrupted connections) now precedes close in runtime_worker_finish and runtime_reject_inline, so hello-conflict, capacity and disconnect acknowledgements reliably reach the peer. - runtime: CLOSE_INTENT wire frame. A Windows named-pipe client has no transport half-close, so close_begin now announces departure with an explicit frame (ordered after APPLICATION_CANCEL, before the local interrupt); the server releases the client's admission on receipt instead of waiting for the handle to close. Admission-drop timing is now identical to POSIX shutdown() semantics on every platform. - runtime(win): client close cancellation. close_begin serializes with request publication under the send lock, best-effort sends the active token's APPLICATION_CANCEL frame, then interrupts local I/O; the server cancels MCP/subprocess work promptly. Contract tests accept both correct outcomes (interrupted transport or decoded CANCELLED). - runtime: activation acknowledgement ordering. The activation ACK is the requester's license to act on "snapshotted and draining", so every connection interrupt is now initiated before the ACK is sent; a session could previously get one more request serviced after the requester observed the ACK. - service(win): deadline-bounded private-file prepare. The conflict-log prepare retry loop (100 x Sleep(2), which rounds up to the ~16 ms timer granularity) burned ~1.6 s against permanently obstructed paths, stalling hello rejections past the client's timeout. The retry budget is now a 250 ms deadline; transient share collisions still retry. - subprocess(win): cmd.exe /C payload encoder quotes metacharacters correctly (root cause of the git-on-Windows failure cluster). - watcher: SHA-256 buffer sizing (CBM_SZ_64 -> CBM_SZ_128) and a native Windows stop/unwatch cancellation test with exact-image verification. - httpd: send_all writes in bounded 64 KiB slices. A single giant nonblocking send() on Windows is absorbed wholesale into AFD kernel buffering regardless of SO_SNDBUF, so send deadlines and interrupts could never engage against a slow peer (and the full payload was pinned in nonpaged pool). Slicing restores a deterministic backpressure point; a test hook pins SO_SNDBUF for the deadline and interrupt tests. - ui/http: shutdown lifecycle — interrupt checks, response-wide send deadline, explicit connection states, refusal to free a server while a listener-owned connection is active. Windows long-path support: - Central path-aware wide conversion (canonicalize via GetFullPathNameW and prepend the extended-length prefix for absolute paths >=240) at the compat chokepoints (cbm_fopen/compat_fs/mkstemp/mkdtemp), sqlite store opens, and the daemon build-fingerprint/log paths. Deep managed installs (a 64-hex generation directory routinely exceeds MAX_PATH) now index, stage and activate correctly. - activation transaction: its own file APIs and the component-walking ancestry validators now operate in the extended-length namespace; the launcher path is canonicalized (and prefixed when deep) once at entry so every downstream exact-string comparison stays form-consistent. - Executable self-resolution uses the wide APIs (GetModuleFileNameW, GetFileAttributesW) so non-ASCII install paths survive argv[0] resolution. Windows launcher install/uninstall transaction: - FileRenameInfoEx names are NUL-terminated in an over-allocated buffer. FileNameLength governs per the contract, but filter drivers read FileName as NUL-terminated and appended adjacent heap bytes to created names — a flaky, garbage-suffixed rename target. Both the CLI and the launcher rename helpers are fixed. - Uninstall retires state via rename-aside (.cbm -> .cbm-retired-v1-<tag>-<pid>) with the retired tag shortened to 16 hex chars so the bare rename target stays under the FileRenameInfoEx NT-conversion ceiling at guard depths; 64 bits still uniquely identify the generation. - When the running launcher's mapped generation backings pin .cbm against rename, the backings are relocated to activation-<pid>-N .retired tombstones beside the install (a mapped image may be renamed, never deleted; the launcher's liveness-guarded sweep reclaims stale tombstones). Every relocation is recorded, and a FAILED uninstall reverses the moves after restoring .cbm — via MoveFileExW with extended-length paths on both arguments, since the deep generation target is beyond the handle-based rename's bare-path reach — so a restored install keeps its generation backings and stays runnable. - After a committed uninstall the retired tree's backings are relocated out so the tree is shallow enough for the detached cleanup's rd, and the cleanup's working directory strips the extended-length prefix (CreateProcessW lpCurrentDirectory silently ignores prefixed paths). - Files created under Administrators-default-owner directories (CopyFileW destinations, CREATE_NEW tombstones, probe directories) are explicitly owner-stamped so the exact-owner validators hold on runner images; guard fixtures stamp hand-built trees the same way. Diagnostics, tests and infra: - diagnostics: discovery is now an always-delivered JSON control record (new cbm_log_control) that survives CBM_LOG_LEVEL suppression and paths containing spaces; placement honors $TMPDIR with /tmp fallback via a diagnostics-local helper; the soak parser reads the JSON record; documented in docs/CONFIGURATION.md. Red-first coverage for suppressed log levels, TMPDIR-with-spaces, and native Windows output-contract assertions. - tests(win): daemon_ipc/daemon_frontend fixtures now build endpoint parents with production-shaped ancestry (LocalAppData on Windows, via th_secure_runtime_parent_new) — the runtime ancestry validation correctly refuses temp roots whose ancestors grant mutation rights to Authenticated Users (C:/msys64/tmp, GitHub-runner work dirs) — and drive the documented startup-owner publication flow before reading generation-bound endpoint addresses. This turns the 26 Windows failures previously visible in CI's full-test job green without weakening any validation. - tests(win): the launcher guard covers the full permanent-launcher contract including failed-uninstall restore and immediate reinstall after uninstall; new daemon lifecycle and reworked hook-augment guards run the start/recycle/stop flow end to end. - tests: CBM_SKIP_PERF is now actually consumed by the test runner (it was set by CI but never read, so perf suites ran everywhere); four throughput/bench suites are classified as perf, the heavy store_arch suite moved to the slow-timeout tier, and two wall-clock-sensitive assertions were rewritten as invariant checks with coarse hang-detector backstops. - build/test infra: build-dir safety contract, UI dev-proxy security contract, soak daemon-recovery contract, path-safety helper, the Windows VM worktree-sync contract wired into scripts/test.sh, and vm/win.sh guards building its clean embedded-UI product in an isolated BUILD_DIR so it cannot clobber the incremental test build. provision-windows.sh now installs Node.js for the guards UI build. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 3 months ago | |
fix: preserve semantic graph relationships Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 3 months ago | |
test(harness): show sanitizer reports in the suite failure summary A Windows sanitizer job went red on one suite that aborted under AddressSanitizer, and the job log showed an empty "every failure site" section followed by the shadow-byte legend. The summary in scripts/run-tests-parallel.sh searched the suite log for "FAIL" only. A sanitizer report never contains that word, and its useful part (error kind, access size, top frames) is at the start of the report, far above the last 15 lines. The summary now lives in scripts/suite-failure-report.sh, which the harness sources. Between the unchanged failure-site and last-15-lines sections it prints, for a failing suite, the test that was running and the first sanitizer report. The framework flushes each test name before the test body runs, so the last test line ahead of a report is the test that was running; a report that comes after the last test finished is labelled as such instead of blaming a test. The report is shown from its header (AddressSanitizer, LeakSanitizer, ThreadSanitizer, MemorySanitizer, UndefinedBehaviorSanitizer) to its own SUMMARY line, at most 40 lines, without the shadow dump and the legend; a recoverable "runtime error:" line is shown with its stack only. Further sanitizer headers, "runtime error:" lines and SUMMARY lines follow (at most 10), then signal and timeout lines (at most 5). A suite that dies with no report and no completion summary gets the running test named. A trap-mode UBSan build stops on an illegal instruction and prints nothing, so that name is the only evidence its log holds. A log with an assertion failure and no report prints exactly what it printed before. The same file keeps the logs a red run needs. An EXIT trap in the harness copies the log of every suite that did not record rc=0, plus results.txt, into test-logs/failed/. The trap covers the scheduler giving up and the union guard as well as the normal end, and it never changes the exit status. scripts/run-test-wave.py reported pass=0 for a suite that died without its summary line. In that case, and only then, it now counts the PASS markers in the log. A suite with a summary line is never recounted, so the totals of a green run are unchanged. tests/test_failure_report_contract.sh drives the real functions with fabricated suite logs and runs as Step 0i2 of scripts/test.sh. It fails on the previous behaviour in every sanitizer, running-test and result-line check and passes on it in the unchanged-output checks. The awk code uses no interval expressions and no character classes; the test passes with BSD awk and grep on macOS and with mawk and GNU grep on Ubuntu. scripts/README.md names the new internal file. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 26 days ago | |
test(harness): show sanitizer reports in the suite failure summary A Windows sanitizer job went red on one suite that aborted under AddressSanitizer, and the job log showed an empty "every failure site" section followed by the shadow-byte legend. The summary in scripts/run-tests-parallel.sh searched the suite log for "FAIL" only. A sanitizer report never contains that word, and its useful part (error kind, access size, top frames) is at the start of the report, far above the last 15 lines. The summary now lives in scripts/suite-failure-report.sh, which the harness sources. Between the unchanged failure-site and last-15-lines sections it prints, for a failing suite, the test that was running and the first sanitizer report. The framework flushes each test name before the test body runs, so the last test line ahead of a report is the test that was running; a report that comes after the last test finished is labelled as such instead of blaming a test. The report is shown from its header (AddressSanitizer, LeakSanitizer, ThreadSanitizer, MemorySanitizer, UndefinedBehaviorSanitizer) to its own SUMMARY line, at most 40 lines, without the shadow dump and the legend; a recoverable "runtime error:" line is shown with its stack only. Further sanitizer headers, "runtime error:" lines and SUMMARY lines follow (at most 10), then signal and timeout lines (at most 5). A suite that dies with no report and no completion summary gets the running test named. A trap-mode UBSan build stops on an illegal instruction and prints nothing, so that name is the only evidence its log holds. A log with an assertion failure and no report prints exactly what it printed before. The same file keeps the logs a red run needs. An EXIT trap in the harness copies the log of every suite that did not record rc=0, plus results.txt, into test-logs/failed/. The trap covers the scheduler giving up and the union guard as well as the normal end, and it never changes the exit status. scripts/run-test-wave.py reported pass=0 for a suite that died without its summary line. In that case, and only then, it now counts the PASS markers in the log. A suite with a summary line is never recounted, so the totals of a green run are unchanged. tests/test_failure_report_contract.sh drives the real functions with fabricated suite logs and runs as Step 0i2 of scripts/test.sh. It fails on the previous behaviour in every sanitizer, running-test and result-line check and passes on it in the unchanged-output checks. The awk code uses no interval expressions and no character classes; the test passes with BSD awk and grep on macOS and with mawk and GNU grep on Ubuntu. scripts/README.md names the new internal file. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 26 days ago | |
feat(qa): memory + CPU waste sanitizer over the memory core Answers a question the existing tooling could not: of everything we allocate, how much is never used? Lanes, all driven by scripts/memwaste.sh: event every allocation observed through the memory core — size, age, churn, and whether the bytes were ever written access SanitizerCoverage load/store instrumentation, so "allocated and never touched" is measured rather than inferred scaling k vs 2k replicas through clang profile counters, to catch super-linear work per file padding allocator slack per size class ratchet the numbers recorded per venue so a regression is visible The event lane needed a foreign-free counter: on Windows the override sees frees for blocks it never allocated (they come from the CRT), and counting those as untracked tripped the soundness gate. mem_events now separates them (14 daemon / 5 worker on the Go corpus). Two portability fixes the lanes surfaced, both real: - the enable decision must not be cached before `environ` exists. Under -fsanitize-coverage the UBSan preinit runs dlsym -> malloc -> our wrapper before glibc sets up environ, so the lane silently disabled itself and produced no dump at all on Linux. - the access map needs 48 bits: arm64 Linux hands out addresses above 2^47, which a 47-bit map dropped on the floor. mem_profile is retired into this: it measured a strict subset and had its own sampling story. Local-only by decision — not a PR-CI gate. _memwaste.yml runs the lanes on all three platforms report-only, and pr.yml/release.yml carry the wiring. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 1 month ago | |
feat(qa): memory + CPU waste sanitizer over the memory core Answers a question the existing tooling could not: of everything we allocate, how much is never used? Lanes, all driven by scripts/memwaste.sh: event every allocation observed through the memory core — size, age, churn, and whether the bytes were ever written access SanitizerCoverage load/store instrumentation, so "allocated and never touched" is measured rather than inferred scaling k vs 2k replicas through clang profile counters, to catch super-linear work per file padding allocator slack per size class ratchet the numbers recorded per venue so a regression is visible The event lane needed a foreign-free counter: on Windows the override sees frees for blocks it never allocated (they come from the CRT), and counting those as untracked tripped the soundness gate. mem_events now separates them (14 daemon / 5 worker on the Go corpus). Two portability fixes the lanes surfaced, both real: - the enable decision must not be cached before `environ` exists. Under -fsanitize-coverage the UBSan preinit runs dlsym -> malloc -> our wrapper before glibc sets up environ, so the lane silently disabled itself and produced no dump at all on Linux. - the access map needs 48 bits: arm64 Linux hands out addresses above 2^47, which a 47-bit map dropped on the floor. mem_profile is retired into this: it measured a strict subset and had its own sampling story. Local-only by decision — not a PR-CI gate. _memwaste.yml runs the lanes on all three platforms report-only, and pr.yml/release.yml carry the wiring. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 1 month ago | |
Secure Lens history reads and verify shipped browser dependencies Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 27 days ago | |
Secure Lens history reads and verify shipped browser dependencies Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 27 days ago | |
fix(test): isolate harness daemon runtimes Signed-off-by: astandrik <astandrik@yandex-team.ru> | 2 months ago | |
fix(test): isolate harness daemon runtimes Signed-off-by: astandrik <astandrik@yandex-team.ru> | 2 months ago | |
fix(test): isolate harness daemon runtimes Signed-off-by: astandrik <astandrik@yandex-team.ru> | 2 months ago | |
fix(ci): audit authenticated loopback readiness Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 2 months ago | |
fix(test): isolate harness daemon runtimes Signed-off-by: astandrik <astandrik@yandex-team.ru> | 2 months ago | |
fix(security): allow the bare wget token the shell pass compares The v0.12.0 release smoke blocked every binary in its string audit: "BLOCKED: Dangerous commands found in binary: wget". The line is the bare string "wget" that src/pipeline/pass_shell.c compares a script's program name with, and records as the `via` of a curl / wget HTTP_CALLS edge (3b93db06). The binary never runs it. Allow that whole line only, like the rst grammar's bare "telnet". A line that carries a wget command (flags, a target, a URL) stays blocked; the contract test now pins both cases. Proof on the release run's own artifacts (run 38644765106, main 8df0639c): linux-amd64-portable and darwin-arm64 binaries BLOCKED before, all three audits OK after. test_security_strings_allowlist.sh: 49 passed; with the allowlist line reverted, the new bare-token case fails. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 24 days ago | |
Audit all Lens source URLs with scoped fixture exceptions Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 27 days ago | |
fix(release): externalize runtime assets and harden VT verification Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 2 months ago | |
fix(setup): install through install.sh / install.ps1, one install path scripts/setup.sh and scripts/setup-windows.ps1 carried their own copy of the release download: resolve the latest tag, fetch the archive, unpack it into the install directory. Two implementations of the same step drifted apart: the setup scripts did not follow the installers as those gained their checks, their architecture selection and their staging directory. Both setup scripts now fetch the installer from the origin and branch they are themselves served from and hand the download and install over to it, so there is exactly one implementation and the setup scripts behave like the installers: the same conditions stop them with a non-zero exit before anything is installed, setup-windows.ps1 selects the archive through the installer's architecture detection instead of always amd64 (CBM_ARCH pins it) and stages in a freshly created directory, and the file is pure ASCII again. CBM_DOWNLOAD_URL, the installers' download-base override, also moves the installer fetch so both scripts can be driven against a local fixture. Everything the setup scripts do beyond installing the binary is unchanged: the --from-source / -FromSource builds, the interactive Claude Code configuration, the PATH hint, the git-hook activation and the uninstall notes. tests/test_setup_scripts_contract.sh runs setup.sh against a loopback fixture release built from a stub binary, in file mode and in curl | bash mode, covers the cases in which the installer refuses, and pins the static contract for setup-windows.ps1. It is Step 0z of scripts/test.sh. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 1 month ago | |
fix(setup): install through install.sh / install.ps1, one install path scripts/setup.sh and scripts/setup-windows.ps1 carried their own copy of the release download: resolve the latest tag, fetch the archive, unpack it into the install directory. Two implementations of the same step drifted apart: the setup scripts did not follow the installers as those gained their checks, their architecture selection and their staging directory. Both setup scripts now fetch the installer from the origin and branch they are themselves served from and hand the download and install over to it, so there is exactly one implementation and the setup scripts behave like the installers: the same conditions stop them with a non-zero exit before anything is installed, setup-windows.ps1 selects the archive through the installer's architecture detection instead of always amd64 (CBM_ARCH pins it) and stages in a freshly created directory, and the file is pure ASCII again. CBM_DOWNLOAD_URL, the installers' download-base override, also moves the installer fetch so both scripts can be driven against a local fixture. Everything the setup scripts do beyond installing the binary is unchanged: the --from-source / -FromSource builds, the interactive Claude Code configuration, the PATH hint, the git-hook activation and the uninstall notes. tests/test_setup_scripts_contract.sh runs setup.sh against a loopback fixture release built from a stub binary, in file mode and in curl | bash mode, covers the cases in which the installer refuses, and pins the static contract for setup-windows.ps1. It is Step 0z of scripts/test.sh. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 1 month ago | |
fix(test-infra): bind the smoke fixture server without a reverse-DNS lookup The macOS Intel leg failed the fixture contract on every run while every other platform passed. Removing the readiness fsync did not change it; the hardened diagnostics named the real cause on the first remote run: waited=30.0s, exit_status=alive, port_file=absent, staged_temp_files=none, empty startup log -- the process was healthy but had never reached publish_port. http.server.HTTPServer.server_bind() resolves socket.getfqdn(host). On a host whose resolver does not answer for the bind address that call blocks for the resolver timeout, so the constructor never returns and no port is ever published. The fixture now binds through a subclass that keeps the threading server but skips the FQDN resolution, which only feeds CGI-style variables this fixture never serves. Proven both directions locally by forcing socket.getfqdn to hang: the subclass publishes its port immediately, the stock server never does. That forcing hook is kept as a permanent contract guard, so the reverse-DNS dependency cannot return without turning the gate red on every platform rather than on one runner nobody can reproduce. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 3 months ago | |
fix(smoke): keep MCP tool inventory exact Signed-off-by: Tamer <tamer.elbakkali@gmail.com> | 1 month ago | |
test(cli): isolate OMP env overrides Signed-off-by: DavidHLP <lysf15520112973@163.com> | 2 months ago | |
Merge origin/main into carry/1862-install-hook-optout Maintainer refresh of #1862 (persistent install-hook opt-out, direction decided in #753) onto main a5b248f4. The first refresh is 2577a86a and carries the semantic fixes for shell-free Claude hooks. Textual conflicts, all resolved by keeping both sides: - src/cli/cli.h: main added CBM_CONFIG_DOC_CANDIDATES on the line where the PR added CBM_CONFIG_INSTALL_HOOKS. Both defines are kept, in alphabetical order. - src/cli/cli.c CONFIG_KEYS[]: main added the watch_non_git row where the PR added install_hooks. Main row stays first, followed by the install_hooks row. - src/cli/cli.c Codex hook install: main (#2719) declared pretool_written in front of the hooks.json branch the PR prefixed with its --no-hooks removal branch. The declaration stays first, then the PR removal branch, then main hooks.json and config.toml upsert branches unchanged. Check: for every file the PR touches, the added and removed lines of the merged tree against main equal those of the PR against its old base. No compile-level semantic conflict: no function the PR calls changed its signature, and the PR re-signs no function main calls. The new Codex PreToolUse hook from #2719 is gated by the opt-out in the next commit. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 6 days ago | |
fix(ci): give the wrapper smoke a Go toolchain and a deletable work tree The v0.12.1 release smoke failed on every leg in Phase 13w, the package-wrapper install gate added in #2664: - macOS: the runner images have no Go on PATH, and SMOKE_REQUIRE_WRAPPERS=1 rightly fails without one. pr-smoke and the three release smoke jobs now set up Go from pkg/go/go.mod, with the same pinned step the wrapper test job uses. - Windows: `mkdir -m 700` changes the mode in a second step, which MSYS2 cannot do under the smoke profile's owner-only DACL. mktemp -d sets the mode at creation. - Linux: the runner's older Go downloaded the 1.26.1 toolchain into the smoke profile's HOME. Go writes its module cache read-only, so the harness could not delete its profile afterwards. All Go state now stays in the wrapper's own work tree, and its cleanup restores write permission before removing it. npm and PyPI passed on every leg that reached them; the gate itself is unchanged. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 22 days ago | |
fix(soak): resolve the .exe whenever it exists, not only when -x fails Both hosted-runner Windows soak legs failed at Phase 1/diagnostics with an empty frontend stderr while the daemon's own log showed a healthy diagnostics.start: the workflows pass the suffix-less binary path, msys resolves it transparently (so the old 'append .exe only when the plain name is not executable' branch never fired), and soak-test keys BOTH native-Windows gates — the cygpath'd CBM_CACHE_DIR and the coproc stdio branch — off the literal .exe suffix. A suffix-less native binary therefore received a POSIX-form cache path: daemon logs mis-rooted (diagnostics grep zero, arm64) and the frontend's cache path handling died silently (x64 SIGPIPE at first write). Reproduced on the VM's CLANG64 environment with a suffix-less invocation (same FAIL) and red->green proven: with the normalization the identical invocation runs the full quick soak to PASSED. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 3 months ago | |
fix(soak): the per-call read follows the soak's own latency ceiling The soak's query-leak leg failed on GitHub's windows-11-arm runner in the v0.12.0 release run (38460145605) and in the 10-12 dry run (38204190338), both times on one search_code call. On that runner image the PowerShell-backed search_code answers in ~23 s per call (mean 23.3 s over 23 calls). One call crossed the harness's fixed 30 s read, counted as a failed operation, and failed the leg, even though the harness's own latency check (Check 4) allows 60 s. It is the runner environment, not the product: the same leg on a local Windows ARM64 VM gives search_code 1.10 s mean for v0.11.0 and 1.06 s for the v0.12.0 candidate artifact, both PASSED. Linux runs it in ~50 ms, macOS in ~150-230 ms, Windows x64 in ~0.9 s. One constant, SOAK_MAX_QUERY_LATENCY_S (60), now drives the latency check. The per-call read waits 30 s past it, so the measured latency, not a timeout, gives the verdict; a reply that never comes still ends the read and fails the call. Checked: bash -n; tests/test_soak_daemon_recovery_contract.sh and tests/test_smoke_fixture_contract.sh pass; the patched harness on the Windows ARM64 VM (query-leak, 2 min, candidate binary) PASSED. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 25 days ago | |
fix(ci): preserve executable modes for failure report scripts Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 26 days ago | |
feat(index): add worker resource watchdogs Discovery limits bound what indexing accepts, not what it then costs. A repository well inside those bounds can still exhaust the host through parser memory, or simply never finish, and a supervised worker that hangs leaves the parent waiting with nothing to report. Add index_max_rss_mb and index_max_duration_seconds, enforced by the parent against the worker process tree rather than the worker process alone, so a runaway child cannot hide behind a small parent. Resident memory is sampled through the platform interface on macOS, Linux and Windows. Crossing a limit terminates the tree and yields one trusted, structured terminal result that attributes the failure to the resource that caused it. Both limits default to off. A measurement that cannot be taken fails the attempt instead of passing it: a watchdog that quietly stops watching is worse than no watchdog at all. The shell fixture that stands in for the supervisor names the two new keys. The worker accepts only a policy that spells out every key it knows, which is what keeps a stale supervisor from starting a worker it cannot bound. Signed-off-by: 刘冲 <mail@liuchong.dev> | 20 days ago | |
revert(release): re-embed runtime assets into the single shipped binary Externalizing the integration templates (#1492/#1493) and the UI bundle (#1501/#1503) was done to reduce the Microsoft `Wacatac.B!ml` surface. It did not work: across dry runs the flagged artifact count stayed at ~3 and the detections merely moved between artifacts. Dissection of run 31286803592 shows there is no structural cause to fix. The verdicts split across every axis at once — linux-amd64 (dynamic) flagged while linux-amd64-portable (static) is clean, but linux-arm64 (dynamic) clean while linux-arm64-portable (static) is flagged. The two macOS binaries have identical segment structure and split clean/flagged. Siblings from one build landed in different variant buckets (.B vs .C). Entropy is low everywhere (code_vectors.bin 4.166, grammar tables 3.464 bits/byte, against 7.5-8.0 for packed payloads), so the packed-payload hypothesis is excluded too. So the complexity bought nothing, and installation goes back to being self-contained: one binary that carries its own UI and agent integration templates, with no adjacent data file that has to resolve before `install` works. Only the UI-capable composition ships from now on, under the historical unsuffixed archive name. Removed: src/ui/asset_pack.{c,h}, asset_pack_stub.c, asset_manifest_stub.c, scripts/pack-ui-assets.mjs, src/cli/integration_assets.{c,h}, assets/cbm-integrations.json, scripts/gen-integrations-hash.sh, the --verify-runtime-assets probe (nothing adjacent left to verify), and the composition gates A6/A7 whose property is now deliberately inverted. Restored: scripts/embed-frontend.sh, src/ui/embedded_{assets.h,stub.c}, the compiled-in hook/adapter template bodies, and the embed/EMBED_OBJS build path. Kept from the reverted commits, re-applied by hand where a wholesale file restore would have dropped them: - cbm_module_path_utf8() in both self-path sites. GetModuleFileNameA renders through the ANSI code page and mangles non-ASCII install paths. - the /__cbm/ui-readiness HMAC proof, secure_random and cbm_hmac_sha256, so `daemon start --open` still waits for a genuine CBM listener. - X-Content-Type-Options: nosniff on served assets. - the MinGW noexecstack gate, -lbcrypt, and the cppcheck/zip CI fixes. Archives are now codebase-memory-mcp-<os>-<arch>[-portable] with exactly four members (binary, LICENSE, installer, THIRD_PARTY_NOTICES.md). That restores the names every static package manifest already points at — aur, chocolatey, homebrew, scoop, winget and glama were all broken by the -ui- rename. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 2 months ago | |
Merge origin/main into fix/lint-mem-gate-warning-suffix Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 5 days ago | |
fix(launcher): name the object and check behind every security refusal The launcher answered every ownership/ACL problem with one generic "launcher ownership or access policy is unsafe", which made the CI guard failures (and any field report) undiagnosable: the same message covers an untrusted owner SID, a mutation-capable ACE, a reparse point, a hard-link clone, and a plain open failure across the whole ancestry walk. Record the failing check at its site - offending path, owner or ACE SID in SDDL string form, rights mask, Win32 error - and print it once alongside the refusal. No policy changes; refusals stay refusals. The interactive MCP smoke now also dumps daemon-conflicts.ndjson next to the daemon log tail on failure, so an admission conflict names itself instead of surfacing as a bare 30 s client timeout. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 3 months ago | |
fix(mcp): address QA round 1 findings - Use O_NONBLOCK + clearerr() in Phase 2 fgetc probe to preserve the 60s idle eviction timeout when both kernel fd and FILE* buffer are empty (fgetc on a blocking fd would otherwise block indefinitely, bypassing Phase 3 poll timeout and preventing cbm_mcp_server_evict_idle) - Add #include <fcntl.h> for fcntl()/O_NONBLOCK - Fix comment: "two-phase" → "three-phase" (implementation has 3 phases) - Improve Python integration test: verify id:1 (initialize) and id:2 (tools/list) response IDs are both present, not just "tools" substring Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> | 7 months ago | |
chore(ui): drop sharp and its LGPL libvips; the license gate allows no exception The v0.12.0 release run's license gate blocked @img/sharp-wasm32@0.35.5 (Apache-2.0 AND LGPL-3.0-or-later AND MIT). sharp sat in graph-ui's tree only because @huggingface/transformers lists it for its Node image path. CBM Lens runs Transformers in the browser (a tokenizer and a causal LM on WebGPU), never processes images, and the Transformers web build already replaces sharp with an empty object. Yet sharp installed 14 LGPL packages (10 @img/sharp-libvips-*, 3 @img/sharp-win32-*, @img/sharp-wasm32), and the gate let them through two ways: - a reviewed exception for exactly @img/sharp-wasm32@0.35.0 ("installed but not shipped"), which the 0.35.5 bump in #2632 tripped; - collect() skipped every os/cpu-restricted package as "build tooling", which hid all ten LGPL libvips binaries. A non-permissive license has no place in the tree, shipped or not: - sharp now resolves to graph-ui/stubs/sharp, our own MIT stand-in that throws if anything ever calls it (direct dependency file:./stubs/sharp plus override "sharp": "$sharp"). The whole sharp family leaves the lockfile: 441 -> 411 packages, zero LGPL/GPL left (audited). - ui_browser_licenses.py loses the exception and keeps only the browser build evidence check; license-gate-check-npm.py validates that evidence unconditionally, checks platform-restricted packages like any other, and has no exclusion path; gen-ui-licenses.py drops the exception section. - tests: the exception tests become guards: an unshipped LGPL package blocks, a platform-restricted LGPL package blocks, a platform-restricted MIT package passes. Against the old scripts both LGPL guards fail. The shipped browser bundle is unchanged: rebuilt, it differs from main's only by the build's own -dirty version marker. License gate stage 3 on this tree: 211 production packages, all allow-listed. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 25 days ago | |
fix(scripts): use /usr/bin/env bash shebangs (NixOS has no /bin/bash) On NixOS (and other non-FHS systems) /bin/bash does not exist, so scripts with an absolute shebang fail to run. Switch the remaining holdouts to /usr/bin/env bash: eleven scripts/*.sh, test-infrastructure/run.sh, and the three Claude Code hook scripts emitted by src/cli/cli.c (gate, session reminder, subagent reminder). Distilled from PR #674, with parser-test coverage preserved: the infra_parse_shell* fixtures in tests/test_pipeline.c intentionally keep #!/bin/bash so absolute-path shebang extraction stays covered, and tests/repro fixtures are untouched. Also replace the GitHub-PAT-shaped fixture string flagged in the #674 thread with an obviously fake placeholder (ghp_FAKE...) that still matches the ghp_ + 36-alnum secret detector. Co-authored-by: Sandro Jäckel <sandro.jaeckel@gmail.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 4 months ago | |
Merge branch 'main' into fix/mimalloc-3.5.3-preinit-free | 11 days ago |
scripts/ — the canonical leg entries
The doctrine (enforced, not advisory): a venue may provision a machine;
only a canonical leg script may exercise the product. Local CI, PR CI, the
dry run and the release all call the SAME files below — venues differ only in
host specs, architecture and inputs, never in what a leg does. Platform
differences (arm64 sanitizer flags, native Windows behavior, the Linux portable
binary) live INSIDE these scripts, once, never per-venue.
tests/test_venue_parity_contract.sh (run as Step 0j of every test leg) turns
any violation — inline harness logic in a workflow, a bypassed wrapper, a
missing --help — into a red build.
Every entry answers --help (authoritative, agent-oriented: modes, env, exit
codes) and rejects unknown flags with exit 2 + Please consult --help.
The legs
| leg | entry | what a run gives you |
|---|---|---|
| test | test.sh |
DEFAULT = the venue leg: static contracts (Step 0a–0r) + CLEAN ASan+UBSan build + all suites via the parallel harness + prod-binary guards. --suites a,b = iteration mode (incremental rebuild, subset, seconds). --tsan = the ThreadSanitizer leg. --contracts-only = the static contract steps alone (PR CI's contracts job for docs-only and non-product changes). CLANGARM64 gets CI's trap-UBSan flags automatically. |
| package wrappers | ci/test-package-wrappers.sh |
Go, npm, and PyPI runtime-set publication/lock suites on the current host. CI runs the same entry on Linux and Windows so both platform lock implementations gate. |
| build | build.sh |
CLEAN production runtime set (native executable + authenticated integration asset; --with-ui adds one content-addressed UI pack). ccache via env.sh makes repeats fast; CCACHE_COMPILERCHECK=content guarantees a hit is byte-identical to a cold compile — never stale. --version, STATIC=1, BUILD_DIR=. |
| lint | lint.sh |
clang-tidy + cppcheck + clang-format (+ no-skips policy). --ci = the CI gate set (no clang-tidy). Drives the same make targets as make lint/lint-ci. |
| smoke (unix) | smoke-local.sh |
Stages a full release fixture, serves it on a kernel-assigned port, runs smoke-test.sh (ALL phases incl. download/install/update E2E) inside a disposable HOME/XDG/TMP sandbox. ui variant makes a missing verified UI pack a FAILURE. CBM_SMOKE_ARTIFACT_DIR = smoke an extracted release artifact verbatim (release mode). |
| smoke (windows) | ../test-infrastructure/vm/vm-smoke.sh |
Same verified runtime-set contract on the real Windows VM, plus the user-PATH registry guard (prepare/verify/cleanup). |
| smoke-invariants | smoke-invariants.sh |
Production-path resilience battery (MCP handshake, all tools invocable, malformed-input handling, supervised crash/hang recovery) — no fixture server or install E2E. smoke.yml runs an explicitly seam-enabled build on the WIDEST source matrix; release artifacts remain seam-free and use the release-shaped smoke legs above. |
| soak | soak-legs.sh |
The release-gating soak SEQUENCE: quick then query-leak (the #581 detector — never reindexes, so RSS growth = query-path leak), each guarded by a completion-summary check. --legs quick for the ASan single-leg variant. Duration is per leg. |
Internal harnesses — never called directly by a venue (the contract forbids
it): smoke-test.sh (phases; wrappers provide fixture server + sandbox),
soak-test.sh (one soak run; soak-legs.sh provides the sequence + guards),
run-tests-parallel.sh (reached through test.sh), which sources
suite-failure-report.sh for its end-of-run failure summary (failure sites,
sanitizer report, running test, last lines) and for test-logs/failed/, the
logs of every suite that did not end green.
Conventions
- Exit codes: 0 = pass · 2 = usage error · 90 = guard (a run died without its completion summary — never counts as green) · anything else = the leg's real failure.
- Iteration is a flag, not a side-tool: the fast paths (
--suites,--legs) are modes of the SAME entry the gates run, so a dev loop can never drift from the venue behaviour. - Env sandboxing: the smoke wrappers neutralize every agent-config destination override; a smoke can never scribble on your real config.
Recommended workflows
- Iterating on a change:
scripts/test.sh --suites <suite>(seconds, incremental, same ASan+UBSan flags as the gate). List suites:build/c/test-runner --list-suites. Debugging a Windows-on-ARM trap: re-run withSANITIZE=for a plain build, or use the emulatedwin.sh ubsan-*pair for full diagnostics. - Before any push (the 3-OS ladder):
scripts/test.sh(macOS, full) →./test-infrastructure/run.sh full(Linux + TSan + smoke) →test-infrastructure/vm/win.sh test-par+guards+smoke-install(+soakwhen the change touches memory/daemon paths). - Concurrency-touching change: add
scripts/test.sh --tsanearly — the same leg CI gates on. - Release-shaped verification:
CBM_SMOKE_ARTIFACT_DIR=<extracted artifact> scripts/smoke-local.sh <binary> [ui]smokes exactly what would ship. - A leg is red in CI but green locally: first suspect environment shape,
not code — the preflights (
win.shautomatic;scripts/ci/preflight-docker.sh) andtest-infrastructure/README.md's residuals list cover the knowable differences.
See scripts/ci/README.md for the CI plumbing and
test-infrastructure/README.md for the venue map.