| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
Merge pull request #2523 from DeusData/fix/issue-514-constructor-fields-v2 fix(typescript): resolve constructor properties and new-initialized fields | 1 天前 | |
fix(csharp): refresh grammar for collection expressions in conditionals (#1748) The vendored tree-sitter-c-sharp (88366631d598) predated upstream's C# 12 collection-expression support (tree-sitter-c-sharp #402, first released in v0.23.4). A `[...]` literal could only be parsed as an element_binding_expression, so `c ? [] : x`, wrapped `? []` ternaries and both-branch collection conditionals had no valid parse. Error recovery then ate the surrounding statement, the file was reported as parse_partial, and nearby declarations could fall out of the graph. Refresh the generated grammar bundle byte-for-byte from upstream tag v0.23.5 (cac6d5fb595f5811a076336682d5d595ac1c9e85). The ABI stays at 15, the MIT LICENSE is byte-identical, and scanner.c is content-identical. The only new node types are collection_expression, collection_element, expression_element and spread_element. No node type is removed or renamed, and the C# extractor and LSP depend on none of the old or new names. MANIFEST.md and the vendored checksums are updated. Add parse_coverage regressions for the reported variants plus a `? null` control. A second test checks that calls inside a collection expression (element and spread) are still extracted. A malformed-source guard checks that broken C# is still flagged. On dotnet/runtime (60108ba6), 4 files are no longer parse_partial and no file becomes newly partial. Utf8JsonWriterTests.cs gets its class back, and 216 orphaned Functions are re-attached as Methods. Two files that were already partial for unrelated unsupported syntax (the preview `closed` modifier, `async` as an identifier) recover differently and lose 46 and 10 definitions. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 4 天前 | |
WIP: strict linting + RAM-first pipeline (lint fixes pending) | 6 个月前 | |
Add C-native pipeline with blocking clang-tidy, cppcheck, clang-format linting Complete C implementation of the indexing pipeline (src/), parallel worker pool, graph buffer with merge support, SQLite writer, and 1893-test suite. Linter setup: zero warnings from clang-tidy (all checks enabled, no NOLINT suppressions), cppcheck, and clang-format. All issues fixed at source — proper headers for external linkage, GROW_ARRAY macro restructured to eliminate type parameter, null-deref paths guarded, named intermediates for suspicious-argument checks. | 6 个月前 | |
fix(arena): NULL-arena guard in cbm_arena_alloc; skip vendored/ dirs - cbm_arena_alloc returns NULL on a NULL arena (both arena.c copies) instead of dereferencing it — defense-in-depth against the NULL-arena type-allocation path that could crash the LSP type layer. - discover: add "vendored" to the always-skip directory list. Distilled from #374, taking the parts not already on main. The PR's ts_lsp tuple-arena allocation, the C/C++ template formal-count clamp, and the type_rep unbound-param preservation it also carried all landed independently in v0.7.0 / via #322 / #360, so only these two defensive improvements remain. Relates to #390. | 4 个月前 | |
feat(mem): memory on the core, waste cut, results spilled under budget Second increment on the memory core. The first landed the core, the linter and phase attribution with zero production call sites; this one migrates the memory that matters, proves where it goes, removes the waste the proof exposed, and makes the memory budget a promise: over budget, extraction results go to disk and the run completes at the floor instead of aborting. Every number below is from a real index on this machine (M5 Pro, 48 GB) with CBM_MEM_PHASES=1 and the release build. WHAT THE PROOF SAID (Go corpus: 21,875 files, 345 MB of source) The graph buffer, blamed for the peak since the 2026-07 research, is 3% of the memory: 0.55 GB peak. The per-file extraction results are the memory: 14.8 GB written into their arenas, of which 3.4 GB is reachable. The rest is temporaries -- cbm_node_text copies at 496 call sites, per-node QN strings, abandoned generations of every growable array -- that an arena cannot free and that the result kept alive until resolve ended, because the result owned the arena. The arenas were charged 28.8 GB of capacity for those 14.8 GB. On the kernel the same shape was 8.9 GB used by 89,731 results, and resolve grew it to 13.3 GB used / 17.2 GB capacity because the C walk allocated its working set in the result arena and stored arena pointers into the shared registry. After that, 7 GB of the kernel's post-resolve memory had no class at all. WHAT CHANGED Attribution. The graph buffer (87 raw sites -> 0), arena blocks, the tree-sitter slab allocator, the bound tree-sitter/SQLite allocators, the semantic path (pass_semantic_edges, pass_similarity, minhash: 80 sites), the hash table (Verstable, a class per table via CTX_TY: cbm_ht_create_in) and CBM_DYN_ARRAY allocate through the core. Classes arena, ts_tree, hash_table and dyn_array join the table. Two cross-boundary frees of buffer-owned strings (pass_importance, pass_complexity) go through a new setter, cbm_gbuf_node_set_properties_json. The linter baseline drops 4086 -> 3916. Result compaction (internal/cbm/result_compact.c). At the end of a file's extraction, everything reachable from its result is measured, copied into one exact-size arena (strings interned by content within the file) and the working arena is handed back. Later appends restart growth at the default block (cbm_arena_init_exact, arena grow_size). The stale duplicate internal/cbm/arena.h, which shared the foundation header's include guard and won or lost by include order, is a shim now. Record slimming. CBMCall 320 -> 72 bytes: the eight inline argument slots (256 bytes, mostly empty) are allocated on first capture. CBMUsage 48 -> 40. The per-file overlay contract (src/pipeline/pass_lsp_cross.c). One lifecycle for every language's cross-file resolve: a scratch arena, a per-file overlay registry chained to the sealed shared base, the walk against the overlay, resolved calls appended into the result, scratch destroyed. type_registry gains chain-aware iterators (every yielded index belongs to it->reg) and copy-on-write refinement (cbm_registry_func/type_for_update), and the C, C#, Go and Python walks use them. This is what an ASan heap-use-after-free in the lsp_resolution_probe suite demanded: the C walk stored per-call arena objects into the shared registry, and a kernel run with a naive scratch arena lost 1,045 edges to the resulting garbage reads. It also ends unsynchronized writes into the shared registry from parallel resolve workers. The 7 GB with no class, found and cleaned. gbuf_index tables and keys were 2.5 GB live and 7.1 GB at peak: the 18 worker buffers built the five secondary indexes they never query (cbm_gbuf_new_worker builds none), and 18M per-key index arrays started at 8 slots (cbm_da_push_min starts them at 2); edge property JSON is 81% "{}" (interned). The semantic pass allocated a 512-token stride per function up front (packed tokens with offsets: cbm_sem_corpus_add_docs_batch). Edge dedup keys are 128-bit hashes instead of sprintf'd strings (edge_key_map_t). Each worker keeps one reusable working arena between files (cbm_work_arena_take/give: rewind, not free) and every phase mark returns freed pages to the OS. Budget metric. The over-budget check compares the charged footprint -- max(phys_footprint, mimalloc commit) on macOS -- not resident_size: after extraction the kernel worker sat at 17.4 GB RSS with 5.5 GB charged, because pages mimalloc has purged stay resident until the OS wants them, and phys_footprint under-reports after MADV_FREE_REUSABLE cycles. Spill / admission control (internal/cbm/result_spill.c, the extract gate in pass_parallel.c). At 15/16 of the budget (or over it, or with CBM_MEM_SPILL=1) the gate latches spill mode -- early, because the gate sees the crossing per file pull and the workers' in-flight files carry the charge past the line before the first sweep lands: every compacted result is parked on disk as header + block in one append-only file per writer under <cache>/spill/, the results already cached are swept out by every worker, and registry build, def collection, surface rows, resolve and the infra-route passes load a result only for the moment they read it (relocated by base delta). The abort path stays closed while a sweep runs or a cached result remains; it fires only at the floor -- graph + registries + in-flight files. A result that owns sub-results stays in memory; a retained parse tree is dropped with the parked result and re-parsed on load. Every consumer of the result cache reads a slot through cbm_pipeline_result_acquire()/release() (pipeline_internal.h): a parked slot is NULL in the array, and the passes that indexed it directly lost every __route__infra__ node and its INFRA_MAPS edges before the contract. The other things the transient loads found: a def label that was the result's own pointer (SIGBUS in the Go cross registry builder), two JVM helpers that passed their input through, a dedup table whose keys borrowed sr->value (the insert spun on freed memory), and a latch that was visible before the store was open (a peer found nothing to park and failed the run while its neighbour wrote 9 GB). The arena rewind reuse branch had defeated the nblocks = CBM_ARENA_MAX_BLOCKS OOM seam the suites use; it reuses only blocks that exist. The incremental and probe routes still index the cache array themselves and therefore never spill (ctx->spill_allowed stays false there; follow-up). The semantic pass under a budget (src/pipeline/pass_semantic_edges.c). The pass held every function's tokens, the corpus's per-document token ids and the vectors being stored at once: 4.9 GB on the kernel, on top of the floor, and nothing sized it to the headroom. Now, when the charged footprint plus that transient would cross the budget, phases 2-4 run per batch of functions: tokenize -> count into the corpus -> free; finalize once; tokenize again -> vectorize -> store -> free, with pages returned to the OS after every batch. Tokenization is deterministic, so both passes see the same tokens and the graph does not change; the run pays with a second tokenize. CBM_SEM_BATCH=<n> forces the batch size (the batched-equals-unbatched test). The corpus frees its per-document token ids at the end of finalize (the co-occurrence pass is their only reader), and mem.semantic.step lines (CBM_MEM_PHASES=1) put the class's live/peak bytes and the charged footprint at every sub-phase. The semantic corpus and tokenizer allocate through the core (54 raw sites -> 0). This is not attribution only: cbm_sem_tokenize handed out libc strdup blocks that pass_semantic_edges freed with cbm_free -- mi_free in the production build -- since the pass moved onto the core. No libc-backed test build can see a cross-allocator free, so the core now refuses a block it does not own (mi_is_in_heap_region) when CBM_MEM_PHASES=1 is set, fatally: every proof run in this PR ran with it armed, and the Go and kernel indexes report zero foreign blocks. Accounting without contention. The core charged every allocation and free to shared per-class atomics; with 18 workers on 18 cores every block bounced the same three cache lines, and the bench against the shipped v0.10.8 showed it as CPU time rising faster than wall time for the same graph: Kotlin 38 -> 121 s CPU (+58% wall), TypeScript 210 -> 596 s (+67%), Go 177 -> 312 s. The deltas are thread-local now and reach the shared counters once per 256 KB or 512 blocks of change per thread, when a parallel-for worker ends, and whenever a reader looks (its own thread first). Phase marks read after the join and are exact; a class peak is a diagnostic and can be low by threads x 256 KB. Result: Kotlin 10.9 s / 43 s CPU (the residual +22% wall against v0.10.8 tracks its +34% nodes), TypeScript 33.9 s (v0.10.8: 33.4), Go 48.4 s (v0.10.8: 52.6), Java 81.1 s (v0.10.8: 79.8); graphs unchanged. SQLite on a heap of its own. The same bench put the kernel's write step at 132 s where v0.10.8 needed 9.7 s, all of it in one coverage publish statement. A stack sample named the cost: mimalloc's free-page search and its periodic heap collect, walking the graph's pages -- millions of them once the graph lives on the core -- for every statement-journal chunk SQLite allocates. SQLite now allocates from a mimalloc heap of its own per thread (mi_heap_new; mi_free is heap-agnostic, so nothing else changes), and the periodic collect is set to the maximum mimalloc allows since the explicit release at phase marks and spill sweeps is where memory goes back. Coverage publish 132 -> 9.4 s; profiled kernel 376 -> 248 s (v0.10.8: 296 s). Formatting without a process lock. On macOS every vsnprintf takes the process locale under an unfair lock (localeconv_l inside __vfprintf); eighteen workers formatting type names on the C# corpus collapsed into it -- the 10 MB JIT test files took 68 s each instead of under a second -- and cbm_arena_sprintf paid it twice per string (size pass, write pass). Each thread now formats with a C locale object of its own (vsnprintf_l), and the common short string is formatted once into a stack buffer. Per-file budgets that cover the whole file. The 5 s per-file budget covered the parse only; the LSP walks after it had none. C# JIT stress files (a 23 MB single expression among them) parsed inside the budget -- the old wall-clock parse timeout used to drop them, which hid the rest -- and then held a worker for 346 s each in the usage walk (tree-sitter's ts_node_parent descends from the root: quadratic on a deep tree), and the cross-file resolve ran on the same trees. Three rules now, one site for every language (cbm_extract_file_ex, honoured by cbm_pxc_dispatch_file): a parse that used more than half the budget disqualifies the file from the LSP walks (lsp_skipped); the unified walk checks its thread CPU time every 1024 nodes against six budgets and stops there, keeping what it found (walk_truncated, implies lsp_skipped); a file whose parse plus walk spent the budget is skipped by the LSP walks as well. Each decision is logged with the path (extract.lsp.skipped, extract.walk.truncated). C# extraction 355 -> 39 s; the whole index 463 -> 128 s wall and 27 -> 11 GB peak against v0.10.8 with 299 more nodes. The walk cuts exactly the five JIT stress files (hugeexpr1, hugeSimpleExpr1, HugeArray1, HugeField1/2); the 15 MB System.Runtime.Intrinsics reference file walks to the end and, with four generic-nesting JIT regression tests, only skips the LSP walk under the parse-plus-walk rule. The crash behind the C# bench. dotnet/runtime is 42,555 files, and its resolve phase died with SIGBUS on every run of this branch. A libc-backed ASan build of the server named it: heap-use-after-free in c_adl_resolve, reading a type name that another worker had allocated in its per-file scratch arena and freed at the end of its file. The C++ class walk refines a method's return type when the declaration is more specific than the pre-registered one (NAMED -> pointer/reference/template), and it did that by casting the chained lookup result to non-const and writing a scratch-arena signature into it -- into the sealed shared base when the entry lived there. The overlay contract had exactly one bypass, and it was this cast; every lookup returns const, so a grep for the cast is the audit. The refinement now goes through cbm_registry_func_for_update: copy-on-write into the overlay, the base untouched. Test: clsp_method_return_refinement_is_copy_on_write (a sealed base with a NAMED method return, an in-class pointer declaration; the base signature pointer is unchanged and the overlay copy carries the refinement plus a marker only a copy has) -- RED on the cast, GREEN on the accessor. The fixed ASan build then indexed the whole corpus clean: 42,555 files, 1,224,981 nodes, 5,794,320 edges, worker exit 0, no report. Production build backs the core with mimalloc explicitly (never the global override, which macOS's two-level namespace forbids); the test build keeps libc so the sanitizers see every block -- the same split the tree-sitter and SQLite bindings already use. PROOF Go peak RSS 16.7 -> 4.0-5.2 GB; result arenas 14,804/28,817 -> 856/856 MB; wall 55 -> 52-60 s; graph inside the pre-existing run-to-run jitter (gRPC Route nodes: two identical runs differ by 21). Go CBM_MEM_SPILL=1: peak RSS 1.7-2.3 GB, 21,875 results parked (950 MB on disk), graph identical: 39 infra routes, 158 INFRA_MAPS, 21,875 surface rows, nodes/edges inside the Route jitter. Go CBM_MEM_BUDGET_MB=1200: spill latched at 1,204 MB, the first sweep took the charge to 768 MB, the run completed at 1.99 GB peak RSS. Kernel DEFAULT budget (24,576 MB): aborted before, completes now without spilling. Extraction 35.2 -> 16.5 GB RSS (5.6 GB footprint, 15.3 GB committed); results 8.9 GB = capacity; resolve tracked 24.6 -> 19.1 GB; peak RSS 27.3 -> 20.1 GB. Nodes identical (8,529,729); edges inside the +-400 run-to-run band the kernel shows between identical runs. Kernel CBM_MEM_BUDGET_MB=16000: aborted before, completes now through the spill path. Latched at 17.2 GB charged; 89,731 results parked (9.3 GB on disk, 370,188 loads); extraction end 6.2 GB RSS / 5.1 GB committed; resolve 8.6 GB RSS / 11.5 GB committed; peak RSS 16.6 GB (the semantic pass plateau, 13.3 GB RSS, is graph-derived and outside what spill can move). Nodes identical; edges 15,773,598. Kernel CBM_MEM_BUDGET_MB=15000, before the semantic batching: completes, every phase mark under 15,000 MB by every metric, but the process RSS high-water mark reached 16.9 GB inside the semantic pass (class peak 4.9 GB against 0.8 GB live). Kernel CBM_MEM_BUDGET_MB=15000, with it: completes; the charged footprint stays under 15,000 MB through the whole semantic pass (14,018 -> 14,381 MB across its sub-phases, 6 batches of 125,644 functions); every phase mark is under budget by every metric (max footprint 12.6 GB, max commit 11.5 GB); the process RSS high-water mark is 15.65 GB and is set in extraction, at the moment the gate latches (15,002 MB charged) and before the first sweep lands -- a 4% overshoot of in-flight work, no longer the semantic pass. Nodes identical, edges 15,772,913, zero foreign blocks. Kernel CBM_MEM_BUDGET_MB=15000, with the anticipatory latch: spill mode enters at 14,062 MB (near_budget); the charged footprint never exceeds 11.8 GB at any mark; the process RSS high-water mark is 15.1 GB in extraction and 15.3 GB during the final write-out -- 1-2% over, the in-flight window and the dump transient. Nodes identical, edges 15,772,543, zero foreign blocks. Go CBM_MEM_BUDGET_MB=1200 with it: latched at 1,131 MB, extraction peak 998 MB RSS, graph identical to the baseline. BENCH AGAINST THE SHIPPED v0.10.8 (same machine, same driver, 2026-09-14) corpus wall s peak RSS GB nodes / edges / CALLS perl 5.7 -> 5.3 0.09 -> 0.05 -3 / -130 / -3 php 6.3 -> 5.8 0.66 -> 0.20 +21.9% / +1.4% / +23.8% rust 6.8 -> 6.3 1.15 -> 0.67 +1 / +56 / -369 c 5.9 -> 5.7 0.16 -> 0.19 identical kotlin 8.9 -> 9.4 1.55 -> 0.96 +33.5% / -22.8% / +12.1% django 10.9 -> 10.2 3.13 -> 1.12 +6 / -14.5% / -28.4% go 52.6 -> 42.9 17.2 -> 3.9 +14.2% / -10.3% / +2.2% java 79.8 -> 73.8 27.6 -> 11.0 -121 / -1.9% / -0.6% csharp 463.3 -> 128.1 27.0 -> 11.2 +299 / -0.9% / -0.6% typescript 33.4 -> 33.6 12.0 -> 2.5 -186 / -3.5% / +130 kernel 294.8 -> 233.3 27.5 -> 25.1 -171 / -1.3% / -0.1% No corpus is slower beyond noise; CPU time follows wall (kernel 2,442 -> 1,403 s). Every graph delta against v0.10.8 is main between the release and the branch base (339b3f40), not this PR: a build of the base commit run through the same driver reproduces the final graphs exactly on django and kotlin, within the pre-existing gRPC Route jitter on go (30 nodes, 27 edges, CALLS identical), and on C# with the same node count. This PR's own C# difference against the base is 5 CALLS (the five truncated JIT stress files) plus one ijwhost swap and 563 edges of 5.79 M; the base takes 448 s / 29.7 GB for that corpus. Diagnostics that made this measurable stay in: the extract.arenas and extract.census lines (behind CBM_MEM_PHASES=1), the CBM_MEM_RELEASE=1 release-to-OS probe, and peak_charged_mb on every mem.phase line -- the high-water mark of the budget metric itself (cbm_mem_peak_charged), next to the RSS peak that counts pages already purged but not yet reclaimed. Tests: extract_compact_* and extract_spill_round_trip_keeps_every_field (extraction), parallel_spill_mode_builds_the_same_graph (parallel: every file parked, node count and every edge type equal to the in-memory run), pipeline_semantic_batched_matches_unbatched (pipeline: one repo indexed unbatched and with CBM_SEM_BATCH=5 -- byte-identical node vectors, token vectors and SEMANTICALLY_RELATED edges), extract_lsp_skipped_when_parse_used_ its_budget_share and extract_walk_truncated_at_its_cpu_budget (extraction: the seams mark a file / stop the walk, the LSP walk and the dispatcher skip it, the defs stay), registry_overlay_chain_iterates_ and_copies_on_write (c_lsp), arena exact-init/growth, mem charged/footprint/ pressure. Not in this increment: run-to-run edge jitter (+-400 on the kernel, Route class on Go) predates this work and has its own item; the incremental route spilling; a tighter result encoding (def ids instead of QN strings in resolved calls) is the next lever on the 9 GB the kernel's results still take. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 22 天前 | |
feat(identity): callable identity plumbing, no language enabled (#2061) Overloads that share a base QN (Java/C#/C++/Kotlin/Swift/Scala/ObjC) merge into one graph node today, and that node collects the CALLS of every overload. The agreed cure is one signature-qualified callable identity for all overloading languages: callable_qn := base_qn [<tparams>] (params) [cvref] This change lands the plumbing with every language still at CBM_CALLABLE_ID_NONE, so the graph is byte-identical; each language is switched on later in its own change (with its index-format bump). - internal/cbm/callable_sig.{c,h}: the per-language mode side table (NONE|TYPED|LABELED_TYPED|LABELED|ARITY; kept out of CBMLangSpec like cbm_string_dispatch_suffixes, since ~160 positional rows would otherwise all need the member under -Wmissing-field-initializers), the shared builder cbm_callable_sig[_mode] for the def and the call-scope QN, and the inverse cbm_qn_callable_base_len[_named]. Suffix contract: no '.', '::' or '->'; whitespace-normalized; comments/annotations/attributes dropped; qualified types keep their last segment; '->' spelled '=>'; variadic '[]' (Java) / '~'; C++ top-level cv dropped, arrays decay, cvref kept; <= 200 bytes, longer ones keep whole leading entries and end in '#<fnv1a64 of the full suffix>)'. The builder is iterative (cursor walk, bounded declarator loop) and self-checks against the inverse. - The inverse is anchored to the node's bare name wherever the name is known: real QNs already end in ')' without being callables (a Java field whose name is its declarator `X = f()`, make `$(VAR)` targets), and only the anchor makes the inverse exact on them. - Leaf splitters strip a suffix first (no-op today): registry by-name key (indexed by the bare name for a suffixed callable), cbm_lsp_bare_segment, cbm_pipeline_qn_class_method_tail, pxc_last_component, pxc_qn_leaf, go_index_visit_node, collect_prod_defs, sg_qn_prefix_len. - New unused fields: CBMDefinition.qn_sig_off, CBMResolvedCall.callee_sig (relocated by result compaction), CBMRegisteredFunc.sig, CBMLSPDef.sig. The lsp_surface codec is unchanged (no format bump in this change). - get_code_snippet and trace_path gain base-QN tiers (cbm_store_find_nodes_by_qn_base) that find nothing until suffixes exist. - store: cbm_store_find_nodes_by_qn_suffix built its LIKE pattern in a 512-byte buffer (a long suffix was truncated into a pattern that matched nothing) and did not escape '_'/'%' (`my_func` also matched `myXfunc`). The pattern is now heap-built, escaped, and matched with ESCAPE '\'. Proof: full sorted node+edge+lsp_surface dumps are byte-identical before and after on 7 corpora (this repo, elasticsearch libs, Exposed, dotnet System.Linq, System.Collections, coreclr/gc, a C corpus). Refs #2061 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 5 天前 | |
feat(identity): callable identity plumbing, no language enabled (#2061) Overloads that share a base QN (Java/C#/C++/Kotlin/Swift/Scala/ObjC) merge into one graph node today, and that node collects the CALLS of every overload. The agreed cure is one signature-qualified callable identity for all overloading languages: callable_qn := base_qn [<tparams>] (params) [cvref] This change lands the plumbing with every language still at CBM_CALLABLE_ID_NONE, so the graph is byte-identical; each language is switched on later in its own change (with its index-format bump). - internal/cbm/callable_sig.{c,h}: the per-language mode side table (NONE|TYPED|LABELED_TYPED|LABELED|ARITY; kept out of CBMLangSpec like cbm_string_dispatch_suffixes, since ~160 positional rows would otherwise all need the member under -Wmissing-field-initializers), the shared builder cbm_callable_sig[_mode] for the def and the call-scope QN, and the inverse cbm_qn_callable_base_len[_named]. Suffix contract: no '.', '::' or '->'; whitespace-normalized; comments/annotations/attributes dropped; qualified types keep their last segment; '->' spelled '=>'; variadic '[]' (Java) / '~'; C++ top-level cv dropped, arrays decay, cvref kept; <= 200 bytes, longer ones keep whole leading entries and end in '#<fnv1a64 of the full suffix>)'. The builder is iterative (cursor walk, bounded declarator loop) and self-checks against the inverse. - The inverse is anchored to the node's bare name wherever the name is known: real QNs already end in ')' without being callables (a Java field whose name is its declarator `X = f()`, make `$(VAR)` targets), and only the anchor makes the inverse exact on them. - Leaf splitters strip a suffix first (no-op today): registry by-name key (indexed by the bare name for a suffixed callable), cbm_lsp_bare_segment, cbm_pipeline_qn_class_method_tail, pxc_last_component, pxc_qn_leaf, go_index_visit_node, collect_prod_defs, sg_qn_prefix_len. - New unused fields: CBMDefinition.qn_sig_off, CBMResolvedCall.callee_sig (relocated by result compaction), CBMRegisteredFunc.sig, CBMLSPDef.sig. The lsp_surface codec is unchanged (no format bump in this change). - get_code_snippet and trace_path gain base-QN tiers (cbm_store_find_nodes_by_qn_base) that find nothing until suffixes exist. - store: cbm_store_find_nodes_by_qn_suffix built its LIKE pattern in a 512-byte buffer (a long suffix was truncated into a pattern that matched nothing) and did not escape '_'/'%' (`my_func` also matched `myXfunc`). The pattern is now heap-built, escaped, and matched with ESCAPE '\'. Proof: full sorted node+edge+lsp_surface dumps are byte-identical before and after on 7 corpora (this repo, elasticsearch libs, Exposed, dotnet System.Linq, System.Collections, coreclr/gc, a C corpus). Refs #2061 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 5 天前 | |
fix(python): retain typed fields across file boundaries Carry declared instance-field types through extraction and result compaction into the shared Python type registry so chained calls through imported classes resolve their receiver types. Refs #1277 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 3 天前 | |
Merge pull request #2459 from DeusData/fix/doc-comment-capture fix(extract): capture complete doc comments across languages | 1 天前 | |
Merge pull request #2354 from DeusData/fix/issue-2235 fix(extract): read the url of a JS/TS request-config object (#2235) | 5 天前 | |
Merge pull request #2013 from shafty023/fix/1997-extract-scratch-arena fix(extract): AST traversal stacks no longer outlive the file they walk | 1 个月前 | |
feat(dbt): extract dbt model lineage from Jinja-templated SQL A dbt model is an ordinary .sql file whose dependencies are written {{ ref('other_model') }} or {{ source('group','table') }}, never as literal table names. The SQL grammar cannot read those -- FROM {{ ref( 'x') }} is a parse error to it -- so the dependency structure of an entire dbt project was invisible to the graph. extract_dbt.c runs as a sub-extractor inside the normal indexing pipeline. Per qualifying file it emits one Model definition (named by the file stem, dbt's own model identity) plus one usage per ref()/ source() call; pass_usages resolves those into model -> relation lineage edges like any other reference. Model is a relation label alongside Table and View, which is what makes the rest work without new plumbing: registry seeding, the central relation veto, the incremental surface hash, search ranking and the architecture queries all pick it up from cbm_label_is_relation. Sharing one label class also means a model's source('raw','customers') resolves onto a Table declared in a plain DDL migration in the same repository, so dbt lineage and SQL lineage form one graph rather than two, while the veto keeps model names out of every non-lineage consumer. The pass is self-gating: SQL files only, and only those carrying a real ref()/source() call. The dbt builtins are themselves the evidence, which is cheaper than a dbt_project.yml lookup and more precise -- templated SQL that is not dbt (an Airflow {{ ds }} parameter) produces no Model node and no usages even inside a dbt repository. {% macro %} definitions are deliberately excluded: the vendored tree-sitter-jinja2 grammar has no node types for {% %} statements, so they can only be recovered by a hand-written scanner that cannot see comments or find {% endmacro %} for a correct span. Filed as follow-up rather than approximated. Tests cover the lineage, the last-string-argument semantics of both builtins, the gate (against a plain-SQL control extraction), plain DDL staying untouched, and an end-to-end pipeline case asserting model -> model across files, model -> Table onto plain DDL, and cross-language isolation. Disabling the pass reddens three of them; removing Model from the relation set breaks lineage outright. Implements the lineage half of #575. Co-authored-by: alexisperinger-ux <alexis.peringer@iss-stoxx.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 1 个月前 | |
Merge pull request #2459 from DeusData/fix/doc-comment-capture fix(extract): capture complete doc comments across languages | 1 天前 | |
fix(extract): AST traversal stacks no longer outlive the file they walk Fixes #2010. Refs #1997. Indexing a 14k-file TypeScript repository holds 13.3 GB resident at post_extract on Linux aarch64, and about 3.1 GB of it is AST traversal scratch that no CBMFileResult points at. ts_nstack_init cut its stacks from ctx->arena, which is result->arena. In the parallel path that arena is stored into ec->result_cache[file_idx] (pass_parallel.c:963) and freed only after parallel_resolve and the infra passes (pipeline.c:1324), so a structure written for a one-file lifetime is in fact held for the whole index, once per file, concurrently. The header claimed otherwise and was wrong: "freed when the arena is destroyed at end of file extraction". The largest instance is channels. CHAN_STACK_CAP is 4096 and sizeof(TSNode) is 32, so scan_string_consts_js (extract_channels.c:104) and extract_channels_js (:376) take 128 KB each, per JavaScript or TypeScript file. Across 12,673 such files that is 3,168 MB, and it produced 23 Channel nodes out of 164,684. CBMExtractCtx gains a scratch arena, created and destroyed by the cbm_extract_file_ex call that builds the context. Nothing else changes about where anything lives: ctx->arena still owns every string a CBMFileResult points at, and ctx->scratch owns only traversal stacks, which nothing points at. Nothing a CBMFileResult points at moved. A context built without a scratch, and a file whose scratch arena fails to allocate, both fall back to ctx->arena, which is exactly the behaviour that shipped before this change, so no path can be handed a NULL arena and silently lose nodes. The signature change is what makes the mistake unrepresentable. ts_nstack_init takes the context rather than an arena, so handing it ctx->arena is an incompatible pointer type; push and push_children read the arena the stack recorded, so a stale argument is an arity error. Both fail under -Werror rather than silently retaining memory. The 55 init sites are NOT one uniform sed. 29 pass ctx->arena and 26 pass a local alias, and that alias is shared with result allocations: in parse_zig_imports (extract_imports.c:1093) the same `a` feeds cbm_node_text, strip_quotes, path_last and cbm_imports_push (extract_imports.c:1101, 1106, 1108, 1109). Rebinding it would have moved import names and module paths into memory that dies with the call while the result kept pointing at them. Each site was edited on its own and no alias was rebound. Re-derive the set with: grep -rn 'ts_nstack_init' internal/cbm/extract_*.c grep -rn 'ts_nstack_push' internal/cbm/extract_*.c parse_lisp_imports used its alias only for the stack, so the alias goes too. push_nested_class_nodes and push_class_body_children took an arena purely to reach the stack, so they take the context now; the one call site passes ctx. Two measurements, from two different builds, and they are not the same number. A macOS census build, which instruments every cbm_arena_alloc and sums the blocks behind every live CBMFileResult arena, reports the retained per-file arena set dropping from 11,226 MB mapped to 5,680 MB, and blocks from 48,045 to 34,186. Those are mapped bytes, not resident: that build logs mem.allocator.bound_populations_only, so untouched block tails never commit. The resident figures come from Linux aarch64, gcc 13.3, static build, where the build logs mem.allocator.owned classes=all and mapped equals resident. Five runs per side on the full 13,999-file tree: mem.collect post_extract rss_mb 13293 -> 6752 (-49.2%), mem.phase peak_mb 14840 -> 8565 (-42.3%). nodes 164684, parse_partial 280 and registry entries/defines/imports 787954/804999/100027 identical on all ten runs; edges 835869-835874 against 835867-835881, overlapping ranges, so run to run noise. Wall clock means 35651 ms against 30502 ms, but the ranges overlap and the host had other load, so the only supportable claim is no regression. ObjectScript Studio Export on a small fixture: nodes 18, edges 25, both sides. Scratch block size is 512 KB, chosen from measurement. arena_grow fires on one file in 12,000 at both 512 KB and 1 MB, and constantly at 256 KB, where the two channel stacks alone are exactly 262144 bytes. 512 KB therefore costs half the resident block per worker for the same growth behaviour, and stays under MI_LARGE_MAX_OBJ_SIZE in the vendored mimalloc, which is MI_LARGE_PAGE_SIZE/8, exactly 524288 bytes (types.h:426, with MI_ENABLE_LARGE_PAGES defaulting to 1 at :115 and not overridden by this build), so a 512 KB block is still bin-allocated from a 4 MiB large page. Growth is not free at that size for the same reason: arena_grow doubles to 1 MiB, above that bound and therefore a singleton OS allocation. One file in twelve thousand pays it, which is why it is accepted. The largest single file's scratch high water mark on this corpus is 33.5 MB, now transient rather than retained. Tests. Nothing in the suite asserted on result->arena size, block count or total_alloc after cbm_extract_file, which is why this went unnoticed for the life of extract_node_stack.h. traversal_stack_not_in_result_arena_issue2010 pins the byte budget: extracting "export const x = 1;" as TypeScript charged 365984 bytes to the result arena before and charges 87456 after, a difference of 278528, exactly the two 4096-entry channel walks plus the 512-entry ES import walk at 32 bytes per TSNode. Of the 87456 that remain, 7680 is the defs item array at GROW_ARRAY's starting capacity of 32 times sizeof(CBMDefinition) 240 and the other 79776 is everything else the extraction interns, none of it traversal scratch; the 128 KB bound is therefore above the new figure with room and a factor of four below the old one. extract_traversal_stacks_come_from_ctx_scratch_issue2010 in test_mem.c pins the lifetime, since a byte budget alone would also be satisfied by shrinking CHAN_STACK_CAP: it builds the extraction context directly over two arenas it owns, runs cbm_extract_channels, and asserts the scratch took at least the 262144 bytes of the two walks while the result arena did not. extract_c_macro_hidden_call_survives_preprocessed_pass_issue2010 is not a scratch test. pp_ctx carries ctx->scratch so every context in the file is uniform, but nothing reads it there: pp_ctx reaches only cbm_extract_unified and cbm_run_c_lsp, and neither extract_unified.c nor anything under internal/cbm/lsp/ includes extract_node_stack.h, so no traversal stack is built on that path today. The test guards the macro-expansion path itself, which had no assertion on a call that exists only after expansion. ObjectScript composite extraction is already covered by pipeline_objectscript_export_preserves_calls_sequential_parallel and iris_export_xml_multi_class, so no new test is added there; a per-call arena is correct on that path by construction, since each call owns and destroys its own. Local: scripts/test.sh, 7558 passed, 0 failed, 8 skipped, 139 suites. Alternatives considered Thread-local scratch arena, reset per file, with a depth guard for nesting. Rejected on three counts. This codebase never uses thread-local storage for owned bulk memory; every existing _Thread_local here is a small cache, a guard or a log buffer, and every per-worker resource is a field in a context struct. It would have needed cbm_arena_reset, which has no production caller and a documented subtlety with grown block sizes (arena.c:213-217). And extraction threads are created per cbm_parallel_for call and joined (worker_pool.c:71,88) with no thread-exit hook in this project's own code, so the arena would have leaked once per thread per dispatch and turned the leak lane red. Save and restore mark and release on the arena. Rejected because a per-call arena already makes nesting correct by construction, and a mark type plus two functions would have to be added to both arena headers. Per-walk arenas, one per traversal rather than one per file. Rejected on cost: 55 sites would each need an explicit span, for the same bound. Right-sizing the initial capacities alone. CHAN_STACK_CAP at 256 rather than 4096 would cut those two sites from 3,168 MB to about 198 MB, which is real, but it tunes the constant while the lifetime defect remains at all 55 sites, and growth is not free: 256 doubling to 4096 abandons 122,880 bytes, and one file in this corpus took 33.5 MB of stacks. The cap and the scratch block size are coupled, which is a further reason to change one at a time. Worth doing separately. Routing to a scratch inside extract_node_stack.h while keeping the arena parameters. Rejected because a parameter that is ignored lies about the API, and the alias hazard above would stay a review obligation instead of a compile error. Freeing the per-file result arena earlier. Rejected: the results are read by parallel_resolve and the infra passes. That is the retention half #1925 addresses, and this change is independent of it. Signed-off-by: Daniel Ochoa <daniel.ochoa@closedloop.ai> | 1 个月前 | |
fix(pipeline): resolve PSR-4 imports to exact class files Preserve import kinds and resolve class imports through Composer prefixes to the existing target class file, including autoload-dev mappings. Refs #1186 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 2 天前 | |
fix(k8s): restore Resource label so the K8s pipeline produces nodes + INFRA_MAPS An earlier relabel (extract_k8s.c Resource->Class, to satisfy labels-valid) left pass_k8s.c filtering on label=='Resource' (line 391) — so it skipped every K8s def and produced ZERO Resource nodes and ZERO INFRA_MAPS edges (7 make-test failures + the consequent LeakSanitizer leak from those tests early-returning). Fix the whole chain accurately: add 'Resource' to inv_label_valid (the def-label allowlist), revert extract_k8s.c to emit 'Resource' (manifest + kustomize), so pass_k8s.c matches -> Resource nodes + INFRA_MAPS restored. K8s repro goldens updated Class->Resource to match. Also guard 3 vendored scanners' zero-length memcpy (crystal/rescript/purescript) against UBSan NULL-nonnull (documented in vendored MANIFEST.md). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 3 个月前 | |
perf: cut the allocation churn and repeated work the sanitizer found Every item here was measured on a real corpus before and after, not reasoned about. Go corpus, event lane, first run -> now: 245.5 M allocations -> 134.8 M, 190 GB cumulative -> 29.3 GB, 158 GB never written -> 9.2 GB, 132 M churn pairs -> 35.7 M, 93 GB of allocator byte work -> 0.84 GB. Wall clock: Go 40.7 s -> 19.3 s, linux kernel 239.6 s -> 194.6 s, and all ten LSP languages faster with none slower. Graph output is unchanged, proven rather than assumed: per-class multiset digests over the full kernel index (8,563,471 nodes identical, 18/20 node classes and 18/24 edge classes byte-identical, the rest differing only in the classes a same-binary re-run also perturbs through worker merge order). The changes, each with its own reason: arena an exact arena that is appended to later restarts growth at 8 KB instead of the 64 KB default — the cross-file pass adds a few resolved calls to a compacted result, so the default block was ~60 KB untouched per result, 0.5 GB of the Go worker peak. cbm_arena_init_lazy defers the first block for arenas that are usually never used at all. scratch per-thread reuse instead of allocate-and-free per unit of work: result_compact (8 MB kept), the cross-file pass (4 MB), extract_defs' walk stack, and the sqlite index cells (a 1 MB arena replacing 12.1 M malloc/free pairs on Go). Walk frames moving into scratch is why pass_k8s now opens the keep window explicitly: without it every YAML file opened a 512 KB scratch on the main thread, +3 GB. sqlite index cells are written in place — building the record and copying it into the cell was a second allocation per entry, 16 M record buffers of pure churn on Go. registry is_test_qn and common_prefix_len in one pass, per-bucket test flags computed once instead of per candidate comparison, and the reachability keys arena allocated lazily. minhash the unique-trigram set is sized from the trigram count and skips repeats, and LSH queries reuse one seen-set per worker instead of allocating per query. parallel lsp_surface rows and the k8s manifest prep now run across workers (graph writes stay sequential and ordered), the file node is hoisted out of register_and_link_def, and configlink prepares dependency imports once with a length prefilter. go_lsp one sealed stdlib registry shared by all workers, published by CAS, instead of one per worker. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 18 天前 | |
Merge pull request #2013 from shafty023/fix/1997-extract-scratch-arena fix(extract): AST traversal stacks no longer outlive the file they walk | 1 个月前 | |
fix(extract): AST traversal stacks no longer outlive the file they walk Fixes #2010. Refs #1997. Indexing a 14k-file TypeScript repository holds 13.3 GB resident at post_extract on Linux aarch64, and about 3.1 GB of it is AST traversal scratch that no CBMFileResult points at. ts_nstack_init cut its stacks from ctx->arena, which is result->arena. In the parallel path that arena is stored into ec->result_cache[file_idx] (pass_parallel.c:963) and freed only after parallel_resolve and the infra passes (pipeline.c:1324), so a structure written for a one-file lifetime is in fact held for the whole index, once per file, concurrently. The header claimed otherwise and was wrong: "freed when the arena is destroyed at end of file extraction". The largest instance is channels. CHAN_STACK_CAP is 4096 and sizeof(TSNode) is 32, so scan_string_consts_js (extract_channels.c:104) and extract_channels_js (:376) take 128 KB each, per JavaScript or TypeScript file. Across 12,673 such files that is 3,168 MB, and it produced 23 Channel nodes out of 164,684. CBMExtractCtx gains a scratch arena, created and destroyed by the cbm_extract_file_ex call that builds the context. Nothing else changes about where anything lives: ctx->arena still owns every string a CBMFileResult points at, and ctx->scratch owns only traversal stacks, which nothing points at. Nothing a CBMFileResult points at moved. A context built without a scratch, and a file whose scratch arena fails to allocate, both fall back to ctx->arena, which is exactly the behaviour that shipped before this change, so no path can be handed a NULL arena and silently lose nodes. The signature change is what makes the mistake unrepresentable. ts_nstack_init takes the context rather than an arena, so handing it ctx->arena is an incompatible pointer type; push and push_children read the arena the stack recorded, so a stale argument is an arity error. Both fail under -Werror rather than silently retaining memory. The 55 init sites are NOT one uniform sed. 29 pass ctx->arena and 26 pass a local alias, and that alias is shared with result allocations: in parse_zig_imports (extract_imports.c:1093) the same `a` feeds cbm_node_text, strip_quotes, path_last and cbm_imports_push (extract_imports.c:1101, 1106, 1108, 1109). Rebinding it would have moved import names and module paths into memory that dies with the call while the result kept pointing at them. Each site was edited on its own and no alias was rebound. Re-derive the set with: grep -rn 'ts_nstack_init' internal/cbm/extract_*.c grep -rn 'ts_nstack_push' internal/cbm/extract_*.c parse_lisp_imports used its alias only for the stack, so the alias goes too. push_nested_class_nodes and push_class_body_children took an arena purely to reach the stack, so they take the context now; the one call site passes ctx. Two measurements, from two different builds, and they are not the same number. A macOS census build, which instruments every cbm_arena_alloc and sums the blocks behind every live CBMFileResult arena, reports the retained per-file arena set dropping from 11,226 MB mapped to 5,680 MB, and blocks from 48,045 to 34,186. Those are mapped bytes, not resident: that build logs mem.allocator.bound_populations_only, so untouched block tails never commit. The resident figures come from Linux aarch64, gcc 13.3, static build, where the build logs mem.allocator.owned classes=all and mapped equals resident. Five runs per side on the full 13,999-file tree: mem.collect post_extract rss_mb 13293 -> 6752 (-49.2%), mem.phase peak_mb 14840 -> 8565 (-42.3%). nodes 164684, parse_partial 280 and registry entries/defines/imports 787954/804999/100027 identical on all ten runs; edges 835869-835874 against 835867-835881, overlapping ranges, so run to run noise. Wall clock means 35651 ms against 30502 ms, but the ranges overlap and the host had other load, so the only supportable claim is no regression. ObjectScript Studio Export on a small fixture: nodes 18, edges 25, both sides. Scratch block size is 512 KB, chosen from measurement. arena_grow fires on one file in 12,000 at both 512 KB and 1 MB, and constantly at 256 KB, where the two channel stacks alone are exactly 262144 bytes. 512 KB therefore costs half the resident block per worker for the same growth behaviour, and stays under MI_LARGE_MAX_OBJ_SIZE in the vendored mimalloc, which is MI_LARGE_PAGE_SIZE/8, exactly 524288 bytes (types.h:426, with MI_ENABLE_LARGE_PAGES defaulting to 1 at :115 and not overridden by this build), so a 512 KB block is still bin-allocated from a 4 MiB large page. Growth is not free at that size for the same reason: arena_grow doubles to 1 MiB, above that bound and therefore a singleton OS allocation. One file in twelve thousand pays it, which is why it is accepted. The largest single file's scratch high water mark on this corpus is 33.5 MB, now transient rather than retained. Tests. Nothing in the suite asserted on result->arena size, block count or total_alloc after cbm_extract_file, which is why this went unnoticed for the life of extract_node_stack.h. traversal_stack_not_in_result_arena_issue2010 pins the byte budget: extracting "export const x = 1;" as TypeScript charged 365984 bytes to the result arena before and charges 87456 after, a difference of 278528, exactly the two 4096-entry channel walks plus the 512-entry ES import walk at 32 bytes per TSNode. Of the 87456 that remain, 7680 is the defs item array at GROW_ARRAY's starting capacity of 32 times sizeof(CBMDefinition) 240 and the other 79776 is everything else the extraction interns, none of it traversal scratch; the 128 KB bound is therefore above the new figure with room and a factor of four below the old one. extract_traversal_stacks_come_from_ctx_scratch_issue2010 in test_mem.c pins the lifetime, since a byte budget alone would also be satisfied by shrinking CHAN_STACK_CAP: it builds the extraction context directly over two arenas it owns, runs cbm_extract_channels, and asserts the scratch took at least the 262144 bytes of the two walks while the result arena did not. extract_c_macro_hidden_call_survives_preprocessed_pass_issue2010 is not a scratch test. pp_ctx carries ctx->scratch so every context in the file is uniform, but nothing reads it there: pp_ctx reaches only cbm_extract_unified and cbm_run_c_lsp, and neither extract_unified.c nor anything under internal/cbm/lsp/ includes extract_node_stack.h, so no traversal stack is built on that path today. The test guards the macro-expansion path itself, which had no assertion on a call that exists only after expansion. ObjectScript composite extraction is already covered by pipeline_objectscript_export_preserves_calls_sequential_parallel and iris_export_xml_multi_class, so no new test is added there; a per-call arena is correct on that path by construction, since each call owns and destroys its own. Local: scripts/test.sh, 7558 passed, 0 failed, 8 skipped, 139 suites. Alternatives considered Thread-local scratch arena, reset per file, with a depth guard for nesting. Rejected on three counts. This codebase never uses thread-local storage for owned bulk memory; every existing _Thread_local here is a small cache, a guard or a log buffer, and every per-worker resource is a field in a context struct. It would have needed cbm_arena_reset, which has no production caller and a documented subtlety with grown block sizes (arena.c:213-217). And extraction threads are created per cbm_parallel_for call and joined (worker_pool.c:71,88) with no thread-exit hook in this project's own code, so the arena would have leaked once per thread per dispatch and turned the leak lane red. Save and restore mark and release on the arena. Rejected because a per-call arena already makes nesting correct by construction, and a mark type plus two functions would have to be added to both arena headers. Per-walk arenas, one per traversal rather than one per file. Rejected on cost: 55 sites would each need an explicit span, for the same bound. Right-sizing the initial capacities alone. CHAN_STACK_CAP at 256 rather than 4096 would cut those two sites from 3,168 MB to about 198 MB, which is real, but it tunes the constant while the lifetime defect remains at all 55 sites, and growth is not free: 256 doubling to 4096 abandons 122,880 bytes, and one file in this corpus took 33.5 MB of stacks. The cap and the scratch block size are coupled, which is a further reason to change one at a time. Worth doing separately. Routing to a scratch inside extract_node_stack.h while keeping the arena parameters. Rejected because a parameter that is ignored lies about the API, and the alias hazard above would stay a review obligation instead of a compile error. Freeing the per-file result arena earlier. Rejected: the results are read by parallel_resolve and the infra passes. That is the retention half #1925 addresses, and this change is independent of it. Signed-off-by: Daniel Ochoa <daniel.ochoa@closedloop.ai> | 1 个月前 | |
Merge pull request #2013 from shafty023/fix/1997-extract-scratch-arena fix(extract): AST traversal stacks no longer outlive the file they walk | 1 个月前 | |
fix(extract): no clock decides what ends up in the graph Indexing the same C# corpus twice on the same machine produced 5,794,336 edges and then 5,794,328. Every other edge type matched exactly, every semantic checksum matched (447,064 functions, 81,534 tokens, identical vec/idf/toks), and the difference was ten WRITES edges out of one generated file. Three places in extraction decided graph content from a clock: * the unified AST walk stopped on a thread-CPU deadline; * a file whose parse spent more than half its budget was refused the per-file and cross-file LSP walks; * so was a file whose parse plus walk exceeded the budget. The first is the one that bit. Any file large enough to reach the deadline is cut at whatever node the clock lands on, and five generated .NET JIT tests reached it on every run: hugeSimpleExpr1.cs stopped at node 2,660,352 in one run and 2,574,336 in the next, out of 2,774,794. That file's tail is field assignments, so the two runs disagreed by exactly ten WRITES edges. The graph was a function of how fast the machine happened to be running, and nothing said so: result->walk_truncated was set and no code anywhere read it. The deadline existed because those files cost 18-48 us per visited node, and that turns out to be one fixable thing. csharp_callable_value_site() was the only language branch in call_reference_candidate_site() still climbing with ts_node_parent, which tree-sitter answers by descending from the ROOT, scanning children at every level; every other language already climbs on the walk cursor. On expressions megabytes deep that was ~85% of the entire unified walk (sampled: stamp_usage_site -> ts_node_parent -> ts_node_child_with_descendant). It now takes the cursor and the parent the caller already resolved, and is_direct_argument_value_cursor() is split so a caller that has stepped the cursor onto the parent can continue from there rather than paying for it twice. With that fixed, the budgets are not needed to make indexing finish, so they are gone rather than made deterministic. A cutoff in nodes would at least be reproducible, but it still answers "what is in this repo?" with "as much as we felt like reading". CBM_WALK_MAX_NODES reinstates one for an operator who wants it; the default is no budget. Removing the two LSP-skip rules was checked against the file and the path they were added for (a 23 MB single-expression JIT test that once cost 354 s in the walk and then crashed the cross-file resolve): the whole 12k-file C# corpus twice with cross-file resolve, that file alone, and synthetic C# nested 10,000 levels deep — no crash, and past roughly that depth tree-sitter stops producing a usable tree, so a tree deep enough to endanger a recursive walk never reaches one. C# corpus, before -> after: edges 5,794,318 / 5,794,328 / 5,794,336 -> 5,794,873 every run truncated 5 files, unreported -> 0 time ~115 s -> 97-101 s Deterministic, about 550 edges more complete (the tails nobody was walking), and faster than the build that was cutting them off. All ten bench corpora now return identical nodes and edges across two runs. A truncated walk and a skipped LSP walk are now reported in the coverage report with how far the walk got ("1673216/4133946 nodes walked"), like a partial parse. A file the walk abandoned halfway used to be reported as fully indexed, which is a graph quietly lying about its coverage. Tests: extract_csharp_argument_values_use_the_walk_cursor asserts no occurrence falls back to the root-descending parent lookup — 128 fallbacks without the fix and 0 with it. Its fixture parenthesises the argument on purpose: a bare Sink(Target) needs no climb at all, so it cannot tell the two walks apart and passed either way while the bug was in place. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 16 天前 | |
fix(python): carry bare-call parameter bindings in the walk Replaces the cursor ascent from 0b0d143c. That commit fixed a real hang -- ts_node_parent() is O(depth), so a parent chain is O(depth^2) per call and O(depth^3) across a file where every level of f(f(f(...))) is itself a bare call, which hung stack_overflow_b's 30,000-deep fixture at 900s. But it fixed it by making each hop cheap rather than by removing the per-call walk, so the cost stayed O(depth) per call and needed a 64-ancestor cap to stay bounded. That cap FAILS OPEN. Past 64 ancestors the guard stopped suppressing, so deep-but-ordinary code silently kept the fabricated edges the guard exists to remove -- and nothing failed when it did. Carry the answer instead. The unified walk binds a def's or lambda's parameters when it opens that scope and unwinds them when it closes, so the guard is an O(1) map lookup. This is the lesson CBMWalkScope already records one struct away: carry walk state, never recompute it per node. - CBMParamSlot is a name -> active-count map, not a set. `def outer(run): def inner(run):` binds one name twice and leaving the inner scope must not unbind the outer. - Bindings are pushed at the end of push_boundary_scopes, after every scope push for that node, so they land in the frame just pushed -- covering function_definition (push_function_scope) and lambda (push_lexical_boundary) alike -- and pop_expired_scopes unwinds to the frame's entry height. - Allocation failure sets py_param_tracking_failed and the lookup answers false forever after: a lost suppression, never a lost edge. The repo's other binding table cannot serve this. CBMLexicalBinding is qsort-ed in cbm_finalize_lexical_usages AFTER the walk and its active_end stays 0 until then, so its binary search is invalid from handle_calls, which runs mid-walk and before handle_usages. extract_python_bare_call_scope_walk_is_bounded pinned the cap's fail-open contract, which no longer exists, and is replaced by extract_python_bare_call_flag_is_depth_independent. It keeps the shallow and 200-ancestor cases -- the deep one now asserts the call IS flagged, so reintroducing a cap fails it -- and adds an unwind case: `handler` as both a parameter and a module-level def, asserting the two shadowed calls are flagged and the sibling call after that scope closed is NOT. A binding outliving its frame would destroy that true edge, the one direction this guard must never fail in. It passed on the first run. extraction 327 passed; pipeline registry parallel complexity 407 passed; stack_overflow_b 6 passed in 38s (46s with the cursor ascent, 900s rc=124 before it), lsp_python_deep_nesting_no_crash PASS. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 1 个月前 | |
fix(rescript): stop binary .res resources hanging the indexer (#2176) A 4.7 MB binary Godot resource (piko_walk_mesh.res) never finished indexing: `.res` maps to ReScript, so the resource was parsed as ReScript source and the definitions pass spun. Three defects, each fixed: 1. Quadratic usage walk (the hang). The ReScript `let` binding check (is_first_named_part_of, shared with Julia, Typst, Elm, PureScript and Nickel) climbed from every named leaf with ts_node_parent, which descends from the root and scans each level's children. A binary file parses into a flat error tree (~170k root children per MB), so each climb cost O(root children) and the file O(n^2): 1 MB spent 46 s in cbm_extract_unified, 4.7 MB never returned. The check now climbs the unified walk's own cursor (O(1) per hop), like the other occurrence classifiers; the node-based climb stays only for callers outside the walk. 1 MB: 46 s -> 0.03 s; 4.7 MB: 2.2 s end to end. 2. Quadratic lexing in error recovery. tree-sitter retries the lexer at every byte of an unparseable stretch with every external token valid; the ReScript scanner's template-string loop then ran to the next backtick/$/backslash/NUL (end of file if none) and discarded the result. That cost O(n^2) inside lexing, where the parse budget's progress callback never runs, so such files were dropped by the 5 s CPU budget instead of parsed -- plain text too (100 KB of non-ReScript text: 15 s -> 0.18 s). Local scanner patch: skip the template branch when NEWLINE is also valid, which only the ERROR state allows. Valid files parse to identical trees. Recorded in vendored/grammars/MANIFEST.md. 3. Binary content parsed as source. `.res` is also Godot's binary resource format and the Windows compiled-resource format. Following the .m/.cls/.frm/.inc/.cfc precedent, cbm_disambiguate_res reads the first 4 KB and reports a file holding a NUL byte as unsupported; ReScript text never contains one. Tests (deterministic work counters, no clocks): - extraction::extract_rescript_let_bindings_use_the_walk_cursor -- zero root-descending parent hops at 128 and 1024 statements (before: 515 / 4099). - complexity::complexity_rescript_error_recovery_lexing_is_linear -- lexer bytes pulled through a chunked TSInput, 4 KB vs 8 KB, ratio must be linear (before 3.99, after 2.00). - language::lang_res_binary_resource_unsupported and lang_res_rescript_stays_rescript. Fixes #2176 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 10 天前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add ArkTS (.ets) language support via first-party tree-sitter-typescript fork Routing .ets through the plain TypeScript grammar produced no definition node for any @Component struct (0/19 across the OpenHarmony sample corpus), parsed struct members as statement labels and build() as a call expression, and registered ArkUI builtins (Column, Row, ListItem, ...) as user function definitions with 28% fabricated cross-file CALLS edges. This adds a dedicated ArkTS grammar that keeps everything TypeScript already handles and models the ArkTS syntax TS cannot parse. Grammar (tools/tree-sitter-arkts/, generated parser vendored to internal/cbm/vendored/grammars/arkts/, ABI 15 via tree-sitter-cli 0.25.10): first-party derivative of tree-sitter/tree-sitter-typescript v0.23.2 @ 75b3874edb2d (the commit our vendored typescript/tsx come from) on the tree-sitter-javascript v0.23.1 base @ 3a837b6f3658 (the version upstream's package-lock pins). LICENSE carries the upstream MIT text verbatim plus a derivative notice; the external scanner is a symbol-rename trampoline over a byte-identical copy of the already-vendored typescript scanner header. ArkTS additions: decorated struct component declarations with class-shaped bodies; UI-DSL trailing-closure component calls with attribute chains after the closing brace; decorators on function declarations (@Builder, @Extend(T), @Styles); @Extend/@Styles leading-dot attribute-chain bodies; anonymous stateStyles style blocks; import lazy named imports. Wiring: CBM_LANG_ARKTS enum + lang spec (struct_declaration as a class kind labeled Struct, public_field_definition fields), .ets extension, LANG_NAMES/userconfig entries, grammar shim, and TS-family joins across the extractors and the name-resolution guards (parse_es_imports, JS keywords/test-file/func-kind/scope tables, TS heritage/param/receiver/ namespace/type-ref/channel/usage paths, js_ts_family and the weak-method- match suppression). Field definitions now also carry their decorators, so @State/@Prop/@Link members are findable with their roles. Tests: grammar corpus (35 cases), per-language regression + label golden + import net + extension cases, seven ArkTS extraction tests including the phantom-builtin regression (ArkUI builtins must never be definitions), and capability-ledger / call-node-manifest / call-argument-matrix rows. On the 14-file OpenHarmony sample corpus the TS-grammar baseline had 12/14 files with parse errors (231 ERROR nodes); the arkts grammar parses 14/14 clean, and an end-to-end index yields all 19 components as Struct nodes, 68 methods, 119 fields, and zero phantom builtin definitions. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 1 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
feat(lang): add CFML / ColdFusion support (cfscript + tag dialects) Vendor the MIT-licensed cfmleditor/tree-sitter-cfml grammars and wire both CFML dialects through the pipeline: - .cfc components -> cfscript grammar (CBM_LANG_CFSCRIPT): a JS-like grammar, so function/method declarations, calls and imports extract with the shared JS/TS arrays. Components' functions become Function nodes. - .cfm templates -> cfml tag grammar (CBM_LANG_CFML): tag-based (HTML-derived). Embedded <cfscript> functions extract via function_declaration; tag-based <cffunction name="..."> nodes are handled by a dedicated walker branch that reads the name attribute (cf_function_tag has no name field). Shared scanner headers are vendored at vendored/common/ so the grammar's ../../common/ includes resolve without editing vendored sources. Adds an ASCII case-insensitive compare for CFML's case-insensitive attributes. Tests cover both dialects and the extension mapping. LICENSEs vendored. Closes #38 | 4 个月前 | |
feat(lang): add CFML / ColdFusion support (cfscript + tag dialects) Vendor the MIT-licensed cfmleditor/tree-sitter-cfml grammars and wire both CFML dialects through the pipeline: - .cfc components -> cfscript grammar (CBM_LANG_CFSCRIPT): a JS-like grammar, so function/method declarations, calls and imports extract with the shared JS/TS arrays. Components' functions become Function nodes. - .cfm templates -> cfml tag grammar (CBM_LANG_CFML): tag-based (HTML-derived). Embedded <cfscript> functions extract via function_declaration; tag-based <cffunction name="..."> nodes are handled by a dedicated walker branch that reads the name attribute (cf_function_tag has no name field). Shared scanner headers are vendored at vendored/common/ so the grammar's ../../common/ includes resolve without editing vendored sources. Adds an ASCII case-insensitive compare for CFML's case-insensitive attributes. Tests cover both dialects and the extension mapping. LICENSEs vendored. Closes #38 | 4 个月前 | |
feat(lang): add Chialisp support with a first-party tree-sitter grammar Chialisp (.clsp/.clib/.clinc) is the Chia smart-coin language. The only public grammar, Quexington/tree-sitter-chialisp, cannot parse it: on the five reference files from chia-blockchain@main it produced 3-34 ERROR nodes each. Three defects account for all of them, and each desynchronised the rest of the file rather than losing one node: * comments were `/;.*\r\n/` -- CRLF-only, while real files are LF; * `(include foo.clib)` was rejected because of the dot in the filename; * `(defconstant NAME <expr>)` accepted only a primitive after the name. So the grammar is written here instead, in tools/tree-sitter-chialisp/, and files as first-party: its vendored LICENSE is the project's own, byte-identical to the repository root, carrying no third-party copyright. It is deliberately GENERIC -- source_file / list / symbol / string / number / hex / dot / comment, modelling the clvm_tools reader rather than the Chialisp form vocabulary. `mod`, `defun`, `defconstant` and `include` are ordinary head symbols; which lists are definitions is decided in extract_defs.c. A dialect that adds a form (cl-21 -> cl-23 -> cl-26 each added several) therefore needs no regenerated parser. Acceptance: all five reference files parse with zero ERROR and zero MISSING nodes; the three defects above are pinned as tests. Extraction, and the four decisions in it: * `Constant` is admitted to the cross-file name registry (cbm_label_is_registry_symbol). Chialisp's condition codes live as `(defconstant ...)` in a shared .clib and are referenced by name from every puzzle; labelled but unregistered they are seeded nowhere, and roughly half of a Chialisp graph would be present as nodes yet never the target of an edge. All three registry-seeding sites route through that one predicate, so they cannot diverge. No other language emits `Constant`, so the blast radius is exactly Chialisp. Pinned by a two-file graph test that goes red when the label is removed from the predicate. * `export` / `namespace` are NOT definition heads. `(export foo)` names a function already defined in the same file, so treating it as a def head mints a second node for one symbol. They stay in the not-a-call filter. * A parameter list is a `list` too, but it binds rather than invokes. Without that gate `(defun check_conditions (HEIGHTLOCK conditions) ...)` mints a phantom call to HEIGHTLOCK, and real Chialisp is mostly such definitions. * `(include *standard-cl-26*)` is a dialect directive, not a file dependency; recording it would invent a dependency on a file that does not exist. Three helpers that the defs, calls and unified walks all need -- the quote-data gate, comment-skipping named-child access, and the Chialisp def-head set -- live in helpers.c rather than being copied per translation unit. They previously existed as byte-identical duplicates that had already drifted: extract_defs.c skipped comments when reading a def head while its mirror in extract_unified.c did not, so a comment between a head and its name desynchronised defs from call scope. Both now use the same accessor. Ledger: 161 -> 162 grammars across the seven published surfaces, ABI-14 86 -> 87, first-party 13 -> 14, CBM_LANG_COUNT 164 -> 165. Distilled from PR #919 by darkverbito, whose extraction logic -- the CLVM not-a-call vocabulary, the quoted-data gate, the filename-derived `mod` name and the include/embed import handling -- is kept and is the substance of this change. The grammar that PR vendored is not used. Co-authored-by: darkverbito <8049985+darkverbito@users.noreply.github.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 1 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add MATLAB, Lean 4, FORM, and Magma language support (59 -> 63) - MATLAB: vendored from acristoffers/tree-sitter-matlab, definitions-only (no call graph due to A(1) indexing ambiguity), .m file disambiguation via GitHub Linguist heuristics in discover.go - Lean 4: vendored from Julian/tree-sitter-lean, definitions-only (Lean 4 syntax is runtime-extensible), ABI 13 backward-compatible - FORM: custom tree-sitter grammar written from scratch for symbolic manipulation language used in particle physics (CERN/Nikhef) - Magma: custom tree-sitter grammar written from scratch for computer algebra system (Univ. Sydney) Integration touchpoints per language: cbm.h enum, cbm.go map, lang.go constants, lang_specs.c (extern + arrays + table + switch), helpers.c (func_kinds + module_parents + test detection), extract_defs.c (resolve_func_name for MATLAB/Lean), extract_imports.c, lang/<name>.go spec, discover.go (.m heuristic), regression_test.go (9 new tests). | 6 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
fix(parser): use alias-safe Haskell arrays Preload the shared Tree-sitter array helpers before the pinned Haskell scanner so optimized builds do not retain a stale contents pointer across realloc. Keep a compile-time signature guard to prevent regression. Fixes #1231 Signed-off-by: wargloom <wargloom@gmail.com> | 2 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add MATLAB, Lean 4, FORM, and Magma language support (59 -> 63) - MATLAB: vendored from acristoffers/tree-sitter-matlab, definitions-only (no call graph due to A(1) indexing ambiguity), .m file disambiguation via GitHub Linguist heuristics in discover.go - Lean 4: vendored from Julian/tree-sitter-lean, definitions-only (Lean 4 syntax is runtime-extensible), ABI 13 backward-compatible - FORM: custom tree-sitter grammar written from scratch for symbolic manipulation language used in particle physics (CERN/Nikhef) - Magma: custom tree-sitter grammar written from scratch for computer algebra system (Univ. Sydney) Integration touchpoints per language: cbm.h enum, cbm.go map, lang.go constants, lang_specs.c (extern + arrays + table + switch), helpers.c (func_kinds + module_parents + test detection), extract_defs.c (resolve_func_name for MATLAB/Lean), extract_imports.c, lang/<name>.go spec, discover.go (.m heuristic), regression_test.go (9 new tests). | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add MATLAB, Lean 4, FORM, and Magma language support (59 -> 63) - MATLAB: vendored from acristoffers/tree-sitter-matlab, definitions-only (no call graph due to A(1) indexing ambiguity), .m file disambiguation via GitHub Linguist heuristics in discover.go - Lean 4: vendored from Julian/tree-sitter-lean, definitions-only (Lean 4 syntax is runtime-extensible), ABI 13 backward-compatible - FORM: custom tree-sitter grammar written from scratch for symbolic manipulation language used in particle physics (CERN/Nikhef) - Magma: custom tree-sitter grammar written from scratch for computer algebra system (Univ. Sydney) Integration touchpoints per language: cbm.h enum, cbm.go map, lang.go constants, lang_specs.c (extern + arrays + table + switch), helpers.c (func_kinds + module_parents + test detection), extract_defs.c (resolve_func_name for MATLAB/Lean), extract_imports.c, lang/<name>.go spec, discover.go (.m heuristic), regression_test.go (9 new tests). | 6 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add MATLAB, Lean 4, FORM, and Magma language support (59 -> 63) - MATLAB: vendored from acristoffers/tree-sitter-matlab, definitions-only (no call graph due to A(1) indexing ambiguity), .m file disambiguation via GitHub Linguist heuristics in discover.go - Lean 4: vendored from Julian/tree-sitter-lean, definitions-only (Lean 4 syntax is runtime-extensible), ABI 13 backward-compatible - FORM: custom tree-sitter grammar written from scratch for symbolic manipulation language used in particle physics (CERN/Nikhef) - Magma: custom tree-sitter grammar written from scratch for computer algebra system (Univ. Sydney) Integration touchpoints per language: cbm.h enum, cbm.go map, lang.go constants, lang_specs.c (extern + arrays + table + switch), helpers.c (func_kinds + module_parents + test detection), extract_defs.c (resolve_func_name for MATLAB/Lean), extract_imports.c, lang/<name>.go spec, discover.go (.m heuristic), regression_test.go (9 new tests). | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
chore(grammar): vendor Mojo tree-sitter parser Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 3 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
feat(objectscript): add InterSystems IRIS ObjectScript language support Add ObjectScript (InterSystems IRIS / Caché) as a supported language, covering the UDL class format (.cls), MAC/INT routines (.mac/.int/.rtn), include/macro files (.inc), and IRIS Studio Export XML. Definition extraction (extract_defs.c): Class, Method, ClassMethod, Property, Parameter, Index, Trigger (with body text), XData, Storage, and Query members as graph nodes; base classes from the Extends clause. Call dispatch resolution (extract_calls.c) — four ObjectScript patterns that are structurally invisible to text search: 1. ##class(Pkg.Class).Method() explicit cross-class call 2. ..Method() relative-dot self-call (the dominant intra-class form; large impact on CALLS completeness) 3. $$$Macro macro expansion via a per-project table built from .inc files 4. type inference from %New/%OpenId + declared return types Ensemble production topology (pass_ensemble_routing.c): EnsembleItem nodes per production component and ROUTES_TO edges resolved from ProductionDefinition XData, plus WorkMgr .Queue("##class(X).method") dispatch — all parsed statically at index time, no live IRIS required. Language detection (language.c): .mac/.int/.rtn map to ObjectScript routine directly; .cls (shared with Apex) and .inc (shared with BitBake) are disambiguated by content, defaulting to the existing language on any doubt so neither Apex nor BitBake detection regresses. The two new per-project tables (macros, return types) are threaded through a new internal cbm_extract_file_ex() so the public cbm_extract_file() signature is unchanged. The tree-sitter grammars for ObjectScript UDL and routine are vendored in internal/cbm/vendored/grammars/objectscript_{udl,routine}/ from https://github.com/intersystems/tree-sitter-objectscript (MIT, ABI 15). Refs #462 Signed-off-by: Thomas Dyar <tdyar@intersystems.com> | 2 个月前 | |
feat(objectscript): add InterSystems IRIS ObjectScript language support Add ObjectScript (InterSystems IRIS / Caché) as a supported language, covering the UDL class format (.cls), MAC/INT routines (.mac/.int/.rtn), include/macro files (.inc), and IRIS Studio Export XML. Definition extraction (extract_defs.c): Class, Method, ClassMethod, Property, Parameter, Index, Trigger (with body text), XData, Storage, and Query members as graph nodes; base classes from the Extends clause. Call dispatch resolution (extract_calls.c) — four ObjectScript patterns that are structurally invisible to text search: 1. ##class(Pkg.Class).Method() explicit cross-class call 2. ..Method() relative-dot self-call (the dominant intra-class form; large impact on CALLS completeness) 3. $$$Macro macro expansion via a per-project table built from .inc files 4. type inference from %New/%OpenId + declared return types Ensemble production topology (pass_ensemble_routing.c): EnsembleItem nodes per production component and ROUTES_TO edges resolved from ProductionDefinition XData, plus WorkMgr .Queue("##class(X).method") dispatch — all parsed statically at index time, no live IRIS required. Language detection (language.c): .mac/.int/.rtn map to ObjectScript routine directly; .cls (shared with Apex) and .inc (shared with BitBake) are disambiguated by content, defaulting to the existing language on any doubt so neither Apex nor BitBake detection regresses. The two new per-project tables (macros, return types) are threaded through a new internal cbm_extract_file_ex() so the public cbm_extract_file() signature is unchanged. The tree-sitter grammars for ObjectScript UDL and routine are vendored in internal/cbm/vendored/grammars/objectscript_{udl,routine}/ from https://github.com/intersystems/tree-sitter-objectscript (MIT, ABI 15). Refs #462 Signed-off-by: Thomas Dyar <tdyar@intersystems.com> | 2 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
feat(lang): add Pine Script support via vendored kvarenzn/tree-sitter-pine (#273) Adds Pine Script (TradingView's indicator/strategy DSL) as a first-class language using the project's existing language-add pipeline: - scripts/new-languages.json: register pine (display "PineScript", ts_func tree_sitter_pine, repo kvarenzn/tree-sitter-pine, extension .pine, has_scanner=true, module_root source_file) - internal/cbm/vendored/grammars/pine/: vendored from https://github.com/kvarenzn/tree-sitter-pine via scripts/vendor-grammar.sh — parser.c, scanner.c, tree_sitter/ headers. License is ISC (declared in upstream package.json; written into vendored LICENSE for clarity since upstream has no LICENSE file) - internal/cbm/grammar_pine.c: 4-line wrapper generated by scripts/generate-lang-code.py wrappers - internal/cbm/cbm.h: CBM_LANG_PINE before CBM_LANG_COUNT - internal/cbm/lang_specs.c: 7 node-type arrays (func / class / module / call / var / branch / assign) cross-checked against kvarenzn's grammar.js, plus the spec table entry - src/discover/language.c: .pine -> CBM_LANG_PINE extension dispatch and "PineScript" name scripts/audit-grammar-security.sh on the vendored grammar: no dangerous includes (only string/stdint/stdbool/stdlib), no system / popen / exec / socket / mmap / dlopen calls, no constructor/destructor attributes, no inline asm. The 148-line scanner.c only handles INDENT/DEDENT/NEWLINE tokens via the standard tree-sitter lexer API. Distilled from #273 rather than vendored verbatim because the original PR's parser.c was a plain 46k-line drop without provenance, no LICENSE attribution, and bundled an unrelated Makefile change (scripts/embed-frontend.sh -> sh scripts/embed-frontend.sh) that didn't belong to a Pine Script feature. The node-type choices in the lang spec match the original PR's selections, all of which were verified to be real grammar.js rules. Full suite: 2840 passed, 0 failed under ASan + UBSan. Closes #273. Co-authored-by: vinay-veerappa <vinay.veerappa@gmail.com> | 4 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
feat(lang): add Oracle PL/SQL support (distilled from #1033) Vendor AndreasMaierDe/tree-sitter-plsql @ 28aebef209be (MIT, ABI 14, no external scanner) and wire CBM_LANG_PLSQL end to end: - extraction onto existing labels only: package spec/body -> Class, procedure/function (create/definition/declaration) -> Function, file -> Module; callee extraction for ref_call incl. the package-qualified pkg.name form; identifier usage vocabulary with the upstream 'parameter' kind (a ref_call ARGUMENT wrapper) carved out of the whole-binding rule - discover: .pks/.pkb/.pck/.pls/.plb/.plsql/.fnc/.trg/.bdy/.tps/.tpb (.sql stays SQL and .prc stays FORM, both asserted negatively) - ledgers: CALL_WITHOUT_REFERENCE_VOCAB(PLSQL), call-node manifest ref_call row (historical total 219 -> 220), call-argument matrix B row - tests: extraction triple (package body + package-qualified call, standalone function, honest negative pin of the upstream CREATE TYPE ... AS OBJECT ERROR-node limitation), grammar-regression case, label golden Class:1,Function:1,Module:1, MUST_EXTRACT_DEFS, extension mapping, bug-repro board battery row - MANIFEST.md (community-niche verdict, local include-quote patch row), THIRD_PARTY.md section, refreshed vendored checksums PL/SQL support was contributed by Oguz (@ouzsrcm) in #1033; re-vendored from upstream per vendoring policy with his language wiring distilled on top. Co-authored-by: Oğuz <oguzhan.saricam@gmail.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 1 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
feat(lang): add Qt QML language support Vendor the MIT-licensed tree-sitter-qmljs grammar (a TypeScript superset with declarative ui_* nodes) and wire .qml files through the full pipeline: language enum, extension + display-name mapping, grammar shim, and a language spec that reuses the JS/TS function/call/branch arrays and adds QML class/field/import types (ui_inline_component, ui_property, ui_signal, ui_import). QML files now produce Function nodes (incl. JS functions inside QML objects), class/interface/enum nodes, imports and CALLS edges, so call chains through a Qt project's .qml files can be traced. LICENSE vendored alongside the grammar. Closes #42 | 4 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
Add Wolfram Language support (64 languages) + fix caller attribution - Vendor LumaKernel/tree-sitter-wolfram grammar (ABI 13, parser.c + scanner.c) - Register Wolfram language: .wl/.wls extensions, set_delayed_top/set_top/ set_delayed/set as function types, apply as call type, get_top + Needs[] as import types - Add CBM_LANG_WOLFRAM enum, lang spec entry, and grammar bridge - Extract defs: resolve function name from apply(user_symbol("f"), ...) LHS of set/set_delayed definitions; recurse into function bodies for nested defs - Extract calls: skip apply nodes that are definition LHS (not a call) - Extract imports: handle << "file.wl" (get_top) and Needs["Package`"] - Fix caller attribution: add func_kinds_wolfram to func_kinds_for_lang and add Wolfram name extraction in compute_func_qn (extract_unified.c) so the scope stack correctly tracks enclosing function for all call sites - Add MATLAB function_call + command call extraction (63 -> 64 with Wolfram) - Add Lean 4 apply call extraction with type-position filter - Add regression tests for Wolfram, MATLAB, Lean calls + FORM pipeline test | 6 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
Add 89 new tree-sitter grammars (66 to 155 languages) Vendor, wire up, and compile 89 new tree-sitter grammars, expanding language support from 66 to 155 languages. All grammars pass security audit (no dangerous patterns in scanner.c files). New programming languages (31): Solidity, Typst, GDScript, Gleam, PowerShell, Pascal, D, Nim, Scheme, Fennel, Fish, AWK, Zsh, Tcl, Ada, Agda, Racket, Odin, ReScript, PureScript, Nickel, Crystal, Teal, Hare, Pony, Luau, Janet, Sway, NASM, Assembly, TLA+, Pkl, Cairo, Move, Squirrel, ISPC, FunC, Smali New config/data/IDL formats (31): Just, Astro, Blade, Go Template, Templ, Liquid, Jinja2, Prisma, Hyprlang, DotEnv, Diff, WGSL, KDL, JSON5, Jsonnet, RON, Thrift, Cap'n Proto, Properties, SSH Config, BibTeX, Starlark, Bicep, CSV, Requirements, HLSL, VHDL, SystemVerilog, DeviceTree, Linker Script, GN, Kconfig, BitBake, TableGen, Slang, LLVM IR, Smithy, WIT, Go Mod, Mermaid, RST, Beancount, Puppet, PO, Regex, JSDoc, gitattributes, gitignore, Apex, SOQL, SOSL Infrastructure: - scripts/new-languages.json: manifest for all new languages - scripts/generate-lang-code.py: generates boilerplate from manifest - scripts/audit-grammar-security.sh: pre-vendoring security scanner - Fixed angle-bracket includes in 18 grammars - Fixed PureScript scanner const mismatch - Fixed VHDL scanner void* API signatures - Fixed RST tree_sitter_rst/ subdirectory include paths - Copied crystal unicode.c (extra scanner dependency) All new languages start with minimal lang specs (module_types only). Function/class/call extraction specs to be refined incrementally. | 5 个月前 | |
fix(extract): AST walker stacks grow instead of silently capping content Three iterative AST walkers kept their pending nodes in fixed-size stacks and silently stopped pushing children once full: - cbm_compute_complexity and count_branching_iter (helpers.c, BRANCHING_STACK_CAP 4096) -> cyclomatic, cognitive, loop and access metrics - extract_body_ident_tokens (extract_defs.c, BT_STACK 512) -> the body-token sample that feeds semantic search / SEMANTICALLY_RELATED Children are pushed last-to-first, so a full stack dropped a wide body's FIRST children. The metrics undercounted, and the token sampler read a mid-body window instead of the first identifiers. That let a work cap decide graph content. Real input: tree-sitter-llvm's generated ts_lex (a 4654-case switch) reported cyclomatic 11133 / cognitive 29303. It really has 8776 if + 4654 case + 1 default + 1 switch = 13432 branches, which is what it reports now (cognitive 35639). On the Linux kernel no function reaches the old 4096 cap, so complexity is byte-identical for every function. Three functions with bodies over 512 pending nodes now sample their tokens from the start of the body (for example rimob_panel_on), which moves 11 SEMANTICALLY_RELATED edges. Index time is unchanged within noise (main 214.4/207.9 s, fix 197.6/192.2 s). Fix: cbm_walk_stack_reserve()/cbm_walk_stack_release() in helpers.c. Each walker keeps a small on-stack first chunk (512 frames) and spills to the heap through the memory core (cbm_alloc/cbm_realloc/cbm_free, class EXTRACT) only for deep or wide trees. Traversal order is unchanged. If the stack cannot grow, the failure is logged ("extract.walk_stack_alloc_failed") and the walker returns a documented sentinel instead of a truncated result: CBM_WALK_METRIC_UNAVAILABLE (-1) for the metrics, or no token sample. The body-token sampler's caps of 128 identifiers and a 2048-byte buffer are its sampling contract and stay. The 512-entry stack was never part of that contract, because it broke the "first identifiers in source order" property. Tests (extraction suite): a C function with 5000 top-level if/while statements must report exact cyclomatic/cognitive/loop counts through both the extractor and cbm_count_branching, and a 600-call body must sample from f0 on. All three are RED on main (4095 / 4095 / sample starting at f91) and GREEN with the fix. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 8 天前 | |
fix(extract): AST walker stacks grow instead of silently capping content Three iterative AST walkers kept their pending nodes in fixed-size stacks and silently stopped pushing children once full: - cbm_compute_complexity and count_branching_iter (helpers.c, BRANCHING_STACK_CAP 4096) -> cyclomatic, cognitive, loop and access metrics - extract_body_ident_tokens (extract_defs.c, BT_STACK 512) -> the body-token sample that feeds semantic search / SEMANTICALLY_RELATED Children are pushed last-to-first, so a full stack dropped a wide body's FIRST children. The metrics undercounted, and the token sampler read a mid-body window instead of the first identifiers. That let a work cap decide graph content. Real input: tree-sitter-llvm's generated ts_lex (a 4654-case switch) reported cyclomatic 11133 / cognitive 29303. It really has 8776 if + 4654 case + 1 default + 1 switch = 13432 branches, which is what it reports now (cognitive 35639). On the Linux kernel no function reaches the old 4096 cap, so complexity is byte-identical for every function. Three functions with bodies over 512 pending nodes now sample their tokens from the start of the body (for example rimob_panel_on), which moves 11 SEMANTICALLY_RELATED edges. Index time is unchanged within noise (main 214.4/207.9 s, fix 197.6/192.2 s). Fix: cbm_walk_stack_reserve()/cbm_walk_stack_release() in helpers.c. Each walker keeps a small on-stack first chunk (512 frames) and spills to the heap through the memory core (cbm_alloc/cbm_realloc/cbm_free, class EXTRACT) only for deep or wide trees. Traversal order is unchanged. If the stack cannot grow, the failure is logged ("extract.walk_stack_alloc_failed") and the walker returns a documented sentinel instead of a truncated result: CBM_WALK_METRIC_UNAVAILABLE (-1) for the metrics, or no token sample. The body-token sampler's caps of 128 identifiers and a 2048-byte buffer are its sampling contract and stay. The 512-entry stack was never part of that contract, because it broke the "first identifiers in source order" property. Tests (extraction suite): a C function with 5000 top-level if/while statements must report exact cyclomatic/cognitive/loop counts through both the extractor and cbm_count_branching, and a 600-call body must sample from f0 on. All three are RED on main (4095 / 4095 / sample starting at f91) and GREEN with the fix. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 8 天前 | |
feat(objectscript): add InterSystems IRIS ObjectScript language support Add ObjectScript (InterSystems IRIS / Caché) as a supported language, covering the UDL class format (.cls), MAC/INT routines (.mac/.int/.rtn), include/macro files (.inc), and IRIS Studio Export XML. Definition extraction (extract_defs.c): Class, Method, ClassMethod, Property, Parameter, Index, Trigger (with body text), XData, Storage, and Query members as graph nodes; base classes from the Extends clause. Call dispatch resolution (extract_calls.c) — four ObjectScript patterns that are structurally invisible to text search: 1. ##class(Pkg.Class).Method() explicit cross-class call 2. ..Method() relative-dot self-call (the dominant intra-class form; large impact on CALLS completeness) 3. $$$Macro macro expansion via a per-project table built from .inc files 4. type inference from %New/%OpenId + declared return types Ensemble production topology (pass_ensemble_routing.c): EnsembleItem nodes per production component and ROUTES_TO edges resolved from ProductionDefinition XData, plus WorkMgr .Queue("##class(X).method") dispatch — all parsed statically at index time, no live IRIS required. Language detection (language.c): .mac/.int/.rtn map to ObjectScript routine directly; .cls (shared with Apex) and .inc (shared with BitBake) are disambiguated by content, defaulting to the existing language on any doubt so neither Apex nor BitBake detection regresses. The two new per-project tables (macros, return types) are threaded through a new internal cbm_extract_file_ex() so the public cbm_extract_file() signature is unchanged. The tree-sitter grammars for ObjectScript UDL and routine are vendored in internal/cbm/vendored/grammars/objectscript_{udl,routine}/ from https://github.com/intersystems/tree-sitter-objectscript (MIT, ABI 15). Refs #462 Signed-off-by: Thomas Dyar <tdyar@intersystems.com> | 2 个月前 | |
feat(objectscript): add InterSystems IRIS ObjectScript language support Add ObjectScript (InterSystems IRIS / Caché) as a supported language, covering the UDL class format (.cls), MAC/INT routines (.mac/.int/.rtn), include/macro files (.inc), and IRIS Studio Export XML. Definition extraction (extract_defs.c): Class, Method, ClassMethod, Property, Parameter, Index, Trigger (with body text), XData, Storage, and Query members as graph nodes; base classes from the Extends clause. Call dispatch resolution (extract_calls.c) — four ObjectScript patterns that are structurally invisible to text search: 1. ##class(Pkg.Class).Method() explicit cross-class call 2. ..Method() relative-dot self-call (the dominant intra-class form; large impact on CALLS completeness) 3. $$$Macro macro expansion via a per-project table built from .inc files 4. type inference from %New/%OpenId + declared return types Ensemble production topology (pass_ensemble_routing.c): EnsembleItem nodes per production component and ROUTES_TO edges resolved from ProductionDefinition XData, plus WorkMgr .Queue("##class(X).method") dispatch — all parsed statically at index time, no live IRIS required. Language detection (language.c): .mac/.int/.rtn map to ObjectScript routine directly; .cls (shared with Apex) and .inc (shared with BitBake) are disambiguated by content, defaulting to the existing language on any doubt so neither Apex nor BitBake detection regresses. The two new per-project tables (macros, return types) are threaded through a new internal cbm_extract_file_ex() so the public cbm_extract_file() signature is unchanged. The tree-sitter grammars for ObjectScript UDL and routine are vendored in internal/cbm/vendored/grammars/objectscript_{udl,routine}/ from https://github.com/intersystems/tree-sitter-objectscript (MIT, ABI 15). Refs #462 Signed-off-by: Thomas Dyar <tdyar@intersystems.com> | 2 个月前 | |
fix(dart): index extension declarations and their members (#1457) The Dart language spec listed class_definition, enum_declaration, mixin_declaration and type_alias as container kinds, but not extension_declaration. The def walk never treated `extension X on Y { }` as a container, and extract_func_def could not name the bare `method_signature` wrappers in the extension body. So the extension and every member were dropped: only the Module, File, the extended class and top-level functions made it into the graph. - lang_specs.c: add extension_declaration to dart_class_types. A named extension becomes a Class node, and its members become Methods with DEFINES_METHOD edges from it. In-body calls get the method as caller. - extract_defs.c: an unnamed `extension on T { }` has no name to attach its members to. extract_class_def would return before extract_class_methods and drop them. Walk such an extension's members as file-level Functions instead, unwrapping method_signature to its named function_signature. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 10 天前 | |
fix(extract): string-dispatch calls in dead code path, never linked to target Python calls of the form iris_obj.classMethodValue('Pkg.Class', 'Method') emitted no CALLS edge. The detection existed in walk_calls, which is dead code — the pipeline uses cbm_extract_unified → handle_calls, not walk_calls. Move the IRIS Native API string-dispatch detection to handle_calls. When a Python call's callee ends in .classMethodValue, .classMethodVoid, .classMethodBoolean, or .classMethodObject, and both string-literal arguments can be extracted, emit a synthetic CALLS edge to Arg1.Arg2. The pattern generalizes beyond this specific API: any library that takes class/method names as string literals follows the same shape. Regression test: python_iris_classMethodValue in test_extraction.c. Signed-off-by: Thomas Dyar <tdyar@intersystems.com> | 3 个月前 | |
feat(perl-lsp): wire dispatch, unity build, and Makefile - cbm.c: include lsp/perl_lsp.h and dispatch CBM_LANG_PERL -> cbm_run_perl_lsp - lsp_all.c: unity-include perl_lsp.c + generated/perl_stdlib_data.c - Makefile.cbm: register TEST_PERL_LSP_SRCS and append to ALL_TEST_SRCS - tests/test_perl_lsp.c: placeholder suite so the Makefile var resolves (full suite + test_main.c registration land in plan 22-04) Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com> | 2 个月前 | |
WIP: strict linting + RAM-first pipeline (lint fixes pending) | 6 个月前 | |
Add C-native pipeline with blocking clang-tidy, cppcheck, clang-format linting Complete C implementation of the indexing pipeline (src/), parallel worker pool, graph buffer with merge support, SQLite writer, and 1893-test suite. Linter setup: zero warnings from clang-tidy (all checks enabled, no NOLINT suppressions), cppcheck, and clang-format. All issues fixed at source — proper headers for external linkage, GROW_ARRAY macro restructured to eliminate type parameter, null-deref paths guarded, named intermediates for suspicious-argument checks. | 6 个月前 | |
feat(objectscript): add InterSystems IRIS ObjectScript language support Add ObjectScript (InterSystems IRIS / Caché) as a supported language, covering the UDL class format (.cls), MAC/INT routines (.mac/.int/.rtn), include/macro files (.inc), and IRIS Studio Export XML. Definition extraction (extract_defs.c): Class, Method, ClassMethod, Property, Parameter, Index, Trigger (with body text), XData, Storage, and Query members as graph nodes; base classes from the Extends clause. Call dispatch resolution (extract_calls.c) — four ObjectScript patterns that are structurally invisible to text search: 1. ##class(Pkg.Class).Method() explicit cross-class call 2. ..Method() relative-dot self-call (the dominant intra-class form; large impact on CALLS completeness) 3. $$$Macro macro expansion via a per-project table built from .inc files 4. type inference from %New/%OpenId + declared return types Ensemble production topology (pass_ensemble_routing.c): EnsembleItem nodes per production component and ROUTES_TO edges resolved from ProductionDefinition XData, plus WorkMgr .Queue("##class(X).method") dispatch — all parsed statically at index time, no live IRIS required. Language detection (language.c): .mac/.int/.rtn map to ObjectScript routine directly; .cls (shared with Apex) and .inc (shared with BitBake) are disambiguated by content, defaulting to the existing language on any doubt so neither Apex nor BitBake detection regresses. The two new per-project tables (macros, return types) are threaded through a new internal cbm_extract_file_ex() so the public cbm_extract_file() signature is unchanged. The tree-sitter grammars for ObjectScript UDL and routine are vendored in internal/cbm/vendored/grammars/objectscript_{udl,routine}/ from https://github.com/intersystems/tree-sitter-objectscript (MIT, ABI 15). Refs #462 Signed-off-by: Thomas Dyar <tdyar@intersystems.com> | 2 个月前 | |
feat(objectscript): add InterSystems IRIS ObjectScript language support Add ObjectScript (InterSystems IRIS / Caché) as a supported language, covering the UDL class format (.cls), MAC/INT routines (.mac/.int/.rtn), include/macro files (.inc), and IRIS Studio Export XML. Definition extraction (extract_defs.c): Class, Method, ClassMethod, Property, Parameter, Index, Trigger (with body text), XData, Storage, and Query members as graph nodes; base classes from the Extends clause. Call dispatch resolution (extract_calls.c) — four ObjectScript patterns that are structurally invisible to text search: 1. ##class(Pkg.Class).Method() explicit cross-class call 2. ..Method() relative-dot self-call (the dominant intra-class form; large impact on CALLS completeness) 3. $$$Macro macro expansion via a per-project table built from .inc files 4. type inference from %New/%OpenId + declared return types Ensemble production topology (pass_ensemble_routing.c): EnsembleItem nodes per production component and ROUTES_TO edges resolved from ProductionDefinition XData, plus WorkMgr .Queue("##class(X).method") dispatch — all parsed statically at index time, no live IRIS required. Language detection (language.c): .mac/.int/.rtn map to ObjectScript routine directly; .cls (shared with Apex) and .inc (shared with BitBake) are disambiguated by content, defaulting to the existing language on any doubt so neither Apex nor BitBake detection regresses. The two new per-project tables (macros, return types) are threaded through a new internal cbm_extract_file_ex() so the public cbm_extract_file() signature is unchanged. The tree-sitter grammars for ObjectScript UDL and routine are vendored in internal/cbm/vendored/grammars/objectscript_{udl,routine}/ from https://github.com/intersystems/tree-sitter-objectscript (MIT, ABI 15). Refs #462 Signed-off-by: Thomas Dyar <tdyar@intersystems.com> | 2 个月前 | |
fix(parsing): recover export-macro-hidden type definitions (#1989) Collect a bounded set of conventional export macro candidates and inject empty definitions into the existing C/C++ preprocessing second pass without overriding explicit caller definitions. Conservatively reconcile remapped definitions to recover hidden classes, structs, enums, free functions, and inline methods while suppressing matching base-class phantom callables. Add focused regression coverage for supported suffixes, ordinary all-caps negative controls, candidate limits, comments, strings, raw strings, overlong names, explicit define priority, and C/C++ extraction. Rebase onto main 92abefa3 and route the temporary superseded-definition bitmap through the extraction memory class to comply with the current allocation contract. Preserve the error-region dropped counter initialization. Signed-off-by: XIYBHK <xiybhk@163.com> | 15 天前 | |
fix(parsing): recover export-macro-hidden type definitions (#1989) Collect a bounded set of conventional export macro candidates and inject empty definitions into the existing C/C++ preprocessing second pass without overriding explicit caller definitions. Conservatively reconcile remapped definitions to recover hidden classes, structs, enums, free functions, and inline methods while suppressing matching base-class phantom callables. Add focused regression coverage for supported suffixes, ordinary all-caps negative controls, candidate limits, comments, strings, raw strings, overlong names, explicit define priority, and C/C++ extraction. Rebase onto main 92abefa3 and route the temporary superseded-definition bitmap through the extraction memory class to comply with the current allocation contract. Preserve the error-region dropped counter initialization. Signed-off-by: XIYBHK <xiybhk@163.com> | 15 天前 | |
Merge pull request #2459 from DeusData/fix/doc-comment-capture fix(extract): capture complete doc comments across languages | 1 天前 | |
test(spill): make spill test verdicts independent of host free disk extraction::extract_spill_round_trip_keeps_every_field and parallel::parallel_spill_mode_builds_the_same_graph failed on any machine with less than 10 GB free: cbm_result_spill_open refuses to spill onto a nearly full disk (mem.spill.refused reason=low_disk), so the store never opened. Three pipeline tests took the same refused-store path there and passed without spilling at all (backpressure_futile_nap_disengages, over_budget_after_futility_fails_whole_and_preserves_db, spill_resolves_namespace_imports_like_memory). A verdict must be a pure function of code, test, platform and seed, not of how full the disk is. Add a free-space pin to result_spill that exists only in test-seam builds (CBM_ENABLE_TEST_SEAMS). Production still asks the disk, and its 10 GB floor and half-of-free cap are unchanged. The test runner pins 64 GiB at the top of main, before worker re-exec dispatch, so every suite and re-exec'd index worker sees the same disk. A new test, extraction::extract_spill_refuses_below_the_free_disk_floor, pins the guard itself: one byte under the floor refuses, the floor opens. Verified with cbm_fs_free_bytes temporarily forced to 9 GB: both tests RED 3/3 before, green 3/3 after, RED again with the runner pin removed; the refusal test goes RED with the production guard disabled. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 8 天前 | |
test(spill): make spill test verdicts independent of host free disk extraction::extract_spill_round_trip_keeps_every_field and parallel::parallel_spill_mode_builds_the_same_graph failed on any machine with less than 10 GB free: cbm_result_spill_open refuses to spill onto a nearly full disk (mem.spill.refused reason=low_disk), so the store never opened. Three pipeline tests took the same refused-store path there and passed without spilling at all (backpressure_futile_nap_disengages, over_budget_after_futility_fails_whole_and_preserves_db, spill_resolves_namespace_imports_like_memory). A verdict must be a pure function of code, test, platform and seed, not of how full the disk is. Add a free-space pin to result_spill that exists only in test-seam builds (CBM_ENABLE_TEST_SEAMS). Production still asks the disk, and its 10 GB floor and half-of-free cap are unchanged. The test runner pins 64 GiB at the top of main, before worker re-exec dispatch, so every suite and re-exec'd index worker sees the same disk. A new test, extraction::extract_spill_refuses_below_the_free_disk_floor, pins the guard itself: one byte under the floor refuses, the floor opens. Verified with cbm_fs_free_bytes temporarily forced to 9 GB: both tests RED 3/3 before, green 3/3 after, RED again with the runner pin removed; the refusal test goes RED with the production guard disabled. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 8 天前 | |
Merge pull request #2255 from DeusData/distill/1245-route-literal-exts fix(routes): a document or log path is not an HTTP route literal | 1 天前 | |
fix(pipeline): key GraphQL Routes by operation name, not operation text (#598) When a call resolved to a GraphQL client, emit_graphql_edge named the Route after the whole first string argument, stripping only a leading "query " or "mutation " (the mutation strip was also one byte short, so names kept a leading space). An anonymous `query { ... }` therefore became one long Route, and even a named operation carried its variables and selection set in the name ("GetUser($id: ID!) { user(id: $id) { name } }"). Every distinct operation text minted its own Route node, and names were cut off at 256 bytes. Add cbm_service_pattern_graphql_operation(), a small GraphQL document scanner. It skips comments, strings, block strings, `${...}` template interpolations and leading fragment definitions, and reports the first operation's type and name without copying any operation text. Routes are now keyed as follows: named operation __graphql__<Name> name <Name> anonymous query / mutation / __graphql__<type>__anonymous name (anonymous <type>) subscription, incl. the `{ ... }` query shorthand no GraphQL document __graphql__operation__anonymous name (anonymous operation) (URL, bare word, no string argument) Keys are deterministic and bounded by the GraphQL Name grammar. The GRAPHQL_CALLS edge keeps "operation" (now the Route name) and gains "operation_type". The old fallback to the callee name ("client.query") is gone: a callee is not an operation. The sequential path (pass_calls.c) emits no GRAPHQL_CALLS at all, so it had no copy of this logic to fix. Proof on graphql-python/gql (374a70df, Python GraphQL client, index-only): GraphQL Routes 170 -> 16, longest QN 266 -> 34 bytes, 106 -> 0 Routes carrying operation text. GRAPHQL_CALLS 2643 -> 1158 with the same 984 distinct callers: edges are one per (caller, operation) pair, so calls that used to hit distinct text-keyed Routes now share their operation's Route. Tests (edge_types_probe): graphql_route_keyed_by_operation_name_issue598 asserts the exact GraphQL Route set (QN + name) on the parallel path. graphql_operation_identity_parser_issue598 covers braces inside comments, strings and interpolations, fragments before the operation, anonymous operations with variables, name truncation and non-document arguments. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 10 天前 | |
fix(sql): keep literal INSERT rows out of the SQL parse (#1735) A mysqldump-style .sql file is a few CREATE TABLEs followed by megabytes of `INSERT ... VALUES (..),(..),...` rows. Tree-sitter built a full tree for every row (~70 bytes of tree per source byte; 0.26 s/MB with MySQL \' escapes, which the grammar does not know and error-recovers from) until the 5 s parse budget ran out, and the whole file, tables included, was skipped as "parse timeout". The reporter's database/sql/world.sql was dropped this way. The rows contribute nothing to the graph. A linear, quote- and comment-aware pre-scan (internal/cbm/sql_values.c) finds INSERT/REPLACE ... VALUES lists and excludes every tuple after the first that consists only of literals: strings (standard '' and MySQL \' escapes, X'..'/B'..'/N'.., _charset introducers), numbers, NULL, TRUE, FALSE, DEFAULT. A tuple holding a subquery, a function call, an identifier or a quoted identifier stays in the parse, and so does one whose string spans a raw newline. The excluded bytes go to ts_parser_set_included_ranges as the complement, so byte offsets and line/column positions of all kept text are unchanged. The ranges are cleared right after the parse because the parser is thread-local and reused. This is a syntax rule, not a work cap: the same file always gives the same ranges, and the parse budget is unchanged. Measured on generated mysqldump-shaped dumps, release build, same machine, index worker's own numbers (base = origin/main): dump extraction pipeline peak RSS graph 8 MB std '' 1310 -> 23 ms 1431 -> 139 ms 625 -> 37 MB identical 8 MB MySQL \' 2712 -> 24 ms 2827 -> 146 ms 474 -> 37 MB identical 32 MB MySQL \' 5145 -> 82 ms 5467 -> 396 ms 1132 -> 62 MB timeout -> indexed 59 MB MySQL \' 5130 -> 138 ms 5667 -> 674 ms 1157 -> 89 MB timeout -> indexed "identical" = every stored node and edge, Module end line included. For the 32/59 MB dumps the base run skipped the file as "parse timeout" (Module + 3 Tables + their DEFINES/USAGE missing). The fixed graph matches the 8 MB graph except for the Module's end line. With standard escapes the extraction and the stored graph are identical to the full parse. With MySQL \' escapes the full parse's error recovery swallows rows next to a misread escape, a subquery row among them, and the cut parse keeps those rows. So it loses nothing the full parse found outside the dropped rows and recovers what the full parse lost. The only things the full parse extracted INSIDE the dropped rows were junk usages (DEFAULT and _binary read as identifiers, fragments of misread strings) that name nothing. Tests (parse_coverage, pipeline; a CBM_TEST_SQL_FULL_PARSE_ON seam turns the exclusion off for one file so the same source can be compared both ways): - scanner cases: comments, strings, dollar quotes, ON DUPLICATE KEY UPDATE, VALUES outside INSERT, unclosed tuples, exact kept positions - extraction equals the full parse on a small dump (standard escapes); nothing lost outside dropped rows (MySQL escapes) - the subquery and function-call rows are still parsed - the tree does not grow when the row count doubles (a count, not a clock) - a 25 MB MySQL dump under the production budget is indexed, not "parse timeout" (asserts the outcome, not the time) - stored nodes and edges of a pipeline index equal the full parse's Refs #1735 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 10 天前 | |
fix(sql): keep literal INSERT rows out of the SQL parse (#1735) A mysqldump-style .sql file is a few CREATE TABLEs followed by megabytes of `INSERT ... VALUES (..),(..),...` rows. Tree-sitter built a full tree for every row (~70 bytes of tree per source byte; 0.26 s/MB with MySQL \' escapes, which the grammar does not know and error-recovers from) until the 5 s parse budget ran out, and the whole file, tables included, was skipped as "parse timeout". The reporter's database/sql/world.sql was dropped this way. The rows contribute nothing to the graph. A linear, quote- and comment-aware pre-scan (internal/cbm/sql_values.c) finds INSERT/REPLACE ... VALUES lists and excludes every tuple after the first that consists only of literals: strings (standard '' and MySQL \' escapes, X'..'/B'..'/N'.., _charset introducers), numbers, NULL, TRUE, FALSE, DEFAULT. A tuple holding a subquery, a function call, an identifier or a quoted identifier stays in the parse, and so does one whose string spans a raw newline. The excluded bytes go to ts_parser_set_included_ranges as the complement, so byte offsets and line/column positions of all kept text are unchanged. The ranges are cleared right after the parse because the parser is thread-local and reused. This is a syntax rule, not a work cap: the same file always gives the same ranges, and the parse budget is unchanged. Measured on generated mysqldump-shaped dumps, release build, same machine, index worker's own numbers (base = origin/main): dump extraction pipeline peak RSS graph 8 MB std '' 1310 -> 23 ms 1431 -> 139 ms 625 -> 37 MB identical 8 MB MySQL \' 2712 -> 24 ms 2827 -> 146 ms 474 -> 37 MB identical 32 MB MySQL \' 5145 -> 82 ms 5467 -> 396 ms 1132 -> 62 MB timeout -> indexed 59 MB MySQL \' 5130 -> 138 ms 5667 -> 674 ms 1157 -> 89 MB timeout -> indexed "identical" = every stored node and edge, Module end line included. For the 32/59 MB dumps the base run skipped the file as "parse timeout" (Module + 3 Tables + their DEFINES/USAGE missing). The fixed graph matches the 8 MB graph except for the Module's end line. With standard escapes the extraction and the stored graph are identical to the full parse. With MySQL \' escapes the full parse's error recovery swallows rows next to a misread escape, a subquery row among them, and the cut parse keeps those rows. So it loses nothing the full parse found outside the dropped rows and recovers what the full parse lost. The only things the full parse extracted INSIDE the dropped rows were junk usages (DEFAULT and _binary read as identifiers, fragments of misread strings) that name nothing. Tests (parse_coverage, pipeline; a CBM_TEST_SQL_FULL_PARSE_ON seam turns the exclusion off for one file so the same source can be compared both ways): - scanner cases: comments, strings, dollar quotes, ON DUPLICATE KEY UPDATE, VALUES outside INSERT, unclosed tuples, exact kept positions - extraction equals the full parse on a small dump (standard escapes); nothing lost outside dropped rows (MySQL escapes) - the subquery and function-call rows are still parsed - the tree does not grow when the row count doubles (a count, not a clock) - a 25 MB MySQL dump under the production budget is indexed, not "parse timeout" (asserts the outcome, not the time) - stored nodes and edges of a pipeline index equal the full parse's Refs #1735 Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 10 天前 | |
fix(sqlite-writer): never end a B-tree level with a zero-cell interior page (#1783) The bulk writer builds each interior level greedily: it fills a page with separator cells until the next one no longer fits, uses that child as the page's right child, and starts the next page after it. When a page filled up exactly at children[child_count - 2], only the last child was left, so the next (and last) page of that level was written with ZERO cells and just a right-child pointer. SQLite rejects a non-root page with nCell < 1 as SQLITE_CORRUPT the moment a cursor descends into it (getAndInitPage). Because that page is always the last one of its level, the failing descent is the rightmost path -- exactly what the coverage-shadow wipe takes: "DELETE ... WHERE project = ?1" with "<project>::missed", a key that sorts after every row of "<project>". The publish then failed with "coverage_shadow_wipe: database disk image is malformed". It needs an index with more than one interior page on a level (large graphs or long keys) AND the page-fill boundary landing on the one bad position, which is why it looked scale-dependent and repo-specific; the same DBs also fail PRAGMA integrity_check. When a page fills with exactly one child left, give its last cell back, so the next page holds one cell plus the last child as its right child. The removed bytes are zeroed, keeping the output deterministic. Test: sqlite_writer::sw_interior_level_never_ends_with_empty_page writes 2..120 rows with 12000-byte names (one and two interior levels) and runs the wipe-shaped DELETE through idx_nodes_name; rows 31-35, 61-65 and 91-95 failed with SQLITE_CORRUPT before the fix. Real input: 41 generated markdown repos with long headings indexed by v0.11.0 -> 5 databases fail the "::missed" seek and integrity_check; with the fix 0 of 41. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 10 天前 | |
fix(graph): IMPORTS edges keyed by local_name across buffer, store and dump Distills PR #770 (graph-buffer dedup key) and completes the edge-uniqueness contract it left partial: the contract lives in THREE places, and changing only the buffer ships databases that violate their own UNIQUE constraint (PRAGMA integrity_check: non-unique entry in sqlite_autoindex_edges_1). A single 'import { A, B } from ./lib' produced ONE IMPORTS edge: the graph buffer dedups edges on (source_id, target_id, type) and merge-replaces properties on collision, so the second symbol silently overwrote the first. pass_calls.c, pass_usages.c, pass_semantic.c and pass_lsp_cross.c parse one local_name per IMPORTS edge for cross-file resolution, so the dropped symbol's calls failed to resolve too — not just "who imports X" queries. Changes, kept in sync across all three sites: - graph_buffer.c (from #770): make_edge_key() folds local_name into the dedup key for IMPORTS edges only; all three key call-sites updated. Hardened beyond #770: EDGE_KEY_BUF bumped to 256 and oversized local_names are re-keyed with an FNV-1a hash of the full name instead of being silently truncated (two long names sharing a prefix must not collide back into one edge). - store.c: edges gains local_name_gen, a VIRTUAL generated column (IMPORTS -> coalesce(json_extract(properties,'$.local_name'),''), else '' — NOT NULL because NULLs never conflict in a UNIQUE index); uniqueness widened to UNIQUE(source_id, target_id, type, local_name_gen) and the insert upsert's conflict target matches. init_schema probes pre-#768 DBs (no local_name_gen) and fails the open: SQLite cannot ALTER a table constraint in place, and an unopenable DB already takes the existing repair path — full index deletes + rebuilds, artifact import refuses and falls back to a reindex. Read-only query opens skip init_schema and keep working. - sqlite_writer.c/.h: dump DDL matches the widened schema; the hand-built sqlite_autoindex_edges_1 comparator and entry builder include the local_name column; CBMDumpEdge carries local_name extracted via real JSON parsing (yyjson) so index entries match json_extract's unescaped values exactly, per the idx_edges_url_path precedent. - artifact.h: CBM_ARTIFACT_SCHEMA_VERSION 1 -> 2 so old binaries refuse artifacts carrying the widened schema (their 3-column conflict target can no longer prepare against it). Tests (reproduce-first, all red on the unfixed code): - gbuf tier: multi-symbol import -> 2 IMPORTS edges; long-local_name truncation guard. - store tier: distinct local_name coexists as 2 rows, same local_name still upserts, non-IMPORTS dedup unchanged. - writer tier: dumped DB passes integrity_check with 2 sibling imports and exposes matching local_name_gen values. - end-to-end: TS fixture through the real pipeline -> 2 queryable IMPORTS edges AND integrity_check ok; with only the buffer half of the fix this test still fails on integrity_check, proving the schema half is required. Closes #768. Co-authored-by: Alexandros Pappas <11921291+apappas1129@users.noreply.github.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 3 个月前 | |
v0.5: CBM tree-sitter engine, 59 vendored language grammars - Replace Go-binding tree-sitter with vendored C grammars compiled via CGo (internal/cbm/); eliminates all Go module/linker fragility - Add 32 new languages: Clojure, CMake, COBOL, Common Lisp, CUDA, Elm, Emacs Lisp, Fortran, F#, GLSL, GraphQL, INI, JSON, Julia, Makefile, Markdown, Meson, Nix, Protobuf, Svelte, Verilog, Vim Script, Vue, XML (plus C, C#, Erlang, SQL, YAML quality improvements) - Adaptive worker concurrency and mmap prefetch for faster indexing - Graph buffer: batched SQLite writes for 3-5x throughput improvement - Fix cancellation test: use 50ms timeout (pipeline now indexes 212-file Erlang repo in ~400ms, well under the old 2s deadline) - Fix isTestFunction: add "Test" prefix for C#, "test" prefix for Scala | 6 个月前 | |
fix: size the compress path with size_t and prune the cache from the walk Two loose ends from the bounds work. The decompression side already took size_t so a >2 GiB capacity could not wrap through int, but compression still took int for both the source length and the destination capacity, and the artifact export cast a size_t database size down to reach it. A database past 2 GiB would have handed the encoder a negative length. Both lengths and the bound helper are size_t now, and the function returns int64_t like its decompressing counterpart. The discovery walk now prunes the cache directory by absolute path. A custom CBM_CACHE_DIR may sit inside a repository — tests do it routinely — and walking into it pulls every other project's graph database into this project's file list. This is the narrow form of a concern that was briefly implemented as refusing any root that contained the cache; refusing a whole root was too blunt, and not walking the cache is what the concern actually asks for. The test fails without the prune: a .go file planted under the cache is otherwise discovered, and "cache" is not in the built-in skip list, so the assertion is not vacuous. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 1 个月前 | |
fix: size the compress path with size_t and prune the cache from the walk Two loose ends from the bounds work. The decompression side already took size_t so a >2 GiB capacity could not wrap through int, but compression still took int for both the source length and the destination capacity, and the artifact export cast a size_t database size down to reach it. A database past 2 GiB would have handed the encoder a negative length. Both lengths and the bound helper are size_t now, and the function returns int64_t like its decompressing counterpart. The discovery walk now prunes the cache directory by absolute path. A custom CBM_CACHE_DIR may sit inside a repository — tests do it routinely — and walking into it pulls every other project's graph database into this project's file list. This is the narrow form of a concern that was briefly implemented as refusing any root that contained the cache; refusing a whole root was too blunt, and not walking the cache is what the concern actually asks for. The test fails without the prune: a .go file planted under the cache is otherwise discovered, and "cache" is not in the built-in skip list, so the assertion is not vacuous. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com> | 1 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 1 天前 | ||
| 4 天前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 4 个月前 | ||
| 22 天前 | ||
| 5 天前 | ||
| 5 天前 | ||
| 3 天前 | ||
| 1 天前 | ||
| 5 天前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 1 天前 | ||
| 1 个月前 | ||
| 2 天前 | ||
| 3 个月前 | ||
| 18 天前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 16 天前 | ||
| 1 个月前 | ||
| 10 天前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 1 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 4 个月前 | ||
| 4 个月前 | ||
| 1 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 2 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 3 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 2 个月前 | ||
| 2 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 4 个月前 | ||
| 5 个月前 | ||
| 1 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 4 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 8 天前 | ||
| 8 天前 | ||
| 2 个月前 | ||
| 2 个月前 | ||
| 10 天前 | ||
| 3 个月前 | ||
| 2 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 2 个月前 | ||
| 2 个月前 | ||
| 15 天前 | ||
| 15 天前 | ||
| 1 天前 | ||
| 8 天前 | ||
| 8 天前 | ||
| 1 天前 | ||
| 10 天前 | ||
| 10 天前 | ||
| 10 天前 | ||
| 10 天前 | ||
| 3 个月前 | ||
| 6 个月前 | ||
| 1 个月前 | ||
| 1 个月前 |