| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
chore: optimizes slow tests in CI/CD (#6029) ## Results The PR reduced unit-test time (3.13, native, 22.0.0, ubuntu-latest, false) from [33m 18s](https://github.com/Eventual-Inc/Daft/actions/runs/20981828603/job/60308365885?pr=6023) to 13m 44s — saving about 20 mins which is nice. The test_context is still slow, but it has very few tests, a big win is parallelization and taking >5 mins off just by skipping the setup for skipped tests. This also reduced the ray tests from 45 mins to 27 mins, and there should be plenty more savings here. Then disabling windows eliminated a 50+ min action. text ================================================================================ OVERALL SUMMARY ================================================================================ Tests Analyzed: 9620 Total Time: 941.32s Average Time: 0.05s Max Time: 14.70s Min Time: 0.01s It took 188s to run make test on my mac which is nice; it also crushed my machine while it was running 🥲 ## Changes Made * Adds -n auto to pytest for parallelize testing on multiple cores * Skips generating fixtures (expensive?) by not actually creating the tests - we were creating test data _then_ skipping for several thousand join tests taking ~5 mins of setup work just to skip. * Memoizes data generation for the window tests (often the slowest for me) * Reduces the data generation size and faster comparison algo (before was 10,000 data points with a O(n^2) algo, now it's 1,000 data points with an O(nlogn) comparison algo) * Disables the RAY_DASHBOARD for the context setup commands, and other minor things to speedup ray init * Skips windows on PRs (>50 minutes) only enabled on main, worthwhile trade-off because we will still catch windows-specific failures when merging to main which is quite rare to be fair. ## Related Issues N/A | 8 个月前 | |
chore: optimizes slow tests in CI/CD (#6029) ## Results The PR reduced unit-test time (3.13, native, 22.0.0, ubuntu-latest, false) from [33m 18s](https://github.com/Eventual-Inc/Daft/actions/runs/20981828603/job/60308365885?pr=6023) to 13m 44s — saving about 20 mins which is nice. The test_context is still slow, but it has very few tests, a big win is parallelization and taking >5 mins off just by skipping the setup for skipped tests. This also reduced the ray tests from 45 mins to 27 mins, and there should be plenty more savings here. Then disabling windows eliminated a 50+ min action. text ================================================================================ OVERALL SUMMARY ================================================================================ Tests Analyzed: 9620 Total Time: 941.32s Average Time: 0.05s Max Time: 14.70s Min Time: 0.01s It took 188s to run make test on my mac which is nice; it also crushed my machine while it was running 🥲 ## Changes Made * Adds -n auto to pytest for parallelize testing on multiple cores * Skips generating fixtures (expensive?) by not actually creating the tests - we were creating test data _then_ skipping for several thousand join tests taking ~5 mins of setup work just to skip. * Memoizes data generation for the window tests (often the slowest for me) * Reduces the data generation size and faster comparison algo (before was 10,000 data points with a O(n^2) algo, now it's 1,000 data points with an O(nlogn) comparison algo) * Disables the RAY_DASHBOARD for the context setup commands, and other minor things to speedup ray init * Skips windows on PRs (>50 minutes) only enabled on main, worthwhile trade-off because we will still catch windows-specific failures when merging to main which is quite rare to be fair. ## Related Issues N/A | 8 个月前 | |
fix: Broadcast literal expressions in aggregations to match input length (#6155) ## Changes Made Fixes incorrect results when aggregating literal expressions (e.g., count(lit(1)), sum(lit(1))). ### Root Cause RecordBatch::eval_expression for Literal(1) returns a single-element Series [1], but aggregation functions expect the input Series length to match the RecordBatch row count. This caused: 1. **Non-grouped aggregation** (e.g., df.agg(count(lit(1)))): counted the 1-element Series, always returning 1 2. **Grouped aggregation with data-accessing functions** (e.g., groupby.agg(sum(lit(1)))): panic with "Out of bounds" when group indices exceeded the Series length 3. **Window functions**: after hash-partitioning, when all rows share the same partition key, the single-partition optimization sets groups=None, falling into case 1 ## Related Issues Closes #5685 | 7 个月前 | |
fix: use union instead of append_column in window agg to fix schema mismatch (#6178) ## Changes Made In WindowPartitionAndOrderBySink::finalize, the Agg branch used append_column with the full output schema (params.original_schema), even though only one window column had been added so far. This caused a SchemaMismatch error when an Agg expression (sum, mean, etc.) appeared before an Offset expression (lag/lead) on the same window spec. The fix switches the Agg branch to use union (via RecordBatch::from_nonempty_columns + union), matching what the Rank, Offset, and RowNumber branches already do. This builds the schema from the actual columns present rather than using a pre-specified full output schema. Also adds a regression test for this case. ## Related Issues This fixes the nightly [NIGHTLY] Notebook Checker CI failure that has been occurring since PR #5547 was merged in November 2025. | 7 个月前 | |
chore: bump mypy and ruff in pre-commit (#5836) ## Changes Made Bump mypy to 1.19.1 and ruff to 0.14.10 and resolve pre-commit errors. Rationale: my local ruff version is much higher than that in the project pre-commit, so I am getting a lot of linter warnings in my IDE. It is also about time to upgrade lints to utilize newer Python language features. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 9 个月前 | |
chore: bump mypy and ruff in pre-commit (#5836) ## Changes Made Bump mypy to 1.19.1 and ruff to 0.14.10 and resolve pre-commit errors. Rationale: my local ruff version is much higher than that in the project pre-commit, so I am getting a lot of linter warnings in my IDE. It is also about time to upgrade lints to utilize newer Python language features. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 9 个月前 | |
feat(swordfish): plan caching (#6278) ## Summary Introduces fingerprint-based plan caching so multiple executions of the same logical plan share a single pipeline, rather than building a new one each time. ### Key changes - ** PipelineMessage enum** — Replaces raw MicroPartition in all pipeline channels with Morsel { input_id, partition } and Flush(input_id), allowing a single pipeline to multiplex data from multiple logical inputs - **NativeExecutor becomes stateful** — Maintains a HashMap<u64, PlanState> keyed by plan fingerprint; repeated calls to run() with the same fingerprint reuse the existing pipeline - **MessageRouter** — Routes pipeline output to per-input_id unbounded channels so each caller gets only its own results - **try_finish() lifecycle API** — Callers signal input completion and collect per-input stats; pipeline is torn down when the last input finishes - **Per-input runtime stats** — RuntimeStatsManager tracks (NodeID, InputId) pairs with merge support for aggregated views - **next_event helper** — Shared tokio::select! loop abstraction used by blocking sinks, streaming sinks, intermediate ops, and joins - **Simplified BlockingSink trait** — Removed BlockingSinkFinalizeOutput::HasMoreOutput (was dead code); finalize now returns Vec<MicroPartition> directly; make_state takes InputId - **Simplified StreamingSink trait** — Removed StreamingSinkOutput::HasMoreOutput variant - **All node types updated** — Sources, intermediate ops, blocking sinks, streaming sinks, joins, and concat all handle Morsel/Flush per-input routing - **Python integration** — native_executor.py and flotilla.py call try_finish() for lifecycle management ### Files changed ~49 files across daft-local-execution (pipeline, sources, sinks, joins, stats), daft-distributed, common-metrics, and Python runners. ## Test plan - [ ] Full test suite with DAFT_RUNNER=native - [ ] Full test suite with DAFT_RUNNER=ray - [ ] Concurrent query execution sharing the same plan fingerprint - [ ] Verify per-input stats are correctly reported and merged - [ ] TPC-H benchmarks to check for performance regressions 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: R. C. Howell <5731503+rchowell@users.noreply.github.com> | 6 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 8 个月前 | ||
| 8 个月前 | ||
| 7 个月前 | ||
| 7 个月前 | ||
| 9 个月前 | ||
| 9 个月前 | ||
| 6 个月前 |