| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
feat: support various image hash functions (aHash, dHash, pHash, wHash, crop-resistant hash) for deduplication (#6338) Implements 5 image hashing algorithms for image deduplication: aHash (Average Hash): Compares each pixel to the mean intensity dHash (Difference Hash): Compares adjacent pixel differences pHash (Perceptual Hash): DCT-based frequency domain hashing wHash (Wavelet Hash): Haar wavelet transform-based hashing Crop-resistant Hash: Robust against cropping transformations ## Changes Made Added hash methods to CowImage with helper functions (dct2d_32x32, haar_2d) Added ImageHashOps trait and implementations for ImageArray / FixedShapeImageArray Added series-level functions and ScalarUDF registrations Exposed Python APIs: image_ahash, image_dhash, image_phash, image_whash, image_crop_resistant_hash Added corresponding methods to SeriesImageNamespace Added tests covering all 5 algorithms Fixed-size hashes return FixedSizeBinary(8), crop-resistant hash returns variable-length Binary. ## Related Issues Closes #4889 | 5 个月前 | |
chore(io): replace python source shim on the rust side (#6556) ## Motivation This removes the python ScanOperator shim in favor of a ScanOperator implementation in Rust which is backed by either a Python *or* Rust data source. This allows us to move the shim one step lower, and now we actually have Rust traits + python bridges for the DataSource and DataSourceTask interfaces. This is an incremental step towards full streaming sources and consolidating the various scanning types and implementations to this interface. ## Changes Made - Removes python _DataSourceShim - Cleanup visibility modifiers and conversion methods for daft-catalog Catalog, Table, Provider - Adds native factory methods for native data source tasks, which are a work-in-progress and just today's scan tasks. - We have a Rust PyDataSourceWrapper trait which is able to implement our existing ScanOperator backed by a Python DataSource | 5 个月前 | |
chore: Disable cargo check in local dev (#6439) ## Changes Made I believe the cargo check pre-commit commands are useless in local development. They should run with cargo clippy anyways (since check just ensures that the crate compiles without errors and maybe warnings). Plus, because we do it 2 times with different flags, it causes a lot of recompilation which just makes it slow. So i disabled them for git commits. They are still enabled when you do make precommit and in CI to be safe. | 6 个月前 | |
chore(io): replace python source shim on the rust side (#6556) ## Motivation This removes the python ScanOperator shim in favor of a ScanOperator implementation in Rust which is backed by either a Python *or* Rust data source. This allows us to move the shim one step lower, and now we actually have Rust traits + python bridges for the DataSource and DataSourceTask interfaces. This is an incremental step towards full streaming sources and consolidating the various scanning types and implementations to this interface. ## Changes Made - Removes python _DataSourceShim - Cleanup visibility modifiers and conversion methods for daft-catalog Catalog, Table, Provider - Adds native factory methods for native data source tasks, which are a work-in-progress and just today's scan tasks. - We have a Rust PyDataSourceWrapper trait which is able to implement our existing ScanOperator backed by a Python DataSource | 5 个月前 | |
feat: add daft-checkpoint crate with CheckpointStore trait and in-mem… (#6511) Introduces a source-oriented checkpoint store for tracking which input rows have been processed and what output files were produced. This enables skipping already-processed rows on re-run (progress tracking) and knowing which files to commit after a crash (recovery for 2PC sinks like Iceberg). Core abstractions: - CheckpointStore async trait with stage/checkpoint/commit lifecycle - CheckpointId (UUID), Checkpoint (metadata with timestamps), FileMetadata (opaque blob) - Lifecycle: staged (invisible) → checkpointed (visible) → committed - checkpoint() and mark_committed() are idempotent for crash recovery In-memory implementation gated behind test-utils feature flag. 10 tests covering lifecycle, idempotency, error paths, orphan behavior, edge cases, and trait object safety. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> | 5 个月前 | |
chore(lints): add workspace lints inheritance to missing subcrates (#6330) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> - Add lints = { workspace = true } to Common/macros, Daft-ai, Daft-catalog, Daft-cli - Align subcrate lint configuration with workspace standards to ensure consistency ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> --------- Co-authored-by: zhouhai <zhouhai@dewu.com> | 6 个月前 | |
feat: Add WARC reader (#3871) Adds a reader for .warc and .warc.gz files. Currently optimized for reading from S3. Some numbers: - Downloading a single common crawl file from S3, e.g. s3://commoncrawl/crawl-data/CC-MAIN-2018-17/segments/1524125937193.1/warc/CC-MAIN-20180420081400-20180420101400-00000.warc.gz, takes ~16.5s. - Gunzipping this file and processing it with fastwarc takes ~33s. - With swordfish, collecting that same file as a daft dataframe takes ~20s on an m7g.4xlarge instance. - With swordfish, collecting 10 common crawl files take ~3min 12s. - With swordfish, processing (read then sum on content length) 1 common crawl file takes ~20s. - With swordfish, processing 2 common crawl files still takes ~20s. - With swordfish, processing 10 common crawl files takes ~40s. This is because we've set the max number of parallel reads to 8. So 10 scan tasks take 2x20s to read. If we increase the max number of parallel reads to 10, the runtime drops to ~30s. - With ray, collecting 1 file takes ~25s. - **Unfortunately, with ray, collecting 10 files caused the instance to become unresponsive.** Followup work: - Extracting fields from the warc_headers json is not very fast. We can do better here by allowing users to specify the metadata headers that they want to extract. --------- Co-authored-by: Sammy Sidhu <sammy.sidhu@gmail.com> Co-authored-by: Colin Ho <colinho@Colins-MBP.localdomain> | 1 年前 | |
fix: hide dashboard result preview for unsupported queries (Flotilla) (#6557) ## Summary The results preview tab was added to the dashboard in #6327 but only supports Swordfish queries since Flotilla never calls on_result_out. This hides the tab for Flotilla queries (rather than misleadingly displaying a disabled tab which will never become enabled) until we add support for them (follow-up: #6559). Note: for a while running Flotilla queries would actually panic the dashboard server process in RecordBatch::from_ipc_stream, but this was coincidentally fixed on main by #6541. This PR addresses the remaining issues: avoiding the empty serialization in the first place, and hiding the broken UI. ## Test plan - [x] cargo check passes for daft-context, daft-dashboard - [x] tsc --noEmit passes for dashboard frontend - [x] Manual: run a Flotilla query and verify dashboard doesn't panic, Results tab is hidden - [x] Manual: run a Swordfish query and verify Results tab still works --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> | 5 个月前 | |
perf: reduce binary size by de-monomorphizing sort and dispatch paths (#6541) ## Changes Made Incremental pass on #5623 to reduce binary size in daft-core by de-monomorphizing sort and dispatch-heavy paths. Measured with maturin build --release, then unzipping the wheel and inspecting daft.abi3.so. Before: 238,827,265 bytes After: 237,987,585 bytes Delta: -839,680 bytes (~0.35%) Main changes: - de-monomorphize sort dispatch - share comparison, logical, arithmetic, hashing, and search_sorted dispatch - add #[inline(never)] on selected generic cold paths Validated with existing test suites, including pre-commit, doctests, TPCH/SQL TPCH on native and ray, Gravitino integration on native and ray, the TPCH-related benchmark suite, targeted comparison/hash/aggregation checks, and the recordbatch IPC regression test. AI usage: used AI assistance for search/review support; all changes, measurements, and validation were verified locally. ## Related Issues Refs #5623 --------- Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> | 5 个月前 | |
perf: introduce Column type with ScalarColumn for O(1) scalar storage in RecordBatch (#6444) ## Changes Made Introduces a Column enum in daft-recordbatch that allows RecordBatch to store scalar (constant) columns without physically broadcasting them to full-length arrays. ### Problem When Daft evaluates expressions with literals (e.g., col("x") + lit(42)), the literal gets broadcast to a full-length array — a scalar value like 42 across 1M rows allocates a 1M-element Int64Array. This is wasteful both in memory and over the wire. ### Solution A new Column type wraps either a materialized Series or a ScalarColumn that stores a single Literal value + logical length: rust pub enum Column { Series(Series), Scalar(ScalarColumn), } ScalarColumn uses OnceLock<Series> for lazy materialization — the full array is only constructed when an operation actually requires element-wise access via as_materialized_series(), and the result is cached. ### What stays compact (no materialization) Row-preserving operations on scalar columns adjust the logical length in O(1): - slice, head, filter, take, broadcast - cast, as_physical (operate on the scalar value directly) - concat (if all values match, stays scalar) - size_bytes, Hash, PartialEq (scalar-aware) ### What materializes (unchanged behavior) Most ops/ code (joins, groupby, hash, sort, etc.) materializes via as_materialized_series() before operating. This PR lays the groundwork — behavior is identical to before, just with the Column abstraction in place. ### Public API - get_column(idx) -> &Series — **unchanged**. Returns a &Series via lazy materialization. Zero blast radius for external callers. - columns() -> &[Column] — return type changed from &[Series]. Callers that need &Series add .as_materialized_series(). ### Follow-ups - **Reduce materialization points**: audit as_materialized_series() calls and operate on scalars directly where possible - **Expression eval**: have Expr::Literal produce Column::Scalar instead of a length-1 Series - **REE wire format**: encode scalar columns as RunEndEncoded Arrow arrays for compact IPC/shuffle serialization | 5 个月前 | |
fix: hide dashboard result preview for unsupported queries (Flotilla) (#6557) ## Summary The results preview tab was added to the dashboard in #6327 but only supports Swordfish queries since Flotilla never calls on_result_out. This hides the tab for Flotilla queries (rather than misleadingly displaying a disabled tab which will never become enabled) until we add support for them (follow-up: #6559). Note: for a while running Flotilla queries would actually panic the dashboard server process in RecordBatch::from_ipc_stream, but this was coincidentally fixed on main by #6541. This PR addresses the remaining issues: avoiding the empty serialization in the first place, and hiding the broken UI. ## Test plan - [x] cargo check passes for daft-context, daft-dashboard - [x] tsc --noEmit passes for dashboard frontend - [x] Manual: run a Flotilla query and verify dashboard doesn't panic, Results tab is hidden - [x] Manual: run a Swordfish query and verify Results tab still works --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> | 5 个月前 | |
| 6 个月前 | ||
feat: Add a random shuffle operation (#6481) ## Changes Made Implement a global row-rearraging shuffle operation, similar to https://huggingface.co/docs/datasets/process#shuffle. All it does is generate a random integer per row and then sort by that integer. Rather than implementing a custom distributed and local operation, it is all done within expressions and the optimizer. Note, I still made it an explicit logical node because: 1. Its easier to understand in explain outputs 2. I don't think we should always assume that it optimizes the same way that sort does. For example, it might be reasonable to push limits through a shuffle, but not allowed for sorts. ## Related Issues First half of #2612. Second half to add a batch_size version. --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 5 个月前 | |
fix: short-circuit evaluation for coalesce (#6525) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> Fixes #4069 by adding proper short-circuit behavior to coalesce. - Added Expr::Coalesce with short-circuit logic - Implemented early-exit in record batch evaluation - Preserved semantics during UDF optimization - Added short-circuit tests ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> Closes #4069 --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 5 个月前 | |
refactor(ext): use single, publishable crate for daft-ext types (#6452) ## Changes Made This consolidates the daft-ext types into one crate so it's easy to publish as just 'daft-ext'. Splitting into individual crates so prematurely just adds complexity. ## Related Issues N/A --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Desmond Cheong <desmondcheongzx@gmail.com> | 5 个月前 | |
refactor(ext): use single, publishable crate for daft-ext types (#6452) ## Changes Made This consolidates the daft-ext types into one crate so it's easy to publish as just 'daft-ext'. Splitting into individual crates so prematurely just adds complexity. ## Related Issues N/A --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Desmond Cheong <desmondcheongzx@gmail.com> | 5 个月前 | |
refactor(ext): use single, publishable crate for daft-ext types (#6452) ## Changes Made This consolidates the daft-ext types into one crate so it's easy to publish as just 'daft-ext'. Splitting into individual crates so prematurely just adds complexity. ## Related Issues N/A --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Desmond Cheong <desmondcheongzx@gmail.com> | 5 个月前 | |
chore: Upgrade PyO3 to 0.28 (#6383) FIxed some warnings that popped up. | 6 个月前 | |
perf: Rewrite decode(..., "utf-8") as a cast (#6537) ## Changes Made We have LogicalPlan optimizations for simplifying casting expressions. For example, if we cast a string -> string, we eliminate the cast operation. We don't with decode because the behavior is encoded within the expression definition. Note, we could special-case decode in the optimizer, but I think its cleaner to do it in Python because parsing is kind of ugly. Lmk if you disagree though. I think its fair to say that converting decodes to cast may be confusing in explain plans. Also, I think this will help ClickBench a bunch, since this was blocking filter pushdowns --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 5 个月前 | |
refactor(arrow2): remove a bunch of from impls that used arrow2 (#6137) ## Changes Made removes a lot of From impls that used arrow2, and were honestly quite unintuitive. Replaced them with explicit method calls like from_slice and from_vec instead. Additional notes: I made an attempt to consolidate all of these apis for consistency. Most of the daft arrays now all follow the same pattern. - from_iter(name, IntoIterator<Option<T>>) - from_values(name, IntoIterator<T>) - from_slice(name, &[T]) - from_vec(name, Vec<T>) the only exception is DataArray<T: DaftPhysicalType> which uses field instead of name - from_iter(field, IntoIterator<Option<T>>) - from_field_and_values(field, IntoIterator<T>) and this is currently ONLY because of Decimal128 which can change the precision and scale, and therefore needs a whole new datatype. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 7 个月前 | |
perf: Optimize list_* aggregation functions (#6494) ## Changes Made Optimize list_* operations (list_min, list_max, etc) by replacing the existing Series -> concat algorithm with an aggregator based approach. Note, I'm not particularly happy with this approach, cause it still seems like a bunch of overhead to create a bunch of mini vectors, but its a pretty small diff and provably works. Tested on a m8g.2xlarge using the connected components for dedupe script (https://github.com/Eventual-Inc/dedupe/blob/main/cc.py). - Original: 62.80s - PR: 20.82s | 5 个月前 | |
refactor(arrow2): goodbye arrow2 (#6363) ## Changes Made <img width="497" height="469" alt="image" src="https://github.com/user-attachments/assets/3fd455e7-0801-4994-94fb-fe48eeb1e142" /> ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 6 个月前 | |
feat: convert and replace timezone (#6106) ## Changes Made - When converting time zone: - Output error when from_timezone is not provided for timestamp without tz info. - Ignore from_timezone if timestamp already has tz info. - When replacing time zone, drop timezone in timestamp if timezone is not provided. - Create new ParsedTimezone enum to handle different timezone types (offset and tz name). - Use new Daft array building semantics. ## Related Issues Closes #4096. | 5 个月前 | |
| 7 个月前 | ||
perf(upload): use a zero copy approach for url_upload (#6334) ## Changes Made Eliminate per-row memcpy in url_upload by creating bytes::Bytes slices that share ownership of the underlying arrow buffer instead of calling .to_vec() on every element. This was specifically mentioned in a TODO as the optimal path forward: `` // Alternatively, we can find a way of creating a bytes::Bytes that just references the underlying // arrow2 buffer, without making a copy. This would be the ideal case. ` ### How it works Arrow-rs Buffer is internally reference-counted. We wrap it in a newtype (ArrowBuffer) that implements AsRef<[u8]>, pass it to bytes::Bytes::from_owner (one cheap Arc bump), and then bytes::Bytes::slice` for each element — all zero-copy. Previously every row was fully copied into a new Vec<u8> before upload. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 6 个月前 | |
feat: Add string strip function (#6372) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> Add strip() function to Daft's string functions. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> https://github.com/Eventual-Inc/Daft/issues/3792 | 5 个月前 | |
feat: added concat_ws function (#6543) GitHub tracking issue: [Eventual-Inc/Daft#3792](https://github.com/Eventual-Inc/Daft/issues/3792) Joining strings with a separator is one of the most common operations in data pipelines - building file paths, composite keys, CSV-like fields, display strings. Daft has concat today, but it propagates nulls and has no separator support. Users who need concat_ws semantics currently have to manually interleave lit(sep) between columns and handle nulls themselves. | 5 个月前 | |
chore: Remove some extra & duplicate Rust packages (#6435) ## Changes Made Found some extra duplicate packages, so removed them and refactored to compile. I was hoping to remove the ve-tos-rust-sdk library as well (since it requires reqwests 0.11 and we use 0.12, which causes us to pull at least 5 duplicates), but I need OpenDAL to release a new version with TOS support | 6 个月前 | |
feat: support various image hash functions (aHash, dHash, pHash, wHash, crop-resistant hash) for deduplication (#6338) Implements 5 image hashing algorithms for image deduplication: aHash (Average Hash): Compares each pixel to the mean intensity dHash (Difference Hash): Compares adjacent pixel differences pHash (Perceptual Hash): DCT-based frequency domain hashing wHash (Wavelet Hash): Haar wavelet transform-based hashing Crop-resistant Hash: Robust against cropping transformations ## Changes Made Added hash methods to CowImage with helper functions (dct2d_32x32, haar_2d) Added ImageHashOps trait and implementations for ImageArray / FixedShapeImageArray Added series-level functions and ScalarUDF registrations Exposed Python APIs: image_ahash, image_dhash, image_phash, image_whash, image_crop_resistant_hash Added corresponding methods to SeriesImageNamespace Added tests covering all 5 algorithms Fixed-size hashes return FixedSizeBinary(8), crop-resistant hash returns variable-length Binary. ## Related Issues Closes #4889 | 5 个月前 | |
| 6 个月前 | ||
chore: remove _FixEmptyStructArrays workaround (#6478) ## Changes Made Removes the _FixEmptyStructArrays workaround from the codebase. This was legacy code that converted empty struct arrays (pa.struct([])) into single-field placeholder structs (pa.struct({"": pa.null()})) on ingestion, then reversed the transformation on output. Arrow now properly supports empty structs, and the root cause in Daft — StructArray deriving its length from the first child (yielding 0 for zero-field structs) — is fixed directly. **Core fix:** - StructArray::new now derives length from the null buffer when children is empty - Added StructArray::new_empty constructor for zero-field structs with explicit length - StructArray::to_arrow uses arrow::array::StructArray::new_empty_fields for empty structs **Removed placeholder machinery:** - Deleted daft/arrow_utils.py (_FixEmptyStructArrays, remove_empty_struct_placeholders, ensure_* helpers) - Removed all callers in series.py, recordbatch.py, ray_runner.py - Removed remove_empty_struct_placeholders call from Rust FFI path (series.rs) - Removed {String::new() => Literal::Null} placeholder pattern in pydict_to_struct_lit / pytuple_to_struct_lit - Removed placeholder field filters in literal-to-Python conversion, repr, JSON inference, and get_lit ## Related Issues Follow-up to #6378 which kept _FixEmptyStructArrays due to internal limitations that are now resolved. | 5 个月前 | |
chore: Move Flight Server to Rust (#6519) ## Changes Made Provide in Rust so we have direct access to it for optimizations | 5 个月前 | |
feat: Add a random shuffle operation (#6481) ## Changes Made Implement a global row-rearraging shuffle operation, similar to https://huggingface.co/docs/datasets/process#shuffle. All it does is generate a random integer per row and then sort by that integer. Rather than implementing a custom distributed and local operation, it is all done within expressions and the optimizer. Note, I still made it an explicit logical node because: 1. Its easier to understand in explain outputs 2. I don't think we should always assume that it optimizes the same way that sort does. For example, it might be reasonable to push limits through a shuffle, but not allowed for sorts. ## Related Issues First half of #2612. Second half to add a batch_size version. --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 5 个月前 | |
fix: short-circuit evaluation for coalesce (#6525) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> Fixes #4069 by adding proper short-circuit behavior to coalesce. - Added Expr::Coalesce with short-circuit logic - Implemented early-exit in record batch evaluation - Preserved semantics during UDF optimization - Added short-circuit tests ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> Closes #4069 --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 5 个月前 | |
chore: Collection of random Swordfish cleanups (#6454) ## Changes Made A bunch of random cleanup / chores that I did in Swordfish as I was cleaning up the PR to add real progress bars. Rather than clutter that PR, I split it out into this one. Some things in particular: * Made RuntimeStats an associated type in local PipelineNodes * Changed the RuntimeStats trait a bit in response * Made operators return MicroPartitions directly instead of Arc<MicroPartition> & making MicroPartitions cloneable * I think they weren't originally because of the Unloaded option, but I don't think that's necessary anymore. Feel free to lmk if you disagree * Removed TimedFutures and instead recycled the timers we use for dynamic batching. This actually ended up fixing a bug we saw with the timers. | 5 个月前 | |
refactor(arrow2): remove arrow2 from minhash (#6149) ## Changes Made Removed arrow2 usage from array ops minhash | 7 个月前 | |
perf(parquet): yield individual batches in streaming path to fix morsel cache locality (#6558) The arrow-rs parquet reader's streaming path ( local_parquet_stream_arrowrs) called decode_single_rg per row group, which set with_batch_size(128K) on the reader -- correctly producing multiple ~128K batches -- but then immediately concatenated them back into a single ~500K batch via concat_or_empty. This sent one large batch per row group through the pipeline, defeating the morsel size that downstream operators depend on for cache locality. On AMD EPYC (512KB L2 per core), evaluating many aggregation expressions on the same column causes the 128K-row source buffer (256KB) to stay hot in L2 across all evaluations. At 500K rows, the cast output (4MB) evicts the source from cache, forcing each expression to re-fetch from L3/memory. The fix extracts reader setup into build_rg_reader and has the streaming path iterate over individual batches, sending each ~128K batch through the channel. The bulk read path (decode_single_rg) is unchanged. Benchmarked on c6a.4xlarge (16 vCPUs, AMD EPYC) with 100 ClickBench parquet files (~100M rows total, Int16 column, 90 SUM aggregations): | Workload | v0.7.4 | v0.7.5 (regressed) | After fix | |----------|--------|--------------------|-----------| | 90 SUMs | 0.576s | 1.226s | **0.652s** | | 90 SUM(col+i) | 1.084s | 4.643s | **1.714s** | Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> | 5 个月前 | |
fix: short-circuit evaluation for coalesce (#6525) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> Fixes #4069 by adding proper short-circuit behavior to coalesce. - Added Expr::Coalesce with short-circuit logic - Implemented early-exit in record batch evaluation - Preserved semantics during UDF optimization - Added short-circuit tests ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> Closes #4069 --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 5 个月前 | |
| 7 个月前 | ||
chore(io): replace python source shim on the rust side (#6556) ## Motivation This removes the python ScanOperator shim in favor of a ScanOperator implementation in Rust which is backed by either a Python *or* Rust data source. This allows us to move the shim one step lower, and now we actually have Rust traits + python bridges for the DataSource and DataSourceTask interfaces. This is an incremental step towards full streaming sources and consolidating the various scanning types and implementations to this interface. ## Changes Made - Removes python _DataSourceShim - Cleanup visibility modifiers and conversion methods for daft-catalog Catalog, Table, Provider - Adds native factory methods for native data source tasks, which are a work-in-progress and just today's scan tasks. - We have a Rust PyDataSourceWrapper trait which is able to implement our existing ScanOperator backed by a Python DataSource | 5 个月前 | |
feat: convert and replace timezone (#6106) ## Changes Made - When converting time zone: - Output error when from_timezone is not provided for timestamp without tz info. - Ignore from_timezone if timestamp already has tz info. - When replacing time zone, drop timezone in timestamp if timezone is not provided. - Create new ParsedTimezone enum to handle different timezone types (offset and tz name). - Use new Daft array building semantics. ## Related Issues Closes #4096. | 5 个月前 | |
chore(io): replace python source shim on the rust side (#6556) ## Motivation This removes the python ScanOperator shim in favor of a ScanOperator implementation in Rust which is backed by either a Python *or* Rust data source. This allows us to move the shim one step lower, and now we actually have Rust traits + python bridges for the DataSource and DataSourceTask interfaces. This is an incremental step towards full streaming sources and consolidating the various scanning types and implementations to this interface. ## Changes Made - Removes python _DataSourceShim - Cleanup visibility modifiers and conversion methods for daft-catalog Catalog, Table, Provider - Adds native factory methods for native data source tasks, which are a work-in-progress and just today's scan tasks. - We have a Rust PyDataSourceWrapper trait which is able to implement our existing ScanOperator backed by a Python DataSource | 5 个月前 | |
chore: Move Flight Server to Rust (#6519) ## Changes Made Provide in Rust so we have direct access to it for optimizations | 5 个月前 | |
refactor(arrow-rs): Upgrade arrow-rs to 57.1.0 (#5969) ## Changes Made When going through some of the arrow2 uses (like in IPC, Flight, JSON, etc), I noticed that there are a couple of useful features in the latest version of arrow-rs that will help us with the migration. So I upgraded it | 8 个月前 | |
fix: short-circuit evaluation for coalesce (#6525) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> Fixes #4069 by adding proper short-circuit behavior to coalesce. - Added Expr::Coalesce with short-circuit logic - Implemented early-exit in record batch evaluation - Preserved semantics during UDF optimization - Added short-circuit tests ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> Closes #4069 --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 5 个月前 | |
perf: introduce Column type with ScalarColumn for O(1) scalar storage in RecordBatch (#6444) ## Changes Made Introduces a Column enum in daft-recordbatch that allows RecordBatch to store scalar (constant) columns without physically broadcasting them to full-length arrays. ### Problem When Daft evaluates expressions with literals (e.g., col("x") + lit(42)), the literal gets broadcast to a full-length array — a scalar value like 42 across 1M rows allocates a 1M-element Int64Array. This is wasteful both in memory and over the wire. ### Solution A new Column type wraps either a materialized Series or a ScalarColumn that stores a single Literal value + logical length: rust pub enum Column { Series(Series), Scalar(ScalarColumn), } ScalarColumn uses OnceLock<Series> for lazy materialization — the full array is only constructed when an operation actually requires element-wise access via as_materialized_series(), and the result is cached. ### What stays compact (no materialization) Row-preserving operations on scalar columns adjust the logical length in O(1): - slice, head, filter, take, broadcast - cast, as_physical (operate on the scalar value directly) - concat (if all values match, stays scalar) - size_bytes, Hash, PartialEq (scalar-aware) ### What materializes (unchanged behavior) Most ops/ code (joins, groupby, hash, sort, etc.) materializes via as_materialized_series() before operating. This PR lays the groundwork — behavior is identical to before, just with the Column abstraction in place. ### Public API - get_column(idx) -> &Series — **unchanged**. Returns a &Series via lazy materialization. Zero blast radius for external callers. - columns() -> &[Column] — return type changed from &[Series]. Callers that need &Series add .as_materialized_series(). ### Follow-ups - **Reduce materialization points**: audit as_materialized_series() calls and operate on scalars directly where possible - **Expression eval**: have Expr::Literal produce Column::Scalar instead of a length-1 Series - **REE wire format**: encode scalar columns as RunEndEncoded Arrow arrays for compact IPC/shuffle serialization | 5 个月前 | |
chore: Disable cargo check in local dev (#6439) ## Changes Made I believe the cargo check pre-commit commands are useless in local development. They should run with cargo clippy anyways (since check just ensures that the crate compiles without errors and maybe warnings). Plus, because we do it 2 times with different flags, it causes a lot of recompilation which just makes it slow. So i disabled them for git commits. They are still enabled when you do make precommit and in CI to be safe. | 6 个月前 | |
chore: Disable cargo check in local dev (#6439) ## Changes Made I believe the cargo check pre-commit commands are useless in local development. They should run with cargo clippy anyways (since check just ensures that the crate compiles without errors and maybe warnings). Plus, because we do it 2 times with different flags, it causes a lot of recompilation which just makes it slow. So i disabled them for git commits. They are still enabled when you do make precommit and in CI to be safe. | 6 个月前 | |
chore: Collection of random Swordfish cleanups (#6454) ## Changes Made A bunch of random cleanup / chores that I did in Swordfish as I was cleaning up the PR to add real progress bars. Rather than clutter that PR, I split it out into this one. Some things in particular: * Made RuntimeStats an associated type in local PipelineNodes * Changed the RuntimeStats trait a bit in response * Made operators return MicroPartitions directly instead of Arc<MicroPartition> & making MicroPartitions cloneable * I think they weren't originally because of the Unloaded option, but I don't think that's necessary anymore. Feel free to lmk if you disagree * Removed TimedFutures and instead recycled the timers we use for dynamic batching. This actually ended up fixing a bug we saw with the timers. | 5 个月前 | |
[CHORE] add/fix many clippy lints (#2978) mostly auto-fix; a few manual fixes | 1 年前 | |
chore: Move Flight Server to Rust (#6519) ## Changes Made Provide in Rust so we have direct access to it for optimizations | 5 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 5 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 1 年前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 7 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 7 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 7 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 7 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 8 个月前 | ||
| 5 个月前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 5 个月前 | ||
| 1 年前 | ||
| 5 个月前 |