| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
chore: Remove expression namespaces (#5619) ## Changes Made Bye bye ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 11 个月前 | |
fix: close connections in io tests and address other warnings (#6470) ## Changes Made * Iceberg tests weren't closing the sqlite connection which made our CI logs flooded with warnings. * Several other io tests had the same issue. * Also fixes some additional warnings which are making CI logs noisy. ## Related Issues N/A | 7 个月前 | |
feat: supports passing a projection kwargs in select (#4884) ## Changes Made Adds support for a dict or projections to select which is backwards compatible but consistent with .with_columns. This format makes it easy to list down the columns prior to inserting somewhere else. python In [1]: import daft In [2]: df = daft.from_pydict({"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}) In [3]: df = df.select({ ...: "sum": df["x"] + df["y"] + df["z"], ...: "product": df["x"] * df["y"] * df["z"], ...: }) In [4]: df.show() ╭───────┬─────────╮ │ sum ┆ product │ │ --- ┆ --- │ │ Int64 ┆ Int64 │ ╞═══════╪═════════╡ │ 12 ┆ 28 │ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┤ │ 15 ┆ 80 │ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┤ │ 18 ┆ 162 │ ╰───────┴─────────╯ (Showing first 3 of 3 rows) ## Related Issues Closes #4883 ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 1 年前 | |
| 8 个月前 | ||
chore: Remove expression namespaces (#5619) ## Changes Made Bye bye ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 11 个月前 | |
chore: Remove expression namespaces (#5619) ## Changes Made Bye bye ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 11 个月前 | |
chore(deps): drop pyarrow 8.0.0 support, bump minimum to >= 15.0.0 (#6378) ## Changes Made Bump minimum PyArrow version from >= 8.0.0 to >= 15.0.0 (44 files, -450 lines). - Update version constraints in pyproject.toml and CI test matrix - Remove _FixSliceOffsets workaround (pyarrow < 12.0.0 struct slice offset bug, fixed upstream) - Remove pyarrow_supports_fixed_shape_tensor() and all conditional branches - Remove obsolete version checks (< 12.0.1, < 13.0.0, >= 9.0.0) and try/except imports - Clean up ~50 pytest.mark.skipif markers and unused PYARROW_GE_* constants in test files Note: _FixEmptyStructArrays is intentionally kept — Daft internally cannot handle empty StructArrays, not just an arrow2 FFI issue. ## Related Issues Closes #6347 | 7 个月前 | |
chore: Remove expression namespaces (#5619) ## Changes Made Bye bye ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 11 个月前 | |
chore: Remove expression namespaces (#5619) ## Changes Made Bye bye ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 11 个月前 | |
feat: Add string strip function (#6372) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> Add strip() function to Daft's string functions. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> https://github.com/Eventual-Inc/Daft/issues/3792 | 7 个月前 | |
refactor: rename table to recordbatch (#3771) renames 'table' to 'recordbatch' to free up 'table' for upcoming catalog & table functionality. note for reviewers: a lot of files changed, but this was mostly a find and replace, so there shouldn't be much "logic" to actually review, just more of a sanity check to make sure I didn't terribly mess something up. | 1 年前 | |
chore: change series literal to list (#4896) ## Changes Made Remove LiteralValue::Series and replace it with a LiteralValue::List. I don't think we use series literals as series anymore so this name makes more sense. Additionally I've changed the datatypes/conversions to reflect that this is actually a list-type scalar value. This is part of a larger change that I'm working on to unify type conversions between Python and Daft. Notes: - Removed test_length_mismatch_all_types because length mismatch is now handled by the RecordBatch constructor so there's a different error, which should happen for any expression, so no need to test it specifically for this one. - Removed test_hash_int_array_with_bad_length - same reason as above. - Removed test_table_expr_is_in_different_input_types. It tested behavior that we did not document, so I'm removing it instead of fixing it. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [x] Documented in API Docs (if applicable) - [x] Documented in User Guide (if applicable) - [x] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [x] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) --------- Co-authored-by: Colin Ho <colin.ho99@gmail.com> | 1 年前 | |
refactor: rename table to recordbatch (#3771) renames 'table' to 'recordbatch' to free up 'table' for upcoming catalog & table functionality. note for reviewers: a lot of files changed, but this was mostly a find and replace, so there shouldn't be much "logic" to actually review, just more of a sanity check to make sure I didn't terribly mess something up. | 1 年前 | |
refactor: rename table to recordbatch (#3771) renames 'table' to 'recordbatch' to free up 'table' for upcoming catalog & table functionality. note for reviewers: a lot of files changed, but this was mostly a find and replace, so there shouldn't be much "logic" to actually review, just more of a sanity check to make sure I didn't terribly mess something up. | 1 年前 | |
refactor: rename table to recordbatch (#3771) renames 'table' to 'recordbatch' to free up 'table' for upcoming catalog & table functionality. note for reviewers: a lot of files changed, but this was mostly a find and replace, so there shouldn't be much "logic" to actually review, just more of a sanity check to make sure I didn't terribly mess something up. | 1 年前 | |
chore: Collection of random Swordfish cleanups (#6454) ## Changes Made A bunch of random cleanup / chores that I did in Swordfish as I was cleaning up the PR to add real progress bars. Rather than clutter that PR, I split it out into this one. Some things in particular: * Made RuntimeStats an associated type in local PipelineNodes * Changed the RuntimeStats trait a bit in response * Made operators return MicroPartitions directly instead of Arc<MicroPartition> & making MicroPartitions cloneable * I think they weren't originally because of the Unloaded option, but I don't think that's necessary anymore. Feel free to lmk if you disagree * Removed TimedFutures and instead recycled the timers we use for dynamic batching. This actually ended up fixing a bug we saw with the timers. | 7 个月前 | |
refactor(ordinals): move binding step above micropartition + local execution (#4425) ## Changes Made continuation of the refactor, this time moving column binding above the daft-local-execution and daft-micropartition crates. After this, we can start working on binding in the logical plan!! ## Related Issues #4270 ## Checklist - [x] Documented in API Docs (if applicable) - [x] Documented in User Guide (if applicable) - [x] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [x] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 1 年前 | |
| 9 个月前 | ||
chore: Remove expression namespaces (#5619) ## Changes Made Bye bye ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 11 个月前 | |
refactor(arrow2): remove arrow2 from daft-recordbatch (#6231) ## Changes Made removed all usages of arrow2 and daft-arrow from daft-recordbatch except for probeable, as probable requires dyn_compare to be refactored to arrow_rs first. ## Related Issues https://github.com/Eventual-Inc/Daft/pull/6227 | 8 个月前 | |
chore: remove _FixEmptyStructArrays workaround (#6478) ## Changes Made Removes the _FixEmptyStructArrays workaround from the codebase. This was legacy code that converted empty struct arrays (pa.struct([])) into single-field placeholder structs (pa.struct({"": pa.null()})) on ingestion, then reversed the transformation on output. Arrow now properly supports empty structs, and the root cause in Daft — StructArray deriving its length from the first child (yielding 0 for zero-field structs) — is fixed directly. **Core fix:** - StructArray::new now derives length from the null buffer when children is empty - Added StructArray::new_empty constructor for zero-field structs with explicit length - StructArray::to_arrow uses arrow::array::StructArray::new_empty_fields for empty structs **Removed placeholder machinery:** - Deleted daft/arrow_utils.py (_FixEmptyStructArrays, remove_empty_struct_placeholders, ensure_* helpers) - Removed all callers in series.py, recordbatch.py, ray_runner.py - Removed remove_empty_struct_placeholders call from Rust FFI path (series.rs) - Removed {String::new() => Literal::Null} placeholder pattern in pydict_to_struct_lit / pytuple_to_struct_lit - Removed placeholder field filters in literal-to-Python conversion, repr, JSON inference, and get_lit ## Related Issues Follow-up to #6378 which kept _FixEmptyStructArrays due to internal limitations that are now resolved. | 7 个月前 | |
feat: extended hash function to take and hash multiple inputs (#5692) ## Changes Made The 3 core changes are: 1. The Python hash helper now accepts multiple expressions/strings, normalizes bare strings to columns, and forwards them to the native UDF. 2. The Rust hash UDF (src/daft-functions/src/hash.rs) was updated to accept multiple Series: it builds a RecordBatch and uses hash_with(). Single-column and seed handling remain as before. 4. A broadcast-friendly constructor is used so scalar seeds/literals (e.g., hash(a, 0)) are expanded to match row counts. ## Related Issues https://github.com/Eventual-Inc/Daft/issues/4390 --------- Co-authored-by: Colin Ho <chiuhong@usc.edu> | 10 个月前 | |
refactor: rename table to recordbatch (#3771) renames 'table' to 'recordbatch' to free up 'table' for upcoming catalog & table functionality. note for reviewers: a lot of files changed, but this was mostly a find and replace, so there shouldn't be much "logic" to actually review, just more of a sanity check to make sure I didn't terribly mess something up. | 1 年前 | |
fix: Allow is_in to accept sets, tuples, and other iterables (#6115) Previously, is_in only accepted lists. Now it accepts any iterable (sets, tuples, frozensets, ranges, generators, dict_keys, etc.) by converting them to lists before processing. Strings and bytes are explicitly excluded since they are iterable but should not be treated as a sequence of characters/bytes. Fixes #6107 The regression was introduced in https://github.com/Eventual-Inc/Daft/pull/5086 This commit refactored many functions including moving is_in to daft/functions/misc.py. Looking at the implementation that was added, we went from if isinstance(other, Collection): other = [Expression._to_expression(item) for item in other] to if isinstance(other, list): other = [Expression._to_expression(item) for item in other] --------- Co-authored-by: Claude <noreply@anthropic.com> | 8 个月前 | |
refactor(ordinals): use indices in RecordBatch::get_column(s) (#4318) ## Changes Made Follow-up to #4297. This should complete the ordinals refactor for RecordBatch, both on Rust and Python, by removing the ability to directly get columns by name, and replacing calls to deprecated schema methods. The only place in daft-recordbatch that still uses name references are in the Python boundary and the join kernels, because it does common column merging. We want to change that behavior in the future anyway, so I'm leaving it in for now. Also renamed MicroPartition.get_column to MicroPartition.get_column_by_name and added MicroPartition.get_column and MicroPartition.columns for consistency between RecordBatch and MicroPartition APIs. There are still a number of places using MicroPartition.get_column_by_name that I plan to address when the refactor reaches micropartition ## Related Issues #4270 ## Checklist - [x] Documented in API Docs (if applicable) - [x] Documented in User Guide (if applicable) - [x] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [x] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 1 年前 | |
fix: Picking MicroPartitions with duplicate column names (#4696) | 1 年前 | |
feat: Extend hash variants for xxhash (#5276) | 1 年前 | |
perf: Don't concat in micropartition.to_arrow() (#4192) ## Changes Made Don't concat the micropartition in micropartition.to_arrow(). Instead, create a pyarrow table from the record batches via from_batches Additionally, replace more internal references to table to recordbatch ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 1 年前 | |
refactor(ordinals): use indices in RecordBatch::get_column(s) (#4318) ## Changes Made Follow-up to #4297. This should complete the ordinals refactor for RecordBatch, both on Rust and Python, by removing the ability to directly get columns by name, and replacing calls to deprecated schema methods. The only place in daft-recordbatch that still uses name references are in the Python boundary and the join kernels, because it does common column merging. We want to change that behavior in the future anyway, so I'm leaving it in for now. Also renamed MicroPartition.get_column to MicroPartition.get_column_by_name and added MicroPartition.get_column and MicroPartition.columns for consistency between RecordBatch and MicroPartition APIs. There are still a number of places using MicroPartition.get_column_by_name that I plan to address when the refactor reaches micropartition ## Related Issues #4270 ## Checklist - [x] Documented in API Docs (if applicable) - [x] Documented in User Guide (if applicable) - [x] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [x] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 1 年前 | |
refactor: rename table to recordbatch (#3771) renames 'table' to 'recordbatch' to free up 'table' for upcoming catalog & table functionality. note for reviewers: a lot of files changed, but this was mostly a find and replace, so there shouldn't be much "logic" to actually review, just more of a sanity check to make sure I didn't terribly mess something up. | 1 年前 | |
refactor: rename table to recordbatch (#3771) renames 'table' to 'recordbatch' to free up 'table' for upcoming catalog & table functionality. note for reviewers: a lot of files changed, but this was mostly a find and replace, so there shouldn't be much "logic" to actually review, just more of a sanity check to make sure I didn't terribly mess something up. | 1 年前 | |
fix: sorting on a literal value and aggregation with order-by (#5547) | 11 个月前 | |
refactor(arrow2): migrate groups to arrow-rs (#6185) ## Changes Made Migrate groups to arrow-rs and add a few test cases. --------- Co-authored-by: universalmind303 <cory.grinstead@gmail.com> | 8 个月前 | |
refactor(arrow-rs): Remove arrow2 Index generics usages (#5761) ## Changes Made Arrow-rs doesn't have the notion of an Index type like arrow2 does. Instead, most of the functions either except a UInt64Array or do internal downcasting from a dyn Array. Most of the times, we actually just force a UInt64Array anyways, like in argsort, and most of the take implementations, so I just made it a hard requirement. Note that I did modify some take tests because we no longer support take on non u64 arrays. If you feel like this would be a regression, I will add it back by adding a cast in PySeries::take | 10 个月前 | |
chore: bump mypy and ruff in pre-commit (#5836) ## Changes Made Bump mypy to 1.19.1 and ruff to 0.14.10 and resolve pre-commit errors. Rationale: my local ruff version is much higher than that in the project pre-commit, so I am getting a lot of linter warnings in my IDE. It is also about time to upgrade lints to utilize newer Python language features. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 10 个月前 | |
chore: Remove expression namespaces (#5619) ## Changes Made Bye bye ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 11 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 11 个月前 | ||
| 7 个月前 | ||
| 1 年前 | ||
| 8 个月前 | ||
| 11 个月前 | ||
| 11 个月前 | ||
| 7 个月前 | ||
| 11 个月前 | ||
| 11 个月前 | ||
| 7 个月前 | ||
| 1 年前 | ||
| 1 年前 | ||
| 1 年前 | ||
| 1 年前 | ||
| 1 年前 | ||
| 7 个月前 | ||
| 1 年前 | ||
| 9 个月前 | ||
| 11 个月前 | ||
| 8 个月前 | ||
| 7 个月前 | ||
| 10 个月前 | ||
| 1 年前 | ||
| 8 个月前 | ||
| 1 年前 | ||
| 1 年前 | ||
| 1 年前 | ||
| 1 年前 | ||
| 1 年前 | ||
| 1 年前 | ||
| 1 年前 | ||
| 11 个月前 | ||
| 8 个月前 | ||
| 10 个月前 | ||
| 10 个月前 | ||
| 11 个月前 |