| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
[RUST] Split series tests (#749) * It is currently difficult to understand test coverage across types and ops because everything is in one file * Splits test_series.py into individual files for each family of operations for easier contribution of tests Co-authored-by: Jay Chia <jaychia94@gmail.com@users.noreply.github.com> | 3 年前 | |
feat: add is_t and inner property methods to DataType (#4141) ## Changes Made - adds an is_T method for every variant on DataType - adds a method to extract all inner types. In rust, it returns None if the value doesn't exist, and in python it raises an AttributeException if the value doesnt exist for that variant ## Related Issues - closes https://github.com/Eventual-Inc/Daft/issues/4031 - closes https://github.com/Eventual-Inc/Daft/issues/4032 ## Checklist - [x] Documented in API Docs (if applicable) - [n/a] Documented in User Guide (if applicable) - [n/a] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [n/a] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 1 年前 | |
chore: Upgrade Ruff ruleset to 3.9 and add from __future__ import annotations (#4393) | 1 年前 | |
fix(series): unable to cast directly from string to fixed_size_binary (#6521) ## Problem Direct casting from Utf8 to FixedSizeBinary(n) fails, even though a two-step cast (Utf8 -> Binary -> FixedSizeBinary(n)) works. ## Root Cause There was no explicit cast bridge for Utf8 -> FixedSizeBinary, so the implementation fell through to an Arrow cast path that does not support this conversion. ## Solution Add a narrow Rust cast fast-path that routes Utf8 -> FixedSizeBinary(n) through Binary, then reuses the existing Binary -> FixedSizeBinary length validation. ## Tests - DAFT_RUNNER=native make test EXTRA_ARGS='-v tests/series/test_cast.py::test_cast_utf8_to_fixed_size_binary tests/series/test_cast.py::test_cast_utf8_to_fixed_size_binary_fails_with_variable_length tests/series/test_cast.py::test_cast_binary_to_fixed_size_binary tests/series/test_cast.py::test_cast_binary_to_fixed_size_binary_fails_with_variable_length' ## Impact - Fixes Issue #3749. - Enables expected direct cast behavior without API or architecture changes. - Preserves error behavior for variable-length values. Co-authored-by: Lord of Abyss <pancx@chinatelecom.cn> | 6 个月前 | |
| 8 个月前 | ||
chore(deps): drop pyarrow 8.0.0 support, bump minimum to >= 15.0.0 (#6378) ## Changes Made Bump minimum PyArrow version from >= 8.0.0 to >= 15.0.0 (44 files, -450 lines). - Update version constraints in pyproject.toml and CI test matrix - Remove _FixSliceOffsets workaround (pyarrow < 12.0.0 struct slice offset bug, fixed upstream) - Remove pyarrow_supports_fixed_shape_tensor() and all conditional branches - Remove obsolete version checks (< 12.0.1, < 13.0.0, >= 9.0.0) and try/except imports - Clean up ~50 pytest.mark.skipif markers and unused PYARROW_GE_* constants in test files Note: _FixEmptyStructArrays is intentionally kept — Daft internally cannot handle empty StructArrays, not just an arrow2 FFI issue. ## Related Issues Closes #6347 | 7 个月前 | |
fix: Use large PyArrow types for casting from Python series (#4625) Co-authored-by: Colin Ho <colin.ho99@gmail.com> | 1 年前 | |
| 2 年前 | ||
chore(deps): drop pyarrow 8.0.0 support, bump minimum to >= 15.0.0 (#6378) ## Changes Made Bump minimum PyArrow version from >= 8.0.0 to >= 15.0.0 (44 files, -450 lines). - Update version constraints in pyproject.toml and CI test matrix - Remove _FixSliceOffsets workaround (pyarrow < 12.0.0 struct slice offset bug, fixed upstream) - Remove pyarrow_supports_fixed_shape_tensor() and all conditional branches - Remove obsolete version checks (< 12.0.1, < 13.0.0, >= 9.0.0) and try/except imports - Clean up ~50 pytest.mark.skipif markers and unused PYARROW_GE_* constants in test files Note: _FixEmptyStructArrays is intentionally kept — Daft internally cannot handle empty StructArrays, not just an arrow2 FFI issue. ## Related Issues Closes #6347 | 7 个月前 | |
refactor(exprs): 1 of N -- simplify expr logic (#4271) ## Changes Made Move is_nan, is_inf, not_nan, and fill_nan from direct Series methods to expression-based implementations. Add RecordBatch creation methods for PySeries lists. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 1 年前 | |
chore: change series literal to list (#4896) ## Changes Made Remove LiteralValue::Series and replace it with a LiteralValue::List. I don't think we use series literals as series anymore so this name makes more sense. Additionally I've changed the datatypes/conversions to reflect that this is actually a list-type scalar value. This is part of a larger change that I'm working on to unify type conversions between Python and Daft. Notes: - Removed test_length_mismatch_all_types because length mismatch is now handled by the RecordBatch constructor so there's a different error, which should happen for any expression, so no need to test it specifically for this one. - Removed test_hash_int_array_with_bad_length - same reason as above. - Removed test_table_expr_is_in_different_input_types. It tested behavior that we did not document, so I'm removing it instead of fixing it. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [x] Documented in API Docs (if applicable) - [x] Documented in User Guide (if applicable) - [x] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [x] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) --------- Co-authored-by: Colin Ho <colin.ho99@gmail.com> | 1 年前 | |
fix(repr): avoid crashing notebook UI when visualizing large strings (#4789) ## Changes Made The following should not crash the Jupyter UI: py import daft df = daft.from_pydict({ 'x': ['foo' * 10000000] }) df.show() This is just a first start, there's more work to do here to lazy fetch data. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> N/A ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 1 年前 | |
chore(deps): drop pyarrow 8.0.0 support, bump minimum to >= 15.0.0 (#6378) ## Changes Made Bump minimum PyArrow version from >= 8.0.0 to >= 15.0.0 (44 files, -450 lines). - Update version constraints in pyproject.toml and CI test matrix - Remove _FixSliceOffsets workaround (pyarrow < 12.0.0 struct slice offset bug, fixed upstream) - Remove pyarrow_supports_fixed_shape_tensor() and all conditional branches - Remove obsolete version checks (< 12.0.1, < 13.0.0, >= 9.0.0) and try/except imports - Clean up ~50 pytest.mark.skipif markers and unused PYARROW_GE_* constants in test files Note: _FixEmptyStructArrays is intentionally kept — Daft internally cannot handle empty StructArrays, not just an arrow2 FFI issue. ## Related Issues Closes #6347 | 7 个月前 | |
feat: support various image hash functions (aHash, dHash, pHash, wHash, crop-resistant hash) for deduplication (#6338) Implements 5 image hashing algorithms for image deduplication: aHash (Average Hash): Compares each pixel to the mean intensity dHash (Difference Hash): Compares adjacent pixel differences pHash (Perceptual Hash): DCT-based frequency domain hashing wHash (Wavelet Hash): Haar wavelet transform-based hashing Crop-resistant Hash: Robust against cropping transformations ## Changes Made Added hash methods to CowImage with helper functions (dct2d_32x32, haar_2d) Added ImageHashOps trait and implementations for ImageArray / FixedShapeImageArray Added series-level functions and ScalarUDF registrations Exposed Python APIs: image_ahash, image_dhash, image_phash, image_whash, image_crop_resistant_hash Added corresponding methods to SeriesImageNamespace Added tests covering all 5 algorithms Fixed-size hashes return FixedSizeBinary(8), crop-resistant hash returns variable-length Binary. ## Related Issues Closes #4889 | 6 个月前 | |
| 8 个月前 | ||
feat: Extend hash variants for xxhash (#5276) | 11 个月前 | |
[RUST] Split series tests (#749) * It is currently difficult to understand test coverage across types and ops because everything is in one file * Splits test_series.py into individual files for each family of operations for easier contribution of tests Co-authored-by: Jay Chia <jaychia94@gmail.com@users.noreply.github.com> | 3 年前 | |
fix(iceberg): partition field names and day transform type (#4181) ## Changes Made - When constructing Iceberg partition fields for writing, use partition field name specified in Iceberg metadata - Day partitioning expression now returns Date type, in order to conform with other Iceberg implementations ## Related Issues Resolves #4157 ## Checklist - [x] Documented in API Docs (if applicable) - [x] Documented in User Guide (if applicable) - [x] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [x] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 1 年前 | |
feat: Add support for Series[start:end] (#5815) ## Changes Made Add support for Series[start:end] <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues #4771 <!-- Link to related GitHub issues, e.g., "Closes #123" --> Co-authored-by: wangzheyan <wangzheyan@bytedance.com> Co-authored-by: Colin Ho <colin.ho99@gmail.com> | 10 个月前 | |
chore(deps): drop pyarrow 8.0.0 support, bump minimum to >= 15.0.0 (#6378) ## Changes Made Bump minimum PyArrow version from >= 8.0.0 to >= 15.0.0 (44 files, -450 lines). - Update version constraints in pyproject.toml and CI test matrix - Remove _FixSliceOffsets workaround (pyarrow < 12.0.0 struct slice offset bug, fixed upstream) - Remove pyarrow_supports_fixed_shape_tensor() and all conditional branches - Remove obsolete version checks (< 12.0.1, < 13.0.0, >= 9.0.0) and try/except imports - Clean up ~50 pytest.mark.skipif markers and unused PYARROW_GE_* constants in test files Note: _FixEmptyStructArrays is intentionally kept — Daft internally cannot handle empty StructArrays, not just an arrow2 FFI issue. ## Related Issues Closes #6347 | 7 个月前 | |
[BUG] Slice array if necessary when casting from list to fixed size list (#2415) Resolves https://github.com/Eventual-Inc/Daft/issues/2394 When we call into_partitions on a dataframe with list arrays, the list arrays are sliced. However, the implementation of [list array slice](https://github.com/Eventual-Inc/Daft/blob/main/src/daft-core/src/array/list_array.rs#L139) simply slices the offsets, and leaves the data unchanged. If the column is then casted to a fixed size list array (and there is no validity bitmap), we will just copy the data, which leads to a series length mismatch error. This PR makes a fix to slice the data during a cast. Example: import daft import numpy as np import pandas as pd N_ROWS = 4 N_ELEMENTS_IN_LIST = 2 N_PARTITIONS = 2 # Create synthetic parquet num = [i for i in range(N_ROWS)] fixed_list = [np.random.randint(0, 100, size=N_ELEMENTS_IN_LIST).tolist() for _ in range(N_ROWS)] df = pd.DataFrame({"fixed_list": fixed_list, "num": num}) df.to_parquet("fixed_list.parquet", engine="pyarrow") # Read and cast the relevant column df = daft.read_parquet("fixed_list.parquet").into_partitions(N_PARTITIONS) dtype = daft.DataType.from_numpy_dtype("int8") fixed_size_list_dtype = daft.DataType.fixed_size_list(dtype, N_ELEMENTS_IN_LIST) df = df.with_column("fixed_list", df["fixed_list"].cast(fixed_size_list_dtype)) iter_df = iter(df) s = next(iter_df) print(s) Todo: - [x] - Add tests | 2 年前 | |
[CHORE] refactors for ruff [1/n] (#2120) | 2 年前 | |
chore(deps): drop pyarrow 8.0.0 support, bump minimum to >= 15.0.0 (#6378) ## Changes Made Bump minimum PyArrow version from >= 8.0.0 to >= 15.0.0 (44 files, -450 lines). - Update version constraints in pyproject.toml and CI test matrix - Remove _FixSliceOffsets workaround (pyarrow < 12.0.0 struct slice offset bug, fixed upstream) - Remove pyarrow_supports_fixed_shape_tensor() and all conditional branches - Remove obsolete version checks (< 12.0.1, < 13.0.0, >= 9.0.0) and try/except imports - Clean up ~50 pytest.mark.skipif markers and unused PYARROW_GE_* constants in test files Note: _FixEmptyStructArrays is intentionally kept — Daft internally cannot handle empty StructArrays, not just an arrow2 FFI issue. ## Related Issues Closes #6347 | 7 个月前 | |
| 7 个月前 | ||
chore(deps): drop pyarrow 8.0.0 support, bump minimum to >= 15.0.0 (#6378) ## Changes Made Bump minimum PyArrow version from >= 8.0.0 to >= 15.0.0 (44 files, -450 lines). - Update version constraints in pyproject.toml and CI test matrix - Remove _FixSliceOffsets workaround (pyarrow < 12.0.0 struct slice offset bug, fixed upstream) - Remove pyarrow_supports_fixed_shape_tensor() and all conditional branches - Remove obsolete version checks (< 12.0.1, < 13.0.0, >= 9.0.0) and try/except imports - Clean up ~50 pytest.mark.skipif markers and unused PYARROW_GE_* constants in test files Note: _FixEmptyStructArrays is intentionally kept — Daft internally cannot handle empty StructArrays, not just an arrow2 FFI issue. ## Related Issues Closes #6347 | 7 个月前 | |
feat: convert and replace timezone (#6106) ## Changes Made - When converting time zone: - Output error when from_timezone is not provided for timestamp without tz info. - Ignore from_timezone if timestamp already has tz info. - When replacing time zone, drop timezone in timestamp if timezone is not provided. - Create new ParsedTimezone enum to handle different timezone types (offset and tz name). - Use new Daft array building semantics. ## Related Issues Closes #4096. | 6 个月前 | |
chore(deps): drop pyarrow 8.0.0 support, bump minimum to >= 15.0.0 (#6378) ## Changes Made Bump minimum PyArrow version from >= 8.0.0 to >= 15.0.0 (44 files, -450 lines). - Update version constraints in pyproject.toml and CI test matrix - Remove _FixSliceOffsets workaround (pyarrow < 12.0.0 struct slice offset bug, fixed upstream) - Remove pyarrow_supports_fixed_shape_tensor() and all conditional branches - Remove obsolete version checks (< 12.0.1, < 13.0.0, >= 9.0.0) and try/except imports - Clean up ~50 pytest.mark.skipif markers and unused PYARROW_GE_* constants in test files Note: _FixEmptyStructArrays is intentionally kept — Daft internally cannot handle empty StructArrays, not just an arrow2 FFI issue. ## Related Issues Closes #6347 | 7 个月前 | |
feat: Add string strip function (#6372) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> Add strip() function to Daft's string functions. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> https://github.com/Eventual-Inc/Daft/issues/3792 | 6 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 3 年前 | ||
| 1 年前 | ||
| 1 年前 | ||
| 6 个月前 | ||
| 8 个月前 | ||
| 7 个月前 | ||
| 1 年前 | ||
| 2 年前 | ||
| 7 个月前 | ||
| 1 年前 | ||
| 1 年前 | ||
| 1 年前 | ||
| 7 个月前 | ||
| 6 个月前 | ||
| 8 个月前 | ||
| 11 个月前 | ||
| 3 年前 | ||
| 1 年前 | ||
| 10 个月前 | ||
| 7 个月前 | ||
| 2 年前 | ||
| 2 年前 | ||
| 7 个月前 | ||
| 7 个月前 | ||
| 7 个月前 | ||
| 6 个月前 | ||
| 7 个月前 | ||
| 6 个月前 |