| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 9 个月前 | ||
feat: support various image hash functions (aHash, dHash, pHash, wHash, crop-resistant hash) for deduplication (#6338) Implements 5 image hashing algorithms for image deduplication: aHash (Average Hash): Compares each pixel to the mean intensity dHash (Difference Hash): Compares adjacent pixel differences pHash (Perceptual Hash): DCT-based frequency domain hashing wHash (Wavelet Hash): Haar wavelet transform-based hashing Crop-resistant Hash: Robust against cropping transformations ## Changes Made Added hash methods to CowImage with helper functions (dct2d_32x32, haar_2d) Added ImageHashOps trait and implementations for ImageArray / FixedShapeImageArray Added series-level functions and ScalarUDF registrations Exposed Python APIs: image_ahash, image_dhash, image_phash, image_whash, image_crop_resistant_hash Added corresponding methods to SeriesImageNamespace Added tests covering all 5 algorithms Fixed-size hashes return FixedSizeBinary(8), crop-resistant hash returns variable-length Binary. ## Related Issues Closes #4889 | 6 个月前 | |
feat(expressions): support count(mode='all') without expr (#6358) ## Changes Made This PR adds support for calling daft.functions.count(mode="all") without passing an expression, enabling free-standing row-count aggregations similar to SQL COUNT(*). ### What changed - Updated daft.functions.count in daft/functions/agg.py to accept an optional expr argument. - Added explicit behavior for expr=None: - mode="all" is allowed and maps to count(*) semantics via col("*"). - mode="valid" or mode="null" now raises a clear ValueError when no expression is provided. - Preserved backward compatibility for all existing count(expr, mode=...) and count(lit(...)) usages. ### Tests added In tests/dataframe/test_aggregations.py: - Global aggregation with count(mode="all") and no expression. - Grouped aggregation with count(mode="all") and no expression. - Empty DataFrame behavior (0 result). - Error path validation for no-expression calls with non-all modes. ### Validation run - DAFT_RUNNER=native make test EXTRA_ARGS="-v tests/dataframe/test_aggregations.py" - DAFT_RUNNER=native make test EXTRA_ARGS="-v tests/sql/test_sql.py tests/sql/test_exprs.py" ## Related Issues Closes #5526 Co-authored-by: Lord of Abyss <pancx@chinatelecom.cn> | 7 个月前 | |
docs: Update daft.File API and add docstrings, improve error handling in tests (#5877) - Updated error messages in AudioFile and related functions to provide clearer instructions for installing required modules (soundfile and librosa). - Enhanced docstrings to include detailed metadata descriptions for audio files. - Added new documentation files for daft.File types and updated existing documentation links for better navigation. - Improved test cases to ensure informative error handling when dependencies are missing. This change aims to improve user experience by providing clearer guidance on required dependencies and enhancing the overall documentation structure. | 9 个月前 | |
perf: Rewrite decode(..., "utf-8") as a cast (#6537) ## Changes Made We have LogicalPlan optimizations for simplifying casting expressions. For example, if we cast a string -> string, we eliminate the cast operation. We don't with decode because the behavior is encoded within the expression definition. Note, we could special-case decode in the optimizer, but I think its cleaner to do it in Python because parsing is kind of ugly. Lmk if you disagree though. I think its fair to say that converting decodes to cast may be confusing in explain plans. Also, I think this will help ClickBench a bunch, since this was blocking filter pushdowns --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 6 个月前 | |
feat: migrate first batch of daft functions (#5086) ## Changes Made First phase of our daft.functions migration + Expression namespacing deprecation! These changes should all be uncontroversial, and all of [the comments about them](https://www.notion.so/Expressions-Migration-Tasks-246091bd8634806a8f24eb573d8d3b3e?source=copy_link) have been resolved. Thus, I'm hoping that the reviews for this PR will be a little easier since they should only need to focus on the structural changes rather than the naming ones. I'm leaving the ones with unresolved discussions for future PRs, but this covers something like 95% of expressions. In addition, since the existing tests call the expression methods, which in turn call daft.functions, passing those tests should indicate both that the new functions work and that the existing methods have not been broken. All of the functions are straightforward Python-only changes except for two: - daft.functions.length has to work on binary, string, as well as list types. prior to that they were three separate functions, so I had to write a new scalar function on the Rust side - similarly, daft.functions.concat works for both string and binary, whereas the original functions were separate. I implemented this one by implementing Expression.__add__ between two binary expressions, and then just used the add implementation in the new concat scalar function ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [x] Documented in API Docs (if applicable) - [x] Documented in User Guide (if applicable) - [x] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [x] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) --------- Co-authored-by: Srinivas Lade <srinulade1@gmail.com> | 1 年前 | |
chore: Remove expression namespaces (#5619) ## Changes Made Bye bye ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 10 个月前 | |
feat: convert and replace timezone (#6106) ## Changes Made - When converting time zone: - Output error when from_timezone is not provided for timestamp without tz info. - Ignore from_timezone if timestamp already has tz info. - When replacing time zone, drop timezone in timestamp if timezone is not provided. - Create new ParsedTimezone enum to handle different timezone types (offset and tz name). - Use new Daft array building semantics. ## Related Issues Closes #4096. | 6 个月前 | |
| 8 个月前 | ||
feat: Add guess_mime_type scalar expression for MIME type detection from bytes (#5883) | 9 个月前 | |
feat: support various image hash functions (aHash, dHash, pHash, wHash, crop-resistant hash) for deduplication (#6338) Implements 5 image hashing algorithms for image deduplication: aHash (Average Hash): Compares each pixel to the mean intensity dHash (Difference Hash): Compares adjacent pixel differences pHash (Perceptual Hash): DCT-based frequency domain hashing wHash (Wavelet Hash): Haar wavelet transform-based hashing Crop-resistant Hash: Robust against cropping transformations ## Changes Made Added hash methods to CowImage with helper functions (dct2d_32x32, haar_2d) Added ImageHashOps trait and implementations for ImageArray / FixedShapeImageArray Added series-level functions and ScalarUDF registrations Exposed Python APIs: image_ahash, image_dhash, image_phash, image_whash, image_crop_resistant_hash Added corresponding methods to SeriesImageNamespace Added tests covering all 5 algorithms Fixed-size hashes return FixedSizeBinary(8), crop-resistant hash returns variable-length Binary. ## Related Issues Closes #4889 | 6 个月前 | |
fix: add ignore_empty_and_null parameter to .explode() (#6047) Add an optional ignore_empty parameter to explode() that allows users to drop rows with empty lists or null values instead of producing null rows. - Default behavior (ignore_empty=False) is unchanged: empty lists and nulls produce a single null row - When ignore_empty=True: empty lists and nulls are dropped entirely, matching PyArrow/DuckDB behavior ## Changes Made - Added ignore_empty parameter to Python API (DataFrame.explode, Expression.explode, daft.functions.explode) - Propagated the parameter through logical plan, physical plan, and execution layers - Updated Rust core implementation in daft-functions-list and daft-recordbatch - Added 11 test cases covering basic usage and edge cases (all empty lists, all nulls, FixedSizeList, nested lists, etc.) ## Related Issues #5752 --------- Co-authored-by: Srinivas Lade <srinulade1@gmail.com> Co-authored-by: desmondcheongzx <desmondcheongzx@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> | 8 个月前 | |
chore: deprecate llm_generate function (#5603) ## Changes Made deprecates llm_generate in favor of prompt ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 11 个月前 | |
fix: short-circuit evaluation for coalesce (#6525) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> Fixes #4069 by adding proper short-circuit behavior to coalesce. - Added Expr::Coalesce with short-circuit logic - Implemented early-exit in record batch evaluation - Preserved semantics during UDF optimization - Added short-circuit tests ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> Closes #4069 --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 6 个月前 | |
chore: Remove expression namespaces (#5619) ## Changes Made Bye bye ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 10 个月前 | |
feat: migrate nested constructors, get, slice, to_unix_epoch, partitioning functions (#5233) | 1 年前 | |
fix(cast): handle whitespace in string-to-number casting (#5955) ## Changes Made - Handle whitespace trimming for Utf8 -> numeric casting in cast.rs - Add Rust unit tests for whitespace handling in cast operations - Simplify run_process to return raw subprocess stdout - Remove | tr -d ' ' workaround from tests since cast now handles whitespace ## Motivation String-to-number casting in Daft was strict about whitespace, returning None for inputs like " 3 ". This differed from: - Python's int() - Pandas - NumPy - DuckDB All of which handle leading/trailing whitespace gracefully. ### Comparison (Numeric) | Engine | ' 3 ' → int | |--------|-----------------| | Python int() | 3 | | Pandas | 3 | | DuckDB | 3 | | Polars | None | | Daft (before) | None | | **Daft (after)** | **3** | ## Implementation The fix is implemented in src/daft-core/src/array/ops/cast.rs: - Before calling arrow2's cast, check if we're casting Utf8 -> numeric - If so, create a trimmed copy of the string array - Pass the trimmed data to arrow2's existing cast logic Trade-off: Temporarily duplicates string data during cast, but gains code reuse and maintainability. ## Scope This PR only affects **numeric** casting (int, float). ### Date Parsing Date parsing already handles whitespace correctly - chrono::NaiveDate::from_str (used by arrow2) is lenient with whitespace. Added a test to document this existing behavior. ## Related Issues Fixes the root cause of #5942 Related to #5738 | 9 个月前 | |
| 8 个月前 | ||
feat: added concat_ws function (#6543) GitHub tracking issue: [Eventual-Inc/Daft#3792](https://github.com/Eventual-Inc/Daft/issues/3792) Joining strings with a separator is one of the most common operations in data pipelines - building file paths, composite keys, CSV-like fields, display strings. Daft has concat today, but it propagates nulls and has no separator support. Users who need concat_ws semantics currently have to manually interleave lit(sep) between columns and handle nulls themselves. | 6 个月前 | |
refactor: Cleanup Dtype Names (#5400) | 1 年前 | |
chore: Remove expression namespaces (#5619) ## Changes Made Bye bye ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 10 个月前 | |
fix: Clean up UDF display name in progress bar and plans (#5810) ## Changes Made Fix a couple of things with how names for UDFs are displayed in progress bars: * Don't use the unique id for the name, use a clean UDF name * Expose a name_override parameter so we can override built-in UDFs with custom names instead. Useful for the AI functions <img width="722" height="57" alt="Screenshot 2025-12-12 at 3 15 10 PM" src="https://github.com/user-attachments/assets/694cb578-aa1e-446e-93d8-4ea143e72aca" /> | 9 个月前 | |
chore: bump mypy and ruff in pre-commit (#5836) ## Changes Made Bump mypy to 1.19.1 and ruff to 0.14.10 and resolve pre-commit errors. Rationale: my local ruff version is much higher than that in the project pre-commit, so I am getting a lot of linter warnings in my IDE. It is also about time to upgrade lints to utilize newer Python language features. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 9 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 9 个月前 | ||
| 6 个月前 | ||
| 7 个月前 | ||
| 9 个月前 | ||
| 6 个月前 | ||
| 1 年前 | ||
| 10 个月前 | ||
| 6 个月前 | ||
| 8 个月前 | ||
| 9 个月前 | ||
| 6 个月前 | ||
| 8 个月前 | ||
| 11 个月前 | ||
| 6 个月前 | ||
| 10 个月前 | ||
| 1 年前 | ||
| 9 个月前 | ||
| 8 个月前 | ||
| 6 个月前 | ||
| 1 年前 | ||
| 10 个月前 | ||
| 9 个月前 | ||
| 9 个月前 |