| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
fix: remove use_safetensors=True to allow model fallback (#6489) ## Changes Made Forcing safetensors was causing unit tests to fail because these weights weren't available; removing this let's the resolution fallback to pytorch ## Related Issues https://github.com/Eventual-Inc/Daft/actions/runs/23555955545/job/68582692936 | 6 个月前 | |
feat!: add iceberg table support with the gravitino catalog (#6509) ## Changes Made This refactors the Gravitino catalog implementation to match our Glue implementation. It looks at table information to determine which concrete Table implementation to use. This also refactors the Catalog creation a little bit to once again match Glue. I'm not proud of the Catalog.from_* thing with the objects, and I like the pyiceberg choice of the top-level load_ methods, hence why later implementations like Glue use that pattern, so I've updated Gravitino to follow this. This PR also properly internalizes/encapsulates the client, I wasn't interested in having our own Gravitino client vs. using the python one, but the customer required this -- so I just hid the client and updated the docs accordingly. Also a tiny fix on an old typo of mine from the video reader: read_glob_paths/from_glob_paths. ## Related Issues - Closes #6318 - Closes #6319 | 6 个月前 | |
chore: Move Flight Server to Rust (#6519) ## Changes Made Provide in Rust so we have direct access to it for optimizations | 6 个月前 | |
fix(dataframe): explain plan does not make sense when using a write_ op (#6564) ## Problem After calling write_*, explain(show_all=True) collapses to an in-memory source and loses upstream plan context (e.g. filters before write). ## Root Cause write_parquet / write_csv / write_json build the returned DataFrame from write_df._builder after collect(). With cached results, _builder resolves to an in-memory scan builder. ## Solution Build the returned DataFrame from write_df._get_current_builder() so the original logical plan is preserved, while keeping result cache/preview/metadata wiring unchanged. ## Tests - DAFT_RUNNER=native make test EXTRA_ARGS="-v tests/dataframe/test_explain.py::test_explain_after_write_preserves_upstream_plan" ## Impact Fixes explainability for write chains by preserving upstream logical plan visibility without changing write output behavior. --------- Co-authored-by: Lord of Abyss <pancx@chinatelecom.cn> | 6 个月前 | |
fix: Support Union and Optional in DataType.infer_from_type (#6304) ## Changes Made Noticed it when I was writing some other test, and got really annoying so I just dealt with it. Note that the workarounds seem kind of bad, so I might take a better look at it at another point. Just a temp solution. | 7 个月前 | |
chore: Move Flight Server to Rust (#6519) ## Changes Made Provide in Rust so we have direct access to it for optimizations | 6 个月前 | |
fix: short-circuit evaluation for coalesce (#6525) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> Fixes #4069 by adding proper short-circuit behavior to coalesce. - Added Expr::Coalesce with short-circuit logic - Implemented early-exit in record batch evaluation - Preserved semantics during UDF optimization - Added short-circuit tests ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> Closes #4069 --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 6 个月前 | |
fix(video): add missing pillow dependency for video keyframes (#6355) Based on @everettVT's branch. video_keyframes() fails with a confusing DaftError::ValueError Need at least 1 series to perform concat when Pillow is not installed. This primarily affects Python 3.12+ users because the daft[video] extra didn't include pillow as a dependency, and Pillow isn't transitively installed on those versions. The root cause is that VideoFile.keyframes() calls frame.to_image() (which requires PIL) without checking for its availability first. When every row fails, the errors get swallowed and ListArray::try_from hits an empty concat instead of surfacing the actual ModuleNotFoundError. To fix this, we add pillow==12.1.1 to the video extra in pyproject.toml (matching the pin used by the google, openai, and transformers extras) and add an early import check in VideoFile.keyframes() that raises a clear ImportError directing users to pip install daft[video]. Fixes #6064 Co-authored-by: Everett Kleven <ekleven@vangelis.tech> | 7 个月前 | |
fix: short-circuit evaluation for coalesce (#6525) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> Fixes #4069 by adding proper short-circuit behavior to coalesce. - Added Expr::Coalesce with short-circuit logic - Implemented early-exit in record batch evaluation - Preserved semantics during UDF optimization - Added short-circuit tests ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> Closes #4069 --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 6 个月前 | |
docs: Enable Linting of docstrings (#3506) | 1 年前 | |
chore(io): replace python source shim on the rust side (#6556) ## Motivation This removes the python ScanOperator shim in favor of a ScanOperator implementation in Rust which is backed by either a Python *or* Rust data source. This allows us to move the shim one step lower, and now we actually have Rust traits + python bridges for the DataSource and DataSourceTask interfaces. This is an incremental step towards full streaming sources and consolidating the various scanning types and implementations to this interface. ## Changes Made - Removes python _DataSourceShim - Cleanup visibility modifiers and conversion methods for daft-catalog Catalog, Table, Provider - Adds native factory methods for native data source tasks, which are a work-in-progress and just today's scan tasks. - We have a Rust PyDataSourceWrapper trait which is able to implement our existing ScanOperator backed by a Python DataSource | 6 个月前 | |
feat: add support for writing _SUCCESS files in parquet operations (#6090) ## Changes Made - Add support for writing a _SUCCESS file upon successful completion of parquet write operations. - Add parameter to existing write_parquet method. df.write_parquet("/path/to/output", write_success_file=True) <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues Closes #4085 <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 6 个月前 | |
feat: Add pow expression (#5237) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues #4704 <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) --------- Co-authored-by: Colin Ho <colin.ho99@gmail.com> | 10 个月前 | |
fix: respect scheme aliases in Iceberg writes (#6512) ## Changes Made Iceberg uses a FileProvider interface to create the data_file full URI matching the scheme of the table location itself. However Daft doesn't have a single interface like this, so our read path was supporting custom schemes while the write path wasn't using the aliases. This meant we were writing s3:// in the manifests instead of the custom scheme (bad). This fixes that by applying the aliases to the paths in our python filesystem abstraction which is what our Iceberg writer uses. Because this is only python, you can actually monkeypatch this right now without waiting on a new daft release. https://gist.github.com/rchowell/7027d8c23b3674f4c57fe796d8ae429b ## Related Issues If you previously tried to write an Iceberg table with a different object-store scheme, then the manifests are using s3:// rather than your object-store scheme. | 6 个月前 | |
chore: Move Flight Server to Rust (#6519) ## Changes Made Provide in Rust so we have direct access to it for optimizations | 6 个月前 | |
fix(sql): fall back to single scan when partition bounds are degenerate (#6482) ## Changes Made Updated SQL partition planning in read_sql to detect degenerate partition bounds and avoid creating empty partition scans. ## Related Issues | 6 个月前 | |
feat(subscriber): add Event enum and on_event dispatch for subscriber (#6508) ## Changes Made Introduces Event transport enum with OperatorStart/OperatorEnd/Stats variants and a single on_event async method on the Subscriber trait. Ports RuntimeStatsManager to emit all events through on_event, replacing direct per-method calls. Updates dashboard, debug, and python subscribers to implement on_event. | 6 个月前 | |
feat(udf): support ray_options and resource overrides in UDF v2 (#5982) ## Changes Made This PR adds support for passing ray_options and overriding resource requirements in Daft UDF v2 (@daft.func and @daft.cls). Key changes: - Update row_wise_udf and batch_udf bindings to accept ray_options and cpus. - Add override_options and with_concurrency methods to Func class in daft/udf/udf_v2.py for dynamic configuration. - Propagate these options from Python to the Rust Logical Plan. - Update type stubs (.pyi) to match new Rust signatures. <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 7 个月前 | |
chore: bump mypy and ruff in pre-commit (#5836) ## Changes Made Bump mypy to 1.19.1 and ruff to 0.14.10 and resolve pre-commit errors. Rationale: my local ruff version is much higher than that in the project pre-commit, so I am getting a lot of linter warnings in my IDE. It is also about time to upgrade lints to utilize newer Python language features. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 9 个月前 | |
| 6 个月前 | ||
feat: DataFrame.from_files (#6507) | 6 个月前 | |
chore: bump mypy and ruff in pre-commit (#5836) ## Changes Made Bump mypy to 1.19.1 and ruff to 0.14.10 and resolve pre-commit errors. Rationale: my local ruff version is much higher than that in the project pre-commit, so I am getting a lot of linter warnings in my IDE. It is also about time to upgrade lints to utilize newer Python language features. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 9 个月前 | |
chore: Upgrade Ruff ruleset to 3.9 and add from __future__ import annotations (#4393) | 1 年前 | |
chore: Make the partition threshold to use pre_shuffle_merge configurable (#6356) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> When shuffle_algorithm=auto, it currently calculates the geometric mean value of partitions on the Map side and Reduce side, and hardcodes the use of pre_shuffle_merge when the value exceeds 200. This PR turns this threshold into a configuration item to enhance flexibility in task tuning. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> Signed-off-by: plotor <zhenchao.wang@hotmail.com> Co-authored-by: Cory Grinstead <cory.grinstead@gmail.com> | 7 个月前 | |
chore: Remove runner from context (#5628) ## Changes Made Remove runner methods from daft.context, as they are now decoupled from the context. Changed all uses of daft.context.set_runner_... to daft.set_runner_... ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 10 个月前 | |
chore(deps): drop pyarrow 8.0.0 support, bump minimum to >= 15.0.0 (#6378) ## Changes Made Bump minimum PyArrow version from >= 8.0.0 to >= 15.0.0 (44 files, -450 lines). - Update version constraints in pyproject.toml and CI test matrix - Remove _FixSliceOffsets workaround (pyarrow < 12.0.0 struct slice offset bug, fixed upstream) - Remove pyarrow_supports_fixed_shape_tensor() and all conditional branches - Remove obsolete version checks (< 12.0.1, < 13.0.0, >= 9.0.0) and try/except imports - Clean up ~50 pytest.mark.skipif markers and unused PYARROW_GE_* constants in test files Note: _FixEmptyStructArrays is intentionally kept — Daft internally cannot handle empty StructArrays, not just an arrow2 FFI issue. ## Related Issues Closes #6347 | 7 个月前 | |
| 6 个月前 | ||
feat: Implement retry-after mechanism for model apis and udfs (#5769) ## Changes Made Adds a RetryAfterException that can be raised from a UDF, which will supply the UDF retry mechanism with a delay to sleep. This allows us to respect the retry-afters provided in headers from 429 and 503 errors from clients, such as our openai and google clients in the model apis. By default, all of our model apis should already be configured with max_retries = 3, and our exponential backoff has an initial delay of 100ms. It might be worth exploring in the future how we may want to more easily expose these configurations to users. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 10 个月前 | |
fix: Fix async udf with use_process (#5457) ## Changes Made async udfs with use_process=True were failing because the udf process did not have the background event loop initialized. This pr adds a get_or_init for the event loop so that it can be initialized for the udf process. ## Related Issues Closes https://github.com/Eventual-Inc/Daft/issues/5456 ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 11 个月前 | |
chore: Upgrade Ruff ruleset to 3.9 and add from __future__ import annotations (#4393) | 1 年前 | |
chore: bump mypy and ruff in pre-commit (#5836) ## Changes Made Bump mypy to 1.19.1 and ruff to 0.14.10 and resolve pre-commit errors. Rationale: my local ruff version is much higher than that in the project pre-commit, so I am getting a lot of linter warnings in my IDE. It is also about time to upgrade lints to utilize newer Python language features. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 9 个月前 | |
fix: respect scheme aliases in Iceberg writes (#6512) ## Changes Made Iceberg uses a FileProvider interface to create the data_file full URI matching the scheme of the table location itself. However Daft doesn't have a single interface like this, so our read path was supporting custom schemes while the write path wasn't using the aliases. This meant we were writing s3:// in the manifests instead of the custom scheme (bad). This fixes that by applying the aliases to the paths in our python filesystem abstraction which is what our Iceberg writer uses. Because this is only python, you can actually monkeypatch this right now without waiting on a new daft release. https://gist.github.com/rchowell/7027d8c23b3674f4c57fe796d8ae429b ## Related Issues If you previously tried to write an Iceberg table with a different object-store scheme, then the manifests are using s3:// rather than your object-store scheme. | 6 个月前 | |
| 6 个月前 | ||
| 9 个月前 | ||
feat: use human-readable query names instead of UUIDs (#6310) ## Summary - Add daft/naming.py module that generates Docker-style random names (e.g., eager-phoenix-a4f821) using 55 adjectives × 55 nouns × 6 hex digits (~48 billion combinations) - Replace uuid.uuid4() with generate_query_name() in both native and Ray runners - Remove dashboard truncation on query ID column since the new names are short enough to display fully 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> | 7 个月前 | |
[Images] [1/N] Logical type for variable-shaped and fixed-shaped images. (#955) This PR adds a logical type for variable-shaped and fixed-shape images. | 3 年前 | |
feat: add dual-send telemetry to osstelemetry.io (#6540) ## Summary Send telemetry data to osstelemetry.io alongside Scarf for side-by-side comparison during evaluation. ## Changes - ** daft/scarf_telemetry.py**: After the existing Scarf GET request, send the same data to https://osstelemetry.io/data with activity_type (mapped from endpoint name) and package=daft as additional params. The osstelemetry.io call is independent - failures are silently caught and do not affect existing Scarf telemetry. - **docs/telemetry.md**: Updated to mention osstelemetry.io alongside Scarf. - **tests/test_scarf_telemetry.py**: Updated test_scarf_telemetry_basic to verify both the Scarf URL (first call) and the osstelemetry.io URL (second call) using call_args_list instead of call_args. ## Request format The existing Scarf request: GET https://daft.gateway.scarf.sh/daft-import?version=0.3.0&platform=Darwin&python=3.11&arch=arm64 The new osstelemetry.io request: GET https://osstelemetry.io/data?activity_type=daft-import&package=daft&version=0.3.0&platform=Darwin&python=3.11&arch=arm64 For runner events, both include the runner param (e.g. runner=native or runner=ray): GET https://daft.gateway.scarf.sh/daft-runner?version=0.3.0&platform=Darwin&python=3.11&arch=arm64&runner=native GET https://osstelemetry.io/data?activity_type=daft-runner&package=daft&version=0.3.0&platform=Darwin&python=3.11&arch=arm64&runner=native ## Details - Opt-out flags (SCARF_NO_ANALYTICS, DO_NOT_TRACK, DAFT_ANALYTICS_ENABLED) apply to both endpoints - Both calls run in the same daemon thread, so no additional threads are created - Dev builds still skip all telemetry ## Testing Tested locally by: 1. Creating a test branch with the dev-build check commented out and both URLs pointed to localhost servers (port 8001 for Scarf, port 8002 for osstelemetry) 2. Running a simple HTTP server that logs all incoming request paths and query params 3. Running import daft to trigger the import telemetry event Both servers received the expected requests: [SCARF] GET /daft-import version: 0.3.0-dev0 platform: Darwin python: 3.11 arch: arm64 [OSSTELEMETRY] GET /data activity_type: daft-import package: daft version: 0.3.0-dev0 platform: Darwin python: 3.11 arch: arm64 | 6 个月前 | |
chore: bump mypy and ruff in pre-commit (#5836) ## Changes Made Bump mypy to 1.19.1 and ruff to 0.14.10 and resolve pre-commit errors. Rationale: my local ruff version is much higher than that in the project pre-commit, so I am getting a lot of linter warnings in my IDE. It is also about time to upgrade lints to utilize newer Python language features. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 9 个月前 | |
feat: support various image hash functions (aHash, dHash, pHash, wHash, crop-resistant hash) for deduplication (#6338) Implements 5 image hashing algorithms for image deduplication: aHash (Average Hash): Compares each pixel to the mean intensity dHash (Difference Hash): Compares adjacent pixel differences pHash (Perceptual Hash): DCT-based frequency domain hashing wHash (Wavelet Hash): Haar wavelet transform-based hashing Crop-resistant Hash: Robust against cropping transformations ## Changes Made Added hash methods to CowImage with helper functions (dct2d_32x32, haar_2d) Added ImageHashOps trait and implementations for ImageArray / FixedShapeImageArray Added series-level functions and ScalarUDF registrations Exposed Python APIs: image_ahash, image_dhash, image_phash, image_whash, image_crop_resistant_hash Added corresponding methods to SeriesImageNamespace Added tests covering all 5 algorithms Fixed-size hashes return FixedSizeBinary(8), crop-resistant hash returns variable-length Binary. ## Related Issues Closes #4889 | 6 个月前 | |
feat: adds support for daft native extensions via stable C ABI (#6301) ## About This PR creates a native extension framework for Daft and is modeled after Postgres. Extension authors ship a normal pip-installable python package with a bundled dylib (instructions in tutorial). Users import python functions to get full IDE support and the native implementations are linked by loading the extension into a session. ## Design **Crates** - daft-ext - Single dependency for extension authors (published). - daft-ext-abi - Stable #[repr(C)] types defining the ABI contract. - daft-ext-core - Public Rust SDK types (DaftScalarFunction, traits, error types). - daft-ext-macros - Proc macro (#[daft_extension]). - daft-ext-internal - Host-side adapters (ScalarFunctionHandle, module loader). Not published. **Functions** ScalarUDF is Daft's internal trait for scalar functions which cannot cross a dlopen boundary because it uses Rust trait objects with unstable ABI layouts - so FFI_ScalarFunction in daft-ext-abi is the stable C ABI version of this. ScalarFunctionHandle in daft-ext-internal wraps a FFI_ScalarFunction and implements both ScalarUDF and ScalarFunctionFactory. Data crosses the extension boundary using the Arrow C Data Interface (FFI_ArrowArray + FFI_ArrowSchema) and is zero-copy. The extension FFI uses arrow::ffi directly, not the common-arrow-ffi crate which is coupled to PyO3. **Installation** We load shared libs once into the process when someone calls .load_extension on the session. The top-level daft.load_extension will load the given extension module into the active session (from context). All defined functions are scoped to the session in which the extension is loaded. ## Changes - Creates the daft-ext-* crates defined above. - Defines the stable C ABI which minimally wraps arrow ffi. - Creates the daft-ext-internal to bridge host->abi. - Creates the daft-ext-core to bridge extension->abi. - Implements internal Daft traits using the daft-ext-internal types. - Adds a dvector example extension which is a pgvector clone. - Adds a hello example extension with a tutorial document. - Extends the session to support native function registration. - Adds the get_function method for session-backed function resolution mirroring the SQL implementation. Can add more clarity upon reviews. ## Examples - [Hello, World](https://github.com/Eventual-Inc/Daft/tree/564548a93fa99180e4e17f51049e5db60a5949cf/examples/hello) - [dvector (pgvector)](https://github.com/Eventual-Inc/Daft/tree/564548a93fa99180e4e17f51049e5db60a5949cf/examples/dvector) python import daft # Step 1. Import your extension module import hello # Step 2. Load the extension into the current daft session daft.load_extension(hello) # Step 3. Use in your dataframe! df = daft.from_pydict({"name": ["John", "Paul"]}) df = df.select(hello.greet(df["name"])) See the actual [greet rust implementation](https://github.com/Eventual-Inc/Daft/blob/564548a93fa99180e4e17f51049e5db60a5949cf/examples/hello/src/lib.rs). ## Guide See [Daft Extension Guide](https://github.com/Eventual-Inc/Daft/blob/564548a93fa99180e4e17f51049e5db60a5949cf/docs/extensions/index.md). | 7 个月前 | |
chore(deps): drop pyarrow 8.0.0 support, bump minimum to >= 15.0.0 (#6378) ## Changes Made Bump minimum PyArrow version from >= 8.0.0 to >= 15.0.0 (44 files, -450 lines). - Update version constraints in pyproject.toml and CI test matrix - Remove _FixSliceOffsets workaround (pyarrow < 12.0.0 struct slice offset bug, fixed upstream) - Remove pyarrow_supports_fixed_shape_tensor() and all conditional branches - Remove obsolete version checks (< 12.0.1, < 13.0.0, >= 9.0.0) and try/except imports - Clean up ~50 pytest.mark.skipif markers and unused PYARROW_GE_* constants in test files Note: _FixEmptyStructArrays is intentionally kept — Daft internally cannot handle empty StructArrays, not just an arrow2 FFI issue. ## Related Issues Closes #6347 | 7 个月前 | |
chore: bump mypy and ruff in pre-commit (#5836) ## Changes Made Bump mypy to 1.19.1 and ruff to 0.14.10 and resolve pre-commit errors. Rationale: my local ruff version is much higher than that in the project pre-commit, so I am getting a lot of linter warnings in my IDE. It is also about time to upgrade lints to utilize newer Python language features. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 9 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 6 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 7 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 7 个月前 | ||
| 6 个月前 | ||
| 1 年前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 10 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 7 个月前 | ||
| 9 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 9 个月前 | ||
| 1 年前 | ||
| 7 个月前 | ||
| 10 个月前 | ||
| 7 个月前 | ||
| 6 个月前 | ||
| 10 个月前 | ||
| 11 个月前 | ||
| 1 年前 | ||
| 9 个月前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 9 个月前 | ||
| 7 个月前 | ||
| 3 年前 | ||
| 6 个月前 | ||
| 9 个月前 | ||
| 6 个月前 | ||
| 7 个月前 | ||
| 7 个月前 | ||
| 9 个月前 |