| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
chore: use dedicated OSS AWS account (#6442) ## Changes Made Update to use new dedicated OSS AWS account: - S3 buckets for GitHub artifacts, and public datasets - CloudFront distribution Requires: - Update ACTIONS_AWS_IAM_ROLE GitHub secret ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 6 个月前 | |
feat: add clickhouse data sink (#4850) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> Add the clickhouse data sink, and use it to writes the DataFrame to a ClickHouse table ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) --------- Co-authored-by: Desmond Cheong <desmondcheongzx@gmail.com> | 1 年前 | |
chore(deps): drop pyarrow 8.0.0 support, bump minimum to >= 15.0.0 (#6378) ## Changes Made Bump minimum PyArrow version from >= 8.0.0 to >= 15.0.0 (44 files, -450 lines). - Update version constraints in pyproject.toml and CI test matrix - Remove _FixSliceOffsets workaround (pyarrow < 12.0.0 struct slice offset bug, fixed upstream) - Remove pyarrow_supports_fixed_shape_tensor() and all conditional branches - Remove obsolete version checks (< 12.0.1, < 13.0.0, >= 9.0.0) and try/except imports - Clean up ~50 pytest.mark.skipif markers and unused PYARROW_GE_* constants in test files Note: _FixEmptyStructArrays is intentionally kept — Daft internally cannot handle empty StructArrays, not just an arrow2 FFI issue. ## Related Issues Closes #6347 | 7 个月前 | |
fix: respect scheme aliases in Iceberg writes (#6512) ## Changes Made Iceberg uses a FileProvider interface to create the data_file full URI matching the scheme of the table location itself. However Daft doesn't have a single interface like this, so our read path was supporting custom schemes while the write path wasn't using the aliases. This meant we were writing s3:// in the manifests instead of the custom scheme (bad). This fixes that by applying the aliases to the paths in our python filesystem abstraction which is what our Iceberg writer uses. Because this is only python, you can actually monkeypatch this right now without waiting on a new daft release. https://gist.github.com/rchowell/7027d8c23b3674f4c57fe796d8ae429b ## Related Issues If you previously tried to write an Iceberg table with a different object-store scheme, then the manifests are using s3:// rather than your object-store scheme. | 6 个月前 | |
feat(lance): Support lance namespace read and write (#5980) ## Changes Made This PR implements Lance REST Namespace Catalog support, enabling Daft to read from and write to Lance tables managed by REST-compliant services like LanceDB Cloud, Apache Gravitino, and other catalog systems. Unified API with REST Support: - Enhanced daft.read_lance() and df.write_lance() to support REST URIs (rest://namespace/table) - Added LanceRestConfig class for REST service configuration (authentication, timeouts, custom headers) - URI-based routing automatically detects REST vs file-based Lance tables Core Implementation: - LanceRestScanOperator with filter pushdown, column selection, and count operations - REST write operations supporting create, append, and overwrite modes - Catalog management functions for table creation and registration - Comprehensive error handling and logging Enterprise Integration: - Namespace support for multi-tenant environments - API key authentication and custom headers - Compatible with Lance REST Namespace OpenAPI specification ## Related Issues Closes [#5489](https://github.com/Eventual-Inc/Daft/issues/5489) | 8 个月前 | |
chore(deps): drop pyarrow 8.0.0 support, bump minimum to >= 15.0.0 (#6378) ## Changes Made Bump minimum PyArrow version from >= 8.0.0 to >= 15.0.0 (44 files, -450 lines). - Update version constraints in pyproject.toml and CI test matrix - Remove _FixSliceOffsets workaround (pyarrow < 12.0.0 struct slice offset bug, fixed upstream) - Remove pyarrow_supports_fixed_shape_tensor() and all conditional branches - Remove obsolete version checks (< 12.0.1, < 13.0.0, >= 9.0.0) and try/except imports - Clean up ~50 pytest.mark.skipif markers and unused PYARROW_GE_* constants in test files Note: _FixEmptyStructArrays is intentionally kept — Daft internally cannot handle empty StructArrays, not just an arrow2 FFI issue. ## Related Issues Closes #6347 | 7 个月前 | |
feat(mcap): support topic_start_time_resolver and raw-bytes non-seekable reader (#5886) ## Changes Made * Add per-file per-topic start_time resolver and read via iter_messages with decoders disabled. * Use non-seekable stream to avoid seeking differences across FS backends. <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 9 个月前 | |
feat: Add Apache Paimon Lake Format Support (#6450) ## Changes Made [Apache Paimon](https://paimon.apache.org/) is an open lake format designed for streaming and batch unified analytics. Adding Paimon closes the gap for users running Flink-based pipelines who want to use Daft for batch/ML workloads on top of their existing Paimon tables — without copying data out of the lake. This PR introduces daft.read_paimon(table) and df.write_paimon(table, mode) using [pypaimon](https://pypi.org/project/pypaimon/) as the catalog and metadata layer. ## Related Issues Closes #4976 | 6 个月前 | |
[FEAT] Native globbing for other backends (#1460) Adds and tests globbing capabilities for: 1. Local 2. GCS 3. HTTP Azure support is also added, but not tested yet --------- Co-authored-by: Jay Chia <jaychia94@gmail.com@users.noreply.github.com> | 2 年前 | |
chore: improve the stability of unit test (#4565) ## Changes Made ## Related Issues improve some test cases describe in https://github.com/Eventual-Inc/Daft/discussions/4564 FAILED tests/io/test_s3_credentials_refresh.py::test_s3_credentials_refresh - botocore.errorfactory.BucketAlreadyOwnedByYou: An error occurred (BucketAlreadyOwnedByYou) when calling the CreateBucket operati... FAILED tests/io/test_url_upload_local.py::test_upload_local_no_write_permissions - Failed: DID NOT RAISE <class 'daft.exceptions.DaftCoreException'> FAILED tests/recordbatch/numeric/test_numeric.py::test_table_numeric_trigonometry[sinh-False-False] - assert False is True FAILED tests/recordbatch/numeric/test_numeric.py::test_table_numeric_trigonometry[cosh-False-False] - assert False is True FAILED tests/recordbatch/numeric/test_numeric.py::test_table_numeric_trigonometry[arccos-True-False] - assert False is True ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) Co-authored-by: wuzhiping <wuzhiping.007@bytedance.com> | 1 年前 | |
ci: skip tests during publishing of release (#3744) Tests were failing for unknown reasons: https://github.com/Eventual-Inc/Daft/actions/runs/13064922153/job/36455524516 We skip tests now when publishing, and should just use main as the source of truth --------- Co-authored-by: Jay Chia <jaychia@Jays-MacBook-Pro.local> | 1 年前 | |
fix(io): add connection pool semaphore to Azure Blob Storage backend (#6305) Changes Made Add a connection pool semaphore to AzureBlobSource, matching the existing pattern in S3/GCS/TOS backends. Azure was the only major backend missing this, causing unbounded concurrent connections (400-800+) when reading multiple large parquet files in parallel. - Add max_connections_per_io_thread (default 8) to AzureConfig - Add connection_pool_sema to AzureBlobSource with permit lifecycle tied to GetResult::Stream - Extract get_size_internal() to avoid deadlock for GetRange::Suffix (Azure SDK doesn't support native suffix ranges) - Update Python bindings, type stubs, SQL config Related Issues Closes #6279 | 7 个月前 | |
feat: Add Tencent Cloud COS (Cloud Object Storage) support (#6140) ## Changes Made This PR adds support for Tencent Cloud COS (Cloud Object Storage) by leveraging the generic OpenDAL backend infrastructure introduced in #6177. Instead of implementing a dedicated COS source from scratch, we provide a lightweight CosConfig that converts to OpenDAL configuration, reusing the existing OpenDALSource for all I/O operations. ### Architecture mermaid flowchart TD A["User API: IOConfig(cos=CosConfig(region="ap-guangzhou", ...))"] --> B["Config: CosConfig.to_opendal_config(bucket) → BTreeMap<String, String>"] B --> C["I/O: OpenDALSource (generic, supports read/write/list/glob/multipart)"] C --> D["Backend: Apache OpenDAL → Tencent COS"] ### Implementation Details **New Files:** - src/common/io-config/src/cos.rs - CosConfig struct with region, endpoint, credentials, and connection settings. Includes from_env() for automatic environment variable scanning and to_opendal_config() to convert into OpenDAL-compatible config map. **Modified Files:** - src/common/io-config/src/lib.rs - Export CosConfig module - src/common/io-config/src/config.rs - Add cos field to IOConfig struct - src/common/io-config/src/python.rs - Python bindings for CosConfig class with full API support - src/daft-io/src/lib.rs - Route cos:// and cosn:// URL schemes to SourceType::OpenDAL { scheme: "cos" }, with special handling to extract the bucket from the URL and merge CosConfig into the OpenDAL config - daft/io/__init__.py - Export CosConfig to Python API **Key Design Decisions:** - **No dedicated CosSource** — instead of ~1,000 lines of custom I/O code, COS reuses the generic OpenDALSource which already implements ObjectSource (including multipart write support) - **Lightweight CosConfig preserved** — provides a user-friendly Python API with automatic region↔endpoint derivation and environment variable scanning (COS_* / TENCENTCLOUD_*), rather than requiring raw opendal_backends dicts **Supported Features:** - URL schemes: cos://bucket/key and cosn://bucket/key (Hadoop CosN compatible) - Environment variables: COS_* and TENCENTCLOUD_* prefixes for configuration - Operations: read, write (including multipart), list, delete, glob pattern matching - Authentication: permanent keys and STS temporary credentials - Full Python API with CosConfig class **Why OpenDAL:** Since Tencent COS doesn't have an official Rust SDK, we use Apache OpenDAL which provides a unified data access layer supporting 70+ storage backends including Tencent COS. This PR builds on #6177 which already added the generic OpenDAL backend infrastructure. ### Example Usage **Reading from COS with explicit credentials:** ```python import daft from daft.io import IOConfig, CosConfig io_config = IOConfig( cos=CosConfig( region="ap-guangzhou", secret_id="your-secret-id", secret_key="your-secret-key", ) ) df = daft.read_parquet("cos://my-bucket/path/to/data.parquet", io_config=io_config) df.show() | 7 个月前 | |
chore(deps): drop pyarrow 8.0.0 support, bump minimum to >= 15.0.0 (#6378) ## Changes Made Bump minimum PyArrow version from >= 8.0.0 to >= 15.0.0 (44 files, -450 lines). - Update version constraints in pyproject.toml and CI test matrix - Remove _FixSliceOffsets workaround (pyarrow < 12.0.0 struct slice offset bug, fixed upstream) - Remove pyarrow_supports_fixed_shape_tensor() and all conditional branches - Remove obsolete version checks (< 12.0.1, < 13.0.0, >= 9.0.0) and try/except imports - Clean up ~50 pytest.mark.skipif markers and unused PYARROW_GE_* constants in test files Note: _FixEmptyStructArrays is intentionally kept — Daft internally cannot handle empty StructArrays, not just an arrow2 FFI issue. ## Related Issues Closes #6347 | 7 个月前 | |
feat: Add Apache Gravitino virtual file system (gvfs://) write support in io module (#5965) | 9 个月前 | |
| 6 个月前 | ||
feat: Use daft-decoding for hive value deserialization (#4836) ## Changes Made Use the existing daft-decoding code used in csv deserialization for hive partition values as well. Previously, we were using arrow2's utf8 cast to do this, which caused issues as seen in https://github.com/Eventual-Inc/Daft/issues/4821. ## Related Issues Closes https://github.com/Eventual-Inc/Daft/issues/4821 ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) --------- Co-authored-by: Desmond Cheong <desmondcheongzx@gmail.com> | 1 年前 | |
fix: enhance unit tests (#5787) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> enhance tests. --------- Co-authored-by: cancai <caican@xiaomi.com> | 10 个月前 | |
fix: fix column missing issue for jsonl record schema when reading json/jsonl files (#5681) ## Issue daft.read_json('/mnt/test/vedio.audio-00002.jsonl').where( "sound is null" ).count_rows() ray.exceptions.RayTaskError(DaftCoreException): ray::PhysicalScan->UnGroupedAggregate->Project() (pid=xx, ip=xx.xx.xx.xx, actor_id=xxxx, repr=<daft.runners.flotilla.RaySwordfishActor object at xxxxxxx>) daft.exceptions.DaftCoreException: DaftError::FieldNotFound Column "sound" not found in schema: [Field { name: "video_meta", dtype: Struct([Field { name: "target_fps", dtype: Int64, metadata: {} }]), metadata: {} }] ## Changes Made When reading large JSONL files with automatic schema inference enabled, only the first 1M are used by default to infer the schema. In cases where the data is very sparse, some columns may be missing from the inferred schema. If users later attempt to access such missing columns, an error occurs. The purpose of this change is to eliminate the need for users to manually provide a large and complete schema in sparse-column scenarios. By enhancing the semantics of schema hints to support an "add-missing" behavior, the system can automatically add any missing columns to the final schema when they are accessed. This reduces complexity and improves ease of use. <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues https://github.com/Eventual-Inc/Daft/issues/5647 <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## How to test new UTs. | 6 个月前 | |
feat: add support for reading json arrays (#4844) ## Changes Made General approach is that we peek at the first byte of the file. if it's [ we use the array algorithm, if its { we use the default ndjson algo. ⚠️ It's worth noting that this is not highly optimized and _could_ OOM on very large JSON files as they are pulled **entirely** into memory. as a side note: Our JSON reading logic is a total mess and I didn't want to do any unnecessary refactors, so there was a bit of copy/paste in this PR to get things to work. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 1 年前 | |
fix: Expose json_target_filesize execution config (#6382) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> Expose the json_target_filesize configuration to allow users to set the json file size when writing JSON. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> Signed-off-by: plotor <zhenchao.wang@hotmail.com> | 7 个月前 | |
fix(io): fix JSONL byte-range reading and make chunk_size byte-based (#6374) ## Changes Made - Fix stream_json() ignoring byte range for local files, causing data duplication when scan_task_split_and_merge splits a large JSONL file - Change chunk_size semantic from row count to byte size - Clear stale metadata/statistics on byte-range sub-ScanTasks - Delete dead code (read_json_bulk, ChunkError) <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues close #6414 close #4496 <!-- Link to related GitHub issues, e.g., "Closes #123" --> --------- Co-authored-by: wangzheyan <wangzheyan@bytedance.com> | 6 个月前 | |
feat: support split and merge jsonl/ndjson files (#5695) ## Changes Made Support split and merge jsonl/ndjson files, now only support jsonl and ndjson files. Unsupported: 1. json file. 2. compressed files haven't support now. <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues https://github.com/Eventual-Inc/Daft/issues/5615 <!-- Link to related GitHub issues, e.g., "Closes #123" --> Co-authored-by: cancai <caican@xiaomi.com> | 10 个月前 | |
chore(io): replace python source shim on the rust side (#6556) ## Motivation This removes the python ScanOperator shim in favor of a ScanOperator implementation in Rust which is backed by either a Python *or* Rust data source. This allows us to move the shim one step lower, and now we actually have Rust traits + python bridges for the DataSource and DataSourceTask interfaces. This is an incremental step towards full streaming sources and consolidating the various scanning types and implementations to this interface. ## Changes Made - Removes python _DataSourceShim - Cleanup visibility modifiers and conversion methods for daft-catalog Catalog, Table, Provider - Adds native factory methods for native data source tasks, which are a work-in-progress and just today's scan tasks. - We have a Rust PyDataSourceWrapper trait which is able to implement our existing ScanOperator backed by a Python DataSource | 6 个月前 | |
feat(io): support numeric range expansion in glob patterns (#6127) ## Changes Made Add support for bash-style numeric range syntax {start..end} in glob patterns. This allows users to easily select files by numeric sequences: df = daft.read_parquet("s3://bucket/{0..99}.parquet") The implementation expands numeric ranges to alternation patterns that are compatible with the globset crate, e.g. {0..3} becomes {0,1,2,3}. Features: - Basic ranges: {0..10} - Leading zeros preserved: {00..05} -> 00,01,02,03,04,05 - Reverse ranges: {10..0} - Negative numbers: {-5..5} - Multiple ranges in path: {0..1}/{0..2}.csv - Mixed with wildcards: {0..9}_*.parquet Safety: - Maximum range size of 10,000 elements to prevent memory issues - Clear error message when limit exceeded ## Related Issues Closes #2708 | 7 个月前 | |
chore: bump mypy and ruff in pre-commit (#5836) ## Changes Made Bump mypy to 1.19.1 and ruff to 0.14.10 and resolve pre-commit errors. Rationale: my local ruff version is much higher than that in the project pre-commit, so I am getting a lot of linter warnings in my IDE. It is also about time to upgrade lints to utilize newer Python language features. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 9 个月前 | |
feat: async data sources and tasks (#6492) ## Motivation This makes the python DataSource and DataSourceTask interfaces async generators and switches to using RecordBatch. It's a step towards moving off of the legacy ScanOperator and onto the DataSource. This is backwards compatible because the shim is still able to handle the old interface methods, and we have proper warnings. I am shipping this before the end-to-end native DataSource impl. because it unblocks the paimon PR and I wanted to get the async interfaces in ASAP. A nice feature of this is that you can now use native scan tasks in a python DataSource which unblocks being able to migrate existing python ScanOperator implementations over to DataSource. There's a lot of cleanup still on the Rust side, but I wanted to get the public python side in order. ## Changes - Fixed that annoying range redefinition error for good (fingers crossed) - Updated DataSource task fetching to an async generator - Updated DataSourceTask reading to an async record batch generator - Add a _PyDataSourceTask which wraps the existing ScanTask which unblocks native tasks in python DataSource - ScanOperator can smart unwrap the native tasks as before. - ScanOperator is sync and materializes all ScanTasks for now, so that's why we have the drain loop - We won't be materializing ScanTasks in the near future | 6 个月前 | |
chore: add bucket into endpoint for virtual host style (#4740) ## Changes Made Add bucket into endpoint if not set for virtual host style of s3-like object store. Currently daft uses different IO to write data to object store with different format: - parquet & csv: pyarrow fs(s3 c++ sdk) for write, daft object source(s3 rust sdk) for read - lance: arrow object store Some SDK will auto concat bucket and endpoint to a virtual host style endpoint within SDK, e.g. S3 SDK, but arrow object store require the end user passing the virtual host style endpoint if virtual_hosted_style_request is set. https://github.com/apache/arrow-rs-object-store/blob/main/src/aws/builder.rs#L724 so it's better to include the bucket in endpoint if virtual_hosted_style_request is true, so the end user can include or exclude bucket in endpoint in all file format. ## Related Issues ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 1 年前 | |
feat: Apache OpenDAL™ compatible backends (#6177) ## Changes Made Integrates Apache OpenDAL as a catch-all backend so that any OpenDAL-supported storage (62+ backends) works out of the box via IOConfig(backends={...}) Native backends (S3, GCS, Azure, HTTP, HF, TOS, etc.) are unchanged and always take priority Unknown URL schemes now route through OpenDAL instead of erroring immediately, with a helpful error message if no backend config is provided example usage: py io_config = IOConfig( opendal_backends={"oss": {"bucket": "my-bucket", "access_key_id": "...", "access_key_secret": "..."}} ) df = daft.read_parquet("oss://my-bucket/data.parquet", io_config=io_config) more examples: py # using the opendal github backend io_config = IOConfig( opendal_backends={ "github": { "owner": "Eventual-Inc", "repo": "Daft", } } ) df = daft.read_json("github://main/tests/assets/json-data/sample1.json", io_config=io_config) ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> | 7 个月前 | |
feat: add support for writing _SUCCESS files in parquet operations (#6090) ## Changes Made - Add support for writing a _SUCCESS file upon successful completion of parquet write operations. - Add parameter to existing write_parquet method. df.write_parquet("/path/to/output", write_success_file=True) <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues Closes #4085 <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 6 个月前 | |
chore(deps): drop pyarrow 8.0.0 support, bump minimum to >= 15.0.0 (#6378) ## Changes Made Bump minimum PyArrow version from >= 8.0.0 to >= 15.0.0 (44 files, -450 lines). - Update version constraints in pyproject.toml and CI test matrix - Remove _FixSliceOffsets workaround (pyarrow < 12.0.0 struct slice offset bug, fixed upstream) - Remove pyarrow_supports_fixed_shape_tensor() and all conditional branches - Remove obsolete version checks (< 12.0.1, < 13.0.0, >= 9.0.0) and try/except imports - Clean up ~50 pytest.mark.skipif markers and unused PYARROW_GE_* constants in test files Note: _FixEmptyStructArrays is intentionally kept — Daft internally cannot handle empty StructArrays, not just an arrow2 FFI issue. ## Related Issues Closes #6347 | 7 个月前 | |
fix: short-circuit evaluation for coalesce (#6525) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> Fixes #4069 by adding proper short-circuit behavior to coalesce. - Added Expr::Coalesce with short-circuit logic - Implemented early-exit in record batch evaluation - Preserved semantics during UDF optimization - Added short-circuit tests ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> Closes #4069 --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 6 个月前 | |
chore: Upgrade Ruff ruleset to 3.9 and add from __future__ import annotations (#4393) | 1 年前 | |
feat: Add protocol aliases for IOConfig (#6252) ## Changes Made Adds **protocol aliases** to IOConfig: user-defined mappings from custom scheme names to existing schemes. For example, "my-s3" -> "s3" lets organizations use domain-specific protocol names that route to standard backends (including native S3, Azure, GCS — not just OpenDAL). **Python API:** python io_config = IOConfig( protocol_aliases={"my-s3": "s3", "company-store": "gcs"}, s3=S3Config(endpoint_url="https://my-proprietary-endpoint.example.com"), ) daft.read_parquet("my-s3://bucket/path", io_config=io_config) ### Implementation - **src/common/io-config/src/config.rs** — Added protocol_aliases: BTreeMap<String, String> field to IOConfig, display support, and validate_protocol_aliases() that rejects alias keys matching built-in schemes. - **src/daft-io/src/lib.rs** — Added resolve_url_alias() using Cow for zero-allocation on the common (no-alias) path. Integrated into get_source_and_path(), single_url_get(), single_url_put(), and single_url_get_size(). Added 7 Rust unit tests. - **src/common/io-config/src/python.rs** — Added protocol_aliases parameter to IOConfig::new() and replace() with case normalization and validation. Added getter. - **daft/daft/__init__.pyi** — Updated type stubs. - **tests/io/test_protocol_aliases.py** — 9 config tests + 2 integration tests using OpenDAL fs backend. ### Design Decisions - **Single-level resolution** — no chaining, avoids infinite loops - **Built-in scheme protection** — aliasing s3, gcs, etc. as keys is rejected at construction time - **Case-insensitive** — consistent with parse_url() which already lowercases schemes - **Minimal change surface** — parse_url() and its 17+ external callers remain untouched; alias resolution happens in IOClient methods before calling parse_url() ## Related Issues Builds on PR #6177 (OpenDAL support). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> | 7 个月前 | |
feat: adds column set visitor and use in pushdowns (#4929) ## Changes Made * Adds a visitor which collects a column set * Uses this in Pushdowns to return the filter column set. ## Related Issues I considered saving the list from the rust pushdowns object, but a python visitor is more generic and flexible because it can be used outside the context of pushdowns to find all the columns in an arbitrary expression. This also uses a set for unique column names. If/when daft has a notion of bound columns, then the hash will be important for distinctness because for now we rely on unique names within an expression tree. ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 1 年前 | |
chore: Remove the old Ray Runner (#5375) ## Changes Made 🎉🎂🥳 Can finally delete it, Flotilla supports all necessary features. Additional Related Features Removed: * DataFrame.num_partitions: This is computed using the old Ray runner's planner. We could use the new Ray runner, but it seems kind of unnecessary * Context Settings: --------- Co-authored-by: Colin Ho <colin.ho99@gmail.com> | 10 个月前 | |
chore: bump mypy and ruff in pre-commit (#5836) ## Changes Made Bump mypy to 1.19.1 and ruff to 0.14.10 and resolve pre-commit errors. Rationale: my local ruff version is much higher than that in the project pre-commit, so I am getting a lot of linter warnings in my IDE. It is also about time to upgrade lints to utilize newer Python language features. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 9 个月前 | |
chore: bump mypy and ruff in pre-commit (#5836) ## Changes Made Bump mypy to 1.19.1 and ruff to 0.14.10 and resolve pre-commit errors. Rationale: my local ruff version is much higher than that in the project pre-commit, so I am getting a lot of linter warnings in my IDE. It is also about time to upgrade lints to utilize newer Python language features. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 9 个月前 | |
fix: Reraise exceptions from data sinks to handle unserializable exceptions (#4858) ## Changes Made If an exception is thrown in the data sink and is unserializable, we may get errors like File ~/anaconda3/lib/python3.12/site-packages/daft/dataframe/dataframe.py:1304, in DataFrame.write_sink(self, sink) 1302 builder = self._builder.write_datasink(sink.name(), sink) 1303 write_df = DataFrame(builder) -> 1304 write_df.collect() 1306 results = write_df.to_pydict() 1307 assert "write_results" in results ... ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ray/anaconda3/lib/python3.12/site-packages/ray/exceptions.py", line 54, in from_ray_exception raise RuntimeError(msg) from e RuntimeError: Failed to unpickle serialized exception Catching and reraising their string representations allow us to report the actual exception to the user. | 1 年前 | |
fix(io): fix JSONL byte-range reading and make chunk_size byte-based (#6374) ## Changes Made - Fix stream_json() ignoring byte range for local files, causing data duplication when scan_task_split_and_merge splits a large JSONL file - Change chunk_size semantic from row count to byte size - Clear stale metadata/statistics on byte-range sub-ScanTasks - Delete dead code (read_json_bulk, ChunkError) <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues close #6414 close #4496 <!-- Link to related GitHub issues, e.g., "Closes #123" --> --------- Co-authored-by: wangzheyan <wangzheyan@bytedance.com> | 6 个月前 | |
fix: handles schema inference errors when partition columns are NULL (#6467) ## Changes Made The read_sql method fails when partition columns are all NULL because the schema cannot be inferred. This PR passes the schema explicitly to avoid relying on pyarrow type inference. Includes sqlite-backed read_sql tests for this exact scenario. sh pytest ./tests/io/test_sql.py -v ## Related Issues - Closes #6464 - Closes #6462 --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 6 个月前 | |
feat: Add support for reading whole text files to read_text (#6354) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> Add a whole_text option to the read_text API to support reading whole text contents as a single line. Consider scenarios such as inference scenarios where the content of a text might be a complete prompt, in which case it shouldn't be read line by line. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> Signed-off-by: plotor <zhenchao.wang@hotmail.com> | 7 个月前 | |
feat!: Catch transient errors on turbopuffer writes (#5380) ## Changes Made The turbopuffer client library [retries transient errors 4 times with appropriate back-offs and jitters](https://github.com/turbopuffer/turbopuffer-python?tab=readme-ov-file#retries). In the event that we receive a transient error from the client library, instead of causing the entire script to fail, we should simply track these transient errors and report the number of failed rows later on. Non-transient errors should cause a script to fail fast, as it did before. In the future, this should ideally extend to the proposal of splitting results so that we keep track of what rows resulted in errors so that we can retry them later. For now we simply track the number of failed rows. | 11 个月前 | |
chore: Remove expression namespaces (#5619) ## Changes Made Bye bye ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 10 个月前 | |
chore: Remove expression namespaces (#5619) ## Changes Made Bye bye ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 10 个月前 | |
refactor: Cleanup Dtype Names (#5400) | 11 个月前 | |
refactor: write empty dataframe to parquet/json files via native IO (#5682) ## Changes Made Currently, if the final dataframe is empty, will use the pyarrow fs to write the empty parquet/json files, but if the dataframe is not empty, will use the daft native io to write the data, the write behavior is not consistent, especially if we want to implement new storage backend, we need to cover both scenarios. Secondly, the write empty dataframe to json file is not working currently, because only parquet and csv file format supported during write_empty_tabular, the json file format is not implemented via pyarrow fs. BTW, there still remains a issue that we can write a empty json, but cannot read the empty json via daft.read_json() since cannot infer the schema info. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 10 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 6 个月前 | ||
| 1 年前 | ||
| 7 个月前 | ||
| 6 个月前 | ||
| 8 个月前 | ||
| 7 个月前 | ||
| 9 个月前 | ||
| 6 个月前 | ||
| 2 年前 | ||
| 1 年前 | ||
| 1 年前 | ||
| 7 个月前 | ||
| 7 个月前 | ||
| 7 个月前 | ||
| 9 个月前 | ||
| 6 个月前 | ||
| 1 年前 | ||
| 10 个月前 | ||
| 6 个月前 | ||
| 1 年前 | ||
| 7 个月前 | ||
| 6 个月前 | ||
| 10 个月前 | ||
| 6 个月前 | ||
| 7 个月前 | ||
| 9 个月前 | ||
| 6 个月前 | ||
| 1 年前 | ||
| 7 个月前 | ||
| 6 个月前 | ||
| 7 个月前 | ||
| 6 个月前 | ||
| 1 年前 | ||
| 7 个月前 | ||
| 1 年前 | ||
| 10 个月前 | ||
| 9 个月前 | ||
| 9 个月前 | ||
| 1 年前 | ||
| 6 个月前 | ||
| 6 个月前 | ||
| 7 个月前 | ||
| 11 个月前 | ||
| 10 个月前 | ||
| 10 个月前 | ||
| 11 个月前 | ||
| 10 个月前 |