Sstone31415codex_optimize_rowwise_udf_conversion
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
chore: Update dependencies for dependabot alerts (#6002) Various dependency updates to address security warnings. | 8 个月前 | |
feat: Channel-less intermediate op (#5999) ## Changes Made There is currently a bug with IntermediateOps where the worker that produces a HasMoreOutput is not being polled from again, in the maintain_order = True case. ### Recap Quick recap, what is actually going in with intermediate ops, and why does this only affect HasMoreOutput. <img width="1807" height="1578" alt="image" src="https://github.com/user-attachments/assets/b06d84b7-dd7b-443e-99c6-6dd14f05b8e4" /> We have intermediate op workers, that are responsible for actually executing the op on input morsels, a dispatcher that dispatches work to them, and a receiver that receives results. in the maintain order = True case, we use a round robin dispatcher that dispatches work in round robin fashion, and an OrderingAwareReceiver that receives results in a round robin fashion. ### Problem If a worker produces a result with HasMoreOutput, the round robin receiver SHOULD PULL FROM THAT WORKER AGAIN, BUT IT DOES NOT and instead it moves on to the next worker. This can create a deadlock where the worker that produces has more output wants to send again, but the round robin reciever is not awaiting it, and is instead awaiting from some other worker that could be, say, awaiting it's own input. ### Solution Just give the OrderingAwareReceiver info that it should pull from a certain receiver again. Bleh, this is now so messy. So, lets just get rid of the channels altogether. No more channels. We can remodel the intermediate op to a single concurrent state machine. while has_input or has_active_tasks { tokio::select { new_input = input_rx.recv() => active_tasks.spawn(execute(new_input)), result = active_tasks.join_next() => process_result(result) # if has more output just spawn again. } } This allows us to get rid channels within the intermediate op altogether, and only have a single channel connecting intermdiate ops. ### Results This script simulates a long pipeline of connecting intermediate ops. Theoretically less channels means less intermediate data. import daft import os @daft.func def generate_10_mb_data(x: int) -> bytes: return os.urandom(10 * 1024 * 1024) @daft.func def noop(x: bytes) -> bytes: return x df = daft.from_pydict({"id": [i for i in range(100)]}).into_batches(1) # Generate 10 MB of data for each row df = df.with_column("data_0", generate_10_mb_data(daft.col("id"))) # Add some noop udfs to add a lot of intermdiate ops for i in range(1, 10): df = df.with_column(f"data_{i}", noop(daft.col(f"data_{i-1}"))) # Send them to the void for p in df.iter_partitions(): pass Before: <img width="1113" height="367" alt="Screenshot 2026-01-09 at 1 37 21 PM" src="https://github.com/user-attachments/assets/2ed003fc-7764-4f32-8c2f-1ab571d1696a" /> After: <img width="1094" height="390" alt="Screenshot 2026-01-09 at 1 36 47 PM" src="https://github.com/user-attachments/assets/4faa9a91-218b-4c86-be00-983b67e20b98" /> We reduced peak memory by half. ### Future This HasMoreOutput bug affects streaming sinks as well actually. So if this fix works we should implement it on streaming sink as well, and then might as well do so for blocking sinks and we can get rid of the intra-op channels altogether. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 8 个月前 | |
chore: bump pyo3 dependency (#5410) ## Changes Made bumps pyo3 and related dependencies to 26 Pretty much just a find & replace /s/old/new - with_gil -> attach - allow_threads -> detach - PyObject -> Py<PyAny> ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 10 个月前 | |
ci: remove macos from PR test suite (#5142) ## Changes Made We currently use MacOS to run unit tests in PRs because we had an issue with getting Rust code coverage in Ubuntu 22.04. The latest Rust toolchain no longer has this issue, so this PR updates the rust toolchain to the latest nightly and moves the code coverage tests to Ubuntu. The vast majority of the diff here is just linting and lifetime fixes due to the upgrade to the latest nightly Rust + 2024 edition. The rest of the changes are as follows: - moving the code coverage tests in .github/workflows/pr-test-suite.yml to Ubuntu as described above - it was failing due to out of disk space so I also added a disk space remover action - update our Rust versions in rust-toolchain.toml and Cargo.toml - use hashbrown::HashMap instead of std::HashMap because raw_entry_mut was removed from the standard library ## Related Issues https://github.com/Eventual-Inc/Daft/issues/3801 ## Checklist - [x] Documented in API Docs (if applicable) - [x] Documented in User Guide (if applicable) - [x] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [x] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 1 年前 | |
refactor(arrow-rs): Remove arrow2 from daft-writers (#5985) ## Changes Made Removed arrow2 usage from the daft-writers crate. Also did some other cleanup along the way, such as adding minimal timestamp with timezone support for CSV and JSON. Will probably double check support when migrating the readers. | 8 个月前 | |
feat: Support pattern filtering for SHOW TABLES (#5423) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> ### Extended SQL LIKE Pattern Matching in MemoryCatalog - Upgrade sqlparser to extract namespace from query - Translate SQL LIKE patterns to regex (src/daft-catalog/src/pattern.rs) - Supports standard SQL LIKE wildcards: % (zero or more), _ (exactly one), \ (escape) - Updated documentation on SHOW TABLES syntax and pattern behavior ### Testing - Added pattern matching tests to test_sql_show_tables.py and in src/daft-catalog/pattern.rs - cargo test -p daft-catalog pattern:: ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> Closes #4461 Closes #4007 ## Checklist - [x] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 8 个月前 | |
chore: reduce binary size by feature flagging derive(Debug) (#5622) ## Changes Made cargo-llvm-lines showed that 0.7% of our binary size was from Debug impls 40559 (0.7%, 5.8%) 15604 (3.0%, 5.4%) <&T as core::fmt::Debug>::fmt I noticed that this PR reduces the total binary size by about 1.2MB. While pretty small when our uncompressed binary is 180+MB, the small savings add up. So this PR attempts to either remove derive(Debug) or feature flag it: cfg_attr(debug_assertions, derive(Debug)) in some spots we still need it to not break a bunch of things, so for not(debug_assertions), theres a simpler debug impl provided, that usually just debugs the name, not all of the struct fields. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 9 个月前 | |
feat: Add WARC reader (#3871) Adds a reader for .warc and .warc.gz files. Currently optimized for reading from S3. Some numbers: - Downloading a single common crawl file from S3, e.g. s3://commoncrawl/crawl-data/CC-MAIN-2018-17/segments/1524125937193.1/warc/CC-MAIN-20180420081400-20180420101400-00000.warc.gz, takes ~16.5s. - Gunzipping this file and processing it with fastwarc takes ~33s. - With swordfish, collecting that same file as a daft dataframe takes ~20s on an m7g.4xlarge instance. - With swordfish, collecting 10 common crawl files take ~3min 12s. - With swordfish, processing (read then sum on content length) 1 common crawl file takes ~20s. - With swordfish, processing 2 common crawl files still takes ~20s. - With swordfish, processing 10 common crawl files takes ~40s. This is because we've set the max number of parallel reads to 8. So 10 scan tasks take 2x20s to read. If we increase the max number of parallel reads to 10, the runtime drops to ~30s. - With ray, collecting 1 file takes ~25s. - **Unfortunately, with ray, collecting 10 files caused the instance to become unresponsive.** Followup work: - Extracting fields from the warc_headers json is not very fast. We can do better here by allowing users to specify the metadata headers that they want to extract. --------- Co-authored-by: Sammy Sidhu <sammy.sidhu@gmail.com> Co-authored-by: Colin Ho <colinho@Colins-MBP.localdomain> | 1 年前 | |
chore(observability): Refactor progress bar to remove RuntimeStatsSubscriber (#6030) ## Changes Made Again, a change while working on the One True Progress Bar (™️ pending) PR. Originally was going to make the pbar a QuerySubscriber, but that will be really messy, so instead, just refactored it a bit to remove RuntimeStatsSubscriber | 7 个月前 | |
refactor(arrow2): rename validity to nulls to align with arrow-rs (#6027) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 8 个月前 | |
fix: Incorrect buffer pool calculation strategy when reading CSV (#5857) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> Currently, the chunk_size parameter of ScanTask is calculated by the MorselSizeRequirement strategy (as shown below), and its corresponding unit is "rows". However, read_csv treats it as "bytes" during processing, which is inconsistent with the behavior of APIs such as read_parquet and read_json. This results in the data of each batch usually being much smaller than the expected value of the batch_size parameter. rust let chunk_size = match self.morsel_size_requirement { MorselSizeRequirement::Strict(size) => size, MorselSizeRequirement::Flexible(_, upper) => upper, }; For example, in our scenario, the batch_size of the UDF is 1024, but in reality, only 3 rows of data are passed to the UDF each time. Because Daft estimates that each row of data is approximately 300~400 bytes, dividing by 1024 gives that about 3 rows of data need to be read each time. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> Signed-off-by: plotor <zhenchao.wang@hotmail.com> | 8 个月前 | |
chore: Upgrade nextjs (#6015) ## Changes Made Address these CVEs: - https://www.cve.org/CVERecord?id=CVE-2025-55183 - https://www.cve.org/CVERecord?id=CVE-2025-55184 - https://www.cve.org/CVERecord?id=CVE-2025-67779 (because the fix to the above was not fully addressed) | 8 个月前 | |
refactor(arrow-rs): Remove arrow2 from daft-scan and related (#5974) ## Changes Made Another portion split off from the larger PR | 8 个月前 | |
refactor(flotilla): Swordfish task builder (#5976) ## Changes Made Passing SubmittableTask<SwordfishTask> through the distributed pipeline nodes makes it difficult to layer and modify tasks, because they contain not only the plan templates but also things like plan context, node ids, input partitions, notify tokens, cancel tokens, etc etc. Currently the pipeline nodes have to deal with all that when all it really wants to do is add some instruction to the plan template. This PR introduces SwordfishTaskBuilder, which takes care of all the non-instruction related fields and exposes methods like 'map_plan' that the pipeline nodes can use to modify the plan, and it will automatically deal with the changes needed on the other fields, like adding node id. Other minor changes: - Task ids are created upon building a SwordfishTaskBuilder into a SubmittableSwordfishTask. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 8 个月前 | |
codex_optimize_rowwise_udf_conversion | 1 个月前 | |
feat: Add name and path properties to daft.File (#6024) ## Summary - Added path property to daft.File to return the full path or URL - Added name property to daft.File to return the filename (basename) - Both properties work with local paths, URLs (http/https), and cloud storage paths (s3://, etc.) ## Implementation - Added path() and name() methods to PyFileReference in Rust (src/daft-file/src/python.rs) - Exposed them as @property decorators in the Python File class (daft/file/file.py) - The name method intelligently extracts the basename from various path formats using URL parsing and path splitting ## Tests Added comprehensive tests in tests/file/test_file_basics.py: - test_file_path_property: Verifies path property returns full path - test_file_name_property_local_path: Tests filename extraction from local paths - test_file_name_property_with_subdirs: Tests with nested directories - test_file_path_and_name_with_url: Tests with remote URLs - test_file_name_property_various_formats: Parametrized test for s3://, https://, file://, and local paths All 10 test cases pass successfully. ## Example Usage python import daft # Local file f = daft.File("/path/to/data.csv") print(f.path) # /path/to/data.csv print(f.name) # data.csv # S3 file f = daft.File("s3://bucket/path/to/file.parquet") print(f.path) # s3://bucket/path/to/file.parquet print(f.name) # file.parquet # Remote URL f = daft.File("https://example.com/data.json") print(f.path) # https://example.com/data.json print(f.name) # data.json 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com> | 8 个月前 | |
refactor(arrow2): rename validity to nulls to align with arrow-rs (#6027) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 8 个月前 | |
refactor(arrow2): rename validity to nulls to align with arrow-rs (#6027) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 8 个月前 | |
refactor(arrow2): rename validity to nulls to align with arrow-rs (#6027) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 8 个月前 | |
refactor(arrow2): rename validity to nulls to align with arrow-rs (#6027) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 8 个月前 | |
chore: reduce binary size by feature flagging derive(Debug) (#5622) ## Changes Made cargo-llvm-lines showed that 0.7% of our binary size was from Debug impls 40559 (0.7%, 5.8%) 15604 (3.0%, 5.4%) <&T as core::fmt::Debug>::fmt I noticed that this PR reduces the total binary size by about 1.2MB. While pretty small when our uncompressed binary is 180+MB, the small savings add up. So this PR attempts to either remove derive(Debug) or feature flag it: cfg_attr(debug_assertions, derive(Debug)) in some spots we still need it to not break a bunch of things, so for not(debug_assertions), theres a simpler debug impl provided, that usually just debugs the name, not all of the struct fields. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 9 个月前 | |
refactor(arrow2): rename validity to nulls to align with arrow-rs (#6027) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 8 个月前 | |
refactor(arrow2): rename validity to nulls to align with arrow-rs (#6027) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 8 个月前 | |
refactor(arrow2): rename validity to nulls to align with arrow-rs (#6027) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 8 个月前 | |
refactor(arrow2): rename validity to nulls to align with arrow-rs (#6027) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 8 个月前 | |
feat: Extend hash variants for xxhash (#5276) | 10 个月前 | |
refactor(arrow2): rename validity to nulls to align with arrow-rs (#6027) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 8 个月前 | |
chore: Update dependencies for dependabot alerts (#6002) Various dependency updates to address security warnings. | 8 个月前 | |
refactor(arrow-rs): Remove arrow2 from daft-scan and related (#5974) ## Changes Made Another portion split off from the larger PR | 8 个月前 | |
chore(observability): Refactor progress bar to remove RuntimeStatsSubscriber (#6030) ## Changes Made Again, a change while working on the One True Progress Bar (™️ pending) PR. Originally was going to make the pbar a QuerySubscriber, but that will be really messy, so instead, just refactored it a bit to remove RuntimeStatsSubscriber | 7 个月前 | |
feat(agg): support map_groups with v2 udf (#5927) ## Summary Extend grouped aggregation map_groups to support Daft's new UDF system (@daft.func.batch and @daft.cls/@daft.method.batch) in addition to legacy @daft.udf. ## Changes ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 8 个月前 | |
fix: ".*" not handled correctly in SQL planner (#5784) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> 1. Fix handling of struct field wildcards (.*) in the SQL planner. 2. Add test case to validate. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> https://github.com/Eventual-Inc/Daft/issues/4120 | 8 个月前 | |
refactor(arrow-rs): Remove arrow2 from daft-writers (#5985) ## Changes Made Removed arrow2 usage from the daft-writers crate. Also did some other cleanup along the way, such as adding minimal timestamp with timezone support for CSV and JSON. Will probably double check support when migrating the readers. | 8 个月前 | |
ci: remove macos from PR test suite (#5142) ## Changes Made We currently use MacOS to run unit tests in PRs because we had an issue with getting Rust code coverage in Ubuntu 22.04. The latest Rust toolchain no longer has this issue, so this PR updates the rust toolchain to the latest nightly and moves the code coverage tests to Ubuntu. The vast majority of the diff here is just linting and lifetime fixes due to the upgrade to the latest nightly Rust + 2024 edition. The rest of the changes are as follows: - moving the code coverage tests in .github/workflows/pr-test-suite.yml to Ubuntu as described above - it was failing due to out of disk space so I also added a disk space remover action - update our Rust versions in rust-toolchain.toml and Cargo.toml - use hashbrown::HashMap instead of std::HashMap because raw_entry_mut was removed from the standard library ## Related Issues https://github.com/Eventual-Inc/Daft/issues/3801 ## Checklist - [x] Documented in API Docs (if applicable) - [x] Documented in User Guide (if applicable) - [x] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [x] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 1 年前 | |
| 8 个月前 | ||
refactor(arrow2): rename validity to nulls to align with arrow-rs (#6027) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 8 个月前 | |
fix: Fast failure when dashboard is enabled in Ray Runner (#5867) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> The daft-dashboard currently only supports Native Runner, so we should gives a direct "not implemented" error message to users who enabled dashboard in Ray Runner, rather than a strange job failure. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> Signed-off-by: plotor <zhenchao.wang@hotmail.com> | 8 个月前 | |
refactor(arrow-rs): Remove arrow2 from daft-scan and related (#5974) ## Changes Made Another portion split off from the larger PR | 8 个月前 | |
refactor(arrow-rs): Remove arrow2 from daft-scan and related (#5974) ## Changes Made Another portion split off from the larger PR | 8 个月前 | |
chore: reduce binary size by feature flagging derive(Debug) (#5622) ## Changes Made cargo-llvm-lines showed that 0.7% of our binary size was from Debug impls 40559 (0.7%, 5.8%) 15604 (3.0%, 5.4%) <&T as core::fmt::Debug>::fmt I noticed that this PR reduces the total binary size by about 1.2MB. While pretty small when our uncompressed binary is 180+MB, the small savings add up. So this PR attempts to either remove derive(Debug) or feature flag it: cfg_attr(debug_assertions, derive(Debug)) in some spots we still need it to not break a bunch of things, so for not(debug_assertions), theres a simpler debug impl provided, that usually just debugs the name, not all of the struct fields. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 9 个月前 | |
refactor(arrow-rs): Upgrade arrow-rs to 57.1.0 (#5969) ## Changes Made When going through some of the arrow2 uses (like in IPC, Flight, JSON, etc), I noticed that there are a couple of useful features in the latest version of arrow-rs that will help us with the migration. So I upgraded it | 8 个月前 | |
refactor(arrow-rs): Upgrade arrow-rs to 57.1.0 (#5969) ## Changes Made When going through some of the arrow2 uses (like in IPC, Flight, JSON, etc), I noticed that there are a couple of useful features in the latest version of arrow-rs that will help us with the migration. So I upgraded it | 8 个月前 | |
fix: ".*" not handled correctly in SQL planner (#5784) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> 1. Fix handling of struct field wildcards (.*) in the SQL planner. 2. Add test case to validate. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> https://github.com/Eventual-Inc/Daft/issues/4120 | 8 个月前 | |
chore: reduce binary size by feature flagging derive(Debug) (#5622) ## Changes Made cargo-llvm-lines showed that 0.7% of our binary size was from Debug impls 40559 (0.7%, 5.8%) 15604 (3.0%, 5.4%) <&T as core::fmt::Debug>::fmt I noticed that this PR reduces the total binary size by about 1.2MB. While pretty small when our uncompressed binary is 180+MB, the small savings add up. So this PR attempts to either remove derive(Debug) or feature flag it: cfg_attr(debug_assertions, derive(Debug)) in some spots we still need it to not break a bunch of things, so for not(debug_assertions), theres a simpler debug impl provided, that usually just debugs the name, not all of the struct fields. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 9 个月前 | |
refactor(arrow-rs): Remove arrow2 from WARC reader (#5948) | 8 个月前 | |
refactor(swordfish): Channel-less blocking sink (#6023) ## Changes Made Remove channels from blocking sink, review [github.com/Eventual-Inc/Daft/pull/5999](https://github.com/Eventual-Inc/Daft/pull/5999) first. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 8 个月前 | |
[CHORE] add/fix many clippy lints (#2978) mostly auto-fix; a few manual fixes | 1 年前 | |
ci: remove macos from PR test suite (#5142) ## Changes Made We currently use MacOS to run unit tests in PRs because we had an issue with getting Rust code coverage in Ubuntu 22.04. The latest Rust toolchain no longer has this issue, so this PR updates the rust toolchain to the latest nightly and moves the code coverage tests to Ubuntu. The vast majority of the diff here is just linting and lifetime fixes due to the upgrade to the latest nightly Rust + 2024 edition. The rest of the changes are as follows: - moving the code coverage tests in .github/workflows/pr-test-suite.yml to Ubuntu as described above - it was failing due to out of disk space so I also added a disk space remover action - update our Rust versions in rust-toolchain.toml and Cargo.toml - use hashbrown::HashMap instead of std::HashMap because raw_entry_mut was removed from the standard library ## Related Issues https://github.com/Eventual-Inc/Daft/issues/3801 ## Checklist - [x] Documented in API Docs (if applicable) - [x] Documented in User Guide (if applicable) - [x] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [x] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 1 年前 | |
refactor(arrow2): array & series to/from arrow (#5848) ## Changes Made adds the following methods to most array implementations `` rs // convert to type erased ArrayRef // all arrays implement this fn to_arrow(&self) -> DaftResult<ArrayRef>; // convert to a concrete type which varies for each array type // such as DataArray<Int8Type> -> arrow::array::Int8Array // most arrays implement this, but not all of them. // this is also true for the original as_arrow (now as_arrow2) that this is intended to replace. fn as_arrow(&self) -> DaftResult<Self::ArrowOutput> ` and from methods. rs // this is also implemented for all arrays. fn from_arrow<F: Into<Arc<Field>>>(field: F, array: ArrayRef) -> DaftResult<Self>; ## Note for reviewers the bulk of the changes is in the following - src/daft-core/src/array/ops/from_arrow.rs - src/daft-core/src/datatypes/logical.rs - src/daft-core/src/array/*` ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 8 个月前 | |
feat: Add guess_mime_type scalar expression for MIME type detection from bytes (#5883) | 8 个月前 |