| [FEAT] add list.value_counts() (#2902) This PR implements the list.value_counts() function and refactors Map types to use a more explicit key-value structure. It also includes various improvements and bug fixes across the codebase. ## Key changes 1. **Implemented list.value_counts() function** 2. **Refactored Map type representation** - Updated DataType enum in src/daft-schema/src/dtype.rs to use explicit key and value types 3. **Improved error handling and type checking** - Enhanced type checking in various parts of the codebase - Improved error messages for better debugging 4. **Performance optimizations** - Refactored some operations to be more efficient, especially in list and map operations 5. **Code cleanup and minor improvements** - Removed unnecessary clones and improved code readability - Updated comments and documentation | 1 年前 |
| chore: optimizes slow tests in CI/CD (#6029) ## Results The PR reduced unit-test time (3.13, native, 22.0.0, ubuntu-latest, false) from [33m 18s](https://github.com/Eventual-Inc/Daft/actions/runs/20981828603/job/60308365885?pr=6023) to 13m 44s — saving about 20 mins which is nice. The test_context is still slow, but it has very few tests, a big win is parallelization and taking >5 mins off just by skipping the setup for skipped tests. This also reduced the ray tests from 45 mins to 27 mins, and there should be plenty more savings here. Then disabling windows eliminated a 50+ min action. text ================================================================================ OVERALL SUMMARY ================================================================================ Tests Analyzed: 9620 Total Time: 941.32s Average Time: 0.05s Max Time: 14.70s Min Time: 0.01s It took 188s to run make test on my mac which is nice; it also crushed my machine while it was running 🥲 ## Changes Made * Adds -n auto to pytest for parallelize testing on multiple cores * Skips generating fixtures (expensive?) by not actually creating the tests - we were creating test data _then_ skipping for several thousand join tests taking ~5 mins of setup work just to skip. * Memoizes data generation for the window tests (often the slowest for me) * Reduces the data generation size and faster comparison algo (before was 10,000 data points with a O(n^2) algo, now it's 1,000 data points with an O(nlogn) comparison algo) * Disables the RAY_DASHBOARD for the context setup commands, and other minor things to speedup ray init * Skips windows on PRs (>50 minutes) only enabled on main, worthwhile trade-off because we will still catch windows-specific failures when merging to main which is quite rare to be fair. ## Related Issues N/A | 7 个月前 |
| chore: add vscode debug example with env (#4819) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 1 年前 |
| docs: Update readme with new benchmarks (#5281) ## Changes Made ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) --------- Co-authored-by: Desmond Cheong <desmondcheongzx@gmail.com> | 11 个月前 |
| refactor: Remove Unloaded MicroPartitions (#5710) ## Changes Made With the removal of the Ray runner, we no longer use unloaded MicroPartitions for reading ScanTasks, so we can remove that code path entirely. Couple of Notes: * With these changes, I could also update the Partition trait (for MicroPartition and RayPartitionRef) to never error. * I left some code paths to support MicroPartition.read_parquet even though the runners don't use them, only some tests do. If you think I should remove those as well, lmk | 9 个月前 |
| [BUG] Anaconda client don't upload src wheels (#1415) | 2 年前 |
| fix(video): correct keyframe seek timestamp calculation for start_time (#6005) ## Changes Made VideoFile.keyframes() computed an incorrect seek timestamp when start_time > 0. The previous implementation used start_time * time_base, which keeps the effective seek position near 0 for small time_base values (for example 1/30000). As a result, the start_time parameter was effectively ignored and keyframes were decoded from the beginning of the video. This also affected the daft.functions.video_keyframes() UDF, which delegates to VideoFile.keyframes(). Users expecting to extract keyframes from a later time range (for example starting at 10 seconds) instead received frames starting close to time 0. <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues #5949 <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 7 个月前 |
| docs: new section for openai compatible providers (#5748) This pull request updates the documentation to provide clearer guidance on using OpenAI-compatible providers and vLLM Online Serving with Daft's prompt function. It introduces new sections, improves discoverability, and updates example references to reflect recent changes in supported use cases. **Documentation improvements for provider usage:** * Added detailed sections in providers.md for "Prompting with OpenAI-Compatible Providers" and "Prompting with vLLM Online Serving", including code samples for OpenRouter, HuggingFace, Databricks, and vLLM setup. * Updated the summary and navigation in providers.md to include the new provider sections for easier access. * Added cross-links in prompt.md to direct users to the new provider documentation for OpenAI-compatible providers and vLLM Online Serving. **Example and navigation updates:** * Changed the example reference in SUMMARY.md from "Document Processing" to "Voice AI Analytics" to better reflect current supported examples. **General documentation clarity:** * Added a section break in prompt.md for improved readability and flow. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> | 7 个月前 |
| feat: Add support for Metrics tab in quickstart Ray dashboard (#5429) ## Changes Made Adds support for properly rendering "Metrics" tab in Ray dashboard. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 10 个月前 |
| chore(observability): Refactor progress bar to remove RuntimeStatsSubscriber (#6030) ## Changes Made Again, a change while working on the One True Progress Bar (™️ pending) PR. Originally was going to make the pbar a QuerySubscriber, but that will be really messy, so instead, just refactored it a bit to remove RuntimeStatsSubscriber | 7 个月前 |
| fix(observability): Clean up progress bar naming (#6028) ## Changes Made While working on the One True Progress Bar (™️ pending) PR, noticed a couple of things we can clean up. | 7 个月前 |
| refactor(arrow-rs): Remove arrow2 from daft-sketch (#5967) ## Changes Made Remove arrow2 usage from the sketches code path --------- Co-authored-by: universalmind303 <cory.grinstead@gmail.com> | 8 个月前 |
| chore: bump mypy and ruff in pre-commit (#5836) ## Changes Made Bump mypy to 1.19.1 and ruff to 0.14.10 and resolve pre-commit errors. Rationale: my local ruff version is much higher than that in the project pre-commit, so I am getting a lot of linter warnings in my IDE. It is also about time to upgrade lints to utilize newer Python language features. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 8 个月前 |
| [CHORE] ignore 45e2944e (#2979) | 1 年前 |
| chore: ignore all markdown files inside .claude (#5943) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 8 个月前 |
| [CHORE] Add .lldbinit for debugging (#2750) | 2 年前 |
| chore: uv lock check (#6010) ## Changes Made Update uv lock and also add precommit hook to ensure its always up to date. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 7 个月前 |
| fix: Fix docs build (#5066) ## Changes Made https://github.com/Eventual-Inc/Daft/pull/4849 broke the readthedocs build. Unbreak it. | 1 年前 |
| chore: config isort known_third_party to fix import formatting errors (#4840) ## Changes Made When I develop the connectors for daft and lance simultaneously, I debug the code in the same venv environment. At this time, it seems that when formatting, lance is regarded as daft's own implementation. So, I've clarified the isort rules for formatting here. This PR Before: <img width="621" height="47" alt="image" src="https://github.com/user-attachments/assets/a6fa87f0-233e-4a2a-9611-caef26dedf60" /> This PR After: <img width="583" height="34" alt="image" src="https://github.com/user-attachments/assets/5ca44803-3622-4431-b09e-bfd46f9d66c9" /> <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 1 年前 |
| chore: remove ir and proto crates (#5742) ## Changes Made They are unused and taking up space, lets just remove them. ## Related Issues https://github.com/Eventual-Inc/Daft/pull/5736 | 9 个月前 |
| chore: add PR conventions note to AGENTS.md (#5166) ## Changes Made - Add PR conventions note to AGENTS.md - Add CLAUDE.md symlink to AGENTS.md ## Related Issues - Related PR: #5124 ## Checklist - [x] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 11 个月前 |
| [CHORE]: Arrow2 migrate (#2341) | 2 年前 |
| docs: fix broken links causing CI failure (#5832) ## Summary Fix two broken links that have been causing the "Check Broken Links on daft.ai" CI workflow to fail. ### Changes 1. **CONTRIBUTING.md**: Updated docs link from /contributing-to-daft/ to /contributing/development/ (the docs structure was reorganized) 2. **docs/api/ai.md**: Updated GitHub link from daft-data/daft to Eventual-Inc/Daft (the repository was migrated) | 8 个月前 |
| chore: Use tokio channel for swordfish channels (#6035) ## Changes Made Change the crate::channel to tokio channel (only used for inter-operator now) ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 7 个月前 |
| chore: Update dependencies for dependabot alerts (#6002) Various dependency updates to address security warnings. | 7 个月前 |
| Build Wheels in CI (#142) * Leverages cibuildwheel to build wheels for linux and macos targets in github workflow * script to correct macos target name x86_64 -> arm64 * uses semantic versioning for daft * change package name to getdaft * adds apache2 license * seperate out dependancies as groups | 3 年前 |
| chore: optimizes slow tests in CI/CD (#6029) ## Results The PR reduced unit-test time (3.13, native, 22.0.0, ubuntu-latest, false) from [33m 18s](https://github.com/Eventual-Inc/Daft/actions/runs/20981828603/job/60308365885?pr=6023) to 13m 44s — saving about 20 mins which is nice. The test_context is still slow, but it has very few tests, a big win is parallelization and taking >5 mins off just by skipping the setup for skipped tests. This also reduced the ray tests from 45 mins to 27 mins, and there should be plenty more savings here. Then disabling windows eliminated a 50+ min action. text ================================================================================ OVERALL SUMMARY ================================================================================ Tests Analyzed: 9620 Total Time: 941.32s Average Time: 0.05s Max Time: 14.70s Min Time: 0.01s It took 188s to run make test on my mac which is nice; it also crushed my machine while it was running 🥲 ## Changes Made * Adds -n auto to pytest for parallelize testing on multiple cores * Skips generating fixtures (expensive?) by not actually creating the tests - we were creating test data _then_ skipping for several thousand join tests taking ~5 mins of setup work just to skip. * Memoizes data generation for the window tests (often the slowest for me) * Reduces the data generation size and faster comparison algo (before was 10,000 data points with a O(n^2) algo, now it's 1,000 data points with an O(nlogn) comparison algo) * Disables the RAY_DASHBOARD for the context setup commands, and other minor things to speedup ray init * Skips windows on PRs (>50 minutes) only enabled on main, worthwhile trade-off because we will still catch windows-specific failures when merging to main which is quite rare to be fair. ## Related Issues N/A | 7 个月前 |
| chore: Fix a minor ambiguity in the README docs (#5830) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> Change "We <3 developers!" to "We ❤️ developers!". I know this is a rather interesting expression, but some customers have interpreted this sentence as "Daft has no more than 3 full-time developers", mistakenly thinking that Daft is a personal project and thus being hesitant to adopt it in production easily. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> Signed-off-by: plotor <zhenchao.wang@hotmail.com> | 8 个月前 |
| docs: fix typo "Github" --> "GitHub" (#5025) ## Changes Made Fixes GitHub capitalization across the docs. <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 1 年前 |
| chore: remove ir and proto crates (#5742) ## Changes Made They are unused and taking up space, lets just remove them. ## Related Issues https://github.com/Eventual-Inc/Daft/pull/5736 | 9 个月前 |
| feat: Add Apache Gravitino catalog in catalog module (#5694) ## Changes Made Add support for Apache Gravitino as a data catalog in Daft. This PR is adding the catalog class implementation. ## Related Issues The first pr for https://github.com/Eventual-Inc/Daft/issues/5503 --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 9 个月前 |
| chore: optimizes slow tests in CI/CD (#6029) ## Results The PR reduced unit-test time (3.13, native, 22.0.0, ubuntu-latest, false) from [33m 18s](https://github.com/Eventual-Inc/Daft/actions/runs/20981828603/job/60308365885?pr=6023) to 13m 44s — saving about 20 mins which is nice. The test_context is still slow, but it has very few tests, a big win is parallelization and taking >5 mins off just by skipping the setup for skipped tests. This also reduced the ray tests from 45 mins to 27 mins, and there should be plenty more savings here. Then disabling windows eliminated a 50+ min action. text ================================================================================ OVERALL SUMMARY ================================================================================ Tests Analyzed: 9620 Total Time: 941.32s Average Time: 0.05s Max Time: 14.70s Min Time: 0.01s It took 188s to run make test on my mac which is nice; it also crushed my machine while it was running 🥲 ## Changes Made * Adds -n auto to pytest for parallelize testing on multiple cores * Skips generating fixtures (expensive?) by not actually creating the tests - we were creating test data _then_ skipping for several thousand join tests taking ~5 mins of setup work just to skip. * Memoizes data generation for the window tests (often the slowest for me) * Reduces the data generation size and faster comparison algo (before was 10,000 data points with a O(n^2) algo, now it's 1,000 data points with an O(nlogn) comparison algo) * Disables the RAY_DASHBOARD for the context setup commands, and other minor things to speedup ray init * Skips windows on PRs (>50 minutes) only enabled on main, worthwhile trade-off because we will still catch windows-specific failures when merging to main which is quite rare to be fair. ## Related Issues N/A | 7 个月前 |
| ci: remove macos from PR test suite (#5142) ## Changes Made We currently use MacOS to run unit tests in PRs because we had an issue with getting Rust code coverage in Ubuntu 22.04. The latest Rust toolchain no longer has this issue, so this PR updates the rust toolchain to the latest nightly and moves the code coverage tests to Ubuntu. The vast majority of the diff here is just linting and lifetime fixes due to the upgrade to the latest nightly Rust + 2024 edition. The rest of the changes are as follows: - moving the code coverage tests in .github/workflows/pr-test-suite.yml to Ubuntu as described above - it was failing due to out of disk space so I also added a disk space remover action - update our Rust versions in rust-toolchain.toml and Cargo.toml - use hashbrown::HashMap instead of std::HashMap because raw_entry_mut was removed from the standard library ## Related Issues https://github.com/Eventual-Inc/Daft/issues/3801 ## Checklist - [x] Documented in API Docs (if applicable) - [x] Documented in User Guide (if applicable) - [x] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [x] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 1 年前 |
| [CHORE] Add rustfmt config file and run formatter (#2807) # Overview The only file that I added was the rustfmt.toml. In that file, I updated the styles on how imports should be ordered ("StdExternalCrate"). Then I ran the formatter. The formatter changed updated all of our rust files to follow this convention. This PR is obviously huge, but the only changes that are observed are after running cargo fmt on the original code. I have not sneaked anything else into here. You can verify this via 2 ways: 1. Manually inspecting each file to make sure that only the imports have been rearranged (obviously tiring and cumbersome). 2. Checking out the prior commit, adding the *exact same* rustfmt.toml file, running cargo fmt, and checking to make sure your formatted code has *NO* diffs against my commit. (Make sure your rust toolchain is updated to the latest release!). | 1 年前 |
| chore: optimizes slow tests in CI/CD (#6029) ## Results The PR reduced unit-test time (3.13, native, 22.0.0, ubuntu-latest, false) from [33m 18s](https://github.com/Eventual-Inc/Daft/actions/runs/20981828603/job/60308365885?pr=6023) to 13m 44s — saving about 20 mins which is nice. The test_context is still slow, but it has very few tests, a big win is parallelization and taking >5 mins off just by skipping the setup for skipped tests. This also reduced the ray tests from 45 mins to 27 mins, and there should be plenty more savings here. Then disabling windows eliminated a 50+ min action. text ================================================================================ OVERALL SUMMARY ================================================================================ Tests Analyzed: 9620 Total Time: 941.32s Average Time: 0.05s Max Time: 14.70s Min Time: 0.01s It took 188s to run make test on my mac which is nice; it also crushed my machine while it was running 🥲 ## Changes Made * Adds -n auto to pytest for parallelize testing on multiple cores * Skips generating fixtures (expensive?) by not actually creating the tests - we were creating test data _then_ skipping for several thousand join tests taking ~5 mins of setup work just to skip. * Memoizes data generation for the window tests (often the slowest for me) * Reduces the data generation size and faster comparison algo (before was 10,000 data points with a O(n^2) algo, now it's 1,000 data points with an O(nlogn) comparison algo) * Disables the RAY_DASHBOARD for the context setup commands, and other minor things to speedup ray init * Skips windows on PRs (>50 minutes) only enabled on main, worthwhile trade-off because we will still catch windows-specific failures when merging to main which is quite rare to be fair. ## Related Issues N/A | 7 个月前 |