| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
feat: Add string strip function (#6372) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> Add strip() function to Daft's string functions. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> https://github.com/Eventual-Inc/Daft/issues/3792 | 6 个月前 | |
Initializes working rust-main branch * Old tests are migrated to tests-legacy/ - only new, working tests are left in tests/ * Tests that are failing in tests/ are skipped and tagged with [RUST-INT] * Small fixes made for the code to type-check - removal of udf.py and stubbing out of DataFrame.explode() code | 3 年前 | |
chore: Remove expression namespaces (#5619) ## Changes Made Bye bye ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 10 个月前 | |
[DataFrame Creation APIs] Move DataFrame creation APIs to module-level functions. (#867) This PR makes DataFrame creation APIs (e.g. from_pydict(), read_parquet(), etc.) module-level functions rather than DataFrame classmethods. This is more in keeping with other data libraries that have a single central data abstraction (such as our DataFrame) and allows users to create a DataFrame with less typing. ## API ### Before python from daft import DataFrame df = DataFrame.from_pydict(...) ## OR import daft df = daft.DataFrame.from_pydict(...) ### After python import daft df = daft.from_pydict(...) ## Docs Changes This PR also adds a dedicated "Input/Output" API docs page, similar to [pandas](https://pandas.pydata.org/docs/reference/io.html), [Polars](https://pola-rs.github.io/polars/py-polars/html/reference/io.html), [SparkSQL](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/io.html), etc., with 3 top-level groupings (in-memory data, files, and integrations), with the APIs further grouped into e.g. the file type or the integration library, including both the input and the output APIs. I did something [similar for Ray Datasets](https://docs.ray.io/en/latest/data/api/input_output.html) after user feedback around I/O API discoverability, and the reception after the change was all positive. ## Deprecated API Removal This PR also removes the following deprecated APIs ahead of the 0.1.0 release: - DataFrame.from_files() - DataFrame.from_csv() - DataFrame.from_json() - DataFrame.from_parquet() | 3 年前 | |
feat: added concat_ws function (#6543) GitHub tracking issue: [Eventual-Inc/Daft#3792](https://github.com/Eventual-Inc/Daft/issues/3792) Joining strings with a separator is one of the most common operations in data pipelines - building file paths, composite keys, CSV-like fields, display strings. Daft has concat today, but it propagates nulls and has no separator support. Users who need concat_ws semantics currently have to manually interleave lit(sep) between columns and handle nulls themselves. | 6 个月前 | |
| 8 个月前 | ||
fix(optimizer): Prevent limits from being pushed below explodes in non-top-level projections (#5292) ## Changes Made Currently, limits are being pushed past explodes in projections in the edge case where the explode is not a top-level projection. That's not correct. e.g. import daft from daft.functions import abs, explode daft.from_pydict({"list": [[1, 2], [3, 4]]}).select(abs(explode(daft.col("list")))).limit(2).show() currently returns ╭───────╮ │ list │ │ --- │ │ Int64 │ ╞═══════╡ │ 1 │ ├╌╌╌╌╌╌╌┤ │ 2 │ ├╌╌╌╌╌╌╌┤ │ 3 │ ├╌╌╌╌╌╌╌┤ │ 4 │ ╰───────╯ when it should only return ╭───────╮ │ list │ │ --- │ │ Int64 │ ╞═══════╡ │ 1 │ ├╌╌╌╌╌╌╌┤ │ 2 │ ╰───────╯ This PR fixes this issue. ## Related Issues Closes #5291 Co-authored-by: Srinivas Lade <srinulade1@gmail.com> | 10 个月前 | |
fix: Map Literal <-> Python Dict conversion (#6084) ## Changes Made Return map literals as Python dicts instead of lists of tuples. Update tests to expect dicts for map outputs and value_counts results. ## Related Issues #6081 --------- Co-authored-by: desmondcheongzx <desmondcheongzx@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> | 7 个月前 | |
refactor!: split column expression into unresolved and resolved types (#3804) BREAKING CHANGE: Python struct get syntactic sugar is no longer allowed (ex: col("a.b") -> col("a").struct.get("b"). I did not think it was worth continuing to maintain what is essentially a small compiler for an undocumented and somewhat unclear feature. I would have to make some significant code changes to get it to work with unresolved columns, and I figured I would just rip it out. In the future I'd like to support something like this instead: col("a")["b"]. --- This PR is the first step to non-equality joins in Daft. We introduce an UnresolvedColumn type so that we can resolve it in the logical plan builder into a ResolvedColumn::Basic, ResolvedColumn::JoinSide (for joins), or ResolvedColumn::OuterRef (for subqueries). In addition, we introduce a SubqueryAlias logical op that is used to determine the sources for UnresolvedColumns with plan_ref set. Along with this PR is a refactor of our SQL planner. The SubqueryAlias op is now used to encode naming and scoping information, so the Relation struct is no longer necessary. Additionally, I discovered that we can significantly simplify SQLPlanner::plan_non_agg_query and SQLPlanner::plan_aggregate_query. I believe the prior complexity came due to the fact that in many clauses (GROUP BY, ORDER BY, etc), you can reference both columns in the input tables as well as aliases in the select clause. However, the planner's bound_columns already solves this issue. This may yield slightly less performant plans due to redundant expressions, but that should be handled by the optimizer instead of the SQL planner. Moreover, I was able to remove a sort from plan_aggregate_query, which may yield both performance and correctness improvements, since the prior code assumed you can split a sort up into two, however that may lead to different results due to sort orders (ORDER BY a, b != ORDER BY b, a) as well as whether our sort impl is stable or unstable. | 1 年前 | |
chore: Upgrade Ruff ruleset to 3.9 and add from __future__ import annotations (#4393) | 1 年前 | |
feat: Add guess_mime_type scalar expression for MIME type detection from bytes (#5883) | 8 个月前 | |
feat(swordfish): plan caching (#6278) ## Summary Introduces fingerprint-based plan caching so multiple executions of the same logical plan share a single pipeline, rather than building a new one each time. ### Key changes - ** PipelineMessage enum** — Replaces raw MicroPartition in all pipeline channels with Morsel { input_id, partition } and Flush(input_id), allowing a single pipeline to multiplex data from multiple logical inputs - **NativeExecutor becomes stateful** — Maintains a HashMap<u64, PlanState> keyed by plan fingerprint; repeated calls to run() with the same fingerprint reuse the existing pipeline - **MessageRouter** — Routes pipeline output to per-input_id unbounded channels so each caller gets only its own results - **try_finish() lifecycle API** — Callers signal input completion and collect per-input stats; pipeline is torn down when the last input finishes - **Per-input runtime stats** — RuntimeStatsManager tracks (NodeID, InputId) pairs with merge support for aggregated views - **next_event helper** — Shared tokio::select! loop abstraction used by blocking sinks, streaming sinks, intermediate ops, and joins - **Simplified BlockingSink trait** — Removed BlockingSinkFinalizeOutput::HasMoreOutput (was dead code); finalize now returns Vec<MicroPartition> directly; make_state takes InputId - **Simplified StreamingSink trait** — Removed StreamingSinkOutput::HasMoreOutput variant - **All node types updated** — Sources, intermediate ops, blocking sinks, streaming sinks, joins, and concat all handle Morsel/Flush per-input routing - **Python integration** — native_executor.py and flotilla.py call try_finish() for lifecycle management ### Files changed ~49 files across daft-local-execution (pipeline, sources, sinks, joins, stats), daft-distributed, common-metrics, and Python runners. ## Test plan - [ ] Full test suite with DAFT_RUNNER=native - [ ] Full test suite with DAFT_RUNNER=ray - [ ] Concurrent query execution sharing the same plan fingerprint - [ ] Verify per-input stats are correctly reported and merged - [ ] TPC-H benchmarks to check for performance regressions 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: R. C. Howell <5731503+rchowell@users.noreply.github.com> | 6 个月前 | |
chore: change series literal to list (#4896) ## Changes Made Remove LiteralValue::Series and replace it with a LiteralValue::List. I don't think we use series literals as series anymore so this name makes more sense. Additionally I've changed the datatypes/conversions to reflect that this is actually a list-type scalar value. This is part of a larger change that I'm working on to unify type conversions between Python and Daft. Notes: - Removed test_length_mismatch_all_types because length mismatch is now handled by the RecordBatch constructor so there's a different error, which should happen for any expression, so no need to test it specifically for this one. - Removed test_hash_int_array_with_bad_length - same reason as above. - Removed test_table_expr_is_in_different_input_types. It tested behavior that we did not document, so I'm removing it instead of fixing it. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [x] Documented in API Docs (if applicable) - [x] Documented in User Guide (if applicable) - [x] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [x] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) --------- Co-authored-by: Colin Ho <colin.ho99@gmail.com> | 1 年前 | |
| 8 个月前 | ||
chore: Remove expression namespaces (#5619) ## Changes Made Bye bye ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 10 个月前 | |
chore: Remove expression namespaces (#5619) ## Changes Made Bye bye ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 10 个月前 | |
chore: Remove expression namespaces (#5619) ## Changes Made Bye bye ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 10 个月前 | |
chore: Remove expression namespaces (#5619) ## Changes Made Bye bye ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 10 个月前 | |
chore: Remove expression namespaces (#5619) ## Changes Made Bye bye ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 10 个月前 | |
fix: short-circuit evaluation for coalesce (#6525) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> Fixes #4069 by adding proper short-circuit behavior to coalesce. - Added Expr::Coalesce with short-circuit logic - Implemented early-exit in record batch evaluation - Preserved semantics during UDF optimization - Added short-circuit tests ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> Closes #4069 --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 6 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 6 个月前 | ||
| 3 年前 | ||
| 10 个月前 | ||
| 3 年前 | ||
| 6 个月前 | ||
| 8 个月前 | ||
| 10 个月前 | ||
| 7 个月前 | ||
| 1 年前 | ||
| 1 年前 | ||
| 8 个月前 | ||
| 6 个月前 | ||
| 1 年前 | ||
| 8 个月前 | ||
| 10 个月前 | ||
| 10 个月前 | ||
| 10 个月前 | ||
| 10 个月前 | ||
| 10 个月前 | ||
| 6 个月前 |