| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
[RUST] Migrate legacy tests (#775) Migration of legacy tests from tests_legacy/ to tests/ * tests_legacy/null_tests/ are now moved into their respective tests in tests/dataframe/*.py * Deleted tests_legacy/expression_operators/* tests, which should now be tested on the Series-level * Deleted tests_legacy/experimental/* tests * Deleted tests_legacy/internal since these are no longer used from Python * Deleted tests_legacy/runners since these are redundant and don't serve much purpose --------- Co-authored-by: Jay Chia <jaychia94@gmail.com@users.noreply.github.com> | 3 年前 | |
[RUST] Migrate legacy tests (#775) Migration of legacy tests from tests_legacy/ to tests/ * tests_legacy/null_tests/ are now moved into their respective tests in tests/dataframe/*.py * Deleted tests_legacy/expression_operators/* tests, which should now be tested on the Series-level * Deleted tests_legacy/experimental/* tests * Deleted tests_legacy/internal since these are no longer used from Python * Deleted tests_legacy/runners since these are redundant and don't serve much purpose --------- Co-authored-by: Jay Chia <jaychia94@gmail.com@users.noreply.github.com> | 3 年前 | |
chore: Remove the old Ray Runner (#5375) ## Changes Made 🎉🎂🥳 Can finally delete it, Flotilla supports all necessary features. Additional Related Features Removed: * DataFrame.num_partitions: This is computed using the old Ray runner's planner. We could use the new Ray runner, but it seems kind of unnecessary * Context Settings: --------- Co-authored-by: Colin Ho <colin.ho99@gmail.com> | 10 个月前 | |
chore: bump mypy and ruff in pre-commit (#5836) ## Changes Made Bump mypy to 1.19.1 and ruff to 0.14.10 and resolve pre-commit errors. Rationale: my local ruff version is much higher than that in the project pre-commit, so I am getting a lot of linter warnings in my IDE. It is also about time to upgrade lints to utilize newer Python language features. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 9 个月前 | |
chore: Upgrade Ruff ruleset to 3.9 and add from __future__ import annotations (#4393) | 1 年前 | |
[BUG] Enable groupby with alias for native executor (#2917) Fixed this already for the current translation path back in https://github.com/Eventual-Inc/Daft/pull/2790, doing the same for swordfish --------- Co-authored-by: Colin Ho <colinho@Colins-MacBook-Pro.local> Co-authored-by: Colin Ho <colinho@Colins-MBP.localdomain> | 2 年前 | |
perf: Call async udfs asynchronously (#5451) ## Changes Made Call async UDFs from swordfish asynchronously using pyo3-async-runtimes. This allows async UDFs to actually run asynchronously without blocking a thread. Consider the following example, where we run 2 async udfs. On main, currently only a total of num_cores across _both_ async udfs can run concurrently, but with this PR, we can run all concurrently as long as they have input ready. Running on my macbook the speed up is ~20s -> ~2s. import daft import asyncio @daft.func async def my_udf(foo: int) -> int: await asyncio.sleep(1) return foo + 1 df = ( daft.from_pydict({"foo": [i for i in range(1000)]}).into_batches(10) .select(my_udf(daft.col("foo"))) .select(my_udf(daft.col("foo"))) .collect() ) ## Implementation Details 1. Convert def call_async_batch to an async def call_async_batch 2. Add an async fn call_async to row wise python udf that uses pyo3_async_runtimes to call async def call_async_batch 3. Create a new async_udf streaming sink. 4. Call the new async fn call_async when evaluating the udf in async_udf operator, spawn the future on a joinset and poll it during calls to execute. Notes: - We have to store a reference to a running asyncio event loop in order to use pyo3-async-runtimes. This works natively on flotilla because we use async Ray actors, but on swordfish we need to create one ourselves, so I opted to create a background event loop on a separate python thread. - I verified that this approach works even on Jupyter which by default runs it's own event loop - The joinset approach will allow the udfs to be driven asynchronously, but their results will only be retrieved during calls to execute / finalize. ## Todos - yield incrementally from streaming sink finalize - consider using orderedjoinset to maintain order - batch async udfs - user defined concurrency control ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 11 个月前 | |
[CHORE] Swordfish specific test fixtures (#3164) This PR sets up a few swordfish related test fixtures, specifically: - Parameterize default_morsel_size = [1, None] for dataframe tests that do any into/repartitioning. This is to make sure that the operator parallelism is working. - Setup iteration tests in test_iter.py - Makes the ordering assertions stricter on some tests. E.g. some tests do assert df.sort(col) == expected, but there are other columns in df that may not be sorted, and this won't be enough if morsel_size = 1. This isn't a problem with swordfish but the test, where the sort should actually involve more columns. Big note: There was a problem with pivot not getting applied correctly. This is because a dataframe pivot operation comprises of an agg + the actual pivoting, but previously the pivot was implemented as an intermediate operator, and the results of the agg were getting buffered. In order for the pivot to work it has to receive all values with the same group_by keys. This PR implements simplifies Pivot as a BlockingSink, so all the work is in there. --------- Co-authored-by: Colin Ho <colinho@Colins-MacBook-Pro.local> | 1 年前 | |
feat: support count(1) in dataframe and choose the cheap column (#4977) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> Not count(1) in dataframe is not supported: In [49]: df.count(1).show() --------------------------------------------------------------------------- TypeError Traceback (most recent call last) Cell In[49], line 1 ----> 1 df.count(1).show() [... skipping hidden 1 frame] File /data00/code/tmp2/Daft/daft/dataframe/dataframe.py:3011, in DataFrame.count(self, *cols) 3008 raise ValueError("Cannot call count() with both * and column names") 3010 # Otherwise, perform a column-wise count on the specified columns -> 3011 return self._apply_agg_fn(Expression.count, cols) File /data00/code/tmp2/Daft/daft/dataframe/dataframe.py:2854, in DataFrame._apply_agg_fn(self, fn, cols, group_by) 2852 groupby_name_set = set() if group_by is None else group_by.to_name_set() 2853 cols = tuple(c for c in self.column_names if c not in groupby_name_set) -> 2854 exprs = self._wildcard_inputs_to_expressions(cols) 2855 return self._agg([fn(c) for c in exprs], group_by) File /data00/code/tmp2/Daft/daft/dataframe/dataframe.py:1596, in DataFrame._wildcard_inputs_to_expressions(self, columns) 1594 """Handles wildcard argument column inputs.""" 1595 column_input: Iterable[ColumnInputType] = columns[0] if len(columns) == 1 else columns # type: ignore -> 1596 return column_inputs_to_expressions(column_input) File /data00/code/tmp2/Daft/daft/utils.py:126, in column_inputs_to_expressions(columns) 123 from daft.expressions import col 125 column_iter: Iterable[ColumnInputType] = [columns] if is_column_input(columns) else columns # type: ignore --> 126 return [col(c) if isinstance(c, str) else c for c in column_iter] TypeError: 'int' object is not iterable Using this pr: In [5]: df.count(1).show() ╭────────╮ │ count │ │ --- │ │ UInt64 │ ╞════════╡ │ 4 │ ╰────────╯ At the same time, there is also a certain improvement in performance. Before modification, the performance data is as follows -------------------------------------------------------------------------------------------------------- benchmark: 4 tests ------------------------------------------------------------------------------------------------------- Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- test_count_with_col_name[float_id-float] 18.2832 (1.0) 24.8761 (1.07) 20.1458 (1.0) 2.6823 (1.34) 19.3201 (1.0) 2.0831 (1.0) 1;1 49.6381 (1.0) 5 1 test_count_with_col_name[int_id-int] 18.6567 (1.02) 23.2190 (1.0) 21.0213 (1.04) 1.9988 (1.0) 21.3333 (1.10) 3.6173 (1.74) 2;0 47.5707 (0.96) 5 1 test_count_with_col_name[string_content-string] 314.5036 (17.20) 633.3133 (27.28) 434.0078 (21.54) 120.1076 (60.09) 414.9809 (21.48) 120.6531 (57.92) 1;0 2.3041 (0.05) 5 1 test_count_with_col_name[binary_content-binary] 1,836.5554 (100.45) 1,889.4419 (81.37) 1,864.8847 (92.57) 22.0420 (11.03) 1,857.1332 (96.12) 34.7329 (16.67) 2;0 0.5362 (0.01) 5 1 ----- Use this pr, the performance indicators are as follows: ------------------------------------------------------------------------------------------------ benchmark: 4 tests ----------------------------------------------------------------------------------------------- Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- test_count_with_col_name[int_id-int] 18.3580 (1.0) 25.2071 (1.00) 21.3138 (1.0) 2.7985 (1.68) 20.9401 (1.0) 4.5452 (2.61) 2;0 46.9180 (1.0) 5 1 test_count_with_col_name[float_id-float] 20.5159 (1.12) 25.1621 (1.0) 22.8351 (1.07) 1.6660 (1.0) 22.9081 (1.09) 1.7429 (1.0) 2;0 43.7923 (0.93) 5 1 test_count_with_col_name[binary_content-binary] 21.8774 (1.19) 27.1461 (1.08) 23.9783 (1.13) 2.1795 (1.31) 24.3946 (1.16) 3.2378 (1.86) 1;0 41.7043 (0.89) 5 1 test_count_with_col_name[string_content-string] 22.6195 (1.23) 30.9669 (1.23) 25.7760 (1.21) 3.5412 (2.13) 23.9304 (1.14) 5.4121 (3.11) 1;0 38.7958 (0.83) 5 1 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The benchmark test cases are as follows: from __future__ import annotations import pytest import daft PATH = "/tmp/test_count_1" # Consolidated test configuration with built-in IDs TEST_CASES = [ pytest.param("int_id", "int", id="first_col_int"), pytest.param("float_id", "float", id="first_col_float"), pytest.param("string_content", "string", id="first_col_string"), pytest.param("binary_content", "binary", id="first_col_binary"), ] def generate_data(): # Adjust column order to ensure the target column is first data = { "int_id": [1] * 1000, "float_id": [0.001] * 1000, "string_content": ["a"* 100000] * 1000, "binary_content": [b"a" * 1000000] * 1000, } for case in TEST_CASES: param, _ = case.values col_order = [param] + ["int_id", "float_id", "string_content", "binary_content"] col_order = list(dict.fromkeys(col_order)) df = daft.from_pydict({k: data[k] for k in col_order}) path = f"{PATH}_{param}" df.write_parquet(path, write_mode = "overwrite") generate_data() @pytest.mark.parametrize("col_name, _", [case.values[:2] for case in TEST_CASES]) def test_count_with_col_name(benchmark, col_name, _): """Benchmark count(*) with different first column layouts.""" def operation(): path = f"{PATH}_{col_name}" df = daft.read_parquet(path) return df.count().to_pydict() result = benchmark.pedantic(operation, rounds=5, warmup_rounds=1) assert result["count"][0] == 1000 (Showing first 1 of 1 rows) ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) --------- Co-authored-by: Colin Ho <colin.ho99@gmail.com> Co-authored-by: EC2 Default User <ec2-user@ip-172-31-22-159.us-west-2.compute.internal> | 1 年前 | |
chore: Upgrade Ruff ruleset to 3.9 and add from __future__ import annotations (#4393) | 1 年前 | |
chore: Remove expression namespaces (#5619) ## Changes Made Bye bye ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 10 个月前 | |
ci: Remove Tests for the Old Ray Runner (#5374) Co-authored-by: Colin Ho <colin.ho99@gmail.com> | 11 个月前 | |
chore: Upgrade Ruff ruleset to 3.9 and add from __future__ import annotations (#4393) | 1 年前 | |
feat: Add DISTINCT ON / drop_duplicates on a subset of columns (#4513) | 1 年前 | |
fix: Optimize the display information of Join nodes in query plan (#5617) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> Currently, when displaying the query plan containing a "join" node using explain, we can't intuitively distinguish between the left and right tables, especially in the case of a Broadcast Join, which requires the is_swapped field to assist in judgment. This PR mainly unifies the information format and elements of the Join nodes displayed in explain, and explicitly identifies the Broadcaster and Receiver for Broadcast Join. For example: text == Physical Plan == * BroadcastJoin | Type: Inner | Left: Join key = col(1: l_name), Role = Receiver | Right: Join key = col(1: s_name), Role = Broadcaster | Null equals nulls: [false] |\ | * ScanTaskSource: | | Num Scan Tasks = 100 | | Estimated Scan Bytes = 728327 | | Pushdowns: {filter: not(is_null(col(l_name)))} | | Schema: {id#Int64, l_name#Utf8, l_email#Utf8} | | Scan Tasks: [ ... ] | * Project: col(0: id) as right.id, col(1: s_name), col(2: s_email) | Resource request = None | * ScanTaskSource: | Num Scan Tasks = 10 | Estimated Scan Bytes = 67702 | Pushdowns: {filter: not(is_null(col(s_name)))} | Schema: {id#Int64, s_name#Utf8, s_email#Utf8} | Scan Tasks: [ ... ] ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly Signed-off-by: plotor <zhenchao.wang@hotmail.com> | 9 个月前 | |
| 9 个月前 | ||
[FEAT]: Dataframe.filter method (#2853) closes https://github.com/Eventual-Inc/Daft/issues/2846 Note, this does not rename 'where' to 'filter' but instead just adds an alias. We can revisit at a later date if we want to deprecate the where method. | 2 年前 | |
feat: f"" string style string concatenation (#4500) ## Changes Made Implemented f"" string style string concatenation (see linked issue for more detail) and added some tests. Probs want to also add this to the docs at some point? ## Related Issues https://github.com/Eventual-Inc/Daft/issues/4001 ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 1 年前 | |
refactor!: split column expression into unresolved and resolved types (#3804) BREAKING CHANGE: Python struct get syntactic sugar is no longer allowed (ex: col("a.b") -> col("a").struct.get("b"). I did not think it was worth continuing to maintain what is essentially a small compiler for an undocumented and somewhat unclear feature. I would have to make some significant code changes to get it to work with unresolved columns, and I figured I would just rip it out. In the future I'd like to support something like this instead: col("a")["b"]. --- This PR is the first step to non-equality joins in Daft. We introduce an UnresolvedColumn type so that we can resolve it in the logical plan builder into a ResolvedColumn::Basic, ResolvedColumn::JoinSide (for joins), or ResolvedColumn::OuterRef (for subqueries). In addition, we introduce a SubqueryAlias logical op that is used to determine the sources for UnresolvedColumns with plan_ref set. Along with this PR is a refactor of our SQL planner. The SubqueryAlias op is now used to encode naming and scoping information, so the Relation struct is no longer necessary. Additionally, I discovered that we can significantly simplify SQLPlanner::plan_non_agg_query and SQLPlanner::plan_aggregate_query. I believe the prior complexity came due to the fact that in many clauses (GROUP BY, ORDER BY, etc), you can reference both columns in the input tables as well as aliases in the select clause. However, the planner's bound_columns already solves this issue. This may yield slightly less performant plans due to redundant expressions, but that should be handled by the optimizer instead of the SQL planner. Moreover, I was able to remove a sort from plan_aggregate_query, which may yield both performance and correctness improvements, since the prior code assumed you can split a sort up into two, however that may lead to different results due to sort orders (ORDER BY a, b != ORDER BY b, a) as well as whether our sort impl is stable or unstable. | 1 年前 | |
docs: categorize functions and make each function its own page (#4838) ## Changes Made Change Daft functions docs to categorize them, as I wanted to do with expressions before. Now, the functions are split into different files under daft/functions/, and if you add a function, it will automatically be added to the functions page in the category corresponding to the file it's in! Screenshots: <img width="1584" height="1106" alt="image" src="https://github.com/user-attachments/assets/c5f1d309-1c57-4571-97b6-d303328bfb5a" /> <img width="1584" height="1104" alt="image" src="https://github.com/user-attachments/assets/4edc95bc-f2a1-4d09-97cb-b15eff960a02" /> ## Related Issues #4737 #4824 ## Checklist - [x] Documented in API Docs (if applicable) - [x] Documented in User Guide (if applicable) - [x] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [x] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 1 年前 | |
chore: Remove the old Ray Runner (#5375) ## Changes Made 🎉🎂🥳 Can finally delete it, Flotilla supports all necessary features. Additional Related Features Removed: * DataFrame.num_partitions: This is computed using the old Ray runner's planner. We could use the new Ray runner, but it seems kind of unnecessary * Context Settings: --------- Co-authored-by: Colin Ho <colin.ho99@gmail.com> | 10 个月前 | |
feat: Flotilla into partitions (#4963) ## Changes Made Implements into_partitions for flotilla. First collect all input tasks. Then either coalesces inputs or splits them such that exactly num_partitions partitions are emitted from this operator. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 1 年前 | |
feat: dynamic batching per operator (#5676) ## Changes Made adds new dynamic batching functionality for pipeline operators. Now each operator has an additional associated type of BatchingStrategy which indicates how that specific operator should be batched. By default this uses the StaticBatchingStrategy which retains the functionality prior to this PR. Summary of this PR - Operator specifies strategy via associated type and batching_strategy function - by default, everything uses the StaticBatchingStrategy - Workers send updates to the batching_context - Dispatcher uses the batching strategy to split morsels into smaller or bigger batches depending on the operator's batching strategy. Will use the messages from the workers to adjust the batch size on the fly. - RowBasedBuffer modified to be able to accomodate a "target batch size" An operator can specify a custom batching strategy such as the newly introduced LatencyConstrainedBatchingStrategy which will batch the morsels based off latency requirements. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 10 个月前 | |
chore: optimizes slow tests in CI/CD (#6029) ## Results The PR reduced unit-test time (3.13, native, 22.0.0, ubuntu-latest, false) from [33m 18s](https://github.com/Eventual-Inc/Daft/actions/runs/20981828603/job/60308365885?pr=6023) to 13m 44s — saving about 20 mins which is nice. The test_context is still slow, but it has very few tests, a big win is parallelization and taking >5 mins off just by skipping the setup for skipped tests. This also reduced the ray tests from 45 mins to 27 mins, and there should be plenty more savings here. Then disabling windows eliminated a 50+ min action. text ================================================================================ OVERALL SUMMARY ================================================================================ Tests Analyzed: 9620 Total Time: 941.32s Average Time: 0.05s Max Time: 14.70s Min Time: 0.01s It took 188s to run make test on my mac which is nice; it also crushed my machine while it was running 🥲 ## Changes Made * Adds -n auto to pytest for parallelize testing on multiple cores * Skips generating fixtures (expensive?) by not actually creating the tests - we were creating test data _then_ skipping for several thousand join tests taking ~5 mins of setup work just to skip. * Memoizes data generation for the window tests (often the slowest for me) * Reduces the data generation size and faster comparison algo (before was 10,000 data points with a O(n^2) algo, now it's 1,000 data points with an O(nlogn) comparison algo) * Disables the RAY_DASHBOARD for the context setup commands, and other minor things to speedup ray init * Skips windows on PRs (>50 minutes) only enabled on main, worthwhile trade-off because we will still catch windows-specific failures when merging to main which is quite rare to be fair. ## Related Issues N/A | 8 个月前 | |
| 9 个月前 | ||
feat!: unify Python -> Daft type conversions (#5201) BREAKING CHANGES: - list[object] element inputs to daft.from_pydict and similar methods are now converted into List[Python] instead of Python - Before: daft.from_pydict({"x": [[object()], [object(), object()]]}) -> schema of (x: Python) - After: schema of (x: List[Python]) - Python int inputs less than 2^31-1 are now converted to Int64 instead of Int32 - Flat numpy array inputs are converted into Tensor instead of List - I believe we used to be able to infer types from dicts or tuples of types. I've removed them and instead added the ability to infer from TypedDict and tuple[T1, T2, ...]. They were not documented anyway ## Changes Made Unified and documented the conversions from Python objects and types to Daft. The core of the conversion logic lies in: - Literal::from_pyobj in src/daft-core/lit/python.rs, which is now the ultimate source of truth for our Python to Daft conversions - DataType.infer_from_type in daft/datatype.py, which maps Python types to Daft types for things like inferring a UDF return type. - Since Python types do not always have sufficient information to derive the full Daft type, this function deviates slightly from Literal::from_pyobj in those cases. Users can also use DataType.infer_from_object to check the logic in Literal::from_pyobj. Other changes: - Added type conversion tables and moved them to docs/api/datatypes/type_conversions.md - Changed the Rust NdArray type from a trait to an enum so that code in daft-core can match on the variants. - Removed a significant amount of PythonArray casting logic, changing them to go through Literal::from_pyobj instead to make everything consistent and cleaner. - Removed ability to run concat agg on Python-type columns. This was only used because we could not serialize List[Python] so we just kept everything as Python lists. But that is fixed with #5270 - Changed Literal::Struct to store mapping from name to item instead of field to item, since the dtype of a field can be inferred from the item. Some UDFs that now work: - returning tuple py >>> import daft >>> @daft.func ... def tuple_udf(x) -> tuple[str, int]: ... return (str(x), int(x)) ... >>> df = daft.from_pydict({"input": [1, 2, 3]}) >>> df.select(tuple_udf(df["input"])).show() ╭─────────────────────────────╮ │ input │ │ --- │ │ Struct[_0: Utf8, _1: Int64] │ ╞═════════════════════════════╡ │ {_0: 1, │ │ _1: 1, │ │ } │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ {_0: 2, │ │ _1: 2, │ │ } │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ {_0: 3, │ │ _1: 3, │ │ } │ ╰─────────────────────────────╯ (Showing first 3 of 3 rows) - returning list of numpy array: py >>> import daft >>> import numpy as np >>> import numpy.typing as npt >>> @daft.func ... def array_list_udf(x) -> list[npt.NDArray[np.uint]]: ... return [np.array([x], dtype=np.uint), np.array([x*2, x*3], dtype=np.uint)] ... >>> df = daft.from_pydict({"input": [1, 2, 3]}) >>> df.select(array_list_udf(df["input"])).show() ╭────────────────────────────────╮ │ input │ │ --- │ │ List[Tensor(UInt64)] │ ╞════════════════════════════════╡ │ [<Tensor shape=(1)>, <Tensor … │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ [<Tensor shape=(1)>, <Tensor … │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ [<Tensor shape=(1)>, <Tensor … │ ╰────────────────────────────────╯ (Showing first 3 of 3 rows) ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [x] Documented in API Docs (if applicable) - [x] Documented in User Guide (if applicable) - [x] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [x] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 1 年前 | |
feat(agg): support map_groups with v2 udf (#5927) ## Summary Extend grouped aggregation map_groups to support Daft's new UDF system (@daft.func.batch and @daft.cls/@daft.method.batch) in addition to legacy @daft.udf. ## Changes ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 9 个月前 | |
chore: Remove expression namespaces (#5619) ## Changes Made Bye bye ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 10 个月前 | |
fix(observability): Clean up progress bar naming (#6028) ## Changes Made While working on the One True Progress Bar (™️ pending) PR, noticed a couple of things we can clean up. | 8 个月前 | |
feat(flotilla): Hash Join (#4700) | 1 年前 | |
feat(flotilla): Distributed Pivot (#5199) | 1 年前 | |
feat: No truncate in .collect preview (#5632) ## Changes Made Follow on from https://github.com/Eventual-Inc/Daft/pull/5597. Remove truncation logic when previewing with .collect(num_preview_rows=N) ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 10 个月前 | |
feat: creates daft-ir and daft-proto (#4539) ## Changes Made * Creates a daft-ir crate for a consolidate IR view + from/to proto implementations (WIP). * Creates a daft-proto crate which holds daft's IR protobuf definitions. * Adds initial schema, datatype, expression (rex), and logical plan (rel) definitions. ## TODOs. * The exports (and structure) are not finalized. * Effectively all relational operator (rel) definitions. * Testing. testing. testing. ## Related Issues * n/a ## Checklist - [n/a] Documented in API Docs (if applicable) - [n/a] Documented in User Guide (if applicable) - [n/a] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [n/a] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 1 年前 | |
chore: Hint users to use .collect when printing empty dataframes (#5616) ## Changes Made Add a hint to the user to use .collect if they print an unmaterialized dataframe. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 10 个月前 | |
feat: Streaming sample by size (#5663) ## Changes Made Implement a streaming sample by size algorithm for swordfish. - For sampling without replacement, we assign a random number to each row, and simply keep around the rows with the lowest N random numbers. - For sampling with replacement where size=N, we create N slots. For each slot, assign a random number to each row. If the random number is smaller than the existing one in that slot, replace it. This PR only implements for swordfish. For flotilla, we will need to do some merging of local samples, this will be in the next PR. Essentially, we keep the random numbers per row / slot, merge the data onto a single node, and then pick the N rows with the smallest random numbers (per slot if with replacement). ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 10 个月前 | |
feat: supports passing a projection kwargs in select (#4884) ## Changes Made Adds support for a dict or projections to select which is backwards compatible but consistent with .with_columns. This format makes it easy to list down the columns prior to inserting somewhere else. python In [1]: import daft In [2]: df = daft.from_pydict({"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}) In [3]: df = df.select({ ...: "sum": df["x"] + df["y"] + df["z"], ...: "product": df["x"] * df["y"] * df["z"], ...: }) In [4]: df.show() ╭───────┬─────────╮ │ sum ┆ product │ │ --- ┆ --- │ │ Int64 ┆ Int64 │ ╞═══════╪═════════╡ │ 12 ┆ 28 │ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┤ │ 15 ┆ 80 │ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌┤ │ 18 ┆ 162 │ ╰───────┴─────────╯ (Showing first 3 of 3 rows) ## Related Issues Closes #4883 ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 1 年前 | |
fix(cast): handle whitespace in string-to-number casting (#5955) ## Changes Made - Handle whitespace trimming for Utf8 -> numeric casting in cast.rs - Add Rust unit tests for whitespace handling in cast operations - Simplify run_process to return raw subprocess stdout - Remove | tr -d ' ' workaround from tests since cast now handles whitespace ## Motivation String-to-number casting in Daft was strict about whitespace, returning None for inputs like " 3 ". This differed from: - Python's int() - Pandas - NumPy - DuckDB All of which handle leading/trailing whitespace gracefully. ### Comparison (Numeric) | Engine | ' 3 ' → int | |--------|-----------------| | Python int() | 3 | | Pandas | 3 | | DuckDB | 3 | | Polars | None | | Daft (before) | None | | **Daft (after)** | **3** | ## Implementation The fix is implemented in src/daft-core/src/array/ops/cast.rs: - Before calling arrow2's cast, check if we're casting Utf8 -> numeric - If so, create a trimmed copy of the string array - Pass the trimmed data to arrow2's existing cast logic Trade-off: Temporarily duplicates string data during cast, but gains code reuse and maintainability. ## Scope This PR only affects **numeric** casting (int, float). ### Date Parsing Date parsing already handles whitespace correctly - chrono::NaiveDate::from_str (used by arrow2) is lenient with whitespace. Added a test to document this existing behavior. ## Related Issues Fixes the root cause of #5942 Related to #5738 | 9 个月前 | |
feat: union ops (#3872) adds the following to both dataframe and sql - union (distinct union) - union_all - union_by_name (distinct union by name) - union_all_by_name closes https://github.com/Eventual-Inc/Daft/issues/3855 | 1 年前 | |
chore: bump mypy and ruff in pre-commit (#5836) ## Changes Made Bump mypy to 1.19.1 and ruff to 0.14.10 and resolve pre-commit errors. Rationale: my local ruff version is much higher than that in the project pre-commit, so I am getting a lot of linter warnings in my IDE. It is also about time to upgrade lints to utilize newer Python language features. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 9 个月前 | |
feat: No truncate in .collect preview (#5632) ## Changes Made Follow on from https://github.com/Eventual-Inc/Daft/pull/5597. Remove truncation logic when previewing with .collect(num_preview_rows=N) ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 10 个月前 | |
chore: bump mypy and ruff in pre-commit (#5836) ## Changes Made Bump mypy to 1.19.1 and ruff to 0.14.10 and resolve pre-commit errors. Rationale: my local ruff version is much higher than that in the project pre-commit, so I am getting a lot of linter warnings in my IDE. It is also about time to upgrade lints to utilize newer Python language features. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 9 个月前 | |
feat: sample api supports precise sampling by size params (#5600) ## Changes Made The sample api of daft added a new parameter named size, supporting precise sampling by the number of rows and aligning the semantics of pandas. https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.sample.html Now daft, like pandas, supports sampling based on fraction and size, and is particularly useful in scenarios where precise sampling is required. <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues https://github.com/Eventual-Inc/Daft/issues/5599 <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Does this PR introduce any user-facing change? Yes. the sample api added a new parameter named size. ## How to test new UTs. ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly --------- Co-authored-by: cancai <caican@xiaomi.com> Co-authored-by: Mi Code <micode@xiaomi.com> | 10 个月前 | |
chore: Upgrade Ruff ruleset to 3.9 and add from __future__ import annotations (#4393) | 1 年前 | |
refactor: Cleanup Dtype Names (#5400) | 11 个月前 | |
chore: Remove expression namespaces (#5619) ## Changes Made Bye bye ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 10 个月前 | |
chore: Upgrade Ruff ruleset to 3.9 and add from __future__ import annotations (#4393) | 1 年前 | |
[FEAT] [New Query Planner] All functional tests pass + add to CI. (#1274) This PR ensures that all functional tests pass for the new query planner and adds the new query planner to the CI job matrix; all tests are covered and pass except for the Python query planner optimization tests. The use_new_planner test fixture is removed, since DAFT_NEW_QUERY_PLANNER=1 make test will now work with expected test coverage. | 3 年前 | |
chore: Upgrade Ruff ruleset to 3.9 and add from __future__ import annotations (#4393) | 1 年前 | |
chore: Remove expression namespaces (#5619) ## Changes Made Bye bye ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 10 个月前 | |
[FEAT] Enable concat for swordfish (#2976) Implement concat as a streaming sink in swordfish. Concatenating two dataframes in a streaming executor is pretty simple. You just stream the left side, then stream the right side. This PR implements just that. --------- Co-authored-by: Colin Ho <colinho@Colins-MBP.localdomain> Co-authored-by: Colin Ho <colinho@Colins-MacBook-Pro.local> | 1 年前 | |
chore: Upgrade Ruff ruleset to 3.9 and add from __future__ import annotations (#4393) | 1 年前 | |
chore: Upgrade Ruff ruleset to 3.9 and add from __future__ import annotations (#4393) | 1 年前 | |
[BUG] with_column with existing column name should not reorder columns (#2381) | 2 年前 | |
feat: with_column(s)_renamed expression for DataFrame (#3732) [issue #3705](https://github.com/Eventual-Inc/Daft/issues/3705) Add with_column_renamed and with_columns_renamed to DataFrame API (no SQL). def with_column_renamed(self, existing: str, new: str) -> "DataFrame": """Renames a column in the current DataFrame. Example: >>> import daft >>> df = daft.from_pydict({"x": [1, 2, 3], "y": [4, 5, 6]}) >>> df.with_column_renamed("x", "foo").show() ╭───────┬───────╮ │ foo ┆ y │ │ --- ┆ --- │ │ Int64 ┆ Int64 │ ╞═══════╪═══════╡ │ 1 ┆ 4 │ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤ │ 2 ┆ 5 │ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤ │ 3 ┆ 6 │ ╰───────┴───────╯ <BLANKLINE> (Showing first 3 of 3 rows) Args: existing (str): name of the existing column to rename new (str): new name for the column Returns: DataFrame: DataFrame with the column renamed. """ def with_columns_renamed(self, colsMap: Dict[str, str]) -> "DataFrame": """Renames multiple columns in the current DataFrame. Example: >>> import daft >>> df = daft.from_pydict({"x": [1, 2, 3], "y": [4, 5, 6]}) >>> df.with_columns_renamed({"x": "foo", "y": "bar"}).show() ╭───────┬───────╮ │ foo ┆ bar │ │ --- ┆ --- │ │ Int64 ┆ Int64 │ ╞═══════╪═══════╡ │ 1 ┆ 4 │ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤ │ 2 ┆ 5 │ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤ │ 3 ┆ 6 │ ╰───────┴───────╯ <BLANKLINE> (Showing first 3 of 3 rows) """ | 1 年前 | |
[BUG] with_column with existing column name should not reorder columns (#2381) | 2 年前 | |
feat: with_column(s)_renamed expression for DataFrame (#3732) [issue #3705](https://github.com/Eventual-Inc/Daft/issues/3705) Add with_column_renamed and with_columns_renamed to DataFrame API (no SQL). def with_column_renamed(self, existing: str, new: str) -> "DataFrame": """Renames a column in the current DataFrame. Example: >>> import daft >>> df = daft.from_pydict({"x": [1, 2, 3], "y": [4, 5, 6]}) >>> df.with_column_renamed("x", "foo").show() ╭───────┬───────╮ │ foo ┆ y │ │ --- ┆ --- │ │ Int64 ┆ Int64 │ ╞═══════╪═══════╡ │ 1 ┆ 4 │ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤ │ 2 ┆ 5 │ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤ │ 3 ┆ 6 │ ╰───────┴───────╯ <BLANKLINE> (Showing first 3 of 3 rows) Args: existing (str): name of the existing column to rename new (str): new name for the column Returns: DataFrame: DataFrame with the column renamed. """ def with_columns_renamed(self, colsMap: Dict[str, str]) -> "DataFrame": """Renames multiple columns in the current DataFrame. Example: >>> import daft >>> df = daft.from_pydict({"x": [1, 2, 3], "y": [4, 5, 6]}) >>> df.with_columns_renamed({"x": "foo", "y": "bar"}).show() ╭───────┬───────╮ │ foo ┆ bar │ │ --- ┆ --- │ │ Int64 ┆ Int64 │ ╞═══════╪═══════╡ │ 1 ┆ 4 │ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤ │ 2 ┆ 5 │ ├╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌┤ │ 3 ┆ 6 │ ╰───────┴───────╯ <BLANKLINE> (Showing first 3 of 3 rows) """ | 1 年前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 3 年前 | ||
| 3 年前 | ||
| 10 个月前 | ||
| 9 个月前 | ||
| 1 年前 | ||
| 2 年前 | ||
| 11 个月前 | ||
| 1 年前 | ||
| 1 年前 | ||
| 1 年前 | ||
| 10 个月前 | ||
| 11 个月前 | ||
| 1 年前 | ||
| 1 年前 | ||
| 9 个月前 | ||
| 9 个月前 | ||
| 2 年前 | ||
| 1 年前 | ||
| 1 年前 | ||
| 1 年前 | ||
| 10 个月前 | ||
| 1 年前 | ||
| 10 个月前 | ||
| 8 个月前 | ||
| 9 个月前 | ||
| 1 年前 | ||
| 9 个月前 | ||
| 10 个月前 | ||
| 8 个月前 | ||
| 1 年前 | ||
| 1 年前 | ||
| 10 个月前 | ||
| 1 年前 | ||
| 10 个月前 | ||
| 10 个月前 | ||
| 1 年前 | ||
| 9 个月前 | ||
| 1 年前 | ||
| 9 个月前 | ||
| 10 个月前 | ||
| 9 个月前 | ||
| 10 个月前 | ||
| 1 年前 | ||
| 11 个月前 | ||
| 10 个月前 | ||
| 1 年前 | ||
| 3 年前 | ||
| 1 年前 | ||
| 10 个月前 | ||
| 1 年前 | ||
| 1 年前 | ||
| 1 年前 | ||
| 2 年前 | ||
| 1 年前 | ||
| 2 年前 | ||
| 1 年前 |