| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
Image Expressions fix (#1001) * Implements the Image Expressions instead of the todo marcros | 3 年前 | |
[RUST] Enable cookbook tests (#763) * Migrates our cookbook tests from legacy_tests/ to tests/ * Simplifies repartitioning logic for test readability * Added some fixes to typing for .count() --------- Co-authored-by: Jay Chia <jaychia94@gmail.com@users.noreply.github.com> | 3 年前 | |
chore: Upgrade Ruff ruleset to 3.9 and add from __future__ import annotations (#4393) | 1 年前 | |
chore: Remove expression namespaces (#5619) ## Changes Made Bye bye ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 10 个月前 | |
docs: Enable Linting of docstrings (#3506) | 1 年前 | |
fix: Don't recompute df for count rows if results exist (#4778) ## Changes Made Don't recompute count_rows if results are already computed ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 1 年前 | |
ci: Remove Tests for the Old Ray Runner (#5374) Co-authored-by: Colin Ho <colin.ho99@gmail.com> | 11 个月前 | |
docs: Enable Linting of docstrings (#3506) | 1 年前 | |
docs: Enable Linting of docstrings (#3506) | 1 年前 | |
fix: Set default ImageMode in decode_image to RGB (#5827) ## Changes Made Set the default ImageMode for decode_image to RGB. Currently, the behavior is to infer the image mode at a per-image level, which means we could have 8-bit and 16-bit images in the same column, which will error because the underlying datatypes (uint8 vs uint16) are different. The solution to this is to force decode into a specific image mode. This PR simply makes that the default. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 9 个月前 | |
feat(flotilla): Flotilla sort merge join (#5369) ## Changes Made Implement smj for flotilla. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly --------- Co-authored-by: Srinivas Lade <srinulade1@gmail.com> | 11 个月前 | |
feat!: unify Python -> Daft type conversions (#5201) BREAKING CHANGES: - list[object] element inputs to daft.from_pydict and similar methods are now converted into List[Python] instead of Python - Before: daft.from_pydict({"x": [[object()], [object(), object()]]}) -> schema of (x: Python) - After: schema of (x: List[Python]) - Python int inputs less than 2^31-1 are now converted to Int64 instead of Int32 - Flat numpy array inputs are converted into Tensor instead of List - I believe we used to be able to infer types from dicts or tuples of types. I've removed them and instead added the ability to infer from TypedDict and tuple[T1, T2, ...]. They were not documented anyway ## Changes Made Unified and documented the conversions from Python objects and types to Daft. The core of the conversion logic lies in: - Literal::from_pyobj in src/daft-core/lit/python.rs, which is now the ultimate source of truth for our Python to Daft conversions - DataType.infer_from_type in daft/datatype.py, which maps Python types to Daft types for things like inferring a UDF return type. - Since Python types do not always have sufficient information to derive the full Daft type, this function deviates slightly from Literal::from_pyobj in those cases. Users can also use DataType.infer_from_object to check the logic in Literal::from_pyobj. Other changes: - Added type conversion tables and moved them to docs/api/datatypes/type_conversions.md - Changed the Rust NdArray type from a trait to an enum so that code in daft-core can match on the variants. - Removed a significant amount of PythonArray casting logic, changing them to go through Literal::from_pyobj instead to make everything consistent and cleaner. - Removed ability to run concat agg on Python-type columns. This was only used because we could not serialize List[Python] so we just kept everything as Python lists. But that is fixed with #5270 - Changed Literal::Struct to store mapping from name to item instead of field to item, since the dtype of a field can be inferred from the item. Some UDFs that now work: - returning tuple py >>> import daft >>> @daft.func ... def tuple_udf(x) -> tuple[str, int]: ... return (str(x), int(x)) ... >>> df = daft.from_pydict({"input": [1, 2, 3]}) >>> df.select(tuple_udf(df["input"])).show() ╭─────────────────────────────╮ │ input │ │ --- │ │ Struct[_0: Utf8, _1: Int64] │ ╞═════════════════════════════╡ │ {_0: 1, │ │ _1: 1, │ │ } │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ {_0: 2, │ │ _1: 2, │ │ } │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ {_0: 3, │ │ _1: 3, │ │ } │ ╰─────────────────────────────╯ (Showing first 3 of 3 rows) - returning list of numpy array: py >>> import daft >>> import numpy as np >>> import numpy.typing as npt >>> @daft.func ... def array_list_udf(x) -> list[npt.NDArray[np.uint]]: ... return [np.array([x], dtype=np.uint), np.array([x*2, x*3], dtype=np.uint)] ... >>> df = daft.from_pydict({"input": [1, 2, 3]}) >>> df.select(array_list_udf(df["input"])).show() ╭────────────────────────────────╮ │ input │ │ --- │ │ List[Tensor(UInt64)] │ ╞════════════════════════════════╡ │ [<Tensor shape=(1)>, <Tensor … │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ [<Tensor shape=(1)>, <Tensor … │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ [<Tensor shape=(1)>, <Tensor … │ ╰────────────────────────────────╯ (Showing first 3 of 3 rows) ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [x] Documented in API Docs (if applicable) - [x] Documented in User Guide (if applicable) - [x] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [x] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 11 个月前 | |
chore: Remove expression namespaces (#5619) ## Changes Made Bye bye ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 10 个月前 | |
chore: Remove expression namespaces (#5619) ## Changes Made Bye bye ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 10 个月前 | |
refactor: write empty dataframe to parquet/json files via native IO (#5682) ## Changes Made Currently, if the final dataframe is empty, will use the pyarrow fs to write the empty parquet/json files, but if the dataframe is not empty, will use the daft native io to write the data, the write behavior is not consistent, especially if we want to implement new storage backend, we need to cover both scenarios. Secondly, the write empty dataframe to json file is not working currently, because only parquet and csv file format supported during write_empty_tabular, the json file format is not implemented via pyarrow fs. BTW, there still remains a issue that we can write a empty json, but cannot read the empty json via daft.read_json() since cannot infer the schema info. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 9 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 3 年前 | ||
| 3 年前 | ||
| 1 年前 | ||
| 10 个月前 | ||
| 1 年前 | ||
| 1 年前 | ||
| 11 个月前 | ||
| 1 年前 | ||
| 1 年前 | ||
| 9 个月前 | ||
| 11 个月前 | ||
| 11 个月前 | ||
| 10 个月前 | ||
| 10 个月前 | ||
| 9 个月前 |