| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 10 个月前 | ||
ci: remove macos from PR test suite (#5142) ## Changes Made We currently use MacOS to run unit tests in PRs because we had an issue with getting Rust code coverage in Ubuntu 22.04. The latest Rust toolchain no longer has this issue, so this PR updates the rust toolchain to the latest nightly and moves the code coverage tests to Ubuntu. The vast majority of the diff here is just linting and lifetime fixes due to the upgrade to the latest nightly Rust + 2024 edition. The rest of the changes are as follows: - moving the code coverage tests in .github/workflows/pr-test-suite.yml to Ubuntu as described above - it was failing due to out of disk space so I also added a disk space remover action - update our Rust versions in rust-toolchain.toml and Cargo.toml - use hashbrown::HashMap instead of std::HashMap because raw_entry_mut was removed from the standard library ## Related Issues https://github.com/Eventual-Inc/Daft/issues/3801 ## Checklist - [x] Documented in API Docs (if applicable) - [x] Documented in User Guide (if applicable) - [x] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [x] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 1 年前 | |
chore: reduce binary size by feature flagging derive(Debug) (#5622) ## Changes Made cargo-llvm-lines showed that 0.7% of our binary size was from Debug impls 40559 (0.7%, 5.8%) 15604 (3.0%, 5.4%) <&T as core::fmt::Debug>::fmt I noticed that this PR reduces the total binary size by about 1.2MB. While pretty small when our uncompressed binary is 180+MB, the small savings add up. So this PR attempts to either remove derive(Debug) or feature flag it: cfg_attr(debug_assertions, derive(Debug)) in some spots we still need it to not break a bunch of things, so for not(debug_assertions), theres a simpler debug impl provided, that usually just debugs the name, not all of the struct fields. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 11 个月前 | |
feat!: unify Python -> Daft type conversions (#5201) BREAKING CHANGES: - list[object] element inputs to daft.from_pydict and similar methods are now converted into List[Python] instead of Python - Before: daft.from_pydict({"x": [[object()], [object(), object()]]}) -> schema of (x: Python) - After: schema of (x: List[Python]) - Python int inputs less than 2^31-1 are now converted to Int64 instead of Int32 - Flat numpy array inputs are converted into Tensor instead of List - I believe we used to be able to infer types from dicts or tuples of types. I've removed them and instead added the ability to infer from TypedDict and tuple[T1, T2, ...]. They were not documented anyway ## Changes Made Unified and documented the conversions from Python objects and types to Daft. The core of the conversion logic lies in: - Literal::from_pyobj in src/daft-core/lit/python.rs, which is now the ultimate source of truth for our Python to Daft conversions - DataType.infer_from_type in daft/datatype.py, which maps Python types to Daft types for things like inferring a UDF return type. - Since Python types do not always have sufficient information to derive the full Daft type, this function deviates slightly from Literal::from_pyobj in those cases. Users can also use DataType.infer_from_object to check the logic in Literal::from_pyobj. Other changes: - Added type conversion tables and moved them to docs/api/datatypes/type_conversions.md - Changed the Rust NdArray type from a trait to an enum so that code in daft-core can match on the variants. - Removed a significant amount of PythonArray casting logic, changing them to go through Literal::from_pyobj instead to make everything consistent and cleaner. - Removed ability to run concat agg on Python-type columns. This was only used because we could not serialize List[Python] so we just kept everything as Python lists. But that is fixed with #5270 - Changed Literal::Struct to store mapping from name to item instead of field to item, since the dtype of a field can be inferred from the item. Some UDFs that now work: - returning tuple py >>> import daft >>> @daft.func ... def tuple_udf(x) -> tuple[str, int]: ... return (str(x), int(x)) ... >>> df = daft.from_pydict({"input": [1, 2, 3]}) >>> df.select(tuple_udf(df["input"])).show() ╭─────────────────────────────╮ │ input │ │ --- │ │ Struct[_0: Utf8, _1: Int64] │ ╞═════════════════════════════╡ │ {_0: 1, │ │ _1: 1, │ │ } │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ {_0: 2, │ │ _1: 2, │ │ } │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ {_0: 3, │ │ _1: 3, │ │ } │ ╰─────────────────────────────╯ (Showing first 3 of 3 rows) - returning list of numpy array: py >>> import daft >>> import numpy as np >>> import numpy.typing as npt >>> @daft.func ... def array_list_udf(x) -> list[npt.NDArray[np.uint]]: ... return [np.array([x], dtype=np.uint), np.array([x*2, x*3], dtype=np.uint)] ... >>> df = daft.from_pydict({"input": [1, 2, 3]}) >>> df.select(array_list_udf(df["input"])).show() ╭────────────────────────────────╮ │ input │ │ --- │ │ List[Tensor(UInt64)] │ ╞════════════════════════════════╡ │ [<Tensor shape=(1)>, <Tensor … │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ [<Tensor shape=(1)>, <Tensor … │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ [<Tensor shape=(1)>, <Tensor … │ ╰────────────────────────────────╯ (Showing first 3 of 3 rows) ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [x] Documented in API Docs (if applicable) - [x] Documented in User Guide (if applicable) - [x] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [x] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 1 年前 | |
chore: Drop Python 3.9 (#5479) ## Changes Made Python 3.9 is EOL starting from Nov 1, so lets drop it and make 3.10 the minimum. Also, start testing ranges 3.10 to 3.13 --------- Co-authored-by: Colin Ho <colin.ho99@gmail.com> | 11 个月前 |