develop/ai_data_engineering

文件	最后提交记录	最后更新时间
src	chore: bump pyo3 dependency (#5410) ## Changes Made bumps pyo3 and related dependencies to 26 Pretty much just a find & replace /s/old/new - with_gil -> attach - allow_threads -> detach - PyObject -> Py<PyAny> ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly	1 年前
Cargo.toml	feat!: unify Python -> Daft type conversions (#5201) BREAKING CHANGES: - list[object] element inputs to daft.from_pydict and similar methods are now converted into List[Python] instead of Python - Before: daft.from_pydict({"x": [[object()], [object(), object()]]}) -> schema of (x: Python) - After: schema of (x: List[Python]) - Python int inputs less than 2^31-1 are now converted to Int64 instead of Int32 - Flat numpy array inputs are converted into Tensor instead of List - I believe we used to be able to infer types from dicts or tuples of types. I've removed them and instead added the ability to infer from TypedDict and tuple[T1, T2, ...]. They were not documented anyway ## Changes Made Unified and documented the conversions from Python objects and types to Daft. The core of the conversion logic lies in: - Literal::from_pyobj in src/daft-core/lit/python.rs, which is now the ultimate source of truth for our Python to Daft conversions - DataType.infer_from_type in daft/datatype.py, which maps Python types to Daft types for things like inferring a UDF return type. - Since Python types do not always have sufficient information to derive the full Daft type, this function deviates slightly from Literal::from_pyobj in those cases. Users can also use DataType.infer_from_object to check the logic in Literal::from_pyobj. Other changes: - Added type conversion tables and moved them to docs/api/datatypes/type_conversions.md - Changed the Rust NdArray type from a trait to an enum so that code in daft-core can match on the variants. - Removed a significant amount of PythonArray casting logic, changing them to go through Literal::from_pyobj instead to make everything consistent and cleaner. - Removed ability to run concat agg on Python-type columns. This was only used because we could not serialize List[Python] so we just kept everything as Python lists. But that is fixed with #5270 - Changed Literal::Struct to store mapping from name to item instead of field to item, since the dtype of a field can be inferred from the item. Some UDFs that now work: - returning tuple py >>> import daft >>> @daft.func ... def tuple_udf(x) -> tuple[str, int]: ... return (str(x), int(x)) ... >>> df = daft.from_pydict({"input": [1, 2, 3]}) >>> df.select(tuple_udf(df["input"])).show() ╭─────────────────────────────╮ │ input │ │ --- │ │ Struct[_0: Utf8, _1: Int64] │ ╞═════════════════════════════╡ │ {_0: 1, │ │ _1: 1, │ │ } │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ {_0: 2, │ │ _1: 2, │ │ } │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ {_0: 3, │ │ _1: 3, │ │ } │ ╰─────────────────────────────╯ (Showing first 3 of 3 rows) - returning list of numpy array: py >>> import daft >>> import numpy as np >>> import numpy.typing as npt >>> @daft.func ... def array_list_udf(x) -> list[npt.NDArray[np.uint]]: ... return [np.array([x], dtype=np.uint), np.array([x*2, x*3], dtype=np.uint)] ... >>> df = daft.from_pydict({"input": [1, 2, 3]}) >>> df.select(array_list_udf(df["input"])).show() ╭────────────────────────────────╮ │ input │ │ --- │ │ List[Tensor(UInt64)] │ ╞════════════════════════════════╡ │ [<Tensor shape=(1)>, <Tensor … │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ [<Tensor shape=(1)>, <Tensor … │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ [<Tensor shape=(1)>, <Tensor … │ ╰────────────────────────────────╯ (Showing first 3 of 3 rows) ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [x] Documented in API Docs (if applicable) - [x] Documented in User Guide (if applicable) - [x] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [x] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review)	1 年前