| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
feat: Support pattern filtering for SHOW TABLES (#5423) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> ### Extended SQL LIKE Pattern Matching in MemoryCatalog - Upgrade sqlparser to extract namespace from query - Translate SQL LIKE patterns to regex (src/daft-catalog/src/pattern.rs) - Supports standard SQL LIKE wildcards: % (zero or more), _ (exactly one), \ (escape) - Updated documentation on SHOW TABLES syntax and pattern behavior ### Testing - Added pattern matching tests to test_sql_show_tables.py and in src/daft-catalog/pattern.rs - cargo test -p daft-catalog pattern:: ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> Closes #4461 Closes #4007 ## Checklist - [x] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 9 个月前 | |
feat!: unify Python -> Daft type conversions (#5201) BREAKING CHANGES: - list[object] element inputs to daft.from_pydict and similar methods are now converted into List[Python] instead of Python - Before: daft.from_pydict({"x": [[object()], [object(), object()]]}) -> schema of (x: Python) - After: schema of (x: List[Python]) - Python int inputs less than 2^31-1 are now converted to Int64 instead of Int32 - Flat numpy array inputs are converted into Tensor instead of List - I believe we used to be able to infer types from dicts or tuples of types. I've removed them and instead added the ability to infer from TypedDict and tuple[T1, T2, ...]. They were not documented anyway ## Changes Made Unified and documented the conversions from Python objects and types to Daft. The core of the conversion logic lies in: - Literal::from_pyobj in src/daft-core/lit/python.rs, which is now the ultimate source of truth for our Python to Daft conversions - DataType.infer_from_type in daft/datatype.py, which maps Python types to Daft types for things like inferring a UDF return type. - Since Python types do not always have sufficient information to derive the full Daft type, this function deviates slightly from Literal::from_pyobj in those cases. Users can also use DataType.infer_from_object to check the logic in Literal::from_pyobj. Other changes: - Added type conversion tables and moved them to docs/api/datatypes/type_conversions.md - Changed the Rust NdArray type from a trait to an enum so that code in daft-core can match on the variants. - Removed a significant amount of PythonArray casting logic, changing them to go through Literal::from_pyobj instead to make everything consistent and cleaner. - Removed ability to run concat agg on Python-type columns. This was only used because we could not serialize List[Python] so we just kept everything as Python lists. But that is fixed with #5270 - Changed Literal::Struct to store mapping from name to item instead of field to item, since the dtype of a field can be inferred from the item. Some UDFs that now work: - returning tuple py >>> import daft >>> @daft.func ... def tuple_udf(x) -> tuple[str, int]: ... return (str(x), int(x)) ... >>> df = daft.from_pydict({"input": [1, 2, 3]}) >>> df.select(tuple_udf(df["input"])).show() ╭─────────────────────────────╮ │ input │ │ --- │ │ Struct[_0: Utf8, _1: Int64] │ ╞═════════════════════════════╡ │ {_0: 1, │ │ _1: 1, │ │ } │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ {_0: 2, │ │ _1: 2, │ │ } │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ {_0: 3, │ │ _1: 3, │ │ } │ ╰─────────────────────────────╯ (Showing first 3 of 3 rows) - returning list of numpy array: py >>> import daft >>> import numpy as np >>> import numpy.typing as npt >>> @daft.func ... def array_list_udf(x) -> list[npt.NDArray[np.uint]]: ... return [np.array([x], dtype=np.uint), np.array([x*2, x*3], dtype=np.uint)] ... >>> df = daft.from_pydict({"input": [1, 2, 3]}) >>> df.select(array_list_udf(df["input"])).show() ╭────────────────────────────────╮ │ input │ │ --- │ │ List[Tensor(UInt64)] │ ╞════════════════════════════════╡ │ [<Tensor shape=(1)>, <Tensor … │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ [<Tensor shape=(1)>, <Tensor … │ ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤ │ [<Tensor shape=(1)>, <Tensor … │ ╰────────────────────────────────╯ (Showing first 3 of 3 rows) ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [x] Documented in API Docs (if applicable) - [x] Documented in User Guide (if applicable) - [x] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [x] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 1 年前 | |
docs: migrate sphinx api docs to mkdocs (#3980) ## Part 1 - [x] Dataframe Creation - [x] Dataframe - [x] Expressions (namespaces: url, float, datetime, string, list, struct, map, projection, image, partitioning, json, embedding, binary) - [x] Functions - [ ] SQL (all sql funcs) - [x] Sessions - [x] Catalogs - [x] Tables - [x] Schema - [x] Data types - [x] Groupby - [x] UDFs - [x] Series - [x] Configuration (s3config, s3credentials, gcsconfig, azureconfig) - [x] Miscellaneous ## Part 2 - [x] rename mkdocs folder to user-guide to differentiate between user guide and api docs ^ update: all user guide docs will live in docs and all api docs will live in docs/api_docs, this is so that build site opens on user guide - [x] update relative links between user guide and api docs - [x] Terminology - [x] Quickstart - [x] Core Concepts - [x] SQL Reference - [x] Sessions - [x] Catalogs - [x] Spark Connect - [x] Distributed Computing - [x] Advanced - [x] Integrations - [x] Ray - [x] Unity - [x] Apache Iceberg - [x] Delta Lake - [x] Apache Hudi - [x] AWS (blocked by s3config) - [x] Azure (blocked by azureconfig) - [x] SQL - [x] Hugging Face Datasets - [x] S3 Tables - [x] Resources - [x] Migration Guide ## Part 3 - [ ] reformat docstrings and examples for api docs https://github.com/Eventual-Inc/Daft/pull/4148 - [ ] fix warning/note boxes from original docstring - [ ] fix random headings that shouldn't be - [ ] how can we better group the integrations section? ## Part 4 / Clean Up (after complete migration is done) - [x] remove from mkdocs.yml - mkdocs-simple-hooks: hooks: on_pre_build: "docs.hooks:make_api_docs" - [x] remove docs/hooks.py file - [ ] delete docs/Makefile (these are sphinx build instructions) - [ ] remove sphinx folders in docs/.gitignore and only keep site - [ ] remove sphinx mentions in .github/workflows/build-docs.yml - [x] update Makefile with make docs to build docs and make serve to build in development serve - [x] update docs/CONTRIBUTING.md with build command make docs and serve command make serve in Daft dir not in docs dir - [ ] search todo(docs - cc) for missing items - [ ] remove docs/source/... in .pre-commit-config.yml | 1 年前 | |
chore: Remove expression namespaces (#5619) ## Changes Made Bye bye ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 10 个月前 | |
| 1 年前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 9 个月前 | ||
| 1 年前 | ||
| 1 年前 | ||
| 10 个月前 | ||
| 1 年前 |