| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
feat!: add iceberg table support with the gravitino catalog (#6509) ## Changes Made This refactors the Gravitino catalog implementation to match our Glue implementation. It looks at table information to determine which concrete Table implementation to use. This also refactors the Catalog creation a little bit to once again match Glue. I'm not proud of the Catalog.from_* thing with the objects, and I like the pyiceberg choice of the top-level load_ methods, hence why later implementations like Glue use that pattern, so I've updated Gravitino to follow this. This PR also properly internalizes/encapsulates the client, I wasn't interested in having our own Gravitino client vs. using the python one, but the customer required this -- so I just hid the client and updated the docs accordingly. Also a tiny fix on an old typo of mine from the video reader: read_glob_paths/from_glob_paths. ## Related Issues - Closes #6318 - Closes #6319 | 6 个月前 | |
| 7 个月前 | ||
| 7 个月前 | ||
feat: add support for pyiceberg 0.11.0 (#6200) ## Changes Made Makes required changes to support pyiceberg v0.11.0 which now properly supports Decimal type (see https://github.com/apache/iceberg-python/pull/2515). Excludes v0.9.1 and v0.10.0 that have the Decimal type issue. There is a breaking change in pyiceberg that requires a partition field to have a different name than any field in the schema. Updates to automatically name field by using internal pyiceberg naming conventions for partition fields. Specific changes required: - Use _PartitionNameGenerator in pyiceberg to generate the partition field name based on the source field and field ID when creating a table. - Fix can_cast_types to handle Extension. - Update Dockerfile for iceberg integration tests to use 0.11.0. - Update get_data_files to use pyiceberg InspectTable to get data files as opposed to using the now position-based Record. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> --------- Co-authored-by: gmweaver <gmweaver.usc@gmail.com> Co-authored-by: desmondcheongzx <desmondcheongzx@gmail.com> | 8 个月前 | |
feat!: add iceberg table support with the gravitino catalog (#6509) ## Changes Made This refactors the Gravitino catalog implementation to match our Glue implementation. It looks at table information to determine which concrete Table implementation to use. This also refactors the Catalog creation a little bit to once again match Glue. I'm not proud of the Catalog.from_* thing with the objects, and I like the pyiceberg choice of the top-level load_ methods, hence why later implementations like Glue use that pattern, so I've updated Gravitino to follow this. This PR also properly internalizes/encapsulates the client, I wasn't interested in having our own Gravitino client vs. using the python one, but the customer required this -- so I just hid the client and updated the docs accordingly. Also a tiny fix on an old typo of mine from the video reader: read_glob_paths/from_glob_paths. ## Related Issues - Closes #6318 - Closes #6319 | 6 个月前 | |
feat: support creating partitioned tables in Iceberg via the Catalog interface. (#4951) ## Changes Made Support creating partitioned tables in Iceberg via the Catalog interface. Added function to translate daft's PartitionFields to a pyiceberg PartitionSpec and handled partition field list if passed as argument in _create_table function. ## Related Issues #4876 ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) --------- Co-authored-by: redpheonixx <“amit220698@gmail.com”> | 1 年前 | |
feat: Add Apache Paimon Lake Format Support (#6450) ## Changes Made [Apache Paimon](https://paimon.apache.org/) is an open lake format designed for streaming and batch unified analytics. Adding Paimon closes the gap for users running Flink-based pipelines who want to use Daft for batch/ML workloads on top of their existing Paimon tables — without copying data out of the lake. This PR introduces daft.read_paimon(table) and df.write_paimon(table, mode) using [pypaimon](https://pypi.org/project/pypaimon/) as the catalog and metadata layer. ## Related Issues Closes #4976 | 6 个月前 | |
fix: Postgres overwrite table should enable RLS and set up pgvector automatically (#5657) ## Changes Made Previously we didn't use the drop, create, append table functionality in the Table.overwrite() method. This resulted in duplicate code that we didn't implement correctly. We can instead simply rely on existing implementations that already work. --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 10 个月前 | |
feat: support creating partitioned tables in Iceberg via the Catalog interface. (#4951) ## Changes Made Support creating partitioned tables in Iceberg via the Catalog interface. Added function to translate daft's PartitionFields to a pyiceberg PartitionSpec and handled partition field list if passed as argument in _create_table function. ## Related Issues #4876 ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) --------- Co-authored-by: redpheonixx <“amit220698@gmail.com”> | 1 年前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 6 个月前 | ||
| 7 个月前 | ||
| 7 个月前 | ||
| 8 个月前 | ||
| 6 个月前 | ||
| 1 年前 | ||
| 6 个月前 | ||
| 10 个月前 | ||
| 1 年前 |