develop/ai_data_engineering_base

GGitHubfeat: Add Apache Gravitino virtual file system (gvfs://) write support in io module (#5965)

2855a34d创建于 1月10日历史提交

文件	最后提交记录	最后更新时间
ai	chore: bump mypy and ruff in pre-commit (#5836) ## Changes Made Bump mypy to 1.19.1 and ruff to 0.14.10 and resolve pre-commit errors. Rationale: my local ruff version is much higher than that in the project pre-commit, so I am getting a lot of linter warnings in my IDE. It is also about time to upgrade lints to utilize newer Python language features. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" -->	8 个月前
delta_lake	chore: Remove the old Ray Runner (#5375) ## Changes Made 🎉🎂🥳 Can finally delete it, Flotilla supports all necessary features. Additional Related Features Removed: * DataFrame.num_partitions: This is computed using the old Ray runner's planner. We could use the new Ray runner, but it seems kind of unnecessary * Context Settings: --------- Co-authored-by: Colin Ho <colin.ho99@gmail.com>	10 个月前
gravitino	feat: Add Apache Gravitino virtual file system (gvfs://) write support in io module (#5965)	8 个月前
iceberg	feat: Add Apache Gravitino virtual file system (gvfs://) read support in io module (#5766) ## Changes Made Add Gravitino virtual file system (gvfs://) support read support. So that user can use "gvfs://filesets/catalog/schema/fileset_name" path to access s3 (and other cloud storages) location. This PR only implements the s3 as the first step. To ensure the functionality, the integration test cases are added with a MinIO service to simulate s3. ## Related Issues This is the second pr for https://github.com/Eventual-Inc/Daft/issues/5503	8 个月前
io	fix: remove flaky datasets from read_huggingface tests (#5926) This pull request makes a minor adjustment to the HuggingFace dataset integration tests by removing a dataset from the test parameterization. This helps keep the test suite focused and potentially avoids issues with unavailable or problematic datasets. * Removed the ("nebius/SWE-rebench", "instance_id") entry from the test parameterization in test_read_huggingface to streamline test coverage.## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" -->	8 个月前
ray	fix: Supporting fractional gpu count on class udf (#5840) ## Changes Made <!-- Describe what changes were made and why. Include implementation details if necessary. --> Supporting fractional gpu count in class udf. def cls( class_: type | None = None, *, gpus: float = 0.0, use_process: bool | None = None, max_concurrency: int | None = None, max_retries: int | None = None, on_error: Literal["raise", "log", "ignore"] | None = None, ) -> type | Callable[[type], type]: before this, the datatype of gpus in @daft.cls was int, but the underlying rust implementation was float. The declarations on the python side and the rust side are inconsistent. And for small models, we should support fine-grained resource Settings, such as 0.5 gpu. Declarations of the float type make it clearer and accurate. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> https://github.com/Eventual-Inc/Daft/issues/5837 --------- Co-authored-by: cancai <caican@xiaomi.com> Co-authored-by: Colin Ho <chiuhong@usc.edu>	8 个月前
sql	fix: Postgres overwrite table should enable RLS and set up pgvector automatically (#5657) ## Changes Made Previously we didn't use the drop, create, append table functionality in the Table.overwrite() method. This resulted in duplicate code that we didn't implement correctly. We can instead simply rely on existing implementations that already work. --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>	9 个月前
unity	chore: Remove expression namespaces (#5619) ## Changes Made Bye bye ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly	10 个月前
__init__.py	Initializes working rust-main branch * Old tests are migrated to tests-legacy/ - only new, working tests are left in tests/ * Tests that are failing in tests/ are skipped and tagged with [RUST-INT] * Small fixes made for the code to type-check - removal of udf.py and stubbing out of DataFrame.explode() code	3 年前
conftest.py	[CHORE] Add TPC-H questions 11-22 to benchmarks (currently skipped) (#2299)	2 年前
test_file_tokenize.py	chore: Remove expression namespaces (#5619) ## Changes Made Bye bye ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly	10 个月前
test_file_udfs_s3.py	chore: Remove expression namespaces (#5619) ## Changes Made Bye bye ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly	10 个月前
test_tpch.py	[CHORE] Add TPC-H questions 11-22 to benchmarks (currently skipped) (#2299)	2 年前