| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
feat: Add Catalog and Table implementations for PostgreSQL (#5487) ## Changes Made Allows us to do catalog = daft.catalog.Catalog.from_postgres("postgresql://user:pass@uri:5432/postgres") table = catalog.get_table("myschema.mytable") df = table.read() ... output_table = catalog.create_table_if_not_exists("myschema.output") output_table.write(df) Depends on https://github.com/Eventual-Inc/Daft/pull/5471 for distributed writes and the df.write_sql() interface. ## Key decisions, limitations, and notes - Embeddings support is currently provided via the pgvector extension and the pgvector python client library. - The following types are currently not supported: - Duration - Interval - The following types currently fallback to JSONB: - Multidimensional lists - Structs - Maps - Images - Tensors - No connection pooling - Writes happen on a single node (https://github.com/Eventual-Inc/Daft/pull/5471 implements distributed writes) - Uses psycopg and COPY for writes. ADBC would potentially be a better choice here but we ignore it for now due to its limited type support. - Instead of assuming that public is the default schema, we delegate table resolution to Postgres' [Schema Search Path](https://www.postgresql.org/docs/current/ddl-schemas.html#DDL-SCHEMAS-PATH) --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 11 个月前 | |
[FEAT] read_sql (#1943) Closes #1560 Adds a new read method: read_sql(sql: str, url: str), which executes a given sql query on a given database url, and creates a Dataframe from the results. Drive bys: - Added a from_arrow cast from pyarrow date64 to daft timestamp (both are millisecond precision) to allow connectorx reads with timestamp columns. Features: - Uses [connector-x](https://github.com/sfu-db/connector-x/tree/main), which reads directly into pyarrow via rust when supported, else fallback to [SQL alchemy ](https://docs.sqlalchemy.org/en/20/orm/quickstart.html) - Partitioned reads via limit and offset when supported - Integration tests for Postgres, MySQL, Trino - Pushdowns into base SQL query | 2 年前 | |
chore: Upgrade Ruff ruleset to 3.9 and add from __future__ import annotations (#4393) | 1 年前 | |
fix: Postgres overwrite table should enable RLS and set up pgvector automatically (#5657) ## Changes Made Previously we didn't use the drop, create, append table functionality in the Table.overwrite() method. This resulted in duplicate code that we didn't implement correctly. We can instead simply rely on existing implementations that already work. --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 10 个月前 | |
chore: Remove the old Ray Runner (#5375) ## Changes Made 🎉🎂🥳 Can finally delete it, Flotilla supports all necessary features. Additional Related Features Removed: * DataFrame.num_partitions: This is computed using the old Ray runner's planner. We could use the new Ray runner, but it seems kind of unnecessary * Context Settings: --------- Co-authored-by: Colin Ho <colin.ho99@gmail.com> | 10 个月前 | |
[CHORE] Add tpch test for read sql (#2026) This PR adds tpch tests via read_sql. Our tpch integration test setup already creates a sqlite table, which makes it easy to run read_sql against for complex query validation. As a drive by in addition to adding tpch tests, this PR gates the Date and Timestamp literal to sql translations. This is because the semantics for date and timestamp literals were not as general as expected. | 2 年前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 11 个月前 | ||
| 2 年前 | ||
| 1 年前 | ||
| 10 个月前 | ||
| 10 个月前 | ||
| 2 年前 |