GGitHubchore: removes unnecessary pushdown builder code (#6403 )

6bec12d3创建于 3月17日历史提交

文件	最后提交记录	最后更新时间
hello	feat: adds support for daft native extensions via stable C ABI (#6301) ## About This PR creates a native extension framework for Daft and is modeled after Postgres. Extension authors ship a normal pip-installable python package with a bundled dylib (instructions in tutorial). Users import python functions to get full IDE support and the native implementations are linked by loading the extension into a session. ## Design Crates - `daft-ext` - Single dependency for extension authors (published). - `daft-ext-abi` - Stable #[repr(C)] types defining the ABI contract. - `daft-ext-core` - Public Rust SDK types (DaftScalarFunction, traits, error types). - `daft-ext-macros` - Proc macro (#[daft_extension]). - `daft-ext-internal` - Host-side adapters (ScalarFunctionHandle, module loader). Not published. Functions `ScalarUDF` is Daft's internal trait for scalar functions which cannot cross a `dlopen` boundary because it uses Rust trait objects with unstable ABI layouts - so `FFI_ScalarFunction` in `daft-ext-abi` is the stable C ABI version of this. `ScalarFunctionHandle` in `daft-ext-internal` wraps a `FFI_ScalarFunction` and implements both `ScalarUDF` and `ScalarFunctionFactory`. Data crosses the extension boundary using the Arrow C Data Interface (`FFI_ArrowArray` + `FFI_ArrowSchema`) and is zero-copy. The extension FFI uses `arrow::ffi` directly, not the `common-arrow-ffi` crate which is coupled to PyO3. Installation We load shared libs once into the process when someone calls `.load_extension` on the session. The top-level `daft.load_extension` will load the given extension module into the active session (from context). All defined functions are scoped to the session in which the extension is loaded. ## Changes - Creates the daft-ext-* crates defined above. - Defines the stable C ABI which minimally wraps arrow ffi. - Creates the daft-ext-internal to bridge host->abi. - Creates the daft-ext-core to bridge extension->abi. - Implements internal Daft traits using the daft-ext-internal types. - Adds a `dvector` example extension which is a pgvector clone. - Adds a `hello` example extension with a tutorial document. - Extends the session to support native function registration. - Adds the `get_function` method for session-backed function resolution mirroring the SQL implementation. Can add more clarity upon reviews. ## Examples - [Hello, World](https://github.com/Eventual-Inc/Daft/tree/564548a93fa99180e4e17f51049e5db60a5949cf/examples/hello) - [dvector (pgvector)](https://github.com/Eventual-Inc/Daft/tree/564548a93fa99180e4e17f51049e5db60a5949cf/examples/dvector) `python import daft # Step 1. Import your extension module import hello # Step 2. Load the extension into the current daft session daft.load_extension(hello) # Step 3. Use in your dataframe! df = daft.from_pydict({"name": ["John", "Paul"]}) df = df.select(hello.greet(df["name"]))` See the actual [greet rust implementation](https://github.com/Eventual-Inc/Daft/blob/564548a93fa99180e4e17f51049e5db60a5949cf/examples/hello/src/lib.rs). ## Guide See [Daft Extension Guide](https://github.com/Eventual-Inc/Daft/blob/564548a93fa99180e4e17f51049e5db60a5949cf/docs/extensions/index.md).	7 个月前
src	chore: removes unnecessary pushdown builder code (#6403)	7 个月前
tests	feat: adds support for daft native extensions via stable C ABI (#6301) ## About This PR creates a native extension framework for Daft and is modeled after Postgres. Extension authors ship a normal pip-installable python package with a bundled dylib (instructions in tutorial). Users import python functions to get full IDE support and the native implementations are linked by loading the extension into a session. ## Design Crates - `daft-ext` - Single dependency for extension authors (published). - `daft-ext-abi` - Stable #[repr(C)] types defining the ABI contract. - `daft-ext-core` - Public Rust SDK types (DaftScalarFunction, traits, error types). - `daft-ext-macros` - Proc macro (#[daft_extension]). - `daft-ext-internal` - Host-side adapters (ScalarFunctionHandle, module loader). Not published. Functions `ScalarUDF` is Daft's internal trait for scalar functions which cannot cross a `dlopen` boundary because it uses Rust trait objects with unstable ABI layouts - so `FFI_ScalarFunction` in `daft-ext-abi` is the stable C ABI version of this. `ScalarFunctionHandle` in `daft-ext-internal` wraps a `FFI_ScalarFunction` and implements both `ScalarUDF` and `ScalarFunctionFactory`. Data crosses the extension boundary using the Arrow C Data Interface (`FFI_ArrowArray` + `FFI_ArrowSchema`) and is zero-copy. The extension FFI uses `arrow::ffi` directly, not the `common-arrow-ffi` crate which is coupled to PyO3. Installation We load shared libs once into the process when someone calls `.load_extension` on the session. The top-level `daft.load_extension` will load the given extension module into the active session (from context). All defined functions are scoped to the session in which the extension is loaded. ## Changes - Creates the daft-ext-* crates defined above. - Defines the stable C ABI which minimally wraps arrow ffi. - Creates the daft-ext-internal to bridge host->abi. - Creates the daft-ext-core to bridge extension->abi. - Implements internal Daft traits using the daft-ext-internal types. - Adds a `dvector` example extension which is a pgvector clone. - Adds a `hello` example extension with a tutorial document. - Extends the session to support native function registration. - Adds the `get_function` method for session-backed function resolution mirroring the SQL implementation. Can add more clarity upon reviews. ## Examples - [Hello, World](https://github.com/Eventual-Inc/Daft/tree/564548a93fa99180e4e17f51049e5db60a5949cf/examples/hello) - [dvector (pgvector)](https://github.com/Eventual-Inc/Daft/tree/564548a93fa99180e4e17f51049e5db60a5949cf/examples/dvector) `python import daft # Step 1. Import your extension module import hello # Step 2. Load the extension into the current daft session daft.load_extension(hello) # Step 3. Use in your dataframe! df = daft.from_pydict({"name": ["John", "Paul"]}) df = df.select(hello.greet(df["name"]))` See the actual [greet rust implementation](https://github.com/Eventual-Inc/Daft/blob/564548a93fa99180e4e17f51049e5db60a5949cf/examples/hello/src/lib.rs). ## Guide See [Daft Extension Guide](https://github.com/Eventual-Inc/Daft/blob/564548a93fa99180e4e17f51049e5db60a5949cf/docs/extensions/index.md).	7 个月前
.gitignore	feat: adds support for daft native extensions via stable C ABI (#6301) ## About This PR creates a native extension framework for Daft and is modeled after Postgres. Extension authors ship a normal pip-installable python package with a bundled dylib (instructions in tutorial). Users import python functions to get full IDE support and the native implementations are linked by loading the extension into a session. ## Design Crates - `daft-ext` - Single dependency for extension authors (published). - `daft-ext-abi` - Stable #[repr(C)] types defining the ABI contract. - `daft-ext-core` - Public Rust SDK types (DaftScalarFunction, traits, error types). - `daft-ext-macros` - Proc macro (#[daft_extension]). - `daft-ext-internal` - Host-side adapters (ScalarFunctionHandle, module loader). Not published. Functions `ScalarUDF` is Daft's internal trait for scalar functions which cannot cross a `dlopen` boundary because it uses Rust trait objects with unstable ABI layouts - so `FFI_ScalarFunction` in `daft-ext-abi` is the stable C ABI version of this. `ScalarFunctionHandle` in `daft-ext-internal` wraps a `FFI_ScalarFunction` and implements both `ScalarUDF` and `ScalarFunctionFactory`. Data crosses the extension boundary using the Arrow C Data Interface (`FFI_ArrowArray` + `FFI_ArrowSchema`) and is zero-copy. The extension FFI uses `arrow::ffi` directly, not the `common-arrow-ffi` crate which is coupled to PyO3. Installation We load shared libs once into the process when someone calls `.load_extension` on the session. The top-level `daft.load_extension` will load the given extension module into the active session (from context). All defined functions are scoped to the session in which the extension is loaded. ## Changes - Creates the daft-ext-* crates defined above. - Defines the stable C ABI which minimally wraps arrow ffi. - Creates the daft-ext-internal to bridge host->abi. - Creates the daft-ext-core to bridge extension->abi. - Implements internal Daft traits using the daft-ext-internal types. - Adds a `dvector` example extension which is a pgvector clone. - Adds a `hello` example extension with a tutorial document. - Extends the session to support native function registration. - Adds the `get_function` method for session-backed function resolution mirroring the SQL implementation. Can add more clarity upon reviews. ## Examples - [Hello, World](https://github.com/Eventual-Inc/Daft/tree/564548a93fa99180e4e17f51049e5db60a5949cf/examples/hello) - [dvector (pgvector)](https://github.com/Eventual-Inc/Daft/tree/564548a93fa99180e4e17f51049e5db60a5949cf/examples/dvector) `python import daft # Step 1. Import your extension module import hello # Step 2. Load the extension into the current daft session daft.load_extension(hello) # Step 3. Use in your dataframe! df = daft.from_pydict({"name": ["John", "Paul"]}) df = df.select(hello.greet(df["name"]))` See the actual [greet rust implementation](https://github.com/Eventual-Inc/Daft/blob/564548a93fa99180e4e17f51049e5db60a5949cf/examples/hello/src/lib.rs). ## Guide See [Daft Extension Guide](https://github.com/Eventual-Inc/Daft/blob/564548a93fa99180e4e17f51049e5db60a5949cf/docs/extensions/index.md).	7 个月前
Cargo.toml	refactor: decouple extension ABI from arrow-rs with Arrow C Data Interfaces (#6337) Remove arrow-rs dependency from daft-ext-abi by introducing owned Arrow C Data Interface types (ArrowSchema, ArrowArray, ArrowArrayStream) that are layout compatible with the C ABI. This allows extensions to use any Arrow implementation (arrow-rs, arrow2, etc.) without coupling to a specific version. ## Changes Made - Add ArrowSchema, ArrowArray, ArrowArrayStream C types to daft-ext-abi - Add ArrowData (schema + array pair) for safe FFI data transfer - Move FFI conversion helpers into daft-ext-abi/src/ffi/arrow.rs - Update DaftScalarFunction trait to use ABI types instead of arrow-rs - Update DaftSession trait to use ABI types - Remove arrow-rs dependency from daft-ext-abi Cargo.toml - Add arrow.rs helpers in daft-ext-core for arrow-rs <-> ABI conversions - Update hello example to use new ABI types ## Related Issues - Follow-up to #6301	7 个月前
README.md	feat: adds support for daft native extensions via stable C ABI (#6301) ## About This PR creates a native extension framework for Daft and is modeled after Postgres. Extension authors ship a normal pip-installable python package with a bundled dylib (instructions in tutorial). Users import python functions to get full IDE support and the native implementations are linked by loading the extension into a session. ## Design Crates - `daft-ext` - Single dependency for extension authors (published). - `daft-ext-abi` - Stable #[repr(C)] types defining the ABI contract. - `daft-ext-core` - Public Rust SDK types (DaftScalarFunction, traits, error types). - `daft-ext-macros` - Proc macro (#[daft_extension]). - `daft-ext-internal` - Host-side adapters (ScalarFunctionHandle, module loader). Not published. Functions `ScalarUDF` is Daft's internal trait for scalar functions which cannot cross a `dlopen` boundary because it uses Rust trait objects with unstable ABI layouts - so `FFI_ScalarFunction` in `daft-ext-abi` is the stable C ABI version of this. `ScalarFunctionHandle` in `daft-ext-internal` wraps a `FFI_ScalarFunction` and implements both `ScalarUDF` and `ScalarFunctionFactory`. Data crosses the extension boundary using the Arrow C Data Interface (`FFI_ArrowArray` + `FFI_ArrowSchema`) and is zero-copy. The extension FFI uses `arrow::ffi` directly, not the `common-arrow-ffi` crate which is coupled to PyO3. Installation We load shared libs once into the process when someone calls `.load_extension` on the session. The top-level `daft.load_extension` will load the given extension module into the active session (from context). All defined functions are scoped to the session in which the extension is loaded. ## Changes - Creates the daft-ext-* crates defined above. - Defines the stable C ABI which minimally wraps arrow ffi. - Creates the daft-ext-internal to bridge host->abi. - Creates the daft-ext-core to bridge extension->abi. - Implements internal Daft traits using the daft-ext-internal types. - Adds a `dvector` example extension which is a pgvector clone. - Adds a `hello` example extension with a tutorial document. - Extends the session to support native function registration. - Adds the `get_function` method for session-backed function resolution mirroring the SQL implementation. Can add more clarity upon reviews. ## Examples - [Hello, World](https://github.com/Eventual-Inc/Daft/tree/564548a93fa99180e4e17f51049e5db60a5949cf/examples/hello) - [dvector (pgvector)](https://github.com/Eventual-Inc/Daft/tree/564548a93fa99180e4e17f51049e5db60a5949cf/examples/dvector) `python import daft # Step 1. Import your extension module import hello # Step 2. Load the extension into the current daft session daft.load_extension(hello) # Step 3. Use in your dataframe! df = daft.from_pydict({"name": ["John", "Paul"]}) df = df.select(hello.greet(df["name"]))` See the actual [greet rust implementation](https://github.com/Eventual-Inc/Daft/blob/564548a93fa99180e4e17f51049e5db60a5949cf/examples/hello/src/lib.rs). ## Guide See [Daft Extension Guide](https://github.com/Eventual-Inc/Daft/blob/564548a93fa99180e4e17f51049e5db60a5949cf/docs/extensions/index.md).	7 个月前
pyproject.toml	feat: adds support for daft native extensions via stable C ABI (#6301) ## About This PR creates a native extension framework for Daft and is modeled after Postgres. Extension authors ship a normal pip-installable python package with a bundled dylib (instructions in tutorial). Users import python functions to get full IDE support and the native implementations are linked by loading the extension into a session. ## Design Crates - `daft-ext` - Single dependency for extension authors (published). - `daft-ext-abi` - Stable #[repr(C)] types defining the ABI contract. - `daft-ext-core` - Public Rust SDK types (DaftScalarFunction, traits, error types). - `daft-ext-macros` - Proc macro (#[daft_extension]). - `daft-ext-internal` - Host-side adapters (ScalarFunctionHandle, module loader). Not published. Functions `ScalarUDF` is Daft's internal trait for scalar functions which cannot cross a `dlopen` boundary because it uses Rust trait objects with unstable ABI layouts - so `FFI_ScalarFunction` in `daft-ext-abi` is the stable C ABI version of this. `ScalarFunctionHandle` in `daft-ext-internal` wraps a `FFI_ScalarFunction` and implements both `ScalarUDF` and `ScalarFunctionFactory`. Data crosses the extension boundary using the Arrow C Data Interface (`FFI_ArrowArray` + `FFI_ArrowSchema`) and is zero-copy. The extension FFI uses `arrow::ffi` directly, not the `common-arrow-ffi` crate which is coupled to PyO3. Installation We load shared libs once into the process when someone calls `.load_extension` on the session. The top-level `daft.load_extension` will load the given extension module into the active session (from context). All defined functions are scoped to the session in which the extension is loaded. ## Changes - Creates the daft-ext-* crates defined above. - Defines the stable C ABI which minimally wraps arrow ffi. - Creates the daft-ext-internal to bridge host->abi. - Creates the daft-ext-core to bridge extension->abi. - Implements internal Daft traits using the daft-ext-internal types. - Adds a `dvector` example extension which is a pgvector clone. - Adds a `hello` example extension with a tutorial document. - Extends the session to support native function registration. - Adds the `get_function` method for session-backed function resolution mirroring the SQL implementation. Can add more clarity upon reviews. ## Examples - [Hello, World](https://github.com/Eventual-Inc/Daft/tree/564548a93fa99180e4e17f51049e5db60a5949cf/examples/hello) - [dvector (pgvector)](https://github.com/Eventual-Inc/Daft/tree/564548a93fa99180e4e17f51049e5db60a5949cf/examples/dvector) `python import daft # Step 1. Import your extension module import hello # Step 2. Load the extension into the current daft session daft.load_extension(hello) # Step 3. Use in your dataframe! df = daft.from_pydict({"name": ["John", "Paul"]}) df = df.select(hello.greet(df["name"]))` See the actual [greet rust implementation](https://github.com/Eventual-Inc/Daft/blob/564548a93fa99180e4e17f51049e5db60a5949cf/examples/hello/src/lib.rs). ## Guide See [Daft Extension Guide](https://github.com/Eventual-Inc/Daft/blob/564548a93fa99180e4e17f51049e5db60a5949cf/docs/extensions/index.md).	7 个月前
setup.py	feat: adds support for daft native extensions via stable C ABI (#6301) ## About This PR creates a native extension framework for Daft and is modeled after Postgres. Extension authors ship a normal pip-installable python package with a bundled dylib (instructions in tutorial). Users import python functions to get full IDE support and the native implementations are linked by loading the extension into a session. ## Design Crates - `daft-ext` - Single dependency for extension authors (published). - `daft-ext-abi` - Stable #[repr(C)] types defining the ABI contract. - `daft-ext-core` - Public Rust SDK types (DaftScalarFunction, traits, error types). - `daft-ext-macros` - Proc macro (#[daft_extension]). - `daft-ext-internal` - Host-side adapters (ScalarFunctionHandle, module loader). Not published. Functions `ScalarUDF` is Daft's internal trait for scalar functions which cannot cross a `dlopen` boundary because it uses Rust trait objects with unstable ABI layouts - so `FFI_ScalarFunction` in `daft-ext-abi` is the stable C ABI version of this. `ScalarFunctionHandle` in `daft-ext-internal` wraps a `FFI_ScalarFunction` and implements both `ScalarUDF` and `ScalarFunctionFactory`. Data crosses the extension boundary using the Arrow C Data Interface (`FFI_ArrowArray` + `FFI_ArrowSchema`) and is zero-copy. The extension FFI uses `arrow::ffi` directly, not the `common-arrow-ffi` crate which is coupled to PyO3. Installation We load shared libs once into the process when someone calls `.load_extension` on the session. The top-level `daft.load_extension` will load the given extension module into the active session (from context). All defined functions are scoped to the session in which the extension is loaded. ## Changes - Creates the daft-ext-* crates defined above. - Defines the stable C ABI which minimally wraps arrow ffi. - Creates the daft-ext-internal to bridge host->abi. - Creates the daft-ext-core to bridge extension->abi. - Implements internal Daft traits using the daft-ext-internal types. - Adds a `dvector` example extension which is a pgvector clone. - Adds a `hello` example extension with a tutorial document. - Extends the session to support native function registration. - Adds the `get_function` method for session-backed function resolution mirroring the SQL implementation. Can add more clarity upon reviews. ## Examples - [Hello, World](https://github.com/Eventual-Inc/Daft/tree/564548a93fa99180e4e17f51049e5db60a5949cf/examples/hello) - [dvector (pgvector)](https://github.com/Eventual-Inc/Daft/tree/564548a93fa99180e4e17f51049e5db60a5949cf/examples/dvector) `python import daft # Step 1. Import your extension module import hello # Step 2. Load the extension into the current daft session daft.load_extension(hello) # Step 3. Use in your dataframe! df = daft.from_pydict({"name": ["John", "Paul"]}) df = df.select(hello.greet(df["name"]))` See the actual [greet rust implementation](https://github.com/Eventual-Inc/Daft/blob/564548a93fa99180e4e17f51049e5db60a5949cf/examples/hello/src/lib.rs). ## Guide See [Daft Extension Guide](https://github.com/Eventual-Inc/Daft/blob/564548a93fa99180e4e17f51049e5db60a5949cf/docs/extensions/index.md).	7 个月前

Hello Native Extension

A minimal Daft native extension example. See the extension guide for details.

Quick start

# Install the extension in the project .venv
uv pip install -e .

# Run the tests!
pytest -v