main

文件	最后提交记录	最后更新时间
dvector	feat: adds support for daft native extensions via stable C ABI (#6301) ## About This PR creates a native extension framework for Daft and is modeled after Postgres. Extension authors ship a normal pip-installable python package with a bundled dylib (instructions in tutorial). Users import python functions to get full IDE support and the native implementations are linked by loading the extension into a session. ## Design **Crates** - daft-ext - Single dependency for extension authors (published). - daft-ext-abi - Stable #[repr(C)] types defining the ABI contract. - daft-ext-core - Public Rust SDK types (DaftScalarFunction, traits, error types). - daft-ext-macros - Proc macro (#[daft_extension]). - daft-ext-internal - Host-side adapters (ScalarFunctionHandle, module loader). Not published. **Functions** ScalarUDF is Daft's internal trait for scalar functions which cannot cross a dlopen boundary because it uses Rust trait objects with unstable ABI layouts - so FFI_ScalarFunction in daft-ext-abi is the stable C ABI version of this. ScalarFunctionHandle in daft-ext-internal wraps a FFI_ScalarFunction and implements both ScalarUDF and ScalarFunctionFactory. Data crosses the extension boundary using the Arrow C Data Interface (FFI_ArrowArray + FFI_ArrowSchema) and is zero-copy. The extension FFI uses arrow::ffi directly, not the common-arrow-ffi crate which is coupled to PyO3. **Installation** We load shared libs once into the process when someone calls .load_extension on the session. The top-level daft.load_extension will load the given extension module into the active session (from context). All defined functions are scoped to the session in which the extension is loaded. ## Changes - Creates the daft-ext-* crates defined above. - Defines the stable C ABI which minimally wraps arrow ffi. - Creates the daft-ext-internal to bridge host->abi. - Creates the daft-ext-core to bridge extension->abi. - Implements internal Daft traits using the daft-ext-internal types. - Adds a dvector example extension which is a pgvector clone. - Adds a hello example extension with a tutorial document. - Extends the session to support native function registration. - Adds the get_function method for session-backed function resolution mirroring the SQL implementation. Can add more clarity upon reviews. ## Examples - [Hello, World](https://github.com/Eventual-Inc/Daft/tree/564548a93fa99180e4e17f51049e5db60a5949cf/examples/hello) - [dvector (pgvector)](https://github.com/Eventual-Inc/Daft/tree/564548a93fa99180e4e17f51049e5db60a5949cf/examples/dvector) python import daft # Step 1. Import your extension module import hello # Step 2. Load the extension into the current daft session daft.load_extension(hello) # Step 3. Use in your dataframe! df = daft.from_pydict({"name": ["John", "Paul"]}) df = df.select(hello.greet(df["name"])) See the actual [greet rust implementation](https://github.com/Eventual-Inc/Daft/blob/564548a93fa99180e4e17f51049e5db60a5949cf/examples/hello/src/lib.rs). ## Guide See [Daft Extension Guide](https://github.com/Eventual-Inc/Daft/blob/564548a93fa99180e4e17f51049e5db60a5949cf/docs/extensions/index.md).	6 个月前
hello	chore: removes unnecessary pushdown builder code (#6403)	6 个月前
lance_rest_example.py	feat(lance): Support lance namespace read and write (#5980) ## Changes Made This PR implements Lance REST Namespace Catalog support, enabling Daft to read from and write to Lance tables managed by REST-compliant services like LanceDB Cloud, Apache Gravitino, and other catalog systems. Unified API with REST Support: - Enhanced daft.read_lance() and df.write_lance() to support REST URIs (rest://namespace/table) - Added LanceRestConfig class for REST service configuration (authentication, timeouts, custom headers) - URI-based routing automatically detects REST vs file-based Lance tables Core Implementation: - LanceRestScanOperator with filter pushdown, column selection, and count operations - REST write operations supporting create, append, and overwrite modes - Catalog management functions for table creation and registration - Comprehensive error handling and logging Enterprise Integration: - Namespace support for multi-tenant environments - API key authentication and custom headers - Compatible with Lance REST Namespace OpenAPI specification ## Related Issues Closes [#5489](https://github.com/Eventual-Inc/Daft/issues/5489)	7 个月前