| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
chore(deps): Bump dependency group with conflict resolution (#6208) ## Summary Applies the Dependabot 58-package group bump from #6206 with fixes for dependency resolution conflicts that were causing CI to fail. ### 1. Optional extras upper bounds relaxed - clickhouse_connect: <0.11.0 → <0.12.0 - deltalake: <1.3.0 → <1.5.0 - google-genai: <1.53.0 → <1.64.0 - huggingface-hub: <1.2.0 → <1.5.0 - pylance: <0.40.0 → <2.1.0 - openai: <2.9.0 → <2.21.0 - sqlglot: <28.1.0 → <28.11.0 (postgres + sql extras) - ray: <2.53.0 → <2.54.0 - torch: <2.10.0 → <2.11.0 - torchvision: <0.25.0 → <0.26.0 - httpx (unity): <=0.27.2 → <=0.28.1 - av (video): <16.1.0 → <16.2.0 ### 2. Incompatible dev pin bumps reverted Several bumped packages have transitive dependencies that conflict with each other: - **vllm** 0.15.1 → 0.11.0: requires torch==2.9.1, numpy>=2, huggingface-hub<1.0 - **torch/torchvision** 2.10.0/0.25.0 → 2.8.0/0.23.0: must match vllm - **opencv-python** 4.13.0.92 → 4.10.0.84: requires numpy>=2 - **huggingface-hub** 1.4.1 → 0.36.0: vllm/transformers require <1.0 - **boto3/boto3-stubs** 1.42.49 → 1.36.2/1.38.46: no aiobotocore version supports botocore>=1.42.43 - **s3fs/gcsfs/adlfs** 2026.2.0 → originals: require fsspec==2026.2.0, but datasets==4.5.0 needs fsspec<=2025.10.0 - **pillow** 12.1.1 → 11.0.0: mkdocs-material[imaging] requires <12.0 ### 3. Other fixes - **pyiceberg** dev pin: 0.7.0 → 0.9.0 (supports tenacity>=9, within existing < 0.9.1 constraint) - Fixed stale comment referencing old requirements-dev.txt location and version Closes #6206 ## Test plan - [ ] CI dependency resolution (uv sync) succeeds - [ ] Doctests pass - [ ] Existing test suite passes 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> | 7 个月前 | |
feat(ai): Add HTTP URL direct passthrough for images and videos in prompt function (#6182) ## Changes Made This PR adds support for HTTP/HTTPS URL direct passthrough for images and videos in the prompt function. Previously, all media content had to be downloaded and base64-encoded before being sent to the LLM. Now, publicly accessible URLs are passed directly to the model, significantly reducing I/O overhead and memory consumption. <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 7 个月前 | |
feat: Improve model api typing (#5809) ## Changes Made Types the **options param in the model APIs using TypedDicts. Each model api now has it's own specific TypedDict that indicates the a subset of the possible options that can be passed, providing better typing hints for the user and their IDEs. For most of the model apis, the options include max_retries, on_error, and batch_size, which are subsequently passed into the UDF. Provider specific options can be subclassed, for example the OpenAIPromptOptions is a subclass of PromptOptions, and has an additional field for use_chat_completions, which is specific to openai prompting. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 9 个月前 | |
feat: experimental vllm provider (#5443) ## Changes Made Adds the experimental VLLMPrefixCachedProvider for daft.functions.ai.prompt. Does async batching and prefix routing. When using the VLLMPrefixCachedProvider, prompt will create a VLLMExpr instead of a UDF, which Daft will turn into a custom VLLM operator. This operator is implemented as a streaming sink, and I had to make some minor changes to our streaming sink APIs to make the async batching mechanism work. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly | 11 个月前 | |
feat: model resource plumbing for ml model-based functions (#4902) ## Changes Made * Adds embed_text daft expression * Adds TextEmbedder protocol * Adds Provider base class * Adds Descriptor[T] for making deferred initialization an API invariant * Removed batched vs. non-batched protocols to simplify * Removed lifecycle methods from the protocols **SentenceTransformers** * Adds SentenceTransformersTexEmbedder implementation * Adds SentenceTransformersProvider implementation Will implement for additional providers upon review. * Unit tests * Add OpenAI provider support * Add VLLM provider support ## Related Issues n/a ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to docs/mkdocs.yml navigation - [ ] Documentation builds and is formatted properly (tag @/ccmao1130 for docs review) | 1 年前 | |
fix: Fix Pydantic cloudpickle serialization in Google Colab (#5705) ## Summary Fixes #5696. Unblocks #5691. When using prompt() with return_format=PydanticModel in Google Colab, cloudpickle fails with: TypeError: self.handle cannot be converted to a Python object for pickling ### Root Cause When a Pydantic model is defined in a Colab notebook cell, Python's class creation machinery captures references to the notebook's execution frame globals. These globals include IPython/ZMQ internals (socket handles) that cannot be pickled. ### Solution Recreate Pydantic models using pydantic.create_model() before the model is passed into the UDF pipeline. The recreated model has a fresh, minimal class namespace rather than inheriting the Colab notebook's polluted globals. Key changes: - Add _is_colab() detection (cached at module load) - Add clean_pydantic_model() to recursively recreate models - Only apply cleaning when running in Google Colab Note: The cleaned model preserves field names, types, and defaults, which is sufficient since return_format is only used for JSON schema extraction. ### Supported Cases - Simple Pydantic models - Nested models (Parent with list[Child]) - Self-referential models (both list[TreeNode] | None and direct child: TreeNode | None) ## Test Plan - [x] Local unit tests pass (tests/ai/test_colab_compat.py) - [x] Standalone validation in Google Colab using [this test script](https://github.com/Eventual-Inc/Daft/blob/d67b618dcb38b6994b86b07ea00d4ba280325cad/colab_test_create_model.py) - [x] End-to-end validation in Google Colab with custom-built wheel ### Colab End-to-End Testing (Custom Wheel) Since Daft has Rust dependencies that can't be easily built in Colab, full end-to-end testing requires building a Linux x86_64 wheel locally and uploading it: #### 1. Build Linux wheel (on macOS with Docker + QEMU) bash # Build dashboard frontend first cd src/daft-dashboard/frontend && bun install && bun run build && cd - # Build Linux x86_64 wheel using maturin in Docker docker run --rm --platform linux/amd64 \ -e GITHUB_ACTIONS=true \ -v $(pwd):/io -w /io \ ghcr.io/pyo3/maturin:v1.7.4 \ build --release --out /io/dist-linux #### 2. Upload and install in Colab python from google.colab import files uploaded = files.upload() # Upload the .whl file !pip install daft-*.whl #### 3. Test the fix python import daft from daft.functions import prompt from pydantic import BaseModel class Result(BaseModel): answer: bool df = daft.from_pydict({"text": ["hello", "world"]}) df = df.with_column("upper", df["text"].upper()) df.show(2) import openai try: df = df.with_column("analysis", prompt(["Analyze:", df["text"]], model="gpt-4o-mini", provider="openai", return_format=Result)) except openai.OpenAIError: pass # No API key, that's fine print("Serialization worked!") This confirms the fix works in the actual Colab environment where the original issue occurs - the Pydantic model can now be serialized without the TypeError: self.handle cannot be converted to a Python object for pickling error. | 10 个月前 | |
feat: Improve model api typing (#5809) ## Changes Made Types the **options param in the model APIs using TypedDicts. Each model api now has it's own specific TypedDict that indicates the a subset of the possible options that can be passed, providing better typing hints for the user and their IDEs. For most of the model apis, the options include max_retries, on_error, and batch_size, which are subsequently passed into the UDF. Provider specific options can be subclassed, for example the OpenAIPromptOptions is a subclass of PromptOptions, and has an additional field for use_chat_completions, which is specific to openai prompting. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 9 个月前 | |
fix: Check for missing dependencies in OpenAI provider (#5747) This pull request strengthens clarifies AI provider dependencies by ensuring that required Python packages are explicitly checked at runtime and that users receive clear error messages if dependencies are missing. It also updates the optional dependency groups in pyproject.toml to more accurately reflect all required packages. Additional tests have been added to verify these import checks. **Dependency checks and error handling improvements:** * Added explicit runtime checks for required dependencies (numpy, Pillow, and provider-specific packages) in the initializers and image-processing methods of GoogleProvider, OpenAIProvider, and TransformersProvider. If a dependency is missing, a clear ImportError is raised with installation instructions. [[1]](diffhunk://#diff-73ce0c6a43abc1ee9eae9d2767d39cebae806d360ffe6e9c9258e68230c97b9dR28-R41) [[2]](diffhunk://#diff-a71920270695c38c8ab0c8fef4fe2180864fcba648939996d579c0ccf76dedd6R28-R41) [[3]](diffhunk://#diff-eac9d893500be2f016e57a39f9c1bf4956d866e64f06d364392806da65e88acfR30-R52) [[4]](diffhunk://#diff-eac9d893500be2f016e57a39f9c1bf4956d866e64f06d364392806da65e88acfR84-R94) [[5]](diffhunk://#diff-5595ceb5f68a4ab6550e1011b5dc7aca9ef4d825e1b3a8835a33a0a2575277aaR151-R155) [[6]](diffhunk://#diff-162214dc695d914a531cc1513ffb599b4d0780fc66d52d074afb5cfe830bb73aR134-R138) * Refactored ProviderImportError in provider.py to include the relevant dependency group tag and improved error messages. Updated all provider loader functions to use this new error handling, specifying all required dependencies for each provider. **Dependency group and lazy import updates:** * Updated the [project.optional-dependencies] in pyproject.toml to include all necessary dependencies for each provider group, such as adding numpy and Pillow to google and openai, and Pillow to transformers. Also added a vllm group. [[1]](diffhunk://#diff-50c86b7ed8ac2cf95bd48334961bf0530cdc77b5a56f852c5c61b89d735fd711L39-R39) [[2]](diffhunk://#diff-50c86b7ed8ac2cf95bd48334961bf0530cdc77b5a56f852c5c61b89d735fd711L48-R56) [[3]](diffhunk://#diff-50c86b7ed8ac2cf95bd48334961bf0530cdc77b5a56f852c5c61b89d735fd711R65) * Added torchvision as a lazy import in dependencies.py and included it in the list of exported dependencies. [[1]](diffhunk://#diff-f90de260c6dde7a325353b8be65ebcf610591df4181a824485dae16b7f96e1c1R25) [[2]](diffhunk://#diff-f90de260c6dde7a325353b8be65ebcf610591df4181a824485dae16b7f96e1c1R44) [[3]](diffhunk://#diff-f90de260c6dde7a325353b8be65ebcf610591df4181a824485dae16b7f96e1c1R66) **Testing enhancements:** * Added tests for GoogleProvider and OpenAIProvider to verify that an ImportError is raised if numpy or Pillow is missing. These tests use mocking to simulate missing dependencies and check for the correct error messages. [[1]](diffhunk://#diff-ab5372ee857c4cd16caccebebc79ea72de6e963c75a36394b18f6d3c534f8d88R3-R6) [[2]](diffhunk://#diff-ab5372ee857c4cd16caccebebc79ea72de6e963c75a36394b18f6d3c534f8d88R42-R47) [[3]](diffhunk://#diff-d3dd86cf39eebb5abaec579cfbedd909bd310bd0b272a0b317ae9275ea9cd825R3-R4) [[4]](diffhunk://#diff-d3dd86cf39eebb5abaec579cfbedd909bd310bd0b272a0b317ae9275ea9cd825R110-R115) [[5]](diffhunk://#diff-cd6942ac52ad4987db8dab130f21a0ed1047a92cfb180b1a90db93ea1593b1cfR498-R512) [[6]](diffhunk://#diff-eca3e35ba0571a8008c16f9abf4a23e70222c4aad298fcd42af4f51169cc81feR505-R518) --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> | 10 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 7 个月前 | ||
| 7 个月前 | ||
| 9 个月前 | ||
| 11 个月前 | ||
| 1 年前 | ||
| 10 个月前 | ||
| 9 个月前 | ||
| 10 个月前 |