GGitHubfeat: experimental vllm provider (#5443 )

4829f43b创建于 2025年11月4日历史提交

文件	最后提交记录	最后更新时间
README.md	feat: experimental vllm provider (#5443) ## Changes Made Adds the experimental `VLLMPrefixCachedProvider` for `daft.functions.ai.prompt`. Does async batching and prefix routing. When using the `VLLMPrefixCachedProvider`, `prompt` will create a `VLLMExpr` instead of a UDF, which Daft will turn into a custom VLLM operator. This operator is implemented as a streaming sink, and I had to make some minor changes to our streaming sink APIs to make the async batching mechanism work. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to `docs/mkdocs.yml` navigation - [ ] Documentation builds and is formatted properly	11 个月前
config.py	feat: experimental vllm provider (#5443) ## Changes Made Adds the experimental `VLLMPrefixCachedProvider` for `daft.functions.ai.prompt`. Does async batching and prefix routing. When using the `VLLMPrefixCachedProvider`, `prompt` will create a `VLLMExpr` instead of a UDF, which Daft will turn into a custom VLLM operator. This operator is implemented as a streaming sink, and I had to make some minor changes to our streaming sink APIs to make the async batching mechanism work. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to `docs/mkdocs.yml` navigation - [ ] Documentation builds and is formatted properly	11 个月前
continuous-batch-sorted.py	feat: experimental vllm provider (#5443) ## Changes Made Adds the experimental `VLLMPrefixCachedProvider` for `daft.functions.ai.prompt`. Does async batching and prefix routing. When using the `VLLMPrefixCachedProvider`, `prompt` will create a `VLLMExpr` instead of a UDF, which Daft will turn into a custom VLLM operator. This operator is implemented as a streaming sink, and I had to make some minor changes to our streaming sink APIs to make the async batching mechanism work. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to `docs/mkdocs.yml` navigation - [ ] Documentation builds and is formatted properly	11 个月前
continuous-batch.py	feat: experimental vllm provider (#5443) ## Changes Made Adds the experimental `VLLMPrefixCachedProvider` for `daft.functions.ai.prompt`. Does async batching and prefix routing. When using the `VLLMPrefixCachedProvider`, `prompt` will create a `VLLMExpr` instead of a UDF, which Daft will turn into a custom VLLM operator. This operator is implemented as a streaming sink, and I had to make some minor changes to our streaming sink APIs to make the async batching mechanism work. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to `docs/mkdocs.yml` navigation - [ ] Documentation builds and is formatted properly	11 个月前
generate_data.ipynb	feat: experimental vllm provider (#5443) ## Changes Made Adds the experimental `VLLMPrefixCachedProvider` for `daft.functions.ai.prompt`. Does async batching and prefix routing. When using the `VLLMPrefixCachedProvider`, `prompt` will create a `VLLMExpr` instead of a UDF, which Daft will turn into a custom VLLM operator. This operator is implemented as a streaming sink, and I had to make some minor changes to our streaming sink APIs to make the async batching mechanism work. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to `docs/mkdocs.yml` navigation - [ ] Documentation builds and is formatted properly	11 个月前
naive-batch-sorted.py	feat: experimental vllm provider (#5443) ## Changes Made Adds the experimental `VLLMPrefixCachedProvider` for `daft.functions.ai.prompt`. Does async batching and prefix routing. When using the `VLLMPrefixCachedProvider`, `prompt` will create a `VLLMExpr` instead of a UDF, which Daft will turn into a custom VLLM operator. This operator is implemented as a streaming sink, and I had to make some minor changes to our streaming sink APIs to make the async batching mechanism work. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to `docs/mkdocs.yml` navigation - [ ] Documentation builds and is formatted properly	11 个月前
naive-batch.py	feat: experimental vllm provider (#5443) ## Changes Made Adds the experimental `VLLMPrefixCachedProvider` for `daft.functions.ai.prompt`. Does async batching and prefix routing. When using the `VLLMPrefixCachedProvider`, `prompt` will create a `VLLMExpr` instead of a UDF, which Daft will turn into a custom VLLM operator. This operator is implemented as a streaming sink, and I had to make some minor changes to our streaming sink APIs to make the async batching mechanism work. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to `docs/mkdocs.yml` navigation - [ ] Documentation builds and is formatted properly	11 个月前
prefix-bucketing.py	feat: experimental vllm provider (#5443) ## Changes Made Adds the experimental `VLLMPrefixCachedProvider` for `daft.functions.ai.prompt`. Does async batching and prefix routing. When using the `VLLMPrefixCachedProvider`, `prompt` will create a `VLLMExpr` instead of a UDF, which Daft will turn into a custom VLLM operator. This operator is implemented as a streaming sink, and I had to make some minor changes to our streaming sink APIs to make the async batching mechanism work. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to `docs/mkdocs.yml` navigation - [ ] Documentation builds and is formatted properly	11 个月前
ray-data.py	feat: experimental vllm provider (#5443) ## Changes Made Adds the experimental `VLLMPrefixCachedProvider` for `daft.functions.ai.prompt`. Does async batching and prefix routing. When using the `VLLMPrefixCachedProvider`, `prompt` will create a `VLLMExpr` instead of a UDF, which Daft will turn into a custom VLLM operator. This operator is implemented as a streaming sink, and I had to make some minor changes to our streaming sink APIs to make the async batching mechanism work. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> ## Checklist - [ ] Documented in API Docs (if applicable) - [ ] Documented in User Guide (if applicable) - [ ] If adding a new documentation page, doc is added to `docs/mkdocs.yml` navigation - [ ] Documentation builds and is formatted properly	11 个月前

vLLM Benchmarks

This directory contains benchmarks for running batch inference with vLLM.

Setup

Install the dependencies:

vllm
daft
ray

Run the generate_data.ipynb notebook to generate the datasets.
Update config.py with the correct dataset path and other parameters.
Run the benchmark scripts on a Ray cluster.

Benchmarks

naive-batch.py: Simple batch inference using a Daft batch function and vLLM's LLM class.
naive-batch-sorted.py: Same as naive-batch.py, but with the prompts sorted.
continuous-batch.py: Continuous batching using the vllm-prefix-caching provider, with prefix routing disabled.
continuous-batch-sorted.py: Same as continuous-batch.py, but with the prompts sorted.
prefix-bucketing.py: Both continuous batching and prefix bucketing using the vllm-prefix-caching provider.
ray-data.py: Ray Data batching using ray.data.llm.build_llm_processor.