| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
Added lora support proposal (#216) * added lora support proposal Signed-off-by: Chen Wang <Chen.Wang1@ibm.com> * updated lora-k8s-arch.png Signed-off-by: Chen Wang <Chen.Wang1@ibm.com> --------- Signed-off-by: Chen Wang <Chen.Wang1@ibm.com> | 1 年前 | |
Adding contributing guidelines (#120) Signed-off-by: Shaoting <shaotingf@uchicago.edu> | 1 年前 | |
[Feat][Router] Add disaggregated prefill orchestrated routing (#777) * [Feat][Router] Add disaggregated prefill orchestrated routing Implements support for disaggregated prefill as outlined in the 2025 Q1 roadmap. This enables prefill/decode disaggregation with router-orchestrated KV cache transfer. Closes #26 Signed-off-by: Yahav <yahavb@amazon.com> * [CI/Build] Lower Python version requirement to 3.10 for Neuron SDK compatibility Signed-off-by: Yahav <yahavb@amazon.com> * [Feat][Router] Address PR review feedback for disaggregated prefill orchestrated routing - Remove dead code (handle_orchestrated_request method in routing_logic.py) - Fix prefill request to use max_tokens=1 per proposal spec - Use shared aiohttp client instead of creating new session per request - Fix streaming to yield chunks immediately (true streaming) - Remove redundant isinstance check for DisaggregatedPrefillOrchestratedRouter - Use router's _find_endpoints method to avoid code duplication Signed-off-by: Yahav <yahavb@amazon.com> * fix: use kv_transfer_params instead of disagg_prefill_resp - Add kv_transfer_params to prefill request to enable disaggregated mode - Extract kv_transfer_params from prefill response and forward to decode - Set remote_host to prefill endpoint for KV cache retrieval Signed-off-by: Yahav <yahavb@amazon.com> * docs: add example for disaggregated_prefill_orchestrated mode - Add README with usage instructions and configuration notes - Add sanitized Kubernetes manifests (router, prefill, decode) - Include example curl command and expected router logs Signed-off-by: Yahav <yahavb@amazon.com> * style: fix black formatting Signed-off-by: Yahav <yahavb@amazon.com> * style: fix markdownlint errors in README.md Signed-off-by: Yahav <yahavb@amazon.com> * style: fix markdownlint errors in proposal doc Signed-off-by: Yahav <yahavb@amazon.com> * docs: clean up DisaggregatedPrefillOrchestratedRouter docstring Signed-off-by: Yahav <yahavb@amazon.com> * feat: return 503 with distinct codes for prefill/decode unavailability - PREFILL_SERVICE_UNAVAILABLE: No prefill endpoints discovered - DECODE_SERVICE_UNAVAILABLE: No decode endpoints discovered This allows automated tests to distinguish transient startup issues from real bugs. Signed-off-by: Yahav <yahavb@amazon.com> * revert: restore requires-python = 3.12 Signed-off-by: Yahav <yahavb@amazon.com> * fix: replace angle bracket placeholders with uppercase format Angle brackets like <your-pvc-name> are interpreted as shell redirections by shellcheck, causing CI failures. Use uppercase format instead: YOUR-PVC-NAME, YOUR-MODEL-PATH, etc. Signed-off-by: Yahav <yahavb@amazon.com> * fix: remove trailing whitespace from YAML files Signed-off-by: Yahav <yahavb@amazon.com> --------- Signed-off-by: Yahav <yahavb@amazon.com> Co-authored-by: Rui Zhang <51696593+ruizhang0101@users.noreply.github.com> | 6 个月前 | |
Added lora support proposal (#216) * added lora support proposal Signed-off-by: Chen Wang <Chen.Wang1@ibm.com> * updated lora-k8s-arch.png Signed-off-by: Chen Wang <Chen.Wang1@ibm.com> --------- Signed-off-by: Chen Wang <Chen.Wang1@ibm.com> | 1 年前 | |
feat(oci): Add Oracle Cloud Infrastructure (OKE) deployment support (#794) * Update calendar link for community meetings (#783) * Update calendar link for community meetings Signed-off-by: Rui Zhang <51696593+zerofishnoodles@users.noreply.github.com> * Update calendar link for community event Signed-off-by: Rui Zhang <51696593+zerofishnoodles@users.noreply.github.com> * Update calendar link for community meetings Signed-off-by: Rui Zhang <51696593+zerofishnoodles@users.noreply.github.com> --------- Signed-off-by: Rui Zhang <51696593+zerofishnoodles@users.noreply.github.com> Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com> * Update hte documentation of the semantic router deployment to use helm charts and helm command (#786) Signed-off-by: szedan <szedan@redhat.com> Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com> * Fix incorrect import path in batch processor initialization (#784) * Fix incorrect import path in batch processor initialization Fixes #766 The import statement for LocalBatchProcessor was using an incorrect path (vllm_router.batch.local_processor) that does not exist in the codebase. This caused a ModuleNotFoundError when users tried to enable the batch API with the --enable_batch_api flag. Changed the import to use the correct path: vllm_router.services.batch_service.local_processor This unblocks the batch API functionality which was previously unusable due to this import error. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: Keyu Chen <54015474+keyuchen21@users.noreply.github.com> * Fix black formatting for import statement Signed-off-by: Keyu Chen <54015474+keyuchen21@users.noreply.github.com> --------- Signed-off-by: Keyu Chen <54015474+keyuchen21@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com> Co-authored-by: Rui Zhang <51696593+zerofishnoodles@users.noreply.github.com> Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com> * [Build][Router] Update aiohttp (#793) * build: Update aiohttp to 3.13.3 Implements #792 Signed-off-by: Shern Shiou Tan <shernshiou@gmail.com> * fix: Int type for health_check_interval and lmcache_worker_timeout Signed-off-by: Shern Shiou Tan <shernshiou@gmail.com> * style: Rename to lmcache_health_check_interval for consistency Signed-off-by: Shern Shiou Tan <shernshiou@gmail.com> --------- Signed-off-by: Shern Shiou Tan <shernshiou@gmail.com> Co-authored-by: Rui Zhang <51696593+zerofishnoodles@users.noreply.github.com> Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com> * feat(oci): Add Oracle Cloud Infrastructure (OKE) deployment support Add comprehensive OCI support for deploying vLLM production-stack on Oracle Kubernetes Engine (OKE). Features: - One-click deployment via entry_point.sh (setup/cleanup commands) - Private cluster support with OCI Bastion for secure access - GPU node pool with A10/A100/H100 shape support - OCI Block Volume StorageClass for model persistence - NAT Gateway for private subnet internet access - Comprehensive prerequisite validation (oci, jq, kubectl, helm) Documentation: - README with environment variables and GPU shape reference - Step-by-step tutorial for OKE deployment - RFC proposal document - RDMA multi-node configuration guide - OCI Data Science integration path Tested on us-phoenix-1 with VM.GPU.A10.1 shape. Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com> * Fix bugs discovered during end-to-end testing of OCI deployment - Fix get_gpu_image() to use oci_cmd wrapper instead of raw oci command (required for non-DEFAULT OCI profiles) - Add kubeconfig profile fix for non-DEFAULT OCI CLI profiles (kubectl exec auth now includes --profile argument) - Replace fixed sleep times with polling in cleanup: - Node pool deletion now polls until all pools are deleted - Cluster deletion now polls cluster state until DELETED - Add retry logic for subnet deletion (handles dependencies from cluster resources that may still be terminating) Tested successfully with private OKE cluster + bastion in us-phoenix-1. Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com> * Skip kubectl-dependent steps for private clusters For private clusters, the API endpoint is not reachable without an SSH tunnel through the bastion. The script now: - Creates all infrastructure (VCN, cluster, bastion, GPU node pool) - Configures kubeconfig for private endpoint - Prints SSH tunnel setup instructions - Prints manual commands to run after tunnel is established The NVIDIA device plugin, storage class, and Helm deployment steps are skipped for private clusters since they require kubectl connectivity. Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com> * Fix security list for OKE GPU node registration Add Kubernetes pods CIDR (10.244.0.0/16) and services CIDR (10.96.0.0/16) to the security list ingress rules. Also add ICMP type 3 code 4 for path MTU discovery. Without these rules, GPU worker nodes fail to register with the OKE control plane and get stuck in "installing software" state with a 20-minute timeout error. Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com> * Address Gemini code review comments for OCI deployment High Priority: - Restrict SSH access to VCN CIDR only (security improvement) Medium Priority: - Improve cleanup error handling with explicit warnings - Add OCI documentation link for K8s version support - Better helm/kubectl checks before operations The deploy_model.py --wait issue was already correctly implemented using --no-wait with store_false pattern. Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com> * Address code review feedback for OCI deployment Critical: - Scope PVC/PV deletion to vLLM-labeled resources only (clean_up.sh) - Make bastion client CIDR configurable via BASTION_CLIENT_CIDR env var with warning when using insecure default (0.0.0.0/0) High Priority: - Use Kubernetes secrets for hf_token instead of hardcoded values - Remove unnecessary GPU request from Ray head deployment (RDMA docs) Medium Priority: - Clarify BYOC vs model artifact deployment approaches (data-science docs) - Use server-side filtering in get_existing_deployment (deploy_model.py) - Simplify SSH tunnel instructions with clearer guidance - Fix K8s version inconsistency (v1.30.1 -> v1.31.10) in tutorial - Fix kubectl custom-columns quoting syntax error in tutorial Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com> * Remove data-science and rdma folders for parity with other providers AWS, GCP, and Azure deployments only include basic OKE/EKS/GKE/AKS scripts. Remove OCI-specific extras (Data Science Model Deployment, RDMA multi-node) to keep the PR focused on core OKE deployment. These can be added in future PRs if needed. Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com> * Update documentation after removing data-science and rdma folders - Remove references to rdma/ and data-science/ directories from README - Add BASTION_CLIENT_CIDR to environment variables table - Fix tutorial cleanup to use label selector for PVC deletion Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com> * fix(oci): improve GPU disk expansion with proper LVM steps - Rewrite expand_gpu_disk() with step-by-step verification - Use oraclelinux:8 image (has LVM tools) instead of busybox - Run growpart explicitly first - critical step that must happen before pvresize/lvextend have any effect - Stream logs during expansion to show progress - Add clear troubleshooting guidance if expansion fails - Update header comments with key learnings about disk expansion Key insight: pvresize/lvextend do nothing if growpart hasn't expanded the partition first. The script now ensures proper ordering and verification of each step. Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com> * docs(oci): improve disk expansion documentation with troubleshooting - Rewrite Step 10 (disk expansion) with detailed explanation - Add table showing the 4-step expansion process and verification - Add both script and manual expansion options - Add comprehensive troubleshooting section for expansion failures - Update Key Gotchas with new sections: - Disk Expansion Order Matters (CRITICAL) - Expansion Pod Timing - Kubelet Refresh After Expansion The documentation now clearly explains why expansion fails when steps are out of order or incomplete. Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com> * config(oci): update to use openai/gpt-oss-20b model - Change from Llama-3.1-8B-Instruct to openai/gpt-oss-20b - Increase memory to 24Gi for MoE model - Remove HF token requirement (Apache 2.0 licensed) - Increase max-model-len to 8192 GPT-OSS-20B is a MoE model with only 3.6B active parameters, fitting comfortably on a single A10 GPU (24GB). Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com> * docs(oci): add disk expansion step to OCI tutorial - Add Step 8 for GPU node filesystem expansion (critical) - Add note about private cluster requiring bastion access - Update step numbers (8->9, 9->10, 10->11, 11->12) - Add reference to README for troubleshooting - Update summary to include disk expansion step Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com> * fix: resolve markdownlint errors in OCI documentation - Add blank lines around fenced code blocks - Add blank lines around lists and tables - Add language specifications to code blocks Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com> * Update Slack channel link in README (#798) - the slack channel is invite only, so the original link doesn't work!!!! - the new one works and tested: https://communityinviter.com/apps/vllm-dev/join-vllm-developers-slack Signed-off-by: Keyu Chen <54015474+keyuchen21@users.noreply.github.com> Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com> * [Doc] Remove official email link from README (#805) Signed-off-by: Rui Zhang <51696593+zerofishnoodles@users.noreply.github.com> Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com> * fix(oci): use parameter expansion instead of sed for version prefix strip Replace echo | sed with bash parameter expansion ${var#v} to satisfy shellcheck SC2001. Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com> * fix(oci): remove dead code and fix namespace bug in clean_up.sh Remove unused SCRIPT_DIR and OCI_COMPARTMENT_ID variables (the script only uses helm/kubectl, not the OCI CLI). Fix custom resource deletion to use --all-namespaces instead of --all, matching the get check so resources in non-default namespaces are actually deleted. Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com> --------- Signed-off-by: Rui Zhang <51696593+zerofishnoodles@users.noreply.github.com> Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com> Signed-off-by: szedan <szedan@redhat.com> Signed-off-by: Keyu Chen <54015474+keyuchen21@users.noreply.github.com> Signed-off-by: Shern Shiou Tan <shernshiou@gmail.com> Co-authored-by: Rui Zhang <51696593+zerofishnoodles@users.noreply.github.com> Co-authored-by: Senan Zedan <szedan@redhat.com> Co-authored-by: Keyu Chen <54015474+keyuchen21@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com> Co-authored-by: Shern Shiou Tan <shernshiou@gmail.com> | 7 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 1 年前 | ||
| 1 年前 | ||
| 6 个月前 | ||
| 1 年前 | ||
| 7 个月前 |