GGitHubfix tutorial dead url (#947)
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
[helm] document every values, udpate json schema and various fix (#886) * fix(helm) update sharedPvcStorage so it correctly create PVC and/or PV Signed-off-by: enneitex <etienne.divet@gmail.com> * fix(helm) various fix to align values, templates and README. update json schema Signed-off-by: enneitex <etienne.divet@gmail.com> * fix(helm) use correct type for lmcacheControllerPort Signed-off-by: enneitex <etienne.divet@gmail.com> * fix(helm) rayCluster: fix templating and add tests Signed-off-by: enneitex <etienne.divet@gmail.com> * fix(helm) deployment-router: simplify templating and add tests Signed-off-by: enneitex <etienne.divet@gmail.com> * feat(helm) improve schema, add tests and CI validation Signed-off-by: enneitex <etienne.divet@gmail.com> feat(helm) improve testing around rayCluster and engine deployment Signed-off-by: enneitex <etienne.divet@gmail.com> ci: retrigger Signed-off-by: enneitex <etienne.divet@gmail.com> fix(tutorials) rename ScaledObject file name Signed-off-by: enneitex <etienne.divet@gmail.com> fix pre-commit Signed-off-by: enneitex <etienne.divet@gmail.com> --------- Signed-off-by: enneitex <etienne.divet@gmail.com> Co-authored-by: Rui Zhang <51696593+ruizhang0101@users.noreply.github.com> | 4 个月前 | |
Fix OCI OKE deployment script (entry_point.sh) — end-to-end tested (#811) * fix(oci): refactor entry_point.sh and align README with end-to-end tested deployment Key changes to entry_point.sh: - Refactor expand_gpu_disk to single kubectl exec with retry logic - Replace kubectl wait -l app=gpu with polling loop (handles no-match) - Add || true guards for kubectl exec exit code 137 (SIGKILL after success) - Fix set -euo pipefail compatibility when no GPU nodes exist yet - Use kubectl config view --minify -o jsonpath for kubeconfig cluster name - Remove --wait from helm deploy (router CrashLoops until patched) - Use nsenter instead of chroot for kubelet restart (systemd bus access) - Add CPU_BOOT_VOLUME_GB=100 to prevent DiskPressure from router image - Revert image queries to !contains('aarch64') — OKE images lack x86_64 - Cleanup deletes all clusters with matching name, not just the first - Fix stdout pollution breaking filesystem size verification Key changes to README.md: - Use kubectl config view --minify for kubeconfig (matches entry_point.sh) - Replace separate restart-kubelet pod with nsenter via expand-gpu-disk pod - Add CPU_BOOT_VOLUME_GB to environment variables table - Document 4 additional gotchas (exit code 137, CPU disk pressure, OCI image naming, kubeconfig context vs cluster name) Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com> * docs: add OCI security callouts Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com> --------- Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com> Co-authored-by: Rui Zhang <51696593+ruizhang0101@users.noreply.github.com> | 7 个月前 | |
[Bugfix][Dashboard] Dedup healthy_pods_total across router replicas (#943) count by(endpoint) (vllm:healthy_pods_total) returns N samples per backend with N router replicas — each router emits its own time-series with distinct scrape-target labels. The "Available vLLM instances" stat ends up multiplied. sum(max by (server) (vllm:healthy_pods_total)) collapses replicas first, so the count matches actual healthy backends regardless of router count. Fixes #644 Signed-off-by: Mikhail Basov <Michael.S.Sinclair@protonmail.com> Co-authored-by: Rui Zhang <51696593+ruizhang0101@users.noreply.github.com> | 3 个月前 | |
Bugfix/482 helm rayspec fix (#483) * [bugfix] Bugfixed raySpec preventing multiple deployments specified with modelSpecs. Signed-off-by: ahinsutime <ahinsutime@gmail.com> * [Doc] Updated tutorial document and values for raySpec. Signed-off-by: ahinsutime <ahinsutime@gmail.com> * [Bugfix] Updated helm chart version to sync with changes due to bugfix. Signed-off-by: ahinsutime <ahinsutime@gmail.com> * [Doc] Added guideline to deploy both ray cluster and deployments. Fixed typos. Added more example values. Signed-off-by: ahinsutime <ahinsutime@gmail.com> * [Bugfix] Fixed configmap conflicts by distinguishing configmap names. Signed-off-by: ahinsutime <ahinsutime@gmail.com> --------- Signed-off-by: ahinsutime <ahinsutime@gmail.com> Co-authored-by: Yuhan Liu <32589867+YuhanLiu11@users.noreply.github.com> | 1 年前 | |
update install script kubectl command to find kuberay-operator pod globally (#460) Signed-off-by: googs1025 <googs1025@gmail.com> Co-authored-by: Yuhan Liu <32589867+YuhanLiu11@users.noreply.github.com> | 1 年前 | |
[Doc] Kubernetes in Docker (kind) tutorial (#534) * created two files for use in kind portion of 00 tutorial Signed-off-by: Lucas T <lucastbooms@gmail.com> * new branch due to commit issue Signed-off-by: Lucas T <lucastbooms@gmail.com> * added install-kind script Signed-off-by: Lucas T <lucastbooms@gmail.com> * created kind cluster script Signed-off-by: Lucas T <lucastbooms@gmail.com> * removed original kind md Signed-off-by: Lucas T <lucastbooms@gmail.com> * removed initial sh script Signed-off-by: Lucas T <lucastbooms@gmail.com> * whitespace issues Signed-off-by: Lucas T <lucastbooms@gmail.com> * additional formatting Signed-off-by: Lucas T <lucastbooms@gmail.com> * additional formatting -- will use pre-commit hooks in future Signed-off-by: Lucas T <lucastbooms@gmail.com> * gemini suggestions Signed-off-by: Lucas T <lucastbooms@gmail.com> --------- Signed-off-by: Lucas T <lucastbooms@gmail.com> Co-authored-by: lucast2021 <lucast2021@headroyce.org> Co-authored-by: Yuhan Liu <32589867+YuhanLiu11@users.noreply.github.com> | 1 年前 | |
[Doc] Added documentation about uninstalling previous minikube installation. (#405) Signed-off-by: insukim1994 <insu.kim@moreh.io> | 1 年前 | |
[Feat] Added a tutorial document for deploying production stack on amd gpus (#364) * [Feat] Added a tutorial document for deploying production stack on amd gpus. Initial commit. Signed-off-by: insukim1994 <insu.kim@moreh.io> * [Feat] Refined tutorial documentation format and sentences. Added real test example output. Signed-off-by: insukim1994 <insu.kim@moreh.io> * [Feat] Added a guide refers to the link for amd gpu related setup prerequisites. Signed-off-by: insukim1994 <insu.kim@moreh.io> * [Feat] Renamed filename for minimal helm example tutorial on amd gpu node (review updates). Signed-off-by: insukim1994 <insu.kim@moreh.io> --------- Signed-off-by: insukim1994 <insu.kim@moreh.io> | 1 年前 | |
[Doc Improvement]Fix tutorial hyper link (#355) * update doc links for tutorial 00, 01 Signed-off-by: Dong Wang <dongw2019@gmail.com> * update doc links for tutorial 00, 01 Signed-off-by: Dong Wang <dongw2019@gmail.com> * revert back to upstream version Signed-off-by: Dong Wang <dongw2019@gmail.com> * update doc link for remaining tutorials Signed-off-by: Dong Wang <dongw2019@gmail.com> * typo Signed-off-by: Dong Wang <dongw2019@gmail.com> --------- Signed-off-by: Dong Wang <dongw2019@gmail.com> | 1 年前 | |
[bugfix] deprecate disable log request (#885) * bugfix: deprecate disable log request Signed-off-by: Rui Zhang <zrfishnoodles@gmail.com> * Update helm/README.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Rui Zhang <51696593+ruizhang0101@users.noreply.github.com> --------- Signed-off-by: Rui Zhang <zrfishnoodles@gmail.com> Signed-off-by: Rui Zhang <51696593+ruizhang0101@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> | 5 个月前 | |
[Doc Improvement]Fix tutorial hyper link (#355) * update doc links for tutorial 00, 01 Signed-off-by: Dong Wang <dongw2019@gmail.com> * update doc links for tutorial 00, 01 Signed-off-by: Dong Wang <dongw2019@gmail.com> * revert back to upstream version Signed-off-by: Dong Wang <dongw2019@gmail.com> * update doc link for remaining tutorials Signed-off-by: Dong Wang <dongw2019@gmail.com> * typo Signed-off-by: Dong Wang <dongw2019@gmail.com> --------- Signed-off-by: Dong Wang <dongw2019@gmail.com> | 1 年前 | |
[Doc Improvement]Fix tutorial hyper link (#355) * update doc links for tutorial 00, 01 Signed-off-by: Dong Wang <dongw2019@gmail.com> * update doc links for tutorial 00, 01 Signed-off-by: Dong Wang <dongw2019@gmail.com> * revert back to upstream version Signed-off-by: Dong Wang <dongw2019@gmail.com> * update doc link for remaining tutorials Signed-off-by: Dong Wang <dongw2019@gmail.com> * typo Signed-off-by: Dong Wang <dongw2019@gmail.com> --------- Signed-off-by: Dong Wang <dongw2019@gmail.com> | 1 年前 | |
[Doc Improvement]Fix tutorial hyper link (#355) * update doc links for tutorial 00, 01 Signed-off-by: Dong Wang <dongw2019@gmail.com> * update doc links for tutorial 00, 01 Signed-off-by: Dong Wang <dongw2019@gmail.com> * revert back to upstream version Signed-off-by: Dong Wang <dongw2019@gmail.com> * update doc link for remaining tutorials Signed-off-by: Dong Wang <dongw2019@gmail.com> * typo Signed-off-by: Dong Wang <dongw2019@gmail.com> --------- Signed-off-by: Dong Wang <dongw2019@gmail.com> | 1 年前 | |
[Bugfix] Fixing bugs caused by LMCache docker image update and static service discovery (#456) * Fixing bugs caused by LMCache docker image update Signed-off-by: YuhanLiu11 <yliu738@wisc.edu> * Fixing bugs caused by LMCache docker image update Signed-off-by: YuhanLiu11 <yliu738@wisc.edu> --------- Signed-off-by: YuhanLiu11 <yliu738@wisc.edu> | 1 年前 | |
[bugfix] deprecate disable log request (#885) * bugfix: deprecate disable log request Signed-off-by: Rui Zhang <zrfishnoodles@gmail.com> * Update helm/README.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Rui Zhang <51696593+ruizhang0101@users.noreply.github.com> --------- Signed-off-by: Rui Zhang <zrfishnoodles@gmail.com> Signed-off-by: Rui Zhang <51696593+ruizhang0101@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> | 5 个月前 | |
[bugfix] deprecate disable log request (#885) * bugfix: deprecate disable log request Signed-off-by: Rui Zhang <zrfishnoodles@gmail.com> * Update helm/README.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Rui Zhang <51696593+ruizhang0101@users.noreply.github.com> --------- Signed-off-by: Rui Zhang <zrfishnoodles@gmail.com> Signed-off-by: Rui Zhang <51696593+ruizhang0101@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> | 5 个月前 | |
fix tutorial dead url (#947) Signed-off-by: Jinjia <jinjiaKarl@outlook.com> | 3 个月前 | |
[Document, Feat] basic HPA support and tutorials (#209) * [Add] basic HPA support and tutorials Signed-off-by: ApostaC <yihua98@uchicago.edu> * fix shellcheck errors Signed-off-by: ApostaC <yihua98@uchicago.edu> --------- Signed-off-by: ApostaC <yihua98@uchicago.edu> | 1 年前 | |
[bugfix] deprecate disable log request (#885) * bugfix: deprecate disable log request Signed-off-by: Rui Zhang <zrfishnoodles@gmail.com> * Update helm/README.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Rui Zhang <51696593+ruizhang0101@users.noreply.github.com> --------- Signed-off-by: Rui Zhang <zrfishnoodles@gmail.com> Signed-off-by: Rui Zhang <51696593+ruizhang0101@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> | 5 个月前 | |
[Feat] Router: Add OpenTelemetry tracing support with W3C context propagation (#772) Signed-off-by: Fang Han <fhan0520@gmail.com> | 8 个月前 | |
Update tool support (#352) Signed-off-by: YuhanLiu11 <yliu738@wisc.edu> | 1 年前 | |
[Feat] Adding a tutorial for using vLLM v1 in production stack (#390) * Adding vLLM v1 tutorial Signed-off-by: YuhanLiu11 <yliu738@wisc.edu> * Bump helm chart version Signed-off-by: YuhanLiu11 <yliu738@wisc.edu> * Fixing yaml file format Signed-off-by: YuhanLiu11 <yliu738@wisc.edu> * Fixing yaml file format Signed-off-by: YuhanLiu11 <yliu738@wisc.edu> * fix yaml formate Signed-off-by: YuhanLiu11 <yliu738@wisc.edu> --------- Signed-off-by: YuhanLiu11 <yliu738@wisc.edu> | 1 年前 | |
[helm] document every values, udpate json schema and various fix (#886) * fix(helm) update sharedPvcStorage so it correctly create PVC and/or PV Signed-off-by: enneitex <etienne.divet@gmail.com> * fix(helm) various fix to align values, templates and README. update json schema Signed-off-by: enneitex <etienne.divet@gmail.com> * fix(helm) use correct type for lmcacheControllerPort Signed-off-by: enneitex <etienne.divet@gmail.com> * fix(helm) rayCluster: fix templating and add tests Signed-off-by: enneitex <etienne.divet@gmail.com> * fix(helm) deployment-router: simplify templating and add tests Signed-off-by: enneitex <etienne.divet@gmail.com> * feat(helm) improve schema, add tests and CI validation Signed-off-by: enneitex <etienne.divet@gmail.com> feat(helm) improve testing around rayCluster and engine deployment Signed-off-by: enneitex <etienne.divet@gmail.com> ci: retrigger Signed-off-by: enneitex <etienne.divet@gmail.com> fix(tutorials) rename ScaledObject file name Signed-off-by: enneitex <etienne.divet@gmail.com> fix pre-commit Signed-off-by: enneitex <etienne.divet@gmail.com> --------- Signed-off-by: enneitex <etienne.divet@gmail.com> Co-authored-by: Rui Zhang <51696593+ruizhang0101@users.noreply.github.com> | 4 个月前 | |
[Doc] Update disagg prefill tutorial docs (#497) * Update disagg prefill tutorial docs Signed-off-by: zhuohangu <zhuohan@uchicago.edu> * [Doc] Remove extra blank line Signed-off-by: zhuohangu <zhuohan@uchicago.edu> * [Doc] Add NVLink requirement to prerequisites Signed-off-by: zhuohangu <zhuohan@uchicago.edu> --------- Signed-off-by: zhuohangu <zhuohan@uchicago.edu> | 1 年前 | |
[Bugfix] kv aware routing for lmcache 0.3.9 (#697) * [bugfix] fix kvaware routing Signed-off-by: Rui Zhang <zrfishnoodles@gmail.com> * Modify CI to be compatible Signed-off-by: Rui Zhang <zrfishnoodles@gmail.com> * Revert "fix dynamic config" This reverts commit 9ac02f992e58fa102a891d3dfe903142bf913f0b. Signed-off-by: Rui Zhang <zrfishnoodles@gmail.com> * modify CI Signed-off-by: Rui Zhang <zrfishnoodles@gmail.com> * bugfix: fix bug for kvaware routing to be compatiable with lmcache 0.3.9 Signed-off-by: Rui Zhang <rzhan229@ucsc.edu> * bugfix: fix ci Signed-off-by: Rui Zhang <rzhan229@ucsc.edu> * fix ci Signed-off-by: Rui Zhang <rzhan229@ucsc.edu> * modify ci Signed-off-by: Rui Zhang <rzhan229@ucsc.edu> * modify ci Signed-off-by: Rui Zhang <rzhan229@ucsc.edu> * modify ci Signed-off-by: Rui Zhang <rzhan229@ucsc.edu> * modify ci Signed-off-by: Rui Zhang <rzhan229@ucsc.edu> * update controller manager init Signed-off-by: Rui Zhang <rzhan229@ucsc.edu> * increase startup probe time Signed-off-by: Rui Zhang <rzhan229@ucsc.edu> * modify debug messaage Signed-off-by: Rui Zhang <rzhan229@ucsc.edu> --------- Signed-off-by: Rui Zhang <zrfishnoodles@gmail.com> Signed-off-by: Rui Zhang <rzhan229@ucsc.edu> | 9 个月前 | |
[Doc] Adding doc for KV aware routing and Prefix aware routing (#480) * Adding doc for pd Signed-off-by: YuhanLiu11 <yliu738@wisc.edu> * Adding doc for pd Signed-off-by: YuhanLiu11 <yliu738@wisc.edu> * Adding doc for pd Signed-off-by: YuhanLiu11 <yliu738@wisc.edu> * adding docs for kv aware routing Signed-off-by: YuhanLiu11 <yliu738@wisc.edu> * Adding doc page for kv aware routing Signed-off-by: YuhanLiu11 <yliu738@wisc.edu> * Adding docs for prefix aware routing Signed-off-by: YuhanLiu11 <yliu738@wisc.edu> * Adding docs for prefix aware routing Signed-off-by: YuhanLiu11 <yliu738@wisc.edu> * Adding docs for prefix aware routing Signed-off-by: YuhanLiu11 <yliu738@wisc.edu> * Adding docs for prefix aware routing Signed-off-by: YuhanLiu11 <yliu738@wisc.edu> * Adding docs for prefix aware routing Signed-off-by: YuhanLiu11 <yliu738@wisc.edu> * Adding docs for prefix aware routing Signed-off-by: YuhanLiu11 <yliu738@wisc.edu> * Adding docs for prefix aware routing Signed-off-by: YuhanLiu11 <yliu738@wisc.edu> --------- Signed-off-by: YuhanLiu11 <yliu738@wisc.edu> | 1 年前 | |
Add support for sleep and wake_up endpoints (#498) Signed-off-by: Braulio Dumba <Braulio.Dumba@ibm.com> Co-authored-by: Braulio Dumba <Braulio.Dumba@ibm.com> | 1 年前 | |
[helm] document every values, udpate json schema and various fix (#886) * fix(helm) update sharedPvcStorage so it correctly create PVC and/or PV Signed-off-by: enneitex <etienne.divet@gmail.com> * fix(helm) various fix to align values, templates and README. update json schema Signed-off-by: enneitex <etienne.divet@gmail.com> * fix(helm) use correct type for lmcacheControllerPort Signed-off-by: enneitex <etienne.divet@gmail.com> * fix(helm) rayCluster: fix templating and add tests Signed-off-by: enneitex <etienne.divet@gmail.com> * fix(helm) deployment-router: simplify templating and add tests Signed-off-by: enneitex <etienne.divet@gmail.com> * feat(helm) improve schema, add tests and CI validation Signed-off-by: enneitex <etienne.divet@gmail.com> feat(helm) improve testing around rayCluster and engine deployment Signed-off-by: enneitex <etienne.divet@gmail.com> ci: retrigger Signed-off-by: enneitex <etienne.divet@gmail.com> fix(tutorials) rename ScaledObject file name Signed-off-by: enneitex <etienne.divet@gmail.com> fix pre-commit Signed-off-by: enneitex <etienne.divet@gmail.com> --------- Signed-off-by: enneitex <etienne.divet@gmail.com> Co-authored-by: Rui Zhang <51696593+ruizhang0101@users.noreply.github.com> | 4 个月前 | |
[Doc] Adding tutorial for Gateway Inference Extension support (#570) * Adding tutorial for GIE Signed-off-by: YuhanLiu11 <yliu738@wisc.edu> * format checking Signed-off-by: YuhanLiu11 <yliu738@wisc.edu> * fixing shell format checker Signed-off-by: YuhanLiu11 <yliu738@wisc.edu> * fixing comments from gemini Signed-off-by: YuhanLiu11 <yliu738@wisc.edu> --------- Signed-off-by: YuhanLiu11 <yliu738@wisc.edu> Co-authored-by: Shaoting <shaotingf@uchicago.edu> | 1 年前 | |
[Feat][Helm] Add HTTPRoute template for Gateway API support (#610) Add a route template to allow for configuring k8s Gateway API HTTP (or other) routes. See https://gateway-api.sigs.k8s.io/ Signed-off-by: Zackeus Bengtsson <github@zackeus.no> Co-authored-by: Yuhan Liu <32589867+YuhanLiu11@users.noreply.github.com> | 1 年前 | |
Feat/implement streaming path in audio transcription (#926) * implement streaming path in audio transcription endpoint Signed-off-by: WaelRabah11 <wael.rabah@multiversecomputing.com> * Refactored proxy_multipart_request to share post call, headers, and error handling between streaming and non-streaming paths. Stats hooks (on_new_request/on_request_response/on_request_complete) now fire on both paths. Tutorial fixed to use with open(...). Tests pass (8/8). Signed-off-by: WaelRabah11 <wael.rabah@multiversecomputing.com> * Update src/vllm_router/services/request_service/request.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Wael Rabah <47258198+WaelRabah@users.noreply.github.com> * update streaming example, refactor try/except to reduce nesting and catch json decode errors Signed-off-by: WaelRabah11 <wael.rabah@multiversecomputing.com> * Added backend_response.close() to the non-streaming finally block Signed-off-by: WaelRabah11 <wael.rabah@multiversecomputing.com> * fix outdated test Signed-off-by: WaelRabah11 <wael.rabah@multiversecomputing.com> --------- Signed-off-by: WaelRabah11 <wael.rabah@multiversecomputing.com> Signed-off-by: Wael Rabah <47258198+WaelRabah@users.noreply.github.com> Co-authored-by: WaelRabah11 <wael.rabah@multiversecomputing.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> | 4 个月前 | |
Update hte documentation of the semantic router deployment to use helm charts and helm command (#786) Signed-off-by: szedan <szedan@redhat.com> | 7 个月前 | |
Add pre-commit based linting and formatting (#35) * Add pre-commit workflow * Add actionlint * Add generic hooks * Add black, isort, shellcheck * Add requirements and markdown linting * Add toml * Add Dockerfile * Add codespell * Use Node.js version of markdownlint * Add requirements-lint.txt * Use CLI version of Node.js markdownlint * Add pre-commit instructions to Contributing * pre-commit run -a automatic fixes * Exclude helm templates from check-yaml * Comment hooks that require installed tools * Make codespell happy * Make actionlint happy * Disable shellcheck until it can be installed properly * Make markdownlint happy * Add note about running pre-commit --------- Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> | 1 年前 |
vLLM Production Stack Tutorials
Welcome to the tutorials for vLLM Production Stack! This series of tutorials is designed to guide you through setting up and utilizing the vLLM production stack efficiently. Whether you're new to Kubernetes, Helm, or vLLM, or looking to deepen your understanding of advanced features like multi-model management and KV cache offloading, this series has you covered.
Table of Contents
-
Install Kubernetes Environment Learn how to set up a Kubernetes environment as the foundation for running vLLM Production Stack.
-
Minimal Helm Installation A step-by-step guide for deploying vLLM Production Stack using Helm with minimal configuration.
-
Basic vLLM Configuration Learn how to customize vLLM options when using vLLM Production Stack.
-
Load Model from Persistent Volume Discover how to load models from a persistent volume to ensure efficient resource usage.
-
Launch Multiple Models Learn how to deploy and manage multiple models simultaneously in your vLLM environment.
-
Offload KV Cache Understand how to offload the KV cache to CPU to improve the performance in production use cases.
Getting Started
These tutorials are designed to be followed sequentially for beginners, but you can also jump to a specific tutorial based on your needs. Each tutorial includes:
- Prerequisites
- Detailed steps
- Commands to execute
- Expected outputs
- Explanations to enhance your understanding
Feedback and Contributions
If you encounter any issues or have suggestions for improving these tutorials, feel free to contribute by opening a pull request or an issue on our GitHub repository.
Happy learning!