Added vLLM support to IndexTTS for faster inference.
| Files | Last commit | Last update |
|---|---|---|
| 1 year ago | ||
| 1 year ago | ||
| 1 year ago | ||
| 1 year ago | ||
| 1 year ago | ||
| 1 year ago | ||
| 1 year ago | ||
| 1 year ago | ||
| 1 year ago | ||
| 1 year ago | ||
| 1 year ago | ||
| 1 year ago | ||
| 1 year ago | ||
| 1 year ago | ||
| 1 year ago | ||
| 1 year ago | ||
| 1 year ago | ||
| 1 year ago | ||
| 1 year ago | ||
| 1 year ago | ||
| 1 year ago | ||
| 1 year ago | ||
| 1 year ago | ||
| 1 year ago | ||
| 1 year ago | ||
| 1 year ago |
Introduction
This project re-implements the GPT model's inference from index-tts using the vllm library, accelerating the inference process of index-tts.
Inference speed improvement (Index-TTS-v1/v1.5) on a single RTX 4090:
- RTF (Real-Time Factor) for a single request: ≈0.3 -> ≈0.1
- GPT model decode speed for a single request: ≈90 tokens/s -> ≈280 tokens/s
- Concurrency: With
gpu_memory_utilizationset to 0.25 (approx. 5GB VRAM), it can handle a concurrency of around 16 without pressure (refer tosimple_test.pyfor the benchmark script).
Update Log
-
[2025-09-22] Added support for vllm v1. Compatibility with IndexTTS2 is in progress.
-
[2025-09-28] Supported web UI inference for IndexTTS2 and organized the weight files for easier deployment! \0.0/ ; However, the current version doesn't seem to accelerate the GPT of IndexTTS2, which is under investigation.
-
[2025-09-29] Resolved the issue of ineffective GPT model inference acceleration for IndexTTS2.
-
[2025-10-09] Compatible with IndexTTS2 API calls, please refer to API; APIs for v1/1.5 and the OpenAI-compatible interfaces may still have bugs, to be fixed later.
-
[2025-10-19] Supported vllm inference for qwen0.6bemo4-merge.
TODO list
-
Concurrency optimization for V2 API: Currently, only the gpt2 model inference is parallel, while other modules run serially. The s2mel inference has a large overhead (requiring 25 DiT iterations), which significantly impacts concurrency performance.
-
Acceleration of s2mel inference.
Usage Steps
1. Clone this project
git clone https://github.com/Ksuriuri/index-tts-vllm.git
cd index-tts-vllm
2. Create and activate a conda environment
conda create -n index-tts-vllm python=3.12
conda activate index-tts-vllm
3. Install PyTorch
PyTorch version 2.8.0 is required (corresponding to vllm 0.10.2). For specific installation instructions, please refer to the PyTorch official website.
4. Install dependencies
pip install -r requirements.txt
5. Download model weights
Automatic Download (Recommended)
Download the corresponding version of the model weights to the checkpoints/ directory:
# Index-TTS
modelscope download --model kusuriuri/Index-TTS-vLLM --local_dir ./checkpoints/Index-TTS-vLLM
# IndexTTS-1.5
modelscope download --model kusuriuri/Index-TTS-1.5-vLLM --local_dir ./checkpoints/Index-TTS-1.5-vLLM
# IndexTTS-2
modelscope download --model kusuriuri/IndexTTS-2-vLLM --local_dir ./checkpoints/IndexTTS-2-vLLM
Manual Download
- ModelScope: Index-TTS | IndexTTS-1.5 | IndexTTS-2
Convert original weights yourself (Optional, not recommended)
You can use convert_hf_format.sh to convert the official weight files yourself:
bash convert_hf_format.sh /path/to/your/model_dir
6. Launch the web UI!
Run the corresponding version (the first launch may take longer due to CUDA kernel compilation for bigvgan):
# Index-TTS 1.0
python webui.py
# IndexTTS-1.5
python webui.py --version 1.5
# IndexTTS-2
python webui_v2.py
API
An API interface is encapsulated using FastAPI. Here is an example of how to start it:
# Index-TTS-1.0/1.5
python api_server.py
# IndexTTS-2
python api_server_v2.py
Startup Parameters
--model_dir: Required, path to the model weights.--host: Server IP address, defaults to0.0.0.0.--port: Server port, defaults to6006.--gpu_memory_utilization: vllm GPU memory utilization rate, defaults to0.25.
API Request Examples
- For v1/1.5, please refer to
api_example.py. - For v2, please refer to
api_example_v2.py.
OpenAI API
- Added
/audio/speechAPI path for compatibility with the OpenAI interface. - Added
/audio/voicesAPI path to get the list of voices/characters.
For details, see: createSpeech
New Features
- v1/v1.5: Supports multi-character audio mixing: You can input multiple reference audios, and the TTS output voice will be a mix of these reference audios. (Inputting multiple reference audios may lead to an unstable output voice; you can try multiple times to get a satisfactory voice and then use it as a reference audio).
Performance
Word Error Rate (WER) Results for IndexTTS and Baseline Models on the seed-test
| model | zh | en |
|---|---|---|
| Human | 1.254 | 2.143 |
| index-tts (num_beams=3) | 1.005 | 1.943 |
| index-tts (num_beams=1) | 1.107 | 2.032 |
| index-tts-vllm | 1.12 | 1.987 |
Maintains the performance of the original project.
Concurrency Test
Refer to simple_test.py. The API service must be started first.