AAlexTYJAdd streaming ASR inference
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
Add Gradio ASR demo with video support and demo audio/video files - Add gradio_asr_demo_api_video.py: Gradio web UI supporting audio/video upload, streaming output, hotwords, and Cloudflare tunnel - Add demo/asr_demo/: demo audio and video files for the Gradio interface Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> | 5 个月前 | |
add VibeVoice-Realtime | 9 个月前 | |
add experimental multi-lingual speakers | 8 个月前 | |
fix: use weights_only=True with safe_globals for voice preset loading (CWE-502) The voice preset .pt files in demo/web/app.py and demo/realtime_model_inference_from_file.py were loaded with weights_only=False, allowing arbitrary code execution if a malicious .pt file is placed in the preset directory. Switch to weights_only=True with a safe_globals context manager allowlisting only BaseModelOutputWithPast and DynamicCache (the classes actually present in the preset files). Verified that: - Existing voice presets load with identical content - Malicious .pt files are rejected with UnpicklingError - End-to-end TTS inference output is unchanged Reported-by: MSRC | 4 个月前 | |
more experimental voices | 8 个月前 | |
fix: use weights_only=True with safe_globals for voice preset loading (CWE-502) The voice preset .pt files in demo/web/app.py and demo/realtime_model_inference_from_file.py were loaded with weights_only=False, allowing arbitrary code execution if a malicious .pt file is placed in the preset directory. Switch to weights_only=True with a safe_globals context manager allowlisting only BaseModelOutputWithPast and DynamicCache (the classes actually present in the preset files). Verified that: - Existing voice presets load with identical content - Malicious .pt files are rejected with UnpicklingError - End-to-end TTS inference output is unchanged Reported-by: MSRC | 4 个月前 | |
feat(asr): add MPS/Apple Silicon support to Gradio ASR demo (#344) * feat(asr): add MPS/Apple Silicon support to Gradio ASR demo The Gradio ASR demo only detected CUDA and fell back to CPU, ignoring Apple Metal Performance Shaders entirely. On Apple Silicon Macs the demo was essentially unusable—model inference ran on CPU at extreme slowness. Changes: - Add _detect_device_and_attn() helper that auto-detects the best available device (CUDA > MPS > CPU) and picks a compatible attention implementation (flash_attention_2 for CUDA when available, sdpa otherwise). This mirrors the pattern already used in the TTS demo (realtime_model_inference_from_file.py). - Update VibeVoiceASRInference.__init__ to handle MPS device loading: load model on CPU first then move to MPS, since device_map='mps' is not supported by Accelerate. - Update initialize_model() to use float32 for MPS (and CPU), matching the file inference script's dtype selection. - Add --device CLI argument (auto|cuda|mps|xpu|cpu) with sensible auto default, and change --attn_implementation default from flash_attention_2 to auto. - Add 14 regression tests covering auto-detection, explicit device selection, MPS-unavailable fallback, attention implementation resolution, dtype selection, and CLI argument parsing. Refs #339 * chore: remove test file per reviewer request Drop tests/test_gradio_asr_device_detection.py to keep PR scope to the Gradio demo changes only. --------- Co-authored-by: voidborne-d <voidborne-d@users.noreply.github.com> Co-authored-by: d 🔹 <258577966+voidborne-d@users.noreply.github.com> | 4 个月前 | |
fix: use COMMON_AUDIO_EXTS for --audio_dir file discovery The --audio_dir flag hardcodes 6 file extensions (wav, mp3, flac, mp4, m4a, webm) while audio_utils supports 25+ formats via FFmpeg. Files like .ogg, .opus, .aac, .wma are silently ignored. Use COMMON_AUDIO_EXTS from audio_utils with case-insensitive matching, consistent with the gradio demo (vibevoice_asr_gradio_demo.py:50). | 4 个月前 | |
Add streaming ASR inference VibeVoice-ASR-Streaming transcribes while the audio is still arriving, emitting text once per audio chunk instead of waiting for the recording to end. Three pieces make that work on the transformers path: - streaming_generate / init_streaming_state / streaming_generate_step on the ASR model, so a caller can either hand over a whole tensor and iterate chunks, or drive the KV cache chunk by chunk as audio arrives. - text_chunk_end_id on the tokenizer, read from the vocabulary and left as None when absent, so non-streaming checkpoints are unaffected. - The processor now prefers a checkpoint's own tokenizer files, because <|text_chunk_end|> lives there and not in the base LM vocabulary; a language_model_pretrained_name that gets overridden is warned about rather than dropped silently. Adds a file-based inference script and an in-process FastAPI demo page, plus docs/vibevoice-asr-streaming.md. | 13 天前 | |
Add streaming ASR inference VibeVoice-ASR-Streaming transcribes while the audio is still arriving, emitting text once per audio chunk instead of waiting for the recording to end. Three pieces make that work on the transformers path: - streaming_generate / init_streaming_state / streaming_generate_step on the ASR model, so a caller can either hand over a whole tensor and iterate chunks, or drive the KV cache chunk by chunk as audio arrives. - text_chunk_end_id on the tokenizer, read from the vocabulary and left as None when absent, so non-streaming checkpoints are unaffected. - The processor now prefers a checkpoint's own tokenizer files, because <|text_chunk_end|> lives there and not in the base LM vocabulary; a language_model_pretrained_name that gets overridden is warned about rather than dropped silently. Adds a file-based inference script and an in-process FastAPI demo page, plus docs/vibevoice-asr-streaming.md. | 13 天前 | |
streaming use transformers==4.51.3 | 7 个月前 | |
more experimental voices | 8 个月前 |