| feat(qwen3): bundle Qwen3 tokenizer for offline single-file/GGUF encoders (#9338) * feat(qwen3): bundle Qwen3 tokenizer for offline single-file/GGUF encoders" -m "Single-file (safetensors) and GGUF Qwen3 encoder checkpoints used by Anima (0.6B) and Z-Image (4B/8B) ship weights only — no tokenizer files. The loader pulled the tokenizer from Qwen/Qwen3-4B on HuggingFace, which fails offline / airgapped and whenever the HF cache is not persisted (e.g. Docker without a cache volume). Vendor the self-contained Qwen3 fast tokenizer (Apache-2.0, from Qwen/Qwen3-4B) in the package and load it locally, mirroring the bundled T5-XXL tokenizer (#9244). The Qwen3 BPE tokenizer is identical across the 0.6B/4B/8B variants, so a single copy serves every Qwen3 encoder. Removes the HuggingFace download path from both the checkpoint and GGUF loaders. * fix(qwen3): gzip bundled tokenizer to pass LFS check The vendored Qwen3 tokenizer.json is ~11MB, over the repo's 10MB lfs-warning threshold, failing the "lfs checks" CI job. Git LFS is unsuitable here since the file must ship inside the wheel for offline use. Vendor it gzip-compressed (~2MB) instead and decompress into a temp dir at load time. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(qwen3): fix stale tokenizer-loader comments and method name The single-file/GGUF Qwen3 loaders now use the vendored tokenizer, but the call-site comments still described the removed HuggingFace download path and the method was still named _load_tokenizer_with_offline_fallback despite having no fallback. Rename to _load_bundled_tokenizer and update the comments to match. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(qwen3): restore chat_template in bundled tokenizer config The vendored tokenizer_config.json was missing the chat_template that Qwen/Qwen3-4B ships. The Z-Image text encoder formats prompts via tokenizer.apply_chat_template(), which raises ValueError: Cannot use chat template functions because tokenizer.chat_template is not set ... so GGUF/single-file Qwen3 encoders failed at encode time. The old HF-download path pulled the full config (template included), so this was a regression introduced by bundling. Restore the exact upstream Qwen3-4B chat_template and add a regression test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> | 1 个月前 |