| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
Make the notebooks smoke job reach the check it exists for (#10125) | 3 天前 | |
Use Unsloth for bare Studio product references (#9732) * Studio: use Unsloth for product shorthand * Tests: restore case-folding fixture path * Complete Unsloth product reference rename * Tests: restore internal Studio references | 9 天前 | |
Use Unsloth for bare Studio product references (#9732) * Studio: use Unsloth for product shorthand * Tests: restore case-folding fixture path * Complete Unsloth product reference rename * Tests: restore internal Studio references | 9 天前 | |
Studio: update in the background and restart when ready (#9890) * Studio: prepare desktop updates in the background and swap them in at restart * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep a staged update off the live install and add the missing locale keys * Studio: relocate the staged launcher, bound the idle wait and make rollback exact * Studio: stop the staged-update tests racing their own background deleters * Studio: --stage default is a truthy typer sentinel off the CLI * Studio: keep the staged update inside the gate and rename invariants the PR moved * Studio: bound the health probe and keep the rollback while the runtime is in use * Studio: gate the staged clone, reject reused pids, and keep the bundle off the heap * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: evict staged-only entries when undoing activation, and clean the bundle after single-instance * Studio: read startup markers, corroborate the bare pid, and gate the staged backend version * isolate staged python execution * match adopted stages to update versions * retain verified desktop update bundles * adopt active desktop bundle downloads * discard withdrawn staged updates * journal staged runtime activation * isolate staged native helpers * serialize posix runtime updates * stop staged update process groups * verify staged backend before confirmation * roll back failed staged backends * label classic update fallback accurately * tighten updater comments * relaunch after macos staged rollback * finalize mixed-version staged launchers * transfer backend runtime gate ownership * retry transient staged backend probes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * scrub stale backend gate handoffs * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * persist failed version before rollback completion * preserve staged launcher through verification * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * wait for staged backend warmup * keep staged helper builds isolated * preserve staged ownership after app exit * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * avoid windows updater dll locks * restore foreground git bootstrap --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Maheswar Kumar <110882203+mahiatlinux@users.noreply.github.com> | 6 天前 | |
Reduce and tighten code comments and docstrings repo-wide (#6095) Trim and tighten code comments and docstrings across the repository. Comment-only: every changed file verified code-identical to main via AST/token comparison. | 2 个月前 | |
Reduce and tighten code comments and docstrings repo-wide (#6095) Trim and tighten code comments and docstrings across the repository. Comment-only: every changed file verified code-identical to main via AST/token comparison. | 2 个月前 | |
Studio: add image generation, editing workflows and LoRA training with Unsloth GGUFs (#6763) * Tighten comments in the image stack tests and scripts * Close video single-file, training reservation, and image mount-resume gaps Route on-device single-checkpoint video folders through the single_file loader: a bare local .safetensors directory (no model_index.json) is advertised as a pipeline with no filename, so validation rejected it before it could load. Reinterpret the pick as a single_file load of the sole checkpoint, mirroring the image load route. Treat a reserved-but-not-yet-spawned LLM training start as active in is_training_active() so /images/load, /video/load, and /diffusion/start cannot race the reserved run for VRAM during the pre-spawn free window. Mirrors the diffusion training service reservation. Resume an in-flight image generation on the Images page mount: probe generate-progress, re-enter the poll loop, and refresh the gallery on completion so a run started elsewhere is reflected and its saved image appears without a manual refresh. Seed resident image defaults from the resolved base_repo rather than a possibly path-shaped repo_id so the first resident generation uses the right recipe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Publish image generation active state before pre-denoise setup generate() assigned self._gen only at the pipe() call, after deferred compile, LoRA resolution/application, and ControlNet download/build had run. Across that setup window generate_progress() reported inactive even though _generate_lock was held, so a reloaded page's mount probe showed idle and let a second generate queue behind the first. Publish an active step-0 _GenState the moment the generation lock is acquired, before the setup work, and clear it in the outer finally so a setup-time error cannot leave the UI stuck active. Mirrors the video backend's queued phase and the training start guard. * Studio: fix diffusion install ownership, dataset upload atomicity, gallery pagination, and teardown races install_sd_cpp_prebuilt: only write the .unsloth-studio-owned marker when the install created the target directory or it was empty. Adopting a pre-existing, unowned, non-empty directory (a user's own stable-diffusion.cpp checkout) made it eligible for the uninstaller's recursive delete. routes/training upload: make the multi-file promotion transactional. Back up each displaced original and roll every destination back on any failure, so a mid-loop rename error can no longer partially overwrite the live dataset. routes/training _resolve_dataset_folder: reject a symlinked dataset directory and prove the resolved folder stays under the datasets root, so image read/caption/delete cannot escape the root through a link. routes/training delete: escape glob metacharacters in the thumbnail filename so deleting an image named like [ab].png removes only its own thumbnails. image_gallery / video_gallery listing: filter records against the response schema inside the pager via a valid callback, so offset/limit/has_more all count over accepted records. A leading schema-invalid record no longer returns an empty page with has_more=true and stalls infinite scroll at offset 0. image_gallery / video_gallery save: publish via a temp file plus atomic rename (the sidecar is the video pair's commit marker) and clean up on failure, so a partial write never surfaces a truncated PNG or strands an orphan MP4. diffusion_train_common discovery: treat an empty caption sidecar as a metadata tombstone that still falls through to the dreambooth instance prompt, so clearing every metadata caption no longer fails with no captioned images found. diffusion backend unload: wait for an in-flight denoise to exit before tearing down process-wide patches and state, mirroring the load path. diffusion_engine_router: serialize the whole check/unload/publish transition so a concurrent selection cannot return the engine being unloaded. uninstall.ps1: gate the default sd.cpp process stop on the owner marker so a user's own sd-server is not terminated for a directory we then keep. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reject native batch seeds outside the JSON-safe range * Refuse sd.cpp install into unowned non-empty target dir When the install target already exists, is non-empty and lacks the .unsloth-studio-owned marker (a user's own stable-diffusion.cpp checkout, or unrelated files beside a custom Studio root), install() previously still extracted the release into it. Skipping the ownership marker only stopped the uninstaller from deleting the directory; extraction still merged binaries into the user's working tree and could overwrite same-named files. Fail up front with a clear message pointing the user at a fresh/empty location before any download or extraction, leaving their directory untouched. Update the ownership test suite to assert the refusal. * Studio: gate dataset uploads on the symlink check and surface local video single-file checkpoints * Tighten comments and docstrings added by the image-generation fixes * Studio: close arbiter load-registration race and surface native progress + local pipeline folders Publish native sd.cpp generate progress (_gen) before LoRA resolution so a reload probe reads active during setup, matching the diffusers path. Register the diffusion/video GPU load under the arbiter lock (acquire_for now takes a register callback) so a competing acquire cannot evict an owner before its load is marked in-flight and let two loaders allocate VRAM at once. Admit local diffusers pipeline folders (root model_index.json, weights in component subdirs) in the local model scan so they reach task tagging and the On Device picker. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: include marker-owned custom sd.cpp roots in the uninstall stop scan; harden Pester install against the nuget.exe PSGallery bootstrap * Studio: surface local pipeline scan roots, tag single-file checkpoints by filename, mark companion-only pipelines partial - _scan_models_dir: admit a scan folder that is itself a diffusers pipeline (root model_index.json, weights in transformer/ vae/ subdirs). _is_model_directory rejects such a root, so the child scan would list the component subdirs as bogus models and hide the real pipeline; treat the root as one model via _local_pipeline_index. - _local_is_diffusers / _local_model_task: include the sole checkpoint filename in the family-detection needles (_local_family_needles, resolved via resolve_local_single_file). A generically named folder holding one loadable qwen-image-*.safetensors / ltx-*.safetensors identifies its family only from the filename; the load route already resolves that file, so tag it or the task-scoped picker (which rejects task=null) hides the on-device model. - list_cached_models: mark a companion-only base snapshot partial. A GGUF image load prefetches the base repo's VAE / text-encoder / model_index.json but skips the transformer (the GGUF supplies it); the snapshot has a pipeline manifest yet is not a loadable BF16 pipeline, and _cached_repo_partial misses it. _repo_pipeline_missing_denoiser flags a pipeline snapshot whose transformer/ or unet/ component carries no weight, so the picker drops it instead of advertising it as fully on-device. * Studio: preserve foreign gallery files, force safetensors on remote ControlNets, and close dataset/seed/GPU gaps Gallery clear/delete now scope to Studio-owned files: image_gallery and video_gallery skip PNGs / MP4s without a readable recipe (a hand-dropped or orphan file the listing already hides), so clear() and a guessed-id delete no longer destroy files the gallery never surfaced. Remote ControlNets now force use_safetensors: a bare owner/name reaches from_pretrained without the base trust gate, and the Hub scan fails open when unavailable, so requiring safetensors closes the pickle deserialization vector. POSIX uninstall now stops resident sd-server / sd-cli under an owned sd.cpp root before removing the tree (marker-gated), mirroring the Windows stop-before-delete scan; a live native server no longer survives unlinking its binary. Diffusion dataset containment: the training-start read path and the discovery picker route bare names through the protected resolver, so a symlinked dataset is rejected / not advertised like the caption/delete routes already do. Uploads gain the inference decode guard (oversized real images 400 before OOMing the trainer) and dataset upload/caption/delete/import are blocked with 409 while a diffusion run is active. JSONL readers (trainer + routes) tolerate non-object JSON and invalid UTF-8 instead of raising AttributeError / 500. LoRA family compatibility is enforced in the shared resolver, not only the picker, so a direct API client cannot apply a mismatched-family adapter. GPU arbiter gains release_if so the image/video unload idle-check and release are atomic against a concurrent same-owner load's registration. Native batch recipes persist the base batch_seed and restore replays from it, so a native batch_index>0 image no longer advances its seed twice. FLUX.2-klein selects its sd.cpp text encoder by variant (4B -> Qwen3-4B, 9B -> Qwen3-8B) instead of the single family default. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: track the loaded GGUF filename so native companion resolution reproduces the load identity * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: gate local-pipeline image tagging on a real family; validate video sidecars before delete/clear * Studio: tighten image-generation fix comments and docstrings * Studio: gate gallery serve/export on ownership; keep image progress active until persisted; reserve diffusion training before the dataset scan * Studio: restore Reapply target on async image/video load errors; recheck training state before dataset commit; reclaim partial sd.cpp installs on retry Images/Video: a background model load that fails AFTER starting (error/eviction during download) leaves the previous pipeline resident, but handleLoad had already overwritten lastLoad.current with the failed pick, so "Reapply to loaded model" reloaded the failed model. Carry the prior Reapply target into the poller and restore it on the async error/null paths, mirroring the quant rollback. Training: an in-flight diffusion dataset upload passed _require_diffusion_dataset_mutable() at entry but could still commit files after a concurrent /diffusion/start reserved the training slot, mutating the dataset underneath the trainer. Re-check the interlock immediately before the commit phase; a 409 there leaves the staged temps for the finally to clean. sd.cpp install: an interrupted extraction (disk full, killed process, a raising post-extract cudart fetch) left the target non-empty with no owner marker, so the next lazy install tripped the "not a Studio-managed directory" refusal and wedged native install. Write the ownership marker before the partial writes when the target is reclaimable, so a retry recognises the debris as ours and re-extracts. * fp8 DiT quant: floor the dynamic activation scale with activation_value_lb An all-zero activation token row makes the dynamic per-row fp8 scale 0, which turns the quantized data to NaN and the render to black frames on torchao's plain-torch kernel path. The fused fbgemm/mslk quantize kernels clamp zero rows internally, so the bug only reproduces on machines without them, which is most user environments. Zero rows are real inputs, not a corner case: Wan 2.2 zero-pads its text conditioning, and Hunyuan-1.5 and Qwen-Image regenerate zero rows inside their transformer blocks every step. Pass activation_value_lb=1e-12 to Float8DynamicActivationFloat8WeightConfig whenever the installed torchao supports the kwarg (Float8Tensor rework, 0.13+), checked via inspect.signature so older torchao keeps exactly the current behaviour; the existing Float8MMConfig fallback chain is unchanged. Verified on GPU: with the forced plain-torch kernel path a zero-row input NaNs without the floor and stays finite with it, and end to end on HunyuanVideo-1.5 fp8 goes from a solid black frame (LPIPS 1.00) to a normal render (LPIPS 0.225); on Wan the floor matches the condition_embedder exclusion (LPIPS 0.211 vs 0.206). Same-seed renders with fused kernels present are unaffected, and pre-quantized fp8 checkpoints stay valid since weight scales are untouched. * Wire hosted pre-quantized DiT checkpoints into the image families Point prequant_repos for flux.1, flux.2-klein, flux.2-dev, qwen-image (int8 only there; fp8 is family-denied), z-image and krea-2 at the unsloth/<Model>-FP8 Hub repos carrying gate-validated int8 and fp8 transformer checkpoints, so the fast quant path loads the small pre-quantized file instead of materialising the dense bf16 transformer and quantising on device. Measured on FLUX.2-dev int8: build peak drops from 60.7 GB (dense + quantize) to 30.7 GB (hosted prequant), identical 30.7 GB resident after either path since loading a checkpoint is bit-identical to on-the-fly quantisation. The hosted repos name files <Model>-<SCHEME>.pt, so resolve_prequant_source now derives that model-name filename from the repo id (scheme suffix stripped case-insensitively) and carries the legacy transformer_<scheme>.pt as a fallback the resolver tries when the primary 404s, keeping older repos loadable. Wiring a repo also exposed a fallback hazard: with a prequant source present, the dense-fit preflight used to be skipped entirely, so a failed prequant download would fall through to the dense bf16 load the memory plan never budgeted, OOMing after eviction. The preflight now always runs and gates an allow_dense_fallback flag through _load_dense_quant_pipeline: a dense misfit still skips the fast path when no prequant exists, but with one it proceeds and a prequant failure raises to the GGUF build instead of loading dense. The same flag is set when the auto-policy replans an offloaded GGUF against a prequant-sized transient. Tests updated to the new filename convention plus new coverage for the derivation and the legacy-name fallback; the prequant-skips-refit test now asserts the re-check runs and forbids the dense fallback. Verified end to end on GPU: z-image int8 resolves the hosted repo, downloads the model-name file and renders (6.8s load, 5.9 GB peak). * Route krea-2 through its per-component loader on the transformer-quant fast path _assemble_pipe used Pipeline.from_pretrained for every family, but the krea repo ships transformers-5.x configs and no top-level tokenizer files, so the tokenizer dies with vocab_file=None. The pre-quantized checkpoint loaded fine and then the assembly crashed, dropping the load to the GGUF build, which krea-2 cannot take (Krea2Transformer2DModel has no from_single_file). Assemble per-component via load_krea2_pipeline like the pipeline-kind and single-file paths already do. Verified live: Krea-2-Turbo int8 and fp8 hosted prequant loads now assemble and render through the Studio images tab. * Keep Qwen-Image's text-stream linears bf16 on int8 (short prompts break torch._int_mm) Qwen-Image's MMDiT runs every text-stream Linear at M = actual prompt tokens: the Qwen2.5-VL embeds are not padded to a fixed length like FLUX's 512-token T5. A short prompt (13 tokens) or the near-empty negative prompt drives torch._int_mm below its M > 16 floor and the first denoise step raises 'self.size(0) needs to be greater than 16, but got 13' (measured on B200 through the Studio images tab). Add per-family int8 exclusions (txt_in, add_q/k/v_proj, to_add_out, txt_mlp) for qwen-image and qwen-image-edit, threaded through exclude_tokens_for_scheme(scheme, family) and the prequant checkpoint validation, so a checkpoint baked under the old token list is rejected and re-quantised instead of loaded crashing. The text stream runs at M = tens vs the image stream's M ~ 4k, so the exclusion costs nothing; the rebuilt hosted checkpoint gates 28/28 PASS with LPIPS mean 0.057 (was 0.069). * Harden the diffusion memory plan against transient free-VRAM undercounts A cold FLUX.2-dev int8 load on an idle 183 GB B200 planned offload=model (companions exceed budget) and silently served the GGUF as-is; the identical retry went resident and engaged the hosted prequant. The plan arithmetic was byte-identical across both loads (required 90,228 MiB, resident needs free of about 124 GB); the only divergent input was torch.cuda.mem_get_info, which is device-wide and instantaneous: a transient foreign CUDA context briefly held about 100 GB at the first snapshot, and the planner trusted that single read. Three changes: - settled_snapshot_device_memory: on cuda, synchronize + empty_cache (best-effort) and take the MAX free over up to 3 spaced reads. A transient can only shrink free, so the max rejects transient undercounts while a persistent tenant still caps every read. _plan_memory now uses it. - plan_fits_total_capacity + one replan retry: when the dense/prequant candidate fits TOTAL device capacity under the standard reserve and the 0.85 resident margin, an offload verdict can only stem from the free reading, so the loader re-snapshots and replans once before declining the fast path. Explicit balanced/low_vram modes skip the retry (they offload by mode). - diffusion.transformer_quant_declined log line with required/budget/free and the plan reasons, so the next decline is diagnosable from the server log (previously silent). Verified: cold FLUX.2-dev int8 first load in a fresh server now engages the hosted prequant resident (offload=none). * Add FLUX.2 Klein and FLUX.2-dev DiT LoRA training Register flux.2-klein and flux.2-dev in the DiT trainer following the upstream DreamBooth references: latents train patchified and batch-norm normalized from the VAE posterior mode, the packed forward reuses step-invariant position ids, and the guidance vector (3.5) is gated on the variant's guidance_embeds config. Conditioning stacks load per variant (Mistral via Flux2Pipeline for dev, Qwen3 via Flux2KleinPipeline for Klein) and are encoded and freed before the transformer lands on the device. The fused single-stream to_qkv_mlp_proj joins the attention projections in the LoRA targets; the single-stream out projection stays dense because its to_out suffix would also match the double-stream ModuleList container. Wire both families through the training registry (family set, labels, VRAM notes, rank 16 / lr 1e-4 defaults, bf16-only preflight), mark them trainable with train base repos in the family registry, add FLUX.2-dev to the gated-repo token check, and trust both official bases for training downloads. Verified on B200: 30-step klein int8 (19.6s) and nf4 (20.9s) and dev int8 (52.0s) runs train with finite decreasing loss and the saved adapters apply on the bf16 base pipeline (weight 0 reproduces the base image exactly, weight 1 visibly restyles it). * Support LoRA adapters on torchao int8/fp8 quantized image pipelines Adapters are baked at load time: they attach to the dense transformer, then quantize_ converts only the frozen base linears (the lora_ side path is excluded by name), then the loader compiles. Post-quant PEFT injection is not possible on a manually quantized module, so the prequant shortcut is skipped for a baked load and the memory plan is sized for the dense build (force_dense on the quant candidate). At generation time the baked topology is frozen: weight tweaks and disabling (scale 0 reproduces the quantized base exactly) go through set_adapters, while adding or removing adapters returns a clean 400 telling the client to reload with the new selection. supports_lora now returns True for int8/fp8 diffusers loads (checked before the gguf-kind early return, since the quant fast path keeps the picker kind); nvfp4/mxfp8 and GGUF-via-diffusers stay blocked. The load request model takes an optional loras list, threaded through begin_load on both engines (native ignores it and keeps applying LoRA at generation). Verified end to end on GPU: Z-Image GGUF picker + int8 + trained adapter loads through the API, bake marker logged, weight 1.0 vs 0 renders differ visibly, weight 0.5 accepted live, unknown adapter rejected as 400. Affected suites: 296 passed. * Add FLUX.1 Krea dev to the image model catalog Krea's guidance-distilled FLUX.1-dev finetune keeps the exact dev layout, so it runs under the existing flux.1 family unchanged. Wire it up end to end: - Catalog group with the gated official bf16 pipeline and the open QuantStack GGUF quants; the gated artifact is skipped on auto-routing when undownloaded. - Trust the official repo for non-GGUF from_pretrained loads, next to the other black-forest-labs bases. - Generation defaults: 28 steps at guidance 4.5 per the model card. The generic "krea" defaults key (Krea-2-Turbo's 8-step no-CFG recipe) used to swallow the id, which would have produced garbage output; the new flux.1-krea key precedes it on both the backend table and the images page table. - The flux.1 prequant checkpoints are schnell-based; the loader's baked base_model_id validation refuses them for the Krea-dev base, so int8/fp8 requests dense-quantize instead (covered by existing prequant tests). * Resolve pre-quantized checkpoints per base variant One family entry covers several published variants whose weights differ (flux.1: schnell, dev, Krea-dev), but prequant resolution was keyed on (family, scheme) alone, so only the default base could ever be served: the loader's baked base_model_id validation correctly refused the schnell checkpoint for dev and Krea-dev bases and every such load paid the dense download plus on-the-fly quantise. Add an optional prequant_variant_repos table on DiffusionFamily as (base_repo, scheme, repo_id) triples and thread the resolved base repo through resolve_prequant_source / usable_prequant_source and their three call sites (load fast path, memory-plan probe, auto-policy candidate). A base without its own entry keeps returning the family default, preserving the existing refuse-then-dense behavior exactly. Wire the flux.1 variants: the gate-validated unsloth/FLUX.1-dev-FP8 checkpoints (built in the earlier campaign but never reachable) and the new unsloth/FLUX.1-Krea-dev-FP8. * Add the Lumina Image 2.0 family to the image catalog Alpha-VLLM/Lumina-Image-2.0 is a 2.6B single-stream DiT with a Gemma2-2B encoder and a standard 16-channel VAE, all transformers-4.x-compatible, so the generic from_pretrained pipeline path loads it as a new lumina-2 family: - Family entry (Lumina2Pipeline / Lumina2Transformer2DModel), aliased to lumina-image-2.0 / lumina-image-2 / lumina2. No bare lumina alias: Lumina-Next checkpoints are a different arch and must stay unknown rather than crash mid-load. bf16-only upstream, so the fp16 fallback stays off like z-image. - Trust the official repo for non-GGUF loads; bf16 component table entry (ships fp32, ~5.2 GB transformer + 5.2 GB encoder bf16-resident). - Generation defaults 50 steps / guidance 4.0 per the model card, and the generate call passes the card's cfg_trunc_ratio=0.25 itself (family-gated, signature-gated): the pipeline default (1.0) runs the CFG double-forward on every step and oversaturates output. - Catalog group with the single ungated bf16 pipeline artifact (11 GB resident) plus routing assertions; images page defaults row. - No GGUF artifact: none exists upstream (only finetune/LLM quants), so the dense transformer_quant fast path (GGUF-kind-only) stays unreachable for now. Offline probes of the future prequant campaign: int8 and fp8 both engage and render cleanly (fp8 LPIPS 0.11 vs bf16, int8 0.33 from 50-step trajectory drift with intact quality), so neither scheme is family-denied. * Wire the hosted Lumina Image 2.0 int8/fp8 checkpoints Gate-validated against same-seed bf16 renders (28/28 pairs per scheme, zero failures): int8 LPIPS mean 0.146 / SSIM 0.937, fp8 LPIPS mean 0.116 / SSIM 0.946. Uploaded to unsloth/Lumina-Image-2.0-FP8 following the existing checkpoint repo conventions. * Add the HunyuanImage 2.1 family to the image backend The hunyuanvideo-community diffusers mirror carries the full stack in standard layout: a 17B dual-stream DiT (32.5 GB bf16), a Qwen2.5-VL text encoder, a ByT5 glyph encoder, the 32x HunyuanImage VAE, and guider/ocr_guider components (AdaptiveProjectedMixGuidance) that diffusers 0.39 loads natively, so the generic from_pretrained pipeline path covers everything with no per-component assembly. Family notes: - The call's guidance knob is distilled_guidance_scale (there is no guidance_scale kwarg), so cfg_kwarg routes the UI value there; real CFG runs inside the repo's guider at its baked scale. Defaults follow the card recipe: 50 steps, 3.25. - 2K-native: verified live at both 1024 and 2048. - Coexists with the HunyuanImage-3.0 structured exclusion (3.0 has no diffusers pipeline and stays excluded with its stated reason). - int8/fp8 dense quantization verified live (LPIPS 0.186 both vs same-seed bf16); a short prompt does not trip the int8 torch._int_mm minimum on this arch, so no family exclude entry is needed. - bf16 component table for the memory planner: (32.5, 16.3, 0.8) GB. * Surface HunyuanImage 2.1 in the image model catalog Catalog group with the open bf16 mirror pipeline (~50 GB resident, so a bare click on a consumer card routes to the QuantStack GGUF quants, which load and render through the generic GGUF path, verified live) plus the images page defaults (50 steps, guidance 3.25 feeding distilled_guidance_scale). * Add the HiDream-I1 family to the image backend A 17B MoE DiT (16 double + 32 single layers, 4 routed experts) with four text encoders, on HiDreamImagePipeline (diffusers 0.39). One family covers the open Full / Dev / Fast repos (same arch); per-variant generation defaults follow the upstream inference recipes (Full 50 steps at guidance 5, the distilled Dev 28 and Fast 16 guidance-free). The repos name a Llama-3.1-8B text_encoder_4 in their model_index but do not ship its weights; the official example passes the gated meta-llama repo in by hand. The loader instead assembles the component from the open unsloth mirror (byte-identical weights, already inside the non-GGUF trust gate), injected at the three pipeline from_pretrained sites, with output_hidden_states matching the official example. Memory planning counts the assembled TE4: 34.2 GB DiT + 28.8 GB encoders, ~63 GB bf16-resident. * Surface HiDream I1 in the image model catalog One catalog group with the three official bf16 pipelines (Full, plus the Dev and Fast distillations as labeled artifacts) at their ~63 GB resident size, so auto-routing keeps this a datacenter-GPU pick. city96's GGUF is deliberately not wired: the GGUF path would need the same Llama TE4 assembly for very small demand. Images-page defaults mirror the backend table with the variant keys ahead of the generic hidream key. * Pin the measured HiDream quant verdict in tests int8 and fp8 both engage and render cleanly on this family, including short prompts on int8: the routed MoE expert Linears only ever see the concatenated image+text stream (M far above the torch._int_mm minimum), so no deny entry and no family exclude tokens are warranted. Assert that so a future table edit cannot silently regress the measured behavior. * Wire the hosted HunyuanImage 2.1 int8/fp8 checkpoints Verified bit-identical to on-the-fly quantize: all 1264 state dict tensors (456 quantized) dequantize equal between the loaded checkpoint and a fresh quantize_ pass, so quality matches the runtime Dtype path exactly. Same-seed LPIPS suite means (0.35 int8 / 0.28 fp8) blend trajectory divergence with this family's own run-to-run nondeterminism (identical weights and seed reproduce a 17/255 mean pixel delta through the 50-step guider pipeline); per-case hard checks pass and the drift is compositional, reviewed visually. Uploaded to unsloth/HunyuanImage-2.1-FP8. * Fix silent LoRA drop and wasted transformer prefetch on GGUF quant loads Two live-test findings on the images load path: - transformer_quant with baked LoRAs, when the dense quantized build is declined for memory or fails: the load completed as a plain GGUF with the adapters silently dropped (HTTP success, supports_lora=false after the fact) -- wrong output with no signal. The load now fails with the recovery options (drop the adapters, free VRAM, or pick a smaller model). Weight-0 adapters still count as no bake request, and the plain no-LoRA decline keeps its silent GGUF fallback. - A fresh GGUF load on a small GPU prefetched the base repo's full bf16 transformer shards (~47 GB on Qwen-Image) because the dense-quant prefetch widening only checked scheme viability, not whether the device could ever hold the candidate resident. Gate the widening on total device capacity (reserve + 0.85 margin, the plan_fits_total_capacity bar) so a card that is certain to decline the dense build never pays the download; capable devices keep the prefetch. * Fix video progress under-reporting during load and generate Two live-test findings on the video progress endpoints: - load-progress downloaded_bytes froze mid-download: the counter used scan_cache_dir, which skips in-flight *.incomplete blobs, so it sat at the last completed blob for the whole multi-GB shard pull while the disk kept filling. Count the repo's cache directory directly (completed plus incomplete blobs, snapshot symlinks skipped so nothing is double-counted). - generate-progress reported total_steps=null / fraction=0 while step advanced: the video API only carried the native total field while the image API exposes total_steps and fraction, so one poller could not work against both. Derive the image-compatible aliases in generate_progress and declare them on the response model; the native total stays for back-compat. * Wire the hosted HiDream I1 int8/fp8 checkpoints Gate-validated: all 28 per-case pairs pass per scheme (LPIPS suite means 0.291 int8 / 0.278 fp8, in the 50-step trajectory-divergence band; CLIP delta means 0.007-0.008), and the int8 checkpoint is verified bit-identical to on-the-fly quantize across all 1615 state dict tensors (1073 quantized, max abs diff 0.0). Uploaded to unsloth/HiDream-I1-Full-FP8. * Add a pre-cast text-encoder loader for the layerwise fp8 scheme The runtime text_encoder_quant=fp8 path downloads the full bf16 text encoder and layerwise-casts it in place on every fresh load. For the heavyweight encoders (LTX's Gemma3-27B ~50 GB, FLUX.2-dev's Mistral-24B ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) that download dominates load time on a fresh machine. diffusion_te_prequant.py loads a pre-cast fp8-storage state dict instead: meta-init the encoder skeleton from the checkpoint's te_class, load_state_dict(assign=True), rebuild on CPU if non-persistent buffers stay on meta, then re-apply the same layerwise cast to install the upcast hooks. The cast is a deterministic storage transform, so the loaded encoder is bit-identical to dense-load-then-cast by construction. v1 hosts the layerwise fp8 storage scheme only: its state dict is plain tensors (torch.load(weights_only=True), no pickle execution). The dynamic-compute schemes (fp8_dynamic, int8, nvfp4) build torchao subclass wrappers at runtime and are deliberately not hosted. Checkpoints validate format, scheme, component and base_model_id before use and any problem falls back to the dense download and cast. Local path overrides reuse the DiT prequant allowlist env var. Families opt in via a new te_prequant_repos (scheme, component, repo_id) field on both DiffusionFamily and VideoFamily; the field defaults empty so nothing changes until a gate-validated artifact is wired. * Inject hosted pre-cast text encoders during pipeline assembly Wire te_prequant_pipe_kwargs into the three pipeline assembly sites: the diffusion full-pipeline branch, the diffusion transformer-only and GGUF branch (where the companion TE is the big remaining download), and the shared video assembly path before the pipeline/component split. Injection is gated exactly like the runtime cast (mode normalized to fp8, device supported, family not denied), so it can never engage where quantize_text_encoders would not; the later quantize_text_encoders call re-applies the cast idempotently and keeps status reporting truthful. With no hosted checkpoint configured the call returns {} and assembly loads the dense encoder as before. * Add the pre-cast text-encoder checkpoint builder Applies the runtime layerwise fp8 storage cast to a model's dense text encoder once and saves the cast state dict with baked metadata (format tag, base_model_id, family, scheme, component, te_class, versions) in the layout diffusion_te_prequant.py validates. Resolves the encoder class from the checkpoint's config.architectures so the recorded te_class matches what the pipeline instantiates. CPU-runnable: the cast touches storage dtypes only. * Test the pre-cast text-encoder load path Hermetic CPU coverage for diffusion_te_prequant: the checkpoint filename convention, family-table resolution by scheme and component with malformed entries skipped, resolution priority (path override, hosted repo, none) and the fp8-only scheme gate, the checkpoint validation matrix (wrong format, missing state_dict, wrong scheme, wrong component, wrong or missing base_model_id) with base case folding, the local-path allowlist refusal and missing-file fallback, and the assembly injection gating (mode, hosted entry, device support, family deny, load failure, successful injection). Also pins the te_prequant_repos field on both family dataclasses and that no family ships a hosted TE checkpoint until the campaign wires one. * Fix pre-cast TE checkpoint loading and engagement reporting Two bugs found while building the hosted checkpoints: - The builder recorded torch.__version__ (a TorchVersion object) in the checkpoint metadata, so torch.load(weights_only=True) rejected every artifact and the loader silently fell back to the dense download. Record plain strings. - Re-applying the layerwise fp8 cast to an injected pre-cast encoder raised on the duplicate hook registration, making quantize_text_encoders report the engaged cast as failed (status showed no TE quant while the encoder ran fp8). _cast_fp8 now returns early when the hooks are already installed. Also corrects the LTX TE size note: Gemma3-12B stored fp32 (~49 GB), not 27B. * Wire the hosted pre-cast fp8 text encoders qwen-image and flux.2-dev (diffusion) and ltx-2 (video) now resolve a hosted pre-cast fp8 text encoder from their unsloth -FP8 repos: - unsloth/Qwen-Image-FP8: Qwen2.5-VL-7B, 16.6 GB dense -> 8.8 GB - unsloth/FLUX.2-dev-FP8: Mistral-Small-24B, 48.0 GB dense -> 24.7 GB - unsloth/LTX-2-FP8: Gemma3-12B, 48.7 GB fp32 store -> 13.2 GB Every checkpoint verified bit-identical to dense-load-then-cast (729 / 585 / 1066 tensors, zero mismatches) and smoke-tested through the real backends with the repo engagement marker. Tests cover the wired entries, the resolver filenames, builder metadata weights_only survival, and the idempotent re-cast. * Report the compute dtype on fp8-cast encoders and inject the pre-cast TE on the dense fast path Two more findings from the hosted-TE GPU smokes: - Module.dtype reports the first floating parameter, which after the layerwise fp8 cast is the fp8 STORAGE dtype. Flux2 derives its prompt embed and latent dtypes from encoder.dtype and feeds them to randn_tensor, which has no fp8 kernel, so ANY flux.2 load with text_encoder_quant=fp8 crashed at generation (pre-existing, runtime cast included). The cast now swaps in a subclass whose dtype property reports the compute dtype; forward behaviour is unchanged. - The dense transformer_quant fast path assembles companions through _assemble_pipe, which never received the pre-cast TE injection, so the hosted encoder engaged on full-pipeline and GGUF builds but not on the fast path. Threaded through like the other two branches. Verified live on B200: qwen-image (full pipeline), flux.2-dev (GGUF picker with int8 DiT prequant), ltx-2 (video backend) all engage the hosted TE, render non-black, and report text_encoder_quant=fp8 truthfully. * Key the fp8 cast idempotency on an explicit completion marker Hook presence alone cannot distinguish a legitimately pre-cast text encoder from leftover hooks after a cast that failed mid-pass, so the early return now requires the completion marker _cast_fp8 sets once the hooks are fully installed. Leftover partial state keeps failing closed. Also tolerates non-Module encoder doubles in the hook probe and the dtype override. * Extend the fp8 TE quant to HiDream's Llama text_encoder_4 The generic quantize_text_encoders pass only covers text_encoder.._3, so HiDream's HEAVIEST encoder (Llama-3.1-8B TE4, 16.1 GB bf16) always stayed dense. TE4 is assembled separately (hidream_te4_kwargs), so the fp8 path now lives there: when the requested TE quant is layerwise fp8 and the device/family qualify, TE4 prefers the hosted pre-cast checkpoint (unsloth/HiDream-I1-Full-FP8, 8.6 GB) and falls back to dense-load-then- cast; a mid-pass cast failure reloads a fresh dense encoder instead of shipping partial state. The pre-cast loader and builder gain config_subfolder/config_overrides for standalone encoder repos whose config sits at the root and whose pipeline needs forward flags (output_hidden_states/attentions). Verified on B200: bit-identity 291 tensors (225 fp8, 0 mismatches), hosted checkpoint engages through the real backend (marker + status fp8), load 24.3 s vs 48.0 s dense, LPIPS 0.133 mean over 3 same-seed pairs vs the dense-TE render (gate 0.25), non-black frames. * Correct the ltx-2 resident TE estimate to the bf16 cast size The memory plan's bf16_components_gb held 50.4 GB for the LTX text encoder, which is the fp32 hub store of Gemma3-12B (~49 GB download), not what sits on device: the pipeline loads it torch_dtype=bf16, ~24.4 GB resident. The 26 GB over-estimate pushed the auto plan toward offload on cards that fit the real footprint. Comments and the size-table test now pin the resident semantics. * Host pre-cast fp8 text encoders for four more families Round 2 of the hosted TE set, each bit-identical to dense-load-then-cast and gated through the real backend (marker + status fp8 + same-seed LPIPS vs dense TEs): - FLUX.1 T5-XXL (text_encoder_2): 9.52 -> 5.90 GB, one artifact for schnell/dev/Krea-dev (T5 shards byte-identical across all three, verified sha256). 220 tensors, 144 fp8, LPIPS 0.109. - Lumina Gemma2-2B: fp32 hub store 10.46 -> 3.20 GB (3.3x download cut). 288 tensors, 182 fp8, LPIPS 0.041. - Z-Image Qwen3-4B: 8.04 -> 4.41 GB. 399 tensors, 252 fp8, LPIPS 0.112. NOT shared with flux.2-klein-4B: klein retrained layer 35's MLP (verified tensor diff, maxdiff 0.86), so klein hosts no entry. - Krea-2 Qwen3-VL-4B: 8.88 -> 4.83 GB. 713 tensors, 460 fp8, LPIPS 0.082. The constructor-assembled krea pipeline takes the encoder directly (load_krea2_pipeline text_encoder kwarg); the loader remaps 5.x rope_parameters and re-ties weights after assign so the rebuilt encoder matches the builder's structure. HunyuanImage 2.1 reuses the Qwen-Image artifact outright: its Qwen2.5-VL text encoder is byte-identical (every shard sha256, 16,584,414,544 bytes), recorded in the new component-level base-equivalence table the checkpoint validator consults. The injection loop now covers text_encoder.._3 so a family can host several components. Live check: LPIPS 0.123 vs dense. * Report the fp8-cast compute dtype without swapping the encoder class The dtype override swapped encoder.__class__ to a dynamic subclass, which breaks transformers' kwargs-based output recording: a fp8-cast Qwen3VLModel stopped returning hidden_states and every krea-2 generation with text_encoder_quant=fp8 crashed at encode_prompt (regression from the HiDream TE4 change; caught by the krea hosted-TE live smoke). The override is now a property shadowed on the ORIGINAL class that prefers a per-instance compute-dtype attribute, so class identity is preserved and uncast instances keep the stock behaviour. The idempotency test now pins exact class identity and the uncast-sibling fallback. * Pass the calibrated distilled sigma curve to LTX-2.3 8-step runs The 22B distilled DiT was trained against ltx_core's fixed DISTILLED_SIGMA_VALUES, but the diffusers scheduler derives 8-step spacing from resolution-shifted flow matching and lands far off at every reachable mu (second sigma 0.945-0.981 vs 0.99375, tail 0.37-0.61 -> 0.1 vs 0.725 -> 0.42 -> 0). At the distilled default step count the backend now passes the list verbatim, neutralising the scheduler's dynamic shift and terminal stretch for the call (they distort even explicit sigmas) and restoring them afterwards. Other step counts and the dev/base DiT keep the scheduler's own spacing. Live-verified on B200 through the video branch backend: the scheduler holds the exact curve after an 8-step distilled GGUF generation, config restored, healthy clip. Also reword the transformer_quant resolved reason to the measured reality: quant halves resident weights and hosted checkpoints cut load time, while per-step speed is roughly bf16 parity. * Pin the fp8 weight-quantize kernel against silent MSLK switching torchao's Float8Tensor KernelPreference defaults to AUTO, which switches the weight-quantize kernel to MSLK whenever an mslk package is importable on sm90+. Measured on B200: that changes fp8 scale rounding bitwise (8/8 FLUX matrices differ, scales ~55 percent of bytes), so a box that merely gains mslk would break the hosted-prequant bit-identity invariant; the mslk path is also slower under torch.compile (opaque extern call blocks inductor's quantize fusion, FLUX.1 fp8 e2e 1.149 to 1.624 s). Pin KernelPreference.TORCH explicitly, matching current no-mslk behaviour bit for bit; signature-gated for older torchao. GPU-smoked (finite, rel err 0.037) and pinned by test. * Shift Qwen-Image training sigmas to the inference distribution Qwen-Image's scheduler skips its static shift under use_dynamic_shifting, so the DiT trainer was drawing UNSHIFTED uniform-schedule sigmas for it (mean sigma 0.50) while inference always runs the exponential mu = log 3 shift plus the shift_terminal 0.02 stretch. Add a flow_shift config lever: "auto" (the new qwen-image default) rebuilds the training sigma table through the scheduler's own time_shift and stretch_shift_to_terminal so the draw matches the inference distribution exactly (mean sigma 0.72); a numeric value applies the standard linear shift s*u/(1+(s-1)*u); 1.0 keeps the historical identity behavior and stays the default for FLUX, Z-Image and Krea 2. The model timestep conditioning follows the shifted sigma, gathered in fp32 so bf16 rounding never skews it. Also wire two opt-in levers with off defaults: cfg_dropout (per-sample empty-prompt conditioning dropout, encoded alongside the captions before the text encoders are freed) and weighting_scheme="bell" (bsmntw-style mid-schedule Gaussian loss weighting normalized to mean 1). Verified with two 80-step rank-8 bf16 LoRA runs on Qwen/Qwen-Image (identity vs auto, same seed): both converge with finite decreasing loss and produce coherent same-seed previews. Unit tests cover the exact transform, the shifted sampling distribution, per-family defaults and config plumbing. * Add LoRA EMA, a persistent conditioning cache, and aspect bucketing helpers diffusion_train_extras hosts the opt-in training extras: LoRAEMA shadows only the trainable adapter params (warmup-ramped decay, default 0.99, exported as a second adapter under output_dir/ema), PersistentConditioningCache stores latent posterior stats and caption embeddings as safetensors keyed by content hash + family + resolution, and the aspect-ratio bucketing helpers group mixed-aspect datasets into same-area divisor-snapped shapes. The DiT trainer wires the first two behind config flags that default to the current behavior: ema_decay (0 disables) and cond_cache_dir (None disables). A fully warm cache skips loading the VAE and text encoders entirely; a cache hit is bit-identical to a fresh encode, including the per-channel qwen latent normalization. Also fixes the stale _gather_sigmas call in the perf test that still passed the scheduler instead of the sigma table. * Tighten torchao configs and note the FSDP2 design for the DiT trainer nf4 loads now enable double quantization (~0.4 bits/param off the frozen base scales at no fidelity cost), fp8 training uses the rowwise recipe when the torchao build ships it (per-row scaling confines the DiT activation outliers that a tensor-wide scale collapses), and the inference quant filter gains a per-scheme GEMM-tiling divisibility floor (16 for scaled_mm, 32 for MX blocks) so one ragged Linear cannot crash the first denoise after a clean quantize pass. plans/fsdp2_diffusion_design.md records the multi-GPU design: bf16/fp8 over FSDP2 with per-block units, LoRA attached before sharding, int8 out of scope (DTensor over the quantized subclass is undefined), per-family notes. * Batch diffusion inference with per-image seeds, an inference conditioning cache, and GGUF loader fixes Batched generation: /images/generate takes a prompts list (one image per prompt, txt2img only) or a seeds list (one prompt, one image per seed); the legacy batch_size path derives per-image seeds base..base+n-1 like the native engine. Every image gets its own torch.Generator so any batch member replays alone from its gallery recipe; the whole list runs as one forward by default with OOM backoff that halves a failed chunk, and an explicit batch_size caps images per forward. Validated 10-22x over serial engines on 32-image suites with LPIPS deltas within 0.002. Conditioning cache on the inference path: UNSLOTH_DIFFUSION_COND_CACHE_DIR (the inference sibling of the trainers' cond_cache_dir, same persistent store) wraps encode_prompt so repeated prompts skip the text-encoder forward entirely; verified bit-identical outputs. Bypassed while LoRA adapters are attached; tensor-argument calls pass through uncached. Compile cache: GGUF loads fingerprint their own bundles (quant=gguf, a different compiled graph than the dense family) and batched calls register every distinct (w, h, batch) chunk shape they ran, so the heavy GGUF batched warmups (~159 s at batch 32 on 12B-class, ~655 s on 20B CFG-batched) are paid once ever. GGUF loader: strip the sd.cpp model.diffusion_model. container prefix in the single-file converter; diffusers' FLUX.2 converter KeyErrors on it and the Qwen-Image identity mapping strands the model on meta. * Correct batched seed-replay docs to match measured behavior Same-seed images at the same batch shape are bit-identical; a solo regeneration with the recorded seed matches its batched rendition up to batch-size-dependent kernel numerics (mean abs pixel delta about 2.5/255, LPIPS delta under 0.002), not bit-exactly. The previous wording overclaimed bit-identity across batch shapes. * Note that batched bit-identity assumes a settled compiled graph The first generation issued while the deferred compile is still in flight can deviate transiently (observed once on a cold fp8 build: mean abs pixel delta 0.063/255); once the graph is settled, same-seed same-batch-shape images are bit-identical across runs. * Studio sidebar: Image03/FlimSlate icons, More flyout, Train row, New pills - Images uses Image03Icon and Video uses FlimSlateIcon. - New "More" row (MoreHorizontalIcon) opens a right-side flyout on click or hover holding Video, Recipes and Export; the close is delayed 180ms so the pointer can cross the gap. Its SidebarMenuButton deliberately takes `title` rather than `tooltip`: with `tooltip` the button returns a Tooltip root and DropdownMenuTrigger asChild would hand its ref to a non-DOM node. - Dropped the "Train" section heading; Train is now a top-level row between Images and More. data-tour="navbar" moves to the surviving nav group so the product tour keeps its anchor. - "New" pill beside Images and (inside the flyout) Video, via NavBadge. * Studio sidebar: match flyout rows and New pills to the existing scales - More flyout rows dropped their sidebar-row typography and size-icon override, which fought DropdownMenuItem's own scale (text-sm, gap-2.5, px-3 py-2 and size-4 icons) and rendered oversized glyphs and text next to the nav. - New pill reuses the brand "beta" badge recipe (nav-badge font, --ui-font-scale sizing, nav token colours) rather than hardcoded 9px values. - The More row's native title tooltip (an OS box on hover) is replaced by the app's Tooltip, wrapped around DropdownMenuTrigger so both triggers compose onto the same button, and shown only on the collapsed rail like other nav rows. * Settings: pin and reorder the sidebar navigation Adds a "Sidebar navigation" section to Settings -> Appearance, above the existing profile-menu customizer, with the same drag-to-reorder + switch UI. - New sidebarNav preference: one { id, pinned } entry per navigable row (projects, hub, images, train, video, recipes, export), array order = render order. Defaults match the shipped layout, so an untouched install is unchanged. - Unpinning moves a row into the More flyout rather than hiding it, so no page becomes unreachable. New chat and Search stay fixed as actions. - app-sidebar now renders from one navRows descriptor map, so a pinned row and its flyout counterpart cannot drift; the More row appears only when something is unpinned and highlights off whatever it actually holds. - Mirrored in the backend PersonalizationCustomization: without it the model's extra="ignore" would drop the field, and because sync replaces local state with the server's copy once customization is saved, the user's pin order would reset on the next sync. The validator dedupes and back-fills like sidebarMenu but preserves the client's order, since here order is meaningful. Frontend typecheck, i18n parity and catalog checks pass; 32 personalization tests pass, including a round-trip asserting a reordered list survives a save. * Sidebar customizer: drop the Search row, skip More for a lone item - Search is reached from the top bar, so it is no longer previewed as a fixed sidebar nav row; New chat stays. - More now appears only when it would hold two or more rows. A single unpinned row renders inline in its saved order position instead: a flyout wrapping one item costs a click and earns nothing. The customizer's More preview follows the same threshold. * Sidebar settings: hide a lone unpinned tab, match New chat icon, rename Profile menu - With exactly one tab unpinned, both More and that tab are dropped, so nothing is drawn for it (previously it rendered inline). The page stays reachable by URL. - The customizer's New chat preview uses PencilEdit02Icon, the icon the real row renders; Edit03Icon was a different glyph. - "Sidebar menu" is now "Profile menu", described as the shortcuts behind your name at the bottom of the sidebar, so it no longer reads as a second name for the navigation section above it. * Tighten comments in the new sidebar and delete-guard code * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio sidebar: keep the More row highlighted while its panel is open Moving the pointer into the flyout left the row unhighlighted while the panel stayed open. The row now carries data-menu-open, added to the nav hover selector list. Not data-state: the tooltip and menu triggers both write that attribute, so whichever lands last wins. * Images: use the shared pill toggle for Create/Train and pad the panels - Create/Train was the only segmented control on its own Tabs styling. It now uses PillTabs, the same control as the model picker and Hub toggles, pinned to the header row's 34px. PillTabs takes an icon per tab, so the inline-span workaround for TabsTrigger goes away. - pt-3 on both the Create and Train panels, which sat flush against the model selector row. * Images: make the workflow picker a dropdown instead of a 7-up strip Seven workflows in a 340px rail left ~48px each, so the labels crowded and the hints were only reachable as title tooltips. The strip is now a dropdown: the trigger shows the current workflow and its hint, and each row carries its own description. A row the loaded model can't run is disabled and shows the reason in place of the hint, so the gating explains itself. Adding a workflow no longer shrinks the others. * Images: workflow icons, hint under the trigger, more top room, unclipped Train cards - Each workflow carries an icon, shown on the closed trigger and on every row. - The trigger is one line (icon plus name). The selected workflow's description moved below it, where it reads like the Field hints further down the rail. - pt-6 instead of pt-3 on both Create and Train, so the cards clear the model selector row. - The Train right column scrolls while its cards use ring-1, which draws outside the box and was clipped at the scroll edges. p-px gives the ring room. * Images: one-line workflow rows, stronger trigger fill, roomier mode tabs - Dropdown rows are icon plus name only. The selected row's description already shows under the trigger, and a disabled row keeps its reason as a title. - Trigger fill moves to the bg-foreground/[0.07] dark:bg-foreground/[0.12] pair the hub cards use, so it reads against the card in both modes. - Description under the trigger goes from text-ui-10 to text-ui-11p5. - More horizontal padding on the Create / Train tabs. * Images: drop card borders for the composer shadow, keep scrollbars inside, use app controls in Train - Cards lose ring-1 for .panel-soft-surface: the composer's shadow in light, flat in dark, matching .chat-composer-surface and the menus. - Both rails now clip (overflow-hidden) with the scroller inside, so the scrollbar can't ride over the rounded corner. Same shape video-page already uses. - Train's 9 native selects become the app Select, so they no longer open an OS-native menu, and the native file input is hidden behind a Choose images button that reports the count. - Image previews use explicit 8-10px radii: this theme sets --radius to 1.1rem, so rounded-md was 15.6px and the thumbnails read as circles. * Images: one card for controls and preview, chat sliders, wider softer shadow - Controls and preview were two floating cards; they now share one card split by a divider. The Advanced dock stays separate since it toggles. - SliderField wraps Chat's ParamSlider, so the sliders match Chat (label row with the value, full-width neutral track) instead of a green track with a spin box. All 14 call sites keep their props. - panel-soft-surface goes from 0 2px 8px -2px /0.16 to 0 4px 22px -6px /0.10: lighter, spread wider. * Images: flat Create and Train panes, hover-only scrollbars, tidier Train dataset step Both Images tabs now sit on the page background like the Hub: no card, no shadow, no bounding box. A single rule divides the controls rail from the preview canvas (Create) and from the run area (Train), and the settings and previous-runs sections read as panes rather than nested cards. Also: - Scrollbars in these panes use the existing hover-scrollbar recipe, so the thumb only shows while the pane is hovered. - Workflow rows explain themselves with a tooltip after a short hover, which also works on disabled rows, and the descriptions are much shorter. - Training images rows are name plus image count; the license stays on the example card. - The upload step loses its dashed box, the buttons match the sizes around them, and Upload only appears once files are picked. - The empty preview uses the same icon as the Images nav item. * Images: full-height panes, wider settings rail, Create/Train offset from the selector The rule between the panes now runs the whole page height (the row drops its bottom padding and each pane pads its own content), the settings rail is wider on both Create and Train, and the Create/Train switch sits further right of the model selector. * Images: put both tabs on the Hub's centered measure Top bar and content now share mx-auto max-w-1100 with px-5 / sm:px-8, so Create and Train sit at the same width and position as the Hub instead of running edge to edge. * Images: restore the top bar position, drop the panes lower under it * Images: center the mode switch, flip the arrow with the orientation, app tooltips everywhere The Create/Train switch is centered on the page instead of trailing the model selector, with wider buttons. The flip control's arrows now rotate with the orientation and its label says which way the flip goes. Every native title tooltip on the page is now the app's tooltip, so they all get the rounded surface instead of the OS box. * Images Train: plainer field text, no green buttons, columns that stop colliding - The dataset name, trigger prompt, adapter name and custom base fields now say what they are in plain words instead of leaning on example values. - Import, Upload, Back, Back to settings and Train another are outline buttons, not green ones. - Example thumbnails are landscape tiles, so photos are not cropped to chunky squares. - Settings cells get min-w-0 and the select value truncates, so a long option like the nf4 label no longer widens its column into the next one. - The number stepper sits a little further in from the field edge. - Create and Train are wider. * Images Train: roomier example cards with Import on the thumbnail row * Video: same treatment as the Images tabs - No cards: the rail and the canvas sit on the page background, divided by a rule that runs the full page height, on the Hub's centered measure. - Wider rail, chat's sliders, hover-only scrollbars. - Every native title tooltip is now the app's tooltip, including the clip cards. - Reapply and Cancel are outline buttons, the empty state uses the Video nav icon, and the clip tiles are less rounded. * Images and Video: narrower generation rail, matching Train headings Create and Video rails go from 392px to 368px. Train a LoRA and Training settings are now the same size and both in the heading font: the h2 already picks it up from the base rule, so the settings header opts in with font-heading and the weight that rule pins. * Images Train: shorter copy throughout Family notes, example descriptions, precision labels and every helper line are trimmed so they stop wrapping to three lines and colliding with the next column. The nf4 label now fits its select without truncating. * Images Train: a little more spacing between field groups * Images and Video: tighten code comments * Fix training start NameError, the load-order guard test and CPU-only diffusion tests - start_training forwards resume_source_run_id to _start_training_impl, which reads it. Without it every start raised NameError. - Restore main's anchor in the load-marker order test: the file now has an earlier `if config.is_gguf:`, so indexing the first one compared the wrong branch. - The two diffusion tests that reach diffusers now skip when it is absent, matching the CPU repo-test env. - The UI smoke finds nav rows that live in the sidebar's More flyout. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Treat a null metadata caption as no caption str(None) stored the literal "None" as the caption, so a null row counted as captioned and would have trained on that text. Also drops an unused import. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix invalid-UTF-8 500s, the flat Canny map and the dropped DiT knobs read_text raises UnicodeDecodeError, which is not an OSError, so one bad caption sidecar or video sidecar 500d the info, upload and gallery routes. A flat image now yields the all-black edge map instead of its own luminance, and the four DiT loss knobs the trainer implements are declared so model_dump keeps them. * Use the ui font-size tokens instead of raw px text utilities text-[11px] and friends ignore the UI font size preference, which the repo's font-scale contract test enforces. Same rendered size at the default scale. * Fix diffusion dataset 500s, the dropout-1.0 no-op run and the reset base pick Four correctness fixes on the training side: - The labeling grid read caption sidecars under except OSError, but a non-UTF-8 sidecar raises UnicodeDecodeError (a ValueError), so one bad file 500d /diffusion/dataset/{name}/images and the grid could not be opened to repair it. Read it as no caption, matching the info summary. - An image past Pillow's own hard limit raises DecompressionBombError, which derives straight from Exception and so escaped the upload guard's (OSError, UnidentifiedImageError, ValueError) and returned 500 instead of the intended 400. - lora_dropout accepted 1.0, which makes PEFT build nn.Dropout(p=1.0): lora_A and lora_B receive no gradient and the run saves an untrained adapter while reporting normal progress. Bound it below 1.0, matching the LLM request schema. - The train panel re-seeded the base repo on every dataset refresh because the family object identity changes on each info fetch, so an upload or caption save silently replaced the user's chosen base and the run started on a different model. Track the pick and only re-seed on a real family change. * Show the retained failure when a video page mounts after a failed job Mount-time recovery handled only phase=completed, so reloading the page after a multi-minute generation failed left an idle view with no diagnosis: the backend keeps the terminal failed record only until the next job, and nothing else survives the reload. Surface it the same way the poll does, filtering the cancelled sentinel. * Fix batched generation crashes, cache keying and unreplayable recipes Four bugs in the batched inference path, all found by review: - A mixed-prompt batch sent a scalar negative prompt against a prompt list. Z-Image asserts on the length, and Qwen-Image, Krea 2 and FLUX true-CFG encode a batch-1 negative against batch-N latents and fail in the transformer's text/image concat. Broadcast it to match the batch. - The FBCache step-cache reset sat above the chunk loop. diffusers only resets that state at the end of a successful call, so a forward that raised (the OOM the backoff is meant to recover) left its own residual behind and the halved retry died on a shape mismatch. Reset before every forward instead. - The conditioning cache keyed on the checkpoint alone, but a GGUF or single-file load takes its text encoders from the companion base, so the same checkpoint against a different base reused the previous base's embeddings. Key the base too. - Gallery records stored the base seed and the requested batch size even when a prompts/seeds list drove the run, so restoring the second image of seeds=[5, 99] replayed seed 5. List-driven outputs now record as single-image recipes on their own seed. Also bound strength above 0: every img2img pipeline derives its step count from it, so 0 leaves zero denoising steps and either raises or, on SDXL, crashes on empty latents. * Fix quantized-load LoRA bake, prequant family exclusions and outpaint canvas Six review findings across the Images page and model scanning: - The quantized (int8/fp8) load path can only attach LoRA adapters before quantization, but the frontend load request had no loras field, so every generation after such a load was rejected and each reload repeated it. Send the selection with the load. - build_prequant_checkpoint passed no family to the scheme exclusions while recording the family in metadata, so a Qwen int8 artifact baked the short-M text-stream linears and was then rejected wholesale by the loader's family-keyed check. - Registering a bare single-file checkpoint directory produced no On Device row even though the images loader can load it; only its parent worked. Admit that shape when nothing else matched. - Unload left the Reapply target set, so the repair path was skipped and Reapply reloaded the ejected model. Clear it, as the video page does. - Both FLUX.2 bases were trusted for training but not inference, so Deploy to Create rejected every FLUX.2 adapter. - Outpaint allocated the grown canvas before downscaling, exceeding the browser canvas area cap on a large photo; an over-cap canvas is unusable, so Extend silently posted a fully transparent image and mask. Scale the source first. * Send the picked GGUF filename with the quant so diffusion loads fire The variant expander emitted only the quant label, and nothing else in the frontend set ggufFilename, so the Images and Video pages could never take their GGUF branch: both gate it on meta.ggufVariant and meta.ggufFilename, then fall through to the single-file path, which returns because the id is a repo id and not a .gguf name. Every quant pick was a silent dead click, with no load request reaching the backend. The filename was already on the variant row (the picker keys its list on it, and the variant validator requires a non-empty string), so thread it through the click handler. The chat path is unaffected: it reads ggufVariant and never needed the filename. * Version the conditioning cache key and reject non-finite flow_shift Two correctness fixes: - The cache keyed the checkpoint and its companion base by name only, so a Hub repo advancing to a new commit, or a local directory updated in place, kept returning embeddings from the previous text encoder. Pair both with a revision marker: the locally resolved commit sha for a Hub repo, config plus text-encoder file stats for a directory. Neither loads the encoders, so a warm run still keeps them off the GPU. - flow_shift only checked positivity, but JSON accepts 1e309, which floats to inf, and inf <= 0 is False while NaN fails every comparison. The sigma table then evaluates s * u / (1 + (s - 1) * u) as NaN, which poisons every sampled sigma and saves a corrupted adapter while progress looks normal. Require a finite value. * Keep curated models listed, guard the video companion repo, pin diffusers Three review findings: - The picker filtered every catalog member out of Recommended and Hub search on the way to canonical group rows, but nothing renders those rows yet (catalogGroupFitsDevice and groupMatchesQuery are imported and unused). A task-scoped picker's models list is catalogToModelOptions(), i.e. group members exclusively, so both lists came back empty and no curated model could be discovered or downloaded. Keep the artifacts listed until the grouped UI exists. - The video delete guard compared only repo_id, so deleting the companion base of a loaded GGUF video model was allowed even though it supplies the VAE and text encoders. Compare base_repo too, matching what the images guard already does for its companions. - diffusers was declared unversioned while the diffusion stack requires 0.39 (Krea2Pipeline, the cache_context child registries, the Flux2 and Z-Image pipelines), so an upgrade could keep an older release and selecting an advertised model failed until the user upgraded by hand. * Namespace the trainer conditioning cache per checkpoint, bound the learning rate - The trainer keyed its persistent conditioning cache on family and resolution only, while the keys themselves carry just the caption or image content and crop variant. One cache directory reused for two checkpoints, or for the same repo at a new revision, let a warm run skip loading its encoders and train on the other model's embeddings and latent statistics. Namespace on the base checkpoint and its resolved revision as well. The revision helper now lives beside the cache in diffusion_train_extras and the inference wrapper delegates to it, so the two cannot disagree about what counts as the same source. - The diffusion learning rate only checked positivity, but 1e309 floats to inf and satisfies gt, so the route evicted the resident models and started AdamW with an infinite rate: the first step destroys the adapter while progress looks normal and the result is saved. Bound it below 1.0, matching the LLM schema, which rejects inf for the same reason. * Fix GGUF image model picks doing nothing, and pick the train base in the top bar The quant rows never forwarded the .gguf filename, so every hub GGUF pick on Images/Video fell through to a silent return. On Train the top bar now picks the training base instead of a generation model, which is GGUF-only and untrainable. * Pin diffusion and video loads to the live HF cache root Both read huggingface_hub's import-time HF_HUB_CACHE, which changing the cache folder does not update: progress counted the old root while the download wrote to the new one, and from_pretrained could split one model across both. * Add the diffusion download plan endpoint Reports the repos and exact files a pick needs so the download manager can stage them with the loader's own file scope. A plain snapshot would add the packaged root single, transformer shards and fp16 twins the loader never opens. * Add a file-scoped flavour to the Hub download job Lets a consumer that reads a deliberate subset of a repo stage it through the normal download manager. Keyed as "@scope" so it never collides with a quant or with the repo's full snapshot, and the file list rides the registry so an XET to HTTP retry respawns the same scoped job. * Stage image and video downloads through the Hub download manager They downloaded inline inside the load, so they had none of the manager's disk preflight, manifest verification, resume or panel progress. Picks now stage as scoped jobs carrying the loader's own file list, then load from a warm cache. * Fetch staged GGUF checkpoints as scoped jobs, and stop calling diffusion models unsupported A GGUF entry went out as a full snapshot, whose ignore list drops *.gguf: the job finished at once having fetched only docs, and the repo landed on device unloadable. Every entry is scoped now. The Hub also no longer tags image/video models as unsupported (they run on their own pages), and those pickers name what they select. * Apply the picker task filter to local model sections LM Studio, ./models and custom-folder rows ignored it, so the Images picker listed chat GGUFs that 400 on a diffusion load. The backend already tags every local model with a task for this purpose. * Route a chat pick of a diffusion model to the Images or Video page Chat cannot load one, so it was either hidden or failed on load. The unfiltered picker now lists on-device diffusion models and navigates to the page that runs them, passing the repo and quant so that page loads it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Route the real GGUF filename, keep non-GGUF curated models, key scoped downloads by file set Five review findings, four of them ways a click did nothing or fetched the wrong thing: - A chat pick of a diffusion model routed ggufVariant (a label like Q4_K_M) in the search param the target page uses verbatim as the GGUF filename, so the load asked for a file that does not exist. Route ggufFilename; no filename means a curated non-GGUF pick, loaded as a pipeline. - The task-scoped pickers kept only GGUF repos, so the catalog's bf16, bnb-4bit and single-file fp8 artifacts could not be discovered or downloaded on the Images and Video pages even though loadSpecFor knows how to load them. Keep curated artifacts whatever their format, in Recommended and in Hub search. - Both pages deduplicated routed selections on the model alone, and they now stay mounted, so picking the same repo again -- another quant, or the same one after chat evicted it -- returned early without loading or clearing the query string. Key on model and quant. - Every scoped image download shared one @diffusion job key regardless of the requested files, so switching quant mid-download adopted the running job: the UI waited on the first file set, then loaded a file that was never fetched. Include a digest of the file set in the key. - A scoped plan silently dropped requested files missing from Hub metadata, and snapshot_download succeeds when an allow pattern matches nothing, so the job reported completion and triggered a load with required files absent. Fail the job instead. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the scoped download key derivable, and stop the hidden page hijacking a route Four review findings, the first a regression from my own last commit: - Keying scoped download jobs by a digest of the file set broke the download manager: it builds that key client-side (it polls and cancels before any response tells it a key), so it watched and cancelled a key no worker owned and never fired its ready callback. Keep the derivable "@scope" key and refuse the second request instead when a live job on the slot is fetching a different file set -- decided inside the registry claim, under the lock, so a concurrent claim cannot slip past it. The manager records the file set on the job as well, so a sibling quant's transfer is not adopted locally either. - Both diffusion pages read the route query through a loose useSearch and both stay mounted once visited, so the hidden one consumed the other's ?model=: it navigated back to its own route and tried to load, say, an image checkpoint as a video model. Only the visible page consumes it. - The staged download plan was built without the configured HF token or the Advanced values the load itself sends. The token matters most: the backend's Hub metadata lookup is best-effort, so a gated base silently planned no companion entry and the load pulled those multi-GB files inline, outside the manager. The memory/quant controls decide whether the base transformer/ shards are needed at all, and the route dropped memory_mode, cpu_offload, the prequant path and the LoRA selection before asking for the plan. - The video preview kept playing after leaving the page: the keep-alive layout only hides it, and display:none does not pause a media element, so a clip the user unmuted kept its audio going over the next page. Pause on the active transition and do not auto-replay while hidden. Also completes the hand-built request bodies in the hub download tests: the scoped-files field this branch added to the route read as an AttributeError against them, failing five tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Serialize the GPU handoffs, gate DiT training on a GPU, and keep 3.9 installable Six review findings, three of them evict-then-fail orderings: - The chat load reclaimed the GPU without telling the arbiter it existed. A chat load holds no llama-server process until its GGUF has downloaded, which is minutes, so a competing Images/Video acquire in that window found nothing to cancel, took the GPU, and the chat load then spawned onto the same device. It now registers an in-flight marker through acquire_for's register hook (under the arbiter lock, as the image and video loads do), the evictor cancels a marked load, and the route undoes itself if ownership moved while it loaded. - The Hub-download conflict check ran after that handoff, so a GGUF the download manager already owns destroyed the resident Images/Video pipeline and then 409'd, having loaded nothing. It moves above the handoff, together with the marker it handshakes with. - The image load released the engine router's transition lock before registering the load, so a second load choosing the other engine could unload the still-idle engine this one captured; the load then landed on a deactivated engine, where generate, status, unload and the arbiter's evictor can no longer reach it. Registration now happens under that lock and refuses if the engine changed. - Training a DiT family on a host with no GPU was accepted: nf4 is not a CPU fallback, its 4-bit load goes through bitsandbytes, which requires CUDA, XPU or MPS. The start unloaded the working Images pipeline, pulled the text encoders, and only then died in the child. Rejected before the teardown now, and /info stops advertising a precision that always 400s. SDXL keeps its documented fp32-on-CPU path. - Both diffusion pages kept the routed-pick marker forever, so re-picking the same checkpoint (after chat evicted it) neither loaded nor cleared the query string. The marker is released once the query is gone. The Images key also carried a stray NUL byte, which made the file read as binary to grep and other tooling. - diffusers dropped Python 3.9 in 0.38, so the unconditional >=0.39.0 pin left pip no candidate at all on 3.9 and made every install that composes the huggingface extras unresolvable there. The floor is conditional now. Also fixes tests that were already red on the branch: two hand-built request fakes had gone stale against fields this branch added, and the handoff-ordering test only failed on a host with fewer than two GPUs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the GGUF variant contract test against the merged handler signature The assertion pinned the exact single-line call handleVariantClick(v.quant, v.downloaded, expectedBytes, v.filename), but the handler takes (quant, filename, downloaded, sizeBytes) and prettier wraps the call across lines, so the mandatory repository test job failed on every push. Match the call structurally and assert the filename really is forwarded in the handler's argument order. * Stop a background page and a stale record taking the GPU or a download with them Five fixes from a review pass over the diffusion work. delete-finetuned rmtree'd a model the Images or Video engine was holding: every guard on that route is chat-only, and Images loads any local path, so deleting a local diffusion model under the storage root pulled the weights (and the companion VAE / text encoders sd.cpp re-reads each generation) out from under a live pipeline. The cached-model route already refuses this; the trained/exported one now does too, matching by path rather than repo id, and failing open on a chat-only install so it cannot block ordinary deletes. A staged download finishing while its page was hidden loaded the model and evicted whatever the user was actually using: both diffusion pages stay mounted behind the router and a load takes the GPU unconditionally. The pick is now held until its page is on screen again, which is also what chat does. A scoped download could report success having fetched nothing. With Hugging Face metadata unavailable no manifest is written, so verification is a no-op, and snapshot_download returns an existing snapshot folder without downloading when its own repo_info call fails. A repo already on disk from a full snapshot job (which ignores *.gguf) therefore completed with no weights and auto-loaded against them. The requested file list needs no network, so it is checked against the disk directly. The XET to HTTP retry reclaimed the job slot without the scoped file list, and that claim overwrites the stored record, so a later identical scoped start compared an empty list against the real one and 409'd instead of adopting the running download. The DiT accelerator gate probed torch.mps.is_available(), which only exists from torch 2.5 while the supported floor is 2.4. All three probes shared one try/except, so on torch 2.4 the AttributeError read as 'no block' and a CPU-only host still evicted the resident pipeline, downloaded the encoders and died in the child. Each accelerator is probed on its own now, through torch.backends.mps. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stage what an LTX-2.3 load reads, keep a routed file's load kind, drop an unbakeable LoRA Three from the latest review. The video download plan always asked for the wide base file list, so an LTX-2.3 pick staged the 2.0 base's VAEs, vocoder and connectors that the checkpoint supplies itself, while the companion files the 2.3 assembly does read were left out of the plan and pulled inline at load, outside the panel's progress, cancel and disk preflight. The plan now recognises a 2.3 pick by name (the load keeps the authoritative header probe, and under-guessing only falls back to the load-time pull), narrows the base list, and stages the extras in the same entry as the checkpoint so one repo stays one scoped job. A pick routed from the chat picker arrives as ?model= and ?quant= with no picker metadata, so a bare local .gguf or .safetensors was loaded as a pipeline: an explicit model_kind wins over the backend's filename sniffing, so it evicted the resident model and then failed on the missing model_index.json. Both pages now derive the load kind from the path, the same way their own picker handlers do. A torchao int8/fp8 build takes adapters only at load time. Switching artifact inside one family keeps the LoRA selection, since the family did not change, but the load did not bake it, so the next generation was rejected with 'reload the model with the adapter selection' while the picker still showed the adapter as active. The selection is now dropped once per resident build, with a message saying to pick and load again. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cancel an evicted safetensors load, spare the arbiter for CPU-only chat, fetch clips lazily Four fixes from the latest review round: - The GPU arbiter's chat evictor only cancelled the llama.cpp side. The orchestrator publishes active_model_name once its worker reports success, so an in-flight safetensors load was visible only as an entry in loading_models and finished onto the GPU after ownership had transferred. Cancel every pending load, and give the safetensors branch the post-load ownership recheck the GGUF branch already had. - A manual gpu_layers=0 GGUF load runs on the CPU with the GPUs hidden from the child, yet it took the arbiter unconditionally: it cancelled a running image or video generation for a model needing no VRAM, then held CHAT ownership so the next GPU workload unloaded it for nothing. Gate the acquire on the same predicate the launch-time CPU-only mask uses, as the image and video loaders gate on their resolved device. - The staged-download hook subscribes per repo, not per job, so another job on the same repo advanced the staged queue (starting a load whose scoped files were still downloading) or wiped a queue that was still running. Compare the variant each callback carries, like the chat page's auto-load does. - The video gallery fetched every record of a page into an object URL that lives until the page closes: 50 clips at tens to hundreds of MB each, for cards the user may never scroll to. Fetch a clip as its card nears the strip's edge, plus the selected one the player needs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim the comments across the diffusion backend Comment-only pass over the Python this PR touches: drop what the code already says, collapse multi-line explanations that still read on one line, and keep the reasoning that is not recoverable from the code. No code, docstring semantics or behaviour changes; verified with an AST comparison against the previous revision, and the backend suite is unchanged (same 37 environment failures as before: the API integration tests that need a live keyed server, the flash-attn install hooks, and the GPU memory fields). * Invalidate latents on a VAE swap, keep a cut-off generation, surface the EMA adapter - source_revision() scanned the checkpoint root plus text_encoder/tokenizer but not vae, so swapping or fine-tuning the VAE in place left the conditioning cache namespace unchanged and a warm run trained against latents from the old checkpoint. Include the vae directory, like any other component the cached tensors come from. - /images/generate answers only when the images are saved, and secure mode's tunnel caps an origin response near 100 seconds, which a native CPU or a high-step run passes routinely. The page reported failure while the work kept running, and a retry would duplicate it. A lost response (fetch rejection or a gateway status the origin never answered) is now told apart from a refusal: the page waits out generate-progress and reloads the gallery, so the run it started still lands. - The trainer emits the EMA adapter's path with the terminal event, but the state update dropped it, so neither the run history nor either response schema carried it and an enabled EMA left nothing discoverable. Keep it, and show it next to the primary adapter. - weighting_scheme advertised a choice of timestep sampling; sampling is always logit-normal and the flag only selects the bell loss weights. Describe what it does. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Hide unloadable cached rows, hold the dataset interlock, bound a GIF export - The cached-model listing tagged any repo with a model_index.json as text-to-image, so a community pipeline the image loader's trust rule refuses still got a row in the Images picker, and a detected-but-untrusted video repo fell through to that same tag. Gate the image tag on the load path's rule and hide an untrusted video repo outright. - A routed diffusion pick only carries a GGUF filename, which is all the chat picker has, so a curated single-file artifact arrived with no quant and was loaded as a pipeline: from_pretrained on a repo with no model_index.json. Pass the page's own catalog spec into the route pick, so a routed pick resolves to exactly what a direct pick on that page resolves to. - The dataset mutation endpoints checked is_active() and only then handed their filesystem work to a thread, so a start reserving in that gap changed captions or removed images underneath the preflight or the running trainer. The interlock is now registered for the whole request under the lock reserve() uses, and a start refuses while a mutation is open rather than waiting on it. - GIF export held every kept frame as a paletted image before encoding; a clip may be 2048x2048 for 1024 frames, and at the 12 fps target the step is 1, so one export click could allocate over 4 GB and take the backend down. Downscale past 720 px and widen the step to keep at most 300 frames. - seed accepted any Python int, so an out-of-range one passed every preflight, evicted the resident models, spawned the trainer and only then died in torch.manual_seed. Bound it to torch's 64-bit range in the request and config. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the accelerator probes in the DiT family-metadata tests Six tests read family_train_infos() (or a start preflight) without pinning the host probes, so they only held on a machine with a bf16 accelerator: on a GPU-less runner the DiT gate empties precision_modes, turns supports_compile off, and replaces any other preflight message with the no-accelerator note, and all six failed there. A conftest fixture pins both probes for exactly those tests, so they assert the family metadata they are about on every host. The gate's own CPU-only behaviour keeps its dedicated tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Carry the pipeline task into the hub inventory the pickers read - The task-scoped pickers filter On Device rows on a task, and the chat picker routes a diffusion pick by the same field, but those rows come from the /api/hub inventory, which never carried one: the Images and Video pickers listed nothing on device and the routing never fired. Both cached scans and the local listing now tag rows with the classifiers the models API already uses, the schemas and the frontend adapter carry it through, and a row the backend classified as a generation task is exempt from the chat-only guard that was also dropping it. - The local routing map was keyed by model_id while the row click passes id (a filesystem load id for a models_dir or LM Studio entry), so the lookup missed and the pick fell through to the chat loader. Key both. - A staged download whose start answered "error" left its head in place, where the effect never re-runs and onReady never fires, so the pick was stranded until the user reselected. Clear the queue and say so. - Every scoped pick in a repo shares the @diffusion variant, so the variant alone cannot tell two file sets apart: restaging while the first job finished let its completion pass for the new pick and load a checkpoint that had not downloaded. Bind the callbacks to the repo + file set they started, and to the staging generation. - A rejected generate POST does not say whether it reached the backend, so an immediately idle progress read was ambiguous and a submission that never landed looked like a finished image. Require evidence: progress seen active, or a gallery record that was not there before the POST. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the OpenAI image URL fetchable, keep WebM audio, stream example imports Four review items on the diffusion Studio work: - response_format=url returned the bearer-gated gallery route, which a standard image client downloads with no Authorization header, so the default response format was unusable. Mint a short-lived HMAC link instead (the shape RAG already uses for pdf.js) served by a signed route, and leave the gallery route itself bearer-only. - A manual gpu_layers=0 load carrying speculative_type="off" -- a value the UI persists and sends -- read as GPU-bearing, so it took the GPU arbiter and evicted a resident image/video pipeline even though the launcher hides the GPUs for it. Canonicalize the mode and exempt "off". - The curated example import prepared the whole split before the loop stopped at the 10-100 image cap; m1guelpf/nouns is 49,859 rows / 328 MB. Stream instead, with the prepared load kept as a fallback for a repo that cannot stream. - WebM export dropped the audio track an LTX-2 clip carries, silently, on the format offered for web embeds. Mux it as Opus through a resampler + FIFO, and keep exporting the video alone on a build without libopus. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not stub out triton on a GPU host when the Xet backend fails to import The lazy loader retries `import unsloth_zoo.hf_xet_fallback` under UNSLOTH_ZOO_DISABLE_GPU_INIT=1 whenever the first attempt raises. That flag makes unsloth_zoo take its MLX/CPU path, which injects triton and bitsandbytes STUBS into sys.modules for the rest of the process. On a working GPU box whose first import failed for an unrelated reason (a bitsandbytes/CUDA mismatch, say) the retry succeeds, so Studio boots looking healthy and then dies at the first CUDA-only kernel: a GGUF or compiled diffusion generation hits the stub and returns NotImplementedError: Unsloth: 'triton.tools.experimental_descriptor.enable_in_pytorch' was called on Apple Silicon / MLX, where triton is stubbed out. so every image generation 500s with an Apple-Silicon message on a Linux CUDA host, while the load reports success. Found by loading Z-Image-Turbo GGUF through the API on a box where bitsandbytes could not initialise. Gate the retry on the host genuinely having no accelerator. The Xet stall watchdog is optional and already degrades with a warning; a process whose triton is stubbed out is not recoverable. The warning now says why it did not retry. * Fix the lost-generation proof set, the settle timeout and the hub inventory's diffusion gates Seven fixes from the latest review round on the Images page and the hub cache inventory. Images page: - The lost-POST settle path built its "already seen" gallery id set inside the catch, after the request failed. By then the earlier runs of the same batch had already prepended their records, so run 2 could accept run 1's image as proof that its own request reached the backend. The set is now captured once before the first POST and grows with every record the batch produces. - settleLostGeneration fell out of its SETTLE_MAX_MS loop and returned normally, so a wedged generation was counted as done and the next run started against a busy backend. It now throws on timeout. - Restoring a recipe cleared the ControlNet selection but left the workflow tab and the init / mask / reference images pointing at whatever was loaded, so the next Generate conditioned on an unrelated image. It now clears all of them and returns to Create. - The download plan omitted the adapter selection the load itself bakes in. A baked LoRA forces the dense build path, so the plan described a different file set than the load that followed and the rest was pulled inline, outside the download manager. Both now derive the list from one helper. Hub cache inventory: - A download for a repo an Images or Video load is staging was allowed to start: only the llama.cpp loader was consulted. Both diffusion backends already expose loading_repo_ids for the delete guard, and the download guard now reads them too. - A companion-only prefetch (pipeline manifest plus VAE and text encoder, no transformer) passed the snapshot-partial check, since every file its manifest expected did arrive, and was advertised as on-device although from_pretrained cannot load it. - The single-file flag never reached the picker through the hub inventory path, so a checkpoint-only diffusion repo read as a full pipeline and failed after the handoff. The two pipeline-shape helpers now live in hub/utils/inventory_scan.py so /api/models/cached and the hub inventory classify the same repos the same way. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop adopting an unknown scoped download, leaking raced blobs and resurrecting deleted clips Three items from the latest review round. A scoped download job carries a deliberate file subset, and every file set of one repo rides the same "@scope" slot. A client that adopts a live job from the backend had no file list to compare against: the active-downloads response never carried one, so an adopted job's set was unknown and any later scoped request for the same repo read as "already started". Selecting a different checkpoint then waited on the wrong transfer and tried to load a file nobody fetched. The response now publishes the scoped file list, adoption records it, and an unknown set no longer satisfies a scoped request. A gallery record can be deleted while its blob is still downloading. The delete revokes the URL present at that moment, so the fetch that lands afterwards inserted a fresh object URL for a record no card renders and nothing can revoke: a full MP4, tens to hundreds of MB, pinned for the rest of the session, and once per raced fetch. Both galleries now discard a blob whose record went away, with an epoch covering the video page's Clear all. The video backend keeps the last completed job until the next one starts, and the Video page merges that record on mount to cover a job that finished after the gallery fetch. Deleting the clip left the record in place, so every reload prepended a ghost card whose file request 404s until another generation replaced it. Deleting the clip, or clearing the gallery, now clears the matching terminal record, and the page skips a record it deleted itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Resolve revisions from the live cache, serialize dataset imports, and stop pinning every gallery blob Five more items from the review round. The conditioning-cache revision marker read huggingface_hub's import-time HF_HUB_CACHE constant. Studio can move its cache during a session and loading follows the live setting, so after a move the marker went unresolved (or pointed into the previous root) and pulling a new revision of the same checkpoint no longer invalidated the cache: a warm run could reuse the old encoder's embeddings and the old VAE's latents. It now looks in the active Studio cache first and keeps the environment and the library constant as fallbacks, which the trainer subprocess still needs. The dataset interlock counts mutations rather than excluding them, so two imports of different examples into the same empty name both got past the emptiness check. The winner promoted its staging directory atomically; the loser found the folder non-empty, fell back to a per-file move, and merged its images and captions into the winner's dataset. Imports now take a per-folder lock, a second one is refused with 409, and the emptiness check is repeated under the lock. On Windows the sd.cpp asset resolver filtered only by accelerator token, so a Windows arm64 host matched an x64 zip, downloaded and installed it, and failed later when the binary would not run. It now filters by architecture the way the Darwin and Linux branches do. Every gallery page fetched every PNG up front and kept the object URL for the session, so scrolling a large gallery grew memory without bound for tiles the user may never look at. The Images strip now fetches a tile as it nears view, like the Video strip, and keeps the eager path only where IntersectionObserver is unavailable. A 503 carrying a JSON body comes from the application, not a proxy, so it is surfaced as the error it is instead of entering lost-response settlement and being reported as a request that never reached the server. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the class of a failed generation instead of a bare "Image generation failed." Found on a macOS runner: the native renderer aborts inside its own text encoder there, and the page showed only "Image generation failed." with the sd-server backtrace left in the server log, so nothing about the failure reached the user. The failure is now classified into fixed text, out of memory and native-process death, so the message says what happened and what to try. None of the engine's own output is echoed, since a native tail carries local paths and argv; that stays in the log, and an unrecognised failure keeps the original literal. * Treat an undecodable caption sidecar as the tombstone the trainer sees Uploads store .txt and .caption sidecars as raw bytes, so one can hold invalid UTF-8. The trainer treats any existing sidecar, decodable or not, as an empty tombstone and never falls back to the metadata row for that image. The labeling grid and the dataset summary read an undecodable sidecar as absent instead, so both showed a metadata caption that the run would silently replace with the instance prompt, and counted the image as captioned. Both now track sidecar presence separately, so what the user reviews is what the run trains on. * Keep the reason a native server died, not just its backtrace A ggml abort prints its cause first and then a stack trace, so reporting the last twenty captured lines gave twenty addresses and nothing about the failure: on the macOS runner the native server died on an unimplemented Metal op and the message carried only frame pointers. The captured tail now leads with the lines that name a cause and keeps recent context after them, for both the startup failure and the mid-request death. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop staging the dense text encoder for an fp8 video load Two halves of the same gap, found while measuring the LTX-2.3 download plan: - The video download plan and the scoped pre-download never saw text_encoder_quant. An fp8 request loads a hosted pre-cast encoder, so asking for one still staged and downloaded the base repo's dense Gemma3 (48.79 GB of Lightricks/LTX-2 on the 2.3 distilled pick) that the pipeline then never opened. The plan now drops those shards and stages the pre-cast checkpoint instead; their configs stay, since the pre-cast loader still meta-inits the encoder from the base repo's component config. - The LTX-2.3 assembly builds every component itself, so pipe_kwargs (which carries the pre-cast encoder for from_pretrained) never reached it and an fp8 request silently loaded the dense encoder anyway. It is passed across explicitly now. The dense skip is earned, not assumed: only a pre-cast checkpoint that resolves on the Hub lets the plan drop the dense shards, and only one already fetched to disk lets the pull drop them, so an unpublished or gated artifact leaves both exactly as they were. If injection still fails after that, the load tops the dense weights back up rather than handing from_pretrained a snapshot with no encoder in it. Measured against the real Hub on the 2.3 distilled Q4_K_M pick: 67.24 GB before, 18.92 GB with a 0.43 GB stand-in for the pre-cast artifact (the base entry drops from 24 files / 48.79 GB to 13 files / 0.04 GB). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Match the file's typing idiom for flow_shift models/training.py annotates with typing constructs throughout (105 Optional[...], no Union), and flow_shift was the one place using PEP 604. Union[] reads like the rest of the file, and it also drops the runtime evaluation that would raise on Python 3.9. * Bound the gallery blob cache, and three interlock fixes Four review findings, all reproduced first: - The gallery object-URL caches were unbounded. A clip runs from a few MB to a few hundred, both pages stay mounted after their first visit, and entries were only dropped on delete, so scrolling pinned everything for the session. Both pages now share a byte-budgeted LRU (512 MB video / 192 MB images) keyed off the visibility signal the near-viewport fetching already provides. On-screen media, the selected clip or image, and the item just fetched are never evicted, so eviction is invisible and a single item larger than the whole budget cannot evict itself into a refetch loop. - The image, video and chat load guards ran two independent training probes but returned early when the FIRST one raised, so an unreadable LLM backend disabled the diffusion interlock and a load could proceed straight into an active diffusion trainer on the same GPU. The probes are independent now. - An engine switch swallowed a failed teardown and published the new engine anyway, which is exactly the leak the unload exists to prevent: the arbiter's evictor, /images/unload and the next load all resolve through get_active_diffusion_engine(), so the still-resident pipeline (or a live sd-server) became unreachable and the next load allocated on top of it. The switch now fails and leaves the old engine published, so it stays reclaimable. - The native generation timeout was 30 minutes while the Images page waits up to 6 hours (SETTLE_MAX_MS), so slow-but-progressing CPU jobs died deterministically at the deadline. Measured on GPU-less runners, a 512x512 4-step Q2_K generation took 900 s on Linux and 1465 s on Windows, so larger images or step counts clear half an hour easily. The ceiling now matches the page's window and applies to the whole request: chunks of a split batch share one deadline instead of each getting a full budget. Cancellation is unchanged. Declined: gating the huggingfacenotorch extra off Python 3.9 over the conditional diffusers marker. The marker is deliberate and its comment says why: diffusers dropped 3.9 in 0.38, so pinning >=0.39 outright leaves pip no candidate and the whole extra unresolvable there. The pipelines it names live in studio/backend, which cannot install on 3.9 anyway (studio.txt pins matplotlib==3.10.9 and fastmcp>=3.0.2, both requires_python >=3.10), and the extra is the general core one, so the alternative drops 3.9 for library users who never touch Studio. * Close the load-versus-training-start race, and two picker fixes - The image and video load guards read is_active() and only then selected an engine, acquired the arbiter and registered the load. A /train/diffusion/start reserving inside that window freed residents the load had not registered yet, so the trainer came up beside a brand-new pipeline. The service already had exactly the right pattern for this in dataset_mutation, so gpu_load_admission mirrors it: reserve() refuses while an admission is open, an admission refuses once a start is reserved, both decided under the one lock. The span is only the registration, since begin_load returns as soon as the load is registered and _free_gpu_for_diffusion_training preempts an in-flight load from that point. Chat is deliberately not covered: its load spans an eviction plus a multi-minute GGUF load, and it admits models that fit beside training by design, which is a different contract from the diffusion pipeline's all-or-nothing one. - Hugging Face gives the LTX-2 family the image-to-video pipeline_tag (both Lightricks/LTX-2 and unsloth/LTX-2.3-GGUF report it), so a text-to-video-only filter dropped the flagship audio family out of Video Hub search while the rest of the app routed it to Video. - Task-scoped quant fit sized picks against the LARGEST visible device while resolve_diffusion_device_target returns a bare "cuda" and torch places on the current one. On a heterogeneous host that recommended a checkpoint sized for the bigger card and then loaded it onto the smaller one. Fit now uses the device the load actually lands on; identical on a homogeneous host. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Expose the persistent conditioning cache in the start schema DiffusionLoraConfig has carried cond_cache_dir for a while and the DiT trainer acts on it, but DiffusionTrainingStartRequest omitted the field, so Pydantic dropped it silently and every API-driven run fell back to the in-memory cache that is rebuilt from scratch each time. The warm path skips loading the VAE and the multi-GB text encoders on a rerun whose images, captions and resolution are unchanged, so this was a real capability that could not be reached. Contained like output_dir rather than left to the trainer subprocess's cwd, since it is another directory the trainer writes to. Blank or omitted still means the in-memory cache, so it must not resolve to the outputs root. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix diffusion policy and classification issues from review fp8 auto precision defaulted to precise accumulate on any non-consumer GPU, which made fp8 2.05x slower than int8 on RTX 6000 Ada and slower than not quantising at all. NVIDIA's professional whitepapers do publish equal FP8 rates for both accumulate modes there, so the hardware premise held, but the cost is in the cuBLAS path rather than the published rate. Default to fast accumulate: measured on B200 the flag is a no-op (4096^3 _scaled_mm at 3023.8 vs 3041.8 TFLOP/s, bitwise-identical output, 1.213 s vs 1.230 s end to end), so it is a large win where it bites and free where it does not. Precise accumulate stays available via transformer_quant_fast_accum. Z-Image's DiT is a Lumina2 derivative, so unsloth/Z-Image-GGUF and unsloth/Z-Image-Turbo-GGUF both declare general.architecture = "lumina2" and the whole line was tagged image-diffusion-unsupported and hidden from the Images "On Device" list, though validate_load_request loads them. Resolve shared archs from the repo/file name like bare "wan" already does, with a test asserting the picker and the loader agree for every family. The sage attention on-demand install ran an unpinned `pip install sageattention`, but PyPI's newest wheel is 1.0.6 and diffusers refuses anything below 2.1.1: the install always "succeeded", wrote an unusable version into the running venv, and was rejected on the next line. Carry the dispatcher's floor so pip resolves nothing instead. The dense-quant disk gate sized the download from the bf16-RESIDENT table. The fp32 families download twice that (Z-Image: 23,479 MiB against a 21,970 MiB gate), leaving a window where the check passed and the download filled the disk; Ideogram 4 ships fp8 and was overcharged the other way. Size the gate by published bytes, verified against HF sibling metadata for all 12 families. Patch installs went through unsloth_zoo, which refuses to import unless UNSLOTH_IS_PRESENT is set, and that is set by unsloth itself. The server imports unsloth at boot so it never showed there, but any other process ran silently unpatched with every install returning False, which is 13 test failures on a clean environment. Import unsloth and retry once, memoised per process. Also: the GGUF+LoRA refusal pointed at the native engine without saying a GPU host only selects it under UNSLOTH_DIFFUSION_ENGINE=sd_cpp, so the suggestion was unreachable; the gallery recipe recorded loras from the generate request alone, losing a load-time bake; load-progress claimed "40.07 GB downloaded" for a fully cached load; and pickers.tsx imported three catalog-group helpers it never used. Reported by oobabooga. * Keep the sd.cpp text encoder on CPU under Metal macos-14 loads FLUX.2-klein-4B Q2_K natively on mps and then dies on the first generation with exit code -6: ggml_metal_op_encode_impl: error: unsupported op 'RMS_NORM' -> ggml_abort LLMEmbedder::encode_prompt -> LLMRunner::compute -> GGMLRunner::compute ggml's Metal backend gates RMS_NORM on contiguous rows and aborts the process when that does not hold, with no per-op CPU fallback, so any LLM text encoder (Qwen3 for FLUX.2 and Z-Image, T5 for FLUX.1) takes sd-server down. The encoder runs once per prompt while the DiT runs every step, so pinning only the encoder keeps Metal for the part that matters. UNSLOTH_DIFFUSION_SD_CPP_METAL_TE_GPU=1 opts back in once ggml grows the kernel. * Gate the unsloth retry in the diffusion patch backend The retry added for the clean-environment patch failures is not free: importing unsloth pulls torch in behind it, which costs ~940 MB of RSS measured in a process that had neither, and on a host with no accelerator it fails anyway. A cross-platform CI job that had generated fine at ~900 s later died 19 s in with SIGTERM and every 'if: always()' step skipped, which is the runner being torn down rather than a step failing. Retry only when torch is already imported (true of the server and of anything patching a real module, and the condition that stops the retry from being what loads torch), unsloth is installed but not yet imported, and the first failure was the ImportError the sentinel guard raises. The clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only retry the unsloth import where it can succeed The gate still let the retry run on hosts unsloth does not support, which is where it is most harmful: a 7 GB macOS runner lost the Studio server 26 s into a load, and the Linux runner was torn down mid-generation. Neither MPS nor plain CPU can complete the import, so the retry there pays the cost and fails anyway. Require an accelerator unsloth actually supports (CUDA/ROCm via torch.cuda, or XPU), with UNSLOTH_ALLOW_CPU as the documented override, and hoist the predicate to module level so it is tested directly rather than through the import system. On a CPU-only host the retry no longer fires at all; on CUDA the clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Guard old diffusers, stream video exports, record conditioned recipes Three fixes from review. The 0.39-only pipeline classes (Flux2Klein, Z-Image, Krea 2, LTX-2, HunyuanImage) were resolved by getattr deep in the load, so on the older diffusers that packaging still allows on Python 3.9 -- diffusers dropped 3.9 in 0.38 and this project still supports it, so the 0.39 floor has to be conditional or the extra becomes unresolvable -- an advertised model failed with a bare AttributeError after its checkpoint had already been downloaded. Krea 2 already guarded itself this way; assert_pipeline_class_available now runs the same check for every image and video family from validation, before any fetch, and names the version and the fix. WebM export accumulated the whole VP9 output in a BytesIO and returned it as one bytes object that the response held again. The request caps allow 2048x2048 for 1024 frames, so an export runs to hundreds of MB and concurrent clicks could exhaust the process, while the MP4 route beside it already streamed from disk. transcode_to_file encodes to a temp file and the route returns a FileResponse with a background unlink, so nothing large is resident. A conditioned generation's recipe carried only the txt2img fields, so the gallery presented an inpaint or upscale result as a complete Create recipe and restoring it replayed an unrelated text-to-image request. The images themselves are still not persisted (user uploads with their own lifetime), but the workflow and its scalars are, restore reapplies them, and the toast now names the inputs that have to be supplied again instead of silently landing on Create. Reported by Codex. * Per-load video cancel event, family-gated image picker, cond cache refusal A cancelled video load could resume: begin_load cleared the shared cancel event, and unload() drops _loading without waiting for the worker, so the next load cleared the very object the cancelled worker was watching and its multi-gigabyte pull ran on alongside the replacement until the token check at the end. Each load now gets its own threading.Event, passed down through _fetch_te_prequant and _predownload_base, so a cancelled worker stays cancelled. A cached repo with a model_index.json was advertised as text-to-image on the trust rule alone, but validate_load_request also requires a detected image family, so a trusted pipeline of an unsupported class produced a picker row that deterministically 400s. The picker now applies both gates, mirroring the video branch. cond_cache_dir was accepted for sdxl and then ignored: only the DiT trainer reads it, while the SDXL trainer builds a per-run in-memory latent cache, so the promised cross-run reuse never happened. The route now refuses it with a 400 that names the families which do support it, checked against the resolved family so an omitted model_family with an SDXL base is caught too. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the frontend build broken by the gallery blob cache tsc -b failed on the branch head, so npm run build produced no dist and every platform job fell back to --api-only: blob-url-cache.ts(29,15): TS1294: This syntax is not allowed when 'erasableSyntaxOnly' is enabled dataset-labeling-grid.tsx / dataset-showcase.tsx: Argument of type '{ url: string; bytes: number; }' is not assignable to parameter of type 'string' The cache took its budget as a constructor parameter property, which the project's tsconfig forbids, and fetchGalleryObjectUrl now returns the blob size alongside the URL for that budget, which the two dataset thumbnail components still consumed as a bare string. Declare the field explicitly and destructure the URL at both call sites. tsc -b is clean and vite build emits dist again. * Recover from a ggml unsupported-op abort by restarting on the CPU backend ggml checks every node against the device's supports_op and calls GGML_ABORT when one is not implemented, because a single-backend graph has nowhere else to put it: there is no per-op CPU fallback. The whole sd-server dies with SIGABRT mid-generation and the user gets "the native image renderer stopped unexpectedly" with no way forward. Seen on macos-14 arm64 with FLUX.2-klein-4B Q2_K through the cross-platform CI: the text encoder is already pinned to CPU, and the abort moved into the denoise loop instead. ggml_metal_op_encode_impl: error: unsupported op 'MUL_MAT' -> ggml_abort StableDiffusionGGML::sample -> sample_k_diffusion A retry on the same backend would abort identically, so the load is restarted once with --backend cpu (the only flag that changes which backend executes the graph; --offload-to-cpu moves parameters, not compute) and the generation is re-submitted. The same checkpoint then renders slower rather than not at all. Strictly bounded: the signature must carry both the unsupported-op line and ggml_abort, the device must not already be CPU, and it happens once per load, so an OOM kill or a genuine crash still surfaces as itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix two tests that only fail in a full-suite run The 3.10 CI leg resolves PyAV 17, where av.container.OutputContainer is an immutable C type, so the no-libopus export test died on "cannot set 'add_stream' attribute of immutable type" before it asserted anything. Inject the refusal by wrapping the container av.open() returns instead; modules stay patchable on every build. Removing the injection makes the test fail again, so it still covers the branch it is named for. The Xet shim's degraded-path tests drop utils.hf_xet_fallback from sys.modules and import a throwaway copy. Restoring only the sys.modules entry left the utils package attribute bound to the throwaway, and the two disagreed for the rest of the process: a later monkeypatch of the dotted target patched one copy while the code under test imported the other, so the patch did nothing and test_fetch_te_prequant_only_reports_what_it_downloaded reached the real Hub and got a 401. Restore both bindings. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop an ignored-cancel sd-server, guard deletes during diffusion training, repair unusable managed binaries sd-server does not interrupt an in-flight job, so when it ignores a cancel the grace branch abandoned the poll and reported cancellation while the native job kept a core (or the GPU) busy to completion and held the server's job slot. The comment said the caller stops the server, but only unload does that immediately: a superseding load stops it after its multi-gigabyte download, and a load that then fails never gets there. Stop it here, as the deadline branch already does. DELETE /api/models/delete-finetuned checked only the LLM trainer, so it could rmtree the output directory a live diffusion LoRA run was about to write its adapter into. Consult the diffusion training service too, like the dataset mutation and model-load routes. find_sd_*_binary only checks is_file(), so an interrupted extraction (or a prebuilt for the wrong CPU) left a present-but-unrunnable binary the installer never retried: every load probed it, fell back to diffusers, and native inference stayed off until the directory was deleted by hand. Probe it and reinstall, but only for a copy under the installer-owned root -- SD_CLI_PATH, UNSLOTH_SD_CPP_PATH, an in-tree build and anything on PATH are the user's. * Plan the pre-cast text encoder, and make the cross-trainer GPU admission atomic An fp8 text-encoder request loads a hosted PRE-CAST checkpoint, but the image download plan never received text_encoder_quant, so the manager staged the base repo's dense encoder (FLUX.2-dev's Mistral-24B is ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) and the load then pulled the pre-cast file inline, outside the manager's progress and disk preflight. The plan now takes the field, resolves the hosted artifact with the same resolver the injection uses, stages that file, and drops only those components' dense weight shards. The load's own prefetch takes the same treatment, since it paid the same cost. Only a checkpoint that really resolves on the Hub earns the drop, so a gated or renamed artifact still stages the dense encoder the load will fall back to. The two trainers admitted each other with independent check-then-act guards: the diffusion route checks the LLM backend several network-bound preflights before it reserves, and the LLM route checks the diffusion service well before it spawns, so two near-simultaneous starts could both pass and train on one GPU. reserve() now re-tests the LLM backend under its own lock, and the LLM route holds the diffusion service's gpu_load_admission across its spawn, so exactly one of the two wins. Both halves fail open, so a chat-only install still trains. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the Advanced options a staged download planned against Staging does not set busy, so while a multi-gigabyte download runs the user can still change precision, memory mode, speed or the baked LoRA selection. The pending record held only the repo and artifact, and the completed download fired a load that read the CURRENT state: the staged file set could then be missing files that load needs (fetched inline, with no progress and no disk preflight) or hold gigabytes it no longer uses. One snapshot of every Advanced control is now taken when the plan is built, and it travels with the pending record into the load, so the load that runs is the one the download was planned for. * Do not advertise a family the installed diffusers cannot build The newer families (Z-Image, Krea 2, FLUX.2, LTX-2, HunyuanImage) exist only from diffusers 0.39, and 0.39 cannot be installed on Python 3.9 at all -- diffusers dropped 3.9 in 0.38, so the requirement is conditional or the whole extra becomes unresolvable. On such an environment the picker still offered those rows, every pick failed deterministically, and the error's advice to run pip install -U diffusers could not fix it without also upgrading Python. The cached-repo picker now applies the same availability check validate_load_request does, which is keyed on the pipeline class actually present rather than on the Python version, so it is also right for an intentionally pinned older diffusers on 3.10+. Fails open when diffusers cannot be imported at all: that is a different problem and the load path reports it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore the diffusion engine selection after each router test The active engine is module state, and several tests set it by plain assignment because what _activate does to it is the thing under test, so monkeypatch could not undo it. A leaked ENGINE_SD_CPP left get_active_diffusion_engine() handing back the sd.cpp backend for the rest of the process, and every later route that reads the active engine then saw an unloaded model: eight tests in test_openai_images_generations_route.py returned 503 in a full-suite run while passing on their own. The autouse fixture now snapshots and restores it. * Stream gallery clips, and close three races around them Four fixes from the latest review pass. The video gallery downloaded each clip into a blob before it could play, so playback waited on the whole file (tens to hundreds of MB), seeking was limited to what had arrived, and every viewed clip stayed pinned in the webview. The file route already streams and serves ranges; it just could not be a <video src> because it is bearer-gated. Mint a short-lived signed link instead (its own HMAC secret, 12 hour TTL, separate from the image links) and hand it to the element, which then fetches only the ranges it plays. That removes the blob budget, its LRU and every revoke on this page. The sd.cpp readiness probe accepted any process answering on the port, so a foreign server that grabbed the port between the bind check and the spawn was adopted as ours. Confirm the listener is our child before reporting ready, and stay best-effort (psutil missing, an unknown owner, or any probe error still passes) so the check can only reject a definitely foreign process. Dataset import held its lock for the extract but not for the upload path, so two concurrent uploads into the same folder interleaved; take the same lock and return 409. And reject Windows device names (CON, NUL, COM1..9, LPT1..9, with or without an extension) plus trailing periods in dataset names, which are unopenable on Windows. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Build the image download plan for the engine that will load /images/download-plan always asked the diffusers backend, while /images/load picks the engine per host: a GGUF pick on a machine with no usable GPU routes to native sd.cpp, which reads a single-file VAE plus text encoders and never opens the base repo's sharded components. Measured on unsloth/FLUX.2-klein-4B-GGUF (Q2_K): the plan staged 7.66 GB of FLUX.2-klein-4B components the native load discards, and the 7.80 GB sd-cli actually needs was then fetched inline by the loader, outside the download manager's progress and its disk preflight. Z-Image-Turbo is the same shape. The plan now asks whichever engine the load will select. predict_engine() applies the selection policy without any side effect: it activates nothing (staging a download must not unload the resident model) and only locates the binary rather than installing it, but still counts an installable binary as available, since that is what the load does on a fresh host. The native backend gains a download_plan built from the same _asset_specs the loader fetches, returning the same envelope, so the manager stages exactly the files sd-cli opens. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not let a queued generation outlive the model, and three scan fixes Five items from the latest review; four were real. An unload or arbiter eviction only cancels the generation holding _generate_lock. A second request queued behind it holds no cancel event yet, and Python locks are not FIFO, so it could take the lock the instant the active denoise released it, still see a loaded pipeline, and run a whole new denoise after the model was told to go away: the eviction then waits minutes for it and an image lands after the eject. Unload and a superseding load now raise a fence under _lock before they queue, and a generation that wins the lock while one is pending refuses instead. The cached-model scan judged pipeline completeness across every revision, so a repo holding an older complete snapshot plus a newer companion-only one read as complete while the snapshot from_pretrained actually opens has no transformer. Both scans now look at the revision the loader will open. Deleting a dataset image deleted its caption sidecar unconditionally, which for cat.jpg alongside cat.png removed the caption the survivor still resolves to. The sidecar now goes only with the last image of that stem, matching what the thumbnail cleanup beside it already did. Importing an example into a folder that holds no images but does hold files fell back to promoting the staging dir one file at a time, so an interruption left a partial dataset that the image_count check accepts as complete on retry. Those files are folded into the staging dir instead and the promotion stays a single atomic rename. The MPS generator report does not apply: torch.Generator(device="mps") has worked since PyTorch 2.0 (pytorch/pytorch#91348) and the studio installer pins torch>=2.4. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten diffusion comments Collapse the multi-line comment blocks across the image, video, sd.cpp and diffusion-training code to one or two lines each, and drop comments that only restate the statement below them. Comments only, no code or behaviour changes. * Tighten diffusion comments (second pass) Collapse the remaining multi-line comment blocks in the video page, training routes and service, sd.cpp server and installer, memory and speed planners, and the shared request models. Comments only, no code or behaviour changes. * Tighten diffusion comments (third pass) Collapse the remaining multi-line comment blocks in the attention, cache, LoRA, prequant, precision and compile-cache modules, the sd.cpp arg builder and engine, the video routes, the Ideogram 4 assembly, the model picker, and the diffusion test suites. Comments only, no code or behaviour changes. * Restore the dataset when an example import cannot be promoted Promotion folds the folder's pre-existing entries into the staging dir so the swap is one atomic rename. Every failure after that fold left the user with nothing: the 409 path and an os.replace error both fell through to 'finally: shutil.rmtree(staging)', which deleted the entries that had just been moved in there, while the response said 'Nothing was written'. A same-named entry was also unlinked outright before the promotion was known to succeed. Park superseded same-name entries in a rescue dir instead of deleting them, record every move, and restore all of them if any step of the promotion fails. The fold loop itself is covered too: renaming a non-writable directory raises EACCES on POSIX, which previously escaped as a 500 after the earlier entries had already been moved out. A failed rename now maps to the same retryable 409 as the rmdir conflict. Verified with the reported trigger (a non-empty mode-500 directory whose name collides with an imported file): the folder listing is now identical before and after the failed import. * Drain the teardown fence on a failing unload, give each load its own cancel event Two independent leaks on the image path, both already solved elsewhere in the same file. unload() incremented _teardown_waiters, ran _unload_locked() and decremented, with no try/finally, while the superseding-load path used a finally for the same pair. _unload_locked ends in clear_gpu_cache(), whose CUDA branch calls synchronize/empty_cache/ipc_collect unguarded, and a sticky CUDA fault makes those raise. The count then never drained, so every later generation was refused as cancelled for the life of the process, a fresh load included, since begin_load's own increment and decrement are symmetric. unload() is reached from the chat/video GPU handoff, the engine router and two training routes, so one fault during an ordinary handoff wedged image generation until restart. Release it in a finally. The image and native backends each cleared one shared cancel Event on a new load. unload() sets that event to cancel an in-flight multi-GB download and drops _loading in the same breath, so a replacement load is admitted while the cancelled worker is still inside the fetch, and its clear() re-enabled the very object that worker was watching: the cancelled download resumed and ran alongside the replacement. Take a fresh Event per load and thread it to the worker, as the video backend already does, and set it under the lock since begin_load now rebinds the attribute. * Name utf-8 on the diffusion text I/O and the sd.cpp subprocess pipes tests/test_text_io_encoding.py failed on five files this branch adds. Text I/O without an explicit encoding falls back to the Windows ANSI codepage, so a non-ASCII path or manifest value round-trips corrupted, and the three sd.cpp pipes decode the child's UTF-8 output as ANSI on Windows despite already passing errors = 'replace'. Eleven read_text() / write_text() sites across diffusion_compile_cache, diffusion_ideogram4 and diffusion_krea2, plus text = True on the sd-cli version probe, the sd-cli run and the sd-server pipe. * Record the load-time build on a gallery image's recipe A gallery record documents itself as the image's full generation recipe and is embedded in the PNG, but the only load-related field it carried was the repo id. A GGUF repo holds many quants, so that does not say which one made the pixels, and it says nothing about an adapter baked in at load time. The fallback meant to cover the baked case could never fire: with no loras on the request _adjust_baked_loras zeroes every baked adapter and _active_lora_pairs drops zero-weight entries, so active_loras was always empty. A baked-and-disabled build is not the same pipeline as a never-baked one, so the recipe could not reconstruct the image once the model was rebuilt. Persist model_kind, gguf_filename, transformer_quant and the baked adapter names, read off the load state rather than the request, and show them in the recipe popover. The new fields are optional with defaults, which matters because list_gallery_images drops any record that fails validation, so a required field would have emptied every existing gallery; a regression test pins that. * Drop eleven duplicated comment tails, restore the mxfp8 denial note The comment passes collapsed several wrapped blocks onto one line without deleting the last physical line of the original wrap, leaving the tail of each sentence repeated as its own comment underneath. Two of the eleven were re-worded rather than byte-identical, so a strict suffix match missed them. 6e16ad16f also dropped the line justifying the qwen-image mxfp8 denial while that rule stayed live in _FAMILY_SCHEME_DENY, under a header that then documented only fp8 and nvfp4. Restored. Comments only; verified with the AST gate. * Only let the dense-quant fallback use shards the prefetch actually staged The prefetch skips the base repo's transformer/ shards whenever a prequant checkpoint is expected, since that checkpoint replaces them. But a prequant fetch can fail for reasons the planner cannot see: an unpublished, gated or renamed artifact, a hub 5xx, a proxy, a checkpoint the validator rejects. The loader then fell through to from_pretrained(subfolder = 'transformer') and pulled those shards inside the load lock during 'finalizing', after the previous pipeline was already evicted, where the cancel event has no reach, load_progress has already reported bytes_downloaded == bytes_total, and the cache-disk gate had only reserved the small prequant checkpoint. That is verbatim the situation _dense_quant_prefetch_needed's own docstring exists to prevent. Gate the in-loader dense fallback on the shards being staged, read off the returned file list rather than the request so a failed size estimate closes it too, and let the GGUF build take over otherwise, which is what the prequant-sized replan already does one branch over. It is also the invariant the text-encoder path already enforces: only a component that really resolves may have its dense weights dropped from a plan or a prefetch. Adds the missing coverage for the gate's prequant arm, which the existing disk-gate tests never reached. * Refuse a training output_dir that resolves to the outputs root resolve_output_dir strips a leading 'outputs' and drops '' / '.' segments, so '.', './', './.', 'outputs', 'outputs/outputs' and ' . ' all clean away to nothing and land on the outputs root itself rather than a run directory under it. The DiT trainer then writes pytorch_lora_weights.safetensors flat into the root, where scan_trained_models and scan_checkpoints cannot see it (both filter is_dir()), and a second such run overwrites the first. The UI only checks the field is non-empty, so 'outputs' is one plausible run name away. Refuse it with a 400 that says what to do instead. cond_cache_dir collapses the same way but has an honest 'off' to fall back to, so a root-resolving value now means the in-memory cache, which is what the comment above it already promised: otherwise a run drops one flat safetensors per cached latent and caption into the directory trained models live in. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Repair three defects the main merge left behind Git merged all three files without a conflict, but the result was wrong in each case. The tests only started failing once the merged tree was run. routes/inference.py duplicated the GGUF load block: this branch had moved the gguf_load_in_flight marker and the _hub_download_blocks_gguf_load guard under "if config.is_gguf and config.gguf_hf_repo", and the conflict resolution re-added main's copy at the old position, so both ran. Dropped main's copy; the earlier placement is the deliberate one, so a 409 from the hub guard cannot tear down a resident Images or Video pipeline. test_gpu_selection.py still called _hf_offline_if_dns_dead, which main renamed to _hf_offline_if_unreachable_for (#7591). Disjoint edits, so no conflict, but four route-error tests referenced a function that no longer exists. test_gguf_load_cache_reuse.py anchored its ordering assertion with rindex over "if config.is_gguf:", taking the last one before the load marker. That only held while _resolve_inherited_extra_args sat above every such line; main has since hoisted it above the gpu_ids preflight, so the anchor landed between the call and the marker and the assertion compared against an unrelated later call site. The ordering it checks is a property of _load_model_impl as a whole, so it now anchors on the function. The ordering itself is intact: _resolve_inherited_extra_args, then the gguf_load_in_flight marker, then the hub guard, then the chat handoff, then unload_model. * Stop a real sd.cpp install from breaking its own discovery tests The five "nothing is installed" assertions in test_sd_cpp_engine.py cleared SD_CLI_PATH and UNSLOTH_SD_CPP_PATH and patched Path.home, which covers hops 1, 2 and the fallback half of hop 3. It leaves two hops live. Hop 3 goes through managed_install_root(), which honors UNSLOTH_STUDIO_HOME and STUDIO_HOME and resolves to the stable-diffusion.cpp directory beside the studio home. That is the documented way to run side-by-side Studios, so anyone who has one set gets a real sd-cli back from a finder the test expects to return None. Hop 4 is the in-tree developer build, which does the same for anyone who built sd.cpp inside the checkout. Isolated with an autouse fixture rather than another helper call, because reaching the failure needs no fixture: SdCppEngine(binary = None) runs the finder from its constructor, which is why test_generate_raises_when_binary_missing failed too. The fixture points the studio home and the in-tree root at an empty tmp tree, so every hop is answered by the test rather than by the box. The in-tree root moves into a named in_tree_install_root() so it can be pointed somewhere empty; behaviour is unchanged, including the OSError and IndexError guard on an unexpected layout. * Make the sd.cpp uninstall test actually extract the code it tests The two sed ranges anchored the production fragments at column 0, but both blocks sit inside the main removal function and are indented, so each range matched nothing and LOOP_FILE / DEFAULT_FILE came out empty. Sourcing an empty file is a no-op, so the suite reported 4 passed / 7 failed: the seven removal assertions failed because nothing ran, and the four that passed were 'kept' assertions that pass trivially when nothing runs. The shell job auto-discovers tests/sh/test_*.sh and runs each under set -e, and this file is not in its skip list, so it was a deterministic red. Anchor on optional leading whitespace and fail loudly on an empty or _remove_path-less fragment, so a future reshuffle of uninstall.sh cannot make the suite vacuous again. Now 11 passed, 0 failed, against the real removal loop and the real default-mode block. * Agree on what an engine can build, and on what the installer owns Two gates, each half-applied. The unbuildable-family gate had one caller, the image branch of the cached repo picker. The GGUF classifier, the local-model classifier and the video branch had none, so on a diffusers too old for a family the picker still offered its GGUF and the load then failed. Meanwhile the loader asserted the diffusers pipeline class before engine selection, so a GGUF this host routes to native sd.cpp, which instantiates no pipeline class at all, was refused with an upgrade instruction that could not help. Both are wrong one-sidedly: hide a family only when NEITHER engine can build it, and demand the diffusers class only when diffusers is what will load it. One predicate, family_buildable_here, now answers both, so the picker and the loader cannot disagree. The population is real on Python 3.9, whose diffusers ceiling is 0.36: Flux2KleinPipeline, Krea2Pipeline and LTX2Pipeline are all absent there, and FLUX.2-klein GGUF is a repo in this PR's title. That assertion also raised RuntimeError, which /images/load maps to 409, the status that otherwise means a load is already in progress, and which escaped /images/download-plan (it catches ValueError and FileNotFoundError) as a bare 500 with the message lost. It is an unloadable pick like every other, so it raises ValueError and both routes answer 400 with the text intact. The managed-binary repair used a path test for ownership while the installer uses a marker. On a managed root without the marker, which is any install predating it, the repair deleted sd-server and the reinstall was then refused, permanently, because the surviving sd-cli keeps the directory non-empty so the marker can never be claimed: the user went from an unrunnable binary to no binary and no way back. Require the marker before discarding, which is the same definition of ours that uninstall.sh already uses to keep a user's own stable-diffusion.cpp checkout. A genuinely interrupted extraction still self-heals, since install() writes the marker before it extracts. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add gguf and av to the studio extra so the wheel install matches studio.txt This branch added both to studio/backend/requirements/studio.txt, which is what install.sh uses, but never to the studio extra in pyproject.toml, which is what 'pip install unsloth[studio]' uses. Nothing else keeps the two in sync, so tests/studio/install/test_studio_extra_matches_requirements.py failed deterministically, and the CI job runs the whole tests/ tree. The drift is not only a red test: gguf backs diffusers' GGUFQuantizationConfig and av does the MP4 encode and audio mux, so a wheel install got a Studio that cannot read a GGUF or export a video, which is most of what this PR adds. * Fix two test-suite defects: the CPU-only patch gate and a popped module The patch backend reaches unsloth_zoo's helpers only through an ~940 MB 'import unsloth', which a CPU-only host cannot complete, so every patch install returned False and nine arch/eager/compile tests failed there -- exactly the runners the retry was narrowed to protect. unsloth_zoo only wants UNSLOTH_IS_PRESENT in the environment, which costs nothing and needs no accelerator, and the conftest already sets its sibling UNSLOTH_ALLOW_CPU. Set it there too, as run.py and main.py already do at module scope, so no real server ever took the expensive route. CPU-only goes from 9 failed to 24 passed, 1 skipped; the GPU run is unchanged at 29 passed. test_setup_cache_env_hf_home popped utils.hf_cache_settings to model a fresh process and never restored it, so a later import built a second module object and rebound it on the utils package. test_hf_cache_settings then wrote its setting into one object while core.inference.diffusion read the other, the same split-module failure the xet shim already had. Restore both bindings in teardown: 21 pass together, and each file still passes alone. * Restore the setup_fail assertions the main merge reverted A fourth instance of the class 9541cc535 fixed: the merge took main's studio/setup.sh, which #7644 changed to abort through the setup_fail helper so desktop mode still emits [TAURI:ERROR], but kept this branch's older copy of the test, which still asserted the literal 'exit 1'. setup.sh is byte-identical to main here and the only diff in this file was the reverted assertions, so take main's version. 60 passed. * Fence the video teardown, reject non-finite clipping knobs, stop a test installing sd.cpp The video backend never had the teardown fence the image backend documents. Its unload and its superseding-load path both signalled the active generation, then did 'with self._generate_lock: pass' and tore the state down with the lock free. A generation queued behind that barrier holds no cancel event yet, so the signal cannot reach it, and Python locks are not FIFO: it won the lock the instant the barrier released it, read a still-loaded state and denoised a whole clip against the pipeline being freed. Reproduced on both paths, where the queued generation returned a finished MP4. Mirror the image backend: count waiters, refuse a generation while one is pending, and tear down inside the barrier with the counter released in a finally. _teardown_state becomes _teardown_state_locked since the caller now holds both locks. max_grad_norm and snr_gamma were bounded on one side only, so 1e309 floated to inf and passed. clip_grad_norm_ then computes an infinite clip coefficient, clamps it to 1.0 and scales nothing, and min(snr, inf) / snr makes every min-SNR weight 1.0. The run starts, reports normal progress and trains with the requested knob silently disabled. Measured both. NaN already failed the bounds; allow_inf_nan makes that explicit. learning_rate and flow_shift already guard this exact vector. test_load_routes_to_sd_cpp_on_cpu stubbed ensure_sd_cpp_binary but not ensure_sd_server_binary, which select_and_activate_engine probes first with installs enabled, so the unit test downloaded and unpacked 108 MB into the developer's real ~/.unsloth when UNSLOTH_STUDIO_HOME was unset. Now 0 bytes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Key the video schedule reset on the resolved defaults, not the repo id A GGUF repo holds several variants, so another client swapping a distilled build for the base one from the same repo changes the steps and guidance the backend reports while repo_id stays put. modelChanged stayed false, the page kept the previous schedule, and later generations ran an 8-step distilled setting on a model that expects roughly 40 steps and CFG 4. Include the reported defaults in the key. They only move when the resident artifact does, so a manual steps change still survives a status poll. * Retire the release-lag pin now that 2026.7.6 carries the relaxed gates The virgin Windows container row that installs from PyPI on purpose was tolerated with continue-on-error, because a Server Core container has no Microsoft Store and so no winget, and the released studio/setup.ps1 hard-stopped on a winget-only git gate and reached for winget again for the VC++ runtime. #7549 relaxed both, but it landed on 2026-07-28 and the newest wheel was 2026.7.5 from the 23rd, so the row could not get past it. The pin carried its own tripwire for exactly this moment, and it has fired: unsloth 2026.7.6 shipped 2026-07-29 and the released wheel now installs end to end, so the step reported success and the assertion errored with "delete this pin and drop continue-on-error from the Install step so this row gates". Doing that. Both rows now gate unconditionally, and the assertion that enumerated the accepted failure signatures goes with the pin it protected, since there is no longer an accepted failure for it to describe. The other three release-lag mentions in this file are descriptive rather than tolerated-failure pins: they explain why the fedora and ubuntu-nonroot legs run with overlay: true, and both already refuse a triton failure as a regression rather than accepting it as lag. Left alone. * Pin the hosted pre-quant onto the plain-torch fp8 kernel A Z-Image GGUF pick at the DEFAULT speed mode, once the hosted fp8 pre-quant is actually reachable, dies at generate with an HTTP 500: torch._dynamo.exc.Unsupported: Operator does not support running with fake tensors. Developer debug context: unsupported operator: mslk.f8f8bf16_rowwise.default _fp8_config already pins KernelPreference.TORCH when it BUILDS a config, precisely because the default AUTO switches to the MSLK kernel wherever an mslk package is importable (sm90+). A hosted checkpoint escapes that pin completely: the preference is serialized per Float8Tensor, and all 239 weights in the published Z-Image-Turbo-FP8 checkpoint carry AUTO. Loading it re-arms the very kernel the pin exists to avoid, mslk.f8f8bf16_rowwise has no fake impl, and the first compiled generate therefore cannot be traced. Isolated away from the product to be sure of the mechanism: quantise one Linear three ways on this box and compile each. KernelPreference.TORCH eager ok, compiled ok KernelPreference.AUTO eager ok, compiled FAILS on mslk.f8f8bf16_rowwise library default eager ok, compiled FAILS the same way So the pin is correct and necessary, and the only gap is that the hosted path never got it. _validate_checkpoint checks scheme, granularity, base model, min_features, exclude tokens and fast_accum, but not this. Rewriting the preference on load is safe: it selects a matmul kernel, it is not weight data, so the tensors stay bit-identical and the checkpoint's own state_dict_sha256 still describes them. It is also the faster path compiled, since the opaque extern call blocks inductor quantize fusion. Why this went unnoticed: the pre-quant repos are private, so nothing that could not read them ever took this path. It becomes the default the moment they are readable. Verified end to end at 1024x1024, 8 steps, speed_mode default, on the published checkpoint: before HTTP 500 at generate, diffusion.generate_failed after pinned 239 weights to the plain-torch fp8 kernel, load 10.0 s, transformer_quant fp8 with compiled engaged, cold 7.9 s, warm 0.87 0.87 0.87 0.86 0.87 s That warm number also beats the 1.4 s recorded for this shape on 2026-07-26. 159 prequant and transformer-quant tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cover the fp8 kernel pin so the 500 cannot come back quietly The hosted pre-quant re-arming MSLK was only visible as an HTTP 500 on a compiled generate, which needs a GPU, a readable private repo and an importable mslk to reproduce. None of that is available in CI, so the pin would rot unnoticed. Three hermetic cases on _pin_kernel_preference instead: AUTO weights are rewritten and already-TORCH ones are left alone (counting only what changed), a weight that refuses the assignment does not sink the whole load, and with no torchao enum available the checkpoint is left exactly as saved. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: tighten comments across the image generation, video and training code Rewrite the comments added by this branch to be shorter and clearer: collapse multi-line explanations into one or two lines, drop comments that only restate the code, and keep the rationale that explains why a choice was made. No code changes. * Studio: let the pipeline-class guard survive a host with no diffusers Backend CI installs the CPU-only dependency set, which has no diffusers, and runs without a GPU. 13 tests failed there for three separate reasons, one of which is a real product bug rather than a test-environment gap. assert_pipeline_class_available did a bare "import diffusers", so on any host without it the call raised ModuleNotFoundError instead of the ValueError its own contract promises. /images/download-plan catches only ValueError and FileNotFoundError, so that escaped as a bare 500 with the message lost, which is the exact failure the guard exists to prevent. It is reachable in production: the native sd.cpp engine serves GGUF picks on a CPU or Apple host where diffusers is never installed. Absent diffusers now returns, since the check answers "is the installed diffusers new enough for this family" and there is no version to judge; a pick that genuinely needs diffusers still fails in the loader with its own message. The rest were tests asserting through gates unrelated to what they cover: - Three cond_cache_dir route tests drive a DiT family, which the start route refuses without an accelerator. They now take the existing dit_train_host fixture, written for this case, so they keep testing the schema on every host. - test_download_plan_forwards_the_load_time_controls stubs the diffusers planner but sends a GGUF pick, which routes to the native planner on a GPU-less host. It now pins the engine, so it tests kwarg forwarding rather than the hardware. Engine selection keeps its own tests. - The pipeline-class guard test needed a real diffusers only for its sweep over every shipped family. That half is split out and skips on its own; the stub-driven refusal needs no diffusers and still runs. Added a test for the absent-diffusers contract above. 409 pass on a GPU host with diffusers; 301 pass and 1 skips with diffusers blocked and no GPU. * Studio: cut the image generation comments down further Second pass over the comments this branch adds. Collapse the multi-line blocks that still read as paragraphs, rewrite the longest one-liners so they say the same thing in fewer words, and drop a stale comment that had drifted away from the constant it described. No code changes: only comments and whitespace. * Studio: workflow rail, sidebar Video row, and Train settings polish Sidebar - Pin Video under Images by default; persist store bumped to v3 so an untouched install adopts it while a custom arrangement is left alone. - Add "Customize sidebar" to the end of the More flyout, opening Settings > Appearance scrolled to the sidebar nav section. Images - Replace the workflow dropdown with an icon rail down the left edge: all seven workflows visible, keyboard nav, tooltips carry the labels. - Workflows stay selectable with no model loaded, so one can be set up before picking a model. Generate is already gated on a loaded model. - Video moves out of the Create/Train strip into a link at the far right, with a matching Images link on the Video page. - Wider gutters around the settings column, headings matched to Train. Train - Field guidance moves into "i" tooltips; only state that limits a control stays on the page. Steps, LoRA rank and the rest gain hints. - /info reports params, qlora_vram_gb, gated and note as fields, so the family note renders as chips. vram_note is rebuilt from them unchanged for older clients. Shared - Number steppers appear on hover or focus. - Lighter control border (#e9e9e9) on Images and Video, light mode only. * Studio: move Advanced inline on Images and Video - Replace the top-bar toggle and right-docked Advanced panel with a disclosure under Seed, so load-time tuning sits with the settings it affects and opening it no longer shrinks the preview canvas. - Video uses the bordered variant, since offload and memory decide whether a model fits at all. - Open state persists per page in localStorage. - Tighten the Steps unit trigger: 14px chevron, less right padding. * Studio: full width media pages and layout polish - Drop the 1100px cap on Images Create, Images Train and Video, so the preview canvas grows with the window instead of sitting in a band. - Align the workflow rail to the model selector label, widen the gap before its divider, and restore the divider itself. - Advanced: more room above and below, icon and label sized to the slider rows, and the same quiet row on Video as on Images. - Give Video a pane heading and description, matching Images. - More space above Seed on both pages. - More no longer takes the active style when the current page is one of its own rows. * Studio: fix the Extend side toggles The resting outline used a ring, and index.css blanks the ring on the button holding mouse focus, so the side you just clicked lost its border until focus moved. Use a border for the outline and leave the ring to focus-visible. Also separates the two states properly (the off state had no surface of its own), drops the border in dark mode as the inputs do, and adds the aria-pressed and focus-visible styling the buttons were missing. * Studio: drop the border on the Extend side toggles Light mode should carry no border either, so the fill alone marks the state: a muted surface when off, a primary tint when on. Matches the borderless treatment dark mode already had. * Studio: move the Images workflows into the sidebar The workflow switcher was a vertical icon rail on the page. It now lives in the sidebar under Images, so the page keeps that width for the canvas. - Workflows list under the Images row, with a chevron to fold them away. Hovering the row peeks them in a flyout on the standard menu surface. - Clicking Images while already there toggles the list instead of navigating. - The listed workflow carries the highlight, so the Images row drops it. - Create takes its own icon: it was sharing the New chat pencil. - Model hub moves above Projects. Persisted layouts bump to v4 and only adopt the new order where the stored one is still a shipped default. - Images and Video content both start at 32px, clear of the sidebar. * Studio: simplify the Images workflow submenu Dropping the hover flyout: two ways to reach the same seven workflows, one of them overlapping the list right below it, read as clutter. - The flyout is gone. The workflows are rows under Images and nothing else. - On the Images page they are always open, since they are that page's switcher. Elsewhere they stay folded, and the row's chevron opens them. - Create takes the sparkles icon. - Create and Train both pad their settings column to 40px a side, so the two tabs line up with each other and with the model selector. * Studio: dock the generate action and tidy the Create controls The primary action sat at the foot of a long scroll, so it was off screen until you scrolled for it. - Generate, Video's Generate and Train's Start training float at the bottom of their settings column. No bar behind them: hover lightens the fill rather than thinning it, and the disabled state is opaque, so the controls underneath never show through. - Aspect ratios read as names: Square (1:1), Widescreen (16:9) and so on. - Width and height replace their sliders with two compact boxes. Type a size or pick one from the menu; the value still snaps into range and still drives the locked ratio. - Negative prompt gets a quiet disclosure under the prompt. It was there before but only above guidance 0, which is not the default, so it never showed. - The info "i" is smaller across the settings UI. * Studio: reveal Images and Video field hints on hover The "i" next to every field label sat there permanently, which made a column of settings read as busier than it is. Scoped to the diffusion pages by CSS rather than per component: the hints come from the page's own Field and SelectRow, the train panel's FieldLabel, and the chat ParamSlider, which is shared and should not change for chat. Reveals when the pointer is inside the field, on focus-within for keyboard, and stays inert while hidden. InfoHint grows a data-slot so the rule has a stable hook, rather than keying off its aria-label. * Studio: one-line sliders, edge fades, shared negative prompt - Steps, Guidance and the rest put label, track and value on one row. ParamSlider is shared with chat, so this is an opt-in prop and chat keeps the stacked layout. - The settings columns fade at whichever edge they run past, as the sidebar and model picker do, instead of cutting off. New useScrollFades hook drives all three. - Negative prompt moves to a shared component and Video picks it up, so both pages collapse it the same way. - Right gutter sits closer to the rule, Steps takes a bigger break above it, and the Advanced rule sits between its neighbours rather than up against the field above. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: add the API monitor to the sidebar nav Sits under More, below Export. Points at the monitor page, not the API keys dialog the profile menu opens. No store version bump: an id missing from a stored layout is appended with its default pinned state, so existing sidebars gain the row at the end and keep their own order. * Studio: take main's copy of the sidebar spinner test The merge kept this branch's older version, which pins the Recents row to h-[33px]. Main already relaxed that to any height, since row density moves independently of the trailing column the test is about. * Studio: fix FLUX.2 klein size resolution, HunyuanVideo tier routing, and the personalization save Five fixes found while auditing the model registries against the live Hub. FLUX.2 klein 9B loaded against the 4B config. One family covers both klein sizes and defaults to 4B, relying on the base_model card tag for the real base, but the trust allowlist only had klein-4B, so the correct tag was discarded and a 9B GGUF (inner_dim 4096) was loaded against a 4B config (3072). That surfaced as a bare shape mismatch from inside the GGUF quantizer naming neither the file nor the repo. Adds the missing allowlist entries and a header-level size check that fails early with a legible message; the check is fail-open, so an unreadable file, a non-FLUX.2 family, or an unmapped base leaves the load exactly as it was. The sd.cpp text encoder rule matched the literal "klein-9b", so klein-base-9B was handed the 4B encoder. HunyuanVideo-1.5 720p checkpoints routed to the 480p family. Only the literal 720p_t2v path was aliased, so 720p_i2v and every GGUF repack fell through to the generic token and inherited the 480p base repo, which also supplies the VAE and text encoder. The tier is baked into the weights (transformer target_size 640 vs 960, scheduler shift 5.0 vs 9.0) and the two bucket lists are disjoint, so this ran the whole pipeline off-tier. Resolution presets that the checkpoints were never trained for. Wan2.2 TI2V-5B is 720P-only upstream (SUPPORTED_SIZES is exactly 704x1280 and 1280x704, asserted in generate.py), and the HunyuanVideo 480p square preset 624x624 is not a bucket of its tier; 640x640 is. Personalization could never be saved. The frontend ships eight sidebar nav ids and refills every missing one on each save, but the backend Literal had seven and no "api", so every PUT to /api/settings/personalization returned 422. Aligns the server defaults and the Literal with the shipped layout. A catalog row pointing at a file that does not exist. unsloth/Qwen-Image-2512-FP8 holds torch prequant .pt checkpoints, not qwen-image-2512-fp8.safetensors, and fp8 is denied for this family anyway because it renders black, yet the row was the auto-route target on any GPU above 40 GB. Also marks FLUX.1-schnell, Krea-2-Turbo and the two Ideogram repos gated: all four are gated on the Hub today, and schnell being Apache-2.0 does not make it ungated. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: stop a case-only duplicate upload destroying an image, and size the load against the right GPU Two fixes from the review round. A dataset upload containing two names that differ only by case silently loses one on Windows and macOS. The in-batch duplicate check is an exact string compare, so "Cat.png" and "cat.png" both pass, but on a case-folding filesystem they are one destination: the commit step moves the first staged part aside, writes the second over it, then deletes the backup, while the response still reports both as uploaded. Captions collapse the same way and never even reach the image branch. No single folder can hold such a pair, but the Windows and macOS open dialogs flatten search and Recents results across folders into one multi-selectable list, which is a normal way a LoRA dataset gets assembled. Rejecting the pair everywhere would regress Linux, where the two really are different files with their own sidecars, so the check probes the filesystem once per process instead of keying off sys.platform: macOS also ships case-sensitive APFS volumes, and a Linux host can keep its Studio home on an exFAT or NTFS mount. A failed probe answers "case-sensitive", which leaves today's behaviour untouched. The quantization fit budget was sized against the wrong card under a reordering CUDA_VISIBLE_DEVICES. A bare "cuda" load lands on visible ordinal 0, but the reducer took the lowest PHYSICAL index, which the nvidia-smi path reports as index_kind "physical". Under "3,1" that sizes against GPU 1 while the pipeline loads onto GPU 3. The hook's own interface doc already promised the lowest visible ordinal. Ranks by visible_ordinal, falling back to index only for an older backend that omits it. * Studio: rework the diffusion dataset panel and align the pane gutters Upload: - one Upload button beside the dataset name, with an upload icon - picking files uploads them, no second confirm click - Add button beside the Training images dropdown, so images can go into a set that already exists - fall back to the upload form when a selected dataset no longer resolves Labeling grid: - two columns at any width; sm:grid-cols-3 crushed the tiles in a fixed width column - pin the caption size so the Textarea's md:text-sm does not outsize a tile - name the tile hover group; a bare one revealed every tile's Remove at once - Remove is an icon button on the image, not a word over the artwork - drop the amber tile fill, keep the No caption label - match the header, toggle and status text to the section's type scale Gutters: - the run area and the preview canvas now sit 40px off the rule, the gutter the settings column has off the page edge * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: give the dataset preview per-image hover and remove, and wrap it onto rows - each thumbnail is its own button, so it can carry an X that removes just that image; the strip was one button and could not nest a second control - hover lightens the image instead of drawing a border - wraps onto more rows rather than scrolling sideways, which also keeps the bottom corners rounded - samples 12 thumbnails, up from 8, now that a second row is available - a delete refreshes the panel's dataset counts * Studio: show the workflow icon beside the Images and Video headings Images reads it off WORKFLOW_TABS, so it is the same icon the sidebar submenu shows and it follows the active workflow. Video's heading is also Create, so it takes the Create icon rather than the nav row's film slate. * Studio: reject a duplicate LoRA id on the diffusion load path too DiffusionGenerateRequest already refuses a repeated adapter id, with a comment naming the hazard: _resolve_lora_set suffixes colliding adapter names, so the same id resolves the SAME adapter twice and set_adapters stacks both copies past the per-adapter weight bound. DiffusionLoadRequest bounds only the list length, and it matters more there. Generation-time stacking spoils one image; on the load path the adapters are baked into the quantized build before compilation, so the unintended combination rides every image until a reload. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: teach the GGUF reuse test double about holds_no_vram The resident-GGUF fast path consults llama_backend.holds_no_vram before asserting CHAT ownership, but two tests hand it a SimpleNamespace built before that attribute existed, so they raised AttributeError and returned a 500. A real LlamaCppBackend exposes it as a property; the doubles now carry it too. Product code is unchanged: these pass on main and were the branch breaking its own test, not a defect in the guard. --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: michaelhan <michaelhan2050@gmail.com> Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: Michael Han <107991372+shimmyshimmer@users.noreply.github.com> | 1 个月前 | |
Studio: add image generation, editing workflows and LoRA training with Unsloth GGUFs (#6763) * Tighten comments in the image stack tests and scripts * Close video single-file, training reservation, and image mount-resume gaps Route on-device single-checkpoint video folders through the single_file loader: a bare local .safetensors directory (no model_index.json) is advertised as a pipeline with no filename, so validation rejected it before it could load. Reinterpret the pick as a single_file load of the sole checkpoint, mirroring the image load route. Treat a reserved-but-not-yet-spawned LLM training start as active in is_training_active() so /images/load, /video/load, and /diffusion/start cannot race the reserved run for VRAM during the pre-spawn free window. Mirrors the diffusion training service reservation. Resume an in-flight image generation on the Images page mount: probe generate-progress, re-enter the poll loop, and refresh the gallery on completion so a run started elsewhere is reflected and its saved image appears without a manual refresh. Seed resident image defaults from the resolved base_repo rather than a possibly path-shaped repo_id so the first resident generation uses the right recipe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Publish image generation active state before pre-denoise setup generate() assigned self._gen only at the pipe() call, after deferred compile, LoRA resolution/application, and ControlNet download/build had run. Across that setup window generate_progress() reported inactive even though _generate_lock was held, so a reloaded page's mount probe showed idle and let a second generate queue behind the first. Publish an active step-0 _GenState the moment the generation lock is acquired, before the setup work, and clear it in the outer finally so a setup-time error cannot leave the UI stuck active. Mirrors the video backend's queued phase and the training start guard. * Studio: fix diffusion install ownership, dataset upload atomicity, gallery pagination, and teardown races install_sd_cpp_prebuilt: only write the .unsloth-studio-owned marker when the install created the target directory or it was empty. Adopting a pre-existing, unowned, non-empty directory (a user's own stable-diffusion.cpp checkout) made it eligible for the uninstaller's recursive delete. routes/training upload: make the multi-file promotion transactional. Back up each displaced original and roll every destination back on any failure, so a mid-loop rename error can no longer partially overwrite the live dataset. routes/training _resolve_dataset_folder: reject a symlinked dataset directory and prove the resolved folder stays under the datasets root, so image read/caption/delete cannot escape the root through a link. routes/training delete: escape glob metacharacters in the thumbnail filename so deleting an image named like [ab].png removes only its own thumbnails. image_gallery / video_gallery listing: filter records against the response schema inside the pager via a valid callback, so offset/limit/has_more all count over accepted records. A leading schema-invalid record no longer returns an empty page with has_more=true and stalls infinite scroll at offset 0. image_gallery / video_gallery save: publish via a temp file plus atomic rename (the sidecar is the video pair's commit marker) and clean up on failure, so a partial write never surfaces a truncated PNG or strands an orphan MP4. diffusion_train_common discovery: treat an empty caption sidecar as a metadata tombstone that still falls through to the dreambooth instance prompt, so clearing every metadata caption no longer fails with no captioned images found. diffusion backend unload: wait for an in-flight denoise to exit before tearing down process-wide patches and state, mirroring the load path. diffusion_engine_router: serialize the whole check/unload/publish transition so a concurrent selection cannot return the engine being unloaded. uninstall.ps1: gate the default sd.cpp process stop on the owner marker so a user's own sd-server is not terminated for a directory we then keep. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reject native batch seeds outside the JSON-safe range * Refuse sd.cpp install into unowned non-empty target dir When the install target already exists, is non-empty and lacks the .unsloth-studio-owned marker (a user's own stable-diffusion.cpp checkout, or unrelated files beside a custom Studio root), install() previously still extracted the release into it. Skipping the ownership marker only stopped the uninstaller from deleting the directory; extraction still merged binaries into the user's working tree and could overwrite same-named files. Fail up front with a clear message pointing the user at a fresh/empty location before any download or extraction, leaving their directory untouched. Update the ownership test suite to assert the refusal. * Studio: gate dataset uploads on the symlink check and surface local video single-file checkpoints * Tighten comments and docstrings added by the image-generation fixes * Studio: close arbiter load-registration race and surface native progress + local pipeline folders Publish native sd.cpp generate progress (_gen) before LoRA resolution so a reload probe reads active during setup, matching the diffusers path. Register the diffusion/video GPU load under the arbiter lock (acquire_for now takes a register callback) so a competing acquire cannot evict an owner before its load is marked in-flight and let two loaders allocate VRAM at once. Admit local diffusers pipeline folders (root model_index.json, weights in component subdirs) in the local model scan so they reach task tagging and the On Device picker. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: include marker-owned custom sd.cpp roots in the uninstall stop scan; harden Pester install against the nuget.exe PSGallery bootstrap * Studio: surface local pipeline scan roots, tag single-file checkpoints by filename, mark companion-only pipelines partial - _scan_models_dir: admit a scan folder that is itself a diffusers pipeline (root model_index.json, weights in transformer/ vae/ subdirs). _is_model_directory rejects such a root, so the child scan would list the component subdirs as bogus models and hide the real pipeline; treat the root as one model via _local_pipeline_index. - _local_is_diffusers / _local_model_task: include the sole checkpoint filename in the family-detection needles (_local_family_needles, resolved via resolve_local_single_file). A generically named folder holding one loadable qwen-image-*.safetensors / ltx-*.safetensors identifies its family only from the filename; the load route already resolves that file, so tag it or the task-scoped picker (which rejects task=null) hides the on-device model. - list_cached_models: mark a companion-only base snapshot partial. A GGUF image load prefetches the base repo's VAE / text-encoder / model_index.json but skips the transformer (the GGUF supplies it); the snapshot has a pipeline manifest yet is not a loadable BF16 pipeline, and _cached_repo_partial misses it. _repo_pipeline_missing_denoiser flags a pipeline snapshot whose transformer/ or unet/ component carries no weight, so the picker drops it instead of advertising it as fully on-device. * Studio: preserve foreign gallery files, force safetensors on remote ControlNets, and close dataset/seed/GPU gaps Gallery clear/delete now scope to Studio-owned files: image_gallery and video_gallery skip PNGs / MP4s without a readable recipe (a hand-dropped or orphan file the listing already hides), so clear() and a guessed-id delete no longer destroy files the gallery never surfaced. Remote ControlNets now force use_safetensors: a bare owner/name reaches from_pretrained without the base trust gate, and the Hub scan fails open when unavailable, so requiring safetensors closes the pickle deserialization vector. POSIX uninstall now stops resident sd-server / sd-cli under an owned sd.cpp root before removing the tree (marker-gated), mirroring the Windows stop-before-delete scan; a live native server no longer survives unlinking its binary. Diffusion dataset containment: the training-start read path and the discovery picker route bare names through the protected resolver, so a symlinked dataset is rejected / not advertised like the caption/delete routes already do. Uploads gain the inference decode guard (oversized real images 400 before OOMing the trainer) and dataset upload/caption/delete/import are blocked with 409 while a diffusion run is active. JSONL readers (trainer + routes) tolerate non-object JSON and invalid UTF-8 instead of raising AttributeError / 500. LoRA family compatibility is enforced in the shared resolver, not only the picker, so a direct API client cannot apply a mismatched-family adapter. GPU arbiter gains release_if so the image/video unload idle-check and release are atomic against a concurrent same-owner load's registration. Native batch recipes persist the base batch_seed and restore replays from it, so a native batch_index>0 image no longer advances its seed twice. FLUX.2-klein selects its sd.cpp text encoder by variant (4B -> Qwen3-4B, 9B -> Qwen3-8B) instead of the single family default. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: track the loaded GGUF filename so native companion resolution reproduces the load identity * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: gate local-pipeline image tagging on a real family; validate video sidecars before delete/clear * Studio: tighten image-generation fix comments and docstrings * Studio: gate gallery serve/export on ownership; keep image progress active until persisted; reserve diffusion training before the dataset scan * Studio: restore Reapply target on async image/video load errors; recheck training state before dataset commit; reclaim partial sd.cpp installs on retry Images/Video: a background model load that fails AFTER starting (error/eviction during download) leaves the previous pipeline resident, but handleLoad had already overwritten lastLoad.current with the failed pick, so "Reapply to loaded model" reloaded the failed model. Carry the prior Reapply target into the poller and restore it on the async error/null paths, mirroring the quant rollback. Training: an in-flight diffusion dataset upload passed _require_diffusion_dataset_mutable() at entry but could still commit files after a concurrent /diffusion/start reserved the training slot, mutating the dataset underneath the trainer. Re-check the interlock immediately before the commit phase; a 409 there leaves the staged temps for the finally to clean. sd.cpp install: an interrupted extraction (disk full, killed process, a raising post-extract cudart fetch) left the target non-empty with no owner marker, so the next lazy install tripped the "not a Studio-managed directory" refusal and wedged native install. Write the ownership marker before the partial writes when the target is reclaimable, so a retry recognises the debris as ours and re-extracts. * fp8 DiT quant: floor the dynamic activation scale with activation_value_lb An all-zero activation token row makes the dynamic per-row fp8 scale 0, which turns the quantized data to NaN and the render to black frames on torchao's plain-torch kernel path. The fused fbgemm/mslk quantize kernels clamp zero rows internally, so the bug only reproduces on machines without them, which is most user environments. Zero rows are real inputs, not a corner case: Wan 2.2 zero-pads its text conditioning, and Hunyuan-1.5 and Qwen-Image regenerate zero rows inside their transformer blocks every step. Pass activation_value_lb=1e-12 to Float8DynamicActivationFloat8WeightConfig whenever the installed torchao supports the kwarg (Float8Tensor rework, 0.13+), checked via inspect.signature so older torchao keeps exactly the current behaviour; the existing Float8MMConfig fallback chain is unchanged. Verified on GPU: with the forced plain-torch kernel path a zero-row input NaNs without the floor and stays finite with it, and end to end on HunyuanVideo-1.5 fp8 goes from a solid black frame (LPIPS 1.00) to a normal render (LPIPS 0.225); on Wan the floor matches the condition_embedder exclusion (LPIPS 0.211 vs 0.206). Same-seed renders with fused kernels present are unaffected, and pre-quantized fp8 checkpoints stay valid since weight scales are untouched. * Wire hosted pre-quantized DiT checkpoints into the image families Point prequant_repos for flux.1, flux.2-klein, flux.2-dev, qwen-image (int8 only there; fp8 is family-denied), z-image and krea-2 at the unsloth/<Model>-FP8 Hub repos carrying gate-validated int8 and fp8 transformer checkpoints, so the fast quant path loads the small pre-quantized file instead of materialising the dense bf16 transformer and quantising on device. Measured on FLUX.2-dev int8: build peak drops from 60.7 GB (dense + quantize) to 30.7 GB (hosted prequant), identical 30.7 GB resident after either path since loading a checkpoint is bit-identical to on-the-fly quantisation. The hosted repos name files <Model>-<SCHEME>.pt, so resolve_prequant_source now derives that model-name filename from the repo id (scheme suffix stripped case-insensitively) and carries the legacy transformer_<scheme>.pt as a fallback the resolver tries when the primary 404s, keeping older repos loadable. Wiring a repo also exposed a fallback hazard: with a prequant source present, the dense-fit preflight used to be skipped entirely, so a failed prequant download would fall through to the dense bf16 load the memory plan never budgeted, OOMing after eviction. The preflight now always runs and gates an allow_dense_fallback flag through _load_dense_quant_pipeline: a dense misfit still skips the fast path when no prequant exists, but with one it proceeds and a prequant failure raises to the GGUF build instead of loading dense. The same flag is set when the auto-policy replans an offloaded GGUF against a prequant-sized transient. Tests updated to the new filename convention plus new coverage for the derivation and the legacy-name fallback; the prequant-skips-refit test now asserts the re-check runs and forbids the dense fallback. Verified end to end on GPU: z-image int8 resolves the hosted repo, downloads the model-name file and renders (6.8s load, 5.9 GB peak). * Route krea-2 through its per-component loader on the transformer-quant fast path _assemble_pipe used Pipeline.from_pretrained for every family, but the krea repo ships transformers-5.x configs and no top-level tokenizer files, so the tokenizer dies with vocab_file=None. The pre-quantized checkpoint loaded fine and then the assembly crashed, dropping the load to the GGUF build, which krea-2 cannot take (Krea2Transformer2DModel has no from_single_file). Assemble per-component via load_krea2_pipeline like the pipeline-kind and single-file paths already do. Verified live: Krea-2-Turbo int8 and fp8 hosted prequant loads now assemble and render through the Studio images tab. * Keep Qwen-Image's text-stream linears bf16 on int8 (short prompts break torch._int_mm) Qwen-Image's MMDiT runs every text-stream Linear at M = actual prompt tokens: the Qwen2.5-VL embeds are not padded to a fixed length like FLUX's 512-token T5. A short prompt (13 tokens) or the near-empty negative prompt drives torch._int_mm below its M > 16 floor and the first denoise step raises 'self.size(0) needs to be greater than 16, but got 13' (measured on B200 through the Studio images tab). Add per-family int8 exclusions (txt_in, add_q/k/v_proj, to_add_out, txt_mlp) for qwen-image and qwen-image-edit, threaded through exclude_tokens_for_scheme(scheme, family) and the prequant checkpoint validation, so a checkpoint baked under the old token list is rejected and re-quantised instead of loaded crashing. The text stream runs at M = tens vs the image stream's M ~ 4k, so the exclusion costs nothing; the rebuilt hosted checkpoint gates 28/28 PASS with LPIPS mean 0.057 (was 0.069). * Harden the diffusion memory plan against transient free-VRAM undercounts A cold FLUX.2-dev int8 load on an idle 183 GB B200 planned offload=model (companions exceed budget) and silently served the GGUF as-is; the identical retry went resident and engaged the hosted prequant. The plan arithmetic was byte-identical across both loads (required 90,228 MiB, resident needs free of about 124 GB); the only divergent input was torch.cuda.mem_get_info, which is device-wide and instantaneous: a transient foreign CUDA context briefly held about 100 GB at the first snapshot, and the planner trusted that single read. Three changes: - settled_snapshot_device_memory: on cuda, synchronize + empty_cache (best-effort) and take the MAX free over up to 3 spaced reads. A transient can only shrink free, so the max rejects transient undercounts while a persistent tenant still caps every read. _plan_memory now uses it. - plan_fits_total_capacity + one replan retry: when the dense/prequant candidate fits TOTAL device capacity under the standard reserve and the 0.85 resident margin, an offload verdict can only stem from the free reading, so the loader re-snapshots and replans once before declining the fast path. Explicit balanced/low_vram modes skip the retry (they offload by mode). - diffusion.transformer_quant_declined log line with required/budget/free and the plan reasons, so the next decline is diagnosable from the server log (previously silent). Verified: cold FLUX.2-dev int8 first load in a fresh server now engages the hosted prequant resident (offload=none). * Add FLUX.2 Klein and FLUX.2-dev DiT LoRA training Register flux.2-klein and flux.2-dev in the DiT trainer following the upstream DreamBooth references: latents train patchified and batch-norm normalized from the VAE posterior mode, the packed forward reuses step-invariant position ids, and the guidance vector (3.5) is gated on the variant's guidance_embeds config. Conditioning stacks load per variant (Mistral via Flux2Pipeline for dev, Qwen3 via Flux2KleinPipeline for Klein) and are encoded and freed before the transformer lands on the device. The fused single-stream to_qkv_mlp_proj joins the attention projections in the LoRA targets; the single-stream out projection stays dense because its to_out suffix would also match the double-stream ModuleList container. Wire both families through the training registry (family set, labels, VRAM notes, rank 16 / lr 1e-4 defaults, bf16-only preflight), mark them trainable with train base repos in the family registry, add FLUX.2-dev to the gated-repo token check, and trust both official bases for training downloads. Verified on B200: 30-step klein int8 (19.6s) and nf4 (20.9s) and dev int8 (52.0s) runs train with finite decreasing loss and the saved adapters apply on the bf16 base pipeline (weight 0 reproduces the base image exactly, weight 1 visibly restyles it). * Support LoRA adapters on torchao int8/fp8 quantized image pipelines Adapters are baked at load time: they attach to the dense transformer, then quantize_ converts only the frozen base linears (the lora_ side path is excluded by name), then the loader compiles. Post-quant PEFT injection is not possible on a manually quantized module, so the prequant shortcut is skipped for a baked load and the memory plan is sized for the dense build (force_dense on the quant candidate). At generation time the baked topology is frozen: weight tweaks and disabling (scale 0 reproduces the quantized base exactly) go through set_adapters, while adding or removing adapters returns a clean 400 telling the client to reload with the new selection. supports_lora now returns True for int8/fp8 diffusers loads (checked before the gguf-kind early return, since the quant fast path keeps the picker kind); nvfp4/mxfp8 and GGUF-via-diffusers stay blocked. The load request model takes an optional loras list, threaded through begin_load on both engines (native ignores it and keeps applying LoRA at generation). Verified end to end on GPU: Z-Image GGUF picker + int8 + trained adapter loads through the API, bake marker logged, weight 1.0 vs 0 renders differ visibly, weight 0.5 accepted live, unknown adapter rejected as 400. Affected suites: 296 passed. * Add FLUX.1 Krea dev to the image model catalog Krea's guidance-distilled FLUX.1-dev finetune keeps the exact dev layout, so it runs under the existing flux.1 family unchanged. Wire it up end to end: - Catalog group with the gated official bf16 pipeline and the open QuantStack GGUF quants; the gated artifact is skipped on auto-routing when undownloaded. - Trust the official repo for non-GGUF from_pretrained loads, next to the other black-forest-labs bases. - Generation defaults: 28 steps at guidance 4.5 per the model card. The generic "krea" defaults key (Krea-2-Turbo's 8-step no-CFG recipe) used to swallow the id, which would have produced garbage output; the new flux.1-krea key precedes it on both the backend table and the images page table. - The flux.1 prequant checkpoints are schnell-based; the loader's baked base_model_id validation refuses them for the Krea-dev base, so int8/fp8 requests dense-quantize instead (covered by existing prequant tests). * Resolve pre-quantized checkpoints per base variant One family entry covers several published variants whose weights differ (flux.1: schnell, dev, Krea-dev), but prequant resolution was keyed on (family, scheme) alone, so only the default base could ever be served: the loader's baked base_model_id validation correctly refused the schnell checkpoint for dev and Krea-dev bases and every such load paid the dense download plus on-the-fly quantise. Add an optional prequant_variant_repos table on DiffusionFamily as (base_repo, scheme, repo_id) triples and thread the resolved base repo through resolve_prequant_source / usable_prequant_source and their three call sites (load fast path, memory-plan probe, auto-policy candidate). A base without its own entry keeps returning the family default, preserving the existing refuse-then-dense behavior exactly. Wire the flux.1 variants: the gate-validated unsloth/FLUX.1-dev-FP8 checkpoints (built in the earlier campaign but never reachable) and the new unsloth/FLUX.1-Krea-dev-FP8. * Add the Lumina Image 2.0 family to the image catalog Alpha-VLLM/Lumina-Image-2.0 is a 2.6B single-stream DiT with a Gemma2-2B encoder and a standard 16-channel VAE, all transformers-4.x-compatible, so the generic from_pretrained pipeline path loads it as a new lumina-2 family: - Family entry (Lumina2Pipeline / Lumina2Transformer2DModel), aliased to lumina-image-2.0 / lumina-image-2 / lumina2. No bare lumina alias: Lumina-Next checkpoints are a different arch and must stay unknown rather than crash mid-load. bf16-only upstream, so the fp16 fallback stays off like z-image. - Trust the official repo for non-GGUF loads; bf16 component table entry (ships fp32, ~5.2 GB transformer + 5.2 GB encoder bf16-resident). - Generation defaults 50 steps / guidance 4.0 per the model card, and the generate call passes the card's cfg_trunc_ratio=0.25 itself (family-gated, signature-gated): the pipeline default (1.0) runs the CFG double-forward on every step and oversaturates output. - Catalog group with the single ungated bf16 pipeline artifact (11 GB resident) plus routing assertions; images page defaults row. - No GGUF artifact: none exists upstream (only finetune/LLM quants), so the dense transformer_quant fast path (GGUF-kind-only) stays unreachable for now. Offline probes of the future prequant campaign: int8 and fp8 both engage and render cleanly (fp8 LPIPS 0.11 vs bf16, int8 0.33 from 50-step trajectory drift with intact quality), so neither scheme is family-denied. * Wire the hosted Lumina Image 2.0 int8/fp8 checkpoints Gate-validated against same-seed bf16 renders (28/28 pairs per scheme, zero failures): int8 LPIPS mean 0.146 / SSIM 0.937, fp8 LPIPS mean 0.116 / SSIM 0.946. Uploaded to unsloth/Lumina-Image-2.0-FP8 following the existing checkpoint repo conventions. * Add the HunyuanImage 2.1 family to the image backend The hunyuanvideo-community diffusers mirror carries the full stack in standard layout: a 17B dual-stream DiT (32.5 GB bf16), a Qwen2.5-VL text encoder, a ByT5 glyph encoder, the 32x HunyuanImage VAE, and guider/ocr_guider components (AdaptiveProjectedMixGuidance) that diffusers 0.39 loads natively, so the generic from_pretrained pipeline path covers everything with no per-component assembly. Family notes: - The call's guidance knob is distilled_guidance_scale (there is no guidance_scale kwarg), so cfg_kwarg routes the UI value there; real CFG runs inside the repo's guider at its baked scale. Defaults follow the card recipe: 50 steps, 3.25. - 2K-native: verified live at both 1024 and 2048. - Coexists with the HunyuanImage-3.0 structured exclusion (3.0 has no diffusers pipeline and stays excluded with its stated reason). - int8/fp8 dense quantization verified live (LPIPS 0.186 both vs same-seed bf16); a short prompt does not trip the int8 torch._int_mm minimum on this arch, so no family exclude entry is needed. - bf16 component table for the memory planner: (32.5, 16.3, 0.8) GB. * Surface HunyuanImage 2.1 in the image model catalog Catalog group with the open bf16 mirror pipeline (~50 GB resident, so a bare click on a consumer card routes to the QuantStack GGUF quants, which load and render through the generic GGUF path, verified live) plus the images page defaults (50 steps, guidance 3.25 feeding distilled_guidance_scale). * Add the HiDream-I1 family to the image backend A 17B MoE DiT (16 double + 32 single layers, 4 routed experts) with four text encoders, on HiDreamImagePipeline (diffusers 0.39). One family covers the open Full / Dev / Fast repos (same arch); per-variant generation defaults follow the upstream inference recipes (Full 50 steps at guidance 5, the distilled Dev 28 and Fast 16 guidance-free). The repos name a Llama-3.1-8B text_encoder_4 in their model_index but do not ship its weights; the official example passes the gated meta-llama repo in by hand. The loader instead assembles the component from the open unsloth mirror (byte-identical weights, already inside the non-GGUF trust gate), injected at the three pipeline from_pretrained sites, with output_hidden_states matching the official example. Memory planning counts the assembled TE4: 34.2 GB DiT + 28.8 GB encoders, ~63 GB bf16-resident. * Surface HiDream I1 in the image model catalog One catalog group with the three official bf16 pipelines (Full, plus the Dev and Fast distillations as labeled artifacts) at their ~63 GB resident size, so auto-routing keeps this a datacenter-GPU pick. city96's GGUF is deliberately not wired: the GGUF path would need the same Llama TE4 assembly for very small demand. Images-page defaults mirror the backend table with the variant keys ahead of the generic hidream key. * Pin the measured HiDream quant verdict in tests int8 and fp8 both engage and render cleanly on this family, including short prompts on int8: the routed MoE expert Linears only ever see the concatenated image+text stream (M far above the torch._int_mm minimum), so no deny entry and no family exclude tokens are warranted. Assert that so a future table edit cannot silently regress the measured behavior. * Wire the hosted HunyuanImage 2.1 int8/fp8 checkpoints Verified bit-identical to on-the-fly quantize: all 1264 state dict tensors (456 quantized) dequantize equal between the loaded checkpoint and a fresh quantize_ pass, so quality matches the runtime Dtype path exactly. Same-seed LPIPS suite means (0.35 int8 / 0.28 fp8) blend trajectory divergence with this family's own run-to-run nondeterminism (identical weights and seed reproduce a 17/255 mean pixel delta through the 50-step guider pipeline); per-case hard checks pass and the drift is compositional, reviewed visually. Uploaded to unsloth/HunyuanImage-2.1-FP8. * Fix silent LoRA drop and wasted transformer prefetch on GGUF quant loads Two live-test findings on the images load path: - transformer_quant with baked LoRAs, when the dense quantized build is declined for memory or fails: the load completed as a plain GGUF with the adapters silently dropped (HTTP success, supports_lora=false after the fact) -- wrong output with no signal. The load now fails with the recovery options (drop the adapters, free VRAM, or pick a smaller model). Weight-0 adapters still count as no bake request, and the plain no-LoRA decline keeps its silent GGUF fallback. - A fresh GGUF load on a small GPU prefetched the base repo's full bf16 transformer shards (~47 GB on Qwen-Image) because the dense-quant prefetch widening only checked scheme viability, not whether the device could ever hold the candidate resident. Gate the widening on total device capacity (reserve + 0.85 margin, the plan_fits_total_capacity bar) so a card that is certain to decline the dense build never pays the download; capable devices keep the prefetch. * Fix video progress under-reporting during load and generate Two live-test findings on the video progress endpoints: - load-progress downloaded_bytes froze mid-download: the counter used scan_cache_dir, which skips in-flight *.incomplete blobs, so it sat at the last completed blob for the whole multi-GB shard pull while the disk kept filling. Count the repo's cache directory directly (completed plus incomplete blobs, snapshot symlinks skipped so nothing is double-counted). - generate-progress reported total_steps=null / fraction=0 while step advanced: the video API only carried the native total field while the image API exposes total_steps and fraction, so one poller could not work against both. Derive the image-compatible aliases in generate_progress and declare them on the response model; the native total stays for back-compat. * Wire the hosted HiDream I1 int8/fp8 checkpoints Gate-validated: all 28 per-case pairs pass per scheme (LPIPS suite means 0.291 int8 / 0.278 fp8, in the 50-step trajectory-divergence band; CLIP delta means 0.007-0.008), and the int8 checkpoint is verified bit-identical to on-the-fly quantize across all 1615 state dict tensors (1073 quantized, max abs diff 0.0). Uploaded to unsloth/HiDream-I1-Full-FP8. * Add a pre-cast text-encoder loader for the layerwise fp8 scheme The runtime text_encoder_quant=fp8 path downloads the full bf16 text encoder and layerwise-casts it in place on every fresh load. For the heavyweight encoders (LTX's Gemma3-27B ~50 GB, FLUX.2-dev's Mistral-24B ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) that download dominates load time on a fresh machine. diffusion_te_prequant.py loads a pre-cast fp8-storage state dict instead: meta-init the encoder skeleton from the checkpoint's te_class, load_state_dict(assign=True), rebuild on CPU if non-persistent buffers stay on meta, then re-apply the same layerwise cast to install the upcast hooks. The cast is a deterministic storage transform, so the loaded encoder is bit-identical to dense-load-then-cast by construction. v1 hosts the layerwise fp8 storage scheme only: its state dict is plain tensors (torch.load(weights_only=True), no pickle execution). The dynamic-compute schemes (fp8_dynamic, int8, nvfp4) build torchao subclass wrappers at runtime and are deliberately not hosted. Checkpoints validate format, scheme, component and base_model_id before use and any problem falls back to the dense download and cast. Local path overrides reuse the DiT prequant allowlist env var. Families opt in via a new te_prequant_repos (scheme, component, repo_id) field on both DiffusionFamily and VideoFamily; the field defaults empty so nothing changes until a gate-validated artifact is wired. * Inject hosted pre-cast text encoders during pipeline assembly Wire te_prequant_pipe_kwargs into the three pipeline assembly sites: the diffusion full-pipeline branch, the diffusion transformer-only and GGUF branch (where the companion TE is the big remaining download), and the shared video assembly path before the pipeline/component split. Injection is gated exactly like the runtime cast (mode normalized to fp8, device supported, family not denied), so it can never engage where quantize_text_encoders would not; the later quantize_text_encoders call re-applies the cast idempotently and keeps status reporting truthful. With no hosted checkpoint configured the call returns {} and assembly loads the dense encoder as before. * Add the pre-cast text-encoder checkpoint builder Applies the runtime layerwise fp8 storage cast to a model's dense text encoder once and saves the cast state dict with baked metadata (format tag, base_model_id, family, scheme, component, te_class, versions) in the layout diffusion_te_prequant.py validates. Resolves the encoder class from the checkpoint's config.architectures so the recorded te_class matches what the pipeline instantiates. CPU-runnable: the cast touches storage dtypes only. * Test the pre-cast text-encoder load path Hermetic CPU coverage for diffusion_te_prequant: the checkpoint filename convention, family-table resolution by scheme and component with malformed entries skipped, resolution priority (path override, hosted repo, none) and the fp8-only scheme gate, the checkpoint validation matrix (wrong format, missing state_dict, wrong scheme, wrong component, wrong or missing base_model_id) with base case folding, the local-path allowlist refusal and missing-file fallback, and the assembly injection gating (mode, hosted entry, device support, family deny, load failure, successful injection). Also pins the te_prequant_repos field on both family dataclasses and that no family ships a hosted TE checkpoint until the campaign wires one. * Fix pre-cast TE checkpoint loading and engagement reporting Two bugs found while building the hosted checkpoints: - The builder recorded torch.__version__ (a TorchVersion object) in the checkpoint metadata, so torch.load(weights_only=True) rejected every artifact and the loader silently fell back to the dense download. Record plain strings. - Re-applying the layerwise fp8 cast to an injected pre-cast encoder raised on the duplicate hook registration, making quantize_text_encoders report the engaged cast as failed (status showed no TE quant while the encoder ran fp8). _cast_fp8 now returns early when the hooks are already installed. Also corrects the LTX TE size note: Gemma3-12B stored fp32 (~49 GB), not 27B. * Wire the hosted pre-cast fp8 text encoders qwen-image and flux.2-dev (diffusion) and ltx-2 (video) now resolve a hosted pre-cast fp8 text encoder from their unsloth -FP8 repos: - unsloth/Qwen-Image-FP8: Qwen2.5-VL-7B, 16.6 GB dense -> 8.8 GB - unsloth/FLUX.2-dev-FP8: Mistral-Small-24B, 48.0 GB dense -> 24.7 GB - unsloth/LTX-2-FP8: Gemma3-12B, 48.7 GB fp32 store -> 13.2 GB Every checkpoint verified bit-identical to dense-load-then-cast (729 / 585 / 1066 tensors, zero mismatches) and smoke-tested through the real backends with the repo engagement marker. Tests cover the wired entries, the resolver filenames, builder metadata weights_only survival, and the idempotent re-cast. * Report the compute dtype on fp8-cast encoders and inject the pre-cast TE on the dense fast path Two more findings from the hosted-TE GPU smokes: - Module.dtype reports the first floating parameter, which after the layerwise fp8 cast is the fp8 STORAGE dtype. Flux2 derives its prompt embed and latent dtypes from encoder.dtype and feeds them to randn_tensor, which has no fp8 kernel, so ANY flux.2 load with text_encoder_quant=fp8 crashed at generation (pre-existing, runtime cast included). The cast now swaps in a subclass whose dtype property reports the compute dtype; forward behaviour is unchanged. - The dense transformer_quant fast path assembles companions through _assemble_pipe, which never received the pre-cast TE injection, so the hosted encoder engaged on full-pipeline and GGUF builds but not on the fast path. Threaded through like the other two branches. Verified live on B200: qwen-image (full pipeline), flux.2-dev (GGUF picker with int8 DiT prequant), ltx-2 (video backend) all engage the hosted TE, render non-black, and report text_encoder_quant=fp8 truthfully. * Key the fp8 cast idempotency on an explicit completion marker Hook presence alone cannot distinguish a legitimately pre-cast text encoder from leftover hooks after a cast that failed mid-pass, so the early return now requires the completion marker _cast_fp8 sets once the hooks are fully installed. Leftover partial state keeps failing closed. Also tolerates non-Module encoder doubles in the hook probe and the dtype override. * Extend the fp8 TE quant to HiDream's Llama text_encoder_4 The generic quantize_text_encoders pass only covers text_encoder.._3, so HiDream's HEAVIEST encoder (Llama-3.1-8B TE4, 16.1 GB bf16) always stayed dense. TE4 is assembled separately (hidream_te4_kwargs), so the fp8 path now lives there: when the requested TE quant is layerwise fp8 and the device/family qualify, TE4 prefers the hosted pre-cast checkpoint (unsloth/HiDream-I1-Full-FP8, 8.6 GB) and falls back to dense-load-then- cast; a mid-pass cast failure reloads a fresh dense encoder instead of shipping partial state. The pre-cast loader and builder gain config_subfolder/config_overrides for standalone encoder repos whose config sits at the root and whose pipeline needs forward flags (output_hidden_states/attentions). Verified on B200: bit-identity 291 tensors (225 fp8, 0 mismatches), hosted checkpoint engages through the real backend (marker + status fp8), load 24.3 s vs 48.0 s dense, LPIPS 0.133 mean over 3 same-seed pairs vs the dense-TE render (gate 0.25), non-black frames. * Correct the ltx-2 resident TE estimate to the bf16 cast size The memory plan's bf16_components_gb held 50.4 GB for the LTX text encoder, which is the fp32 hub store of Gemma3-12B (~49 GB download), not what sits on device: the pipeline loads it torch_dtype=bf16, ~24.4 GB resident. The 26 GB over-estimate pushed the auto plan toward offload on cards that fit the real footprint. Comments and the size-table test now pin the resident semantics. * Host pre-cast fp8 text encoders for four more families Round 2 of the hosted TE set, each bit-identical to dense-load-then-cast and gated through the real backend (marker + status fp8 + same-seed LPIPS vs dense TEs): - FLUX.1 T5-XXL (text_encoder_2): 9.52 -> 5.90 GB, one artifact for schnell/dev/Krea-dev (T5 shards byte-identical across all three, verified sha256). 220 tensors, 144 fp8, LPIPS 0.109. - Lumina Gemma2-2B: fp32 hub store 10.46 -> 3.20 GB (3.3x download cut). 288 tensors, 182 fp8, LPIPS 0.041. - Z-Image Qwen3-4B: 8.04 -> 4.41 GB. 399 tensors, 252 fp8, LPIPS 0.112. NOT shared with flux.2-klein-4B: klein retrained layer 35's MLP (verified tensor diff, maxdiff 0.86), so klein hosts no entry. - Krea-2 Qwen3-VL-4B: 8.88 -> 4.83 GB. 713 tensors, 460 fp8, LPIPS 0.082. The constructor-assembled krea pipeline takes the encoder directly (load_krea2_pipeline text_encoder kwarg); the loader remaps 5.x rope_parameters and re-ties weights after assign so the rebuilt encoder matches the builder's structure. HunyuanImage 2.1 reuses the Qwen-Image artifact outright: its Qwen2.5-VL text encoder is byte-identical (every shard sha256, 16,584,414,544 bytes), recorded in the new component-level base-equivalence table the checkpoint validator consults. The injection loop now covers text_encoder.._3 so a family can host several components. Live check: LPIPS 0.123 vs dense. * Report the fp8-cast compute dtype without swapping the encoder class The dtype override swapped encoder.__class__ to a dynamic subclass, which breaks transformers' kwargs-based output recording: a fp8-cast Qwen3VLModel stopped returning hidden_states and every krea-2 generation with text_encoder_quant=fp8 crashed at encode_prompt (regression from the HiDream TE4 change; caught by the krea hosted-TE live smoke). The override is now a property shadowed on the ORIGINAL class that prefers a per-instance compute-dtype attribute, so class identity is preserved and uncast instances keep the stock behaviour. The idempotency test now pins exact class identity and the uncast-sibling fallback. * Pass the calibrated distilled sigma curve to LTX-2.3 8-step runs The 22B distilled DiT was trained against ltx_core's fixed DISTILLED_SIGMA_VALUES, but the diffusers scheduler derives 8-step spacing from resolution-shifted flow matching and lands far off at every reachable mu (second sigma 0.945-0.981 vs 0.99375, tail 0.37-0.61 -> 0.1 vs 0.725 -> 0.42 -> 0). At the distilled default step count the backend now passes the list verbatim, neutralising the scheduler's dynamic shift and terminal stretch for the call (they distort even explicit sigmas) and restoring them afterwards. Other step counts and the dev/base DiT keep the scheduler's own spacing. Live-verified on B200 through the video branch backend: the scheduler holds the exact curve after an 8-step distilled GGUF generation, config restored, healthy clip. Also reword the transformer_quant resolved reason to the measured reality: quant halves resident weights and hosted checkpoints cut load time, while per-step speed is roughly bf16 parity. * Pin the fp8 weight-quantize kernel against silent MSLK switching torchao's Float8Tensor KernelPreference defaults to AUTO, which switches the weight-quantize kernel to MSLK whenever an mslk package is importable on sm90+. Measured on B200: that changes fp8 scale rounding bitwise (8/8 FLUX matrices differ, scales ~55 percent of bytes), so a box that merely gains mslk would break the hosted-prequant bit-identity invariant; the mslk path is also slower under torch.compile (opaque extern call blocks inductor's quantize fusion, FLUX.1 fp8 e2e 1.149 to 1.624 s). Pin KernelPreference.TORCH explicitly, matching current no-mslk behaviour bit for bit; signature-gated for older torchao. GPU-smoked (finite, rel err 0.037) and pinned by test. * Shift Qwen-Image training sigmas to the inference distribution Qwen-Image's scheduler skips its static shift under use_dynamic_shifting, so the DiT trainer was drawing UNSHIFTED uniform-schedule sigmas for it (mean sigma 0.50) while inference always runs the exponential mu = log 3 shift plus the shift_terminal 0.02 stretch. Add a flow_shift config lever: "auto" (the new qwen-image default) rebuilds the training sigma table through the scheduler's own time_shift and stretch_shift_to_terminal so the draw matches the inference distribution exactly (mean sigma 0.72); a numeric value applies the standard linear shift s*u/(1+(s-1)*u); 1.0 keeps the historical identity behavior and stays the default for FLUX, Z-Image and Krea 2. The model timestep conditioning follows the shifted sigma, gathered in fp32 so bf16 rounding never skews it. Also wire two opt-in levers with off defaults: cfg_dropout (per-sample empty-prompt conditioning dropout, encoded alongside the captions before the text encoders are freed) and weighting_scheme="bell" (bsmntw-style mid-schedule Gaussian loss weighting normalized to mean 1). Verified with two 80-step rank-8 bf16 LoRA runs on Qwen/Qwen-Image (identity vs auto, same seed): both converge with finite decreasing loss and produce coherent same-seed previews. Unit tests cover the exact transform, the shifted sampling distribution, per-family defaults and config plumbing. * Add LoRA EMA, a persistent conditioning cache, and aspect bucketing helpers diffusion_train_extras hosts the opt-in training extras: LoRAEMA shadows only the trainable adapter params (warmup-ramped decay, default 0.99, exported as a second adapter under output_dir/ema), PersistentConditioningCache stores latent posterior stats and caption embeddings as safetensors keyed by content hash + family + resolution, and the aspect-ratio bucketing helpers group mixed-aspect datasets into same-area divisor-snapped shapes. The DiT trainer wires the first two behind config flags that default to the current behavior: ema_decay (0 disables) and cond_cache_dir (None disables). A fully warm cache skips loading the VAE and text encoders entirely; a cache hit is bit-identical to a fresh encode, including the per-channel qwen latent normalization. Also fixes the stale _gather_sigmas call in the perf test that still passed the scheduler instead of the sigma table. * Tighten torchao configs and note the FSDP2 design for the DiT trainer nf4 loads now enable double quantization (~0.4 bits/param off the frozen base scales at no fidelity cost), fp8 training uses the rowwise recipe when the torchao build ships it (per-row scaling confines the DiT activation outliers that a tensor-wide scale collapses), and the inference quant filter gains a per-scheme GEMM-tiling divisibility floor (16 for scaled_mm, 32 for MX blocks) so one ragged Linear cannot crash the first denoise after a clean quantize pass. plans/fsdp2_diffusion_design.md records the multi-GPU design: bf16/fp8 over FSDP2 with per-block units, LoRA attached before sharding, int8 out of scope (DTensor over the quantized subclass is undefined), per-family notes. * Batch diffusion inference with per-image seeds, an inference conditioning cache, and GGUF loader fixes Batched generation: /images/generate takes a prompts list (one image per prompt, txt2img only) or a seeds list (one prompt, one image per seed); the legacy batch_size path derives per-image seeds base..base+n-1 like the native engine. Every image gets its own torch.Generator so any batch member replays alone from its gallery recipe; the whole list runs as one forward by default with OOM backoff that halves a failed chunk, and an explicit batch_size caps images per forward. Validated 10-22x over serial engines on 32-image suites with LPIPS deltas within 0.002. Conditioning cache on the inference path: UNSLOTH_DIFFUSION_COND_CACHE_DIR (the inference sibling of the trainers' cond_cache_dir, same persistent store) wraps encode_prompt so repeated prompts skip the text-encoder forward entirely; verified bit-identical outputs. Bypassed while LoRA adapters are attached; tensor-argument calls pass through uncached. Compile cache: GGUF loads fingerprint their own bundles (quant=gguf, a different compiled graph than the dense family) and batched calls register every distinct (w, h, batch) chunk shape they ran, so the heavy GGUF batched warmups (~159 s at batch 32 on 12B-class, ~655 s on 20B CFG-batched) are paid once ever. GGUF loader: strip the sd.cpp model.diffusion_model. container prefix in the single-file converter; diffusers' FLUX.2 converter KeyErrors on it and the Qwen-Image identity mapping strands the model on meta. * Correct batched seed-replay docs to match measured behavior Same-seed images at the same batch shape are bit-identical; a solo regeneration with the recorded seed matches its batched rendition up to batch-size-dependent kernel numerics (mean abs pixel delta about 2.5/255, LPIPS delta under 0.002), not bit-exactly. The previous wording overclaimed bit-identity across batch shapes. * Note that batched bit-identity assumes a settled compiled graph The first generation issued while the deferred compile is still in flight can deviate transiently (observed once on a cold fp8 build: mean abs pixel delta 0.063/255); once the graph is settled, same-seed same-batch-shape images are bit-identical across runs. * Studio sidebar: Image03/FlimSlate icons, More flyout, Train row, New pills - Images uses Image03Icon and Video uses FlimSlateIcon. - New "More" row (MoreHorizontalIcon) opens a right-side flyout on click or hover holding Video, Recipes and Export; the close is delayed 180ms so the pointer can cross the gap. Its SidebarMenuButton deliberately takes `title` rather than `tooltip`: with `tooltip` the button returns a Tooltip root and DropdownMenuTrigger asChild would hand its ref to a non-DOM node. - Dropped the "Train" section heading; Train is now a top-level row between Images and More. data-tour="navbar" moves to the surviving nav group so the product tour keeps its anchor. - "New" pill beside Images and (inside the flyout) Video, via NavBadge. * Studio sidebar: match flyout rows and New pills to the existing scales - More flyout rows dropped their sidebar-row typography and size-icon override, which fought DropdownMenuItem's own scale (text-sm, gap-2.5, px-3 py-2 and size-4 icons) and rendered oversized glyphs and text next to the nav. - New pill reuses the brand "beta" badge recipe (nav-badge font, --ui-font-scale sizing, nav token colours) rather than hardcoded 9px values. - The More row's native title tooltip (an OS box on hover) is replaced by the app's Tooltip, wrapped around DropdownMenuTrigger so both triggers compose onto the same button, and shown only on the collapsed rail like other nav rows. * Settings: pin and reorder the sidebar navigation Adds a "Sidebar navigation" section to Settings -> Appearance, above the existing profile-menu customizer, with the same drag-to-reorder + switch UI. - New sidebarNav preference: one { id, pinned } entry per navigable row (projects, hub, images, train, video, recipes, export), array order = render order. Defaults match the shipped layout, so an untouched install is unchanged. - Unpinning moves a row into the More flyout rather than hiding it, so no page becomes unreachable. New chat and Search stay fixed as actions. - app-sidebar now renders from one navRows descriptor map, so a pinned row and its flyout counterpart cannot drift; the More row appears only when something is unpinned and highlights off whatever it actually holds. - Mirrored in the backend PersonalizationCustomization: without it the model's extra="ignore" would drop the field, and because sync replaces local state with the server's copy once customization is saved, the user's pin order would reset on the next sync. The validator dedupes and back-fills like sidebarMenu but preserves the client's order, since here order is meaningful. Frontend typecheck, i18n parity and catalog checks pass; 32 personalization tests pass, including a round-trip asserting a reordered list survives a save. * Sidebar customizer: drop the Search row, skip More for a lone item - Search is reached from the top bar, so it is no longer previewed as a fixed sidebar nav row; New chat stays. - More now appears only when it would hold two or more rows. A single unpinned row renders inline in its saved order position instead: a flyout wrapping one item costs a click and earns nothing. The customizer's More preview follows the same threshold. * Sidebar settings: hide a lone unpinned tab, match New chat icon, rename Profile menu - With exactly one tab unpinned, both More and that tab are dropped, so nothing is drawn for it (previously it rendered inline). The page stays reachable by URL. - The customizer's New chat preview uses PencilEdit02Icon, the icon the real row renders; Edit03Icon was a different glyph. - "Sidebar menu" is now "Profile menu", described as the shortcuts behind your name at the bottom of the sidebar, so it no longer reads as a second name for the navigation section above it. * Tighten comments in the new sidebar and delete-guard code * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio sidebar: keep the More row highlighted while its panel is open Moving the pointer into the flyout left the row unhighlighted while the panel stayed open. The row now carries data-menu-open, added to the nav hover selector list. Not data-state: the tooltip and menu triggers both write that attribute, so whichever lands last wins. * Images: use the shared pill toggle for Create/Train and pad the panels - Create/Train was the only segmented control on its own Tabs styling. It now uses PillTabs, the same control as the model picker and Hub toggles, pinned to the header row's 34px. PillTabs takes an icon per tab, so the inline-span workaround for TabsTrigger goes away. - pt-3 on both the Create and Train panels, which sat flush against the model selector row. * Images: make the workflow picker a dropdown instead of a 7-up strip Seven workflows in a 340px rail left ~48px each, so the labels crowded and the hints were only reachable as title tooltips. The strip is now a dropdown: the trigger shows the current workflow and its hint, and each row carries its own description. A row the loaded model can't run is disabled and shows the reason in place of the hint, so the gating explains itself. Adding a workflow no longer shrinks the others. * Images: workflow icons, hint under the trigger, more top room, unclipped Train cards - Each workflow carries an icon, shown on the closed trigger and on every row. - The trigger is one line (icon plus name). The selected workflow's description moved below it, where it reads like the Field hints further down the rail. - pt-6 instead of pt-3 on both Create and Train, so the cards clear the model selector row. - The Train right column scrolls while its cards use ring-1, which draws outside the box and was clipped at the scroll edges. p-px gives the ring room. * Images: one-line workflow rows, stronger trigger fill, roomier mode tabs - Dropdown rows are icon plus name only. The selected row's description already shows under the trigger, and a disabled row keeps its reason as a title. - Trigger fill moves to the bg-foreground/[0.07] dark:bg-foreground/[0.12] pair the hub cards use, so it reads against the card in both modes. - Description under the trigger goes from text-ui-10 to text-ui-11p5. - More horizontal padding on the Create / Train tabs. * Images: drop card borders for the composer shadow, keep scrollbars inside, use app controls in Train - Cards lose ring-1 for .panel-soft-surface: the composer's shadow in light, flat in dark, matching .chat-composer-surface and the menus. - Both rails now clip (overflow-hidden) with the scroller inside, so the scrollbar can't ride over the rounded corner. Same shape video-page already uses. - Train's 9 native selects become the app Select, so they no longer open an OS-native menu, and the native file input is hidden behind a Choose images button that reports the count. - Image previews use explicit 8-10px radii: this theme sets --radius to 1.1rem, so rounded-md was 15.6px and the thumbnails read as circles. * Images: one card for controls and preview, chat sliders, wider softer shadow - Controls and preview were two floating cards; they now share one card split by a divider. The Advanced dock stays separate since it toggles. - SliderField wraps Chat's ParamSlider, so the sliders match Chat (label row with the value, full-width neutral track) instead of a green track with a spin box. All 14 call sites keep their props. - panel-soft-surface goes from 0 2px 8px -2px /0.16 to 0 4px 22px -6px /0.10: lighter, spread wider. * Images: flat Create and Train panes, hover-only scrollbars, tidier Train dataset step Both Images tabs now sit on the page background like the Hub: no card, no shadow, no bounding box. A single rule divides the controls rail from the preview canvas (Create) and from the run area (Train), and the settings and previous-runs sections read as panes rather than nested cards. Also: - Scrollbars in these panes use the existing hover-scrollbar recipe, so the thumb only shows while the pane is hovered. - Workflow rows explain themselves with a tooltip after a short hover, which also works on disabled rows, and the descriptions are much shorter. - Training images rows are name plus image count; the license stays on the example card. - The upload step loses its dashed box, the buttons match the sizes around them, and Upload only appears once files are picked. - The empty preview uses the same icon as the Images nav item. * Images: full-height panes, wider settings rail, Create/Train offset from the selector The rule between the panes now runs the whole page height (the row drops its bottom padding and each pane pads its own content), the settings rail is wider on both Create and Train, and the Create/Train switch sits further right of the model selector. * Images: put both tabs on the Hub's centered measure Top bar and content now share mx-auto max-w-1100 with px-5 / sm:px-8, so Create and Train sit at the same width and position as the Hub instead of running edge to edge. * Images: restore the top bar position, drop the panes lower under it * Images: center the mode switch, flip the arrow with the orientation, app tooltips everywhere The Create/Train switch is centered on the page instead of trailing the model selector, with wider buttons. The flip control's arrows now rotate with the orientation and its label says which way the flip goes. Every native title tooltip on the page is now the app's tooltip, so they all get the rounded surface instead of the OS box. * Images Train: plainer field text, no green buttons, columns that stop colliding - The dataset name, trigger prompt, adapter name and custom base fields now say what they are in plain words instead of leaning on example values. - Import, Upload, Back, Back to settings and Train another are outline buttons, not green ones. - Example thumbnails are landscape tiles, so photos are not cropped to chunky squares. - Settings cells get min-w-0 and the select value truncates, so a long option like the nf4 label no longer widens its column into the next one. - The number stepper sits a little further in from the field edge. - Create and Train are wider. * Images Train: roomier example cards with Import on the thumbnail row * Video: same treatment as the Images tabs - No cards: the rail and the canvas sit on the page background, divided by a rule that runs the full page height, on the Hub's centered measure. - Wider rail, chat's sliders, hover-only scrollbars. - Every native title tooltip is now the app's tooltip, including the clip cards. - Reapply and Cancel are outline buttons, the empty state uses the Video nav icon, and the clip tiles are less rounded. * Images and Video: narrower generation rail, matching Train headings Create and Video rails go from 392px to 368px. Train a LoRA and Training settings are now the same size and both in the heading font: the h2 already picks it up from the base rule, so the settings header opts in with font-heading and the weight that rule pins. * Images Train: shorter copy throughout Family notes, example descriptions, precision labels and every helper line are trimmed so they stop wrapping to three lines and colliding with the next column. The nf4 label now fits its select without truncating. * Images Train: a little more spacing between field groups * Images and Video: tighten code comments * Fix training start NameError, the load-order guard test and CPU-only diffusion tests - start_training forwards resume_source_run_id to _start_training_impl, which reads it. Without it every start raised NameError. - Restore main's anchor in the load-marker order test: the file now has an earlier `if config.is_gguf:`, so indexing the first one compared the wrong branch. - The two diffusion tests that reach diffusers now skip when it is absent, matching the CPU repo-test env. - The UI smoke finds nav rows that live in the sidebar's More flyout. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Treat a null metadata caption as no caption str(None) stored the literal "None" as the caption, so a null row counted as captioned and would have trained on that text. Also drops an unused import. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix invalid-UTF-8 500s, the flat Canny map and the dropped DiT knobs read_text raises UnicodeDecodeError, which is not an OSError, so one bad caption sidecar or video sidecar 500d the info, upload and gallery routes. A flat image now yields the all-black edge map instead of its own luminance, and the four DiT loss knobs the trainer implements are declared so model_dump keeps them. * Use the ui font-size tokens instead of raw px text utilities text-[11px] and friends ignore the UI font size preference, which the repo's font-scale contract test enforces. Same rendered size at the default scale. * Fix diffusion dataset 500s, the dropout-1.0 no-op run and the reset base pick Four correctness fixes on the training side: - The labeling grid read caption sidecars under except OSError, but a non-UTF-8 sidecar raises UnicodeDecodeError (a ValueError), so one bad file 500d /diffusion/dataset/{name}/images and the grid could not be opened to repair it. Read it as no caption, matching the info summary. - An image past Pillow's own hard limit raises DecompressionBombError, which derives straight from Exception and so escaped the upload guard's (OSError, UnidentifiedImageError, ValueError) and returned 500 instead of the intended 400. - lora_dropout accepted 1.0, which makes PEFT build nn.Dropout(p=1.0): lora_A and lora_B receive no gradient and the run saves an untrained adapter while reporting normal progress. Bound it below 1.0, matching the LLM request schema. - The train panel re-seeded the base repo on every dataset refresh because the family object identity changes on each info fetch, so an upload or caption save silently replaced the user's chosen base and the run started on a different model. Track the pick and only re-seed on a real family change. * Show the retained failure when a video page mounts after a failed job Mount-time recovery handled only phase=completed, so reloading the page after a multi-minute generation failed left an idle view with no diagnosis: the backend keeps the terminal failed record only until the next job, and nothing else survives the reload. Surface it the same way the poll does, filtering the cancelled sentinel. * Fix batched generation crashes, cache keying and unreplayable recipes Four bugs in the batched inference path, all found by review: - A mixed-prompt batch sent a scalar negative prompt against a prompt list. Z-Image asserts on the length, and Qwen-Image, Krea 2 and FLUX true-CFG encode a batch-1 negative against batch-N latents and fail in the transformer's text/image concat. Broadcast it to match the batch. - The FBCache step-cache reset sat above the chunk loop. diffusers only resets that state at the end of a successful call, so a forward that raised (the OOM the backoff is meant to recover) left its own residual behind and the halved retry died on a shape mismatch. Reset before every forward instead. - The conditioning cache keyed on the checkpoint alone, but a GGUF or single-file load takes its text encoders from the companion base, so the same checkpoint against a different base reused the previous base's embeddings. Key the base too. - Gallery records stored the base seed and the requested batch size even when a prompts/seeds list drove the run, so restoring the second image of seeds=[5, 99] replayed seed 5. List-driven outputs now record as single-image recipes on their own seed. Also bound strength above 0: every img2img pipeline derives its step count from it, so 0 leaves zero denoising steps and either raises or, on SDXL, crashes on empty latents. * Fix quantized-load LoRA bake, prequant family exclusions and outpaint canvas Six review findings across the Images page and model scanning: - The quantized (int8/fp8) load path can only attach LoRA adapters before quantization, but the frontend load request had no loras field, so every generation after such a load was rejected and each reload repeated it. Send the selection with the load. - build_prequant_checkpoint passed no family to the scheme exclusions while recording the family in metadata, so a Qwen int8 artifact baked the short-M text-stream linears and was then rejected wholesale by the loader's family-keyed check. - Registering a bare single-file checkpoint directory produced no On Device row even though the images loader can load it; only its parent worked. Admit that shape when nothing else matched. - Unload left the Reapply target set, so the repair path was skipped and Reapply reloaded the ejected model. Clear it, as the video page does. - Both FLUX.2 bases were trusted for training but not inference, so Deploy to Create rejected every FLUX.2 adapter. - Outpaint allocated the grown canvas before downscaling, exceeding the browser canvas area cap on a large photo; an over-cap canvas is unusable, so Extend silently posted a fully transparent image and mask. Scale the source first. * Send the picked GGUF filename with the quant so diffusion loads fire The variant expander emitted only the quant label, and nothing else in the frontend set ggufFilename, so the Images and Video pages could never take their GGUF branch: both gate it on meta.ggufVariant and meta.ggufFilename, then fall through to the single-file path, which returns because the id is a repo id and not a .gguf name. Every quant pick was a silent dead click, with no load request reaching the backend. The filename was already on the variant row (the picker keys its list on it, and the variant validator requires a non-empty string), so thread it through the click handler. The chat path is unaffected: it reads ggufVariant and never needed the filename. * Version the conditioning cache key and reject non-finite flow_shift Two correctness fixes: - The cache keyed the checkpoint and its companion base by name only, so a Hub repo advancing to a new commit, or a local directory updated in place, kept returning embeddings from the previous text encoder. Pair both with a revision marker: the locally resolved commit sha for a Hub repo, config plus text-encoder file stats for a directory. Neither loads the encoders, so a warm run still keeps them off the GPU. - flow_shift only checked positivity, but JSON accepts 1e309, which floats to inf, and inf <= 0 is False while NaN fails every comparison. The sigma table then evaluates s * u / (1 + (s - 1) * u) as NaN, which poisons every sampled sigma and saves a corrupted adapter while progress looks normal. Require a finite value. * Keep curated models listed, guard the video companion repo, pin diffusers Three review findings: - The picker filtered every catalog member out of Recommended and Hub search on the way to canonical group rows, but nothing renders those rows yet (catalogGroupFitsDevice and groupMatchesQuery are imported and unused). A task-scoped picker's models list is catalogToModelOptions(), i.e. group members exclusively, so both lists came back empty and no curated model could be discovered or downloaded. Keep the artifacts listed until the grouped UI exists. - The video delete guard compared only repo_id, so deleting the companion base of a loaded GGUF video model was allowed even though it supplies the VAE and text encoders. Compare base_repo too, matching what the images guard already does for its companions. - diffusers was declared unversioned while the diffusion stack requires 0.39 (Krea2Pipeline, the cache_context child registries, the Flux2 and Z-Image pipelines), so an upgrade could keep an older release and selecting an advertised model failed until the user upgraded by hand. * Namespace the trainer conditioning cache per checkpoint, bound the learning rate - The trainer keyed its persistent conditioning cache on family and resolution only, while the keys themselves carry just the caption or image content and crop variant. One cache directory reused for two checkpoints, or for the same repo at a new revision, let a warm run skip loading its encoders and train on the other model's embeddings and latent statistics. Namespace on the base checkpoint and its resolved revision as well. The revision helper now lives beside the cache in diffusion_train_extras and the inference wrapper delegates to it, so the two cannot disagree about what counts as the same source. - The diffusion learning rate only checked positivity, but 1e309 floats to inf and satisfies gt, so the route evicted the resident models and started AdamW with an infinite rate: the first step destroys the adapter while progress looks normal and the result is saved. Bound it below 1.0, matching the LLM schema, which rejects inf for the same reason. * Fix GGUF image model picks doing nothing, and pick the train base in the top bar The quant rows never forwarded the .gguf filename, so every hub GGUF pick on Images/Video fell through to a silent return. On Train the top bar now picks the training base instead of a generation model, which is GGUF-only and untrainable. * Pin diffusion and video loads to the live HF cache root Both read huggingface_hub's import-time HF_HUB_CACHE, which changing the cache folder does not update: progress counted the old root while the download wrote to the new one, and from_pretrained could split one model across both. * Add the diffusion download plan endpoint Reports the repos and exact files a pick needs so the download manager can stage them with the loader's own file scope. A plain snapshot would add the packaged root single, transformer shards and fp16 twins the loader never opens. * Add a file-scoped flavour to the Hub download job Lets a consumer that reads a deliberate subset of a repo stage it through the normal download manager. Keyed as "@scope" so it never collides with a quant or with the repo's full snapshot, and the file list rides the registry so an XET to HTTP retry respawns the same scoped job. * Stage image and video downloads through the Hub download manager They downloaded inline inside the load, so they had none of the manager's disk preflight, manifest verification, resume or panel progress. Picks now stage as scoped jobs carrying the loader's own file list, then load from a warm cache. * Fetch staged GGUF checkpoints as scoped jobs, and stop calling diffusion models unsupported A GGUF entry went out as a full snapshot, whose ignore list drops *.gguf: the job finished at once having fetched only docs, and the repo landed on device unloadable. Every entry is scoped now. The Hub also no longer tags image/video models as unsupported (they run on their own pages), and those pickers name what they select. * Apply the picker task filter to local model sections LM Studio, ./models and custom-folder rows ignored it, so the Images picker listed chat GGUFs that 400 on a diffusion load. The backend already tags every local model with a task for this purpose. * Route a chat pick of a diffusion model to the Images or Video page Chat cannot load one, so it was either hidden or failed on load. The unfiltered picker now lists on-device diffusion models and navigates to the page that runs them, passing the repo and quant so that page loads it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Route the real GGUF filename, keep non-GGUF curated models, key scoped downloads by file set Five review findings, four of them ways a click did nothing or fetched the wrong thing: - A chat pick of a diffusion model routed ggufVariant (a label like Q4_K_M) in the search param the target page uses verbatim as the GGUF filename, so the load asked for a file that does not exist. Route ggufFilename; no filename means a curated non-GGUF pick, loaded as a pipeline. - The task-scoped pickers kept only GGUF repos, so the catalog's bf16, bnb-4bit and single-file fp8 artifacts could not be discovered or downloaded on the Images and Video pages even though loadSpecFor knows how to load them. Keep curated artifacts whatever their format, in Recommended and in Hub search. - Both pages deduplicated routed selections on the model alone, and they now stay mounted, so picking the same repo again -- another quant, or the same one after chat evicted it -- returned early without loading or clearing the query string. Key on model and quant. - Every scoped image download shared one @diffusion job key regardless of the requested files, so switching quant mid-download adopted the running job: the UI waited on the first file set, then loaded a file that was never fetched. Include a digest of the file set in the key. - A scoped plan silently dropped requested files missing from Hub metadata, and snapshot_download succeeds when an allow pattern matches nothing, so the job reported completion and triggered a load with required files absent. Fail the job instead. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the scoped download key derivable, and stop the hidden page hijacking a route Four review findings, the first a regression from my own last commit: - Keying scoped download jobs by a digest of the file set broke the download manager: it builds that key client-side (it polls and cancels before any response tells it a key), so it watched and cancelled a key no worker owned and never fired its ready callback. Keep the derivable "@scope" key and refuse the second request instead when a live job on the slot is fetching a different file set -- decided inside the registry claim, under the lock, so a concurrent claim cannot slip past it. The manager records the file set on the job as well, so a sibling quant's transfer is not adopted locally either. - Both diffusion pages read the route query through a loose useSearch and both stay mounted once visited, so the hidden one consumed the other's ?model=: it navigated back to its own route and tried to load, say, an image checkpoint as a video model. Only the visible page consumes it. - The staged download plan was built without the configured HF token or the Advanced values the load itself sends. The token matters most: the backend's Hub metadata lookup is best-effort, so a gated base silently planned no companion entry and the load pulled those multi-GB files inline, outside the manager. The memory/quant controls decide whether the base transformer/ shards are needed at all, and the route dropped memory_mode, cpu_offload, the prequant path and the LoRA selection before asking for the plan. - The video preview kept playing after leaving the page: the keep-alive layout only hides it, and display:none does not pause a media element, so a clip the user unmuted kept its audio going over the next page. Pause on the active transition and do not auto-replay while hidden. Also completes the hand-built request bodies in the hub download tests: the scoped-files field this branch added to the route read as an AttributeError against them, failing five tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Serialize the GPU handoffs, gate DiT training on a GPU, and keep 3.9 installable Six review findings, three of them evict-then-fail orderings: - The chat load reclaimed the GPU without telling the arbiter it existed. A chat load holds no llama-server process until its GGUF has downloaded, which is minutes, so a competing Images/Video acquire in that window found nothing to cancel, took the GPU, and the chat load then spawned onto the same device. It now registers an in-flight marker through acquire_for's register hook (under the arbiter lock, as the image and video loads do), the evictor cancels a marked load, and the route undoes itself if ownership moved while it loaded. - The Hub-download conflict check ran after that handoff, so a GGUF the download manager already owns destroyed the resident Images/Video pipeline and then 409'd, having loaded nothing. It moves above the handoff, together with the marker it handshakes with. - The image load released the engine router's transition lock before registering the load, so a second load choosing the other engine could unload the still-idle engine this one captured; the load then landed on a deactivated engine, where generate, status, unload and the arbiter's evictor can no longer reach it. Registration now happens under that lock and refuses if the engine changed. - Training a DiT family on a host with no GPU was accepted: nf4 is not a CPU fallback, its 4-bit load goes through bitsandbytes, which requires CUDA, XPU or MPS. The start unloaded the working Images pipeline, pulled the text encoders, and only then died in the child. Rejected before the teardown now, and /info stops advertising a precision that always 400s. SDXL keeps its documented fp32-on-CPU path. - Both diffusion pages kept the routed-pick marker forever, so re-picking the same checkpoint (after chat evicted it) neither loaded nor cleared the query string. The marker is released once the query is gone. The Images key also carried a stray NUL byte, which made the file read as binary to grep and other tooling. - diffusers dropped Python 3.9 in 0.38, so the unconditional >=0.39.0 pin left pip no candidate at all on 3.9 and made every install that composes the huggingface extras unresolvable there. The floor is conditional now. Also fixes tests that were already red on the branch: two hand-built request fakes had gone stale against fields this branch added, and the handoff-ordering test only failed on a host with fewer than two GPUs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the GGUF variant contract test against the merged handler signature The assertion pinned the exact single-line call handleVariantClick(v.quant, v.downloaded, expectedBytes, v.filename), but the handler takes (quant, filename, downloaded, sizeBytes) and prettier wraps the call across lines, so the mandatory repository test job failed on every push. Match the call structurally and assert the filename really is forwarded in the handler's argument order. * Stop a background page and a stale record taking the GPU or a download with them Five fixes from a review pass over the diffusion work. delete-finetuned rmtree'd a model the Images or Video engine was holding: every guard on that route is chat-only, and Images loads any local path, so deleting a local diffusion model under the storage root pulled the weights (and the companion VAE / text encoders sd.cpp re-reads each generation) out from under a live pipeline. The cached-model route already refuses this; the trained/exported one now does too, matching by path rather than repo id, and failing open on a chat-only install so it cannot block ordinary deletes. A staged download finishing while its page was hidden loaded the model and evicted whatever the user was actually using: both diffusion pages stay mounted behind the router and a load takes the GPU unconditionally. The pick is now held until its page is on screen again, which is also what chat does. A scoped download could report success having fetched nothing. With Hugging Face metadata unavailable no manifest is written, so verification is a no-op, and snapshot_download returns an existing snapshot folder without downloading when its own repo_info call fails. A repo already on disk from a full snapshot job (which ignores *.gguf) therefore completed with no weights and auto-loaded against them. The requested file list needs no network, so it is checked against the disk directly. The XET to HTTP retry reclaimed the job slot without the scoped file list, and that claim overwrites the stored record, so a later identical scoped start compared an empty list against the real one and 409'd instead of adopting the running download. The DiT accelerator gate probed torch.mps.is_available(), which only exists from torch 2.5 while the supported floor is 2.4. All three probes shared one try/except, so on torch 2.4 the AttributeError read as 'no block' and a CPU-only host still evicted the resident pipeline, downloaded the encoders and died in the child. Each accelerator is probed on its own now, through torch.backends.mps. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stage what an LTX-2.3 load reads, keep a routed file's load kind, drop an unbakeable LoRA Three from the latest review. The video download plan always asked for the wide base file list, so an LTX-2.3 pick staged the 2.0 base's VAEs, vocoder and connectors that the checkpoint supplies itself, while the companion files the 2.3 assembly does read were left out of the plan and pulled inline at load, outside the panel's progress, cancel and disk preflight. The plan now recognises a 2.3 pick by name (the load keeps the authoritative header probe, and under-guessing only falls back to the load-time pull), narrows the base list, and stages the extras in the same entry as the checkpoint so one repo stays one scoped job. A pick routed from the chat picker arrives as ?model= and ?quant= with no picker metadata, so a bare local .gguf or .safetensors was loaded as a pipeline: an explicit model_kind wins over the backend's filename sniffing, so it evicted the resident model and then failed on the missing model_index.json. Both pages now derive the load kind from the path, the same way their own picker handlers do. A torchao int8/fp8 build takes adapters only at load time. Switching artifact inside one family keeps the LoRA selection, since the family did not change, but the load did not bake it, so the next generation was rejected with 'reload the model with the adapter selection' while the picker still showed the adapter as active. The selection is now dropped once per resident build, with a message saying to pick and load again. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cancel an evicted safetensors load, spare the arbiter for CPU-only chat, fetch clips lazily Four fixes from the latest review round: - The GPU arbiter's chat evictor only cancelled the llama.cpp side. The orchestrator publishes active_model_name once its worker reports success, so an in-flight safetensors load was visible only as an entry in loading_models and finished onto the GPU after ownership had transferred. Cancel every pending load, and give the safetensors branch the post-load ownership recheck the GGUF branch already had. - A manual gpu_layers=0 GGUF load runs on the CPU with the GPUs hidden from the child, yet it took the arbiter unconditionally: it cancelled a running image or video generation for a model needing no VRAM, then held CHAT ownership so the next GPU workload unloaded it for nothing. Gate the acquire on the same predicate the launch-time CPU-only mask uses, as the image and video loaders gate on their resolved device. - The staged-download hook subscribes per repo, not per job, so another job on the same repo advanced the staged queue (starting a load whose scoped files were still downloading) or wiped a queue that was still running. Compare the variant each callback carries, like the chat page's auto-load does. - The video gallery fetched every record of a page into an object URL that lives until the page closes: 50 clips at tens to hundreds of MB each, for cards the user may never scroll to. Fetch a clip as its card nears the strip's edge, plus the selected one the player needs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim the comments across the diffusion backend Comment-only pass over the Python this PR touches: drop what the code already says, collapse multi-line explanations that still read on one line, and keep the reasoning that is not recoverable from the code. No code, docstring semantics or behaviour changes; verified with an AST comparison against the previous revision, and the backend suite is unchanged (same 37 environment failures as before: the API integration tests that need a live keyed server, the flash-attn install hooks, and the GPU memory fields). * Invalidate latents on a VAE swap, keep a cut-off generation, surface the EMA adapter - source_revision() scanned the checkpoint root plus text_encoder/tokenizer but not vae, so swapping or fine-tuning the VAE in place left the conditioning cache namespace unchanged and a warm run trained against latents from the old checkpoint. Include the vae directory, like any other component the cached tensors come from. - /images/generate answers only when the images are saved, and secure mode's tunnel caps an origin response near 100 seconds, which a native CPU or a high-step run passes routinely. The page reported failure while the work kept running, and a retry would duplicate it. A lost response (fetch rejection or a gateway status the origin never answered) is now told apart from a refusal: the page waits out generate-progress and reloads the gallery, so the run it started still lands. - The trainer emits the EMA adapter's path with the terminal event, but the state update dropped it, so neither the run history nor either response schema carried it and an enabled EMA left nothing discoverable. Keep it, and show it next to the primary adapter. - weighting_scheme advertised a choice of timestep sampling; sampling is always logit-normal and the flag only selects the bell loss weights. Describe what it does. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Hide unloadable cached rows, hold the dataset interlock, bound a GIF export - The cached-model listing tagged any repo with a model_index.json as text-to-image, so a community pipeline the image loader's trust rule refuses still got a row in the Images picker, and a detected-but-untrusted video repo fell through to that same tag. Gate the image tag on the load path's rule and hide an untrusted video repo outright. - A routed diffusion pick only carries a GGUF filename, which is all the chat picker has, so a curated single-file artifact arrived with no quant and was loaded as a pipeline: from_pretrained on a repo with no model_index.json. Pass the page's own catalog spec into the route pick, so a routed pick resolves to exactly what a direct pick on that page resolves to. - The dataset mutation endpoints checked is_active() and only then handed their filesystem work to a thread, so a start reserving in that gap changed captions or removed images underneath the preflight or the running trainer. The interlock is now registered for the whole request under the lock reserve() uses, and a start refuses while a mutation is open rather than waiting on it. - GIF export held every kept frame as a paletted image before encoding; a clip may be 2048x2048 for 1024 frames, and at the 12 fps target the step is 1, so one export click could allocate over 4 GB and take the backend down. Downscale past 720 px and widen the step to keep at most 300 frames. - seed accepted any Python int, so an out-of-range one passed every preflight, evicted the resident models, spawned the trainer and only then died in torch.manual_seed. Bound it to torch's 64-bit range in the request and config. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the accelerator probes in the DiT family-metadata tests Six tests read family_train_infos() (or a start preflight) without pinning the host probes, so they only held on a machine with a bf16 accelerator: on a GPU-less runner the DiT gate empties precision_modes, turns supports_compile off, and replaces any other preflight message with the no-accelerator note, and all six failed there. A conftest fixture pins both probes for exactly those tests, so they assert the family metadata they are about on every host. The gate's own CPU-only behaviour keeps its dedicated tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Carry the pipeline task into the hub inventory the pickers read - The task-scoped pickers filter On Device rows on a task, and the chat picker routes a diffusion pick by the same field, but those rows come from the /api/hub inventory, which never carried one: the Images and Video pickers listed nothing on device and the routing never fired. Both cached scans and the local listing now tag rows with the classifiers the models API already uses, the schemas and the frontend adapter carry it through, and a row the backend classified as a generation task is exempt from the chat-only guard that was also dropping it. - The local routing map was keyed by model_id while the row click passes id (a filesystem load id for a models_dir or LM Studio entry), so the lookup missed and the pick fell through to the chat loader. Key both. - A staged download whose start answered "error" left its head in place, where the effect never re-runs and onReady never fires, so the pick was stranded until the user reselected. Clear the queue and say so. - Every scoped pick in a repo shares the @diffusion variant, so the variant alone cannot tell two file sets apart: restaging while the first job finished let its completion pass for the new pick and load a checkpoint that had not downloaded. Bind the callbacks to the repo + file set they started, and to the staging generation. - A rejected generate POST does not say whether it reached the backend, so an immediately idle progress read was ambiguous and a submission that never landed looked like a finished image. Require evidence: progress seen active, or a gallery record that was not there before the POST. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the OpenAI image URL fetchable, keep WebM audio, stream example imports Four review items on the diffusion Studio work: - response_format=url returned the bearer-gated gallery route, which a standard image client downloads with no Authorization header, so the default response format was unusable. Mint a short-lived HMAC link instead (the shape RAG already uses for pdf.js) served by a signed route, and leave the gallery route itself bearer-only. - A manual gpu_layers=0 load carrying speculative_type="off" -- a value the UI persists and sends -- read as GPU-bearing, so it took the GPU arbiter and evicted a resident image/video pipeline even though the launcher hides the GPUs for it. Canonicalize the mode and exempt "off". - The curated example import prepared the whole split before the loop stopped at the 10-100 image cap; m1guelpf/nouns is 49,859 rows / 328 MB. Stream instead, with the prepared load kept as a fallback for a repo that cannot stream. - WebM export dropped the audio track an LTX-2 clip carries, silently, on the format offered for web embeds. Mux it as Opus through a resampler + FIFO, and keep exporting the video alone on a build without libopus. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not stub out triton on a GPU host when the Xet backend fails to import The lazy loader retries `import unsloth_zoo.hf_xet_fallback` under UNSLOTH_ZOO_DISABLE_GPU_INIT=1 whenever the first attempt raises. That flag makes unsloth_zoo take its MLX/CPU path, which injects triton and bitsandbytes STUBS into sys.modules for the rest of the process. On a working GPU box whose first import failed for an unrelated reason (a bitsandbytes/CUDA mismatch, say) the retry succeeds, so Studio boots looking healthy and then dies at the first CUDA-only kernel: a GGUF or compiled diffusion generation hits the stub and returns NotImplementedError: Unsloth: 'triton.tools.experimental_descriptor.enable_in_pytorch' was called on Apple Silicon / MLX, where triton is stubbed out. so every image generation 500s with an Apple-Silicon message on a Linux CUDA host, while the load reports success. Found by loading Z-Image-Turbo GGUF through the API on a box where bitsandbytes could not initialise. Gate the retry on the host genuinely having no accelerator. The Xet stall watchdog is optional and already degrades with a warning; a process whose triton is stubbed out is not recoverable. The warning now says why it did not retry. * Fix the lost-generation proof set, the settle timeout and the hub inventory's diffusion gates Seven fixes from the latest review round on the Images page and the hub cache inventory. Images page: - The lost-POST settle path built its "already seen" gallery id set inside the catch, after the request failed. By then the earlier runs of the same batch had already prepended their records, so run 2 could accept run 1's image as proof that its own request reached the backend. The set is now captured once before the first POST and grows with every record the batch produces. - settleLostGeneration fell out of its SETTLE_MAX_MS loop and returned normally, so a wedged generation was counted as done and the next run started against a busy backend. It now throws on timeout. - Restoring a recipe cleared the ControlNet selection but left the workflow tab and the init / mask / reference images pointing at whatever was loaded, so the next Generate conditioned on an unrelated image. It now clears all of them and returns to Create. - The download plan omitted the adapter selection the load itself bakes in. A baked LoRA forces the dense build path, so the plan described a different file set than the load that followed and the rest was pulled inline, outside the download manager. Both now derive the list from one helper. Hub cache inventory: - A download for a repo an Images or Video load is staging was allowed to start: only the llama.cpp loader was consulted. Both diffusion backends already expose loading_repo_ids for the delete guard, and the download guard now reads them too. - A companion-only prefetch (pipeline manifest plus VAE and text encoder, no transformer) passed the snapshot-partial check, since every file its manifest expected did arrive, and was advertised as on-device although from_pretrained cannot load it. - The single-file flag never reached the picker through the hub inventory path, so a checkpoint-only diffusion repo read as a full pipeline and failed after the handoff. The two pipeline-shape helpers now live in hub/utils/inventory_scan.py so /api/models/cached and the hub inventory classify the same repos the same way. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop adopting an unknown scoped download, leaking raced blobs and resurrecting deleted clips Three items from the latest review round. A scoped download job carries a deliberate file subset, and every file set of one repo rides the same "@scope" slot. A client that adopts a live job from the backend had no file list to compare against: the active-downloads response never carried one, so an adopted job's set was unknown and any later scoped request for the same repo read as "already started". Selecting a different checkpoint then waited on the wrong transfer and tried to load a file nobody fetched. The response now publishes the scoped file list, adoption records it, and an unknown set no longer satisfies a scoped request. A gallery record can be deleted while its blob is still downloading. The delete revokes the URL present at that moment, so the fetch that lands afterwards inserted a fresh object URL for a record no card renders and nothing can revoke: a full MP4, tens to hundreds of MB, pinned for the rest of the session, and once per raced fetch. Both galleries now discard a blob whose record went away, with an epoch covering the video page's Clear all. The video backend keeps the last completed job until the next one starts, and the Video page merges that record on mount to cover a job that finished after the gallery fetch. Deleting the clip left the record in place, so every reload prepended a ghost card whose file request 404s until another generation replaced it. Deleting the clip, or clearing the gallery, now clears the matching terminal record, and the page skips a record it deleted itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Resolve revisions from the live cache, serialize dataset imports, and stop pinning every gallery blob Five more items from the review round. The conditioning-cache revision marker read huggingface_hub's import-time HF_HUB_CACHE constant. Studio can move its cache during a session and loading follows the live setting, so after a move the marker went unresolved (or pointed into the previous root) and pulling a new revision of the same checkpoint no longer invalidated the cache: a warm run could reuse the old encoder's embeddings and the old VAE's latents. It now looks in the active Studio cache first and keeps the environment and the library constant as fallbacks, which the trainer subprocess still needs. The dataset interlock counts mutations rather than excluding them, so two imports of different examples into the same empty name both got past the emptiness check. The winner promoted its staging directory atomically; the loser found the folder non-empty, fell back to a per-file move, and merged its images and captions into the winner's dataset. Imports now take a per-folder lock, a second one is refused with 409, and the emptiness check is repeated under the lock. On Windows the sd.cpp asset resolver filtered only by accelerator token, so a Windows arm64 host matched an x64 zip, downloaded and installed it, and failed later when the binary would not run. It now filters by architecture the way the Darwin and Linux branches do. Every gallery page fetched every PNG up front and kept the object URL for the session, so scrolling a large gallery grew memory without bound for tiles the user may never look at. The Images strip now fetches a tile as it nears view, like the Video strip, and keeps the eager path only where IntersectionObserver is unavailable. A 503 carrying a JSON body comes from the application, not a proxy, so it is surfaced as the error it is instead of entering lost-response settlement and being reported as a request that never reached the server. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the class of a failed generation instead of a bare "Image generation failed." Found on a macOS runner: the native renderer aborts inside its own text encoder there, and the page showed only "Image generation failed." with the sd-server backtrace left in the server log, so nothing about the failure reached the user. The failure is now classified into fixed text, out of memory and native-process death, so the message says what happened and what to try. None of the engine's own output is echoed, since a native tail carries local paths and argv; that stays in the log, and an unrecognised failure keeps the original literal. * Treat an undecodable caption sidecar as the tombstone the trainer sees Uploads store .txt and .caption sidecars as raw bytes, so one can hold invalid UTF-8. The trainer treats any existing sidecar, decodable or not, as an empty tombstone and never falls back to the metadata row for that image. The labeling grid and the dataset summary read an undecodable sidecar as absent instead, so both showed a metadata caption that the run would silently replace with the instance prompt, and counted the image as captioned. Both now track sidecar presence separately, so what the user reviews is what the run trains on. * Keep the reason a native server died, not just its backtrace A ggml abort prints its cause first and then a stack trace, so reporting the last twenty captured lines gave twenty addresses and nothing about the failure: on the macOS runner the native server died on an unimplemented Metal op and the message carried only frame pointers. The captured tail now leads with the lines that name a cause and keeps recent context after them, for both the startup failure and the mid-request death. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop staging the dense text encoder for an fp8 video load Two halves of the same gap, found while measuring the LTX-2.3 download plan: - The video download plan and the scoped pre-download never saw text_encoder_quant. An fp8 request loads a hosted pre-cast encoder, so asking for one still staged and downloaded the base repo's dense Gemma3 (48.79 GB of Lightricks/LTX-2 on the 2.3 distilled pick) that the pipeline then never opened. The plan now drops those shards and stages the pre-cast checkpoint instead; their configs stay, since the pre-cast loader still meta-inits the encoder from the base repo's component config. - The LTX-2.3 assembly builds every component itself, so pipe_kwargs (which carries the pre-cast encoder for from_pretrained) never reached it and an fp8 request silently loaded the dense encoder anyway. It is passed across explicitly now. The dense skip is earned, not assumed: only a pre-cast checkpoint that resolves on the Hub lets the plan drop the dense shards, and only one already fetched to disk lets the pull drop them, so an unpublished or gated artifact leaves both exactly as they were. If injection still fails after that, the load tops the dense weights back up rather than handing from_pretrained a snapshot with no encoder in it. Measured against the real Hub on the 2.3 distilled Q4_K_M pick: 67.24 GB before, 18.92 GB with a 0.43 GB stand-in for the pre-cast artifact (the base entry drops from 24 files / 48.79 GB to 13 files / 0.04 GB). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Match the file's typing idiom for flow_shift models/training.py annotates with typing constructs throughout (105 Optional[...], no Union), and flow_shift was the one place using PEP 604. Union[] reads like the rest of the file, and it also drops the runtime evaluation that would raise on Python 3.9. * Bound the gallery blob cache, and three interlock fixes Four review findings, all reproduced first: - The gallery object-URL caches were unbounded. A clip runs from a few MB to a few hundred, both pages stay mounted after their first visit, and entries were only dropped on delete, so scrolling pinned everything for the session. Both pages now share a byte-budgeted LRU (512 MB video / 192 MB images) keyed off the visibility signal the near-viewport fetching already provides. On-screen media, the selected clip or image, and the item just fetched are never evicted, so eviction is invisible and a single item larger than the whole budget cannot evict itself into a refetch loop. - The image, video and chat load guards ran two independent training probes but returned early when the FIRST one raised, so an unreadable LLM backend disabled the diffusion interlock and a load could proceed straight into an active diffusion trainer on the same GPU. The probes are independent now. - An engine switch swallowed a failed teardown and published the new engine anyway, which is exactly the leak the unload exists to prevent: the arbiter's evictor, /images/unload and the next load all resolve through get_active_diffusion_engine(), so the still-resident pipeline (or a live sd-server) became unreachable and the next load allocated on top of it. The switch now fails and leaves the old engine published, so it stays reclaimable. - The native generation timeout was 30 minutes while the Images page waits up to 6 hours (SETTLE_MAX_MS), so slow-but-progressing CPU jobs died deterministically at the deadline. Measured on GPU-less runners, a 512x512 4-step Q2_K generation took 900 s on Linux and 1465 s on Windows, so larger images or step counts clear half an hour easily. The ceiling now matches the page's window and applies to the whole request: chunks of a split batch share one deadline instead of each getting a full budget. Cancellation is unchanged. Declined: gating the huggingfacenotorch extra off Python 3.9 over the conditional diffusers marker. The marker is deliberate and its comment says why: diffusers dropped 3.9 in 0.38, so pinning >=0.39 outright leaves pip no candidate and the whole extra unresolvable there. The pipelines it names live in studio/backend, which cannot install on 3.9 anyway (studio.txt pins matplotlib==3.10.9 and fastmcp>=3.0.2, both requires_python >=3.10), and the extra is the general core one, so the alternative drops 3.9 for library users who never touch Studio. * Close the load-versus-training-start race, and two picker fixes - The image and video load guards read is_active() and only then selected an engine, acquired the arbiter and registered the load. A /train/diffusion/start reserving inside that window freed residents the load had not registered yet, so the trainer came up beside a brand-new pipeline. The service already had exactly the right pattern for this in dataset_mutation, so gpu_load_admission mirrors it: reserve() refuses while an admission is open, an admission refuses once a start is reserved, both decided under the one lock. The span is only the registration, since begin_load returns as soon as the load is registered and _free_gpu_for_diffusion_training preempts an in-flight load from that point. Chat is deliberately not covered: its load spans an eviction plus a multi-minute GGUF load, and it admits models that fit beside training by design, which is a different contract from the diffusion pipeline's all-or-nothing one. - Hugging Face gives the LTX-2 family the image-to-video pipeline_tag (both Lightricks/LTX-2 and unsloth/LTX-2.3-GGUF report it), so a text-to-video-only filter dropped the flagship audio family out of Video Hub search while the rest of the app routed it to Video. - Task-scoped quant fit sized picks against the LARGEST visible device while resolve_diffusion_device_target returns a bare "cuda" and torch places on the current one. On a heterogeneous host that recommended a checkpoint sized for the bigger card and then loaded it onto the smaller one. Fit now uses the device the load actually lands on; identical on a homogeneous host. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Expose the persistent conditioning cache in the start schema DiffusionLoraConfig has carried cond_cache_dir for a while and the DiT trainer acts on it, but DiffusionTrainingStartRequest omitted the field, so Pydantic dropped it silently and every API-driven run fell back to the in-memory cache that is rebuilt from scratch each time. The warm path skips loading the VAE and the multi-GB text encoders on a rerun whose images, captions and resolution are unchanged, so this was a real capability that could not be reached. Contained like output_dir rather than left to the trainer subprocess's cwd, since it is another directory the trainer writes to. Blank or omitted still means the in-memory cache, so it must not resolve to the outputs root. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix diffusion policy and classification issues from review fp8 auto precision defaulted to precise accumulate on any non-consumer GPU, which made fp8 2.05x slower than int8 on RTX 6000 Ada and slower than not quantising at all. NVIDIA's professional whitepapers do publish equal FP8 rates for both accumulate modes there, so the hardware premise held, but the cost is in the cuBLAS path rather than the published rate. Default to fast accumulate: measured on B200 the flag is a no-op (4096^3 _scaled_mm at 3023.8 vs 3041.8 TFLOP/s, bitwise-identical output, 1.213 s vs 1.230 s end to end), so it is a large win where it bites and free where it does not. Precise accumulate stays available via transformer_quant_fast_accum. Z-Image's DiT is a Lumina2 derivative, so unsloth/Z-Image-GGUF and unsloth/Z-Image-Turbo-GGUF both declare general.architecture = "lumina2" and the whole line was tagged image-diffusion-unsupported and hidden from the Images "On Device" list, though validate_load_request loads them. Resolve shared archs from the repo/file name like bare "wan" already does, with a test asserting the picker and the loader agree for every family. The sage attention on-demand install ran an unpinned `pip install sageattention`, but PyPI's newest wheel is 1.0.6 and diffusers refuses anything below 2.1.1: the install always "succeeded", wrote an unusable version into the running venv, and was rejected on the next line. Carry the dispatcher's floor so pip resolves nothing instead. The dense-quant disk gate sized the download from the bf16-RESIDENT table. The fp32 families download twice that (Z-Image: 23,479 MiB against a 21,970 MiB gate), leaving a window where the check passed and the download filled the disk; Ideogram 4 ships fp8 and was overcharged the other way. Size the gate by published bytes, verified against HF sibling metadata for all 12 families. Patch installs went through unsloth_zoo, which refuses to import unless UNSLOTH_IS_PRESENT is set, and that is set by unsloth itself. The server imports unsloth at boot so it never showed there, but any other process ran silently unpatched with every install returning False, which is 13 test failures on a clean environment. Import unsloth and retry once, memoised per process. Also: the GGUF+LoRA refusal pointed at the native engine without saying a GPU host only selects it under UNSLOTH_DIFFUSION_ENGINE=sd_cpp, so the suggestion was unreachable; the gallery recipe recorded loras from the generate request alone, losing a load-time bake; load-progress claimed "40.07 GB downloaded" for a fully cached load; and pickers.tsx imported three catalog-group helpers it never used. Reported by oobabooga. * Keep the sd.cpp text encoder on CPU under Metal macos-14 loads FLUX.2-klein-4B Q2_K natively on mps and then dies on the first generation with exit code -6: ggml_metal_op_encode_impl: error: unsupported op 'RMS_NORM' -> ggml_abort LLMEmbedder::encode_prompt -> LLMRunner::compute -> GGMLRunner::compute ggml's Metal backend gates RMS_NORM on contiguous rows and aborts the process when that does not hold, with no per-op CPU fallback, so any LLM text encoder (Qwen3 for FLUX.2 and Z-Image, T5 for FLUX.1) takes sd-server down. The encoder runs once per prompt while the DiT runs every step, so pinning only the encoder keeps Metal for the part that matters. UNSLOTH_DIFFUSION_SD_CPP_METAL_TE_GPU=1 opts back in once ggml grows the kernel. * Gate the unsloth retry in the diffusion patch backend The retry added for the clean-environment patch failures is not free: importing unsloth pulls torch in behind it, which costs ~940 MB of RSS measured in a process that had neither, and on a host with no accelerator it fails anyway. A cross-platform CI job that had generated fine at ~900 s later died 19 s in with SIGTERM and every 'if: always()' step skipped, which is the runner being torn down rather than a step failing. Retry only when torch is already imported (true of the server and of anything patching a real module, and the condition that stops the retry from being what loads torch), unsloth is installed but not yet imported, and the first failure was the ImportError the sentinel guard raises. The clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only retry the unsloth import where it can succeed The gate still let the retry run on hosts unsloth does not support, which is where it is most harmful: a 7 GB macOS runner lost the Studio server 26 s into a load, and the Linux runner was torn down mid-generation. Neither MPS nor plain CPU can complete the import, so the retry there pays the cost and fails anyway. Require an accelerator unsloth actually supports (CUDA/ROCm via torch.cuda, or XPU), with UNSLOTH_ALLOW_CPU as the documented override, and hoist the predicate to module level so it is tested directly rather than through the import system. On a CPU-only host the retry no longer fires at all; on CUDA the clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Guard old diffusers, stream video exports, record conditioned recipes Three fixes from review. The 0.39-only pipeline classes (Flux2Klein, Z-Image, Krea 2, LTX-2, HunyuanImage) were resolved by getattr deep in the load, so on the older diffusers that packaging still allows on Python 3.9 -- diffusers dropped 3.9 in 0.38 and this project still supports it, so the 0.39 floor has to be conditional or the extra becomes unresolvable -- an advertised model failed with a bare AttributeError after its checkpoint had already been downloaded. Krea 2 already guarded itself this way; assert_pipeline_class_available now runs the same check for every image and video family from validation, before any fetch, and names the version and the fix. WebM export accumulated the whole VP9 output in a BytesIO and returned it as one bytes object that the response held again. The request caps allow 2048x2048 for 1024 frames, so an export runs to hundreds of MB and concurrent clicks could exhaust the process, while the MP4 route beside it already streamed from disk. transcode_to_file encodes to a temp file and the route returns a FileResponse with a background unlink, so nothing large is resident. A conditioned generation's recipe carried only the txt2img fields, so the gallery presented an inpaint or upscale result as a complete Create recipe and restoring it replayed an unrelated text-to-image request. The images themselves are still not persisted (user uploads with their own lifetime), but the workflow and its scalars are, restore reapplies them, and the toast now names the inputs that have to be supplied again instead of silently landing on Create. Reported by Codex. * Per-load video cancel event, family-gated image picker, cond cache refusal A cancelled video load could resume: begin_load cleared the shared cancel event, and unload() drops _loading without waiting for the worker, so the next load cleared the very object the cancelled worker was watching and its multi-gigabyte pull ran on alongside the replacement until the token check at the end. Each load now gets its own threading.Event, passed down through _fetch_te_prequant and _predownload_base, so a cancelled worker stays cancelled. A cached repo with a model_index.json was advertised as text-to-image on the trust rule alone, but validate_load_request also requires a detected image family, so a trusted pipeline of an unsupported class produced a picker row that deterministically 400s. The picker now applies both gates, mirroring the video branch. cond_cache_dir was accepted for sdxl and then ignored: only the DiT trainer reads it, while the SDXL trainer builds a per-run in-memory latent cache, so the promised cross-run reuse never happened. The route now refuses it with a 400 that names the families which do support it, checked against the resolved family so an omitted model_family with an SDXL base is caught too. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the frontend build broken by the gallery blob cache tsc -b failed on the branch head, so npm run build produced no dist and every platform job fell back to --api-only: blob-url-cache.ts(29,15): TS1294: This syntax is not allowed when 'erasableSyntaxOnly' is enabled dataset-labeling-grid.tsx / dataset-showcase.tsx: Argument of type '{ url: string; bytes: number; }' is not assignable to parameter of type 'string' The cache took its budget as a constructor parameter property, which the project's tsconfig forbids, and fetchGalleryObjectUrl now returns the blob size alongside the URL for that budget, which the two dataset thumbnail components still consumed as a bare string. Declare the field explicitly and destructure the URL at both call sites. tsc -b is clean and vite build emits dist again. * Recover from a ggml unsupported-op abort by restarting on the CPU backend ggml checks every node against the device's supports_op and calls GGML_ABORT when one is not implemented, because a single-backend graph has nowhere else to put it: there is no per-op CPU fallback. The whole sd-server dies with SIGABRT mid-generation and the user gets "the native image renderer stopped unexpectedly" with no way forward. Seen on macos-14 arm64 with FLUX.2-klein-4B Q2_K through the cross-platform CI: the text encoder is already pinned to CPU, and the abort moved into the denoise loop instead. ggml_metal_op_encode_impl: error: unsupported op 'MUL_MAT' -> ggml_abort StableDiffusionGGML::sample -> sample_k_diffusion A retry on the same backend would abort identically, so the load is restarted once with --backend cpu (the only flag that changes which backend executes the graph; --offload-to-cpu moves parameters, not compute) and the generation is re-submitted. The same checkpoint then renders slower rather than not at all. Strictly bounded: the signature must carry both the unsupported-op line and ggml_abort, the device must not already be CPU, and it happens once per load, so an OOM kill or a genuine crash still surfaces as itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix two tests that only fail in a full-suite run The 3.10 CI leg resolves PyAV 17, where av.container.OutputContainer is an immutable C type, so the no-libopus export test died on "cannot set 'add_stream' attribute of immutable type" before it asserted anything. Inject the refusal by wrapping the container av.open() returns instead; modules stay patchable on every build. Removing the injection makes the test fail again, so it still covers the branch it is named for. The Xet shim's degraded-path tests drop utils.hf_xet_fallback from sys.modules and import a throwaway copy. Restoring only the sys.modules entry left the utils package attribute bound to the throwaway, and the two disagreed for the rest of the process: a later monkeypatch of the dotted target patched one copy while the code under test imported the other, so the patch did nothing and test_fetch_te_prequant_only_reports_what_it_downloaded reached the real Hub and got a 401. Restore both bindings. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop an ignored-cancel sd-server, guard deletes during diffusion training, repair unusable managed binaries sd-server does not interrupt an in-flight job, so when it ignores a cancel the grace branch abandoned the poll and reported cancellation while the native job kept a core (or the GPU) busy to completion and held the server's job slot. The comment said the caller stops the server, but only unload does that immediately: a superseding load stops it after its multi-gigabyte download, and a load that then fails never gets there. Stop it here, as the deadline branch already does. DELETE /api/models/delete-finetuned checked only the LLM trainer, so it could rmtree the output directory a live diffusion LoRA run was about to write its adapter into. Consult the diffusion training service too, like the dataset mutation and model-load routes. find_sd_*_binary only checks is_file(), so an interrupted extraction (or a prebuilt for the wrong CPU) left a present-but-unrunnable binary the installer never retried: every load probed it, fell back to diffusers, and native inference stayed off until the directory was deleted by hand. Probe it and reinstall, but only for a copy under the installer-owned root -- SD_CLI_PATH, UNSLOTH_SD_CPP_PATH, an in-tree build and anything on PATH are the user's. * Plan the pre-cast text encoder, and make the cross-trainer GPU admission atomic An fp8 text-encoder request loads a hosted PRE-CAST checkpoint, but the image download plan never received text_encoder_quant, so the manager staged the base repo's dense encoder (FLUX.2-dev's Mistral-24B is ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) and the load then pulled the pre-cast file inline, outside the manager's progress and disk preflight. The plan now takes the field, resolves the hosted artifact with the same resolver the injection uses, stages that file, and drops only those components' dense weight shards. The load's own prefetch takes the same treatment, since it paid the same cost. Only a checkpoint that really resolves on the Hub earns the drop, so a gated or renamed artifact still stages the dense encoder the load will fall back to. The two trainers admitted each other with independent check-then-act guards: the diffusion route checks the LLM backend several network-bound preflights before it reserves, and the LLM route checks the diffusion service well before it spawns, so two near-simultaneous starts could both pass and train on one GPU. reserve() now re-tests the LLM backend under its own lock, and the LLM route holds the diffusion service's gpu_load_admission across its spawn, so exactly one of the two wins. Both halves fail open, so a chat-only install still trains. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the Advanced options a staged download planned against Staging does not set busy, so while a multi-gigabyte download runs the user can still change precision, memory mode, speed or the baked LoRA selection. The pending record held only the repo and artifact, and the completed download fired a load that read the CURRENT state: the staged file set could then be missing files that load needs (fetched inline, with no progress and no disk preflight) or hold gigabytes it no longer uses. One snapshot of every Advanced control is now taken when the plan is built, and it travels with the pending record into the load, so the load that runs is the one the download was planned for. * Do not advertise a family the installed diffusers cannot build The newer families (Z-Image, Krea 2, FLUX.2, LTX-2, HunyuanImage) exist only from diffusers 0.39, and 0.39 cannot be installed on Python 3.9 at all -- diffusers dropped 3.9 in 0.38, so the requirement is conditional or the whole extra becomes unresolvable. On such an environment the picker still offered those rows, every pick failed deterministically, and the error's advice to run pip install -U diffusers could not fix it without also upgrading Python. The cached-repo picker now applies the same availability check validate_load_request does, which is keyed on the pipeline class actually present rather than on the Python version, so it is also right for an intentionally pinned older diffusers on 3.10+. Fails open when diffusers cannot be imported at all: that is a different problem and the load path reports it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore the diffusion engine selection after each router test The active engine is module state, and several tests set it by plain assignment because what _activate does to it is the thing under test, so monkeypatch could not undo it. A leaked ENGINE_SD_CPP left get_active_diffusion_engine() handing back the sd.cpp backend for the rest of the process, and every later route that reads the active engine then saw an unloaded model: eight tests in test_openai_images_generations_route.py returned 503 in a full-suite run while passing on their own. The autouse fixture now snapshots and restores it. * Stream gallery clips, and close three races around them Four fixes from the latest review pass. The video gallery downloaded each clip into a blob before it could play, so playback waited on the whole file (tens to hundreds of MB), seeking was limited to what had arrived, and every viewed clip stayed pinned in the webview. The file route already streams and serves ranges; it just could not be a <video src> because it is bearer-gated. Mint a short-lived signed link instead (its own HMAC secret, 12 hour TTL, separate from the image links) and hand it to the element, which then fetches only the ranges it plays. That removes the blob budget, its LRU and every revoke on this page. The sd.cpp readiness probe accepted any process answering on the port, so a foreign server that grabbed the port between the bind check and the spawn was adopted as ours. Confirm the listener is our child before reporting ready, and stay best-effort (psutil missing, an unknown owner, or any probe error still passes) so the check can only reject a definitely foreign process. Dataset import held its lock for the extract but not for the upload path, so two concurrent uploads into the same folder interleaved; take the same lock and return 409. And reject Windows device names (CON, NUL, COM1..9, LPT1..9, with or without an extension) plus trailing periods in dataset names, which are unopenable on Windows. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Build the image download plan for the engine that will load /images/download-plan always asked the diffusers backend, while /images/load picks the engine per host: a GGUF pick on a machine with no usable GPU routes to native sd.cpp, which reads a single-file VAE plus text encoders and never opens the base repo's sharded components. Measured on unsloth/FLUX.2-klein-4B-GGUF (Q2_K): the plan staged 7.66 GB of FLUX.2-klein-4B components the native load discards, and the 7.80 GB sd-cli actually needs was then fetched inline by the loader, outside the download manager's progress and its disk preflight. Z-Image-Turbo is the same shape. The plan now asks whichever engine the load will select. predict_engine() applies the selection policy without any side effect: it activates nothing (staging a download must not unload the resident model) and only locates the binary rather than installing it, but still counts an installable binary as available, since that is what the load does on a fresh host. The native backend gains a download_plan built from the same _asset_specs the loader fetches, returning the same envelope, so the manager stages exactly the files sd-cli opens. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not let a queued generation outlive the model, and three scan fixes Five items from the latest review; four were real. An unload or arbiter eviction only cancels the generation holding _generate_lock. A second request queued behind it holds no cancel event yet, and Python locks are not FIFO, so it could take the lock the instant the active denoise released it, still see a loaded pipeline, and run a whole new denoise after the model was told to go away: the eviction then waits minutes for it and an image lands after the eject. Unload and a superseding load now raise a fence under _lock before they queue, and a generation that wins the lock while one is pending refuses instead. The cached-model scan judged pipeline completeness across every revision, so a repo holding an older complete snapshot plus a newer companion-only one read as complete while the snapshot from_pretrained actually opens has no transformer. Both scans now look at the revision the loader will open. Deleting a dataset image deleted its caption sidecar unconditionally, which for cat.jpg alongside cat.png removed the caption the survivor still resolves to. The sidecar now goes only with the last image of that stem, matching what the thumbnail cleanup beside it already did. Importing an example into a folder that holds no images but does hold files fell back to promoting the staging dir one file at a time, so an interruption left a partial dataset that the image_count check accepts as complete on retry. Those files are folded into the staging dir instead and the promotion stays a single atomic rename. The MPS generator report does not apply: torch.Generator(device="mps") has worked since PyTorch 2.0 (pytorch/pytorch#91348) and the studio installer pins torch>=2.4. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten diffusion comments Collapse the multi-line comment blocks across the image, video, sd.cpp and diffusion-training code to one or two lines each, and drop comments that only restate the statement below them. Comments only, no code or behaviour changes. * Tighten diffusion comments (second pass) Collapse the remaining multi-line comment blocks in the video page, training routes and service, sd.cpp server and installer, memory and speed planners, and the shared request models. Comments only, no code or behaviour changes. * Tighten diffusion comments (third pass) Collapse the remaining multi-line comment blocks in the attention, cache, LoRA, prequant, precision and compile-cache modules, the sd.cpp arg builder and engine, the video routes, the Ideogram 4 assembly, the model picker, and the diffusion test suites. Comments only, no code or behaviour changes. * Restore the dataset when an example import cannot be promoted Promotion folds the folder's pre-existing entries into the staging dir so the swap is one atomic rename. Every failure after that fold left the user with nothing: the 409 path and an os.replace error both fell through to 'finally: shutil.rmtree(staging)', which deleted the entries that had just been moved in there, while the response said 'Nothing was written'. A same-named entry was also unlinked outright before the promotion was known to succeed. Park superseded same-name entries in a rescue dir instead of deleting them, record every move, and restore all of them if any step of the promotion fails. The fold loop itself is covered too: renaming a non-writable directory raises EACCES on POSIX, which previously escaped as a 500 after the earlier entries had already been moved out. A failed rename now maps to the same retryable 409 as the rmdir conflict. Verified with the reported trigger (a non-empty mode-500 directory whose name collides with an imported file): the folder listing is now identical before and after the failed import. * Drain the teardown fence on a failing unload, give each load its own cancel event Two independent leaks on the image path, both already solved elsewhere in the same file. unload() incremented _teardown_waiters, ran _unload_locked() and decremented, with no try/finally, while the superseding-load path used a finally for the same pair. _unload_locked ends in clear_gpu_cache(), whose CUDA branch calls synchronize/empty_cache/ipc_collect unguarded, and a sticky CUDA fault makes those raise. The count then never drained, so every later generation was refused as cancelled for the life of the process, a fresh load included, since begin_load's own increment and decrement are symmetric. unload() is reached from the chat/video GPU handoff, the engine router and two training routes, so one fault during an ordinary handoff wedged image generation until restart. Release it in a finally. The image and native backends each cleared one shared cancel Event on a new load. unload() sets that event to cancel an in-flight multi-GB download and drops _loading in the same breath, so a replacement load is admitted while the cancelled worker is still inside the fetch, and its clear() re-enabled the very object that worker was watching: the cancelled download resumed and ran alongside the replacement. Take a fresh Event per load and thread it to the worker, as the video backend already does, and set it under the lock since begin_load now rebinds the attribute. * Name utf-8 on the diffusion text I/O and the sd.cpp subprocess pipes tests/test_text_io_encoding.py failed on five files this branch adds. Text I/O without an explicit encoding falls back to the Windows ANSI codepage, so a non-ASCII path or manifest value round-trips corrupted, and the three sd.cpp pipes decode the child's UTF-8 output as ANSI on Windows despite already passing errors = 'replace'. Eleven read_text() / write_text() sites across diffusion_compile_cache, diffusion_ideogram4 and diffusion_krea2, plus text = True on the sd-cli version probe, the sd-cli run and the sd-server pipe. * Record the load-time build on a gallery image's recipe A gallery record documents itself as the image's full generation recipe and is embedded in the PNG, but the only load-related field it carried was the repo id. A GGUF repo holds many quants, so that does not say which one made the pixels, and it says nothing about an adapter baked in at load time. The fallback meant to cover the baked case could never fire: with no loras on the request _adjust_baked_loras zeroes every baked adapter and _active_lora_pairs drops zero-weight entries, so active_loras was always empty. A baked-and-disabled build is not the same pipeline as a never-baked one, so the recipe could not reconstruct the image once the model was rebuilt. Persist model_kind, gguf_filename, transformer_quant and the baked adapter names, read off the load state rather than the request, and show them in the recipe popover. The new fields are optional with defaults, which matters because list_gallery_images drops any record that fails validation, so a required field would have emptied every existing gallery; a regression test pins that. * Drop eleven duplicated comment tails, restore the mxfp8 denial note The comment passes collapsed several wrapped blocks onto one line without deleting the last physical line of the original wrap, leaving the tail of each sentence repeated as its own comment underneath. Two of the eleven were re-worded rather than byte-identical, so a strict suffix match missed them. 6e16ad16f also dropped the line justifying the qwen-image mxfp8 denial while that rule stayed live in _FAMILY_SCHEME_DENY, under a header that then documented only fp8 and nvfp4. Restored. Comments only; verified with the AST gate. * Only let the dense-quant fallback use shards the prefetch actually staged The prefetch skips the base repo's transformer/ shards whenever a prequant checkpoint is expected, since that checkpoint replaces them. But a prequant fetch can fail for reasons the planner cannot see: an unpublished, gated or renamed artifact, a hub 5xx, a proxy, a checkpoint the validator rejects. The loader then fell through to from_pretrained(subfolder = 'transformer') and pulled those shards inside the load lock during 'finalizing', after the previous pipeline was already evicted, where the cancel event has no reach, load_progress has already reported bytes_downloaded == bytes_total, and the cache-disk gate had only reserved the small prequant checkpoint. That is verbatim the situation _dense_quant_prefetch_needed's own docstring exists to prevent. Gate the in-loader dense fallback on the shards being staged, read off the returned file list rather than the request so a failed size estimate closes it too, and let the GGUF build take over otherwise, which is what the prequant-sized replan already does one branch over. It is also the invariant the text-encoder path already enforces: only a component that really resolves may have its dense weights dropped from a plan or a prefetch. Adds the missing coverage for the gate's prequant arm, which the existing disk-gate tests never reached. * Refuse a training output_dir that resolves to the outputs root resolve_output_dir strips a leading 'outputs' and drops '' / '.' segments, so '.', './', './.', 'outputs', 'outputs/outputs' and ' . ' all clean away to nothing and land on the outputs root itself rather than a run directory under it. The DiT trainer then writes pytorch_lora_weights.safetensors flat into the root, where scan_trained_models and scan_checkpoints cannot see it (both filter is_dir()), and a second such run overwrites the first. The UI only checks the field is non-empty, so 'outputs' is one plausible run name away. Refuse it with a 400 that says what to do instead. cond_cache_dir collapses the same way but has an honest 'off' to fall back to, so a root-resolving value now means the in-memory cache, which is what the comment above it already promised: otherwise a run drops one flat safetensors per cached latent and caption into the directory trained models live in. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Repair three defects the main merge left behind Git merged all three files without a conflict, but the result was wrong in each case. The tests only started failing once the merged tree was run. routes/inference.py duplicated the GGUF load block: this branch had moved the gguf_load_in_flight marker and the _hub_download_blocks_gguf_load guard under "if config.is_gguf and config.gguf_hf_repo", and the conflict resolution re-added main's copy at the old position, so both ran. Dropped main's copy; the earlier placement is the deliberate one, so a 409 from the hub guard cannot tear down a resident Images or Video pipeline. test_gpu_selection.py still called _hf_offline_if_dns_dead, which main renamed to _hf_offline_if_unreachable_for (#7591). Disjoint edits, so no conflict, but four route-error tests referenced a function that no longer exists. test_gguf_load_cache_reuse.py anchored its ordering assertion with rindex over "if config.is_gguf:", taking the last one before the load marker. That only held while _resolve_inherited_extra_args sat above every such line; main has since hoisted it above the gpu_ids preflight, so the anchor landed between the call and the marker and the assertion compared against an unrelated later call site. The ordering it checks is a property of _load_model_impl as a whole, so it now anchors on the function. The ordering itself is intact: _resolve_inherited_extra_args, then the gguf_load_in_flight marker, then the hub guard, then the chat handoff, then unload_model. * Stop a real sd.cpp install from breaking its own discovery tests The five "nothing is installed" assertions in test_sd_cpp_engine.py cleared SD_CLI_PATH and UNSLOTH_SD_CPP_PATH and patched Path.home, which covers hops 1, 2 and the fallback half of hop 3. It leaves two hops live. Hop 3 goes through managed_install_root(), which honors UNSLOTH_STUDIO_HOME and STUDIO_HOME and resolves to the stable-diffusion.cpp directory beside the studio home. That is the documented way to run side-by-side Studios, so anyone who has one set gets a real sd-cli back from a finder the test expects to return None. Hop 4 is the in-tree developer build, which does the same for anyone who built sd.cpp inside the checkout. Isolated with an autouse fixture rather than another helper call, because reaching the failure needs no fixture: SdCppEngine(binary = None) runs the finder from its constructor, which is why test_generate_raises_when_binary_missing failed too. The fixture points the studio home and the in-tree root at an empty tmp tree, so every hop is answered by the test rather than by the box. The in-tree root moves into a named in_tree_install_root() so it can be pointed somewhere empty; behaviour is unchanged, including the OSError and IndexError guard on an unexpected layout. * Make the sd.cpp uninstall test actually extract the code it tests The two sed ranges anchored the production fragments at column 0, but both blocks sit inside the main removal function and are indented, so each range matched nothing and LOOP_FILE / DEFAULT_FILE came out empty. Sourcing an empty file is a no-op, so the suite reported 4 passed / 7 failed: the seven removal assertions failed because nothing ran, and the four that passed were 'kept' assertions that pass trivially when nothing runs. The shell job auto-discovers tests/sh/test_*.sh and runs each under set -e, and this file is not in its skip list, so it was a deterministic red. Anchor on optional leading whitespace and fail loudly on an empty or _remove_path-less fragment, so a future reshuffle of uninstall.sh cannot make the suite vacuous again. Now 11 passed, 0 failed, against the real removal loop and the real default-mode block. * Agree on what an engine can build, and on what the installer owns Two gates, each half-applied. The unbuildable-family gate had one caller, the image branch of the cached repo picker. The GGUF classifier, the local-model classifier and the video branch had none, so on a diffusers too old for a family the picker still offered its GGUF and the load then failed. Meanwhile the loader asserted the diffusers pipeline class before engine selection, so a GGUF this host routes to native sd.cpp, which instantiates no pipeline class at all, was refused with an upgrade instruction that could not help. Both are wrong one-sidedly: hide a family only when NEITHER engine can build it, and demand the diffusers class only when diffusers is what will load it. One predicate, family_buildable_here, now answers both, so the picker and the loader cannot disagree. The population is real on Python 3.9, whose diffusers ceiling is 0.36: Flux2KleinPipeline, Krea2Pipeline and LTX2Pipeline are all absent there, and FLUX.2-klein GGUF is a repo in this PR's title. That assertion also raised RuntimeError, which /images/load maps to 409, the status that otherwise means a load is already in progress, and which escaped /images/download-plan (it catches ValueError and FileNotFoundError) as a bare 500 with the message lost. It is an unloadable pick like every other, so it raises ValueError and both routes answer 400 with the text intact. The managed-binary repair used a path test for ownership while the installer uses a marker. On a managed root without the marker, which is any install predating it, the repair deleted sd-server and the reinstall was then refused, permanently, because the surviving sd-cli keeps the directory non-empty so the marker can never be claimed: the user went from an unrunnable binary to no binary and no way back. Require the marker before discarding, which is the same definition of ours that uninstall.sh already uses to keep a user's own stable-diffusion.cpp checkout. A genuinely interrupted extraction still self-heals, since install() writes the marker before it extracts. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add gguf and av to the studio extra so the wheel install matches studio.txt This branch added both to studio/backend/requirements/studio.txt, which is what install.sh uses, but never to the studio extra in pyproject.toml, which is what 'pip install unsloth[studio]' uses. Nothing else keeps the two in sync, so tests/studio/install/test_studio_extra_matches_requirements.py failed deterministically, and the CI job runs the whole tests/ tree. The drift is not only a red test: gguf backs diffusers' GGUFQuantizationConfig and av does the MP4 encode and audio mux, so a wheel install got a Studio that cannot read a GGUF or export a video, which is most of what this PR adds. * Fix two test-suite defects: the CPU-only patch gate and a popped module The patch backend reaches unsloth_zoo's helpers only through an ~940 MB 'import unsloth', which a CPU-only host cannot complete, so every patch install returned False and nine arch/eager/compile tests failed there -- exactly the runners the retry was narrowed to protect. unsloth_zoo only wants UNSLOTH_IS_PRESENT in the environment, which costs nothing and needs no accelerator, and the conftest already sets its sibling UNSLOTH_ALLOW_CPU. Set it there too, as run.py and main.py already do at module scope, so no real server ever took the expensive route. CPU-only goes from 9 failed to 24 passed, 1 skipped; the GPU run is unchanged at 29 passed. test_setup_cache_env_hf_home popped utils.hf_cache_settings to model a fresh process and never restored it, so a later import built a second module object and rebound it on the utils package. test_hf_cache_settings then wrote its setting into one object while core.inference.diffusion read the other, the same split-module failure the xet shim already had. Restore both bindings in teardown: 21 pass together, and each file still passes alone. * Restore the setup_fail assertions the main merge reverted A fourth instance of the class 9541cc535 fixed: the merge took main's studio/setup.sh, which #7644 changed to abort through the setup_fail helper so desktop mode still emits [TAURI:ERROR], but kept this branch's older copy of the test, which still asserted the literal 'exit 1'. setup.sh is byte-identical to main here and the only diff in this file was the reverted assertions, so take main's version. 60 passed. * Fence the video teardown, reject non-finite clipping knobs, stop a test installing sd.cpp The video backend never had the teardown fence the image backend documents. Its unload and its superseding-load path both signalled the active generation, then did 'with self._generate_lock: pass' and tore the state down with the lock free. A generation queued behind that barrier holds no cancel event yet, so the signal cannot reach it, and Python locks are not FIFO: it won the lock the instant the barrier released it, read a still-loaded state and denoised a whole clip against the pipeline being freed. Reproduced on both paths, where the queued generation returned a finished MP4. Mirror the image backend: count waiters, refuse a generation while one is pending, and tear down inside the barrier with the counter released in a finally. _teardown_state becomes _teardown_state_locked since the caller now holds both locks. max_grad_norm and snr_gamma were bounded on one side only, so 1e309 floated to inf and passed. clip_grad_norm_ then computes an infinite clip coefficient, clamps it to 1.0 and scales nothing, and min(snr, inf) / snr makes every min-SNR weight 1.0. The run starts, reports normal progress and trains with the requested knob silently disabled. Measured both. NaN already failed the bounds; allow_inf_nan makes that explicit. learning_rate and flow_shift already guard this exact vector. test_load_routes_to_sd_cpp_on_cpu stubbed ensure_sd_cpp_binary but not ensure_sd_server_binary, which select_and_activate_engine probes first with installs enabled, so the unit test downloaded and unpacked 108 MB into the developer's real ~/.unsloth when UNSLOTH_STUDIO_HOME was unset. Now 0 bytes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Key the video schedule reset on the resolved defaults, not the repo id A GGUF repo holds several variants, so another client swapping a distilled build for the base one from the same repo changes the steps and guidance the backend reports while repo_id stays put. modelChanged stayed false, the page kept the previous schedule, and later generations ran an 8-step distilled setting on a model that expects roughly 40 steps and CFG 4. Include the reported defaults in the key. They only move when the resident artifact does, so a manual steps change still survives a status poll. * Retire the release-lag pin now that 2026.7.6 carries the relaxed gates The virgin Windows container row that installs from PyPI on purpose was tolerated with continue-on-error, because a Server Core container has no Microsoft Store and so no winget, and the released studio/setup.ps1 hard-stopped on a winget-only git gate and reached for winget again for the VC++ runtime. #7549 relaxed both, but it landed on 2026-07-28 and the newest wheel was 2026.7.5 from the 23rd, so the row could not get past it. The pin carried its own tripwire for exactly this moment, and it has fired: unsloth 2026.7.6 shipped 2026-07-29 and the released wheel now installs end to end, so the step reported success and the assertion errored with "delete this pin and drop continue-on-error from the Install step so this row gates". Doing that. Both rows now gate unconditionally, and the assertion that enumerated the accepted failure signatures goes with the pin it protected, since there is no longer an accepted failure for it to describe. The other three release-lag mentions in this file are descriptive rather than tolerated-failure pins: they explain why the fedora and ubuntu-nonroot legs run with overlay: true, and both already refuse a triton failure as a regression rather than accepting it as lag. Left alone. * Pin the hosted pre-quant onto the plain-torch fp8 kernel A Z-Image GGUF pick at the DEFAULT speed mode, once the hosted fp8 pre-quant is actually reachable, dies at generate with an HTTP 500: torch._dynamo.exc.Unsupported: Operator does not support running with fake tensors. Developer debug context: unsupported operator: mslk.f8f8bf16_rowwise.default _fp8_config already pins KernelPreference.TORCH when it BUILDS a config, precisely because the default AUTO switches to the MSLK kernel wherever an mslk package is importable (sm90+). A hosted checkpoint escapes that pin completely: the preference is serialized per Float8Tensor, and all 239 weights in the published Z-Image-Turbo-FP8 checkpoint carry AUTO. Loading it re-arms the very kernel the pin exists to avoid, mslk.f8f8bf16_rowwise has no fake impl, and the first compiled generate therefore cannot be traced. Isolated away from the product to be sure of the mechanism: quantise one Linear three ways on this box and compile each. KernelPreference.TORCH eager ok, compiled ok KernelPreference.AUTO eager ok, compiled FAILS on mslk.f8f8bf16_rowwise library default eager ok, compiled FAILS the same way So the pin is correct and necessary, and the only gap is that the hosted path never got it. _validate_checkpoint checks scheme, granularity, base model, min_features, exclude tokens and fast_accum, but not this. Rewriting the preference on load is safe: it selects a matmul kernel, it is not weight data, so the tensors stay bit-identical and the checkpoint's own state_dict_sha256 still describes them. It is also the faster path compiled, since the opaque extern call blocks inductor quantize fusion. Why this went unnoticed: the pre-quant repos are private, so nothing that could not read them ever took this path. It becomes the default the moment they are readable. Verified end to end at 1024x1024, 8 steps, speed_mode default, on the published checkpoint: before HTTP 500 at generate, diffusion.generate_failed after pinned 239 weights to the plain-torch fp8 kernel, load 10.0 s, transformer_quant fp8 with compiled engaged, cold 7.9 s, warm 0.87 0.87 0.87 0.86 0.87 s That warm number also beats the 1.4 s recorded for this shape on 2026-07-26. 159 prequant and transformer-quant tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cover the fp8 kernel pin so the 500 cannot come back quietly The hosted pre-quant re-arming MSLK was only visible as an HTTP 500 on a compiled generate, which needs a GPU, a readable private repo and an importable mslk to reproduce. None of that is available in CI, so the pin would rot unnoticed. Three hermetic cases on _pin_kernel_preference instead: AUTO weights are rewritten and already-TORCH ones are left alone (counting only what changed), a weight that refuses the assignment does not sink the whole load, and with no torchao enum available the checkpoint is left exactly as saved. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: tighten comments across the image generation, video and training code Rewrite the comments added by this branch to be shorter and clearer: collapse multi-line explanations into one or two lines, drop comments that only restate the code, and keep the rationale that explains why a choice was made. No code changes. * Studio: let the pipeline-class guard survive a host with no diffusers Backend CI installs the CPU-only dependency set, which has no diffusers, and runs without a GPU. 13 tests failed there for three separate reasons, one of which is a real product bug rather than a test-environment gap. assert_pipeline_class_available did a bare "import diffusers", so on any host without it the call raised ModuleNotFoundError instead of the ValueError its own contract promises. /images/download-plan catches only ValueError and FileNotFoundError, so that escaped as a bare 500 with the message lost, which is the exact failure the guard exists to prevent. It is reachable in production: the native sd.cpp engine serves GGUF picks on a CPU or Apple host where diffusers is never installed. Absent diffusers now returns, since the check answers "is the installed diffusers new enough for this family" and there is no version to judge; a pick that genuinely needs diffusers still fails in the loader with its own message. The rest were tests asserting through gates unrelated to what they cover: - Three cond_cache_dir route tests drive a DiT family, which the start route refuses without an accelerator. They now take the existing dit_train_host fixture, written for this case, so they keep testing the schema on every host. - test_download_plan_forwards_the_load_time_controls stubs the diffusers planner but sends a GGUF pick, which routes to the native planner on a GPU-less host. It now pins the engine, so it tests kwarg forwarding rather than the hardware. Engine selection keeps its own tests. - The pipeline-class guard test needed a real diffusers only for its sweep over every shipped family. That half is split out and skips on its own; the stub-driven refusal needs no diffusers and still runs. Added a test for the absent-diffusers contract above. 409 pass on a GPU host with diffusers; 301 pass and 1 skips with diffusers blocked and no GPU. * Studio: cut the image generation comments down further Second pass over the comments this branch adds. Collapse the multi-line blocks that still read as paragraphs, rewrite the longest one-liners so they say the same thing in fewer words, and drop a stale comment that had drifted away from the constant it described. No code changes: only comments and whitespace. * Studio: workflow rail, sidebar Video row, and Train settings polish Sidebar - Pin Video under Images by default; persist store bumped to v3 so an untouched install adopts it while a custom arrangement is left alone. - Add "Customize sidebar" to the end of the More flyout, opening Settings > Appearance scrolled to the sidebar nav section. Images - Replace the workflow dropdown with an icon rail down the left edge: all seven workflows visible, keyboard nav, tooltips carry the labels. - Workflows stay selectable with no model loaded, so one can be set up before picking a model. Generate is already gated on a loaded model. - Video moves out of the Create/Train strip into a link at the far right, with a matching Images link on the Video page. - Wider gutters around the settings column, headings matched to Train. Train - Field guidance moves into "i" tooltips; only state that limits a control stays on the page. Steps, LoRA rank and the rest gain hints. - /info reports params, qlora_vram_gb, gated and note as fields, so the family note renders as chips. vram_note is rebuilt from them unchanged for older clients. Shared - Number steppers appear on hover or focus. - Lighter control border (#e9e9e9) on Images and Video, light mode only. * Studio: move Advanced inline on Images and Video - Replace the top-bar toggle and right-docked Advanced panel with a disclosure under Seed, so load-time tuning sits with the settings it affects and opening it no longer shrinks the preview canvas. - Video uses the bordered variant, since offload and memory decide whether a model fits at all. - Open state persists per page in localStorage. - Tighten the Steps unit trigger: 14px chevron, less right padding. * Studio: full width media pages and layout polish - Drop the 1100px cap on Images Create, Images Train and Video, so the preview canvas grows with the window instead of sitting in a band. - Align the workflow rail to the model selector label, widen the gap before its divider, and restore the divider itself. - Advanced: more room above and below, icon and label sized to the slider rows, and the same quiet row on Video as on Images. - Give Video a pane heading and description, matching Images. - More space above Seed on both pages. - More no longer takes the active style when the current page is one of its own rows. * Studio: fix the Extend side toggles The resting outline used a ring, and index.css blanks the ring on the button holding mouse focus, so the side you just clicked lost its border until focus moved. Use a border for the outline and leave the ring to focus-visible. Also separates the two states properly (the off state had no surface of its own), drops the border in dark mode as the inputs do, and adds the aria-pressed and focus-visible styling the buttons were missing. * Studio: drop the border on the Extend side toggles Light mode should carry no border either, so the fill alone marks the state: a muted surface when off, a primary tint when on. Matches the borderless treatment dark mode already had. * Studio: move the Images workflows into the sidebar The workflow switcher was a vertical icon rail on the page. It now lives in the sidebar under Images, so the page keeps that width for the canvas. - Workflows list under the Images row, with a chevron to fold them away. Hovering the row peeks them in a flyout on the standard menu surface. - Clicking Images while already there toggles the list instead of navigating. - The listed workflow carries the highlight, so the Images row drops it. - Create takes its own icon: it was sharing the New chat pencil. - Model hub moves above Projects. Persisted layouts bump to v4 and only adopt the new order where the stored one is still a shipped default. - Images and Video content both start at 32px, clear of the sidebar. * Studio: simplify the Images workflow submenu Dropping the hover flyout: two ways to reach the same seven workflows, one of them overlapping the list right below it, read as clutter. - The flyout is gone. The workflows are rows under Images and nothing else. - On the Images page they are always open, since they are that page's switcher. Elsewhere they stay folded, and the row's chevron opens them. - Create takes the sparkles icon. - Create and Train both pad their settings column to 40px a side, so the two tabs line up with each other and with the model selector. * Studio: dock the generate action and tidy the Create controls The primary action sat at the foot of a long scroll, so it was off screen until you scrolled for it. - Generate, Video's Generate and Train's Start training float at the bottom of their settings column. No bar behind them: hover lightens the fill rather than thinning it, and the disabled state is opaque, so the controls underneath never show through. - Aspect ratios read as names: Square (1:1), Widescreen (16:9) and so on. - Width and height replace their sliders with two compact boxes. Type a size or pick one from the menu; the value still snaps into range and still drives the locked ratio. - Negative prompt gets a quiet disclosure under the prompt. It was there before but only above guidance 0, which is not the default, so it never showed. - The info "i" is smaller across the settings UI. * Studio: reveal Images and Video field hints on hover The "i" next to every field label sat there permanently, which made a column of settings read as busier than it is. Scoped to the diffusion pages by CSS rather than per component: the hints come from the page's own Field and SelectRow, the train panel's FieldLabel, and the chat ParamSlider, which is shared and should not change for chat. Reveals when the pointer is inside the field, on focus-within for keyboard, and stays inert while hidden. InfoHint grows a data-slot so the rule has a stable hook, rather than keying off its aria-label. * Studio: one-line sliders, edge fades, shared negative prompt - Steps, Guidance and the rest put label, track and value on one row. ParamSlider is shared with chat, so this is an opt-in prop and chat keeps the stacked layout. - The settings columns fade at whichever edge they run past, as the sidebar and model picker do, instead of cutting off. New useScrollFades hook drives all three. - Negative prompt moves to a shared component and Video picks it up, so both pages collapse it the same way. - Right gutter sits closer to the rule, Steps takes a bigger break above it, and the Advanced rule sits between its neighbours rather than up against the field above. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: add the API monitor to the sidebar nav Sits under More, below Export. Points at the monitor page, not the API keys dialog the profile menu opens. No store version bump: an id missing from a stored layout is appended with its default pinned state, so existing sidebars gain the row at the end and keep their own order. * Studio: take main's copy of the sidebar spinner test The merge kept this branch's older version, which pins the Recents row to h-[33px]. Main already relaxed that to any height, since row density moves independently of the trailing column the test is about. * Studio: fix FLUX.2 klein size resolution, HunyuanVideo tier routing, and the personalization save Five fixes found while auditing the model registries against the live Hub. FLUX.2 klein 9B loaded against the 4B config. One family covers both klein sizes and defaults to 4B, relying on the base_model card tag for the real base, but the trust allowlist only had klein-4B, so the correct tag was discarded and a 9B GGUF (inner_dim 4096) was loaded against a 4B config (3072). That surfaced as a bare shape mismatch from inside the GGUF quantizer naming neither the file nor the repo. Adds the missing allowlist entries and a header-level size check that fails early with a legible message; the check is fail-open, so an unreadable file, a non-FLUX.2 family, or an unmapped base leaves the load exactly as it was. The sd.cpp text encoder rule matched the literal "klein-9b", so klein-base-9B was handed the 4B encoder. HunyuanVideo-1.5 720p checkpoints routed to the 480p family. Only the literal 720p_t2v path was aliased, so 720p_i2v and every GGUF repack fell through to the generic token and inherited the 480p base repo, which also supplies the VAE and text encoder. The tier is baked into the weights (transformer target_size 640 vs 960, scheduler shift 5.0 vs 9.0) and the two bucket lists are disjoint, so this ran the whole pipeline off-tier. Resolution presets that the checkpoints were never trained for. Wan2.2 TI2V-5B is 720P-only upstream (SUPPORTED_SIZES is exactly 704x1280 and 1280x704, asserted in generate.py), and the HunyuanVideo 480p square preset 624x624 is not a bucket of its tier; 640x640 is. Personalization could never be saved. The frontend ships eight sidebar nav ids and refills every missing one on each save, but the backend Literal had seven and no "api", so every PUT to /api/settings/personalization returned 422. Aligns the server defaults and the Literal with the shipped layout. A catalog row pointing at a file that does not exist. unsloth/Qwen-Image-2512-FP8 holds torch prequant .pt checkpoints, not qwen-image-2512-fp8.safetensors, and fp8 is denied for this family anyway because it renders black, yet the row was the auto-route target on any GPU above 40 GB. Also marks FLUX.1-schnell, Krea-2-Turbo and the two Ideogram repos gated: all four are gated on the Hub today, and schnell being Apache-2.0 does not make it ungated. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: stop a case-only duplicate upload destroying an image, and size the load against the right GPU Two fixes from the review round. A dataset upload containing two names that differ only by case silently loses one on Windows and macOS. The in-batch duplicate check is an exact string compare, so "Cat.png" and "cat.png" both pass, but on a case-folding filesystem they are one destination: the commit step moves the first staged part aside, writes the second over it, then deletes the backup, while the response still reports both as uploaded. Captions collapse the same way and never even reach the image branch. No single folder can hold such a pair, but the Windows and macOS open dialogs flatten search and Recents results across folders into one multi-selectable list, which is a normal way a LoRA dataset gets assembled. Rejecting the pair everywhere would regress Linux, where the two really are different files with their own sidecars, so the check probes the filesystem once per process instead of keying off sys.platform: macOS also ships case-sensitive APFS volumes, and a Linux host can keep its Studio home on an exFAT or NTFS mount. A failed probe answers "case-sensitive", which leaves today's behaviour untouched. The quantization fit budget was sized against the wrong card under a reordering CUDA_VISIBLE_DEVICES. A bare "cuda" load lands on visible ordinal 0, but the reducer took the lowest PHYSICAL index, which the nvidia-smi path reports as index_kind "physical". Under "3,1" that sizes against GPU 1 while the pipeline loads onto GPU 3. The hook's own interface doc already promised the lowest visible ordinal. Ranks by visible_ordinal, falling back to index only for an older backend that omits it. * Studio: rework the diffusion dataset panel and align the pane gutters Upload: - one Upload button beside the dataset name, with an upload icon - picking files uploads them, no second confirm click - Add button beside the Training images dropdown, so images can go into a set that already exists - fall back to the upload form when a selected dataset no longer resolves Labeling grid: - two columns at any width; sm:grid-cols-3 crushed the tiles in a fixed width column - pin the caption size so the Textarea's md:text-sm does not outsize a tile - name the tile hover group; a bare one revealed every tile's Remove at once - Remove is an icon button on the image, not a word over the artwork - drop the amber tile fill, keep the No caption label - match the header, toggle and status text to the section's type scale Gutters: - the run area and the preview canvas now sit 40px off the rule, the gutter the settings column has off the page edge * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: give the dataset preview per-image hover and remove, and wrap it onto rows - each thumbnail is its own button, so it can carry an X that removes just that image; the strip was one button and could not nest a second control - hover lightens the image instead of drawing a border - wraps onto more rows rather than scrolling sideways, which also keeps the bottom corners rounded - samples 12 thumbnails, up from 8, now that a second row is available - a delete refreshes the panel's dataset counts * Studio: show the workflow icon beside the Images and Video headings Images reads it off WORKFLOW_TABS, so it is the same icon the sidebar submenu shows and it follows the active workflow. Video's heading is also Create, so it takes the Create icon rather than the nav row's film slate. * Studio: reject a duplicate LoRA id on the diffusion load path too DiffusionGenerateRequest already refuses a repeated adapter id, with a comment naming the hazard: _resolve_lora_set suffixes colliding adapter names, so the same id resolves the SAME adapter twice and set_adapters stacks both copies past the per-adapter weight bound. DiffusionLoadRequest bounds only the list length, and it matters more there. Generation-time stacking spoils one image; on the load path the adapters are baked into the quantized build before compilation, so the unintended combination rides every image until a reload. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: teach the GGUF reuse test double about holds_no_vram The resident-GGUF fast path consults llama_backend.holds_no_vram before asserting CHAT ownership, but two tests hand it a SimpleNamespace built before that attribute existed, so they raised AttributeError and returned a 500. A real LlamaCppBackend exposes it as a property; the doubles now carry it too. Product code is unchanged: these pass on main and were the branch breaking its own test, not a defect in the guard. --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: michaelhan <michaelhan2050@gmail.com> Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: Michael Han <107991372+shimmyshimmer@users.noreply.github.com> | 1 个月前 | |
Use Unsloth for bare Studio product references (#9732) * Studio: use Unsloth for product shorthand * Tests: restore case-folding fixture path * Complete Unsloth product reference rename * Tests: restore internal Studio references | 9 天前 | |
Use Unsloth for bare Studio product references (#9732) * Studio: use Unsloth for product shorthand * Tests: restore case-folding fixture path * Complete Unsloth product reference rename * Tests: restore internal Studio references | 9 天前 | |
Reduce and tighten code comments and docstrings repo-wide (#6095) Trim and tighten code comments and docstrings across the repository. Comment-only: every changed file verified code-identical to main via AST/token comparison. | 2 个月前 | |
Fail CI on a new exec/eval/compile of a value that is not written out (#9999) * Fail CI on a new exec/eval/compile of a value that is not written out The two holes fixed in unsloth-zoo#1108 and unsloth#9777 were the same shape: exec of an HTTPS response body, and exec of a string built from a downloaded config. CodeQL runs on this repository with Python enabled and raises py/path-injection here, but it has never raised py/code-injection, so the class is not covered by what is already running. The rule is the blunt one: the first argument to exec, eval or compile must be a written-out string. The 72 existing call sites are recorded in scripts/exec_literals_baseline.json, each with a reason, so the gate starts green and only new sites fail. An entry carries a digest of the call's own source rather than a line number, plus a count, so moving code does not churn the baseline and a new call cannot hide behind a removed one. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the encoding when reading the workflow file tests/test_source_read_encoding.py gates every read of a checked-in file on an explicit encoding, so the file does not break on Windows the day it gains a non-ASCII byte. --------- Co-authored-by: danielhanchen <elliegouldingstuff@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> | 5 天前 | |
Studio: add image generation, editing workflows and LoRA training with Unsloth GGUFs (#6763) * Tighten comments in the image stack tests and scripts * Close video single-file, training reservation, and image mount-resume gaps Route on-device single-checkpoint video folders through the single_file loader: a bare local .safetensors directory (no model_index.json) is advertised as a pipeline with no filename, so validation rejected it before it could load. Reinterpret the pick as a single_file load of the sole checkpoint, mirroring the image load route. Treat a reserved-but-not-yet-spawned LLM training start as active in is_training_active() so /images/load, /video/load, and /diffusion/start cannot race the reserved run for VRAM during the pre-spawn free window. Mirrors the diffusion training service reservation. Resume an in-flight image generation on the Images page mount: probe generate-progress, re-enter the poll loop, and refresh the gallery on completion so a run started elsewhere is reflected and its saved image appears without a manual refresh. Seed resident image defaults from the resolved base_repo rather than a possibly path-shaped repo_id so the first resident generation uses the right recipe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Publish image generation active state before pre-denoise setup generate() assigned self._gen only at the pipe() call, after deferred compile, LoRA resolution/application, and ControlNet download/build had run. Across that setup window generate_progress() reported inactive even though _generate_lock was held, so a reloaded page's mount probe showed idle and let a second generate queue behind the first. Publish an active step-0 _GenState the moment the generation lock is acquired, before the setup work, and clear it in the outer finally so a setup-time error cannot leave the UI stuck active. Mirrors the video backend's queued phase and the training start guard. * Studio: fix diffusion install ownership, dataset upload atomicity, gallery pagination, and teardown races install_sd_cpp_prebuilt: only write the .unsloth-studio-owned marker when the install created the target directory or it was empty. Adopting a pre-existing, unowned, non-empty directory (a user's own stable-diffusion.cpp checkout) made it eligible for the uninstaller's recursive delete. routes/training upload: make the multi-file promotion transactional. Back up each displaced original and roll every destination back on any failure, so a mid-loop rename error can no longer partially overwrite the live dataset. routes/training _resolve_dataset_folder: reject a symlinked dataset directory and prove the resolved folder stays under the datasets root, so image read/caption/delete cannot escape the root through a link. routes/training delete: escape glob metacharacters in the thumbnail filename so deleting an image named like [ab].png removes only its own thumbnails. image_gallery / video_gallery listing: filter records against the response schema inside the pager via a valid callback, so offset/limit/has_more all count over accepted records. A leading schema-invalid record no longer returns an empty page with has_more=true and stalls infinite scroll at offset 0. image_gallery / video_gallery save: publish via a temp file plus atomic rename (the sidecar is the video pair's commit marker) and clean up on failure, so a partial write never surfaces a truncated PNG or strands an orphan MP4. diffusion_train_common discovery: treat an empty caption sidecar as a metadata tombstone that still falls through to the dreambooth instance prompt, so clearing every metadata caption no longer fails with no captioned images found. diffusion backend unload: wait for an in-flight denoise to exit before tearing down process-wide patches and state, mirroring the load path. diffusion_engine_router: serialize the whole check/unload/publish transition so a concurrent selection cannot return the engine being unloaded. uninstall.ps1: gate the default sd.cpp process stop on the owner marker so a user's own sd-server is not terminated for a directory we then keep. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reject native batch seeds outside the JSON-safe range * Refuse sd.cpp install into unowned non-empty target dir When the install target already exists, is non-empty and lacks the .unsloth-studio-owned marker (a user's own stable-diffusion.cpp checkout, or unrelated files beside a custom Studio root), install() previously still extracted the release into it. Skipping the ownership marker only stopped the uninstaller from deleting the directory; extraction still merged binaries into the user's working tree and could overwrite same-named files. Fail up front with a clear message pointing the user at a fresh/empty location before any download or extraction, leaving their directory untouched. Update the ownership test suite to assert the refusal. * Studio: gate dataset uploads on the symlink check and surface local video single-file checkpoints * Tighten comments and docstrings added by the image-generation fixes * Studio: close arbiter load-registration race and surface native progress + local pipeline folders Publish native sd.cpp generate progress (_gen) before LoRA resolution so a reload probe reads active during setup, matching the diffusers path. Register the diffusion/video GPU load under the arbiter lock (acquire_for now takes a register callback) so a competing acquire cannot evict an owner before its load is marked in-flight and let two loaders allocate VRAM at once. Admit local diffusers pipeline folders (root model_index.json, weights in component subdirs) in the local model scan so they reach task tagging and the On Device picker. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: include marker-owned custom sd.cpp roots in the uninstall stop scan; harden Pester install against the nuget.exe PSGallery bootstrap * Studio: surface local pipeline scan roots, tag single-file checkpoints by filename, mark companion-only pipelines partial - _scan_models_dir: admit a scan folder that is itself a diffusers pipeline (root model_index.json, weights in transformer/ vae/ subdirs). _is_model_directory rejects such a root, so the child scan would list the component subdirs as bogus models and hide the real pipeline; treat the root as one model via _local_pipeline_index. - _local_is_diffusers / _local_model_task: include the sole checkpoint filename in the family-detection needles (_local_family_needles, resolved via resolve_local_single_file). A generically named folder holding one loadable qwen-image-*.safetensors / ltx-*.safetensors identifies its family only from the filename; the load route already resolves that file, so tag it or the task-scoped picker (which rejects task=null) hides the on-device model. - list_cached_models: mark a companion-only base snapshot partial. A GGUF image load prefetches the base repo's VAE / text-encoder / model_index.json but skips the transformer (the GGUF supplies it); the snapshot has a pipeline manifest yet is not a loadable BF16 pipeline, and _cached_repo_partial misses it. _repo_pipeline_missing_denoiser flags a pipeline snapshot whose transformer/ or unet/ component carries no weight, so the picker drops it instead of advertising it as fully on-device. * Studio: preserve foreign gallery files, force safetensors on remote ControlNets, and close dataset/seed/GPU gaps Gallery clear/delete now scope to Studio-owned files: image_gallery and video_gallery skip PNGs / MP4s without a readable recipe (a hand-dropped or orphan file the listing already hides), so clear() and a guessed-id delete no longer destroy files the gallery never surfaced. Remote ControlNets now force use_safetensors: a bare owner/name reaches from_pretrained without the base trust gate, and the Hub scan fails open when unavailable, so requiring safetensors closes the pickle deserialization vector. POSIX uninstall now stops resident sd-server / sd-cli under an owned sd.cpp root before removing the tree (marker-gated), mirroring the Windows stop-before-delete scan; a live native server no longer survives unlinking its binary. Diffusion dataset containment: the training-start read path and the discovery picker route bare names through the protected resolver, so a symlinked dataset is rejected / not advertised like the caption/delete routes already do. Uploads gain the inference decode guard (oversized real images 400 before OOMing the trainer) and dataset upload/caption/delete/import are blocked with 409 while a diffusion run is active. JSONL readers (trainer + routes) tolerate non-object JSON and invalid UTF-8 instead of raising AttributeError / 500. LoRA family compatibility is enforced in the shared resolver, not only the picker, so a direct API client cannot apply a mismatched-family adapter. GPU arbiter gains release_if so the image/video unload idle-check and release are atomic against a concurrent same-owner load's registration. Native batch recipes persist the base batch_seed and restore replays from it, so a native batch_index>0 image no longer advances its seed twice. FLUX.2-klein selects its sd.cpp text encoder by variant (4B -> Qwen3-4B, 9B -> Qwen3-8B) instead of the single family default. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: track the loaded GGUF filename so native companion resolution reproduces the load identity * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: gate local-pipeline image tagging on a real family; validate video sidecars before delete/clear * Studio: tighten image-generation fix comments and docstrings * Studio: gate gallery serve/export on ownership; keep image progress active until persisted; reserve diffusion training before the dataset scan * Studio: restore Reapply target on async image/video load errors; recheck training state before dataset commit; reclaim partial sd.cpp installs on retry Images/Video: a background model load that fails AFTER starting (error/eviction during download) leaves the previous pipeline resident, but handleLoad had already overwritten lastLoad.current with the failed pick, so "Reapply to loaded model" reloaded the failed model. Carry the prior Reapply target into the poller and restore it on the async error/null paths, mirroring the quant rollback. Training: an in-flight diffusion dataset upload passed _require_diffusion_dataset_mutable() at entry but could still commit files after a concurrent /diffusion/start reserved the training slot, mutating the dataset underneath the trainer. Re-check the interlock immediately before the commit phase; a 409 there leaves the staged temps for the finally to clean. sd.cpp install: an interrupted extraction (disk full, killed process, a raising post-extract cudart fetch) left the target non-empty with no owner marker, so the next lazy install tripped the "not a Studio-managed directory" refusal and wedged native install. Write the ownership marker before the partial writes when the target is reclaimable, so a retry recognises the debris as ours and re-extracts. * fp8 DiT quant: floor the dynamic activation scale with activation_value_lb An all-zero activation token row makes the dynamic per-row fp8 scale 0, which turns the quantized data to NaN and the render to black frames on torchao's plain-torch kernel path. The fused fbgemm/mslk quantize kernels clamp zero rows internally, so the bug only reproduces on machines without them, which is most user environments. Zero rows are real inputs, not a corner case: Wan 2.2 zero-pads its text conditioning, and Hunyuan-1.5 and Qwen-Image regenerate zero rows inside their transformer blocks every step. Pass activation_value_lb=1e-12 to Float8DynamicActivationFloat8WeightConfig whenever the installed torchao supports the kwarg (Float8Tensor rework, 0.13+), checked via inspect.signature so older torchao keeps exactly the current behaviour; the existing Float8MMConfig fallback chain is unchanged. Verified on GPU: with the forced plain-torch kernel path a zero-row input NaNs without the floor and stays finite with it, and end to end on HunyuanVideo-1.5 fp8 goes from a solid black frame (LPIPS 1.00) to a normal render (LPIPS 0.225); on Wan the floor matches the condition_embedder exclusion (LPIPS 0.211 vs 0.206). Same-seed renders with fused kernels present are unaffected, and pre-quantized fp8 checkpoints stay valid since weight scales are untouched. * Wire hosted pre-quantized DiT checkpoints into the image families Point prequant_repos for flux.1, flux.2-klein, flux.2-dev, qwen-image (int8 only there; fp8 is family-denied), z-image and krea-2 at the unsloth/<Model>-FP8 Hub repos carrying gate-validated int8 and fp8 transformer checkpoints, so the fast quant path loads the small pre-quantized file instead of materialising the dense bf16 transformer and quantising on device. Measured on FLUX.2-dev int8: build peak drops from 60.7 GB (dense + quantize) to 30.7 GB (hosted prequant), identical 30.7 GB resident after either path since loading a checkpoint is bit-identical to on-the-fly quantisation. The hosted repos name files <Model>-<SCHEME>.pt, so resolve_prequant_source now derives that model-name filename from the repo id (scheme suffix stripped case-insensitively) and carries the legacy transformer_<scheme>.pt as a fallback the resolver tries when the primary 404s, keeping older repos loadable. Wiring a repo also exposed a fallback hazard: with a prequant source present, the dense-fit preflight used to be skipped entirely, so a failed prequant download would fall through to the dense bf16 load the memory plan never budgeted, OOMing after eviction. The preflight now always runs and gates an allow_dense_fallback flag through _load_dense_quant_pipeline: a dense misfit still skips the fast path when no prequant exists, but with one it proceeds and a prequant failure raises to the GGUF build instead of loading dense. The same flag is set when the auto-policy replans an offloaded GGUF against a prequant-sized transient. Tests updated to the new filename convention plus new coverage for the derivation and the legacy-name fallback; the prequant-skips-refit test now asserts the re-check runs and forbids the dense fallback. Verified end to end on GPU: z-image int8 resolves the hosted repo, downloads the model-name file and renders (6.8s load, 5.9 GB peak). * Route krea-2 through its per-component loader on the transformer-quant fast path _assemble_pipe used Pipeline.from_pretrained for every family, but the krea repo ships transformers-5.x configs and no top-level tokenizer files, so the tokenizer dies with vocab_file=None. The pre-quantized checkpoint loaded fine and then the assembly crashed, dropping the load to the GGUF build, which krea-2 cannot take (Krea2Transformer2DModel has no from_single_file). Assemble per-component via load_krea2_pipeline like the pipeline-kind and single-file paths already do. Verified live: Krea-2-Turbo int8 and fp8 hosted prequant loads now assemble and render through the Studio images tab. * Keep Qwen-Image's text-stream linears bf16 on int8 (short prompts break torch._int_mm) Qwen-Image's MMDiT runs every text-stream Linear at M = actual prompt tokens: the Qwen2.5-VL embeds are not padded to a fixed length like FLUX's 512-token T5. A short prompt (13 tokens) or the near-empty negative prompt drives torch._int_mm below its M > 16 floor and the first denoise step raises 'self.size(0) needs to be greater than 16, but got 13' (measured on B200 through the Studio images tab). Add per-family int8 exclusions (txt_in, add_q/k/v_proj, to_add_out, txt_mlp) for qwen-image and qwen-image-edit, threaded through exclude_tokens_for_scheme(scheme, family) and the prequant checkpoint validation, so a checkpoint baked under the old token list is rejected and re-quantised instead of loaded crashing. The text stream runs at M = tens vs the image stream's M ~ 4k, so the exclusion costs nothing; the rebuilt hosted checkpoint gates 28/28 PASS with LPIPS mean 0.057 (was 0.069). * Harden the diffusion memory plan against transient free-VRAM undercounts A cold FLUX.2-dev int8 load on an idle 183 GB B200 planned offload=model (companions exceed budget) and silently served the GGUF as-is; the identical retry went resident and engaged the hosted prequant. The plan arithmetic was byte-identical across both loads (required 90,228 MiB, resident needs free of about 124 GB); the only divergent input was torch.cuda.mem_get_info, which is device-wide and instantaneous: a transient foreign CUDA context briefly held about 100 GB at the first snapshot, and the planner trusted that single read. Three changes: - settled_snapshot_device_memory: on cuda, synchronize + empty_cache (best-effort) and take the MAX free over up to 3 spaced reads. A transient can only shrink free, so the max rejects transient undercounts while a persistent tenant still caps every read. _plan_memory now uses it. - plan_fits_total_capacity + one replan retry: when the dense/prequant candidate fits TOTAL device capacity under the standard reserve and the 0.85 resident margin, an offload verdict can only stem from the free reading, so the loader re-snapshots and replans once before declining the fast path. Explicit balanced/low_vram modes skip the retry (they offload by mode). - diffusion.transformer_quant_declined log line with required/budget/free and the plan reasons, so the next decline is diagnosable from the server log (previously silent). Verified: cold FLUX.2-dev int8 first load in a fresh server now engages the hosted prequant resident (offload=none). * Add FLUX.2 Klein and FLUX.2-dev DiT LoRA training Register flux.2-klein and flux.2-dev in the DiT trainer following the upstream DreamBooth references: latents train patchified and batch-norm normalized from the VAE posterior mode, the packed forward reuses step-invariant position ids, and the guidance vector (3.5) is gated on the variant's guidance_embeds config. Conditioning stacks load per variant (Mistral via Flux2Pipeline for dev, Qwen3 via Flux2KleinPipeline for Klein) and are encoded and freed before the transformer lands on the device. The fused single-stream to_qkv_mlp_proj joins the attention projections in the LoRA targets; the single-stream out projection stays dense because its to_out suffix would also match the double-stream ModuleList container. Wire both families through the training registry (family set, labels, VRAM notes, rank 16 / lr 1e-4 defaults, bf16-only preflight), mark them trainable with train base repos in the family registry, add FLUX.2-dev to the gated-repo token check, and trust both official bases for training downloads. Verified on B200: 30-step klein int8 (19.6s) and nf4 (20.9s) and dev int8 (52.0s) runs train with finite decreasing loss and the saved adapters apply on the bf16 base pipeline (weight 0 reproduces the base image exactly, weight 1 visibly restyles it). * Support LoRA adapters on torchao int8/fp8 quantized image pipelines Adapters are baked at load time: they attach to the dense transformer, then quantize_ converts only the frozen base linears (the lora_ side path is excluded by name), then the loader compiles. Post-quant PEFT injection is not possible on a manually quantized module, so the prequant shortcut is skipped for a baked load and the memory plan is sized for the dense build (force_dense on the quant candidate). At generation time the baked topology is frozen: weight tweaks and disabling (scale 0 reproduces the quantized base exactly) go through set_adapters, while adding or removing adapters returns a clean 400 telling the client to reload with the new selection. supports_lora now returns True for int8/fp8 diffusers loads (checked before the gguf-kind early return, since the quant fast path keeps the picker kind); nvfp4/mxfp8 and GGUF-via-diffusers stay blocked. The load request model takes an optional loras list, threaded through begin_load on both engines (native ignores it and keeps applying LoRA at generation). Verified end to end on GPU: Z-Image GGUF picker + int8 + trained adapter loads through the API, bake marker logged, weight 1.0 vs 0 renders differ visibly, weight 0.5 accepted live, unknown adapter rejected as 400. Affected suites: 296 passed. * Add FLUX.1 Krea dev to the image model catalog Krea's guidance-distilled FLUX.1-dev finetune keeps the exact dev layout, so it runs under the existing flux.1 family unchanged. Wire it up end to end: - Catalog group with the gated official bf16 pipeline and the open QuantStack GGUF quants; the gated artifact is skipped on auto-routing when undownloaded. - Trust the official repo for non-GGUF from_pretrained loads, next to the other black-forest-labs bases. - Generation defaults: 28 steps at guidance 4.5 per the model card. The generic "krea" defaults key (Krea-2-Turbo's 8-step no-CFG recipe) used to swallow the id, which would have produced garbage output; the new flux.1-krea key precedes it on both the backend table and the images page table. - The flux.1 prequant checkpoints are schnell-based; the loader's baked base_model_id validation refuses them for the Krea-dev base, so int8/fp8 requests dense-quantize instead (covered by existing prequant tests). * Resolve pre-quantized checkpoints per base variant One family entry covers several published variants whose weights differ (flux.1: schnell, dev, Krea-dev), but prequant resolution was keyed on (family, scheme) alone, so only the default base could ever be served: the loader's baked base_model_id validation correctly refused the schnell checkpoint for dev and Krea-dev bases and every such load paid the dense download plus on-the-fly quantise. Add an optional prequant_variant_repos table on DiffusionFamily as (base_repo, scheme, repo_id) triples and thread the resolved base repo through resolve_prequant_source / usable_prequant_source and their three call sites (load fast path, memory-plan probe, auto-policy candidate). A base without its own entry keeps returning the family default, preserving the existing refuse-then-dense behavior exactly. Wire the flux.1 variants: the gate-validated unsloth/FLUX.1-dev-FP8 checkpoints (built in the earlier campaign but never reachable) and the new unsloth/FLUX.1-Krea-dev-FP8. * Add the Lumina Image 2.0 family to the image catalog Alpha-VLLM/Lumina-Image-2.0 is a 2.6B single-stream DiT with a Gemma2-2B encoder and a standard 16-channel VAE, all transformers-4.x-compatible, so the generic from_pretrained pipeline path loads it as a new lumina-2 family: - Family entry (Lumina2Pipeline / Lumina2Transformer2DModel), aliased to lumina-image-2.0 / lumina-image-2 / lumina2. No bare lumina alias: Lumina-Next checkpoints are a different arch and must stay unknown rather than crash mid-load. bf16-only upstream, so the fp16 fallback stays off like z-image. - Trust the official repo for non-GGUF loads; bf16 component table entry (ships fp32, ~5.2 GB transformer + 5.2 GB encoder bf16-resident). - Generation defaults 50 steps / guidance 4.0 per the model card, and the generate call passes the card's cfg_trunc_ratio=0.25 itself (family-gated, signature-gated): the pipeline default (1.0) runs the CFG double-forward on every step and oversaturates output. - Catalog group with the single ungated bf16 pipeline artifact (11 GB resident) plus routing assertions; images page defaults row. - No GGUF artifact: none exists upstream (only finetune/LLM quants), so the dense transformer_quant fast path (GGUF-kind-only) stays unreachable for now. Offline probes of the future prequant campaign: int8 and fp8 both engage and render cleanly (fp8 LPIPS 0.11 vs bf16, int8 0.33 from 50-step trajectory drift with intact quality), so neither scheme is family-denied. * Wire the hosted Lumina Image 2.0 int8/fp8 checkpoints Gate-validated against same-seed bf16 renders (28/28 pairs per scheme, zero failures): int8 LPIPS mean 0.146 / SSIM 0.937, fp8 LPIPS mean 0.116 / SSIM 0.946. Uploaded to unsloth/Lumina-Image-2.0-FP8 following the existing checkpoint repo conventions. * Add the HunyuanImage 2.1 family to the image backend The hunyuanvideo-community diffusers mirror carries the full stack in standard layout: a 17B dual-stream DiT (32.5 GB bf16), a Qwen2.5-VL text encoder, a ByT5 glyph encoder, the 32x HunyuanImage VAE, and guider/ocr_guider components (AdaptiveProjectedMixGuidance) that diffusers 0.39 loads natively, so the generic from_pretrained pipeline path covers everything with no per-component assembly. Family notes: - The call's guidance knob is distilled_guidance_scale (there is no guidance_scale kwarg), so cfg_kwarg routes the UI value there; real CFG runs inside the repo's guider at its baked scale. Defaults follow the card recipe: 50 steps, 3.25. - 2K-native: verified live at both 1024 and 2048. - Coexists with the HunyuanImage-3.0 structured exclusion (3.0 has no diffusers pipeline and stays excluded with its stated reason). - int8/fp8 dense quantization verified live (LPIPS 0.186 both vs same-seed bf16); a short prompt does not trip the int8 torch._int_mm minimum on this arch, so no family exclude entry is needed. - bf16 component table for the memory planner: (32.5, 16.3, 0.8) GB. * Surface HunyuanImage 2.1 in the image model catalog Catalog group with the open bf16 mirror pipeline (~50 GB resident, so a bare click on a consumer card routes to the QuantStack GGUF quants, which load and render through the generic GGUF path, verified live) plus the images page defaults (50 steps, guidance 3.25 feeding distilled_guidance_scale). * Add the HiDream-I1 family to the image backend A 17B MoE DiT (16 double + 32 single layers, 4 routed experts) with four text encoders, on HiDreamImagePipeline (diffusers 0.39). One family covers the open Full / Dev / Fast repos (same arch); per-variant generation defaults follow the upstream inference recipes (Full 50 steps at guidance 5, the distilled Dev 28 and Fast 16 guidance-free). The repos name a Llama-3.1-8B text_encoder_4 in their model_index but do not ship its weights; the official example passes the gated meta-llama repo in by hand. The loader instead assembles the component from the open unsloth mirror (byte-identical weights, already inside the non-GGUF trust gate), injected at the three pipeline from_pretrained sites, with output_hidden_states matching the official example. Memory planning counts the assembled TE4: 34.2 GB DiT + 28.8 GB encoders, ~63 GB bf16-resident. * Surface HiDream I1 in the image model catalog One catalog group with the three official bf16 pipelines (Full, plus the Dev and Fast distillations as labeled artifacts) at their ~63 GB resident size, so auto-routing keeps this a datacenter-GPU pick. city96's GGUF is deliberately not wired: the GGUF path would need the same Llama TE4 assembly for very small demand. Images-page defaults mirror the backend table with the variant keys ahead of the generic hidream key. * Pin the measured HiDream quant verdict in tests int8 and fp8 both engage and render cleanly on this family, including short prompts on int8: the routed MoE expert Linears only ever see the concatenated image+text stream (M far above the torch._int_mm minimum), so no deny entry and no family exclude tokens are warranted. Assert that so a future table edit cannot silently regress the measured behavior. * Wire the hosted HunyuanImage 2.1 int8/fp8 checkpoints Verified bit-identical to on-the-fly quantize: all 1264 state dict tensors (456 quantized) dequantize equal between the loaded checkpoint and a fresh quantize_ pass, so quality matches the runtime Dtype path exactly. Same-seed LPIPS suite means (0.35 int8 / 0.28 fp8) blend trajectory divergence with this family's own run-to-run nondeterminism (identical weights and seed reproduce a 17/255 mean pixel delta through the 50-step guider pipeline); per-case hard checks pass and the drift is compositional, reviewed visually. Uploaded to unsloth/HunyuanImage-2.1-FP8. * Fix silent LoRA drop and wasted transformer prefetch on GGUF quant loads Two live-test findings on the images load path: - transformer_quant with baked LoRAs, when the dense quantized build is declined for memory or fails: the load completed as a plain GGUF with the adapters silently dropped (HTTP success, supports_lora=false after the fact) -- wrong output with no signal. The load now fails with the recovery options (drop the adapters, free VRAM, or pick a smaller model). Weight-0 adapters still count as no bake request, and the plain no-LoRA decline keeps its silent GGUF fallback. - A fresh GGUF load on a small GPU prefetched the base repo's full bf16 transformer shards (~47 GB on Qwen-Image) because the dense-quant prefetch widening only checked scheme viability, not whether the device could ever hold the candidate resident. Gate the widening on total device capacity (reserve + 0.85 margin, the plan_fits_total_capacity bar) so a card that is certain to decline the dense build never pays the download; capable devices keep the prefetch. * Fix video progress under-reporting during load and generate Two live-test findings on the video progress endpoints: - load-progress downloaded_bytes froze mid-download: the counter used scan_cache_dir, which skips in-flight *.incomplete blobs, so it sat at the last completed blob for the whole multi-GB shard pull while the disk kept filling. Count the repo's cache directory directly (completed plus incomplete blobs, snapshot symlinks skipped so nothing is double-counted). - generate-progress reported total_steps=null / fraction=0 while step advanced: the video API only carried the native total field while the image API exposes total_steps and fraction, so one poller could not work against both. Derive the image-compatible aliases in generate_progress and declare them on the response model; the native total stays for back-compat. * Wire the hosted HiDream I1 int8/fp8 checkpoints Gate-validated: all 28 per-case pairs pass per scheme (LPIPS suite means 0.291 int8 / 0.278 fp8, in the 50-step trajectory-divergence band; CLIP delta means 0.007-0.008), and the int8 checkpoint is verified bit-identical to on-the-fly quantize across all 1615 state dict tensors (1073 quantized, max abs diff 0.0). Uploaded to unsloth/HiDream-I1-Full-FP8. * Add a pre-cast text-encoder loader for the layerwise fp8 scheme The runtime text_encoder_quant=fp8 path downloads the full bf16 text encoder and layerwise-casts it in place on every fresh load. For the heavyweight encoders (LTX's Gemma3-27B ~50 GB, FLUX.2-dev's Mistral-24B ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) that download dominates load time on a fresh machine. diffusion_te_prequant.py loads a pre-cast fp8-storage state dict instead: meta-init the encoder skeleton from the checkpoint's te_class, load_state_dict(assign=True), rebuild on CPU if non-persistent buffers stay on meta, then re-apply the same layerwise cast to install the upcast hooks. The cast is a deterministic storage transform, so the loaded encoder is bit-identical to dense-load-then-cast by construction. v1 hosts the layerwise fp8 storage scheme only: its state dict is plain tensors (torch.load(weights_only=True), no pickle execution). The dynamic-compute schemes (fp8_dynamic, int8, nvfp4) build torchao subclass wrappers at runtime and are deliberately not hosted. Checkpoints validate format, scheme, component and base_model_id before use and any problem falls back to the dense download and cast. Local path overrides reuse the DiT prequant allowlist env var. Families opt in via a new te_prequant_repos (scheme, component, repo_id) field on both DiffusionFamily and VideoFamily; the field defaults empty so nothing changes until a gate-validated artifact is wired. * Inject hosted pre-cast text encoders during pipeline assembly Wire te_prequant_pipe_kwargs into the three pipeline assembly sites: the diffusion full-pipeline branch, the diffusion transformer-only and GGUF branch (where the companion TE is the big remaining download), and the shared video assembly path before the pipeline/component split. Injection is gated exactly like the runtime cast (mode normalized to fp8, device supported, family not denied), so it can never engage where quantize_text_encoders would not; the later quantize_text_encoders call re-applies the cast idempotently and keeps status reporting truthful. With no hosted checkpoint configured the call returns {} and assembly loads the dense encoder as before. * Add the pre-cast text-encoder checkpoint builder Applies the runtime layerwise fp8 storage cast to a model's dense text encoder once and saves the cast state dict with baked metadata (format tag, base_model_id, family, scheme, component, te_class, versions) in the layout diffusion_te_prequant.py validates. Resolves the encoder class from the checkpoint's config.architectures so the recorded te_class matches what the pipeline instantiates. CPU-runnable: the cast touches storage dtypes only. * Test the pre-cast text-encoder load path Hermetic CPU coverage for diffusion_te_prequant: the checkpoint filename convention, family-table resolution by scheme and component with malformed entries skipped, resolution priority (path override, hosted repo, none) and the fp8-only scheme gate, the checkpoint validation matrix (wrong format, missing state_dict, wrong scheme, wrong component, wrong or missing base_model_id) with base case folding, the local-path allowlist refusal and missing-file fallback, and the assembly injection gating (mode, hosted entry, device support, family deny, load failure, successful injection). Also pins the te_prequant_repos field on both family dataclasses and that no family ships a hosted TE checkpoint until the campaign wires one. * Fix pre-cast TE checkpoint loading and engagement reporting Two bugs found while building the hosted checkpoints: - The builder recorded torch.__version__ (a TorchVersion object) in the checkpoint metadata, so torch.load(weights_only=True) rejected every artifact and the loader silently fell back to the dense download. Record plain strings. - Re-applying the layerwise fp8 cast to an injected pre-cast encoder raised on the duplicate hook registration, making quantize_text_encoders report the engaged cast as failed (status showed no TE quant while the encoder ran fp8). _cast_fp8 now returns early when the hooks are already installed. Also corrects the LTX TE size note: Gemma3-12B stored fp32 (~49 GB), not 27B. * Wire the hosted pre-cast fp8 text encoders qwen-image and flux.2-dev (diffusion) and ltx-2 (video) now resolve a hosted pre-cast fp8 text encoder from their unsloth -FP8 repos: - unsloth/Qwen-Image-FP8: Qwen2.5-VL-7B, 16.6 GB dense -> 8.8 GB - unsloth/FLUX.2-dev-FP8: Mistral-Small-24B, 48.0 GB dense -> 24.7 GB - unsloth/LTX-2-FP8: Gemma3-12B, 48.7 GB fp32 store -> 13.2 GB Every checkpoint verified bit-identical to dense-load-then-cast (729 / 585 / 1066 tensors, zero mismatches) and smoke-tested through the real backends with the repo engagement marker. Tests cover the wired entries, the resolver filenames, builder metadata weights_only survival, and the idempotent re-cast. * Report the compute dtype on fp8-cast encoders and inject the pre-cast TE on the dense fast path Two more findings from the hosted-TE GPU smokes: - Module.dtype reports the first floating parameter, which after the layerwise fp8 cast is the fp8 STORAGE dtype. Flux2 derives its prompt embed and latent dtypes from encoder.dtype and feeds them to randn_tensor, which has no fp8 kernel, so ANY flux.2 load with text_encoder_quant=fp8 crashed at generation (pre-existing, runtime cast included). The cast now swaps in a subclass whose dtype property reports the compute dtype; forward behaviour is unchanged. - The dense transformer_quant fast path assembles companions through _assemble_pipe, which never received the pre-cast TE injection, so the hosted encoder engaged on full-pipeline and GGUF builds but not on the fast path. Threaded through like the other two branches. Verified live on B200: qwen-image (full pipeline), flux.2-dev (GGUF picker with int8 DiT prequant), ltx-2 (video backend) all engage the hosted TE, render non-black, and report text_encoder_quant=fp8 truthfully. * Key the fp8 cast idempotency on an explicit completion marker Hook presence alone cannot distinguish a legitimately pre-cast text encoder from leftover hooks after a cast that failed mid-pass, so the early return now requires the completion marker _cast_fp8 sets once the hooks are fully installed. Leftover partial state keeps failing closed. Also tolerates non-Module encoder doubles in the hook probe and the dtype override. * Extend the fp8 TE quant to HiDream's Llama text_encoder_4 The generic quantize_text_encoders pass only covers text_encoder.._3, so HiDream's HEAVIEST encoder (Llama-3.1-8B TE4, 16.1 GB bf16) always stayed dense. TE4 is assembled separately (hidream_te4_kwargs), so the fp8 path now lives there: when the requested TE quant is layerwise fp8 and the device/family qualify, TE4 prefers the hosted pre-cast checkpoint (unsloth/HiDream-I1-Full-FP8, 8.6 GB) and falls back to dense-load-then- cast; a mid-pass cast failure reloads a fresh dense encoder instead of shipping partial state. The pre-cast loader and builder gain config_subfolder/config_overrides for standalone encoder repos whose config sits at the root and whose pipeline needs forward flags (output_hidden_states/attentions). Verified on B200: bit-identity 291 tensors (225 fp8, 0 mismatches), hosted checkpoint engages through the real backend (marker + status fp8), load 24.3 s vs 48.0 s dense, LPIPS 0.133 mean over 3 same-seed pairs vs the dense-TE render (gate 0.25), non-black frames. * Correct the ltx-2 resident TE estimate to the bf16 cast size The memory plan's bf16_components_gb held 50.4 GB for the LTX text encoder, which is the fp32 hub store of Gemma3-12B (~49 GB download), not what sits on device: the pipeline loads it torch_dtype=bf16, ~24.4 GB resident. The 26 GB over-estimate pushed the auto plan toward offload on cards that fit the real footprint. Comments and the size-table test now pin the resident semantics. * Host pre-cast fp8 text encoders for four more families Round 2 of the hosted TE set, each bit-identical to dense-load-then-cast and gated through the real backend (marker + status fp8 + same-seed LPIPS vs dense TEs): - FLUX.1 T5-XXL (text_encoder_2): 9.52 -> 5.90 GB, one artifact for schnell/dev/Krea-dev (T5 shards byte-identical across all three, verified sha256). 220 tensors, 144 fp8, LPIPS 0.109. - Lumina Gemma2-2B: fp32 hub store 10.46 -> 3.20 GB (3.3x download cut). 288 tensors, 182 fp8, LPIPS 0.041. - Z-Image Qwen3-4B: 8.04 -> 4.41 GB. 399 tensors, 252 fp8, LPIPS 0.112. NOT shared with flux.2-klein-4B: klein retrained layer 35's MLP (verified tensor diff, maxdiff 0.86), so klein hosts no entry. - Krea-2 Qwen3-VL-4B: 8.88 -> 4.83 GB. 713 tensors, 460 fp8, LPIPS 0.082. The constructor-assembled krea pipeline takes the encoder directly (load_krea2_pipeline text_encoder kwarg); the loader remaps 5.x rope_parameters and re-ties weights after assign so the rebuilt encoder matches the builder's structure. HunyuanImage 2.1 reuses the Qwen-Image artifact outright: its Qwen2.5-VL text encoder is byte-identical (every shard sha256, 16,584,414,544 bytes), recorded in the new component-level base-equivalence table the checkpoint validator consults. The injection loop now covers text_encoder.._3 so a family can host several components. Live check: LPIPS 0.123 vs dense. * Report the fp8-cast compute dtype without swapping the encoder class The dtype override swapped encoder.__class__ to a dynamic subclass, which breaks transformers' kwargs-based output recording: a fp8-cast Qwen3VLModel stopped returning hidden_states and every krea-2 generation with text_encoder_quant=fp8 crashed at encode_prompt (regression from the HiDream TE4 change; caught by the krea hosted-TE live smoke). The override is now a property shadowed on the ORIGINAL class that prefers a per-instance compute-dtype attribute, so class identity is preserved and uncast instances keep the stock behaviour. The idempotency test now pins exact class identity and the uncast-sibling fallback. * Pass the calibrated distilled sigma curve to LTX-2.3 8-step runs The 22B distilled DiT was trained against ltx_core's fixed DISTILLED_SIGMA_VALUES, but the diffusers scheduler derives 8-step spacing from resolution-shifted flow matching and lands far off at every reachable mu (second sigma 0.945-0.981 vs 0.99375, tail 0.37-0.61 -> 0.1 vs 0.725 -> 0.42 -> 0). At the distilled default step count the backend now passes the list verbatim, neutralising the scheduler's dynamic shift and terminal stretch for the call (they distort even explicit sigmas) and restoring them afterwards. Other step counts and the dev/base DiT keep the scheduler's own spacing. Live-verified on B200 through the video branch backend: the scheduler holds the exact curve after an 8-step distilled GGUF generation, config restored, healthy clip. Also reword the transformer_quant resolved reason to the measured reality: quant halves resident weights and hosted checkpoints cut load time, while per-step speed is roughly bf16 parity. * Pin the fp8 weight-quantize kernel against silent MSLK switching torchao's Float8Tensor KernelPreference defaults to AUTO, which switches the weight-quantize kernel to MSLK whenever an mslk package is importable on sm90+. Measured on B200: that changes fp8 scale rounding bitwise (8/8 FLUX matrices differ, scales ~55 percent of bytes), so a box that merely gains mslk would break the hosted-prequant bit-identity invariant; the mslk path is also slower under torch.compile (opaque extern call blocks inductor's quantize fusion, FLUX.1 fp8 e2e 1.149 to 1.624 s). Pin KernelPreference.TORCH explicitly, matching current no-mslk behaviour bit for bit; signature-gated for older torchao. GPU-smoked (finite, rel err 0.037) and pinned by test. * Shift Qwen-Image training sigmas to the inference distribution Qwen-Image's scheduler skips its static shift under use_dynamic_shifting, so the DiT trainer was drawing UNSHIFTED uniform-schedule sigmas for it (mean sigma 0.50) while inference always runs the exponential mu = log 3 shift plus the shift_terminal 0.02 stretch. Add a flow_shift config lever: "auto" (the new qwen-image default) rebuilds the training sigma table through the scheduler's own time_shift and stretch_shift_to_terminal so the draw matches the inference distribution exactly (mean sigma 0.72); a numeric value applies the standard linear shift s*u/(1+(s-1)*u); 1.0 keeps the historical identity behavior and stays the default for FLUX, Z-Image and Krea 2. The model timestep conditioning follows the shifted sigma, gathered in fp32 so bf16 rounding never skews it. Also wire two opt-in levers with off defaults: cfg_dropout (per-sample empty-prompt conditioning dropout, encoded alongside the captions before the text encoders are freed) and weighting_scheme="bell" (bsmntw-style mid-schedule Gaussian loss weighting normalized to mean 1). Verified with two 80-step rank-8 bf16 LoRA runs on Qwen/Qwen-Image (identity vs auto, same seed): both converge with finite decreasing loss and produce coherent same-seed previews. Unit tests cover the exact transform, the shifted sampling distribution, per-family defaults and config plumbing. * Add LoRA EMA, a persistent conditioning cache, and aspect bucketing helpers diffusion_train_extras hosts the opt-in training extras: LoRAEMA shadows only the trainable adapter params (warmup-ramped decay, default 0.99, exported as a second adapter under output_dir/ema), PersistentConditioningCache stores latent posterior stats and caption embeddings as safetensors keyed by content hash + family + resolution, and the aspect-ratio bucketing helpers group mixed-aspect datasets into same-area divisor-snapped shapes. The DiT trainer wires the first two behind config flags that default to the current behavior: ema_decay (0 disables) and cond_cache_dir (None disables). A fully warm cache skips loading the VAE and text encoders entirely; a cache hit is bit-identical to a fresh encode, including the per-channel qwen latent normalization. Also fixes the stale _gather_sigmas call in the perf test that still passed the scheduler instead of the sigma table. * Tighten torchao configs and note the FSDP2 design for the DiT trainer nf4 loads now enable double quantization (~0.4 bits/param off the frozen base scales at no fidelity cost), fp8 training uses the rowwise recipe when the torchao build ships it (per-row scaling confines the DiT activation outliers that a tensor-wide scale collapses), and the inference quant filter gains a per-scheme GEMM-tiling divisibility floor (16 for scaled_mm, 32 for MX blocks) so one ragged Linear cannot crash the first denoise after a clean quantize pass. plans/fsdp2_diffusion_design.md records the multi-GPU design: bf16/fp8 over FSDP2 with per-block units, LoRA attached before sharding, int8 out of scope (DTensor over the quantized subclass is undefined), per-family notes. * Batch diffusion inference with per-image seeds, an inference conditioning cache, and GGUF loader fixes Batched generation: /images/generate takes a prompts list (one image per prompt, txt2img only) or a seeds list (one prompt, one image per seed); the legacy batch_size path derives per-image seeds base..base+n-1 like the native engine. Every image gets its own torch.Generator so any batch member replays alone from its gallery recipe; the whole list runs as one forward by default with OOM backoff that halves a failed chunk, and an explicit batch_size caps images per forward. Validated 10-22x over serial engines on 32-image suites with LPIPS deltas within 0.002. Conditioning cache on the inference path: UNSLOTH_DIFFUSION_COND_CACHE_DIR (the inference sibling of the trainers' cond_cache_dir, same persistent store) wraps encode_prompt so repeated prompts skip the text-encoder forward entirely; verified bit-identical outputs. Bypassed while LoRA adapters are attached; tensor-argument calls pass through uncached. Compile cache: GGUF loads fingerprint their own bundles (quant=gguf, a different compiled graph than the dense family) and batched calls register every distinct (w, h, batch) chunk shape they ran, so the heavy GGUF batched warmups (~159 s at batch 32 on 12B-class, ~655 s on 20B CFG-batched) are paid once ever. GGUF loader: strip the sd.cpp model.diffusion_model. container prefix in the single-file converter; diffusers' FLUX.2 converter KeyErrors on it and the Qwen-Image identity mapping strands the model on meta. * Correct batched seed-replay docs to match measured behavior Same-seed images at the same batch shape are bit-identical; a solo regeneration with the recorded seed matches its batched rendition up to batch-size-dependent kernel numerics (mean abs pixel delta about 2.5/255, LPIPS delta under 0.002), not bit-exactly. The previous wording overclaimed bit-identity across batch shapes. * Note that batched bit-identity assumes a settled compiled graph The first generation issued while the deferred compile is still in flight can deviate transiently (observed once on a cold fp8 build: mean abs pixel delta 0.063/255); once the graph is settled, same-seed same-batch-shape images are bit-identical across runs. * Studio sidebar: Image03/FlimSlate icons, More flyout, Train row, New pills - Images uses Image03Icon and Video uses FlimSlateIcon. - New "More" row (MoreHorizontalIcon) opens a right-side flyout on click or hover holding Video, Recipes and Export; the close is delayed 180ms so the pointer can cross the gap. Its SidebarMenuButton deliberately takes `title` rather than `tooltip`: with `tooltip` the button returns a Tooltip root and DropdownMenuTrigger asChild would hand its ref to a non-DOM node. - Dropped the "Train" section heading; Train is now a top-level row between Images and More. data-tour="navbar" moves to the surviving nav group so the product tour keeps its anchor. - "New" pill beside Images and (inside the flyout) Video, via NavBadge. * Studio sidebar: match flyout rows and New pills to the existing scales - More flyout rows dropped their sidebar-row typography and size-icon override, which fought DropdownMenuItem's own scale (text-sm, gap-2.5, px-3 py-2 and size-4 icons) and rendered oversized glyphs and text next to the nav. - New pill reuses the brand "beta" badge recipe (nav-badge font, --ui-font-scale sizing, nav token colours) rather than hardcoded 9px values. - The More row's native title tooltip (an OS box on hover) is replaced by the app's Tooltip, wrapped around DropdownMenuTrigger so both triggers compose onto the same button, and shown only on the collapsed rail like other nav rows. * Settings: pin and reorder the sidebar navigation Adds a "Sidebar navigation" section to Settings -> Appearance, above the existing profile-menu customizer, with the same drag-to-reorder + switch UI. - New sidebarNav preference: one { id, pinned } entry per navigable row (projects, hub, images, train, video, recipes, export), array order = render order. Defaults match the shipped layout, so an untouched install is unchanged. - Unpinning moves a row into the More flyout rather than hiding it, so no page becomes unreachable. New chat and Search stay fixed as actions. - app-sidebar now renders from one navRows descriptor map, so a pinned row and its flyout counterpart cannot drift; the More row appears only when something is unpinned and highlights off whatever it actually holds. - Mirrored in the backend PersonalizationCustomization: without it the model's extra="ignore" would drop the field, and because sync replaces local state with the server's copy once customization is saved, the user's pin order would reset on the next sync. The validator dedupes and back-fills like sidebarMenu but preserves the client's order, since here order is meaningful. Frontend typecheck, i18n parity and catalog checks pass; 32 personalization tests pass, including a round-trip asserting a reordered list survives a save. * Sidebar customizer: drop the Search row, skip More for a lone item - Search is reached from the top bar, so it is no longer previewed as a fixed sidebar nav row; New chat stays. - More now appears only when it would hold two or more rows. A single unpinned row renders inline in its saved order position instead: a flyout wrapping one item costs a click and earns nothing. The customizer's More preview follows the same threshold. * Sidebar settings: hide a lone unpinned tab, match New chat icon, rename Profile menu - With exactly one tab unpinned, both More and that tab are dropped, so nothing is drawn for it (previously it rendered inline). The page stays reachable by URL. - The customizer's New chat preview uses PencilEdit02Icon, the icon the real row renders; Edit03Icon was a different glyph. - "Sidebar menu" is now "Profile menu", described as the shortcuts behind your name at the bottom of the sidebar, so it no longer reads as a second name for the navigation section above it. * Tighten comments in the new sidebar and delete-guard code * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio sidebar: keep the More row highlighted while its panel is open Moving the pointer into the flyout left the row unhighlighted while the panel stayed open. The row now carries data-menu-open, added to the nav hover selector list. Not data-state: the tooltip and menu triggers both write that attribute, so whichever lands last wins. * Images: use the shared pill toggle for Create/Train and pad the panels - Create/Train was the only segmented control on its own Tabs styling. It now uses PillTabs, the same control as the model picker and Hub toggles, pinned to the header row's 34px. PillTabs takes an icon per tab, so the inline-span workaround for TabsTrigger goes away. - pt-3 on both the Create and Train panels, which sat flush against the model selector row. * Images: make the workflow picker a dropdown instead of a 7-up strip Seven workflows in a 340px rail left ~48px each, so the labels crowded and the hints were only reachable as title tooltips. The strip is now a dropdown: the trigger shows the current workflow and its hint, and each row carries its own description. A row the loaded model can't run is disabled and shows the reason in place of the hint, so the gating explains itself. Adding a workflow no longer shrinks the others. * Images: workflow icons, hint under the trigger, more top room, unclipped Train cards - Each workflow carries an icon, shown on the closed trigger and on every row. - The trigger is one line (icon plus name). The selected workflow's description moved below it, where it reads like the Field hints further down the rail. - pt-6 instead of pt-3 on both Create and Train, so the cards clear the model selector row. - The Train right column scrolls while its cards use ring-1, which draws outside the box and was clipped at the scroll edges. p-px gives the ring room. * Images: one-line workflow rows, stronger trigger fill, roomier mode tabs - Dropdown rows are icon plus name only. The selected row's description already shows under the trigger, and a disabled row keeps its reason as a title. - Trigger fill moves to the bg-foreground/[0.07] dark:bg-foreground/[0.12] pair the hub cards use, so it reads against the card in both modes. - Description under the trigger goes from text-ui-10 to text-ui-11p5. - More horizontal padding on the Create / Train tabs. * Images: drop card borders for the composer shadow, keep scrollbars inside, use app controls in Train - Cards lose ring-1 for .panel-soft-surface: the composer's shadow in light, flat in dark, matching .chat-composer-surface and the menus. - Both rails now clip (overflow-hidden) with the scroller inside, so the scrollbar can't ride over the rounded corner. Same shape video-page already uses. - Train's 9 native selects become the app Select, so they no longer open an OS-native menu, and the native file input is hidden behind a Choose images button that reports the count. - Image previews use explicit 8-10px radii: this theme sets --radius to 1.1rem, so rounded-md was 15.6px and the thumbnails read as circles. * Images: one card for controls and preview, chat sliders, wider softer shadow - Controls and preview were two floating cards; they now share one card split by a divider. The Advanced dock stays separate since it toggles. - SliderField wraps Chat's ParamSlider, so the sliders match Chat (label row with the value, full-width neutral track) instead of a green track with a spin box. All 14 call sites keep their props. - panel-soft-surface goes from 0 2px 8px -2px /0.16 to 0 4px 22px -6px /0.10: lighter, spread wider. * Images: flat Create and Train panes, hover-only scrollbars, tidier Train dataset step Both Images tabs now sit on the page background like the Hub: no card, no shadow, no bounding box. A single rule divides the controls rail from the preview canvas (Create) and from the run area (Train), and the settings and previous-runs sections read as panes rather than nested cards. Also: - Scrollbars in these panes use the existing hover-scrollbar recipe, so the thumb only shows while the pane is hovered. - Workflow rows explain themselves with a tooltip after a short hover, which also works on disabled rows, and the descriptions are much shorter. - Training images rows are name plus image count; the license stays on the example card. - The upload step loses its dashed box, the buttons match the sizes around them, and Upload only appears once files are picked. - The empty preview uses the same icon as the Images nav item. * Images: full-height panes, wider settings rail, Create/Train offset from the selector The rule between the panes now runs the whole page height (the row drops its bottom padding and each pane pads its own content), the settings rail is wider on both Create and Train, and the Create/Train switch sits further right of the model selector. * Images: put both tabs on the Hub's centered measure Top bar and content now share mx-auto max-w-1100 with px-5 / sm:px-8, so Create and Train sit at the same width and position as the Hub instead of running edge to edge. * Images: restore the top bar position, drop the panes lower under it * Images: center the mode switch, flip the arrow with the orientation, app tooltips everywhere The Create/Train switch is centered on the page instead of trailing the model selector, with wider buttons. The flip control's arrows now rotate with the orientation and its label says which way the flip goes. Every native title tooltip on the page is now the app's tooltip, so they all get the rounded surface instead of the OS box. * Images Train: plainer field text, no green buttons, columns that stop colliding - The dataset name, trigger prompt, adapter name and custom base fields now say what they are in plain words instead of leaning on example values. - Import, Upload, Back, Back to settings and Train another are outline buttons, not green ones. - Example thumbnails are landscape tiles, so photos are not cropped to chunky squares. - Settings cells get min-w-0 and the select value truncates, so a long option like the nf4 label no longer widens its column into the next one. - The number stepper sits a little further in from the field edge. - Create and Train are wider. * Images Train: roomier example cards with Import on the thumbnail row * Video: same treatment as the Images tabs - No cards: the rail and the canvas sit on the page background, divided by a rule that runs the full page height, on the Hub's centered measure. - Wider rail, chat's sliders, hover-only scrollbars. - Every native title tooltip is now the app's tooltip, including the clip cards. - Reapply and Cancel are outline buttons, the empty state uses the Video nav icon, and the clip tiles are less rounded. * Images and Video: narrower generation rail, matching Train headings Create and Video rails go from 392px to 368px. Train a LoRA and Training settings are now the same size and both in the heading font: the h2 already picks it up from the base rule, so the settings header opts in with font-heading and the weight that rule pins. * Images Train: shorter copy throughout Family notes, example descriptions, precision labels and every helper line are trimmed so they stop wrapping to three lines and colliding with the next column. The nf4 label now fits its select without truncating. * Images Train: a little more spacing between field groups * Images and Video: tighten code comments * Fix training start NameError, the load-order guard test and CPU-only diffusion tests - start_training forwards resume_source_run_id to _start_training_impl, which reads it. Without it every start raised NameError. - Restore main's anchor in the load-marker order test: the file now has an earlier `if config.is_gguf:`, so indexing the first one compared the wrong branch. - The two diffusion tests that reach diffusers now skip when it is absent, matching the CPU repo-test env. - The UI smoke finds nav rows that live in the sidebar's More flyout. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Treat a null metadata caption as no caption str(None) stored the literal "None" as the caption, so a null row counted as captioned and would have trained on that text. Also drops an unused import. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix invalid-UTF-8 500s, the flat Canny map and the dropped DiT knobs read_text raises UnicodeDecodeError, which is not an OSError, so one bad caption sidecar or video sidecar 500d the info, upload and gallery routes. A flat image now yields the all-black edge map instead of its own luminance, and the four DiT loss knobs the trainer implements are declared so model_dump keeps them. * Use the ui font-size tokens instead of raw px text utilities text-[11px] and friends ignore the UI font size preference, which the repo's font-scale contract test enforces. Same rendered size at the default scale. * Fix diffusion dataset 500s, the dropout-1.0 no-op run and the reset base pick Four correctness fixes on the training side: - The labeling grid read caption sidecars under except OSError, but a non-UTF-8 sidecar raises UnicodeDecodeError (a ValueError), so one bad file 500d /diffusion/dataset/{name}/images and the grid could not be opened to repair it. Read it as no caption, matching the info summary. - An image past Pillow's own hard limit raises DecompressionBombError, which derives straight from Exception and so escaped the upload guard's (OSError, UnidentifiedImageError, ValueError) and returned 500 instead of the intended 400. - lora_dropout accepted 1.0, which makes PEFT build nn.Dropout(p=1.0): lora_A and lora_B receive no gradient and the run saves an untrained adapter while reporting normal progress. Bound it below 1.0, matching the LLM request schema. - The train panel re-seeded the base repo on every dataset refresh because the family object identity changes on each info fetch, so an upload or caption save silently replaced the user's chosen base and the run started on a different model. Track the pick and only re-seed on a real family change. * Show the retained failure when a video page mounts after a failed job Mount-time recovery handled only phase=completed, so reloading the page after a multi-minute generation failed left an idle view with no diagnosis: the backend keeps the terminal failed record only until the next job, and nothing else survives the reload. Surface it the same way the poll does, filtering the cancelled sentinel. * Fix batched generation crashes, cache keying and unreplayable recipes Four bugs in the batched inference path, all found by review: - A mixed-prompt batch sent a scalar negative prompt against a prompt list. Z-Image asserts on the length, and Qwen-Image, Krea 2 and FLUX true-CFG encode a batch-1 negative against batch-N latents and fail in the transformer's text/image concat. Broadcast it to match the batch. - The FBCache step-cache reset sat above the chunk loop. diffusers only resets that state at the end of a successful call, so a forward that raised (the OOM the backoff is meant to recover) left its own residual behind and the halved retry died on a shape mismatch. Reset before every forward instead. - The conditioning cache keyed on the checkpoint alone, but a GGUF or single-file load takes its text encoders from the companion base, so the same checkpoint against a different base reused the previous base's embeddings. Key the base too. - Gallery records stored the base seed and the requested batch size even when a prompts/seeds list drove the run, so restoring the second image of seeds=[5, 99] replayed seed 5. List-driven outputs now record as single-image recipes on their own seed. Also bound strength above 0: every img2img pipeline derives its step count from it, so 0 leaves zero denoising steps and either raises or, on SDXL, crashes on empty latents. * Fix quantized-load LoRA bake, prequant family exclusions and outpaint canvas Six review findings across the Images page and model scanning: - The quantized (int8/fp8) load path can only attach LoRA adapters before quantization, but the frontend load request had no loras field, so every generation after such a load was rejected and each reload repeated it. Send the selection with the load. - build_prequant_checkpoint passed no family to the scheme exclusions while recording the family in metadata, so a Qwen int8 artifact baked the short-M text-stream linears and was then rejected wholesale by the loader's family-keyed check. - Registering a bare single-file checkpoint directory produced no On Device row even though the images loader can load it; only its parent worked. Admit that shape when nothing else matched. - Unload left the Reapply target set, so the repair path was skipped and Reapply reloaded the ejected model. Clear it, as the video page does. - Both FLUX.2 bases were trusted for training but not inference, so Deploy to Create rejected every FLUX.2 adapter. - Outpaint allocated the grown canvas before downscaling, exceeding the browser canvas area cap on a large photo; an over-cap canvas is unusable, so Extend silently posted a fully transparent image and mask. Scale the source first. * Send the picked GGUF filename with the quant so diffusion loads fire The variant expander emitted only the quant label, and nothing else in the frontend set ggufFilename, so the Images and Video pages could never take their GGUF branch: both gate it on meta.ggufVariant and meta.ggufFilename, then fall through to the single-file path, which returns because the id is a repo id and not a .gguf name. Every quant pick was a silent dead click, with no load request reaching the backend. The filename was already on the variant row (the picker keys its list on it, and the variant validator requires a non-empty string), so thread it through the click handler. The chat path is unaffected: it reads ggufVariant and never needed the filename. * Version the conditioning cache key and reject non-finite flow_shift Two correctness fixes: - The cache keyed the checkpoint and its companion base by name only, so a Hub repo advancing to a new commit, or a local directory updated in place, kept returning embeddings from the previous text encoder. Pair both with a revision marker: the locally resolved commit sha for a Hub repo, config plus text-encoder file stats for a directory. Neither loads the encoders, so a warm run still keeps them off the GPU. - flow_shift only checked positivity, but JSON accepts 1e309, which floats to inf, and inf <= 0 is False while NaN fails every comparison. The sigma table then evaluates s * u / (1 + (s - 1) * u) as NaN, which poisons every sampled sigma and saves a corrupted adapter while progress looks normal. Require a finite value. * Keep curated models listed, guard the video companion repo, pin diffusers Three review findings: - The picker filtered every catalog member out of Recommended and Hub search on the way to canonical group rows, but nothing renders those rows yet (catalogGroupFitsDevice and groupMatchesQuery are imported and unused). A task-scoped picker's models list is catalogToModelOptions(), i.e. group members exclusively, so both lists came back empty and no curated model could be discovered or downloaded. Keep the artifacts listed until the grouped UI exists. - The video delete guard compared only repo_id, so deleting the companion base of a loaded GGUF video model was allowed even though it supplies the VAE and text encoders. Compare base_repo too, matching what the images guard already does for its companions. - diffusers was declared unversioned while the diffusion stack requires 0.39 (Krea2Pipeline, the cache_context child registries, the Flux2 and Z-Image pipelines), so an upgrade could keep an older release and selecting an advertised model failed until the user upgraded by hand. * Namespace the trainer conditioning cache per checkpoint, bound the learning rate - The trainer keyed its persistent conditioning cache on family and resolution only, while the keys themselves carry just the caption or image content and crop variant. One cache directory reused for two checkpoints, or for the same repo at a new revision, let a warm run skip loading its encoders and train on the other model's embeddings and latent statistics. Namespace on the base checkpoint and its resolved revision as well. The revision helper now lives beside the cache in diffusion_train_extras and the inference wrapper delegates to it, so the two cannot disagree about what counts as the same source. - The diffusion learning rate only checked positivity, but 1e309 floats to inf and satisfies gt, so the route evicted the resident models and started AdamW with an infinite rate: the first step destroys the adapter while progress looks normal and the result is saved. Bound it below 1.0, matching the LLM schema, which rejects inf for the same reason. * Fix GGUF image model picks doing nothing, and pick the train base in the top bar The quant rows never forwarded the .gguf filename, so every hub GGUF pick on Images/Video fell through to a silent return. On Train the top bar now picks the training base instead of a generation model, which is GGUF-only and untrainable. * Pin diffusion and video loads to the live HF cache root Both read huggingface_hub's import-time HF_HUB_CACHE, which changing the cache folder does not update: progress counted the old root while the download wrote to the new one, and from_pretrained could split one model across both. * Add the diffusion download plan endpoint Reports the repos and exact files a pick needs so the download manager can stage them with the loader's own file scope. A plain snapshot would add the packaged root single, transformer shards and fp16 twins the loader never opens. * Add a file-scoped flavour to the Hub download job Lets a consumer that reads a deliberate subset of a repo stage it through the normal download manager. Keyed as "@scope" so it never collides with a quant or with the repo's full snapshot, and the file list rides the registry so an XET to HTTP retry respawns the same scoped job. * Stage image and video downloads through the Hub download manager They downloaded inline inside the load, so they had none of the manager's disk preflight, manifest verification, resume or panel progress. Picks now stage as scoped jobs carrying the loader's own file list, then load from a warm cache. * Fetch staged GGUF checkpoints as scoped jobs, and stop calling diffusion models unsupported A GGUF entry went out as a full snapshot, whose ignore list drops *.gguf: the job finished at once having fetched only docs, and the repo landed on device unloadable. Every entry is scoped now. The Hub also no longer tags image/video models as unsupported (they run on their own pages), and those pickers name what they select. * Apply the picker task filter to local model sections LM Studio, ./models and custom-folder rows ignored it, so the Images picker listed chat GGUFs that 400 on a diffusion load. The backend already tags every local model with a task for this purpose. * Route a chat pick of a diffusion model to the Images or Video page Chat cannot load one, so it was either hidden or failed on load. The unfiltered picker now lists on-device diffusion models and navigates to the page that runs them, passing the repo and quant so that page loads it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Route the real GGUF filename, keep non-GGUF curated models, key scoped downloads by file set Five review findings, four of them ways a click did nothing or fetched the wrong thing: - A chat pick of a diffusion model routed ggufVariant (a label like Q4_K_M) in the search param the target page uses verbatim as the GGUF filename, so the load asked for a file that does not exist. Route ggufFilename; no filename means a curated non-GGUF pick, loaded as a pipeline. - The task-scoped pickers kept only GGUF repos, so the catalog's bf16, bnb-4bit and single-file fp8 artifacts could not be discovered or downloaded on the Images and Video pages even though loadSpecFor knows how to load them. Keep curated artifacts whatever their format, in Recommended and in Hub search. - Both pages deduplicated routed selections on the model alone, and they now stay mounted, so picking the same repo again -- another quant, or the same one after chat evicted it -- returned early without loading or clearing the query string. Key on model and quant. - Every scoped image download shared one @diffusion job key regardless of the requested files, so switching quant mid-download adopted the running job: the UI waited on the first file set, then loaded a file that was never fetched. Include a digest of the file set in the key. - A scoped plan silently dropped requested files missing from Hub metadata, and snapshot_download succeeds when an allow pattern matches nothing, so the job reported completion and triggered a load with required files absent. Fail the job instead. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the scoped download key derivable, and stop the hidden page hijacking a route Four review findings, the first a regression from my own last commit: - Keying scoped download jobs by a digest of the file set broke the download manager: it builds that key client-side (it polls and cancels before any response tells it a key), so it watched and cancelled a key no worker owned and never fired its ready callback. Keep the derivable "@scope" key and refuse the second request instead when a live job on the slot is fetching a different file set -- decided inside the registry claim, under the lock, so a concurrent claim cannot slip past it. The manager records the file set on the job as well, so a sibling quant's transfer is not adopted locally either. - Both diffusion pages read the route query through a loose useSearch and both stay mounted once visited, so the hidden one consumed the other's ?model=: it navigated back to its own route and tried to load, say, an image checkpoint as a video model. Only the visible page consumes it. - The staged download plan was built without the configured HF token or the Advanced values the load itself sends. The token matters most: the backend's Hub metadata lookup is best-effort, so a gated base silently planned no companion entry and the load pulled those multi-GB files inline, outside the manager. The memory/quant controls decide whether the base transformer/ shards are needed at all, and the route dropped memory_mode, cpu_offload, the prequant path and the LoRA selection before asking for the plan. - The video preview kept playing after leaving the page: the keep-alive layout only hides it, and display:none does not pause a media element, so a clip the user unmuted kept its audio going over the next page. Pause on the active transition and do not auto-replay while hidden. Also completes the hand-built request bodies in the hub download tests: the scoped-files field this branch added to the route read as an AttributeError against them, failing five tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Serialize the GPU handoffs, gate DiT training on a GPU, and keep 3.9 installable Six review findings, three of them evict-then-fail orderings: - The chat load reclaimed the GPU without telling the arbiter it existed. A chat load holds no llama-server process until its GGUF has downloaded, which is minutes, so a competing Images/Video acquire in that window found nothing to cancel, took the GPU, and the chat load then spawned onto the same device. It now registers an in-flight marker through acquire_for's register hook (under the arbiter lock, as the image and video loads do), the evictor cancels a marked load, and the route undoes itself if ownership moved while it loaded. - The Hub-download conflict check ran after that handoff, so a GGUF the download manager already owns destroyed the resident Images/Video pipeline and then 409'd, having loaded nothing. It moves above the handoff, together with the marker it handshakes with. - The image load released the engine router's transition lock before registering the load, so a second load choosing the other engine could unload the still-idle engine this one captured; the load then landed on a deactivated engine, where generate, status, unload and the arbiter's evictor can no longer reach it. Registration now happens under that lock and refuses if the engine changed. - Training a DiT family on a host with no GPU was accepted: nf4 is not a CPU fallback, its 4-bit load goes through bitsandbytes, which requires CUDA, XPU or MPS. The start unloaded the working Images pipeline, pulled the text encoders, and only then died in the child. Rejected before the teardown now, and /info stops advertising a precision that always 400s. SDXL keeps its documented fp32-on-CPU path. - Both diffusion pages kept the routed-pick marker forever, so re-picking the same checkpoint (after chat evicted it) neither loaded nor cleared the query string. The marker is released once the query is gone. The Images key also carried a stray NUL byte, which made the file read as binary to grep and other tooling. - diffusers dropped Python 3.9 in 0.38, so the unconditional >=0.39.0 pin left pip no candidate at all on 3.9 and made every install that composes the huggingface extras unresolvable there. The floor is conditional now. Also fixes tests that were already red on the branch: two hand-built request fakes had gone stale against fields this branch added, and the handoff-ordering test only failed on a host with fewer than two GPUs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the GGUF variant contract test against the merged handler signature The assertion pinned the exact single-line call handleVariantClick(v.quant, v.downloaded, expectedBytes, v.filename), but the handler takes (quant, filename, downloaded, sizeBytes) and prettier wraps the call across lines, so the mandatory repository test job failed on every push. Match the call structurally and assert the filename really is forwarded in the handler's argument order. * Stop a background page and a stale record taking the GPU or a download with them Five fixes from a review pass over the diffusion work. delete-finetuned rmtree'd a model the Images or Video engine was holding: every guard on that route is chat-only, and Images loads any local path, so deleting a local diffusion model under the storage root pulled the weights (and the companion VAE / text encoders sd.cpp re-reads each generation) out from under a live pipeline. The cached-model route already refuses this; the trained/exported one now does too, matching by path rather than repo id, and failing open on a chat-only install so it cannot block ordinary deletes. A staged download finishing while its page was hidden loaded the model and evicted whatever the user was actually using: both diffusion pages stay mounted behind the router and a load takes the GPU unconditionally. The pick is now held until its page is on screen again, which is also what chat does. A scoped download could report success having fetched nothing. With Hugging Face metadata unavailable no manifest is written, so verification is a no-op, and snapshot_download returns an existing snapshot folder without downloading when its own repo_info call fails. A repo already on disk from a full snapshot job (which ignores *.gguf) therefore completed with no weights and auto-loaded against them. The requested file list needs no network, so it is checked against the disk directly. The XET to HTTP retry reclaimed the job slot without the scoped file list, and that claim overwrites the stored record, so a later identical scoped start compared an empty list against the real one and 409'd instead of adopting the running download. The DiT accelerator gate probed torch.mps.is_available(), which only exists from torch 2.5 while the supported floor is 2.4. All three probes shared one try/except, so on torch 2.4 the AttributeError read as 'no block' and a CPU-only host still evicted the resident pipeline, downloaded the encoders and died in the child. Each accelerator is probed on its own now, through torch.backends.mps. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stage what an LTX-2.3 load reads, keep a routed file's load kind, drop an unbakeable LoRA Three from the latest review. The video download plan always asked for the wide base file list, so an LTX-2.3 pick staged the 2.0 base's VAEs, vocoder and connectors that the checkpoint supplies itself, while the companion files the 2.3 assembly does read were left out of the plan and pulled inline at load, outside the panel's progress, cancel and disk preflight. The plan now recognises a 2.3 pick by name (the load keeps the authoritative header probe, and under-guessing only falls back to the load-time pull), narrows the base list, and stages the extras in the same entry as the checkpoint so one repo stays one scoped job. A pick routed from the chat picker arrives as ?model= and ?quant= with no picker metadata, so a bare local .gguf or .safetensors was loaded as a pipeline: an explicit model_kind wins over the backend's filename sniffing, so it evicted the resident model and then failed on the missing model_index.json. Both pages now derive the load kind from the path, the same way their own picker handlers do. A torchao int8/fp8 build takes adapters only at load time. Switching artifact inside one family keeps the LoRA selection, since the family did not change, but the load did not bake it, so the next generation was rejected with 'reload the model with the adapter selection' while the picker still showed the adapter as active. The selection is now dropped once per resident build, with a message saying to pick and load again. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cancel an evicted safetensors load, spare the arbiter for CPU-only chat, fetch clips lazily Four fixes from the latest review round: - The GPU arbiter's chat evictor only cancelled the llama.cpp side. The orchestrator publishes active_model_name once its worker reports success, so an in-flight safetensors load was visible only as an entry in loading_models and finished onto the GPU after ownership had transferred. Cancel every pending load, and give the safetensors branch the post-load ownership recheck the GGUF branch already had. - A manual gpu_layers=0 GGUF load runs on the CPU with the GPUs hidden from the child, yet it took the arbiter unconditionally: it cancelled a running image or video generation for a model needing no VRAM, then held CHAT ownership so the next GPU workload unloaded it for nothing. Gate the acquire on the same predicate the launch-time CPU-only mask uses, as the image and video loaders gate on their resolved device. - The staged-download hook subscribes per repo, not per job, so another job on the same repo advanced the staged queue (starting a load whose scoped files were still downloading) or wiped a queue that was still running. Compare the variant each callback carries, like the chat page's auto-load does. - The video gallery fetched every record of a page into an object URL that lives until the page closes: 50 clips at tens to hundreds of MB each, for cards the user may never scroll to. Fetch a clip as its card nears the strip's edge, plus the selected one the player needs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim the comments across the diffusion backend Comment-only pass over the Python this PR touches: drop what the code already says, collapse multi-line explanations that still read on one line, and keep the reasoning that is not recoverable from the code. No code, docstring semantics or behaviour changes; verified with an AST comparison against the previous revision, and the backend suite is unchanged (same 37 environment failures as before: the API integration tests that need a live keyed server, the flash-attn install hooks, and the GPU memory fields). * Invalidate latents on a VAE swap, keep a cut-off generation, surface the EMA adapter - source_revision() scanned the checkpoint root plus text_encoder/tokenizer but not vae, so swapping or fine-tuning the VAE in place left the conditioning cache namespace unchanged and a warm run trained against latents from the old checkpoint. Include the vae directory, like any other component the cached tensors come from. - /images/generate answers only when the images are saved, and secure mode's tunnel caps an origin response near 100 seconds, which a native CPU or a high-step run passes routinely. The page reported failure while the work kept running, and a retry would duplicate it. A lost response (fetch rejection or a gateway status the origin never answered) is now told apart from a refusal: the page waits out generate-progress and reloads the gallery, so the run it started still lands. - The trainer emits the EMA adapter's path with the terminal event, but the state update dropped it, so neither the run history nor either response schema carried it and an enabled EMA left nothing discoverable. Keep it, and show it next to the primary adapter. - weighting_scheme advertised a choice of timestep sampling; sampling is always logit-normal and the flag only selects the bell loss weights. Describe what it does. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Hide unloadable cached rows, hold the dataset interlock, bound a GIF export - The cached-model listing tagged any repo with a model_index.json as text-to-image, so a community pipeline the image loader's trust rule refuses still got a row in the Images picker, and a detected-but-untrusted video repo fell through to that same tag. Gate the image tag on the load path's rule and hide an untrusted video repo outright. - A routed diffusion pick only carries a GGUF filename, which is all the chat picker has, so a curated single-file artifact arrived with no quant and was loaded as a pipeline: from_pretrained on a repo with no model_index.json. Pass the page's own catalog spec into the route pick, so a routed pick resolves to exactly what a direct pick on that page resolves to. - The dataset mutation endpoints checked is_active() and only then handed their filesystem work to a thread, so a start reserving in that gap changed captions or removed images underneath the preflight or the running trainer. The interlock is now registered for the whole request under the lock reserve() uses, and a start refuses while a mutation is open rather than waiting on it. - GIF export held every kept frame as a paletted image before encoding; a clip may be 2048x2048 for 1024 frames, and at the 12 fps target the step is 1, so one export click could allocate over 4 GB and take the backend down. Downscale past 720 px and widen the step to keep at most 300 frames. - seed accepted any Python int, so an out-of-range one passed every preflight, evicted the resident models, spawned the trainer and only then died in torch.manual_seed. Bound it to torch's 64-bit range in the request and config. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the accelerator probes in the DiT family-metadata tests Six tests read family_train_infos() (or a start preflight) without pinning the host probes, so they only held on a machine with a bf16 accelerator: on a GPU-less runner the DiT gate empties precision_modes, turns supports_compile off, and replaces any other preflight message with the no-accelerator note, and all six failed there. A conftest fixture pins both probes for exactly those tests, so they assert the family metadata they are about on every host. The gate's own CPU-only behaviour keeps its dedicated tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Carry the pipeline task into the hub inventory the pickers read - The task-scoped pickers filter On Device rows on a task, and the chat picker routes a diffusion pick by the same field, but those rows come from the /api/hub inventory, which never carried one: the Images and Video pickers listed nothing on device and the routing never fired. Both cached scans and the local listing now tag rows with the classifiers the models API already uses, the schemas and the frontend adapter carry it through, and a row the backend classified as a generation task is exempt from the chat-only guard that was also dropping it. - The local routing map was keyed by model_id while the row click passes id (a filesystem load id for a models_dir or LM Studio entry), so the lookup missed and the pick fell through to the chat loader. Key both. - A staged download whose start answered "error" left its head in place, where the effect never re-runs and onReady never fires, so the pick was stranded until the user reselected. Clear the queue and say so. - Every scoped pick in a repo shares the @diffusion variant, so the variant alone cannot tell two file sets apart: restaging while the first job finished let its completion pass for the new pick and load a checkpoint that had not downloaded. Bind the callbacks to the repo + file set they started, and to the staging generation. - A rejected generate POST does not say whether it reached the backend, so an immediately idle progress read was ambiguous and a submission that never landed looked like a finished image. Require evidence: progress seen active, or a gallery record that was not there before the POST. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the OpenAI image URL fetchable, keep WebM audio, stream example imports Four review items on the diffusion Studio work: - response_format=url returned the bearer-gated gallery route, which a standard image client downloads with no Authorization header, so the default response format was unusable. Mint a short-lived HMAC link instead (the shape RAG already uses for pdf.js) served by a signed route, and leave the gallery route itself bearer-only. - A manual gpu_layers=0 load carrying speculative_type="off" -- a value the UI persists and sends -- read as GPU-bearing, so it took the GPU arbiter and evicted a resident image/video pipeline even though the launcher hides the GPUs for it. Canonicalize the mode and exempt "off". - The curated example import prepared the whole split before the loop stopped at the 10-100 image cap; m1guelpf/nouns is 49,859 rows / 328 MB. Stream instead, with the prepared load kept as a fallback for a repo that cannot stream. - WebM export dropped the audio track an LTX-2 clip carries, silently, on the format offered for web embeds. Mux it as Opus through a resampler + FIFO, and keep exporting the video alone on a build without libopus. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not stub out triton on a GPU host when the Xet backend fails to import The lazy loader retries `import unsloth_zoo.hf_xet_fallback` under UNSLOTH_ZOO_DISABLE_GPU_INIT=1 whenever the first attempt raises. That flag makes unsloth_zoo take its MLX/CPU path, which injects triton and bitsandbytes STUBS into sys.modules for the rest of the process. On a working GPU box whose first import failed for an unrelated reason (a bitsandbytes/CUDA mismatch, say) the retry succeeds, so Studio boots looking healthy and then dies at the first CUDA-only kernel: a GGUF or compiled diffusion generation hits the stub and returns NotImplementedError: Unsloth: 'triton.tools.experimental_descriptor.enable_in_pytorch' was called on Apple Silicon / MLX, where triton is stubbed out. so every image generation 500s with an Apple-Silicon message on a Linux CUDA host, while the load reports success. Found by loading Z-Image-Turbo GGUF through the API on a box where bitsandbytes could not initialise. Gate the retry on the host genuinely having no accelerator. The Xet stall watchdog is optional and already degrades with a warning; a process whose triton is stubbed out is not recoverable. The warning now says why it did not retry. * Fix the lost-generation proof set, the settle timeout and the hub inventory's diffusion gates Seven fixes from the latest review round on the Images page and the hub cache inventory. Images page: - The lost-POST settle path built its "already seen" gallery id set inside the catch, after the request failed. By then the earlier runs of the same batch had already prepended their records, so run 2 could accept run 1's image as proof that its own request reached the backend. The set is now captured once before the first POST and grows with every record the batch produces. - settleLostGeneration fell out of its SETTLE_MAX_MS loop and returned normally, so a wedged generation was counted as done and the next run started against a busy backend. It now throws on timeout. - Restoring a recipe cleared the ControlNet selection but left the workflow tab and the init / mask / reference images pointing at whatever was loaded, so the next Generate conditioned on an unrelated image. It now clears all of them and returns to Create. - The download plan omitted the adapter selection the load itself bakes in. A baked LoRA forces the dense build path, so the plan described a different file set than the load that followed and the rest was pulled inline, outside the download manager. Both now derive the list from one helper. Hub cache inventory: - A download for a repo an Images or Video load is staging was allowed to start: only the llama.cpp loader was consulted. Both diffusion backends already expose loading_repo_ids for the delete guard, and the download guard now reads them too. - A companion-only prefetch (pipeline manifest plus VAE and text encoder, no transformer) passed the snapshot-partial check, since every file its manifest expected did arrive, and was advertised as on-device although from_pretrained cannot load it. - The single-file flag never reached the picker through the hub inventory path, so a checkpoint-only diffusion repo read as a full pipeline and failed after the handoff. The two pipeline-shape helpers now live in hub/utils/inventory_scan.py so /api/models/cached and the hub inventory classify the same repos the same way. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop adopting an unknown scoped download, leaking raced blobs and resurrecting deleted clips Three items from the latest review round. A scoped download job carries a deliberate file subset, and every file set of one repo rides the same "@scope" slot. A client that adopts a live job from the backend had no file list to compare against: the active-downloads response never carried one, so an adopted job's set was unknown and any later scoped request for the same repo read as "already started". Selecting a different checkpoint then waited on the wrong transfer and tried to load a file nobody fetched. The response now publishes the scoped file list, adoption records it, and an unknown set no longer satisfies a scoped request. A gallery record can be deleted while its blob is still downloading. The delete revokes the URL present at that moment, so the fetch that lands afterwards inserted a fresh object URL for a record no card renders and nothing can revoke: a full MP4, tens to hundreds of MB, pinned for the rest of the session, and once per raced fetch. Both galleries now discard a blob whose record went away, with an epoch covering the video page's Clear all. The video backend keeps the last completed job until the next one starts, and the Video page merges that record on mount to cover a job that finished after the gallery fetch. Deleting the clip left the record in place, so every reload prepended a ghost card whose file request 404s until another generation replaced it. Deleting the clip, or clearing the gallery, now clears the matching terminal record, and the page skips a record it deleted itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Resolve revisions from the live cache, serialize dataset imports, and stop pinning every gallery blob Five more items from the review round. The conditioning-cache revision marker read huggingface_hub's import-time HF_HUB_CACHE constant. Studio can move its cache during a session and loading follows the live setting, so after a move the marker went unresolved (or pointed into the previous root) and pulling a new revision of the same checkpoint no longer invalidated the cache: a warm run could reuse the old encoder's embeddings and the old VAE's latents. It now looks in the active Studio cache first and keeps the environment and the library constant as fallbacks, which the trainer subprocess still needs. The dataset interlock counts mutations rather than excluding them, so two imports of different examples into the same empty name both got past the emptiness check. The winner promoted its staging directory atomically; the loser found the folder non-empty, fell back to a per-file move, and merged its images and captions into the winner's dataset. Imports now take a per-folder lock, a second one is refused with 409, and the emptiness check is repeated under the lock. On Windows the sd.cpp asset resolver filtered only by accelerator token, so a Windows arm64 host matched an x64 zip, downloaded and installed it, and failed later when the binary would not run. It now filters by architecture the way the Darwin and Linux branches do. Every gallery page fetched every PNG up front and kept the object URL for the session, so scrolling a large gallery grew memory without bound for tiles the user may never look at. The Images strip now fetches a tile as it nears view, like the Video strip, and keeps the eager path only where IntersectionObserver is unavailable. A 503 carrying a JSON body comes from the application, not a proxy, so it is surfaced as the error it is instead of entering lost-response settlement and being reported as a request that never reached the server. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the class of a failed generation instead of a bare "Image generation failed." Found on a macOS runner: the native renderer aborts inside its own text encoder there, and the page showed only "Image generation failed." with the sd-server backtrace left in the server log, so nothing about the failure reached the user. The failure is now classified into fixed text, out of memory and native-process death, so the message says what happened and what to try. None of the engine's own output is echoed, since a native tail carries local paths and argv; that stays in the log, and an unrecognised failure keeps the original literal. * Treat an undecodable caption sidecar as the tombstone the trainer sees Uploads store .txt and .caption sidecars as raw bytes, so one can hold invalid UTF-8. The trainer treats any existing sidecar, decodable or not, as an empty tombstone and never falls back to the metadata row for that image. The labeling grid and the dataset summary read an undecodable sidecar as absent instead, so both showed a metadata caption that the run would silently replace with the instance prompt, and counted the image as captioned. Both now track sidecar presence separately, so what the user reviews is what the run trains on. * Keep the reason a native server died, not just its backtrace A ggml abort prints its cause first and then a stack trace, so reporting the last twenty captured lines gave twenty addresses and nothing about the failure: on the macOS runner the native server died on an unimplemented Metal op and the message carried only frame pointers. The captured tail now leads with the lines that name a cause and keeps recent context after them, for both the startup failure and the mid-request death. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop staging the dense text encoder for an fp8 video load Two halves of the same gap, found while measuring the LTX-2.3 download plan: - The video download plan and the scoped pre-download never saw text_encoder_quant. An fp8 request loads a hosted pre-cast encoder, so asking for one still staged and downloaded the base repo's dense Gemma3 (48.79 GB of Lightricks/LTX-2 on the 2.3 distilled pick) that the pipeline then never opened. The plan now drops those shards and stages the pre-cast checkpoint instead; their configs stay, since the pre-cast loader still meta-inits the encoder from the base repo's component config. - The LTX-2.3 assembly builds every component itself, so pipe_kwargs (which carries the pre-cast encoder for from_pretrained) never reached it and an fp8 request silently loaded the dense encoder anyway. It is passed across explicitly now. The dense skip is earned, not assumed: only a pre-cast checkpoint that resolves on the Hub lets the plan drop the dense shards, and only one already fetched to disk lets the pull drop them, so an unpublished or gated artifact leaves both exactly as they were. If injection still fails after that, the load tops the dense weights back up rather than handing from_pretrained a snapshot with no encoder in it. Measured against the real Hub on the 2.3 distilled Q4_K_M pick: 67.24 GB before, 18.92 GB with a 0.43 GB stand-in for the pre-cast artifact (the base entry drops from 24 files / 48.79 GB to 13 files / 0.04 GB). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Match the file's typing idiom for flow_shift models/training.py annotates with typing constructs throughout (105 Optional[...], no Union), and flow_shift was the one place using PEP 604. Union[] reads like the rest of the file, and it also drops the runtime evaluation that would raise on Python 3.9. * Bound the gallery blob cache, and three interlock fixes Four review findings, all reproduced first: - The gallery object-URL caches were unbounded. A clip runs from a few MB to a few hundred, both pages stay mounted after their first visit, and entries were only dropped on delete, so scrolling pinned everything for the session. Both pages now share a byte-budgeted LRU (512 MB video / 192 MB images) keyed off the visibility signal the near-viewport fetching already provides. On-screen media, the selected clip or image, and the item just fetched are never evicted, so eviction is invisible and a single item larger than the whole budget cannot evict itself into a refetch loop. - The image, video and chat load guards ran two independent training probes but returned early when the FIRST one raised, so an unreadable LLM backend disabled the diffusion interlock and a load could proceed straight into an active diffusion trainer on the same GPU. The probes are independent now. - An engine switch swallowed a failed teardown and published the new engine anyway, which is exactly the leak the unload exists to prevent: the arbiter's evictor, /images/unload and the next load all resolve through get_active_diffusion_engine(), so the still-resident pipeline (or a live sd-server) became unreachable and the next load allocated on top of it. The switch now fails and leaves the old engine published, so it stays reclaimable. - The native generation timeout was 30 minutes while the Images page waits up to 6 hours (SETTLE_MAX_MS), so slow-but-progressing CPU jobs died deterministically at the deadline. Measured on GPU-less runners, a 512x512 4-step Q2_K generation took 900 s on Linux and 1465 s on Windows, so larger images or step counts clear half an hour easily. The ceiling now matches the page's window and applies to the whole request: chunks of a split batch share one deadline instead of each getting a full budget. Cancellation is unchanged. Declined: gating the huggingfacenotorch extra off Python 3.9 over the conditional diffusers marker. The marker is deliberate and its comment says why: diffusers dropped 3.9 in 0.38, so pinning >=0.39 outright leaves pip no candidate and the whole extra unresolvable there. The pipelines it names live in studio/backend, which cannot install on 3.9 anyway (studio.txt pins matplotlib==3.10.9 and fastmcp>=3.0.2, both requires_python >=3.10), and the extra is the general core one, so the alternative drops 3.9 for library users who never touch Studio. * Close the load-versus-training-start race, and two picker fixes - The image and video load guards read is_active() and only then selected an engine, acquired the arbiter and registered the load. A /train/diffusion/start reserving inside that window freed residents the load had not registered yet, so the trainer came up beside a brand-new pipeline. The service already had exactly the right pattern for this in dataset_mutation, so gpu_load_admission mirrors it: reserve() refuses while an admission is open, an admission refuses once a start is reserved, both decided under the one lock. The span is only the registration, since begin_load returns as soon as the load is registered and _free_gpu_for_diffusion_training preempts an in-flight load from that point. Chat is deliberately not covered: its load spans an eviction plus a multi-minute GGUF load, and it admits models that fit beside training by design, which is a different contract from the diffusion pipeline's all-or-nothing one. - Hugging Face gives the LTX-2 family the image-to-video pipeline_tag (both Lightricks/LTX-2 and unsloth/LTX-2.3-GGUF report it), so a text-to-video-only filter dropped the flagship audio family out of Video Hub search while the rest of the app routed it to Video. - Task-scoped quant fit sized picks against the LARGEST visible device while resolve_diffusion_device_target returns a bare "cuda" and torch places on the current one. On a heterogeneous host that recommended a checkpoint sized for the bigger card and then loaded it onto the smaller one. Fit now uses the device the load actually lands on; identical on a homogeneous host. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Expose the persistent conditioning cache in the start schema DiffusionLoraConfig has carried cond_cache_dir for a while and the DiT trainer acts on it, but DiffusionTrainingStartRequest omitted the field, so Pydantic dropped it silently and every API-driven run fell back to the in-memory cache that is rebuilt from scratch each time. The warm path skips loading the VAE and the multi-GB text encoders on a rerun whose images, captions and resolution are unchanged, so this was a real capability that could not be reached. Contained like output_dir rather than left to the trainer subprocess's cwd, since it is another directory the trainer writes to. Blank or omitted still means the in-memory cache, so it must not resolve to the outputs root. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix diffusion policy and classification issues from review fp8 auto precision defaulted to precise accumulate on any non-consumer GPU, which made fp8 2.05x slower than int8 on RTX 6000 Ada and slower than not quantising at all. NVIDIA's professional whitepapers do publish equal FP8 rates for both accumulate modes there, so the hardware premise held, but the cost is in the cuBLAS path rather than the published rate. Default to fast accumulate: measured on B200 the flag is a no-op (4096^3 _scaled_mm at 3023.8 vs 3041.8 TFLOP/s, bitwise-identical output, 1.213 s vs 1.230 s end to end), so it is a large win where it bites and free where it does not. Precise accumulate stays available via transformer_quant_fast_accum. Z-Image's DiT is a Lumina2 derivative, so unsloth/Z-Image-GGUF and unsloth/Z-Image-Turbo-GGUF both declare general.architecture = "lumina2" and the whole line was tagged image-diffusion-unsupported and hidden from the Images "On Device" list, though validate_load_request loads them. Resolve shared archs from the repo/file name like bare "wan" already does, with a test asserting the picker and the loader agree for every family. The sage attention on-demand install ran an unpinned `pip install sageattention`, but PyPI's newest wheel is 1.0.6 and diffusers refuses anything below 2.1.1: the install always "succeeded", wrote an unusable version into the running venv, and was rejected on the next line. Carry the dispatcher's floor so pip resolves nothing instead. The dense-quant disk gate sized the download from the bf16-RESIDENT table. The fp32 families download twice that (Z-Image: 23,479 MiB against a 21,970 MiB gate), leaving a window where the check passed and the download filled the disk; Ideogram 4 ships fp8 and was overcharged the other way. Size the gate by published bytes, verified against HF sibling metadata for all 12 families. Patch installs went through unsloth_zoo, which refuses to import unless UNSLOTH_IS_PRESENT is set, and that is set by unsloth itself. The server imports unsloth at boot so it never showed there, but any other process ran silently unpatched with every install returning False, which is 13 test failures on a clean environment. Import unsloth and retry once, memoised per process. Also: the GGUF+LoRA refusal pointed at the native engine without saying a GPU host only selects it under UNSLOTH_DIFFUSION_ENGINE=sd_cpp, so the suggestion was unreachable; the gallery recipe recorded loras from the generate request alone, losing a load-time bake; load-progress claimed "40.07 GB downloaded" for a fully cached load; and pickers.tsx imported three catalog-group helpers it never used. Reported by oobabooga. * Keep the sd.cpp text encoder on CPU under Metal macos-14 loads FLUX.2-klein-4B Q2_K natively on mps and then dies on the first generation with exit code -6: ggml_metal_op_encode_impl: error: unsupported op 'RMS_NORM' -> ggml_abort LLMEmbedder::encode_prompt -> LLMRunner::compute -> GGMLRunner::compute ggml's Metal backend gates RMS_NORM on contiguous rows and aborts the process when that does not hold, with no per-op CPU fallback, so any LLM text encoder (Qwen3 for FLUX.2 and Z-Image, T5 for FLUX.1) takes sd-server down. The encoder runs once per prompt while the DiT runs every step, so pinning only the encoder keeps Metal for the part that matters. UNSLOTH_DIFFUSION_SD_CPP_METAL_TE_GPU=1 opts back in once ggml grows the kernel. * Gate the unsloth retry in the diffusion patch backend The retry added for the clean-environment patch failures is not free: importing unsloth pulls torch in behind it, which costs ~940 MB of RSS measured in a process that had neither, and on a host with no accelerator it fails anyway. A cross-platform CI job that had generated fine at ~900 s later died 19 s in with SIGTERM and every 'if: always()' step skipped, which is the runner being torn down rather than a step failing. Retry only when torch is already imported (true of the server and of anything patching a real module, and the condition that stops the retry from being what loads torch), unsloth is installed but not yet imported, and the first failure was the ImportError the sentinel guard raises. The clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only retry the unsloth import where it can succeed The gate still let the retry run on hosts unsloth does not support, which is where it is most harmful: a 7 GB macOS runner lost the Studio server 26 s into a load, and the Linux runner was torn down mid-generation. Neither MPS nor plain CPU can complete the import, so the retry there pays the cost and fails anyway. Require an accelerator unsloth actually supports (CUDA/ROCm via torch.cuda, or XPU), with UNSLOTH_ALLOW_CPU as the documented override, and hoist the predicate to module level so it is tested directly rather than through the import system. On a CPU-only host the retry no longer fires at all; on CUDA the clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Guard old diffusers, stream video exports, record conditioned recipes Three fixes from review. The 0.39-only pipeline classes (Flux2Klein, Z-Image, Krea 2, LTX-2, HunyuanImage) were resolved by getattr deep in the load, so on the older diffusers that packaging still allows on Python 3.9 -- diffusers dropped 3.9 in 0.38 and this project still supports it, so the 0.39 floor has to be conditional or the extra becomes unresolvable -- an advertised model failed with a bare AttributeError after its checkpoint had already been downloaded. Krea 2 already guarded itself this way; assert_pipeline_class_available now runs the same check for every image and video family from validation, before any fetch, and names the version and the fix. WebM export accumulated the whole VP9 output in a BytesIO and returned it as one bytes object that the response held again. The request caps allow 2048x2048 for 1024 frames, so an export runs to hundreds of MB and concurrent clicks could exhaust the process, while the MP4 route beside it already streamed from disk. transcode_to_file encodes to a temp file and the route returns a FileResponse with a background unlink, so nothing large is resident. A conditioned generation's recipe carried only the txt2img fields, so the gallery presented an inpaint or upscale result as a complete Create recipe and restoring it replayed an unrelated text-to-image request. The images themselves are still not persisted (user uploads with their own lifetime), but the workflow and its scalars are, restore reapplies them, and the toast now names the inputs that have to be supplied again instead of silently landing on Create. Reported by Codex. * Per-load video cancel event, family-gated image picker, cond cache refusal A cancelled video load could resume: begin_load cleared the shared cancel event, and unload() drops _loading without waiting for the worker, so the next load cleared the very object the cancelled worker was watching and its multi-gigabyte pull ran on alongside the replacement until the token check at the end. Each load now gets its own threading.Event, passed down through _fetch_te_prequant and _predownload_base, so a cancelled worker stays cancelled. A cached repo with a model_index.json was advertised as text-to-image on the trust rule alone, but validate_load_request also requires a detected image family, so a trusted pipeline of an unsupported class produced a picker row that deterministically 400s. The picker now applies both gates, mirroring the video branch. cond_cache_dir was accepted for sdxl and then ignored: only the DiT trainer reads it, while the SDXL trainer builds a per-run in-memory latent cache, so the promised cross-run reuse never happened. The route now refuses it with a 400 that names the families which do support it, checked against the resolved family so an omitted model_family with an SDXL base is caught too. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the frontend build broken by the gallery blob cache tsc -b failed on the branch head, so npm run build produced no dist and every platform job fell back to --api-only: blob-url-cache.ts(29,15): TS1294: This syntax is not allowed when 'erasableSyntaxOnly' is enabled dataset-labeling-grid.tsx / dataset-showcase.tsx: Argument of type '{ url: string; bytes: number; }' is not assignable to parameter of type 'string' The cache took its budget as a constructor parameter property, which the project's tsconfig forbids, and fetchGalleryObjectUrl now returns the blob size alongside the URL for that budget, which the two dataset thumbnail components still consumed as a bare string. Declare the field explicitly and destructure the URL at both call sites. tsc -b is clean and vite build emits dist again. * Recover from a ggml unsupported-op abort by restarting on the CPU backend ggml checks every node against the device's supports_op and calls GGML_ABORT when one is not implemented, because a single-backend graph has nowhere else to put it: there is no per-op CPU fallback. The whole sd-server dies with SIGABRT mid-generation and the user gets "the native image renderer stopped unexpectedly" with no way forward. Seen on macos-14 arm64 with FLUX.2-klein-4B Q2_K through the cross-platform CI: the text encoder is already pinned to CPU, and the abort moved into the denoise loop instead. ggml_metal_op_encode_impl: error: unsupported op 'MUL_MAT' -> ggml_abort StableDiffusionGGML::sample -> sample_k_diffusion A retry on the same backend would abort identically, so the load is restarted once with --backend cpu (the only flag that changes which backend executes the graph; --offload-to-cpu moves parameters, not compute) and the generation is re-submitted. The same checkpoint then renders slower rather than not at all. Strictly bounded: the signature must carry both the unsupported-op line and ggml_abort, the device must not already be CPU, and it happens once per load, so an OOM kill or a genuine crash still surfaces as itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix two tests that only fail in a full-suite run The 3.10 CI leg resolves PyAV 17, where av.container.OutputContainer is an immutable C type, so the no-libopus export test died on "cannot set 'add_stream' attribute of immutable type" before it asserted anything. Inject the refusal by wrapping the container av.open() returns instead; modules stay patchable on every build. Removing the injection makes the test fail again, so it still covers the branch it is named for. The Xet shim's degraded-path tests drop utils.hf_xet_fallback from sys.modules and import a throwaway copy. Restoring only the sys.modules entry left the utils package attribute bound to the throwaway, and the two disagreed for the rest of the process: a later monkeypatch of the dotted target patched one copy while the code under test imported the other, so the patch did nothing and test_fetch_te_prequant_only_reports_what_it_downloaded reached the real Hub and got a 401. Restore both bindings. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop an ignored-cancel sd-server, guard deletes during diffusion training, repair unusable managed binaries sd-server does not interrupt an in-flight job, so when it ignores a cancel the grace branch abandoned the poll and reported cancellation while the native job kept a core (or the GPU) busy to completion and held the server's job slot. The comment said the caller stops the server, but only unload does that immediately: a superseding load stops it after its multi-gigabyte download, and a load that then fails never gets there. Stop it here, as the deadline branch already does. DELETE /api/models/delete-finetuned checked only the LLM trainer, so it could rmtree the output directory a live diffusion LoRA run was about to write its adapter into. Consult the diffusion training service too, like the dataset mutation and model-load routes. find_sd_*_binary only checks is_file(), so an interrupted extraction (or a prebuilt for the wrong CPU) left a present-but-unrunnable binary the installer never retried: every load probed it, fell back to diffusers, and native inference stayed off until the directory was deleted by hand. Probe it and reinstall, but only for a copy under the installer-owned root -- SD_CLI_PATH, UNSLOTH_SD_CPP_PATH, an in-tree build and anything on PATH are the user's. * Plan the pre-cast text encoder, and make the cross-trainer GPU admission atomic An fp8 text-encoder request loads a hosted PRE-CAST checkpoint, but the image download plan never received text_encoder_quant, so the manager staged the base repo's dense encoder (FLUX.2-dev's Mistral-24B is ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) and the load then pulled the pre-cast file inline, outside the manager's progress and disk preflight. The plan now takes the field, resolves the hosted artifact with the same resolver the injection uses, stages that file, and drops only those components' dense weight shards. The load's own prefetch takes the same treatment, since it paid the same cost. Only a checkpoint that really resolves on the Hub earns the drop, so a gated or renamed artifact still stages the dense encoder the load will fall back to. The two trainers admitted each other with independent check-then-act guards: the diffusion route checks the LLM backend several network-bound preflights before it reserves, and the LLM route checks the diffusion service well before it spawns, so two near-simultaneous starts could both pass and train on one GPU. reserve() now re-tests the LLM backend under its own lock, and the LLM route holds the diffusion service's gpu_load_admission across its spawn, so exactly one of the two wins. Both halves fail open, so a chat-only install still trains. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the Advanced options a staged download planned against Staging does not set busy, so while a multi-gigabyte download runs the user can still change precision, memory mode, speed or the baked LoRA selection. The pending record held only the repo and artifact, and the completed download fired a load that read the CURRENT state: the staged file set could then be missing files that load needs (fetched inline, with no progress and no disk preflight) or hold gigabytes it no longer uses. One snapshot of every Advanced control is now taken when the plan is built, and it travels with the pending record into the load, so the load that runs is the one the download was planned for. * Do not advertise a family the installed diffusers cannot build The newer families (Z-Image, Krea 2, FLUX.2, LTX-2, HunyuanImage) exist only from diffusers 0.39, and 0.39 cannot be installed on Python 3.9 at all -- diffusers dropped 3.9 in 0.38, so the requirement is conditional or the whole extra becomes unresolvable. On such an environment the picker still offered those rows, every pick failed deterministically, and the error's advice to run pip install -U diffusers could not fix it without also upgrading Python. The cached-repo picker now applies the same availability check validate_load_request does, which is keyed on the pipeline class actually present rather than on the Python version, so it is also right for an intentionally pinned older diffusers on 3.10+. Fails open when diffusers cannot be imported at all: that is a different problem and the load path reports it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore the diffusion engine selection after each router test The active engine is module state, and several tests set it by plain assignment because what _activate does to it is the thing under test, so monkeypatch could not undo it. A leaked ENGINE_SD_CPP left get_active_diffusion_engine() handing back the sd.cpp backend for the rest of the process, and every later route that reads the active engine then saw an unloaded model: eight tests in test_openai_images_generations_route.py returned 503 in a full-suite run while passing on their own. The autouse fixture now snapshots and restores it. * Stream gallery clips, and close three races around them Four fixes from the latest review pass. The video gallery downloaded each clip into a blob before it could play, so playback waited on the whole file (tens to hundreds of MB), seeking was limited to what had arrived, and every viewed clip stayed pinned in the webview. The file route already streams and serves ranges; it just could not be a <video src> because it is bearer-gated. Mint a short-lived signed link instead (its own HMAC secret, 12 hour TTL, separate from the image links) and hand it to the element, which then fetches only the ranges it plays. That removes the blob budget, its LRU and every revoke on this page. The sd.cpp readiness probe accepted any process answering on the port, so a foreign server that grabbed the port between the bind check and the spawn was adopted as ours. Confirm the listener is our child before reporting ready, and stay best-effort (psutil missing, an unknown owner, or any probe error still passes) so the check can only reject a definitely foreign process. Dataset import held its lock for the extract but not for the upload path, so two concurrent uploads into the same folder interleaved; take the same lock and return 409. And reject Windows device names (CON, NUL, COM1..9, LPT1..9, with or without an extension) plus trailing periods in dataset names, which are unopenable on Windows. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Build the image download plan for the engine that will load /images/download-plan always asked the diffusers backend, while /images/load picks the engine per host: a GGUF pick on a machine with no usable GPU routes to native sd.cpp, which reads a single-file VAE plus text encoders and never opens the base repo's sharded components. Measured on unsloth/FLUX.2-klein-4B-GGUF (Q2_K): the plan staged 7.66 GB of FLUX.2-klein-4B components the native load discards, and the 7.80 GB sd-cli actually needs was then fetched inline by the loader, outside the download manager's progress and its disk preflight. Z-Image-Turbo is the same shape. The plan now asks whichever engine the load will select. predict_engine() applies the selection policy without any side effect: it activates nothing (staging a download must not unload the resident model) and only locates the binary rather than installing it, but still counts an installable binary as available, since that is what the load does on a fresh host. The native backend gains a download_plan built from the same _asset_specs the loader fetches, returning the same envelope, so the manager stages exactly the files sd-cli opens. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not let a queued generation outlive the model, and three scan fixes Five items from the latest review; four were real. An unload or arbiter eviction only cancels the generation holding _generate_lock. A second request queued behind it holds no cancel event yet, and Python locks are not FIFO, so it could take the lock the instant the active denoise released it, still see a loaded pipeline, and run a whole new denoise after the model was told to go away: the eviction then waits minutes for it and an image lands after the eject. Unload and a superseding load now raise a fence under _lock before they queue, and a generation that wins the lock while one is pending refuses instead. The cached-model scan judged pipeline completeness across every revision, so a repo holding an older complete snapshot plus a newer companion-only one read as complete while the snapshot from_pretrained actually opens has no transformer. Both scans now look at the revision the loader will open. Deleting a dataset image deleted its caption sidecar unconditionally, which for cat.jpg alongside cat.png removed the caption the survivor still resolves to. The sidecar now goes only with the last image of that stem, matching what the thumbnail cleanup beside it already did. Importing an example into a folder that holds no images but does hold files fell back to promoting the staging dir one file at a time, so an interruption left a partial dataset that the image_count check accepts as complete on retry. Those files are folded into the staging dir instead and the promotion stays a single atomic rename. The MPS generator report does not apply: torch.Generator(device="mps") has worked since PyTorch 2.0 (pytorch/pytorch#91348) and the studio installer pins torch>=2.4. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten diffusion comments Collapse the multi-line comment blocks across the image, video, sd.cpp and diffusion-training code to one or two lines each, and drop comments that only restate the statement below them. Comments only, no code or behaviour changes. * Tighten diffusion comments (second pass) Collapse the remaining multi-line comment blocks in the video page, training routes and service, sd.cpp server and installer, memory and speed planners, and the shared request models. Comments only, no code or behaviour changes. * Tighten diffusion comments (third pass) Collapse the remaining multi-line comment blocks in the attention, cache, LoRA, prequant, precision and compile-cache modules, the sd.cpp arg builder and engine, the video routes, the Ideogram 4 assembly, the model picker, and the diffusion test suites. Comments only, no code or behaviour changes. * Restore the dataset when an example import cannot be promoted Promotion folds the folder's pre-existing entries into the staging dir so the swap is one atomic rename. Every failure after that fold left the user with nothing: the 409 path and an os.replace error both fell through to 'finally: shutil.rmtree(staging)', which deleted the entries that had just been moved in there, while the response said 'Nothing was written'. A same-named entry was also unlinked outright before the promotion was known to succeed. Park superseded same-name entries in a rescue dir instead of deleting them, record every move, and restore all of them if any step of the promotion fails. The fold loop itself is covered too: renaming a non-writable directory raises EACCES on POSIX, which previously escaped as a 500 after the earlier entries had already been moved out. A failed rename now maps to the same retryable 409 as the rmdir conflict. Verified with the reported trigger (a non-empty mode-500 directory whose name collides with an imported file): the folder listing is now identical before and after the failed import. * Drain the teardown fence on a failing unload, give each load its own cancel event Two independent leaks on the image path, both already solved elsewhere in the same file. unload() incremented _teardown_waiters, ran _unload_locked() and decremented, with no try/finally, while the superseding-load path used a finally for the same pair. _unload_locked ends in clear_gpu_cache(), whose CUDA branch calls synchronize/empty_cache/ipc_collect unguarded, and a sticky CUDA fault makes those raise. The count then never drained, so every later generation was refused as cancelled for the life of the process, a fresh load included, since begin_load's own increment and decrement are symmetric. unload() is reached from the chat/video GPU handoff, the engine router and two training routes, so one fault during an ordinary handoff wedged image generation until restart. Release it in a finally. The image and native backends each cleared one shared cancel Event on a new load. unload() sets that event to cancel an in-flight multi-GB download and drops _loading in the same breath, so a replacement load is admitted while the cancelled worker is still inside the fetch, and its clear() re-enabled the very object that worker was watching: the cancelled download resumed and ran alongside the replacement. Take a fresh Event per load and thread it to the worker, as the video backend already does, and set it under the lock since begin_load now rebinds the attribute. * Name utf-8 on the diffusion text I/O and the sd.cpp subprocess pipes tests/test_text_io_encoding.py failed on five files this branch adds. Text I/O without an explicit encoding falls back to the Windows ANSI codepage, so a non-ASCII path or manifest value round-trips corrupted, and the three sd.cpp pipes decode the child's UTF-8 output as ANSI on Windows despite already passing errors = 'replace'. Eleven read_text() / write_text() sites across diffusion_compile_cache, diffusion_ideogram4 and diffusion_krea2, plus text = True on the sd-cli version probe, the sd-cli run and the sd-server pipe. * Record the load-time build on a gallery image's recipe A gallery record documents itself as the image's full generation recipe and is embedded in the PNG, but the only load-related field it carried was the repo id. A GGUF repo holds many quants, so that does not say which one made the pixels, and it says nothing about an adapter baked in at load time. The fallback meant to cover the baked case could never fire: with no loras on the request _adjust_baked_loras zeroes every baked adapter and _active_lora_pairs drops zero-weight entries, so active_loras was always empty. A baked-and-disabled build is not the same pipeline as a never-baked one, so the recipe could not reconstruct the image once the model was rebuilt. Persist model_kind, gguf_filename, transformer_quant and the baked adapter names, read off the load state rather than the request, and show them in the recipe popover. The new fields are optional with defaults, which matters because list_gallery_images drops any record that fails validation, so a required field would have emptied every existing gallery; a regression test pins that. * Drop eleven duplicated comment tails, restore the mxfp8 denial note The comment passes collapsed several wrapped blocks onto one line without deleting the last physical line of the original wrap, leaving the tail of each sentence repeated as its own comment underneath. Two of the eleven were re-worded rather than byte-identical, so a strict suffix match missed them. 6e16ad16f also dropped the line justifying the qwen-image mxfp8 denial while that rule stayed live in _FAMILY_SCHEME_DENY, under a header that then documented only fp8 and nvfp4. Restored. Comments only; verified with the AST gate. * Only let the dense-quant fallback use shards the prefetch actually staged The prefetch skips the base repo's transformer/ shards whenever a prequant checkpoint is expected, since that checkpoint replaces them. But a prequant fetch can fail for reasons the planner cannot see: an unpublished, gated or renamed artifact, a hub 5xx, a proxy, a checkpoint the validator rejects. The loader then fell through to from_pretrained(subfolder = 'transformer') and pulled those shards inside the load lock during 'finalizing', after the previous pipeline was already evicted, where the cancel event has no reach, load_progress has already reported bytes_downloaded == bytes_total, and the cache-disk gate had only reserved the small prequant checkpoint. That is verbatim the situation _dense_quant_prefetch_needed's own docstring exists to prevent. Gate the in-loader dense fallback on the shards being staged, read off the returned file list rather than the request so a failed size estimate closes it too, and let the GGUF build take over otherwise, which is what the prequant-sized replan already does one branch over. It is also the invariant the text-encoder path already enforces: only a component that really resolves may have its dense weights dropped from a plan or a prefetch. Adds the missing coverage for the gate's prequant arm, which the existing disk-gate tests never reached. * Refuse a training output_dir that resolves to the outputs root resolve_output_dir strips a leading 'outputs' and drops '' / '.' segments, so '.', './', './.', 'outputs', 'outputs/outputs' and ' . ' all clean away to nothing and land on the outputs root itself rather than a run directory under it. The DiT trainer then writes pytorch_lora_weights.safetensors flat into the root, where scan_trained_models and scan_checkpoints cannot see it (both filter is_dir()), and a second such run overwrites the first. The UI only checks the field is non-empty, so 'outputs' is one plausible run name away. Refuse it with a 400 that says what to do instead. cond_cache_dir collapses the same way but has an honest 'off' to fall back to, so a root-resolving value now means the in-memory cache, which is what the comment above it already promised: otherwise a run drops one flat safetensors per cached latent and caption into the directory trained models live in. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Repair three defects the main merge left behind Git merged all three files without a conflict, but the result was wrong in each case. The tests only started failing once the merged tree was run. routes/inference.py duplicated the GGUF load block: this branch had moved the gguf_load_in_flight marker and the _hub_download_blocks_gguf_load guard under "if config.is_gguf and config.gguf_hf_repo", and the conflict resolution re-added main's copy at the old position, so both ran. Dropped main's copy; the earlier placement is the deliberate one, so a 409 from the hub guard cannot tear down a resident Images or Video pipeline. test_gpu_selection.py still called _hf_offline_if_dns_dead, which main renamed to _hf_offline_if_unreachable_for (#7591). Disjoint edits, so no conflict, but four route-error tests referenced a function that no longer exists. test_gguf_load_cache_reuse.py anchored its ordering assertion with rindex over "if config.is_gguf:", taking the last one before the load marker. That only held while _resolve_inherited_extra_args sat above every such line; main has since hoisted it above the gpu_ids preflight, so the anchor landed between the call and the marker and the assertion compared against an unrelated later call site. The ordering it checks is a property of _load_model_impl as a whole, so it now anchors on the function. The ordering itself is intact: _resolve_inherited_extra_args, then the gguf_load_in_flight marker, then the hub guard, then the chat handoff, then unload_model. * Stop a real sd.cpp install from breaking its own discovery tests The five "nothing is installed" assertions in test_sd_cpp_engine.py cleared SD_CLI_PATH and UNSLOTH_SD_CPP_PATH and patched Path.home, which covers hops 1, 2 and the fallback half of hop 3. It leaves two hops live. Hop 3 goes through managed_install_root(), which honors UNSLOTH_STUDIO_HOME and STUDIO_HOME and resolves to the stable-diffusion.cpp directory beside the studio home. That is the documented way to run side-by-side Studios, so anyone who has one set gets a real sd-cli back from a finder the test expects to return None. Hop 4 is the in-tree developer build, which does the same for anyone who built sd.cpp inside the checkout. Isolated with an autouse fixture rather than another helper call, because reaching the failure needs no fixture: SdCppEngine(binary = None) runs the finder from its constructor, which is why test_generate_raises_when_binary_missing failed too. The fixture points the studio home and the in-tree root at an empty tmp tree, so every hop is answered by the test rather than by the box. The in-tree root moves into a named in_tree_install_root() so it can be pointed somewhere empty; behaviour is unchanged, including the OSError and IndexError guard on an unexpected layout. * Make the sd.cpp uninstall test actually extract the code it tests The two sed ranges anchored the production fragments at column 0, but both blocks sit inside the main removal function and are indented, so each range matched nothing and LOOP_FILE / DEFAULT_FILE came out empty. Sourcing an empty file is a no-op, so the suite reported 4 passed / 7 failed: the seven removal assertions failed because nothing ran, and the four that passed were 'kept' assertions that pass trivially when nothing runs. The shell job auto-discovers tests/sh/test_*.sh and runs each under set -e, and this file is not in its skip list, so it was a deterministic red. Anchor on optional leading whitespace and fail loudly on an empty or _remove_path-less fragment, so a future reshuffle of uninstall.sh cannot make the suite vacuous again. Now 11 passed, 0 failed, against the real removal loop and the real default-mode block. * Agree on what an engine can build, and on what the installer owns Two gates, each half-applied. The unbuildable-family gate had one caller, the image branch of the cached repo picker. The GGUF classifier, the local-model classifier and the video branch had none, so on a diffusers too old for a family the picker still offered its GGUF and the load then failed. Meanwhile the loader asserted the diffusers pipeline class before engine selection, so a GGUF this host routes to native sd.cpp, which instantiates no pipeline class at all, was refused with an upgrade instruction that could not help. Both are wrong one-sidedly: hide a family only when NEITHER engine can build it, and demand the diffusers class only when diffusers is what will load it. One predicate, family_buildable_here, now answers both, so the picker and the loader cannot disagree. The population is real on Python 3.9, whose diffusers ceiling is 0.36: Flux2KleinPipeline, Krea2Pipeline and LTX2Pipeline are all absent there, and FLUX.2-klein GGUF is a repo in this PR's title. That assertion also raised RuntimeError, which /images/load maps to 409, the status that otherwise means a load is already in progress, and which escaped /images/download-plan (it catches ValueError and FileNotFoundError) as a bare 500 with the message lost. It is an unloadable pick like every other, so it raises ValueError and both routes answer 400 with the text intact. The managed-binary repair used a path test for ownership while the installer uses a marker. On a managed root without the marker, which is any install predating it, the repair deleted sd-server and the reinstall was then refused, permanently, because the surviving sd-cli keeps the directory non-empty so the marker can never be claimed: the user went from an unrunnable binary to no binary and no way back. Require the marker before discarding, which is the same definition of ours that uninstall.sh already uses to keep a user's own stable-diffusion.cpp checkout. A genuinely interrupted extraction still self-heals, since install() writes the marker before it extracts. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add gguf and av to the studio extra so the wheel install matches studio.txt This branch added both to studio/backend/requirements/studio.txt, which is what install.sh uses, but never to the studio extra in pyproject.toml, which is what 'pip install unsloth[studio]' uses. Nothing else keeps the two in sync, so tests/studio/install/test_studio_extra_matches_requirements.py failed deterministically, and the CI job runs the whole tests/ tree. The drift is not only a red test: gguf backs diffusers' GGUFQuantizationConfig and av does the MP4 encode and audio mux, so a wheel install got a Studio that cannot read a GGUF or export a video, which is most of what this PR adds. * Fix two test-suite defects: the CPU-only patch gate and a popped module The patch backend reaches unsloth_zoo's helpers only through an ~940 MB 'import unsloth', which a CPU-only host cannot complete, so every patch install returned False and nine arch/eager/compile tests failed there -- exactly the runners the retry was narrowed to protect. unsloth_zoo only wants UNSLOTH_IS_PRESENT in the environment, which costs nothing and needs no accelerator, and the conftest already sets its sibling UNSLOTH_ALLOW_CPU. Set it there too, as run.py and main.py already do at module scope, so no real server ever took the expensive route. CPU-only goes from 9 failed to 24 passed, 1 skipped; the GPU run is unchanged at 29 passed. test_setup_cache_env_hf_home popped utils.hf_cache_settings to model a fresh process and never restored it, so a later import built a second module object and rebound it on the utils package. test_hf_cache_settings then wrote its setting into one object while core.inference.diffusion read the other, the same split-module failure the xet shim already had. Restore both bindings in teardown: 21 pass together, and each file still passes alone. * Restore the setup_fail assertions the main merge reverted A fourth instance of the class 9541cc535 fixed: the merge took main's studio/setup.sh, which #7644 changed to abort through the setup_fail helper so desktop mode still emits [TAURI:ERROR], but kept this branch's older copy of the test, which still asserted the literal 'exit 1'. setup.sh is byte-identical to main here and the only diff in this file was the reverted assertions, so take main's version. 60 passed. * Fence the video teardown, reject non-finite clipping knobs, stop a test installing sd.cpp The video backend never had the teardown fence the image backend documents. Its unload and its superseding-load path both signalled the active generation, then did 'with self._generate_lock: pass' and tore the state down with the lock free. A generation queued behind that barrier holds no cancel event yet, so the signal cannot reach it, and Python locks are not FIFO: it won the lock the instant the barrier released it, read a still-loaded state and denoised a whole clip against the pipeline being freed. Reproduced on both paths, where the queued generation returned a finished MP4. Mirror the image backend: count waiters, refuse a generation while one is pending, and tear down inside the barrier with the counter released in a finally. _teardown_state becomes _teardown_state_locked since the caller now holds both locks. max_grad_norm and snr_gamma were bounded on one side only, so 1e309 floated to inf and passed. clip_grad_norm_ then computes an infinite clip coefficient, clamps it to 1.0 and scales nothing, and min(snr, inf) / snr makes every min-SNR weight 1.0. The run starts, reports normal progress and trains with the requested knob silently disabled. Measured both. NaN already failed the bounds; allow_inf_nan makes that explicit. learning_rate and flow_shift already guard this exact vector. test_load_routes_to_sd_cpp_on_cpu stubbed ensure_sd_cpp_binary but not ensure_sd_server_binary, which select_and_activate_engine probes first with installs enabled, so the unit test downloaded and unpacked 108 MB into the developer's real ~/.unsloth when UNSLOTH_STUDIO_HOME was unset. Now 0 bytes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Key the video schedule reset on the resolved defaults, not the repo id A GGUF repo holds several variants, so another client swapping a distilled build for the base one from the same repo changes the steps and guidance the backend reports while repo_id stays put. modelChanged stayed false, the page kept the previous schedule, and later generations ran an 8-step distilled setting on a model that expects roughly 40 steps and CFG 4. Include the reported defaults in the key. They only move when the resident artifact does, so a manual steps change still survives a status poll. * Retire the release-lag pin now that 2026.7.6 carries the relaxed gates The virgin Windows container row that installs from PyPI on purpose was tolerated with continue-on-error, because a Server Core container has no Microsoft Store and so no winget, and the released studio/setup.ps1 hard-stopped on a winget-only git gate and reached for winget again for the VC++ runtime. #7549 relaxed both, but it landed on 2026-07-28 and the newest wheel was 2026.7.5 from the 23rd, so the row could not get past it. The pin carried its own tripwire for exactly this moment, and it has fired: unsloth 2026.7.6 shipped 2026-07-29 and the released wheel now installs end to end, so the step reported success and the assertion errored with "delete this pin and drop continue-on-error from the Install step so this row gates". Doing that. Both rows now gate unconditionally, and the assertion that enumerated the accepted failure signatures goes with the pin it protected, since there is no longer an accepted failure for it to describe. The other three release-lag mentions in this file are descriptive rather than tolerated-failure pins: they explain why the fedora and ubuntu-nonroot legs run with overlay: true, and both already refuse a triton failure as a regression rather than accepting it as lag. Left alone. * Pin the hosted pre-quant onto the plain-torch fp8 kernel A Z-Image GGUF pick at the DEFAULT speed mode, once the hosted fp8 pre-quant is actually reachable, dies at generate with an HTTP 500: torch._dynamo.exc.Unsupported: Operator does not support running with fake tensors. Developer debug context: unsupported operator: mslk.f8f8bf16_rowwise.default _fp8_config already pins KernelPreference.TORCH when it BUILDS a config, precisely because the default AUTO switches to the MSLK kernel wherever an mslk package is importable (sm90+). A hosted checkpoint escapes that pin completely: the preference is serialized per Float8Tensor, and all 239 weights in the published Z-Image-Turbo-FP8 checkpoint carry AUTO. Loading it re-arms the very kernel the pin exists to avoid, mslk.f8f8bf16_rowwise has no fake impl, and the first compiled generate therefore cannot be traced. Isolated away from the product to be sure of the mechanism: quantise one Linear three ways on this box and compile each. KernelPreference.TORCH eager ok, compiled ok KernelPreference.AUTO eager ok, compiled FAILS on mslk.f8f8bf16_rowwise library default eager ok, compiled FAILS the same way So the pin is correct and necessary, and the only gap is that the hosted path never got it. _validate_checkpoint checks scheme, granularity, base model, min_features, exclude tokens and fast_accum, but not this. Rewriting the preference on load is safe: it selects a matmul kernel, it is not weight data, so the tensors stay bit-identical and the checkpoint's own state_dict_sha256 still describes them. It is also the faster path compiled, since the opaque extern call blocks inductor quantize fusion. Why this went unnoticed: the pre-quant repos are private, so nothing that could not read them ever took this path. It becomes the default the moment they are readable. Verified end to end at 1024x1024, 8 steps, speed_mode default, on the published checkpoint: before HTTP 500 at generate, diffusion.generate_failed after pinned 239 weights to the plain-torch fp8 kernel, load 10.0 s, transformer_quant fp8 with compiled engaged, cold 7.9 s, warm 0.87 0.87 0.87 0.86 0.87 s That warm number also beats the 1.4 s recorded for this shape on 2026-07-26. 159 prequant and transformer-quant tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cover the fp8 kernel pin so the 500 cannot come back quietly The hosted pre-quant re-arming MSLK was only visible as an HTTP 500 on a compiled generate, which needs a GPU, a readable private repo and an importable mslk to reproduce. None of that is available in CI, so the pin would rot unnoticed. Three hermetic cases on _pin_kernel_preference instead: AUTO weights are rewritten and already-TORCH ones are left alone (counting only what changed), a weight that refuses the assignment does not sink the whole load, and with no torchao enum available the checkpoint is left exactly as saved. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: tighten comments across the image generation, video and training code Rewrite the comments added by this branch to be shorter and clearer: collapse multi-line explanations into one or two lines, drop comments that only restate the code, and keep the rationale that explains why a choice was made. No code changes. * Studio: let the pipeline-class guard survive a host with no diffusers Backend CI installs the CPU-only dependency set, which has no diffusers, and runs without a GPU. 13 tests failed there for three separate reasons, one of which is a real product bug rather than a test-environment gap. assert_pipeline_class_available did a bare "import diffusers", so on any host without it the call raised ModuleNotFoundError instead of the ValueError its own contract promises. /images/download-plan catches only ValueError and FileNotFoundError, so that escaped as a bare 500 with the message lost, which is the exact failure the guard exists to prevent. It is reachable in production: the native sd.cpp engine serves GGUF picks on a CPU or Apple host where diffusers is never installed. Absent diffusers now returns, since the check answers "is the installed diffusers new enough for this family" and there is no version to judge; a pick that genuinely needs diffusers still fails in the loader with its own message. The rest were tests asserting through gates unrelated to what they cover: - Three cond_cache_dir route tests drive a DiT family, which the start route refuses without an accelerator. They now take the existing dit_train_host fixture, written for this case, so they keep testing the schema on every host. - test_download_plan_forwards_the_load_time_controls stubs the diffusers planner but sends a GGUF pick, which routes to the native planner on a GPU-less host. It now pins the engine, so it tests kwarg forwarding rather than the hardware. Engine selection keeps its own tests. - The pipeline-class guard test needed a real diffusers only for its sweep over every shipped family. That half is split out and skips on its own; the stub-driven refusal needs no diffusers and still runs. Added a test for the absent-diffusers contract above. 409 pass on a GPU host with diffusers; 301 pass and 1 skips with diffusers blocked and no GPU. * Studio: cut the image generation comments down further Second pass over the comments this branch adds. Collapse the multi-line blocks that still read as paragraphs, rewrite the longest one-liners so they say the same thing in fewer words, and drop a stale comment that had drifted away from the constant it described. No code changes: only comments and whitespace. * Studio: workflow rail, sidebar Video row, and Train settings polish Sidebar - Pin Video under Images by default; persist store bumped to v3 so an untouched install adopts it while a custom arrangement is left alone. - Add "Customize sidebar" to the end of the More flyout, opening Settings > Appearance scrolled to the sidebar nav section. Images - Replace the workflow dropdown with an icon rail down the left edge: all seven workflows visible, keyboard nav, tooltips carry the labels. - Workflows stay selectable with no model loaded, so one can be set up before picking a model. Generate is already gated on a loaded model. - Video moves out of the Create/Train strip into a link at the far right, with a matching Images link on the Video page. - Wider gutters around the settings column, headings matched to Train. Train - Field guidance moves into "i" tooltips; only state that limits a control stays on the page. Steps, LoRA rank and the rest gain hints. - /info reports params, qlora_vram_gb, gated and note as fields, so the family note renders as chips. vram_note is rebuilt from them unchanged for older clients. Shared - Number steppers appear on hover or focus. - Lighter control border (#e9e9e9) on Images and Video, light mode only. * Studio: move Advanced inline on Images and Video - Replace the top-bar toggle and right-docked Advanced panel with a disclosure under Seed, so load-time tuning sits with the settings it affects and opening it no longer shrinks the preview canvas. - Video uses the bordered variant, since offload and memory decide whether a model fits at all. - Open state persists per page in localStorage. - Tighten the Steps unit trigger: 14px chevron, less right padding. * Studio: full width media pages and layout polish - Drop the 1100px cap on Images Create, Images Train and Video, so the preview canvas grows with the window instead of sitting in a band. - Align the workflow rail to the model selector label, widen the gap before its divider, and restore the divider itself. - Advanced: more room above and below, icon and label sized to the slider rows, and the same quiet row on Video as on Images. - Give Video a pane heading and description, matching Images. - More space above Seed on both pages. - More no longer takes the active style when the current page is one of its own rows. * Studio: fix the Extend side toggles The resting outline used a ring, and index.css blanks the ring on the button holding mouse focus, so the side you just clicked lost its border until focus moved. Use a border for the outline and leave the ring to focus-visible. Also separates the two states properly (the off state had no surface of its own), drops the border in dark mode as the inputs do, and adds the aria-pressed and focus-visible styling the buttons were missing. * Studio: drop the border on the Extend side toggles Light mode should carry no border either, so the fill alone marks the state: a muted surface when off, a primary tint when on. Matches the borderless treatment dark mode already had. * Studio: move the Images workflows into the sidebar The workflow switcher was a vertical icon rail on the page. It now lives in the sidebar under Images, so the page keeps that width for the canvas. - Workflows list under the Images row, with a chevron to fold them away. Hovering the row peeks them in a flyout on the standard menu surface. - Clicking Images while already there toggles the list instead of navigating. - The listed workflow carries the highlight, so the Images row drops it. - Create takes its own icon: it was sharing the New chat pencil. - Model hub moves above Projects. Persisted layouts bump to v4 and only adopt the new order where the stored one is still a shipped default. - Images and Video content both start at 32px, clear of the sidebar. * Studio: simplify the Images workflow submenu Dropping the hover flyout: two ways to reach the same seven workflows, one of them overlapping the list right below it, read as clutter. - The flyout is gone. The workflows are rows under Images and nothing else. - On the Images page they are always open, since they are that page's switcher. Elsewhere they stay folded, and the row's chevron opens them. - Create takes the sparkles icon. - Create and Train both pad their settings column to 40px a side, so the two tabs line up with each other and with the model selector. * Studio: dock the generate action and tidy the Create controls The primary action sat at the foot of a long scroll, so it was off screen until you scrolled for it. - Generate, Video's Generate and Train's Start training float at the bottom of their settings column. No bar behind them: hover lightens the fill rather than thinning it, and the disabled state is opaque, so the controls underneath never show through. - Aspect ratios read as names: Square (1:1), Widescreen (16:9) and so on. - Width and height replace their sliders with two compact boxes. Type a size or pick one from the menu; the value still snaps into range and still drives the locked ratio. - Negative prompt gets a quiet disclosure under the prompt. It was there before but only above guidance 0, which is not the default, so it never showed. - The info "i" is smaller across the settings UI. * Studio: reveal Images and Video field hints on hover The "i" next to every field label sat there permanently, which made a column of settings read as busier than it is. Scoped to the diffusion pages by CSS rather than per component: the hints come from the page's own Field and SelectRow, the train panel's FieldLabel, and the chat ParamSlider, which is shared and should not change for chat. Reveals when the pointer is inside the field, on focus-within for keyboard, and stays inert while hidden. InfoHint grows a data-slot so the rule has a stable hook, rather than keying off its aria-label. * Studio: one-line sliders, edge fades, shared negative prompt - Steps, Guidance and the rest put label, track and value on one row. ParamSlider is shared with chat, so this is an opt-in prop and chat keeps the stacked layout. - The settings columns fade at whichever edge they run past, as the sidebar and model picker do, instead of cutting off. New useScrollFades hook drives all three. - Negative prompt moves to a shared component and Video picks it up, so both pages collapse it the same way. - Right gutter sits closer to the rule, Steps takes a bigger break above it, and the Advanced rule sits between its neighbours rather than up against the field above. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: add the API monitor to the sidebar nav Sits under More, below Export. Points at the monitor page, not the API keys dialog the profile menu opens. No store version bump: an id missing from a stored layout is appended with its default pinned state, so existing sidebars gain the row at the end and keep their own order. * Studio: take main's copy of the sidebar spinner test The merge kept this branch's older version, which pins the Recents row to h-[33px]. Main already relaxed that to any height, since row density moves independently of the trailing column the test is about. * Studio: fix FLUX.2 klein size resolution, HunyuanVideo tier routing, and the personalization save Five fixes found while auditing the model registries against the live Hub. FLUX.2 klein 9B loaded against the 4B config. One family covers both klein sizes and defaults to 4B, relying on the base_model card tag for the real base, but the trust allowlist only had klein-4B, so the correct tag was discarded and a 9B GGUF (inner_dim 4096) was loaded against a 4B config (3072). That surfaced as a bare shape mismatch from inside the GGUF quantizer naming neither the file nor the repo. Adds the missing allowlist entries and a header-level size check that fails early with a legible message; the check is fail-open, so an unreadable file, a non-FLUX.2 family, or an unmapped base leaves the load exactly as it was. The sd.cpp text encoder rule matched the literal "klein-9b", so klein-base-9B was handed the 4B encoder. HunyuanVideo-1.5 720p checkpoints routed to the 480p family. Only the literal 720p_t2v path was aliased, so 720p_i2v and every GGUF repack fell through to the generic token and inherited the 480p base repo, which also supplies the VAE and text encoder. The tier is baked into the weights (transformer target_size 640 vs 960, scheduler shift 5.0 vs 9.0) and the two bucket lists are disjoint, so this ran the whole pipeline off-tier. Resolution presets that the checkpoints were never trained for. Wan2.2 TI2V-5B is 720P-only upstream (SUPPORTED_SIZES is exactly 704x1280 and 1280x704, asserted in generate.py), and the HunyuanVideo 480p square preset 624x624 is not a bucket of its tier; 640x640 is. Personalization could never be saved. The frontend ships eight sidebar nav ids and refills every missing one on each save, but the backend Literal had seven and no "api", so every PUT to /api/settings/personalization returned 422. Aligns the server defaults and the Literal with the shipped layout. A catalog row pointing at a file that does not exist. unsloth/Qwen-Image-2512-FP8 holds torch prequant .pt checkpoints, not qwen-image-2512-fp8.safetensors, and fp8 is denied for this family anyway because it renders black, yet the row was the auto-route target on any GPU above 40 GB. Also marks FLUX.1-schnell, Krea-2-Turbo and the two Ideogram repos gated: all four are gated on the Hub today, and schnell being Apache-2.0 does not make it ungated. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: stop a case-only duplicate upload destroying an image, and size the load against the right GPU Two fixes from the review round. A dataset upload containing two names that differ only by case silently loses one on Windows and macOS. The in-batch duplicate check is an exact string compare, so "Cat.png" and "cat.png" both pass, but on a case-folding filesystem they are one destination: the commit step moves the first staged part aside, writes the second over it, then deletes the backup, while the response still reports both as uploaded. Captions collapse the same way and never even reach the image branch. No single folder can hold such a pair, but the Windows and macOS open dialogs flatten search and Recents results across folders into one multi-selectable list, which is a normal way a LoRA dataset gets assembled. Rejecting the pair everywhere would regress Linux, where the two really are different files with their own sidecars, so the check probes the filesystem once per process instead of keying off sys.platform: macOS also ships case-sensitive APFS volumes, and a Linux host can keep its Studio home on an exFAT or NTFS mount. A failed probe answers "case-sensitive", which leaves today's behaviour untouched. The quantization fit budget was sized against the wrong card under a reordering CUDA_VISIBLE_DEVICES. A bare "cuda" load lands on visible ordinal 0, but the reducer took the lowest PHYSICAL index, which the nvidia-smi path reports as index_kind "physical". Under "3,1" that sizes against GPU 1 while the pipeline loads onto GPU 3. The hook's own interface doc already promised the lowest visible ordinal. Ranks by visible_ordinal, falling back to index only for an older backend that omits it. * Studio: rework the diffusion dataset panel and align the pane gutters Upload: - one Upload button beside the dataset name, with an upload icon - picking files uploads them, no second confirm click - Add button beside the Training images dropdown, so images can go into a set that already exists - fall back to the upload form when a selected dataset no longer resolves Labeling grid: - two columns at any width; sm:grid-cols-3 crushed the tiles in a fixed width column - pin the caption size so the Textarea's md:text-sm does not outsize a tile - name the tile hover group; a bare one revealed every tile's Remove at once - Remove is an icon button on the image, not a word over the artwork - drop the amber tile fill, keep the No caption label - match the header, toggle and status text to the section's type scale Gutters: - the run area and the preview canvas now sit 40px off the rule, the gutter the settings column has off the page edge * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: give the dataset preview per-image hover and remove, and wrap it onto rows - each thumbnail is its own button, so it can carry an X that removes just that image; the strip was one button and could not nest a second control - hover lightens the image instead of drawing a border - wraps onto more rows rather than scrolling sideways, which also keeps the bottom corners rounded - samples 12 thumbnails, up from 8, now that a second row is available - a delete refreshes the panel's dataset counts * Studio: show the workflow icon beside the Images and Video headings Images reads it off WORKFLOW_TABS, so it is the same icon the sidebar submenu shows and it follows the active workflow. Video's heading is also Create, so it takes the Create icon rather than the nav row's film slate. * Studio: reject a duplicate LoRA id on the diffusion load path too DiffusionGenerateRequest already refuses a repeated adapter id, with a comment naming the hazard: _resolve_lora_set suffixes colliding adapter names, so the same id resolves the SAME adapter twice and set_adapters stacks both copies past the per-adapter weight bound. DiffusionLoadRequest bounds only the list length, and it matters more there. Generation-time stacking spoils one image; on the load path the adapters are baked into the quantized build before compilation, so the unintended combination rides every image until a reload. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: teach the GGUF reuse test double about holds_no_vram The resident-GGUF fast path consults llama_backend.holds_no_vram before asserting CHAT ownership, but two tests hand it a SimpleNamespace built before that attribute existed, so they raised AttributeError and returned a 500. A real LlamaCppBackend exposes it as a property; the doubles now carry it too. Product code is unchanged: these pass on main and were the branch breaking its own test, not a defect in the guard. --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: michaelhan <michaelhan2050@gmail.com> Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: Michael Han <107991372+shimmyshimmer@users.noreply.github.com> | 1 个月前 | |
Studio: add image generation, editing workflows and LoRA training with Unsloth GGUFs (#6763) * Tighten comments in the image stack tests and scripts * Close video single-file, training reservation, and image mount-resume gaps Route on-device single-checkpoint video folders through the single_file loader: a bare local .safetensors directory (no model_index.json) is advertised as a pipeline with no filename, so validation rejected it before it could load. Reinterpret the pick as a single_file load of the sole checkpoint, mirroring the image load route. Treat a reserved-but-not-yet-spawned LLM training start as active in is_training_active() so /images/load, /video/load, and /diffusion/start cannot race the reserved run for VRAM during the pre-spawn free window. Mirrors the diffusion training service reservation. Resume an in-flight image generation on the Images page mount: probe generate-progress, re-enter the poll loop, and refresh the gallery on completion so a run started elsewhere is reflected and its saved image appears without a manual refresh. Seed resident image defaults from the resolved base_repo rather than a possibly path-shaped repo_id so the first resident generation uses the right recipe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Publish image generation active state before pre-denoise setup generate() assigned self._gen only at the pipe() call, after deferred compile, LoRA resolution/application, and ControlNet download/build had run. Across that setup window generate_progress() reported inactive even though _generate_lock was held, so a reloaded page's mount probe showed idle and let a second generate queue behind the first. Publish an active step-0 _GenState the moment the generation lock is acquired, before the setup work, and clear it in the outer finally so a setup-time error cannot leave the UI stuck active. Mirrors the video backend's queued phase and the training start guard. * Studio: fix diffusion install ownership, dataset upload atomicity, gallery pagination, and teardown races install_sd_cpp_prebuilt: only write the .unsloth-studio-owned marker when the install created the target directory or it was empty. Adopting a pre-existing, unowned, non-empty directory (a user's own stable-diffusion.cpp checkout) made it eligible for the uninstaller's recursive delete. routes/training upload: make the multi-file promotion transactional. Back up each displaced original and roll every destination back on any failure, so a mid-loop rename error can no longer partially overwrite the live dataset. routes/training _resolve_dataset_folder: reject a symlinked dataset directory and prove the resolved folder stays under the datasets root, so image read/caption/delete cannot escape the root through a link. routes/training delete: escape glob metacharacters in the thumbnail filename so deleting an image named like [ab].png removes only its own thumbnails. image_gallery / video_gallery listing: filter records against the response schema inside the pager via a valid callback, so offset/limit/has_more all count over accepted records. A leading schema-invalid record no longer returns an empty page with has_more=true and stalls infinite scroll at offset 0. image_gallery / video_gallery save: publish via a temp file plus atomic rename (the sidecar is the video pair's commit marker) and clean up on failure, so a partial write never surfaces a truncated PNG or strands an orphan MP4. diffusion_train_common discovery: treat an empty caption sidecar as a metadata tombstone that still falls through to the dreambooth instance prompt, so clearing every metadata caption no longer fails with no captioned images found. diffusion backend unload: wait for an in-flight denoise to exit before tearing down process-wide patches and state, mirroring the load path. diffusion_engine_router: serialize the whole check/unload/publish transition so a concurrent selection cannot return the engine being unloaded. uninstall.ps1: gate the default sd.cpp process stop on the owner marker so a user's own sd-server is not terminated for a directory we then keep. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reject native batch seeds outside the JSON-safe range * Refuse sd.cpp install into unowned non-empty target dir When the install target already exists, is non-empty and lacks the .unsloth-studio-owned marker (a user's own stable-diffusion.cpp checkout, or unrelated files beside a custom Studio root), install() previously still extracted the release into it. Skipping the ownership marker only stopped the uninstaller from deleting the directory; extraction still merged binaries into the user's working tree and could overwrite same-named files. Fail up front with a clear message pointing the user at a fresh/empty location before any download or extraction, leaving their directory untouched. Update the ownership test suite to assert the refusal. * Studio: gate dataset uploads on the symlink check and surface local video single-file checkpoints * Tighten comments and docstrings added by the image-generation fixes * Studio: close arbiter load-registration race and surface native progress + local pipeline folders Publish native sd.cpp generate progress (_gen) before LoRA resolution so a reload probe reads active during setup, matching the diffusers path. Register the diffusion/video GPU load under the arbiter lock (acquire_for now takes a register callback) so a competing acquire cannot evict an owner before its load is marked in-flight and let two loaders allocate VRAM at once. Admit local diffusers pipeline folders (root model_index.json, weights in component subdirs) in the local model scan so they reach task tagging and the On Device picker. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: include marker-owned custom sd.cpp roots in the uninstall stop scan; harden Pester install against the nuget.exe PSGallery bootstrap * Studio: surface local pipeline scan roots, tag single-file checkpoints by filename, mark companion-only pipelines partial - _scan_models_dir: admit a scan folder that is itself a diffusers pipeline (root model_index.json, weights in transformer/ vae/ subdirs). _is_model_directory rejects such a root, so the child scan would list the component subdirs as bogus models and hide the real pipeline; treat the root as one model via _local_pipeline_index. - _local_is_diffusers / _local_model_task: include the sole checkpoint filename in the family-detection needles (_local_family_needles, resolved via resolve_local_single_file). A generically named folder holding one loadable qwen-image-*.safetensors / ltx-*.safetensors identifies its family only from the filename; the load route already resolves that file, so tag it or the task-scoped picker (which rejects task=null) hides the on-device model. - list_cached_models: mark a companion-only base snapshot partial. A GGUF image load prefetches the base repo's VAE / text-encoder / model_index.json but skips the transformer (the GGUF supplies it); the snapshot has a pipeline manifest yet is not a loadable BF16 pipeline, and _cached_repo_partial misses it. _repo_pipeline_missing_denoiser flags a pipeline snapshot whose transformer/ or unet/ component carries no weight, so the picker drops it instead of advertising it as fully on-device. * Studio: preserve foreign gallery files, force safetensors on remote ControlNets, and close dataset/seed/GPU gaps Gallery clear/delete now scope to Studio-owned files: image_gallery and video_gallery skip PNGs / MP4s without a readable recipe (a hand-dropped or orphan file the listing already hides), so clear() and a guessed-id delete no longer destroy files the gallery never surfaced. Remote ControlNets now force use_safetensors: a bare owner/name reaches from_pretrained without the base trust gate, and the Hub scan fails open when unavailable, so requiring safetensors closes the pickle deserialization vector. POSIX uninstall now stops resident sd-server / sd-cli under an owned sd.cpp root before removing the tree (marker-gated), mirroring the Windows stop-before-delete scan; a live native server no longer survives unlinking its binary. Diffusion dataset containment: the training-start read path and the discovery picker route bare names through the protected resolver, so a symlinked dataset is rejected / not advertised like the caption/delete routes already do. Uploads gain the inference decode guard (oversized real images 400 before OOMing the trainer) and dataset upload/caption/delete/import are blocked with 409 while a diffusion run is active. JSONL readers (trainer + routes) tolerate non-object JSON and invalid UTF-8 instead of raising AttributeError / 500. LoRA family compatibility is enforced in the shared resolver, not only the picker, so a direct API client cannot apply a mismatched-family adapter. GPU arbiter gains release_if so the image/video unload idle-check and release are atomic against a concurrent same-owner load's registration. Native batch recipes persist the base batch_seed and restore replays from it, so a native batch_index>0 image no longer advances its seed twice. FLUX.2-klein selects its sd.cpp text encoder by variant (4B -> Qwen3-4B, 9B -> Qwen3-8B) instead of the single family default. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: track the loaded GGUF filename so native companion resolution reproduces the load identity * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: gate local-pipeline image tagging on a real family; validate video sidecars before delete/clear * Studio: tighten image-generation fix comments and docstrings * Studio: gate gallery serve/export on ownership; keep image progress active until persisted; reserve diffusion training before the dataset scan * Studio: restore Reapply target on async image/video load errors; recheck training state before dataset commit; reclaim partial sd.cpp installs on retry Images/Video: a background model load that fails AFTER starting (error/eviction during download) leaves the previous pipeline resident, but handleLoad had already overwritten lastLoad.current with the failed pick, so "Reapply to loaded model" reloaded the failed model. Carry the prior Reapply target into the poller and restore it on the async error/null paths, mirroring the quant rollback. Training: an in-flight diffusion dataset upload passed _require_diffusion_dataset_mutable() at entry but could still commit files after a concurrent /diffusion/start reserved the training slot, mutating the dataset underneath the trainer. Re-check the interlock immediately before the commit phase; a 409 there leaves the staged temps for the finally to clean. sd.cpp install: an interrupted extraction (disk full, killed process, a raising post-extract cudart fetch) left the target non-empty with no owner marker, so the next lazy install tripped the "not a Studio-managed directory" refusal and wedged native install. Write the ownership marker before the partial writes when the target is reclaimable, so a retry recognises the debris as ours and re-extracts. * fp8 DiT quant: floor the dynamic activation scale with activation_value_lb An all-zero activation token row makes the dynamic per-row fp8 scale 0, which turns the quantized data to NaN and the render to black frames on torchao's plain-torch kernel path. The fused fbgemm/mslk quantize kernels clamp zero rows internally, so the bug only reproduces on machines without them, which is most user environments. Zero rows are real inputs, not a corner case: Wan 2.2 zero-pads its text conditioning, and Hunyuan-1.5 and Qwen-Image regenerate zero rows inside their transformer blocks every step. Pass activation_value_lb=1e-12 to Float8DynamicActivationFloat8WeightConfig whenever the installed torchao supports the kwarg (Float8Tensor rework, 0.13+), checked via inspect.signature so older torchao keeps exactly the current behaviour; the existing Float8MMConfig fallback chain is unchanged. Verified on GPU: with the forced plain-torch kernel path a zero-row input NaNs without the floor and stays finite with it, and end to end on HunyuanVideo-1.5 fp8 goes from a solid black frame (LPIPS 1.00) to a normal render (LPIPS 0.225); on Wan the floor matches the condition_embedder exclusion (LPIPS 0.211 vs 0.206). Same-seed renders with fused kernels present are unaffected, and pre-quantized fp8 checkpoints stay valid since weight scales are untouched. * Wire hosted pre-quantized DiT checkpoints into the image families Point prequant_repos for flux.1, flux.2-klein, flux.2-dev, qwen-image (int8 only there; fp8 is family-denied), z-image and krea-2 at the unsloth/<Model>-FP8 Hub repos carrying gate-validated int8 and fp8 transformer checkpoints, so the fast quant path loads the small pre-quantized file instead of materialising the dense bf16 transformer and quantising on device. Measured on FLUX.2-dev int8: build peak drops from 60.7 GB (dense + quantize) to 30.7 GB (hosted prequant), identical 30.7 GB resident after either path since loading a checkpoint is bit-identical to on-the-fly quantisation. The hosted repos name files <Model>-<SCHEME>.pt, so resolve_prequant_source now derives that model-name filename from the repo id (scheme suffix stripped case-insensitively) and carries the legacy transformer_<scheme>.pt as a fallback the resolver tries when the primary 404s, keeping older repos loadable. Wiring a repo also exposed a fallback hazard: with a prequant source present, the dense-fit preflight used to be skipped entirely, so a failed prequant download would fall through to the dense bf16 load the memory plan never budgeted, OOMing after eviction. The preflight now always runs and gates an allow_dense_fallback flag through _load_dense_quant_pipeline: a dense misfit still skips the fast path when no prequant exists, but with one it proceeds and a prequant failure raises to the GGUF build instead of loading dense. The same flag is set when the auto-policy replans an offloaded GGUF against a prequant-sized transient. Tests updated to the new filename convention plus new coverage for the derivation and the legacy-name fallback; the prequant-skips-refit test now asserts the re-check runs and forbids the dense fallback. Verified end to end on GPU: z-image int8 resolves the hosted repo, downloads the model-name file and renders (6.8s load, 5.9 GB peak). * Route krea-2 through its per-component loader on the transformer-quant fast path _assemble_pipe used Pipeline.from_pretrained for every family, but the krea repo ships transformers-5.x configs and no top-level tokenizer files, so the tokenizer dies with vocab_file=None. The pre-quantized checkpoint loaded fine and then the assembly crashed, dropping the load to the GGUF build, which krea-2 cannot take (Krea2Transformer2DModel has no from_single_file). Assemble per-component via load_krea2_pipeline like the pipeline-kind and single-file paths already do. Verified live: Krea-2-Turbo int8 and fp8 hosted prequant loads now assemble and render through the Studio images tab. * Keep Qwen-Image's text-stream linears bf16 on int8 (short prompts break torch._int_mm) Qwen-Image's MMDiT runs every text-stream Linear at M = actual prompt tokens: the Qwen2.5-VL embeds are not padded to a fixed length like FLUX's 512-token T5. A short prompt (13 tokens) or the near-empty negative prompt drives torch._int_mm below its M > 16 floor and the first denoise step raises 'self.size(0) needs to be greater than 16, but got 13' (measured on B200 through the Studio images tab). Add per-family int8 exclusions (txt_in, add_q/k/v_proj, to_add_out, txt_mlp) for qwen-image and qwen-image-edit, threaded through exclude_tokens_for_scheme(scheme, family) and the prequant checkpoint validation, so a checkpoint baked under the old token list is rejected and re-quantised instead of loaded crashing. The text stream runs at M = tens vs the image stream's M ~ 4k, so the exclusion costs nothing; the rebuilt hosted checkpoint gates 28/28 PASS with LPIPS mean 0.057 (was 0.069). * Harden the diffusion memory plan against transient free-VRAM undercounts A cold FLUX.2-dev int8 load on an idle 183 GB B200 planned offload=model (companions exceed budget) and silently served the GGUF as-is; the identical retry went resident and engaged the hosted prequant. The plan arithmetic was byte-identical across both loads (required 90,228 MiB, resident needs free of about 124 GB); the only divergent input was torch.cuda.mem_get_info, which is device-wide and instantaneous: a transient foreign CUDA context briefly held about 100 GB at the first snapshot, and the planner trusted that single read. Three changes: - settled_snapshot_device_memory: on cuda, synchronize + empty_cache (best-effort) and take the MAX free over up to 3 spaced reads. A transient can only shrink free, so the max rejects transient undercounts while a persistent tenant still caps every read. _plan_memory now uses it. - plan_fits_total_capacity + one replan retry: when the dense/prequant candidate fits TOTAL device capacity under the standard reserve and the 0.85 resident margin, an offload verdict can only stem from the free reading, so the loader re-snapshots and replans once before declining the fast path. Explicit balanced/low_vram modes skip the retry (they offload by mode). - diffusion.transformer_quant_declined log line with required/budget/free and the plan reasons, so the next decline is diagnosable from the server log (previously silent). Verified: cold FLUX.2-dev int8 first load in a fresh server now engages the hosted prequant resident (offload=none). * Add FLUX.2 Klein and FLUX.2-dev DiT LoRA training Register flux.2-klein and flux.2-dev in the DiT trainer following the upstream DreamBooth references: latents train patchified and batch-norm normalized from the VAE posterior mode, the packed forward reuses step-invariant position ids, and the guidance vector (3.5) is gated on the variant's guidance_embeds config. Conditioning stacks load per variant (Mistral via Flux2Pipeline for dev, Qwen3 via Flux2KleinPipeline for Klein) and are encoded and freed before the transformer lands on the device. The fused single-stream to_qkv_mlp_proj joins the attention projections in the LoRA targets; the single-stream out projection stays dense because its to_out suffix would also match the double-stream ModuleList container. Wire both families through the training registry (family set, labels, VRAM notes, rank 16 / lr 1e-4 defaults, bf16-only preflight), mark them trainable with train base repos in the family registry, add FLUX.2-dev to the gated-repo token check, and trust both official bases for training downloads. Verified on B200: 30-step klein int8 (19.6s) and nf4 (20.9s) and dev int8 (52.0s) runs train with finite decreasing loss and the saved adapters apply on the bf16 base pipeline (weight 0 reproduces the base image exactly, weight 1 visibly restyles it). * Support LoRA adapters on torchao int8/fp8 quantized image pipelines Adapters are baked at load time: they attach to the dense transformer, then quantize_ converts only the frozen base linears (the lora_ side path is excluded by name), then the loader compiles. Post-quant PEFT injection is not possible on a manually quantized module, so the prequant shortcut is skipped for a baked load and the memory plan is sized for the dense build (force_dense on the quant candidate). At generation time the baked topology is frozen: weight tweaks and disabling (scale 0 reproduces the quantized base exactly) go through set_adapters, while adding or removing adapters returns a clean 400 telling the client to reload with the new selection. supports_lora now returns True for int8/fp8 diffusers loads (checked before the gguf-kind early return, since the quant fast path keeps the picker kind); nvfp4/mxfp8 and GGUF-via-diffusers stay blocked. The load request model takes an optional loras list, threaded through begin_load on both engines (native ignores it and keeps applying LoRA at generation). Verified end to end on GPU: Z-Image GGUF picker + int8 + trained adapter loads through the API, bake marker logged, weight 1.0 vs 0 renders differ visibly, weight 0.5 accepted live, unknown adapter rejected as 400. Affected suites: 296 passed. * Add FLUX.1 Krea dev to the image model catalog Krea's guidance-distilled FLUX.1-dev finetune keeps the exact dev layout, so it runs under the existing flux.1 family unchanged. Wire it up end to end: - Catalog group with the gated official bf16 pipeline and the open QuantStack GGUF quants; the gated artifact is skipped on auto-routing when undownloaded. - Trust the official repo for non-GGUF from_pretrained loads, next to the other black-forest-labs bases. - Generation defaults: 28 steps at guidance 4.5 per the model card. The generic "krea" defaults key (Krea-2-Turbo's 8-step no-CFG recipe) used to swallow the id, which would have produced garbage output; the new flux.1-krea key precedes it on both the backend table and the images page table. - The flux.1 prequant checkpoints are schnell-based; the loader's baked base_model_id validation refuses them for the Krea-dev base, so int8/fp8 requests dense-quantize instead (covered by existing prequant tests). * Resolve pre-quantized checkpoints per base variant One family entry covers several published variants whose weights differ (flux.1: schnell, dev, Krea-dev), but prequant resolution was keyed on (family, scheme) alone, so only the default base could ever be served: the loader's baked base_model_id validation correctly refused the schnell checkpoint for dev and Krea-dev bases and every such load paid the dense download plus on-the-fly quantise. Add an optional prequant_variant_repos table on DiffusionFamily as (base_repo, scheme, repo_id) triples and thread the resolved base repo through resolve_prequant_source / usable_prequant_source and their three call sites (load fast path, memory-plan probe, auto-policy candidate). A base without its own entry keeps returning the family default, preserving the existing refuse-then-dense behavior exactly. Wire the flux.1 variants: the gate-validated unsloth/FLUX.1-dev-FP8 checkpoints (built in the earlier campaign but never reachable) and the new unsloth/FLUX.1-Krea-dev-FP8. * Add the Lumina Image 2.0 family to the image catalog Alpha-VLLM/Lumina-Image-2.0 is a 2.6B single-stream DiT with a Gemma2-2B encoder and a standard 16-channel VAE, all transformers-4.x-compatible, so the generic from_pretrained pipeline path loads it as a new lumina-2 family: - Family entry (Lumina2Pipeline / Lumina2Transformer2DModel), aliased to lumina-image-2.0 / lumina-image-2 / lumina2. No bare lumina alias: Lumina-Next checkpoints are a different arch and must stay unknown rather than crash mid-load. bf16-only upstream, so the fp16 fallback stays off like z-image. - Trust the official repo for non-GGUF loads; bf16 component table entry (ships fp32, ~5.2 GB transformer + 5.2 GB encoder bf16-resident). - Generation defaults 50 steps / guidance 4.0 per the model card, and the generate call passes the card's cfg_trunc_ratio=0.25 itself (family-gated, signature-gated): the pipeline default (1.0) runs the CFG double-forward on every step and oversaturates output. - Catalog group with the single ungated bf16 pipeline artifact (11 GB resident) plus routing assertions; images page defaults row. - No GGUF artifact: none exists upstream (only finetune/LLM quants), so the dense transformer_quant fast path (GGUF-kind-only) stays unreachable for now. Offline probes of the future prequant campaign: int8 and fp8 both engage and render cleanly (fp8 LPIPS 0.11 vs bf16, int8 0.33 from 50-step trajectory drift with intact quality), so neither scheme is family-denied. * Wire the hosted Lumina Image 2.0 int8/fp8 checkpoints Gate-validated against same-seed bf16 renders (28/28 pairs per scheme, zero failures): int8 LPIPS mean 0.146 / SSIM 0.937, fp8 LPIPS mean 0.116 / SSIM 0.946. Uploaded to unsloth/Lumina-Image-2.0-FP8 following the existing checkpoint repo conventions. * Add the HunyuanImage 2.1 family to the image backend The hunyuanvideo-community diffusers mirror carries the full stack in standard layout: a 17B dual-stream DiT (32.5 GB bf16), a Qwen2.5-VL text encoder, a ByT5 glyph encoder, the 32x HunyuanImage VAE, and guider/ocr_guider components (AdaptiveProjectedMixGuidance) that diffusers 0.39 loads natively, so the generic from_pretrained pipeline path covers everything with no per-component assembly. Family notes: - The call's guidance knob is distilled_guidance_scale (there is no guidance_scale kwarg), so cfg_kwarg routes the UI value there; real CFG runs inside the repo's guider at its baked scale. Defaults follow the card recipe: 50 steps, 3.25. - 2K-native: verified live at both 1024 and 2048. - Coexists with the HunyuanImage-3.0 structured exclusion (3.0 has no diffusers pipeline and stays excluded with its stated reason). - int8/fp8 dense quantization verified live (LPIPS 0.186 both vs same-seed bf16); a short prompt does not trip the int8 torch._int_mm minimum on this arch, so no family exclude entry is needed. - bf16 component table for the memory planner: (32.5, 16.3, 0.8) GB. * Surface HunyuanImage 2.1 in the image model catalog Catalog group with the open bf16 mirror pipeline (~50 GB resident, so a bare click on a consumer card routes to the QuantStack GGUF quants, which load and render through the generic GGUF path, verified live) plus the images page defaults (50 steps, guidance 3.25 feeding distilled_guidance_scale). * Add the HiDream-I1 family to the image backend A 17B MoE DiT (16 double + 32 single layers, 4 routed experts) with four text encoders, on HiDreamImagePipeline (diffusers 0.39). One family covers the open Full / Dev / Fast repos (same arch); per-variant generation defaults follow the upstream inference recipes (Full 50 steps at guidance 5, the distilled Dev 28 and Fast 16 guidance-free). The repos name a Llama-3.1-8B text_encoder_4 in their model_index but do not ship its weights; the official example passes the gated meta-llama repo in by hand. The loader instead assembles the component from the open unsloth mirror (byte-identical weights, already inside the non-GGUF trust gate), injected at the three pipeline from_pretrained sites, with output_hidden_states matching the official example. Memory planning counts the assembled TE4: 34.2 GB DiT + 28.8 GB encoders, ~63 GB bf16-resident. * Surface HiDream I1 in the image model catalog One catalog group with the three official bf16 pipelines (Full, plus the Dev and Fast distillations as labeled artifacts) at their ~63 GB resident size, so auto-routing keeps this a datacenter-GPU pick. city96's GGUF is deliberately not wired: the GGUF path would need the same Llama TE4 assembly for very small demand. Images-page defaults mirror the backend table with the variant keys ahead of the generic hidream key. * Pin the measured HiDream quant verdict in tests int8 and fp8 both engage and render cleanly on this family, including short prompts on int8: the routed MoE expert Linears only ever see the concatenated image+text stream (M far above the torch._int_mm minimum), so no deny entry and no family exclude tokens are warranted. Assert that so a future table edit cannot silently regress the measured behavior. * Wire the hosted HunyuanImage 2.1 int8/fp8 checkpoints Verified bit-identical to on-the-fly quantize: all 1264 state dict tensors (456 quantized) dequantize equal between the loaded checkpoint and a fresh quantize_ pass, so quality matches the runtime Dtype path exactly. Same-seed LPIPS suite means (0.35 int8 / 0.28 fp8) blend trajectory divergence with this family's own run-to-run nondeterminism (identical weights and seed reproduce a 17/255 mean pixel delta through the 50-step guider pipeline); per-case hard checks pass and the drift is compositional, reviewed visually. Uploaded to unsloth/HunyuanImage-2.1-FP8. * Fix silent LoRA drop and wasted transformer prefetch on GGUF quant loads Two live-test findings on the images load path: - transformer_quant with baked LoRAs, when the dense quantized build is declined for memory or fails: the load completed as a plain GGUF with the adapters silently dropped (HTTP success, supports_lora=false after the fact) -- wrong output with no signal. The load now fails with the recovery options (drop the adapters, free VRAM, or pick a smaller model). Weight-0 adapters still count as no bake request, and the plain no-LoRA decline keeps its silent GGUF fallback. - A fresh GGUF load on a small GPU prefetched the base repo's full bf16 transformer shards (~47 GB on Qwen-Image) because the dense-quant prefetch widening only checked scheme viability, not whether the device could ever hold the candidate resident. Gate the widening on total device capacity (reserve + 0.85 margin, the plan_fits_total_capacity bar) so a card that is certain to decline the dense build never pays the download; capable devices keep the prefetch. * Fix video progress under-reporting during load and generate Two live-test findings on the video progress endpoints: - load-progress downloaded_bytes froze mid-download: the counter used scan_cache_dir, which skips in-flight *.incomplete blobs, so it sat at the last completed blob for the whole multi-GB shard pull while the disk kept filling. Count the repo's cache directory directly (completed plus incomplete blobs, snapshot symlinks skipped so nothing is double-counted). - generate-progress reported total_steps=null / fraction=0 while step advanced: the video API only carried the native total field while the image API exposes total_steps and fraction, so one poller could not work against both. Derive the image-compatible aliases in generate_progress and declare them on the response model; the native total stays for back-compat. * Wire the hosted HiDream I1 int8/fp8 checkpoints Gate-validated: all 28 per-case pairs pass per scheme (LPIPS suite means 0.291 int8 / 0.278 fp8, in the 50-step trajectory-divergence band; CLIP delta means 0.007-0.008), and the int8 checkpoint is verified bit-identical to on-the-fly quantize across all 1615 state dict tensors (1073 quantized, max abs diff 0.0). Uploaded to unsloth/HiDream-I1-Full-FP8. * Add a pre-cast text-encoder loader for the layerwise fp8 scheme The runtime text_encoder_quant=fp8 path downloads the full bf16 text encoder and layerwise-casts it in place on every fresh load. For the heavyweight encoders (LTX's Gemma3-27B ~50 GB, FLUX.2-dev's Mistral-24B ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) that download dominates load time on a fresh machine. diffusion_te_prequant.py loads a pre-cast fp8-storage state dict instead: meta-init the encoder skeleton from the checkpoint's te_class, load_state_dict(assign=True), rebuild on CPU if non-persistent buffers stay on meta, then re-apply the same layerwise cast to install the upcast hooks. The cast is a deterministic storage transform, so the loaded encoder is bit-identical to dense-load-then-cast by construction. v1 hosts the layerwise fp8 storage scheme only: its state dict is plain tensors (torch.load(weights_only=True), no pickle execution). The dynamic-compute schemes (fp8_dynamic, int8, nvfp4) build torchao subclass wrappers at runtime and are deliberately not hosted. Checkpoints validate format, scheme, component and base_model_id before use and any problem falls back to the dense download and cast. Local path overrides reuse the DiT prequant allowlist env var. Families opt in via a new te_prequant_repos (scheme, component, repo_id) field on both DiffusionFamily and VideoFamily; the field defaults empty so nothing changes until a gate-validated artifact is wired. * Inject hosted pre-cast text encoders during pipeline assembly Wire te_prequant_pipe_kwargs into the three pipeline assembly sites: the diffusion full-pipeline branch, the diffusion transformer-only and GGUF branch (where the companion TE is the big remaining download), and the shared video assembly path before the pipeline/component split. Injection is gated exactly like the runtime cast (mode normalized to fp8, device supported, family not denied), so it can never engage where quantize_text_encoders would not; the later quantize_text_encoders call re-applies the cast idempotently and keeps status reporting truthful. With no hosted checkpoint configured the call returns {} and assembly loads the dense encoder as before. * Add the pre-cast text-encoder checkpoint builder Applies the runtime layerwise fp8 storage cast to a model's dense text encoder once and saves the cast state dict with baked metadata (format tag, base_model_id, family, scheme, component, te_class, versions) in the layout diffusion_te_prequant.py validates. Resolves the encoder class from the checkpoint's config.architectures so the recorded te_class matches what the pipeline instantiates. CPU-runnable: the cast touches storage dtypes only. * Test the pre-cast text-encoder load path Hermetic CPU coverage for diffusion_te_prequant: the checkpoint filename convention, family-table resolution by scheme and component with malformed entries skipped, resolution priority (path override, hosted repo, none) and the fp8-only scheme gate, the checkpoint validation matrix (wrong format, missing state_dict, wrong scheme, wrong component, wrong or missing base_model_id) with base case folding, the local-path allowlist refusal and missing-file fallback, and the assembly injection gating (mode, hosted entry, device support, family deny, load failure, successful injection). Also pins the te_prequant_repos field on both family dataclasses and that no family ships a hosted TE checkpoint until the campaign wires one. * Fix pre-cast TE checkpoint loading and engagement reporting Two bugs found while building the hosted checkpoints: - The builder recorded torch.__version__ (a TorchVersion object) in the checkpoint metadata, so torch.load(weights_only=True) rejected every artifact and the loader silently fell back to the dense download. Record plain strings. - Re-applying the layerwise fp8 cast to an injected pre-cast encoder raised on the duplicate hook registration, making quantize_text_encoders report the engaged cast as failed (status showed no TE quant while the encoder ran fp8). _cast_fp8 now returns early when the hooks are already installed. Also corrects the LTX TE size note: Gemma3-12B stored fp32 (~49 GB), not 27B. * Wire the hosted pre-cast fp8 text encoders qwen-image and flux.2-dev (diffusion) and ltx-2 (video) now resolve a hosted pre-cast fp8 text encoder from their unsloth -FP8 repos: - unsloth/Qwen-Image-FP8: Qwen2.5-VL-7B, 16.6 GB dense -> 8.8 GB - unsloth/FLUX.2-dev-FP8: Mistral-Small-24B, 48.0 GB dense -> 24.7 GB - unsloth/LTX-2-FP8: Gemma3-12B, 48.7 GB fp32 store -> 13.2 GB Every checkpoint verified bit-identical to dense-load-then-cast (729 / 585 / 1066 tensors, zero mismatches) and smoke-tested through the real backends with the repo engagement marker. Tests cover the wired entries, the resolver filenames, builder metadata weights_only survival, and the idempotent re-cast. * Report the compute dtype on fp8-cast encoders and inject the pre-cast TE on the dense fast path Two more findings from the hosted-TE GPU smokes: - Module.dtype reports the first floating parameter, which after the layerwise fp8 cast is the fp8 STORAGE dtype. Flux2 derives its prompt embed and latent dtypes from encoder.dtype and feeds them to randn_tensor, which has no fp8 kernel, so ANY flux.2 load with text_encoder_quant=fp8 crashed at generation (pre-existing, runtime cast included). The cast now swaps in a subclass whose dtype property reports the compute dtype; forward behaviour is unchanged. - The dense transformer_quant fast path assembles companions through _assemble_pipe, which never received the pre-cast TE injection, so the hosted encoder engaged on full-pipeline and GGUF builds but not on the fast path. Threaded through like the other two branches. Verified live on B200: qwen-image (full pipeline), flux.2-dev (GGUF picker with int8 DiT prequant), ltx-2 (video backend) all engage the hosted TE, render non-black, and report text_encoder_quant=fp8 truthfully. * Key the fp8 cast idempotency on an explicit completion marker Hook presence alone cannot distinguish a legitimately pre-cast text encoder from leftover hooks after a cast that failed mid-pass, so the early return now requires the completion marker _cast_fp8 sets once the hooks are fully installed. Leftover partial state keeps failing closed. Also tolerates non-Module encoder doubles in the hook probe and the dtype override. * Extend the fp8 TE quant to HiDream's Llama text_encoder_4 The generic quantize_text_encoders pass only covers text_encoder.._3, so HiDream's HEAVIEST encoder (Llama-3.1-8B TE4, 16.1 GB bf16) always stayed dense. TE4 is assembled separately (hidream_te4_kwargs), so the fp8 path now lives there: when the requested TE quant is layerwise fp8 and the device/family qualify, TE4 prefers the hosted pre-cast checkpoint (unsloth/HiDream-I1-Full-FP8, 8.6 GB) and falls back to dense-load-then- cast; a mid-pass cast failure reloads a fresh dense encoder instead of shipping partial state. The pre-cast loader and builder gain config_subfolder/config_overrides for standalone encoder repos whose config sits at the root and whose pipeline needs forward flags (output_hidden_states/attentions). Verified on B200: bit-identity 291 tensors (225 fp8, 0 mismatches), hosted checkpoint engages through the real backend (marker + status fp8), load 24.3 s vs 48.0 s dense, LPIPS 0.133 mean over 3 same-seed pairs vs the dense-TE render (gate 0.25), non-black frames. * Correct the ltx-2 resident TE estimate to the bf16 cast size The memory plan's bf16_components_gb held 50.4 GB for the LTX text encoder, which is the fp32 hub store of Gemma3-12B (~49 GB download), not what sits on device: the pipeline loads it torch_dtype=bf16, ~24.4 GB resident. The 26 GB over-estimate pushed the auto plan toward offload on cards that fit the real footprint. Comments and the size-table test now pin the resident semantics. * Host pre-cast fp8 text encoders for four more families Round 2 of the hosted TE set, each bit-identical to dense-load-then-cast and gated through the real backend (marker + status fp8 + same-seed LPIPS vs dense TEs): - FLUX.1 T5-XXL (text_encoder_2): 9.52 -> 5.90 GB, one artifact for schnell/dev/Krea-dev (T5 shards byte-identical across all three, verified sha256). 220 tensors, 144 fp8, LPIPS 0.109. - Lumina Gemma2-2B: fp32 hub store 10.46 -> 3.20 GB (3.3x download cut). 288 tensors, 182 fp8, LPIPS 0.041. - Z-Image Qwen3-4B: 8.04 -> 4.41 GB. 399 tensors, 252 fp8, LPIPS 0.112. NOT shared with flux.2-klein-4B: klein retrained layer 35's MLP (verified tensor diff, maxdiff 0.86), so klein hosts no entry. - Krea-2 Qwen3-VL-4B: 8.88 -> 4.83 GB. 713 tensors, 460 fp8, LPIPS 0.082. The constructor-assembled krea pipeline takes the encoder directly (load_krea2_pipeline text_encoder kwarg); the loader remaps 5.x rope_parameters and re-ties weights after assign so the rebuilt encoder matches the builder's structure. HunyuanImage 2.1 reuses the Qwen-Image artifact outright: its Qwen2.5-VL text encoder is byte-identical (every shard sha256, 16,584,414,544 bytes), recorded in the new component-level base-equivalence table the checkpoint validator consults. The injection loop now covers text_encoder.._3 so a family can host several components. Live check: LPIPS 0.123 vs dense. * Report the fp8-cast compute dtype without swapping the encoder class The dtype override swapped encoder.__class__ to a dynamic subclass, which breaks transformers' kwargs-based output recording: a fp8-cast Qwen3VLModel stopped returning hidden_states and every krea-2 generation with text_encoder_quant=fp8 crashed at encode_prompt (regression from the HiDream TE4 change; caught by the krea hosted-TE live smoke). The override is now a property shadowed on the ORIGINAL class that prefers a per-instance compute-dtype attribute, so class identity is preserved and uncast instances keep the stock behaviour. The idempotency test now pins exact class identity and the uncast-sibling fallback. * Pass the calibrated distilled sigma curve to LTX-2.3 8-step runs The 22B distilled DiT was trained against ltx_core's fixed DISTILLED_SIGMA_VALUES, but the diffusers scheduler derives 8-step spacing from resolution-shifted flow matching and lands far off at every reachable mu (second sigma 0.945-0.981 vs 0.99375, tail 0.37-0.61 -> 0.1 vs 0.725 -> 0.42 -> 0). At the distilled default step count the backend now passes the list verbatim, neutralising the scheduler's dynamic shift and terminal stretch for the call (they distort even explicit sigmas) and restoring them afterwards. Other step counts and the dev/base DiT keep the scheduler's own spacing. Live-verified on B200 through the video branch backend: the scheduler holds the exact curve after an 8-step distilled GGUF generation, config restored, healthy clip. Also reword the transformer_quant resolved reason to the measured reality: quant halves resident weights and hosted checkpoints cut load time, while per-step speed is roughly bf16 parity. * Pin the fp8 weight-quantize kernel against silent MSLK switching torchao's Float8Tensor KernelPreference defaults to AUTO, which switches the weight-quantize kernel to MSLK whenever an mslk package is importable on sm90+. Measured on B200: that changes fp8 scale rounding bitwise (8/8 FLUX matrices differ, scales ~55 percent of bytes), so a box that merely gains mslk would break the hosted-prequant bit-identity invariant; the mslk path is also slower under torch.compile (opaque extern call blocks inductor's quantize fusion, FLUX.1 fp8 e2e 1.149 to 1.624 s). Pin KernelPreference.TORCH explicitly, matching current no-mslk behaviour bit for bit; signature-gated for older torchao. GPU-smoked (finite, rel err 0.037) and pinned by test. * Shift Qwen-Image training sigmas to the inference distribution Qwen-Image's scheduler skips its static shift under use_dynamic_shifting, so the DiT trainer was drawing UNSHIFTED uniform-schedule sigmas for it (mean sigma 0.50) while inference always runs the exponential mu = log 3 shift plus the shift_terminal 0.02 stretch. Add a flow_shift config lever: "auto" (the new qwen-image default) rebuilds the training sigma table through the scheduler's own time_shift and stretch_shift_to_terminal so the draw matches the inference distribution exactly (mean sigma 0.72); a numeric value applies the standard linear shift s*u/(1+(s-1)*u); 1.0 keeps the historical identity behavior and stays the default for FLUX, Z-Image and Krea 2. The model timestep conditioning follows the shifted sigma, gathered in fp32 so bf16 rounding never skews it. Also wire two opt-in levers with off defaults: cfg_dropout (per-sample empty-prompt conditioning dropout, encoded alongside the captions before the text encoders are freed) and weighting_scheme="bell" (bsmntw-style mid-schedule Gaussian loss weighting normalized to mean 1). Verified with two 80-step rank-8 bf16 LoRA runs on Qwen/Qwen-Image (identity vs auto, same seed): both converge with finite decreasing loss and produce coherent same-seed previews. Unit tests cover the exact transform, the shifted sampling distribution, per-family defaults and config plumbing. * Add LoRA EMA, a persistent conditioning cache, and aspect bucketing helpers diffusion_train_extras hosts the opt-in training extras: LoRAEMA shadows only the trainable adapter params (warmup-ramped decay, default 0.99, exported as a second adapter under output_dir/ema), PersistentConditioningCache stores latent posterior stats and caption embeddings as safetensors keyed by content hash + family + resolution, and the aspect-ratio bucketing helpers group mixed-aspect datasets into same-area divisor-snapped shapes. The DiT trainer wires the first two behind config flags that default to the current behavior: ema_decay (0 disables) and cond_cache_dir (None disables). A fully warm cache skips loading the VAE and text encoders entirely; a cache hit is bit-identical to a fresh encode, including the per-channel qwen latent normalization. Also fixes the stale _gather_sigmas call in the perf test that still passed the scheduler instead of the sigma table. * Tighten torchao configs and note the FSDP2 design for the DiT trainer nf4 loads now enable double quantization (~0.4 bits/param off the frozen base scales at no fidelity cost), fp8 training uses the rowwise recipe when the torchao build ships it (per-row scaling confines the DiT activation outliers that a tensor-wide scale collapses), and the inference quant filter gains a per-scheme GEMM-tiling divisibility floor (16 for scaled_mm, 32 for MX blocks) so one ragged Linear cannot crash the first denoise after a clean quantize pass. plans/fsdp2_diffusion_design.md records the multi-GPU design: bf16/fp8 over FSDP2 with per-block units, LoRA attached before sharding, int8 out of scope (DTensor over the quantized subclass is undefined), per-family notes. * Batch diffusion inference with per-image seeds, an inference conditioning cache, and GGUF loader fixes Batched generation: /images/generate takes a prompts list (one image per prompt, txt2img only) or a seeds list (one prompt, one image per seed); the legacy batch_size path derives per-image seeds base..base+n-1 like the native engine. Every image gets its own torch.Generator so any batch member replays alone from its gallery recipe; the whole list runs as one forward by default with OOM backoff that halves a failed chunk, and an explicit batch_size caps images per forward. Validated 10-22x over serial engines on 32-image suites with LPIPS deltas within 0.002. Conditioning cache on the inference path: UNSLOTH_DIFFUSION_COND_CACHE_DIR (the inference sibling of the trainers' cond_cache_dir, same persistent store) wraps encode_prompt so repeated prompts skip the text-encoder forward entirely; verified bit-identical outputs. Bypassed while LoRA adapters are attached; tensor-argument calls pass through uncached. Compile cache: GGUF loads fingerprint their own bundles (quant=gguf, a different compiled graph than the dense family) and batched calls register every distinct (w, h, batch) chunk shape they ran, so the heavy GGUF batched warmups (~159 s at batch 32 on 12B-class, ~655 s on 20B CFG-batched) are paid once ever. GGUF loader: strip the sd.cpp model.diffusion_model. container prefix in the single-file converter; diffusers' FLUX.2 converter KeyErrors on it and the Qwen-Image identity mapping strands the model on meta. * Correct batched seed-replay docs to match measured behavior Same-seed images at the same batch shape are bit-identical; a solo regeneration with the recorded seed matches its batched rendition up to batch-size-dependent kernel numerics (mean abs pixel delta about 2.5/255, LPIPS delta under 0.002), not bit-exactly. The previous wording overclaimed bit-identity across batch shapes. * Note that batched bit-identity assumes a settled compiled graph The first generation issued while the deferred compile is still in flight can deviate transiently (observed once on a cold fp8 build: mean abs pixel delta 0.063/255); once the graph is settled, same-seed same-batch-shape images are bit-identical across runs. * Studio sidebar: Image03/FlimSlate icons, More flyout, Train row, New pills - Images uses Image03Icon and Video uses FlimSlateIcon. - New "More" row (MoreHorizontalIcon) opens a right-side flyout on click or hover holding Video, Recipes and Export; the close is delayed 180ms so the pointer can cross the gap. Its SidebarMenuButton deliberately takes `title` rather than `tooltip`: with `tooltip` the button returns a Tooltip root and DropdownMenuTrigger asChild would hand its ref to a non-DOM node. - Dropped the "Train" section heading; Train is now a top-level row between Images and More. data-tour="navbar" moves to the surviving nav group so the product tour keeps its anchor. - "New" pill beside Images and (inside the flyout) Video, via NavBadge. * Studio sidebar: match flyout rows and New pills to the existing scales - More flyout rows dropped their sidebar-row typography and size-icon override, which fought DropdownMenuItem's own scale (text-sm, gap-2.5, px-3 py-2 and size-4 icons) and rendered oversized glyphs and text next to the nav. - New pill reuses the brand "beta" badge recipe (nav-badge font, --ui-font-scale sizing, nav token colours) rather than hardcoded 9px values. - The More row's native title tooltip (an OS box on hover) is replaced by the app's Tooltip, wrapped around DropdownMenuTrigger so both triggers compose onto the same button, and shown only on the collapsed rail like other nav rows. * Settings: pin and reorder the sidebar navigation Adds a "Sidebar navigation" section to Settings -> Appearance, above the existing profile-menu customizer, with the same drag-to-reorder + switch UI. - New sidebarNav preference: one { id, pinned } entry per navigable row (projects, hub, images, train, video, recipes, export), array order = render order. Defaults match the shipped layout, so an untouched install is unchanged. - Unpinning moves a row into the More flyout rather than hiding it, so no page becomes unreachable. New chat and Search stay fixed as actions. - app-sidebar now renders from one navRows descriptor map, so a pinned row and its flyout counterpart cannot drift; the More row appears only when something is unpinned and highlights off whatever it actually holds. - Mirrored in the backend PersonalizationCustomization: without it the model's extra="ignore" would drop the field, and because sync replaces local state with the server's copy once customization is saved, the user's pin order would reset on the next sync. The validator dedupes and back-fills like sidebarMenu but preserves the client's order, since here order is meaningful. Frontend typecheck, i18n parity and catalog checks pass; 32 personalization tests pass, including a round-trip asserting a reordered list survives a save. * Sidebar customizer: drop the Search row, skip More for a lone item - Search is reached from the top bar, so it is no longer previewed as a fixed sidebar nav row; New chat stays. - More now appears only when it would hold two or more rows. A single unpinned row renders inline in its saved order position instead: a flyout wrapping one item costs a click and earns nothing. The customizer's More preview follows the same threshold. * Sidebar settings: hide a lone unpinned tab, match New chat icon, rename Profile menu - With exactly one tab unpinned, both More and that tab are dropped, so nothing is drawn for it (previously it rendered inline). The page stays reachable by URL. - The customizer's New chat preview uses PencilEdit02Icon, the icon the real row renders; Edit03Icon was a different glyph. - "Sidebar menu" is now "Profile menu", described as the shortcuts behind your name at the bottom of the sidebar, so it no longer reads as a second name for the navigation section above it. * Tighten comments in the new sidebar and delete-guard code * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio sidebar: keep the More row highlighted while its panel is open Moving the pointer into the flyout left the row unhighlighted while the panel stayed open. The row now carries data-menu-open, added to the nav hover selector list. Not data-state: the tooltip and menu triggers both write that attribute, so whichever lands last wins. * Images: use the shared pill toggle for Create/Train and pad the panels - Create/Train was the only segmented control on its own Tabs styling. It now uses PillTabs, the same control as the model picker and Hub toggles, pinned to the header row's 34px. PillTabs takes an icon per tab, so the inline-span workaround for TabsTrigger goes away. - pt-3 on both the Create and Train panels, which sat flush against the model selector row. * Images: make the workflow picker a dropdown instead of a 7-up strip Seven workflows in a 340px rail left ~48px each, so the labels crowded and the hints were only reachable as title tooltips. The strip is now a dropdown: the trigger shows the current workflow and its hint, and each row carries its own description. A row the loaded model can't run is disabled and shows the reason in place of the hint, so the gating explains itself. Adding a workflow no longer shrinks the others. * Images: workflow icons, hint under the trigger, more top room, unclipped Train cards - Each workflow carries an icon, shown on the closed trigger and on every row. - The trigger is one line (icon plus name). The selected workflow's description moved below it, where it reads like the Field hints further down the rail. - pt-6 instead of pt-3 on both Create and Train, so the cards clear the model selector row. - The Train right column scrolls while its cards use ring-1, which draws outside the box and was clipped at the scroll edges. p-px gives the ring room. * Images: one-line workflow rows, stronger trigger fill, roomier mode tabs - Dropdown rows are icon plus name only. The selected row's description already shows under the trigger, and a disabled row keeps its reason as a title. - Trigger fill moves to the bg-foreground/[0.07] dark:bg-foreground/[0.12] pair the hub cards use, so it reads against the card in both modes. - Description under the trigger goes from text-ui-10 to text-ui-11p5. - More horizontal padding on the Create / Train tabs. * Images: drop card borders for the composer shadow, keep scrollbars inside, use app controls in Train - Cards lose ring-1 for .panel-soft-surface: the composer's shadow in light, flat in dark, matching .chat-composer-surface and the menus. - Both rails now clip (overflow-hidden) with the scroller inside, so the scrollbar can't ride over the rounded corner. Same shape video-page already uses. - Train's 9 native selects become the app Select, so they no longer open an OS-native menu, and the native file input is hidden behind a Choose images button that reports the count. - Image previews use explicit 8-10px radii: this theme sets --radius to 1.1rem, so rounded-md was 15.6px and the thumbnails read as circles. * Images: one card for controls and preview, chat sliders, wider softer shadow - Controls and preview were two floating cards; they now share one card split by a divider. The Advanced dock stays separate since it toggles. - SliderField wraps Chat's ParamSlider, so the sliders match Chat (label row with the value, full-width neutral track) instead of a green track with a spin box. All 14 call sites keep their props. - panel-soft-surface goes from 0 2px 8px -2px /0.16 to 0 4px 22px -6px /0.10: lighter, spread wider. * Images: flat Create and Train panes, hover-only scrollbars, tidier Train dataset step Both Images tabs now sit on the page background like the Hub: no card, no shadow, no bounding box. A single rule divides the controls rail from the preview canvas (Create) and from the run area (Train), and the settings and previous-runs sections read as panes rather than nested cards. Also: - Scrollbars in these panes use the existing hover-scrollbar recipe, so the thumb only shows while the pane is hovered. - Workflow rows explain themselves with a tooltip after a short hover, which also works on disabled rows, and the descriptions are much shorter. - Training images rows are name plus image count; the license stays on the example card. - The upload step loses its dashed box, the buttons match the sizes around them, and Upload only appears once files are picked. - The empty preview uses the same icon as the Images nav item. * Images: full-height panes, wider settings rail, Create/Train offset from the selector The rule between the panes now runs the whole page height (the row drops its bottom padding and each pane pads its own content), the settings rail is wider on both Create and Train, and the Create/Train switch sits further right of the model selector. * Images: put both tabs on the Hub's centered measure Top bar and content now share mx-auto max-w-1100 with px-5 / sm:px-8, so Create and Train sit at the same width and position as the Hub instead of running edge to edge. * Images: restore the top bar position, drop the panes lower under it * Images: center the mode switch, flip the arrow with the orientation, app tooltips everywhere The Create/Train switch is centered on the page instead of trailing the model selector, with wider buttons. The flip control's arrows now rotate with the orientation and its label says which way the flip goes. Every native title tooltip on the page is now the app's tooltip, so they all get the rounded surface instead of the OS box. * Images Train: plainer field text, no green buttons, columns that stop colliding - The dataset name, trigger prompt, adapter name and custom base fields now say what they are in plain words instead of leaning on example values. - Import, Upload, Back, Back to settings and Train another are outline buttons, not green ones. - Example thumbnails are landscape tiles, so photos are not cropped to chunky squares. - Settings cells get min-w-0 and the select value truncates, so a long option like the nf4 label no longer widens its column into the next one. - The number stepper sits a little further in from the field edge. - Create and Train are wider. * Images Train: roomier example cards with Import on the thumbnail row * Video: same treatment as the Images tabs - No cards: the rail and the canvas sit on the page background, divided by a rule that runs the full page height, on the Hub's centered measure. - Wider rail, chat's sliders, hover-only scrollbars. - Every native title tooltip is now the app's tooltip, including the clip cards. - Reapply and Cancel are outline buttons, the empty state uses the Video nav icon, and the clip tiles are less rounded. * Images and Video: narrower generation rail, matching Train headings Create and Video rails go from 392px to 368px. Train a LoRA and Training settings are now the same size and both in the heading font: the h2 already picks it up from the base rule, so the settings header opts in with font-heading and the weight that rule pins. * Images Train: shorter copy throughout Family notes, example descriptions, precision labels and every helper line are trimmed so they stop wrapping to three lines and colliding with the next column. The nf4 label now fits its select without truncating. * Images Train: a little more spacing between field groups * Images and Video: tighten code comments * Fix training start NameError, the load-order guard test and CPU-only diffusion tests - start_training forwards resume_source_run_id to _start_training_impl, which reads it. Without it every start raised NameError. - Restore main's anchor in the load-marker order test: the file now has an earlier `if config.is_gguf:`, so indexing the first one compared the wrong branch. - The two diffusion tests that reach diffusers now skip when it is absent, matching the CPU repo-test env. - The UI smoke finds nav rows that live in the sidebar's More flyout. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Treat a null metadata caption as no caption str(None) stored the literal "None" as the caption, so a null row counted as captioned and would have trained on that text. Also drops an unused import. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix invalid-UTF-8 500s, the flat Canny map and the dropped DiT knobs read_text raises UnicodeDecodeError, which is not an OSError, so one bad caption sidecar or video sidecar 500d the info, upload and gallery routes. A flat image now yields the all-black edge map instead of its own luminance, and the four DiT loss knobs the trainer implements are declared so model_dump keeps them. * Use the ui font-size tokens instead of raw px text utilities text-[11px] and friends ignore the UI font size preference, which the repo's font-scale contract test enforces. Same rendered size at the default scale. * Fix diffusion dataset 500s, the dropout-1.0 no-op run and the reset base pick Four correctness fixes on the training side: - The labeling grid read caption sidecars under except OSError, but a non-UTF-8 sidecar raises UnicodeDecodeError (a ValueError), so one bad file 500d /diffusion/dataset/{name}/images and the grid could not be opened to repair it. Read it as no caption, matching the info summary. - An image past Pillow's own hard limit raises DecompressionBombError, which derives straight from Exception and so escaped the upload guard's (OSError, UnidentifiedImageError, ValueError) and returned 500 instead of the intended 400. - lora_dropout accepted 1.0, which makes PEFT build nn.Dropout(p=1.0): lora_A and lora_B receive no gradient and the run saves an untrained adapter while reporting normal progress. Bound it below 1.0, matching the LLM request schema. - The train panel re-seeded the base repo on every dataset refresh because the family object identity changes on each info fetch, so an upload or caption save silently replaced the user's chosen base and the run started on a different model. Track the pick and only re-seed on a real family change. * Show the retained failure when a video page mounts after a failed job Mount-time recovery handled only phase=completed, so reloading the page after a multi-minute generation failed left an idle view with no diagnosis: the backend keeps the terminal failed record only until the next job, and nothing else survives the reload. Surface it the same way the poll does, filtering the cancelled sentinel. * Fix batched generation crashes, cache keying and unreplayable recipes Four bugs in the batched inference path, all found by review: - A mixed-prompt batch sent a scalar negative prompt against a prompt list. Z-Image asserts on the length, and Qwen-Image, Krea 2 and FLUX true-CFG encode a batch-1 negative against batch-N latents and fail in the transformer's text/image concat. Broadcast it to match the batch. - The FBCache step-cache reset sat above the chunk loop. diffusers only resets that state at the end of a successful call, so a forward that raised (the OOM the backoff is meant to recover) left its own residual behind and the halved retry died on a shape mismatch. Reset before every forward instead. - The conditioning cache keyed on the checkpoint alone, but a GGUF or single-file load takes its text encoders from the companion base, so the same checkpoint against a different base reused the previous base's embeddings. Key the base too. - Gallery records stored the base seed and the requested batch size even when a prompts/seeds list drove the run, so restoring the second image of seeds=[5, 99] replayed seed 5. List-driven outputs now record as single-image recipes on their own seed. Also bound strength above 0: every img2img pipeline derives its step count from it, so 0 leaves zero denoising steps and either raises or, on SDXL, crashes on empty latents. * Fix quantized-load LoRA bake, prequant family exclusions and outpaint canvas Six review findings across the Images page and model scanning: - The quantized (int8/fp8) load path can only attach LoRA adapters before quantization, but the frontend load request had no loras field, so every generation after such a load was rejected and each reload repeated it. Send the selection with the load. - build_prequant_checkpoint passed no family to the scheme exclusions while recording the family in metadata, so a Qwen int8 artifact baked the short-M text-stream linears and was then rejected wholesale by the loader's family-keyed check. - Registering a bare single-file checkpoint directory produced no On Device row even though the images loader can load it; only its parent worked. Admit that shape when nothing else matched. - Unload left the Reapply target set, so the repair path was skipped and Reapply reloaded the ejected model. Clear it, as the video page does. - Both FLUX.2 bases were trusted for training but not inference, so Deploy to Create rejected every FLUX.2 adapter. - Outpaint allocated the grown canvas before downscaling, exceeding the browser canvas area cap on a large photo; an over-cap canvas is unusable, so Extend silently posted a fully transparent image and mask. Scale the source first. * Send the picked GGUF filename with the quant so diffusion loads fire The variant expander emitted only the quant label, and nothing else in the frontend set ggufFilename, so the Images and Video pages could never take their GGUF branch: both gate it on meta.ggufVariant and meta.ggufFilename, then fall through to the single-file path, which returns because the id is a repo id and not a .gguf name. Every quant pick was a silent dead click, with no load request reaching the backend. The filename was already on the variant row (the picker keys its list on it, and the variant validator requires a non-empty string), so thread it through the click handler. The chat path is unaffected: it reads ggufVariant and never needed the filename. * Version the conditioning cache key and reject non-finite flow_shift Two correctness fixes: - The cache keyed the checkpoint and its companion base by name only, so a Hub repo advancing to a new commit, or a local directory updated in place, kept returning embeddings from the previous text encoder. Pair both with a revision marker: the locally resolved commit sha for a Hub repo, config plus text-encoder file stats for a directory. Neither loads the encoders, so a warm run still keeps them off the GPU. - flow_shift only checked positivity, but JSON accepts 1e309, which floats to inf, and inf <= 0 is False while NaN fails every comparison. The sigma table then evaluates s * u / (1 + (s - 1) * u) as NaN, which poisons every sampled sigma and saves a corrupted adapter while progress looks normal. Require a finite value. * Keep curated models listed, guard the video companion repo, pin diffusers Three review findings: - The picker filtered every catalog member out of Recommended and Hub search on the way to canonical group rows, but nothing renders those rows yet (catalogGroupFitsDevice and groupMatchesQuery are imported and unused). A task-scoped picker's models list is catalogToModelOptions(), i.e. group members exclusively, so both lists came back empty and no curated model could be discovered or downloaded. Keep the artifacts listed until the grouped UI exists. - The video delete guard compared only repo_id, so deleting the companion base of a loaded GGUF video model was allowed even though it supplies the VAE and text encoders. Compare base_repo too, matching what the images guard already does for its companions. - diffusers was declared unversioned while the diffusion stack requires 0.39 (Krea2Pipeline, the cache_context child registries, the Flux2 and Z-Image pipelines), so an upgrade could keep an older release and selecting an advertised model failed until the user upgraded by hand. * Namespace the trainer conditioning cache per checkpoint, bound the learning rate - The trainer keyed its persistent conditioning cache on family and resolution only, while the keys themselves carry just the caption or image content and crop variant. One cache directory reused for two checkpoints, or for the same repo at a new revision, let a warm run skip loading its encoders and train on the other model's embeddings and latent statistics. Namespace on the base checkpoint and its resolved revision as well. The revision helper now lives beside the cache in diffusion_train_extras and the inference wrapper delegates to it, so the two cannot disagree about what counts as the same source. - The diffusion learning rate only checked positivity, but 1e309 floats to inf and satisfies gt, so the route evicted the resident models and started AdamW with an infinite rate: the first step destroys the adapter while progress looks normal and the result is saved. Bound it below 1.0, matching the LLM schema, which rejects inf for the same reason. * Fix GGUF image model picks doing nothing, and pick the train base in the top bar The quant rows never forwarded the .gguf filename, so every hub GGUF pick on Images/Video fell through to a silent return. On Train the top bar now picks the training base instead of a generation model, which is GGUF-only and untrainable. * Pin diffusion and video loads to the live HF cache root Both read huggingface_hub's import-time HF_HUB_CACHE, which changing the cache folder does not update: progress counted the old root while the download wrote to the new one, and from_pretrained could split one model across both. * Add the diffusion download plan endpoint Reports the repos and exact files a pick needs so the download manager can stage them with the loader's own file scope. A plain snapshot would add the packaged root single, transformer shards and fp16 twins the loader never opens. * Add a file-scoped flavour to the Hub download job Lets a consumer that reads a deliberate subset of a repo stage it through the normal download manager. Keyed as "@scope" so it never collides with a quant or with the repo's full snapshot, and the file list rides the registry so an XET to HTTP retry respawns the same scoped job. * Stage image and video downloads through the Hub download manager They downloaded inline inside the load, so they had none of the manager's disk preflight, manifest verification, resume or panel progress. Picks now stage as scoped jobs carrying the loader's own file list, then load from a warm cache. * Fetch staged GGUF checkpoints as scoped jobs, and stop calling diffusion models unsupported A GGUF entry went out as a full snapshot, whose ignore list drops *.gguf: the job finished at once having fetched only docs, and the repo landed on device unloadable. Every entry is scoped now. The Hub also no longer tags image/video models as unsupported (they run on their own pages), and those pickers name what they select. * Apply the picker task filter to local model sections LM Studio, ./models and custom-folder rows ignored it, so the Images picker listed chat GGUFs that 400 on a diffusion load. The backend already tags every local model with a task for this purpose. * Route a chat pick of a diffusion model to the Images or Video page Chat cannot load one, so it was either hidden or failed on load. The unfiltered picker now lists on-device diffusion models and navigates to the page that runs them, passing the repo and quant so that page loads it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Route the real GGUF filename, keep non-GGUF curated models, key scoped downloads by file set Five review findings, four of them ways a click did nothing or fetched the wrong thing: - A chat pick of a diffusion model routed ggufVariant (a label like Q4_K_M) in the search param the target page uses verbatim as the GGUF filename, so the load asked for a file that does not exist. Route ggufFilename; no filename means a curated non-GGUF pick, loaded as a pipeline. - The task-scoped pickers kept only GGUF repos, so the catalog's bf16, bnb-4bit and single-file fp8 artifacts could not be discovered or downloaded on the Images and Video pages even though loadSpecFor knows how to load them. Keep curated artifacts whatever their format, in Recommended and in Hub search. - Both pages deduplicated routed selections on the model alone, and they now stay mounted, so picking the same repo again -- another quant, or the same one after chat evicted it -- returned early without loading or clearing the query string. Key on model and quant. - Every scoped image download shared one @diffusion job key regardless of the requested files, so switching quant mid-download adopted the running job: the UI waited on the first file set, then loaded a file that was never fetched. Include a digest of the file set in the key. - A scoped plan silently dropped requested files missing from Hub metadata, and snapshot_download succeeds when an allow pattern matches nothing, so the job reported completion and triggered a load with required files absent. Fail the job instead. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the scoped download key derivable, and stop the hidden page hijacking a route Four review findings, the first a regression from my own last commit: - Keying scoped download jobs by a digest of the file set broke the download manager: it builds that key client-side (it polls and cancels before any response tells it a key), so it watched and cancelled a key no worker owned and never fired its ready callback. Keep the derivable "@scope" key and refuse the second request instead when a live job on the slot is fetching a different file set -- decided inside the registry claim, under the lock, so a concurrent claim cannot slip past it. The manager records the file set on the job as well, so a sibling quant's transfer is not adopted locally either. - Both diffusion pages read the route query through a loose useSearch and both stay mounted once visited, so the hidden one consumed the other's ?model=: it navigated back to its own route and tried to load, say, an image checkpoint as a video model. Only the visible page consumes it. - The staged download plan was built without the configured HF token or the Advanced values the load itself sends. The token matters most: the backend's Hub metadata lookup is best-effort, so a gated base silently planned no companion entry and the load pulled those multi-GB files inline, outside the manager. The memory/quant controls decide whether the base transformer/ shards are needed at all, and the route dropped memory_mode, cpu_offload, the prequant path and the LoRA selection before asking for the plan. - The video preview kept playing after leaving the page: the keep-alive layout only hides it, and display:none does not pause a media element, so a clip the user unmuted kept its audio going over the next page. Pause on the active transition and do not auto-replay while hidden. Also completes the hand-built request bodies in the hub download tests: the scoped-files field this branch added to the route read as an AttributeError against them, failing five tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Serialize the GPU handoffs, gate DiT training on a GPU, and keep 3.9 installable Six review findings, three of them evict-then-fail orderings: - The chat load reclaimed the GPU without telling the arbiter it existed. A chat load holds no llama-server process until its GGUF has downloaded, which is minutes, so a competing Images/Video acquire in that window found nothing to cancel, took the GPU, and the chat load then spawned onto the same device. It now registers an in-flight marker through acquire_for's register hook (under the arbiter lock, as the image and video loads do), the evictor cancels a marked load, and the route undoes itself if ownership moved while it loaded. - The Hub-download conflict check ran after that handoff, so a GGUF the download manager already owns destroyed the resident Images/Video pipeline and then 409'd, having loaded nothing. It moves above the handoff, together with the marker it handshakes with. - The image load released the engine router's transition lock before registering the load, so a second load choosing the other engine could unload the still-idle engine this one captured; the load then landed on a deactivated engine, where generate, status, unload and the arbiter's evictor can no longer reach it. Registration now happens under that lock and refuses if the engine changed. - Training a DiT family on a host with no GPU was accepted: nf4 is not a CPU fallback, its 4-bit load goes through bitsandbytes, which requires CUDA, XPU or MPS. The start unloaded the working Images pipeline, pulled the text encoders, and only then died in the child. Rejected before the teardown now, and /info stops advertising a precision that always 400s. SDXL keeps its documented fp32-on-CPU path. - Both diffusion pages kept the routed-pick marker forever, so re-picking the same checkpoint (after chat evicted it) neither loaded nor cleared the query string. The marker is released once the query is gone. The Images key also carried a stray NUL byte, which made the file read as binary to grep and other tooling. - diffusers dropped Python 3.9 in 0.38, so the unconditional >=0.39.0 pin left pip no candidate at all on 3.9 and made every install that composes the huggingface extras unresolvable there. The floor is conditional now. Also fixes tests that were already red on the branch: two hand-built request fakes had gone stale against fields this branch added, and the handoff-ordering test only failed on a host with fewer than two GPUs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the GGUF variant contract test against the merged handler signature The assertion pinned the exact single-line call handleVariantClick(v.quant, v.downloaded, expectedBytes, v.filename), but the handler takes (quant, filename, downloaded, sizeBytes) and prettier wraps the call across lines, so the mandatory repository test job failed on every push. Match the call structurally and assert the filename really is forwarded in the handler's argument order. * Stop a background page and a stale record taking the GPU or a download with them Five fixes from a review pass over the diffusion work. delete-finetuned rmtree'd a model the Images or Video engine was holding: every guard on that route is chat-only, and Images loads any local path, so deleting a local diffusion model under the storage root pulled the weights (and the companion VAE / text encoders sd.cpp re-reads each generation) out from under a live pipeline. The cached-model route already refuses this; the trained/exported one now does too, matching by path rather than repo id, and failing open on a chat-only install so it cannot block ordinary deletes. A staged download finishing while its page was hidden loaded the model and evicted whatever the user was actually using: both diffusion pages stay mounted behind the router and a load takes the GPU unconditionally. The pick is now held until its page is on screen again, which is also what chat does. A scoped download could report success having fetched nothing. With Hugging Face metadata unavailable no manifest is written, so verification is a no-op, and snapshot_download returns an existing snapshot folder without downloading when its own repo_info call fails. A repo already on disk from a full snapshot job (which ignores *.gguf) therefore completed with no weights and auto-loaded against them. The requested file list needs no network, so it is checked against the disk directly. The XET to HTTP retry reclaimed the job slot without the scoped file list, and that claim overwrites the stored record, so a later identical scoped start compared an empty list against the real one and 409'd instead of adopting the running download. The DiT accelerator gate probed torch.mps.is_available(), which only exists from torch 2.5 while the supported floor is 2.4. All three probes shared one try/except, so on torch 2.4 the AttributeError read as 'no block' and a CPU-only host still evicted the resident pipeline, downloaded the encoders and died in the child. Each accelerator is probed on its own now, through torch.backends.mps. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stage what an LTX-2.3 load reads, keep a routed file's load kind, drop an unbakeable LoRA Three from the latest review. The video download plan always asked for the wide base file list, so an LTX-2.3 pick staged the 2.0 base's VAEs, vocoder and connectors that the checkpoint supplies itself, while the companion files the 2.3 assembly does read were left out of the plan and pulled inline at load, outside the panel's progress, cancel and disk preflight. The plan now recognises a 2.3 pick by name (the load keeps the authoritative header probe, and under-guessing only falls back to the load-time pull), narrows the base list, and stages the extras in the same entry as the checkpoint so one repo stays one scoped job. A pick routed from the chat picker arrives as ?model= and ?quant= with no picker metadata, so a bare local .gguf or .safetensors was loaded as a pipeline: an explicit model_kind wins over the backend's filename sniffing, so it evicted the resident model and then failed on the missing model_index.json. Both pages now derive the load kind from the path, the same way their own picker handlers do. A torchao int8/fp8 build takes adapters only at load time. Switching artifact inside one family keeps the LoRA selection, since the family did not change, but the load did not bake it, so the next generation was rejected with 'reload the model with the adapter selection' while the picker still showed the adapter as active. The selection is now dropped once per resident build, with a message saying to pick and load again. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cancel an evicted safetensors load, spare the arbiter for CPU-only chat, fetch clips lazily Four fixes from the latest review round: - The GPU arbiter's chat evictor only cancelled the llama.cpp side. The orchestrator publishes active_model_name once its worker reports success, so an in-flight safetensors load was visible only as an entry in loading_models and finished onto the GPU after ownership had transferred. Cancel every pending load, and give the safetensors branch the post-load ownership recheck the GGUF branch already had. - A manual gpu_layers=0 GGUF load runs on the CPU with the GPUs hidden from the child, yet it took the arbiter unconditionally: it cancelled a running image or video generation for a model needing no VRAM, then held CHAT ownership so the next GPU workload unloaded it for nothing. Gate the acquire on the same predicate the launch-time CPU-only mask uses, as the image and video loaders gate on their resolved device. - The staged-download hook subscribes per repo, not per job, so another job on the same repo advanced the staged queue (starting a load whose scoped files were still downloading) or wiped a queue that was still running. Compare the variant each callback carries, like the chat page's auto-load does. - The video gallery fetched every record of a page into an object URL that lives until the page closes: 50 clips at tens to hundreds of MB each, for cards the user may never scroll to. Fetch a clip as its card nears the strip's edge, plus the selected one the player needs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim the comments across the diffusion backend Comment-only pass over the Python this PR touches: drop what the code already says, collapse multi-line explanations that still read on one line, and keep the reasoning that is not recoverable from the code. No code, docstring semantics or behaviour changes; verified with an AST comparison against the previous revision, and the backend suite is unchanged (same 37 environment failures as before: the API integration tests that need a live keyed server, the flash-attn install hooks, and the GPU memory fields). * Invalidate latents on a VAE swap, keep a cut-off generation, surface the EMA adapter - source_revision() scanned the checkpoint root plus text_encoder/tokenizer but not vae, so swapping or fine-tuning the VAE in place left the conditioning cache namespace unchanged and a warm run trained against latents from the old checkpoint. Include the vae directory, like any other component the cached tensors come from. - /images/generate answers only when the images are saved, and secure mode's tunnel caps an origin response near 100 seconds, which a native CPU or a high-step run passes routinely. The page reported failure while the work kept running, and a retry would duplicate it. A lost response (fetch rejection or a gateway status the origin never answered) is now told apart from a refusal: the page waits out generate-progress and reloads the gallery, so the run it started still lands. - The trainer emits the EMA adapter's path with the terminal event, but the state update dropped it, so neither the run history nor either response schema carried it and an enabled EMA left nothing discoverable. Keep it, and show it next to the primary adapter. - weighting_scheme advertised a choice of timestep sampling; sampling is always logit-normal and the flag only selects the bell loss weights. Describe what it does. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Hide unloadable cached rows, hold the dataset interlock, bound a GIF export - The cached-model listing tagged any repo with a model_index.json as text-to-image, so a community pipeline the image loader's trust rule refuses still got a row in the Images picker, and a detected-but-untrusted video repo fell through to that same tag. Gate the image tag on the load path's rule and hide an untrusted video repo outright. - A routed diffusion pick only carries a GGUF filename, which is all the chat picker has, so a curated single-file artifact arrived with no quant and was loaded as a pipeline: from_pretrained on a repo with no model_index.json. Pass the page's own catalog spec into the route pick, so a routed pick resolves to exactly what a direct pick on that page resolves to. - The dataset mutation endpoints checked is_active() and only then handed their filesystem work to a thread, so a start reserving in that gap changed captions or removed images underneath the preflight or the running trainer. The interlock is now registered for the whole request under the lock reserve() uses, and a start refuses while a mutation is open rather than waiting on it. - GIF export held every kept frame as a paletted image before encoding; a clip may be 2048x2048 for 1024 frames, and at the 12 fps target the step is 1, so one export click could allocate over 4 GB and take the backend down. Downscale past 720 px and widen the step to keep at most 300 frames. - seed accepted any Python int, so an out-of-range one passed every preflight, evicted the resident models, spawned the trainer and only then died in torch.manual_seed. Bound it to torch's 64-bit range in the request and config. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the accelerator probes in the DiT family-metadata tests Six tests read family_train_infos() (or a start preflight) without pinning the host probes, so they only held on a machine with a bf16 accelerator: on a GPU-less runner the DiT gate empties precision_modes, turns supports_compile off, and replaces any other preflight message with the no-accelerator note, and all six failed there. A conftest fixture pins both probes for exactly those tests, so they assert the family metadata they are about on every host. The gate's own CPU-only behaviour keeps its dedicated tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Carry the pipeline task into the hub inventory the pickers read - The task-scoped pickers filter On Device rows on a task, and the chat picker routes a diffusion pick by the same field, but those rows come from the /api/hub inventory, which never carried one: the Images and Video pickers listed nothing on device and the routing never fired. Both cached scans and the local listing now tag rows with the classifiers the models API already uses, the schemas and the frontend adapter carry it through, and a row the backend classified as a generation task is exempt from the chat-only guard that was also dropping it. - The local routing map was keyed by model_id while the row click passes id (a filesystem load id for a models_dir or LM Studio entry), so the lookup missed and the pick fell through to the chat loader. Key both. - A staged download whose start answered "error" left its head in place, where the effect never re-runs and onReady never fires, so the pick was stranded until the user reselected. Clear the queue and say so. - Every scoped pick in a repo shares the @diffusion variant, so the variant alone cannot tell two file sets apart: restaging while the first job finished let its completion pass for the new pick and load a checkpoint that had not downloaded. Bind the callbacks to the repo + file set they started, and to the staging generation. - A rejected generate POST does not say whether it reached the backend, so an immediately idle progress read was ambiguous and a submission that never landed looked like a finished image. Require evidence: progress seen active, or a gallery record that was not there before the POST. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the OpenAI image URL fetchable, keep WebM audio, stream example imports Four review items on the diffusion Studio work: - response_format=url returned the bearer-gated gallery route, which a standard image client downloads with no Authorization header, so the default response format was unusable. Mint a short-lived HMAC link instead (the shape RAG already uses for pdf.js) served by a signed route, and leave the gallery route itself bearer-only. - A manual gpu_layers=0 load carrying speculative_type="off" -- a value the UI persists and sends -- read as GPU-bearing, so it took the GPU arbiter and evicted a resident image/video pipeline even though the launcher hides the GPUs for it. Canonicalize the mode and exempt "off". - The curated example import prepared the whole split before the loop stopped at the 10-100 image cap; m1guelpf/nouns is 49,859 rows / 328 MB. Stream instead, with the prepared load kept as a fallback for a repo that cannot stream. - WebM export dropped the audio track an LTX-2 clip carries, silently, on the format offered for web embeds. Mux it as Opus through a resampler + FIFO, and keep exporting the video alone on a build without libopus. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not stub out triton on a GPU host when the Xet backend fails to import The lazy loader retries `import unsloth_zoo.hf_xet_fallback` under UNSLOTH_ZOO_DISABLE_GPU_INIT=1 whenever the first attempt raises. That flag makes unsloth_zoo take its MLX/CPU path, which injects triton and bitsandbytes STUBS into sys.modules for the rest of the process. On a working GPU box whose first import failed for an unrelated reason (a bitsandbytes/CUDA mismatch, say) the retry succeeds, so Studio boots looking healthy and then dies at the first CUDA-only kernel: a GGUF or compiled diffusion generation hits the stub and returns NotImplementedError: Unsloth: 'triton.tools.experimental_descriptor.enable_in_pytorch' was called on Apple Silicon / MLX, where triton is stubbed out. so every image generation 500s with an Apple-Silicon message on a Linux CUDA host, while the load reports success. Found by loading Z-Image-Turbo GGUF through the API on a box where bitsandbytes could not initialise. Gate the retry on the host genuinely having no accelerator. The Xet stall watchdog is optional and already degrades with a warning; a process whose triton is stubbed out is not recoverable. The warning now says why it did not retry. * Fix the lost-generation proof set, the settle timeout and the hub inventory's diffusion gates Seven fixes from the latest review round on the Images page and the hub cache inventory. Images page: - The lost-POST settle path built its "already seen" gallery id set inside the catch, after the request failed. By then the earlier runs of the same batch had already prepended their records, so run 2 could accept run 1's image as proof that its own request reached the backend. The set is now captured once before the first POST and grows with every record the batch produces. - settleLostGeneration fell out of its SETTLE_MAX_MS loop and returned normally, so a wedged generation was counted as done and the next run started against a busy backend. It now throws on timeout. - Restoring a recipe cleared the ControlNet selection but left the workflow tab and the init / mask / reference images pointing at whatever was loaded, so the next Generate conditioned on an unrelated image. It now clears all of them and returns to Create. - The download plan omitted the adapter selection the load itself bakes in. A baked LoRA forces the dense build path, so the plan described a different file set than the load that followed and the rest was pulled inline, outside the download manager. Both now derive the list from one helper. Hub cache inventory: - A download for a repo an Images or Video load is staging was allowed to start: only the llama.cpp loader was consulted. Both diffusion backends already expose loading_repo_ids for the delete guard, and the download guard now reads them too. - A companion-only prefetch (pipeline manifest plus VAE and text encoder, no transformer) passed the snapshot-partial check, since every file its manifest expected did arrive, and was advertised as on-device although from_pretrained cannot load it. - The single-file flag never reached the picker through the hub inventory path, so a checkpoint-only diffusion repo read as a full pipeline and failed after the handoff. The two pipeline-shape helpers now live in hub/utils/inventory_scan.py so /api/models/cached and the hub inventory classify the same repos the same way. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop adopting an unknown scoped download, leaking raced blobs and resurrecting deleted clips Three items from the latest review round. A scoped download job carries a deliberate file subset, and every file set of one repo rides the same "@scope" slot. A client that adopts a live job from the backend had no file list to compare against: the active-downloads response never carried one, so an adopted job's set was unknown and any later scoped request for the same repo read as "already started". Selecting a different checkpoint then waited on the wrong transfer and tried to load a file nobody fetched. The response now publishes the scoped file list, adoption records it, and an unknown set no longer satisfies a scoped request. A gallery record can be deleted while its blob is still downloading. The delete revokes the URL present at that moment, so the fetch that lands afterwards inserted a fresh object URL for a record no card renders and nothing can revoke: a full MP4, tens to hundreds of MB, pinned for the rest of the session, and once per raced fetch. Both galleries now discard a blob whose record went away, with an epoch covering the video page's Clear all. The video backend keeps the last completed job until the next one starts, and the Video page merges that record on mount to cover a job that finished after the gallery fetch. Deleting the clip left the record in place, so every reload prepended a ghost card whose file request 404s until another generation replaced it. Deleting the clip, or clearing the gallery, now clears the matching terminal record, and the page skips a record it deleted itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Resolve revisions from the live cache, serialize dataset imports, and stop pinning every gallery blob Five more items from the review round. The conditioning-cache revision marker read huggingface_hub's import-time HF_HUB_CACHE constant. Studio can move its cache during a session and loading follows the live setting, so after a move the marker went unresolved (or pointed into the previous root) and pulling a new revision of the same checkpoint no longer invalidated the cache: a warm run could reuse the old encoder's embeddings and the old VAE's latents. It now looks in the active Studio cache first and keeps the environment and the library constant as fallbacks, which the trainer subprocess still needs. The dataset interlock counts mutations rather than excluding them, so two imports of different examples into the same empty name both got past the emptiness check. The winner promoted its staging directory atomically; the loser found the folder non-empty, fell back to a per-file move, and merged its images and captions into the winner's dataset. Imports now take a per-folder lock, a second one is refused with 409, and the emptiness check is repeated under the lock. On Windows the sd.cpp asset resolver filtered only by accelerator token, so a Windows arm64 host matched an x64 zip, downloaded and installed it, and failed later when the binary would not run. It now filters by architecture the way the Darwin and Linux branches do. Every gallery page fetched every PNG up front and kept the object URL for the session, so scrolling a large gallery grew memory without bound for tiles the user may never look at. The Images strip now fetches a tile as it nears view, like the Video strip, and keeps the eager path only where IntersectionObserver is unavailable. A 503 carrying a JSON body comes from the application, not a proxy, so it is surfaced as the error it is instead of entering lost-response settlement and being reported as a request that never reached the server. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the class of a failed generation instead of a bare "Image generation failed." Found on a macOS runner: the native renderer aborts inside its own text encoder there, and the page showed only "Image generation failed." with the sd-server backtrace left in the server log, so nothing about the failure reached the user. The failure is now classified into fixed text, out of memory and native-process death, so the message says what happened and what to try. None of the engine's own output is echoed, since a native tail carries local paths and argv; that stays in the log, and an unrecognised failure keeps the original literal. * Treat an undecodable caption sidecar as the tombstone the trainer sees Uploads store .txt and .caption sidecars as raw bytes, so one can hold invalid UTF-8. The trainer treats any existing sidecar, decodable or not, as an empty tombstone and never falls back to the metadata row for that image. The labeling grid and the dataset summary read an undecodable sidecar as absent instead, so both showed a metadata caption that the run would silently replace with the instance prompt, and counted the image as captioned. Both now track sidecar presence separately, so what the user reviews is what the run trains on. * Keep the reason a native server died, not just its backtrace A ggml abort prints its cause first and then a stack trace, so reporting the last twenty captured lines gave twenty addresses and nothing about the failure: on the macOS runner the native server died on an unimplemented Metal op and the message carried only frame pointers. The captured tail now leads with the lines that name a cause and keeps recent context after them, for both the startup failure and the mid-request death. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop staging the dense text encoder for an fp8 video load Two halves of the same gap, found while measuring the LTX-2.3 download plan: - The video download plan and the scoped pre-download never saw text_encoder_quant. An fp8 request loads a hosted pre-cast encoder, so asking for one still staged and downloaded the base repo's dense Gemma3 (48.79 GB of Lightricks/LTX-2 on the 2.3 distilled pick) that the pipeline then never opened. The plan now drops those shards and stages the pre-cast checkpoint instead; their configs stay, since the pre-cast loader still meta-inits the encoder from the base repo's component config. - The LTX-2.3 assembly builds every component itself, so pipe_kwargs (which carries the pre-cast encoder for from_pretrained) never reached it and an fp8 request silently loaded the dense encoder anyway. It is passed across explicitly now. The dense skip is earned, not assumed: only a pre-cast checkpoint that resolves on the Hub lets the plan drop the dense shards, and only one already fetched to disk lets the pull drop them, so an unpublished or gated artifact leaves both exactly as they were. If injection still fails after that, the load tops the dense weights back up rather than handing from_pretrained a snapshot with no encoder in it. Measured against the real Hub on the 2.3 distilled Q4_K_M pick: 67.24 GB before, 18.92 GB with a 0.43 GB stand-in for the pre-cast artifact (the base entry drops from 24 files / 48.79 GB to 13 files / 0.04 GB). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Match the file's typing idiom for flow_shift models/training.py annotates with typing constructs throughout (105 Optional[...], no Union), and flow_shift was the one place using PEP 604. Union[] reads like the rest of the file, and it also drops the runtime evaluation that would raise on Python 3.9. * Bound the gallery blob cache, and three interlock fixes Four review findings, all reproduced first: - The gallery object-URL caches were unbounded. A clip runs from a few MB to a few hundred, both pages stay mounted after their first visit, and entries were only dropped on delete, so scrolling pinned everything for the session. Both pages now share a byte-budgeted LRU (512 MB video / 192 MB images) keyed off the visibility signal the near-viewport fetching already provides. On-screen media, the selected clip or image, and the item just fetched are never evicted, so eviction is invisible and a single item larger than the whole budget cannot evict itself into a refetch loop. - The image, video and chat load guards ran two independent training probes but returned early when the FIRST one raised, so an unreadable LLM backend disabled the diffusion interlock and a load could proceed straight into an active diffusion trainer on the same GPU. The probes are independent now. - An engine switch swallowed a failed teardown and published the new engine anyway, which is exactly the leak the unload exists to prevent: the arbiter's evictor, /images/unload and the next load all resolve through get_active_diffusion_engine(), so the still-resident pipeline (or a live sd-server) became unreachable and the next load allocated on top of it. The switch now fails and leaves the old engine published, so it stays reclaimable. - The native generation timeout was 30 minutes while the Images page waits up to 6 hours (SETTLE_MAX_MS), so slow-but-progressing CPU jobs died deterministically at the deadline. Measured on GPU-less runners, a 512x512 4-step Q2_K generation took 900 s on Linux and 1465 s on Windows, so larger images or step counts clear half an hour easily. The ceiling now matches the page's window and applies to the whole request: chunks of a split batch share one deadline instead of each getting a full budget. Cancellation is unchanged. Declined: gating the huggingfacenotorch extra off Python 3.9 over the conditional diffusers marker. The marker is deliberate and its comment says why: diffusers dropped 3.9 in 0.38, so pinning >=0.39 outright leaves pip no candidate and the whole extra unresolvable there. The pipelines it names live in studio/backend, which cannot install on 3.9 anyway (studio.txt pins matplotlib==3.10.9 and fastmcp>=3.0.2, both requires_python >=3.10), and the extra is the general core one, so the alternative drops 3.9 for library users who never touch Studio. * Close the load-versus-training-start race, and two picker fixes - The image and video load guards read is_active() and only then selected an engine, acquired the arbiter and registered the load. A /train/diffusion/start reserving inside that window freed residents the load had not registered yet, so the trainer came up beside a brand-new pipeline. The service already had exactly the right pattern for this in dataset_mutation, so gpu_load_admission mirrors it: reserve() refuses while an admission is open, an admission refuses once a start is reserved, both decided under the one lock. The span is only the registration, since begin_load returns as soon as the load is registered and _free_gpu_for_diffusion_training preempts an in-flight load from that point. Chat is deliberately not covered: its load spans an eviction plus a multi-minute GGUF load, and it admits models that fit beside training by design, which is a different contract from the diffusion pipeline's all-or-nothing one. - Hugging Face gives the LTX-2 family the image-to-video pipeline_tag (both Lightricks/LTX-2 and unsloth/LTX-2.3-GGUF report it), so a text-to-video-only filter dropped the flagship audio family out of Video Hub search while the rest of the app routed it to Video. - Task-scoped quant fit sized picks against the LARGEST visible device while resolve_diffusion_device_target returns a bare "cuda" and torch places on the current one. On a heterogeneous host that recommended a checkpoint sized for the bigger card and then loaded it onto the smaller one. Fit now uses the device the load actually lands on; identical on a homogeneous host. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Expose the persistent conditioning cache in the start schema DiffusionLoraConfig has carried cond_cache_dir for a while and the DiT trainer acts on it, but DiffusionTrainingStartRequest omitted the field, so Pydantic dropped it silently and every API-driven run fell back to the in-memory cache that is rebuilt from scratch each time. The warm path skips loading the VAE and the multi-GB text encoders on a rerun whose images, captions and resolution are unchanged, so this was a real capability that could not be reached. Contained like output_dir rather than left to the trainer subprocess's cwd, since it is another directory the trainer writes to. Blank or omitted still means the in-memory cache, so it must not resolve to the outputs root. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix diffusion policy and classification issues from review fp8 auto precision defaulted to precise accumulate on any non-consumer GPU, which made fp8 2.05x slower than int8 on RTX 6000 Ada and slower than not quantising at all. NVIDIA's professional whitepapers do publish equal FP8 rates for both accumulate modes there, so the hardware premise held, but the cost is in the cuBLAS path rather than the published rate. Default to fast accumulate: measured on B200 the flag is a no-op (4096^3 _scaled_mm at 3023.8 vs 3041.8 TFLOP/s, bitwise-identical output, 1.213 s vs 1.230 s end to end), so it is a large win where it bites and free where it does not. Precise accumulate stays available via transformer_quant_fast_accum. Z-Image's DiT is a Lumina2 derivative, so unsloth/Z-Image-GGUF and unsloth/Z-Image-Turbo-GGUF both declare general.architecture = "lumina2" and the whole line was tagged image-diffusion-unsupported and hidden from the Images "On Device" list, though validate_load_request loads them. Resolve shared archs from the repo/file name like bare "wan" already does, with a test asserting the picker and the loader agree for every family. The sage attention on-demand install ran an unpinned `pip install sageattention`, but PyPI's newest wheel is 1.0.6 and diffusers refuses anything below 2.1.1: the install always "succeeded", wrote an unusable version into the running venv, and was rejected on the next line. Carry the dispatcher's floor so pip resolves nothing instead. The dense-quant disk gate sized the download from the bf16-RESIDENT table. The fp32 families download twice that (Z-Image: 23,479 MiB against a 21,970 MiB gate), leaving a window where the check passed and the download filled the disk; Ideogram 4 ships fp8 and was overcharged the other way. Size the gate by published bytes, verified against HF sibling metadata for all 12 families. Patch installs went through unsloth_zoo, which refuses to import unless UNSLOTH_IS_PRESENT is set, and that is set by unsloth itself. The server imports unsloth at boot so it never showed there, but any other process ran silently unpatched with every install returning False, which is 13 test failures on a clean environment. Import unsloth and retry once, memoised per process. Also: the GGUF+LoRA refusal pointed at the native engine without saying a GPU host only selects it under UNSLOTH_DIFFUSION_ENGINE=sd_cpp, so the suggestion was unreachable; the gallery recipe recorded loras from the generate request alone, losing a load-time bake; load-progress claimed "40.07 GB downloaded" for a fully cached load; and pickers.tsx imported three catalog-group helpers it never used. Reported by oobabooga. * Keep the sd.cpp text encoder on CPU under Metal macos-14 loads FLUX.2-klein-4B Q2_K natively on mps and then dies on the first generation with exit code -6: ggml_metal_op_encode_impl: error: unsupported op 'RMS_NORM' -> ggml_abort LLMEmbedder::encode_prompt -> LLMRunner::compute -> GGMLRunner::compute ggml's Metal backend gates RMS_NORM on contiguous rows and aborts the process when that does not hold, with no per-op CPU fallback, so any LLM text encoder (Qwen3 for FLUX.2 and Z-Image, T5 for FLUX.1) takes sd-server down. The encoder runs once per prompt while the DiT runs every step, so pinning only the encoder keeps Metal for the part that matters. UNSLOTH_DIFFUSION_SD_CPP_METAL_TE_GPU=1 opts back in once ggml grows the kernel. * Gate the unsloth retry in the diffusion patch backend The retry added for the clean-environment patch failures is not free: importing unsloth pulls torch in behind it, which costs ~940 MB of RSS measured in a process that had neither, and on a host with no accelerator it fails anyway. A cross-platform CI job that had generated fine at ~900 s later died 19 s in with SIGTERM and every 'if: always()' step skipped, which is the runner being torn down rather than a step failing. Retry only when torch is already imported (true of the server and of anything patching a real module, and the condition that stops the retry from being what loads torch), unsloth is installed but not yet imported, and the first failure was the ImportError the sentinel guard raises. The clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only retry the unsloth import where it can succeed The gate still let the retry run on hosts unsloth does not support, which is where it is most harmful: a 7 GB macOS runner lost the Studio server 26 s into a load, and the Linux runner was torn down mid-generation. Neither MPS nor plain CPU can complete the import, so the retry there pays the cost and fails anyway. Require an accelerator unsloth actually supports (CUDA/ROCm via torch.cuda, or XPU), with UNSLOTH_ALLOW_CPU as the documented override, and hoist the predicate to module level so it is tested directly rather than through the import system. On a CPU-only host the retry no longer fires at all; on CUDA the clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Guard old diffusers, stream video exports, record conditioned recipes Three fixes from review. The 0.39-only pipeline classes (Flux2Klein, Z-Image, Krea 2, LTX-2, HunyuanImage) were resolved by getattr deep in the load, so on the older diffusers that packaging still allows on Python 3.9 -- diffusers dropped 3.9 in 0.38 and this project still supports it, so the 0.39 floor has to be conditional or the extra becomes unresolvable -- an advertised model failed with a bare AttributeError after its checkpoint had already been downloaded. Krea 2 already guarded itself this way; assert_pipeline_class_available now runs the same check for every image and video family from validation, before any fetch, and names the version and the fix. WebM export accumulated the whole VP9 output in a BytesIO and returned it as one bytes object that the response held again. The request caps allow 2048x2048 for 1024 frames, so an export runs to hundreds of MB and concurrent clicks could exhaust the process, while the MP4 route beside it already streamed from disk. transcode_to_file encodes to a temp file and the route returns a FileResponse with a background unlink, so nothing large is resident. A conditioned generation's recipe carried only the txt2img fields, so the gallery presented an inpaint or upscale result as a complete Create recipe and restoring it replayed an unrelated text-to-image request. The images themselves are still not persisted (user uploads with their own lifetime), but the workflow and its scalars are, restore reapplies them, and the toast now names the inputs that have to be supplied again instead of silently landing on Create. Reported by Codex. * Per-load video cancel event, family-gated image picker, cond cache refusal A cancelled video load could resume: begin_load cleared the shared cancel event, and unload() drops _loading without waiting for the worker, so the next load cleared the very object the cancelled worker was watching and its multi-gigabyte pull ran on alongside the replacement until the token check at the end. Each load now gets its own threading.Event, passed down through _fetch_te_prequant and _predownload_base, so a cancelled worker stays cancelled. A cached repo with a model_index.json was advertised as text-to-image on the trust rule alone, but validate_load_request also requires a detected image family, so a trusted pipeline of an unsupported class produced a picker row that deterministically 400s. The picker now applies both gates, mirroring the video branch. cond_cache_dir was accepted for sdxl and then ignored: only the DiT trainer reads it, while the SDXL trainer builds a per-run in-memory latent cache, so the promised cross-run reuse never happened. The route now refuses it with a 400 that names the families which do support it, checked against the resolved family so an omitted model_family with an SDXL base is caught too. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the frontend build broken by the gallery blob cache tsc -b failed on the branch head, so npm run build produced no dist and every platform job fell back to --api-only: blob-url-cache.ts(29,15): TS1294: This syntax is not allowed when 'erasableSyntaxOnly' is enabled dataset-labeling-grid.tsx / dataset-showcase.tsx: Argument of type '{ url: string; bytes: number; }' is not assignable to parameter of type 'string' The cache took its budget as a constructor parameter property, which the project's tsconfig forbids, and fetchGalleryObjectUrl now returns the blob size alongside the URL for that budget, which the two dataset thumbnail components still consumed as a bare string. Declare the field explicitly and destructure the URL at both call sites. tsc -b is clean and vite build emits dist again. * Recover from a ggml unsupported-op abort by restarting on the CPU backend ggml checks every node against the device's supports_op and calls GGML_ABORT when one is not implemented, because a single-backend graph has nowhere else to put it: there is no per-op CPU fallback. The whole sd-server dies with SIGABRT mid-generation and the user gets "the native image renderer stopped unexpectedly" with no way forward. Seen on macos-14 arm64 with FLUX.2-klein-4B Q2_K through the cross-platform CI: the text encoder is already pinned to CPU, and the abort moved into the denoise loop instead. ggml_metal_op_encode_impl: error: unsupported op 'MUL_MAT' -> ggml_abort StableDiffusionGGML::sample -> sample_k_diffusion A retry on the same backend would abort identically, so the load is restarted once with --backend cpu (the only flag that changes which backend executes the graph; --offload-to-cpu moves parameters, not compute) and the generation is re-submitted. The same checkpoint then renders slower rather than not at all. Strictly bounded: the signature must carry both the unsupported-op line and ggml_abort, the device must not already be CPU, and it happens once per load, so an OOM kill or a genuine crash still surfaces as itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix two tests that only fail in a full-suite run The 3.10 CI leg resolves PyAV 17, where av.container.OutputContainer is an immutable C type, so the no-libopus export test died on "cannot set 'add_stream' attribute of immutable type" before it asserted anything. Inject the refusal by wrapping the container av.open() returns instead; modules stay patchable on every build. Removing the injection makes the test fail again, so it still covers the branch it is named for. The Xet shim's degraded-path tests drop utils.hf_xet_fallback from sys.modules and import a throwaway copy. Restoring only the sys.modules entry left the utils package attribute bound to the throwaway, and the two disagreed for the rest of the process: a later monkeypatch of the dotted target patched one copy while the code under test imported the other, so the patch did nothing and test_fetch_te_prequant_only_reports_what_it_downloaded reached the real Hub and got a 401. Restore both bindings. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop an ignored-cancel sd-server, guard deletes during diffusion training, repair unusable managed binaries sd-server does not interrupt an in-flight job, so when it ignores a cancel the grace branch abandoned the poll and reported cancellation while the native job kept a core (or the GPU) busy to completion and held the server's job slot. The comment said the caller stops the server, but only unload does that immediately: a superseding load stops it after its multi-gigabyte download, and a load that then fails never gets there. Stop it here, as the deadline branch already does. DELETE /api/models/delete-finetuned checked only the LLM trainer, so it could rmtree the output directory a live diffusion LoRA run was about to write its adapter into. Consult the diffusion training service too, like the dataset mutation and model-load routes. find_sd_*_binary only checks is_file(), so an interrupted extraction (or a prebuilt for the wrong CPU) left a present-but-unrunnable binary the installer never retried: every load probed it, fell back to diffusers, and native inference stayed off until the directory was deleted by hand. Probe it and reinstall, but only for a copy under the installer-owned root -- SD_CLI_PATH, UNSLOTH_SD_CPP_PATH, an in-tree build and anything on PATH are the user's. * Plan the pre-cast text encoder, and make the cross-trainer GPU admission atomic An fp8 text-encoder request loads a hosted PRE-CAST checkpoint, but the image download plan never received text_encoder_quant, so the manager staged the base repo's dense encoder (FLUX.2-dev's Mistral-24B is ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) and the load then pulled the pre-cast file inline, outside the manager's progress and disk preflight. The plan now takes the field, resolves the hosted artifact with the same resolver the injection uses, stages that file, and drops only those components' dense weight shards. The load's own prefetch takes the same treatment, since it paid the same cost. Only a checkpoint that really resolves on the Hub earns the drop, so a gated or renamed artifact still stages the dense encoder the load will fall back to. The two trainers admitted each other with independent check-then-act guards: the diffusion route checks the LLM backend several network-bound preflights before it reserves, and the LLM route checks the diffusion service well before it spawns, so two near-simultaneous starts could both pass and train on one GPU. reserve() now re-tests the LLM backend under its own lock, and the LLM route holds the diffusion service's gpu_load_admission across its spawn, so exactly one of the two wins. Both halves fail open, so a chat-only install still trains. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the Advanced options a staged download planned against Staging does not set busy, so while a multi-gigabyte download runs the user can still change precision, memory mode, speed or the baked LoRA selection. The pending record held only the repo and artifact, and the completed download fired a load that read the CURRENT state: the staged file set could then be missing files that load needs (fetched inline, with no progress and no disk preflight) or hold gigabytes it no longer uses. One snapshot of every Advanced control is now taken when the plan is built, and it travels with the pending record into the load, so the load that runs is the one the download was planned for. * Do not advertise a family the installed diffusers cannot build The newer families (Z-Image, Krea 2, FLUX.2, LTX-2, HunyuanImage) exist only from diffusers 0.39, and 0.39 cannot be installed on Python 3.9 at all -- diffusers dropped 3.9 in 0.38, so the requirement is conditional or the whole extra becomes unresolvable. On such an environment the picker still offered those rows, every pick failed deterministically, and the error's advice to run pip install -U diffusers could not fix it without also upgrading Python. The cached-repo picker now applies the same availability check validate_load_request does, which is keyed on the pipeline class actually present rather than on the Python version, so it is also right for an intentionally pinned older diffusers on 3.10+. Fails open when diffusers cannot be imported at all: that is a different problem and the load path reports it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore the diffusion engine selection after each router test The active engine is module state, and several tests set it by plain assignment because what _activate does to it is the thing under test, so monkeypatch could not undo it. A leaked ENGINE_SD_CPP left get_active_diffusion_engine() handing back the sd.cpp backend for the rest of the process, and every later route that reads the active engine then saw an unloaded model: eight tests in test_openai_images_generations_route.py returned 503 in a full-suite run while passing on their own. The autouse fixture now snapshots and restores it. * Stream gallery clips, and close three races around them Four fixes from the latest review pass. The video gallery downloaded each clip into a blob before it could play, so playback waited on the whole file (tens to hundreds of MB), seeking was limited to what had arrived, and every viewed clip stayed pinned in the webview. The file route already streams and serves ranges; it just could not be a <video src> because it is bearer-gated. Mint a short-lived signed link instead (its own HMAC secret, 12 hour TTL, separate from the image links) and hand it to the element, which then fetches only the ranges it plays. That removes the blob budget, its LRU and every revoke on this page. The sd.cpp readiness probe accepted any process answering on the port, so a foreign server that grabbed the port between the bind check and the spawn was adopted as ours. Confirm the listener is our child before reporting ready, and stay best-effort (psutil missing, an unknown owner, or any probe error still passes) so the check can only reject a definitely foreign process. Dataset import held its lock for the extract but not for the upload path, so two concurrent uploads into the same folder interleaved; take the same lock and return 409. And reject Windows device names (CON, NUL, COM1..9, LPT1..9, with or without an extension) plus trailing periods in dataset names, which are unopenable on Windows. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Build the image download plan for the engine that will load /images/download-plan always asked the diffusers backend, while /images/load picks the engine per host: a GGUF pick on a machine with no usable GPU routes to native sd.cpp, which reads a single-file VAE plus text encoders and never opens the base repo's sharded components. Measured on unsloth/FLUX.2-klein-4B-GGUF (Q2_K): the plan staged 7.66 GB of FLUX.2-klein-4B components the native load discards, and the 7.80 GB sd-cli actually needs was then fetched inline by the loader, outside the download manager's progress and its disk preflight. Z-Image-Turbo is the same shape. The plan now asks whichever engine the load will select. predict_engine() applies the selection policy without any side effect: it activates nothing (staging a download must not unload the resident model) and only locates the binary rather than installing it, but still counts an installable binary as available, since that is what the load does on a fresh host. The native backend gains a download_plan built from the same _asset_specs the loader fetches, returning the same envelope, so the manager stages exactly the files sd-cli opens. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not let a queued generation outlive the model, and three scan fixes Five items from the latest review; four were real. An unload or arbiter eviction only cancels the generation holding _generate_lock. A second request queued behind it holds no cancel event yet, and Python locks are not FIFO, so it could take the lock the instant the active denoise released it, still see a loaded pipeline, and run a whole new denoise after the model was told to go away: the eviction then waits minutes for it and an image lands after the eject. Unload and a superseding load now raise a fence under _lock before they queue, and a generation that wins the lock while one is pending refuses instead. The cached-model scan judged pipeline completeness across every revision, so a repo holding an older complete snapshot plus a newer companion-only one read as complete while the snapshot from_pretrained actually opens has no transformer. Both scans now look at the revision the loader will open. Deleting a dataset image deleted its caption sidecar unconditionally, which for cat.jpg alongside cat.png removed the caption the survivor still resolves to. The sidecar now goes only with the last image of that stem, matching what the thumbnail cleanup beside it already did. Importing an example into a folder that holds no images but does hold files fell back to promoting the staging dir one file at a time, so an interruption left a partial dataset that the image_count check accepts as complete on retry. Those files are folded into the staging dir instead and the promotion stays a single atomic rename. The MPS generator report does not apply: torch.Generator(device="mps") has worked since PyTorch 2.0 (pytorch/pytorch#91348) and the studio installer pins torch>=2.4. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten diffusion comments Collapse the multi-line comment blocks across the image, video, sd.cpp and diffusion-training code to one or two lines each, and drop comments that only restate the statement below them. Comments only, no code or behaviour changes. * Tighten diffusion comments (second pass) Collapse the remaining multi-line comment blocks in the video page, training routes and service, sd.cpp server and installer, memory and speed planners, and the shared request models. Comments only, no code or behaviour changes. * Tighten diffusion comments (third pass) Collapse the remaining multi-line comment blocks in the attention, cache, LoRA, prequant, precision and compile-cache modules, the sd.cpp arg builder and engine, the video routes, the Ideogram 4 assembly, the model picker, and the diffusion test suites. Comments only, no code or behaviour changes. * Restore the dataset when an example import cannot be promoted Promotion folds the folder's pre-existing entries into the staging dir so the swap is one atomic rename. Every failure after that fold left the user with nothing: the 409 path and an os.replace error both fell through to 'finally: shutil.rmtree(staging)', which deleted the entries that had just been moved in there, while the response said 'Nothing was written'. A same-named entry was also unlinked outright before the promotion was known to succeed. Park superseded same-name entries in a rescue dir instead of deleting them, record every move, and restore all of them if any step of the promotion fails. The fold loop itself is covered too: renaming a non-writable directory raises EACCES on POSIX, which previously escaped as a 500 after the earlier entries had already been moved out. A failed rename now maps to the same retryable 409 as the rmdir conflict. Verified with the reported trigger (a non-empty mode-500 directory whose name collides with an imported file): the folder listing is now identical before and after the failed import. * Drain the teardown fence on a failing unload, give each load its own cancel event Two independent leaks on the image path, both already solved elsewhere in the same file. unload() incremented _teardown_waiters, ran _unload_locked() and decremented, with no try/finally, while the superseding-load path used a finally for the same pair. _unload_locked ends in clear_gpu_cache(), whose CUDA branch calls synchronize/empty_cache/ipc_collect unguarded, and a sticky CUDA fault makes those raise. The count then never drained, so every later generation was refused as cancelled for the life of the process, a fresh load included, since begin_load's own increment and decrement are symmetric. unload() is reached from the chat/video GPU handoff, the engine router and two training routes, so one fault during an ordinary handoff wedged image generation until restart. Release it in a finally. The image and native backends each cleared one shared cancel Event on a new load. unload() sets that event to cancel an in-flight multi-GB download and drops _loading in the same breath, so a replacement load is admitted while the cancelled worker is still inside the fetch, and its clear() re-enabled the very object that worker was watching: the cancelled download resumed and ran alongside the replacement. Take a fresh Event per load and thread it to the worker, as the video backend already does, and set it under the lock since begin_load now rebinds the attribute. * Name utf-8 on the diffusion text I/O and the sd.cpp subprocess pipes tests/test_text_io_encoding.py failed on five files this branch adds. Text I/O without an explicit encoding falls back to the Windows ANSI codepage, so a non-ASCII path or manifest value round-trips corrupted, and the three sd.cpp pipes decode the child's UTF-8 output as ANSI on Windows despite already passing errors = 'replace'. Eleven read_text() / write_text() sites across diffusion_compile_cache, diffusion_ideogram4 and diffusion_krea2, plus text = True on the sd-cli version probe, the sd-cli run and the sd-server pipe. * Record the load-time build on a gallery image's recipe A gallery record documents itself as the image's full generation recipe and is embedded in the PNG, but the only load-related field it carried was the repo id. A GGUF repo holds many quants, so that does not say which one made the pixels, and it says nothing about an adapter baked in at load time. The fallback meant to cover the baked case could never fire: with no loras on the request _adjust_baked_loras zeroes every baked adapter and _active_lora_pairs drops zero-weight entries, so active_loras was always empty. A baked-and-disabled build is not the same pipeline as a never-baked one, so the recipe could not reconstruct the image once the model was rebuilt. Persist model_kind, gguf_filename, transformer_quant and the baked adapter names, read off the load state rather than the request, and show them in the recipe popover. The new fields are optional with defaults, which matters because list_gallery_images drops any record that fails validation, so a required field would have emptied every existing gallery; a regression test pins that. * Drop eleven duplicated comment tails, restore the mxfp8 denial note The comment passes collapsed several wrapped blocks onto one line without deleting the last physical line of the original wrap, leaving the tail of each sentence repeated as its own comment underneath. Two of the eleven were re-worded rather than byte-identical, so a strict suffix match missed them. 6e16ad16f also dropped the line justifying the qwen-image mxfp8 denial while that rule stayed live in _FAMILY_SCHEME_DENY, under a header that then documented only fp8 and nvfp4. Restored. Comments only; verified with the AST gate. * Only let the dense-quant fallback use shards the prefetch actually staged The prefetch skips the base repo's transformer/ shards whenever a prequant checkpoint is expected, since that checkpoint replaces them. But a prequant fetch can fail for reasons the planner cannot see: an unpublished, gated or renamed artifact, a hub 5xx, a proxy, a checkpoint the validator rejects. The loader then fell through to from_pretrained(subfolder = 'transformer') and pulled those shards inside the load lock during 'finalizing', after the previous pipeline was already evicted, where the cancel event has no reach, load_progress has already reported bytes_downloaded == bytes_total, and the cache-disk gate had only reserved the small prequant checkpoint. That is verbatim the situation _dense_quant_prefetch_needed's own docstring exists to prevent. Gate the in-loader dense fallback on the shards being staged, read off the returned file list rather than the request so a failed size estimate closes it too, and let the GGUF build take over otherwise, which is what the prequant-sized replan already does one branch over. It is also the invariant the text-encoder path already enforces: only a component that really resolves may have its dense weights dropped from a plan or a prefetch. Adds the missing coverage for the gate's prequant arm, which the existing disk-gate tests never reached. * Refuse a training output_dir that resolves to the outputs root resolve_output_dir strips a leading 'outputs' and drops '' / '.' segments, so '.', './', './.', 'outputs', 'outputs/outputs' and ' . ' all clean away to nothing and land on the outputs root itself rather than a run directory under it. The DiT trainer then writes pytorch_lora_weights.safetensors flat into the root, where scan_trained_models and scan_checkpoints cannot see it (both filter is_dir()), and a second such run overwrites the first. The UI only checks the field is non-empty, so 'outputs' is one plausible run name away. Refuse it with a 400 that says what to do instead. cond_cache_dir collapses the same way but has an honest 'off' to fall back to, so a root-resolving value now means the in-memory cache, which is what the comment above it already promised: otherwise a run drops one flat safetensors per cached latent and caption into the directory trained models live in. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Repair three defects the main merge left behind Git merged all three files without a conflict, but the result was wrong in each case. The tests only started failing once the merged tree was run. routes/inference.py duplicated the GGUF load block: this branch had moved the gguf_load_in_flight marker and the _hub_download_blocks_gguf_load guard under "if config.is_gguf and config.gguf_hf_repo", and the conflict resolution re-added main's copy at the old position, so both ran. Dropped main's copy; the earlier placement is the deliberate one, so a 409 from the hub guard cannot tear down a resident Images or Video pipeline. test_gpu_selection.py still called _hf_offline_if_dns_dead, which main renamed to _hf_offline_if_unreachable_for (#7591). Disjoint edits, so no conflict, but four route-error tests referenced a function that no longer exists. test_gguf_load_cache_reuse.py anchored its ordering assertion with rindex over "if config.is_gguf:", taking the last one before the load marker. That only held while _resolve_inherited_extra_args sat above every such line; main has since hoisted it above the gpu_ids preflight, so the anchor landed between the call and the marker and the assertion compared against an unrelated later call site. The ordering it checks is a property of _load_model_impl as a whole, so it now anchors on the function. The ordering itself is intact: _resolve_inherited_extra_args, then the gguf_load_in_flight marker, then the hub guard, then the chat handoff, then unload_model. * Stop a real sd.cpp install from breaking its own discovery tests The five "nothing is installed" assertions in test_sd_cpp_engine.py cleared SD_CLI_PATH and UNSLOTH_SD_CPP_PATH and patched Path.home, which covers hops 1, 2 and the fallback half of hop 3. It leaves two hops live. Hop 3 goes through managed_install_root(), which honors UNSLOTH_STUDIO_HOME and STUDIO_HOME and resolves to the stable-diffusion.cpp directory beside the studio home. That is the documented way to run side-by-side Studios, so anyone who has one set gets a real sd-cli back from a finder the test expects to return None. Hop 4 is the in-tree developer build, which does the same for anyone who built sd.cpp inside the checkout. Isolated with an autouse fixture rather than another helper call, because reaching the failure needs no fixture: SdCppEngine(binary = None) runs the finder from its constructor, which is why test_generate_raises_when_binary_missing failed too. The fixture points the studio home and the in-tree root at an empty tmp tree, so every hop is answered by the test rather than by the box. The in-tree root moves into a named in_tree_install_root() so it can be pointed somewhere empty; behaviour is unchanged, including the OSError and IndexError guard on an unexpected layout. * Make the sd.cpp uninstall test actually extract the code it tests The two sed ranges anchored the production fragments at column 0, but both blocks sit inside the main removal function and are indented, so each range matched nothing and LOOP_FILE / DEFAULT_FILE came out empty. Sourcing an empty file is a no-op, so the suite reported 4 passed / 7 failed: the seven removal assertions failed because nothing ran, and the four that passed were 'kept' assertions that pass trivially when nothing runs. The shell job auto-discovers tests/sh/test_*.sh and runs each under set -e, and this file is not in its skip list, so it was a deterministic red. Anchor on optional leading whitespace and fail loudly on an empty or _remove_path-less fragment, so a future reshuffle of uninstall.sh cannot make the suite vacuous again. Now 11 passed, 0 failed, against the real removal loop and the real default-mode block. * Agree on what an engine can build, and on what the installer owns Two gates, each half-applied. The unbuildable-family gate had one caller, the image branch of the cached repo picker. The GGUF classifier, the local-model classifier and the video branch had none, so on a diffusers too old for a family the picker still offered its GGUF and the load then failed. Meanwhile the loader asserted the diffusers pipeline class before engine selection, so a GGUF this host routes to native sd.cpp, which instantiates no pipeline class at all, was refused with an upgrade instruction that could not help. Both are wrong one-sidedly: hide a family only when NEITHER engine can build it, and demand the diffusers class only when diffusers is what will load it. One predicate, family_buildable_here, now answers both, so the picker and the loader cannot disagree. The population is real on Python 3.9, whose diffusers ceiling is 0.36: Flux2KleinPipeline, Krea2Pipeline and LTX2Pipeline are all absent there, and FLUX.2-klein GGUF is a repo in this PR's title. That assertion also raised RuntimeError, which /images/load maps to 409, the status that otherwise means a load is already in progress, and which escaped /images/download-plan (it catches ValueError and FileNotFoundError) as a bare 500 with the message lost. It is an unloadable pick like every other, so it raises ValueError and both routes answer 400 with the text intact. The managed-binary repair used a path test for ownership while the installer uses a marker. On a managed root without the marker, which is any install predating it, the repair deleted sd-server and the reinstall was then refused, permanently, because the surviving sd-cli keeps the directory non-empty so the marker can never be claimed: the user went from an unrunnable binary to no binary and no way back. Require the marker before discarding, which is the same definition of ours that uninstall.sh already uses to keep a user's own stable-diffusion.cpp checkout. A genuinely interrupted extraction still self-heals, since install() writes the marker before it extracts. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add gguf and av to the studio extra so the wheel install matches studio.txt This branch added both to studio/backend/requirements/studio.txt, which is what install.sh uses, but never to the studio extra in pyproject.toml, which is what 'pip install unsloth[studio]' uses. Nothing else keeps the two in sync, so tests/studio/install/test_studio_extra_matches_requirements.py failed deterministically, and the CI job runs the whole tests/ tree. The drift is not only a red test: gguf backs diffusers' GGUFQuantizationConfig and av does the MP4 encode and audio mux, so a wheel install got a Studio that cannot read a GGUF or export a video, which is most of what this PR adds. * Fix two test-suite defects: the CPU-only patch gate and a popped module The patch backend reaches unsloth_zoo's helpers only through an ~940 MB 'import unsloth', which a CPU-only host cannot complete, so every patch install returned False and nine arch/eager/compile tests failed there -- exactly the runners the retry was narrowed to protect. unsloth_zoo only wants UNSLOTH_IS_PRESENT in the environment, which costs nothing and needs no accelerator, and the conftest already sets its sibling UNSLOTH_ALLOW_CPU. Set it there too, as run.py and main.py already do at module scope, so no real server ever took the expensive route. CPU-only goes from 9 failed to 24 passed, 1 skipped; the GPU run is unchanged at 29 passed. test_setup_cache_env_hf_home popped utils.hf_cache_settings to model a fresh process and never restored it, so a later import built a second module object and rebound it on the utils package. test_hf_cache_settings then wrote its setting into one object while core.inference.diffusion read the other, the same split-module failure the xet shim already had. Restore both bindings in teardown: 21 pass together, and each file still passes alone. * Restore the setup_fail assertions the main merge reverted A fourth instance of the class 9541cc535 fixed: the merge took main's studio/setup.sh, which #7644 changed to abort through the setup_fail helper so desktop mode still emits [TAURI:ERROR], but kept this branch's older copy of the test, which still asserted the literal 'exit 1'. setup.sh is byte-identical to main here and the only diff in this file was the reverted assertions, so take main's version. 60 passed. * Fence the video teardown, reject non-finite clipping knobs, stop a test installing sd.cpp The video backend never had the teardown fence the image backend documents. Its unload and its superseding-load path both signalled the active generation, then did 'with self._generate_lock: pass' and tore the state down with the lock free. A generation queued behind that barrier holds no cancel event yet, so the signal cannot reach it, and Python locks are not FIFO: it won the lock the instant the barrier released it, read a still-loaded state and denoised a whole clip against the pipeline being freed. Reproduced on both paths, where the queued generation returned a finished MP4. Mirror the image backend: count waiters, refuse a generation while one is pending, and tear down inside the barrier with the counter released in a finally. _teardown_state becomes _teardown_state_locked since the caller now holds both locks. max_grad_norm and snr_gamma were bounded on one side only, so 1e309 floated to inf and passed. clip_grad_norm_ then computes an infinite clip coefficient, clamps it to 1.0 and scales nothing, and min(snr, inf) / snr makes every min-SNR weight 1.0. The run starts, reports normal progress and trains with the requested knob silently disabled. Measured both. NaN already failed the bounds; allow_inf_nan makes that explicit. learning_rate and flow_shift already guard this exact vector. test_load_routes_to_sd_cpp_on_cpu stubbed ensure_sd_cpp_binary but not ensure_sd_server_binary, which select_and_activate_engine probes first with installs enabled, so the unit test downloaded and unpacked 108 MB into the developer's real ~/.unsloth when UNSLOTH_STUDIO_HOME was unset. Now 0 bytes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Key the video schedule reset on the resolved defaults, not the repo id A GGUF repo holds several variants, so another client swapping a distilled build for the base one from the same repo changes the steps and guidance the backend reports while repo_id stays put. modelChanged stayed false, the page kept the previous schedule, and later generations ran an 8-step distilled setting on a model that expects roughly 40 steps and CFG 4. Include the reported defaults in the key. They only move when the resident artifact does, so a manual steps change still survives a status poll. * Retire the release-lag pin now that 2026.7.6 carries the relaxed gates The virgin Windows container row that installs from PyPI on purpose was tolerated with continue-on-error, because a Server Core container has no Microsoft Store and so no winget, and the released studio/setup.ps1 hard-stopped on a winget-only git gate and reached for winget again for the VC++ runtime. #7549 relaxed both, but it landed on 2026-07-28 and the newest wheel was 2026.7.5 from the 23rd, so the row could not get past it. The pin carried its own tripwire for exactly this moment, and it has fired: unsloth 2026.7.6 shipped 2026-07-29 and the released wheel now installs end to end, so the step reported success and the assertion errored with "delete this pin and drop continue-on-error from the Install step so this row gates". Doing that. Both rows now gate unconditionally, and the assertion that enumerated the accepted failure signatures goes with the pin it protected, since there is no longer an accepted failure for it to describe. The other three release-lag mentions in this file are descriptive rather than tolerated-failure pins: they explain why the fedora and ubuntu-nonroot legs run with overlay: true, and both already refuse a triton failure as a regression rather than accepting it as lag. Left alone. * Pin the hosted pre-quant onto the plain-torch fp8 kernel A Z-Image GGUF pick at the DEFAULT speed mode, once the hosted fp8 pre-quant is actually reachable, dies at generate with an HTTP 500: torch._dynamo.exc.Unsupported: Operator does not support running with fake tensors. Developer debug context: unsupported operator: mslk.f8f8bf16_rowwise.default _fp8_config already pins KernelPreference.TORCH when it BUILDS a config, precisely because the default AUTO switches to the MSLK kernel wherever an mslk package is importable (sm90+). A hosted checkpoint escapes that pin completely: the preference is serialized per Float8Tensor, and all 239 weights in the published Z-Image-Turbo-FP8 checkpoint carry AUTO. Loading it re-arms the very kernel the pin exists to avoid, mslk.f8f8bf16_rowwise has no fake impl, and the first compiled generate therefore cannot be traced. Isolated away from the product to be sure of the mechanism: quantise one Linear three ways on this box and compile each. KernelPreference.TORCH eager ok, compiled ok KernelPreference.AUTO eager ok, compiled FAILS on mslk.f8f8bf16_rowwise library default eager ok, compiled FAILS the same way So the pin is correct and necessary, and the only gap is that the hosted path never got it. _validate_checkpoint checks scheme, granularity, base model, min_features, exclude tokens and fast_accum, but not this. Rewriting the preference on load is safe: it selects a matmul kernel, it is not weight data, so the tensors stay bit-identical and the checkpoint's own state_dict_sha256 still describes them. It is also the faster path compiled, since the opaque extern call blocks inductor quantize fusion. Why this went unnoticed: the pre-quant repos are private, so nothing that could not read them ever took this path. It becomes the default the moment they are readable. Verified end to end at 1024x1024, 8 steps, speed_mode default, on the published checkpoint: before HTTP 500 at generate, diffusion.generate_failed after pinned 239 weights to the plain-torch fp8 kernel, load 10.0 s, transformer_quant fp8 with compiled engaged, cold 7.9 s, warm 0.87 0.87 0.87 0.86 0.87 s That warm number also beats the 1.4 s recorded for this shape on 2026-07-26. 159 prequant and transformer-quant tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cover the fp8 kernel pin so the 500 cannot come back quietly The hosted pre-quant re-arming MSLK was only visible as an HTTP 500 on a compiled generate, which needs a GPU, a readable private repo and an importable mslk to reproduce. None of that is available in CI, so the pin would rot unnoticed. Three hermetic cases on _pin_kernel_preference instead: AUTO weights are rewritten and already-TORCH ones are left alone (counting only what changed), a weight that refuses the assignment does not sink the whole load, and with no torchao enum available the checkpoint is left exactly as saved. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: tighten comments across the image generation, video and training code Rewrite the comments added by this branch to be shorter and clearer: collapse multi-line explanations into one or two lines, drop comments that only restate the code, and keep the rationale that explains why a choice was made. No code changes. * Studio: let the pipeline-class guard survive a host with no diffusers Backend CI installs the CPU-only dependency set, which has no diffusers, and runs without a GPU. 13 tests failed there for three separate reasons, one of which is a real product bug rather than a test-environment gap. assert_pipeline_class_available did a bare "import diffusers", so on any host without it the call raised ModuleNotFoundError instead of the ValueError its own contract promises. /images/download-plan catches only ValueError and FileNotFoundError, so that escaped as a bare 500 with the message lost, which is the exact failure the guard exists to prevent. It is reachable in production: the native sd.cpp engine serves GGUF picks on a CPU or Apple host where diffusers is never installed. Absent diffusers now returns, since the check answers "is the installed diffusers new enough for this family" and there is no version to judge; a pick that genuinely needs diffusers still fails in the loader with its own message. The rest were tests asserting through gates unrelated to what they cover: - Three cond_cache_dir route tests drive a DiT family, which the start route refuses without an accelerator. They now take the existing dit_train_host fixture, written for this case, so they keep testing the schema on every host. - test_download_plan_forwards_the_load_time_controls stubs the diffusers planner but sends a GGUF pick, which routes to the native planner on a GPU-less host. It now pins the engine, so it tests kwarg forwarding rather than the hardware. Engine selection keeps its own tests. - The pipeline-class guard test needed a real diffusers only for its sweep over every shipped family. That half is split out and skips on its own; the stub-driven refusal needs no diffusers and still runs. Added a test for the absent-diffusers contract above. 409 pass on a GPU host with diffusers; 301 pass and 1 skips with diffusers blocked and no GPU. * Studio: cut the image generation comments down further Second pass over the comments this branch adds. Collapse the multi-line blocks that still read as paragraphs, rewrite the longest one-liners so they say the same thing in fewer words, and drop a stale comment that had drifted away from the constant it described. No code changes: only comments and whitespace. * Studio: workflow rail, sidebar Video row, and Train settings polish Sidebar - Pin Video under Images by default; persist store bumped to v3 so an untouched install adopts it while a custom arrangement is left alone. - Add "Customize sidebar" to the end of the More flyout, opening Settings > Appearance scrolled to the sidebar nav section. Images - Replace the workflow dropdown with an icon rail down the left edge: all seven workflows visible, keyboard nav, tooltips carry the labels. - Workflows stay selectable with no model loaded, so one can be set up before picking a model. Generate is already gated on a loaded model. - Video moves out of the Create/Train strip into a link at the far right, with a matching Images link on the Video page. - Wider gutters around the settings column, headings matched to Train. Train - Field guidance moves into "i" tooltips; only state that limits a control stays on the page. Steps, LoRA rank and the rest gain hints. - /info reports params, qlora_vram_gb, gated and note as fields, so the family note renders as chips. vram_note is rebuilt from them unchanged for older clients. Shared - Number steppers appear on hover or focus. - Lighter control border (#e9e9e9) on Images and Video, light mode only. * Studio: move Advanced inline on Images and Video - Replace the top-bar toggle and right-docked Advanced panel with a disclosure under Seed, so load-time tuning sits with the settings it affects and opening it no longer shrinks the preview canvas. - Video uses the bordered variant, since offload and memory decide whether a model fits at all. - Open state persists per page in localStorage. - Tighten the Steps unit trigger: 14px chevron, less right padding. * Studio: full width media pages and layout polish - Drop the 1100px cap on Images Create, Images Train and Video, so the preview canvas grows with the window instead of sitting in a band. - Align the workflow rail to the model selector label, widen the gap before its divider, and restore the divider itself. - Advanced: more room above and below, icon and label sized to the slider rows, and the same quiet row on Video as on Images. - Give Video a pane heading and description, matching Images. - More space above Seed on both pages. - More no longer takes the active style when the current page is one of its own rows. * Studio: fix the Extend side toggles The resting outline used a ring, and index.css blanks the ring on the button holding mouse focus, so the side you just clicked lost its border until focus moved. Use a border for the outline and leave the ring to focus-visible. Also separates the two states properly (the off state had no surface of its own), drops the border in dark mode as the inputs do, and adds the aria-pressed and focus-visible styling the buttons were missing. * Studio: drop the border on the Extend side toggles Light mode should carry no border either, so the fill alone marks the state: a muted surface when off, a primary tint when on. Matches the borderless treatment dark mode already had. * Studio: move the Images workflows into the sidebar The workflow switcher was a vertical icon rail on the page. It now lives in the sidebar under Images, so the page keeps that width for the canvas. - Workflows list under the Images row, with a chevron to fold them away. Hovering the row peeks them in a flyout on the standard menu surface. - Clicking Images while already there toggles the list instead of navigating. - The listed workflow carries the highlight, so the Images row drops it. - Create takes its own icon: it was sharing the New chat pencil. - Model hub moves above Projects. Persisted layouts bump to v4 and only adopt the new order where the stored one is still a shipped default. - Images and Video content both start at 32px, clear of the sidebar. * Studio: simplify the Images workflow submenu Dropping the hover flyout: two ways to reach the same seven workflows, one of them overlapping the list right below it, read as clutter. - The flyout is gone. The workflows are rows under Images and nothing else. - On the Images page they are always open, since they are that page's switcher. Elsewhere they stay folded, and the row's chevron opens them. - Create takes the sparkles icon. - Create and Train both pad their settings column to 40px a side, so the two tabs line up with each other and with the model selector. * Studio: dock the generate action and tidy the Create controls The primary action sat at the foot of a long scroll, so it was off screen until you scrolled for it. - Generate, Video's Generate and Train's Start training float at the bottom of their settings column. No bar behind them: hover lightens the fill rather than thinning it, and the disabled state is opaque, so the controls underneath never show through. - Aspect ratios read as names: Square (1:1), Widescreen (16:9) and so on. - Width and height replace their sliders with two compact boxes. Type a size or pick one from the menu; the value still snaps into range and still drives the locked ratio. - Negative prompt gets a quiet disclosure under the prompt. It was there before but only above guidance 0, which is not the default, so it never showed. - The info "i" is smaller across the settings UI. * Studio: reveal Images and Video field hints on hover The "i" next to every field label sat there permanently, which made a column of settings read as busier than it is. Scoped to the diffusion pages by CSS rather than per component: the hints come from the page's own Field and SelectRow, the train panel's FieldLabel, and the chat ParamSlider, which is shared and should not change for chat. Reveals when the pointer is inside the field, on focus-within for keyboard, and stays inert while hidden. InfoHint grows a data-slot so the rule has a stable hook, rather than keying off its aria-label. * Studio: one-line sliders, edge fades, shared negative prompt - Steps, Guidance and the rest put label, track and value on one row. ParamSlider is shared with chat, so this is an opt-in prop and chat keeps the stacked layout. - The settings columns fade at whichever edge they run past, as the sidebar and model picker do, instead of cutting off. New useScrollFades hook drives all three. - Negative prompt moves to a shared component and Video picks it up, so both pages collapse it the same way. - Right gutter sits closer to the rule, Steps takes a bigger break above it, and the Advanced rule sits between its neighbours rather than up against the field above. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: add the API monitor to the sidebar nav Sits under More, below Export. Points at the monitor page, not the API keys dialog the profile menu opens. No store version bump: an id missing from a stored layout is appended with its default pinned state, so existing sidebars gain the row at the end and keep their own order. * Studio: take main's copy of the sidebar spinner test The merge kept this branch's older version, which pins the Recents row to h-[33px]. Main already relaxed that to any height, since row density moves independently of the trailing column the test is about. * Studio: fix FLUX.2 klein size resolution, HunyuanVideo tier routing, and the personalization save Five fixes found while auditing the model registries against the live Hub. FLUX.2 klein 9B loaded against the 4B config. One family covers both klein sizes and defaults to 4B, relying on the base_model card tag for the real base, but the trust allowlist only had klein-4B, so the correct tag was discarded and a 9B GGUF (inner_dim 4096) was loaded against a 4B config (3072). That surfaced as a bare shape mismatch from inside the GGUF quantizer naming neither the file nor the repo. Adds the missing allowlist entries and a header-level size check that fails early with a legible message; the check is fail-open, so an unreadable file, a non-FLUX.2 family, or an unmapped base leaves the load exactly as it was. The sd.cpp text encoder rule matched the literal "klein-9b", so klein-base-9B was handed the 4B encoder. HunyuanVideo-1.5 720p checkpoints routed to the 480p family. Only the literal 720p_t2v path was aliased, so 720p_i2v and every GGUF repack fell through to the generic token and inherited the 480p base repo, which also supplies the VAE and text encoder. The tier is baked into the weights (transformer target_size 640 vs 960, scheduler shift 5.0 vs 9.0) and the two bucket lists are disjoint, so this ran the whole pipeline off-tier. Resolution presets that the checkpoints were never trained for. Wan2.2 TI2V-5B is 720P-only upstream (SUPPORTED_SIZES is exactly 704x1280 and 1280x704, asserted in generate.py), and the HunyuanVideo 480p square preset 624x624 is not a bucket of its tier; 640x640 is. Personalization could never be saved. The frontend ships eight sidebar nav ids and refills every missing one on each save, but the backend Literal had seven and no "api", so every PUT to /api/settings/personalization returned 422. Aligns the server defaults and the Literal with the shipped layout. A catalog row pointing at a file that does not exist. unsloth/Qwen-Image-2512-FP8 holds torch prequant .pt checkpoints, not qwen-image-2512-fp8.safetensors, and fp8 is denied for this family anyway because it renders black, yet the row was the auto-route target on any GPU above 40 GB. Also marks FLUX.1-schnell, Krea-2-Turbo and the two Ideogram repos gated: all four are gated on the Hub today, and schnell being Apache-2.0 does not make it ungated. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: stop a case-only duplicate upload destroying an image, and size the load against the right GPU Two fixes from the review round. A dataset upload containing two names that differ only by case silently loses one on Windows and macOS. The in-batch duplicate check is an exact string compare, so "Cat.png" and "cat.png" both pass, but on a case-folding filesystem they are one destination: the commit step moves the first staged part aside, writes the second over it, then deletes the backup, while the response still reports both as uploaded. Captions collapse the same way and never even reach the image branch. No single folder can hold such a pair, but the Windows and macOS open dialogs flatten search and Recents results across folders into one multi-selectable list, which is a normal way a LoRA dataset gets assembled. Rejecting the pair everywhere would regress Linux, where the two really are different files with their own sidecars, so the check probes the filesystem once per process instead of keying off sys.platform: macOS also ships case-sensitive APFS volumes, and a Linux host can keep its Studio home on an exFAT or NTFS mount. A failed probe answers "case-sensitive", which leaves today's behaviour untouched. The quantization fit budget was sized against the wrong card under a reordering CUDA_VISIBLE_DEVICES. A bare "cuda" load lands on visible ordinal 0, but the reducer took the lowest PHYSICAL index, which the nvidia-smi path reports as index_kind "physical". Under "3,1" that sizes against GPU 1 while the pipeline loads onto GPU 3. The hook's own interface doc already promised the lowest visible ordinal. Ranks by visible_ordinal, falling back to index only for an older backend that omits it. * Studio: rework the diffusion dataset panel and align the pane gutters Upload: - one Upload button beside the dataset name, with an upload icon - picking files uploads them, no second confirm click - Add button beside the Training images dropdown, so images can go into a set that already exists - fall back to the upload form when a selected dataset no longer resolves Labeling grid: - two columns at any width; sm:grid-cols-3 crushed the tiles in a fixed width column - pin the caption size so the Textarea's md:text-sm does not outsize a tile - name the tile hover group; a bare one revealed every tile's Remove at once - Remove is an icon button on the image, not a word over the artwork - drop the amber tile fill, keep the No caption label - match the header, toggle and status text to the section's type scale Gutters: - the run area and the preview canvas now sit 40px off the rule, the gutter the settings column has off the page edge * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: give the dataset preview per-image hover and remove, and wrap it onto rows - each thumbnail is its own button, so it can carry an X that removes just that image; the strip was one button and could not nest a second control - hover lightens the image instead of drawing a border - wraps onto more rows rather than scrolling sideways, which also keeps the bottom corners rounded - samples 12 thumbnails, up from 8, now that a second row is available - a delete refreshes the panel's dataset counts * Studio: show the workflow icon beside the Images and Video headings Images reads it off WORKFLOW_TABS, so it is the same icon the sidebar submenu shows and it follows the active workflow. Video's heading is also Create, so it takes the Create icon rather than the nav row's film slate. * Studio: reject a duplicate LoRA id on the diffusion load path too DiffusionGenerateRequest already refuses a repeated adapter id, with a comment naming the hazard: _resolve_lora_set suffixes colliding adapter names, so the same id resolves the SAME adapter twice and set_adapters stacks both copies past the per-adapter weight bound. DiffusionLoadRequest bounds only the list length, and it matters more there. Generation-time stacking spoils one image; on the load path the adapters are baked into the quantized build before compilation, so the unintended combination rides every image until a reload. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: teach the GGUF reuse test double about holds_no_vram The resident-GGUF fast path consults llama_backend.holds_no_vram before asserting CHAT ownership, but two tests hand it a SimpleNamespace built before that attribute existed, so they raised AttributeError and returned a 500. A real LlamaCppBackend exposes it as a property; the doubles now carry it too. Product code is unchanged: these pass on main and were the branch breaking its own test, not a defect in the guard. --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: michaelhan <michaelhan2050@gmail.com> Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: Michael Han <107991372+shimmyshimmer@users.noreply.github.com> | 1 个月前 | |
Report the precision actually in use, and refuse an explicit one we cannot honor (#8165) * Studio: report the precision that actually ran, and refuse one that cannot The loader already knew the truth and threw it away at the API boundary. Status reported the ENGAGED transformer / text-encoder precision, but nothing echoed back what the caller ASKED for, so once a fallback happened the request was gone: the Advanced panel kept its dropdown on FP8 while a Q4_K_M GGUF ran, the "Auto: X" badge was suppressed for exactly the case that needed it (source !== "auto" rendered nothing), and a successfully generated image or saved clip carried no evidence of the precision behind it. Backend - DiffusionResolvedControl gains `requested` (the raw ask, null when left to the backend) and `status` ("applied" | "fell_back" | "unsupported") beside the existing value/source/reason. Both default, so an older payload still parses. - build_resolved_record keeps the request beside the engaged value and derives a mismatch for the controls that answer in the vocabulary they are asked in. memory_mode and attention_backend do not, so they are classified by the call site instead of compared blindly. - Every transformer decline site now records WHY, in the caller's terms: an uncached hosted prequant, a re-plan that still needs offload, a dense-fit miss, a failed quant build, an unsupported scheme, and the wrong load kind. - quantize_text_encoders returns a TEQuantOutcome (mode + reason + status). The int8 -> fp8 downgrade, the offload skip and the unsupported-device path were all bare `return None`; the last one had no log line at all. - Explicit precision fails closed. Host-level impossibilities are refused in begin_load, so /images/load and /video/load answer 409 before anything is evicted; footprint-dependent declines raise inside the load and surface on load-progress. `auto` still falls back silently, and UNSLOTH_DIFFUSION_ALLOW_PRECISION_FALLBACK=1 restores the old behaviour. - Saved output metadata: images add text_encoder_quant / memory_mode / offload_policy; video clips gain the whole build block images already had. All read from the engaged state, none added to the required-key sets, so older PNGs and sidecars still list. Frontend - resolved-precision.ts holds the badge/select decisions as pure functions. A declined request now renders "FP8 -> OFF" in a warning tone with the reason in the tooltip, instead of nothing. - The Advanced selects reseed from the loaded build, so a declined scheme stops advertising itself, and a "Loaded build" summary reports the transformer and text-encoder precision plus the memory mode and resolved offload behaviour. - A 409 refusal is surfaced as a titled, actionable toast; transformer_quant is no longer sent for load kinds that cannot use it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Raise the precision refusal before the GPU handoff, not inside it The 409 exists to preserve two things: the chat model holding the GPU, and the several GB the load would otherwise pull down before failing. The check was made in begin_load, which is too late for both. begin_load runs inside acquire_for, and acquire_for evicts the current owner under the arbiter lock BEFORE it runs the register callback. On the image path it also runs after select_and_activate_engine, which unloads the resident model on an engine switch. So an impossible explicit precision was refused having already destroyed exactly what the refusal was meant to protect: the user got a 409 and an empty GPU. Both checks are now made by the route, alongside the sibling refusals that already run there (the unloadable pick, the gated companion), and before the device is taken. The copy in begin_load stays, since it is the load path's own invariant and other callers reach it directly. Diffusers only, on the image path. The native sd.cpp engine accepts transformer_quant / text_encoder_quant for interface parity and ignores them, so gating that path on a torchao capability would refuse loads that work today. A probe failure leaves pending_name None and skips the route check, which is the pre-existing behaviour rather than a new one. `auto` is never refused, so a caller that left the precision to the backend cannot reach any of this. * Stop the Advanced reseed firing on generation-time record rewrites Two separate ways the resolved record was read too literally. The reseed effect keyed on JSON.stringify(resolved). That record is not load-time-only: the backend rewrites entries of it during GENERATION. speed_mode and attention_backend change when the deferred compile profile engages on the 3rd image, and transformer_cache changes whenever the step-cache threshold flips. Each of those moved the key with no load behind it, so the effect re-ran and overwrote a Precision the user had picked but not yet loaded. An edit made after a load is meant to survive until the next LOAD replaces it. resolvedSeedKey covers only the three controls the effect actually writes, and for attention only the request side, since that is the field the reseed reads for an auto or honored request and the one a rewrite leaves alone. A real reload still re-fires: it always moves a request or an engaged value on one of the three. isResolvedHonored treated every status that was not "applied" as a decline. `status` is typed wider than the backend's union on purpose, so a newer backend can add a value, but that reading threw the tolerance away: an unknown status painted a red "FP8 -> FP8" over a request that was honored, and on memory_mode (asked "low_vram", answered "sequential") a "LOW_VRAM -> SEQUENTIAL" that never happened. Only the two statuses that mean a decline are now read as one. Staying quiet is the safe direction, since the build that adds a status ships the frontend that understands it. * Name the fault behind a refused precision, and stop caching an OOM as one Two things the fail-closed 409 made load-bearing that were fine while a declined explicit scheme fell back quietly. select_transformer_quant_scheme answers None for three different faults and the refusal reported all of them as "'fp8' is not usable for family 'X' on this GPU". Measured here on a B200: torchao could not import at all (cannot import name 'ScalingType' from torch.nn.functional, a torch/torchao version skew), the smoke probe swallowed that, and every explicit scheme was refused with a message blaming Blackwell hardware that runs all of them. A skew is fixed by a pip install; a GPU limit is not; a family the accuracy gate rules out is neither. explain_unusable_ scheme separates the three, shared by the image and video resolvers so they cannot drift. The smoke probe also cached an out-of-memory as a verdict on the scheme. That probe now runs on the ROUTE thread, which is the point of raising the refusal before the GPU handoff, so it meets a full GPU by design: the resident chat model has not been evicted yet. One transient OOM therefore refused that scheme for the rest of the process, on a host that runs it fine seconds later. Allocator failures are no longer remembered; every other failure still is, because those really are properties of the build. torch.OutOfMemoryError subclasses RuntimeError rather than MemoryError and has moved between torch and torch.cuda, so both names are tried with the message as the backstop. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Correct a comment the reseed-key change left behind The dependency is no longer the serialized record; it is the load-time projection of it, which is the whole point of the previous commit. * Close the three places an explicit precision was still reported wrong The native engine was exempt from the precision gate, on the grounds that refusing there would break loads that work today. But the loads it works for are exactly the silent mismatch this change exists to remove: sd.cpp accepts transformer_quant and text_encoder_quant for interface parity, ignores them, and reports null, so an explicit FP8 succeeded having quantised nothing. The diffusers path already refuses on the same CPU-only host, so the exemption also left the two engines disagreeing about one request. It now refuses, with a message naming the engine; auto, none and an omitted value still pass through untouched, and the existing escape hatch waives it. The Loaded build panel labelled any non-GGUF load BF16, but a single-file safetensors keeps whatever precision it was saved in and FP8 checkpoints are explicitly supported, so the one panel whose job is to say what actually loaded was asserting a wrong number. It reads "As in checkpoint" for single_file now, on the video page as well. And the video loader rewrites an omitted transformer_quant to "off" under speed_mode="off" before building the resolved record, so the record claimed the user had pinned bf16: the Auto badge disappeared and the Precision select reseeded to none, leaving quantisation pinned off after a Speed change and reload. The raw request is captured before the rewrite and reported. Four tests, each confirmed against a mutation. * Report the truth on partial casts, native builds and unprovable probes - text-encoder quant that cast one encoder and not its sibling reported "applied" and both loaders let the load through, recording the requested mode while conditioning ran off a mixture; it is now reported as a fallback and refused like any other declined explicit precision. - the refusal message pointed users at "Choose Auto" for text_encoder_quant, which both request models reject, so following it returned a 422. - the native sd.cpp engine reports dtype "gguf" and no model_kind, so the Loaded build panel labelled every native checkpoint BF16; the label rule is now one shared helper covering both pages. - native generation results carried no offload state, so every native image recipe persisted it as null. - the video load route probed CUDA precision before the training guard, which allocated next to a training subprocess for a load about to be refused. - a smoke-probe OOM before the arbiter eviction is not a verdict on the scheme, but it reached the new route gate as one and refused the load the eviction was about to make room for. - the torchao import error is interpolated into the 409 detail and names the absolute file that raised it; paths are stripped there and logged in full. * Do not read a native null text-encoder quant as BF16 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refuse an unhonourable precision before the download plan is staged The load routes refuse an explicit precision this host cannot honour, but the UI plans and stages first, so the refusal arrived after the GGUF and its companions (tens of GB on the video side) had already been pulled. Both checks are network-free, so /images/download-plan and /video/download-plan now make them before building the plan, and map the refusal to the same 409 the load routes give. Also fixes three Loaded-build panel rows that were only correct for diffusers: the dense dtype label no longer calls a float16/float32 load BF16, the attention row names the native sd.cpp engine instead of Native SDPA, and the Memory row renders when an offload is active but no memory mode is set. * Keep the plan-time precision check off the GPU, and off the load's blind spot The plan runs before the load's training guard, so an uncached scheme sent assert_precision_available into its quantise-and-matmul smoke probe and initialised CUDA in the Studio process beside a running trainer. Staging needs no GPU, so the check is skipped while training is active on both the image and video plan routes; the load still refuses the same pick afterwards. The video page also asked for its plan without the selected precision while sending it on the load, so the plan cleared a scheme the load would reject and staged the pipeline first. It now sends it under the same pipeline-only rule. A single_file transformer is no longer labelled 'As in checkpoint': from_single_file is handed the resolved torch_dtype, so an fp8 checkpoint is upcast on load and the panel was hiding the dtype it actually runs in. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep two video route tests off the host's precision support Both assert that a text_encoder_quant reaches the backend, and both now run through a precision gate whose answer depends on the machine: a GPU-less runner refuses fp8 with a 409 and the forwarding under test never happens. They run under the product's own fallback escape hatch instead. * Gate diffusion precision on the engine and mode that actually run Three places reported or refused a precision that was not the one the runtime would use. The load route asked predict_engine which gate to apply, and a probe failure left pending_name None, skipping both arms. Selection could then land on sd.cpp anyway, which accepts the knobs and ignores them, so an explicit fp8 loaded, quantised nothing and reported null. The gate is now re-asked of the engine that was actually activated, and only when the prediction missed, so a correct one is never paid twice. quantize_text_encoders rewrites an int8 request to layerwise fp8 on any family with no keep-bf16 schedule, and that path needs no torchao. Both precision asserts consulted te_quant_supported about the raw int8 and so refused loads the runtime would run and report as fell_back. New effective_te_quant() resolves the downgrade before support is consulted. The Recipe popover's Memory row substituted "auto" for a null memory_mode, which is what the native engine always records, claiming the memory planner had picked a mode on the one path that never runs it. An absent mode now reports the offload alone. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refuse an offload-bound precision up front, and read the video precision live An explicit dense precision with Memory=balanced/low_vram (or the legacy cpu_offload flag) is incompatible on its face: those requests name their offload policy without measuring anything, offload hooks move modules with Module.to(), and torchao tensors do not survive it, so the loader skips the dense build. The strict refusal then landed after the resident image model had been torn down. The pre-handoff gate now takes the memory request and refuses the pair before the GPU is acquired. fast and auto are decided from the measured footprint and are untouched. The video page's loadOrStage is memoized on [stage, pickGuard], so its plain capture of transformerQuant froze at whatever was selected when the callback was built. The ordinary auto to FP8 change then asked the plan with no precision, skipping the pre-download refusal, and staged tens of GB before the load rejected the same pick. It reads through a ref now, the same way handleLoad already does. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Extend the offload precision gate to the encoder and to video The pre-handoff gate refused an offload-bound dense transformer quant but not the two adjacent cases. quantize_text_encoders reports the torchao encoder modes (int8, fp8_dynamic, nvfp4) unsupported once offload is active, for the same reason: the hooks move modules with Module.to() and those tensor subclasses do not survive it. The image gate now refuses them alongside an offload-forcing memory request. Layerwise fp8 is a dtype cast and is untouched. assert_video_precision_available took no memory request at all, so a video load with an explicit precision and balanced or low_vram passed the route preflight and was refused inside load_pipeline, after acquire_for and the teardown had evicted the resident model. It takes memory_mode now and applies both rules. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refuse a torchao encoder mode a host cannot import, and plan with the memory request te_quant_supported only asks the device: a CUDA bf16 host with a broken or absent torchao passed every capability check, and the casters import torchao only after the pipeline has been downloaded and built, so the refusal came through load-progress instead of the pre-load 409. Both gates now ask torchao_quantize_importable() for the torchao-backed encoder modes. Layerwise fp8 is a plain dtype cast and does not need it. The video staged plan sent the precision but not the memory mode, and the route refuses the incompatible pair only when it can see both -- so the plan succeeded and tens of GB were staged before /video/load rejected the same pick. It reads the memory mode through a live ref, like the precision. --------- Co-authored-by: Daniel Han <moonshotaisubstack@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> | 26 天前 | |
Update Install Scripts (#5968) * Update Install Scripts Add SPDX AGPL-3.0 headers to the installer scripts and let the piped web installs take their common options from the environment. - install.sh / install.ps1: read UNSLOTH_NO_TORCH (and UNSLOTH_PYTHON for install.sh) so a piped install needs no positional flags. Flags and the pipe forms still work; an explicit flag wins. - Fix the UNSLOTH_STUDIO_HOME example so the variable sits after the pipe and reaches sh instead of curl. - Add SPDX headers to install.sh, install.ps1, the uninstall scripts, and the MLX install scripts. - Drop the internal test package names from the studio install comments. * Mirror UNSLOTH_PYTHON env var to install.ps1 install.ps1 now reads UNSLOTH_PYTHON to pin the Python version, matching install.sh, and lists all three env vars (UNSLOTH_NO_TORCH, UNSLOTH_PYTHON, UNSLOTH_STUDIO_HOME) in the header examples. The requested version is preferred during detection and used as the winget install target; behavior is unchanged when the variable is unset. | 3 个月前 | |
Studio: drop the mlx-lm 0.31.3 exclusion so current mlx-vlm resolves (#7061) | 28 天前 | |
Pin the ROCm-on-WSL bootstrap to immutable refs (#8540) * Pin the ROCm-on-WSL bootstrap to immutable refs The WSL AMD bootstrap runs unattended and installs with sudo, so both pieces of code it pulls in should be addressed by commit, not by a moving branch: - install.sh fetched the helper from raw.githubusercontent .../unsloth/main, so anything landing on main became root code on affected WSL hosts. It now fetches a pinned commit SHA. - the helper cloned ROCm/librocdxg at develop, then cmake/make/sudo make install. It now builds tag v1.2.2 and verifies the clone resolves to that tag's commit SHA before anything is built or installed. v1.2.2 is what librocdxg develop points at today and the pinned helper commit is the current main copy, so the installed result is unchanged. An explicit UNSLOTH_LIBROCDXG_REF with no UNSLOTH_LIBROCDXG_SHA still builds whatever the operator asked for. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Point the helper pin at current main and forward the librocdxg pin Two follow-ups so WSL users keep getting the current helper and the pinned third-party source applies immediately: - _ROCM_WSL_HELPER_REF now points at current main (a23951b), which carries the helper byte for byte, instead of the older commit that last touched it. - install.sh forwards UNSLOTH_LIBROCDXG_REF/SHA to the helper, so the pin holds even when the fetched helper predates it, and a test keeps the forwarded pin equal to the helper's own defaults. A user-set ref still wins. librocdxg v1.2.2 confirmed as the latest non-prerelease (2026-08-05) and its tag resolves to 4955d128, which is also develop HEAD. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the librocdxg pin a commit so older fetched helpers honour it The forwarded ref was the v1.2.2 tag, which an older fetched helper clones without any SHA check, so the curl | sh path still depended on that tag not moving. Forward the commit itself instead: the older helper's --branch attempt fails, its full clone follows, and its checkout resolves exactly this revision. The helper defaults to the same commit and skips the doomed --branch attempt when the ref is a 40 character commit. * Forward an explicit librocdxg SHA as the ref as well An operator who pins a branch or tag plus its expected commit was still exposed on the piped path: a helper old enough to ignore UNSLOTH_LIBROCDXG_SHA clones the symbolic ref and installs its head unverified. Whenever a SHA is known, ours or theirs, forward it as the ref too, so every helper vintage resolves the one revision that was authorised. * Refuse a helper without the pinned-source check, and fail a lost checkout A helper predating this change resolves the forwarded ref through 'git checkout ... || true', so if the pinned commit ever stopped existing upstream it would fall through and build the repository's default HEAD as root. install.sh now runs only a helper that carries the check, whatever the pin or a user's older checkout supplies, and the pin moves to a revision that has it. In the helper, a failed checkout with no SHA to verify against is now fatal rather than silently building the default branch. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Declare a helper capability contract install.sh gates on a content marker, which only proves a string is present, not that the guarantees behind it are. Declare them explicitly instead so the gate can require a level rather than infer one. * Gate on the declared contract, not on a string being present The marker only proved LIBROCDXG_SHA appeared somewhere, so it accepted a helper that verifies the clone but still swallows a failed ref-only checkout and builds default HEAD. Require the helper to declare contract 2, which stands for both guarantees, and pin a revision that declares it. A test keeps the level install.sh requires equal to the level the helper declares. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten the comments on the ROCm-on-WSL pins --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> | 23 天前 | |
Studio: add image generation, editing workflows and LoRA training with Unsloth GGUFs (#6763) * Tighten comments in the image stack tests and scripts * Close video single-file, training reservation, and image mount-resume gaps Route on-device single-checkpoint video folders through the single_file loader: a bare local .safetensors directory (no model_index.json) is advertised as a pipeline with no filename, so validation rejected it before it could load. Reinterpret the pick as a single_file load of the sole checkpoint, mirroring the image load route. Treat a reserved-but-not-yet-spawned LLM training start as active in is_training_active() so /images/load, /video/load, and /diffusion/start cannot race the reserved run for VRAM during the pre-spawn free window. Mirrors the diffusion training service reservation. Resume an in-flight image generation on the Images page mount: probe generate-progress, re-enter the poll loop, and refresh the gallery on completion so a run started elsewhere is reflected and its saved image appears without a manual refresh. Seed resident image defaults from the resolved base_repo rather than a possibly path-shaped repo_id so the first resident generation uses the right recipe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Publish image generation active state before pre-denoise setup generate() assigned self._gen only at the pipe() call, after deferred compile, LoRA resolution/application, and ControlNet download/build had run. Across that setup window generate_progress() reported inactive even though _generate_lock was held, so a reloaded page's mount probe showed idle and let a second generate queue behind the first. Publish an active step-0 _GenState the moment the generation lock is acquired, before the setup work, and clear it in the outer finally so a setup-time error cannot leave the UI stuck active. Mirrors the video backend's queued phase and the training start guard. * Studio: fix diffusion install ownership, dataset upload atomicity, gallery pagination, and teardown races install_sd_cpp_prebuilt: only write the .unsloth-studio-owned marker when the install created the target directory or it was empty. Adopting a pre-existing, unowned, non-empty directory (a user's own stable-diffusion.cpp checkout) made it eligible for the uninstaller's recursive delete. routes/training upload: make the multi-file promotion transactional. Back up each displaced original and roll every destination back on any failure, so a mid-loop rename error can no longer partially overwrite the live dataset. routes/training _resolve_dataset_folder: reject a symlinked dataset directory and prove the resolved folder stays under the datasets root, so image read/caption/delete cannot escape the root through a link. routes/training delete: escape glob metacharacters in the thumbnail filename so deleting an image named like [ab].png removes only its own thumbnails. image_gallery / video_gallery listing: filter records against the response schema inside the pager via a valid callback, so offset/limit/has_more all count over accepted records. A leading schema-invalid record no longer returns an empty page with has_more=true and stalls infinite scroll at offset 0. image_gallery / video_gallery save: publish via a temp file plus atomic rename (the sidecar is the video pair's commit marker) and clean up on failure, so a partial write never surfaces a truncated PNG or strands an orphan MP4. diffusion_train_common discovery: treat an empty caption sidecar as a metadata tombstone that still falls through to the dreambooth instance prompt, so clearing every metadata caption no longer fails with no captioned images found. diffusion backend unload: wait for an in-flight denoise to exit before tearing down process-wide patches and state, mirroring the load path. diffusion_engine_router: serialize the whole check/unload/publish transition so a concurrent selection cannot return the engine being unloaded. uninstall.ps1: gate the default sd.cpp process stop on the owner marker so a user's own sd-server is not terminated for a directory we then keep. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reject native batch seeds outside the JSON-safe range * Refuse sd.cpp install into unowned non-empty target dir When the install target already exists, is non-empty and lacks the .unsloth-studio-owned marker (a user's own stable-diffusion.cpp checkout, or unrelated files beside a custom Studio root), install() previously still extracted the release into it. Skipping the ownership marker only stopped the uninstaller from deleting the directory; extraction still merged binaries into the user's working tree and could overwrite same-named files. Fail up front with a clear message pointing the user at a fresh/empty location before any download or extraction, leaving their directory untouched. Update the ownership test suite to assert the refusal. * Studio: gate dataset uploads on the symlink check and surface local video single-file checkpoints * Tighten comments and docstrings added by the image-generation fixes * Studio: close arbiter load-registration race and surface native progress + local pipeline folders Publish native sd.cpp generate progress (_gen) before LoRA resolution so a reload probe reads active during setup, matching the diffusers path. Register the diffusion/video GPU load under the arbiter lock (acquire_for now takes a register callback) so a competing acquire cannot evict an owner before its load is marked in-flight and let two loaders allocate VRAM at once. Admit local diffusers pipeline folders (root model_index.json, weights in component subdirs) in the local model scan so they reach task tagging and the On Device picker. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: include marker-owned custom sd.cpp roots in the uninstall stop scan; harden Pester install against the nuget.exe PSGallery bootstrap * Studio: surface local pipeline scan roots, tag single-file checkpoints by filename, mark companion-only pipelines partial - _scan_models_dir: admit a scan folder that is itself a diffusers pipeline (root model_index.json, weights in transformer/ vae/ subdirs). _is_model_directory rejects such a root, so the child scan would list the component subdirs as bogus models and hide the real pipeline; treat the root as one model via _local_pipeline_index. - _local_is_diffusers / _local_model_task: include the sole checkpoint filename in the family-detection needles (_local_family_needles, resolved via resolve_local_single_file). A generically named folder holding one loadable qwen-image-*.safetensors / ltx-*.safetensors identifies its family only from the filename; the load route already resolves that file, so tag it or the task-scoped picker (which rejects task=null) hides the on-device model. - list_cached_models: mark a companion-only base snapshot partial. A GGUF image load prefetches the base repo's VAE / text-encoder / model_index.json but skips the transformer (the GGUF supplies it); the snapshot has a pipeline manifest yet is not a loadable BF16 pipeline, and _cached_repo_partial misses it. _repo_pipeline_missing_denoiser flags a pipeline snapshot whose transformer/ or unet/ component carries no weight, so the picker drops it instead of advertising it as fully on-device. * Studio: preserve foreign gallery files, force safetensors on remote ControlNets, and close dataset/seed/GPU gaps Gallery clear/delete now scope to Studio-owned files: image_gallery and video_gallery skip PNGs / MP4s without a readable recipe (a hand-dropped or orphan file the listing already hides), so clear() and a guessed-id delete no longer destroy files the gallery never surfaced. Remote ControlNets now force use_safetensors: a bare owner/name reaches from_pretrained without the base trust gate, and the Hub scan fails open when unavailable, so requiring safetensors closes the pickle deserialization vector. POSIX uninstall now stops resident sd-server / sd-cli under an owned sd.cpp root before removing the tree (marker-gated), mirroring the Windows stop-before-delete scan; a live native server no longer survives unlinking its binary. Diffusion dataset containment: the training-start read path and the discovery picker route bare names through the protected resolver, so a symlinked dataset is rejected / not advertised like the caption/delete routes already do. Uploads gain the inference decode guard (oversized real images 400 before OOMing the trainer) and dataset upload/caption/delete/import are blocked with 409 while a diffusion run is active. JSONL readers (trainer + routes) tolerate non-object JSON and invalid UTF-8 instead of raising AttributeError / 500. LoRA family compatibility is enforced in the shared resolver, not only the picker, so a direct API client cannot apply a mismatched-family adapter. GPU arbiter gains release_if so the image/video unload idle-check and release are atomic against a concurrent same-owner load's registration. Native batch recipes persist the base batch_seed and restore replays from it, so a native batch_index>0 image no longer advances its seed twice. FLUX.2-klein selects its sd.cpp text encoder by variant (4B -> Qwen3-4B, 9B -> Qwen3-8B) instead of the single family default. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: track the loaded GGUF filename so native companion resolution reproduces the load identity * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: gate local-pipeline image tagging on a real family; validate video sidecars before delete/clear * Studio: tighten image-generation fix comments and docstrings * Studio: gate gallery serve/export on ownership; keep image progress active until persisted; reserve diffusion training before the dataset scan * Studio: restore Reapply target on async image/video load errors; recheck training state before dataset commit; reclaim partial sd.cpp installs on retry Images/Video: a background model load that fails AFTER starting (error/eviction during download) leaves the previous pipeline resident, but handleLoad had already overwritten lastLoad.current with the failed pick, so "Reapply to loaded model" reloaded the failed model. Carry the prior Reapply target into the poller and restore it on the async error/null paths, mirroring the quant rollback. Training: an in-flight diffusion dataset upload passed _require_diffusion_dataset_mutable() at entry but could still commit files after a concurrent /diffusion/start reserved the training slot, mutating the dataset underneath the trainer. Re-check the interlock immediately before the commit phase; a 409 there leaves the staged temps for the finally to clean. sd.cpp install: an interrupted extraction (disk full, killed process, a raising post-extract cudart fetch) left the target non-empty with no owner marker, so the next lazy install tripped the "not a Studio-managed directory" refusal and wedged native install. Write the ownership marker before the partial writes when the target is reclaimable, so a retry recognises the debris as ours and re-extracts. * fp8 DiT quant: floor the dynamic activation scale with activation_value_lb An all-zero activation token row makes the dynamic per-row fp8 scale 0, which turns the quantized data to NaN and the render to black frames on torchao's plain-torch kernel path. The fused fbgemm/mslk quantize kernels clamp zero rows internally, so the bug only reproduces on machines without them, which is most user environments. Zero rows are real inputs, not a corner case: Wan 2.2 zero-pads its text conditioning, and Hunyuan-1.5 and Qwen-Image regenerate zero rows inside their transformer blocks every step. Pass activation_value_lb=1e-12 to Float8DynamicActivationFloat8WeightConfig whenever the installed torchao supports the kwarg (Float8Tensor rework, 0.13+), checked via inspect.signature so older torchao keeps exactly the current behaviour; the existing Float8MMConfig fallback chain is unchanged. Verified on GPU: with the forced plain-torch kernel path a zero-row input NaNs without the floor and stays finite with it, and end to end on HunyuanVideo-1.5 fp8 goes from a solid black frame (LPIPS 1.00) to a normal render (LPIPS 0.225); on Wan the floor matches the condition_embedder exclusion (LPIPS 0.211 vs 0.206). Same-seed renders with fused kernels present are unaffected, and pre-quantized fp8 checkpoints stay valid since weight scales are untouched. * Wire hosted pre-quantized DiT checkpoints into the image families Point prequant_repos for flux.1, flux.2-klein, flux.2-dev, qwen-image (int8 only there; fp8 is family-denied), z-image and krea-2 at the unsloth/<Model>-FP8 Hub repos carrying gate-validated int8 and fp8 transformer checkpoints, so the fast quant path loads the small pre-quantized file instead of materialising the dense bf16 transformer and quantising on device. Measured on FLUX.2-dev int8: build peak drops from 60.7 GB (dense + quantize) to 30.7 GB (hosted prequant), identical 30.7 GB resident after either path since loading a checkpoint is bit-identical to on-the-fly quantisation. The hosted repos name files <Model>-<SCHEME>.pt, so resolve_prequant_source now derives that model-name filename from the repo id (scheme suffix stripped case-insensitively) and carries the legacy transformer_<scheme>.pt as a fallback the resolver tries when the primary 404s, keeping older repos loadable. Wiring a repo also exposed a fallback hazard: with a prequant source present, the dense-fit preflight used to be skipped entirely, so a failed prequant download would fall through to the dense bf16 load the memory plan never budgeted, OOMing after eviction. The preflight now always runs and gates an allow_dense_fallback flag through _load_dense_quant_pipeline: a dense misfit still skips the fast path when no prequant exists, but with one it proceeds and a prequant failure raises to the GGUF build instead of loading dense. The same flag is set when the auto-policy replans an offloaded GGUF against a prequant-sized transient. Tests updated to the new filename convention plus new coverage for the derivation and the legacy-name fallback; the prequant-skips-refit test now asserts the re-check runs and forbids the dense fallback. Verified end to end on GPU: z-image int8 resolves the hosted repo, downloads the model-name file and renders (6.8s load, 5.9 GB peak). * Route krea-2 through its per-component loader on the transformer-quant fast path _assemble_pipe used Pipeline.from_pretrained for every family, but the krea repo ships transformers-5.x configs and no top-level tokenizer files, so the tokenizer dies with vocab_file=None. The pre-quantized checkpoint loaded fine and then the assembly crashed, dropping the load to the GGUF build, which krea-2 cannot take (Krea2Transformer2DModel has no from_single_file). Assemble per-component via load_krea2_pipeline like the pipeline-kind and single-file paths already do. Verified live: Krea-2-Turbo int8 and fp8 hosted prequant loads now assemble and render through the Studio images tab. * Keep Qwen-Image's text-stream linears bf16 on int8 (short prompts break torch._int_mm) Qwen-Image's MMDiT runs every text-stream Linear at M = actual prompt tokens: the Qwen2.5-VL embeds are not padded to a fixed length like FLUX's 512-token T5. A short prompt (13 tokens) or the near-empty negative prompt drives torch._int_mm below its M > 16 floor and the first denoise step raises 'self.size(0) needs to be greater than 16, but got 13' (measured on B200 through the Studio images tab). Add per-family int8 exclusions (txt_in, add_q/k/v_proj, to_add_out, txt_mlp) for qwen-image and qwen-image-edit, threaded through exclude_tokens_for_scheme(scheme, family) and the prequant checkpoint validation, so a checkpoint baked under the old token list is rejected and re-quantised instead of loaded crashing. The text stream runs at M = tens vs the image stream's M ~ 4k, so the exclusion costs nothing; the rebuilt hosted checkpoint gates 28/28 PASS with LPIPS mean 0.057 (was 0.069). * Harden the diffusion memory plan against transient free-VRAM undercounts A cold FLUX.2-dev int8 load on an idle 183 GB B200 planned offload=model (companions exceed budget) and silently served the GGUF as-is; the identical retry went resident and engaged the hosted prequant. The plan arithmetic was byte-identical across both loads (required 90,228 MiB, resident needs free of about 124 GB); the only divergent input was torch.cuda.mem_get_info, which is device-wide and instantaneous: a transient foreign CUDA context briefly held about 100 GB at the first snapshot, and the planner trusted that single read. Three changes: - settled_snapshot_device_memory: on cuda, synchronize + empty_cache (best-effort) and take the MAX free over up to 3 spaced reads. A transient can only shrink free, so the max rejects transient undercounts while a persistent tenant still caps every read. _plan_memory now uses it. - plan_fits_total_capacity + one replan retry: when the dense/prequant candidate fits TOTAL device capacity under the standard reserve and the 0.85 resident margin, an offload verdict can only stem from the free reading, so the loader re-snapshots and replans once before declining the fast path. Explicit balanced/low_vram modes skip the retry (they offload by mode). - diffusion.transformer_quant_declined log line with required/budget/free and the plan reasons, so the next decline is diagnosable from the server log (previously silent). Verified: cold FLUX.2-dev int8 first load in a fresh server now engages the hosted prequant resident (offload=none). * Add FLUX.2 Klein and FLUX.2-dev DiT LoRA training Register flux.2-klein and flux.2-dev in the DiT trainer following the upstream DreamBooth references: latents train patchified and batch-norm normalized from the VAE posterior mode, the packed forward reuses step-invariant position ids, and the guidance vector (3.5) is gated on the variant's guidance_embeds config. Conditioning stacks load per variant (Mistral via Flux2Pipeline for dev, Qwen3 via Flux2KleinPipeline for Klein) and are encoded and freed before the transformer lands on the device. The fused single-stream to_qkv_mlp_proj joins the attention projections in the LoRA targets; the single-stream out projection stays dense because its to_out suffix would also match the double-stream ModuleList container. Wire both families through the training registry (family set, labels, VRAM notes, rank 16 / lr 1e-4 defaults, bf16-only preflight), mark them trainable with train base repos in the family registry, add FLUX.2-dev to the gated-repo token check, and trust both official bases for training downloads. Verified on B200: 30-step klein int8 (19.6s) and nf4 (20.9s) and dev int8 (52.0s) runs train with finite decreasing loss and the saved adapters apply on the bf16 base pipeline (weight 0 reproduces the base image exactly, weight 1 visibly restyles it). * Support LoRA adapters on torchao int8/fp8 quantized image pipelines Adapters are baked at load time: they attach to the dense transformer, then quantize_ converts only the frozen base linears (the lora_ side path is excluded by name), then the loader compiles. Post-quant PEFT injection is not possible on a manually quantized module, so the prequant shortcut is skipped for a baked load and the memory plan is sized for the dense build (force_dense on the quant candidate). At generation time the baked topology is frozen: weight tweaks and disabling (scale 0 reproduces the quantized base exactly) go through set_adapters, while adding or removing adapters returns a clean 400 telling the client to reload with the new selection. supports_lora now returns True for int8/fp8 diffusers loads (checked before the gguf-kind early return, since the quant fast path keeps the picker kind); nvfp4/mxfp8 and GGUF-via-diffusers stay blocked. The load request model takes an optional loras list, threaded through begin_load on both engines (native ignores it and keeps applying LoRA at generation). Verified end to end on GPU: Z-Image GGUF picker + int8 + trained adapter loads through the API, bake marker logged, weight 1.0 vs 0 renders differ visibly, weight 0.5 accepted live, unknown adapter rejected as 400. Affected suites: 296 passed. * Add FLUX.1 Krea dev to the image model catalog Krea's guidance-distilled FLUX.1-dev finetune keeps the exact dev layout, so it runs under the existing flux.1 family unchanged. Wire it up end to end: - Catalog group with the gated official bf16 pipeline and the open QuantStack GGUF quants; the gated artifact is skipped on auto-routing when undownloaded. - Trust the official repo for non-GGUF from_pretrained loads, next to the other black-forest-labs bases. - Generation defaults: 28 steps at guidance 4.5 per the model card. The generic "krea" defaults key (Krea-2-Turbo's 8-step no-CFG recipe) used to swallow the id, which would have produced garbage output; the new flux.1-krea key precedes it on both the backend table and the images page table. - The flux.1 prequant checkpoints are schnell-based; the loader's baked base_model_id validation refuses them for the Krea-dev base, so int8/fp8 requests dense-quantize instead (covered by existing prequant tests). * Resolve pre-quantized checkpoints per base variant One family entry covers several published variants whose weights differ (flux.1: schnell, dev, Krea-dev), but prequant resolution was keyed on (family, scheme) alone, so only the default base could ever be served: the loader's baked base_model_id validation correctly refused the schnell checkpoint for dev and Krea-dev bases and every such load paid the dense download plus on-the-fly quantise. Add an optional prequant_variant_repos table on DiffusionFamily as (base_repo, scheme, repo_id) triples and thread the resolved base repo through resolve_prequant_source / usable_prequant_source and their three call sites (load fast path, memory-plan probe, auto-policy candidate). A base without its own entry keeps returning the family default, preserving the existing refuse-then-dense behavior exactly. Wire the flux.1 variants: the gate-validated unsloth/FLUX.1-dev-FP8 checkpoints (built in the earlier campaign but never reachable) and the new unsloth/FLUX.1-Krea-dev-FP8. * Add the Lumina Image 2.0 family to the image catalog Alpha-VLLM/Lumina-Image-2.0 is a 2.6B single-stream DiT with a Gemma2-2B encoder and a standard 16-channel VAE, all transformers-4.x-compatible, so the generic from_pretrained pipeline path loads it as a new lumina-2 family: - Family entry (Lumina2Pipeline / Lumina2Transformer2DModel), aliased to lumina-image-2.0 / lumina-image-2 / lumina2. No bare lumina alias: Lumina-Next checkpoints are a different arch and must stay unknown rather than crash mid-load. bf16-only upstream, so the fp16 fallback stays off like z-image. - Trust the official repo for non-GGUF loads; bf16 component table entry (ships fp32, ~5.2 GB transformer + 5.2 GB encoder bf16-resident). - Generation defaults 50 steps / guidance 4.0 per the model card, and the generate call passes the card's cfg_trunc_ratio=0.25 itself (family-gated, signature-gated): the pipeline default (1.0) runs the CFG double-forward on every step and oversaturates output. - Catalog group with the single ungated bf16 pipeline artifact (11 GB resident) plus routing assertions; images page defaults row. - No GGUF artifact: none exists upstream (only finetune/LLM quants), so the dense transformer_quant fast path (GGUF-kind-only) stays unreachable for now. Offline probes of the future prequant campaign: int8 and fp8 both engage and render cleanly (fp8 LPIPS 0.11 vs bf16, int8 0.33 from 50-step trajectory drift with intact quality), so neither scheme is family-denied. * Wire the hosted Lumina Image 2.0 int8/fp8 checkpoints Gate-validated against same-seed bf16 renders (28/28 pairs per scheme, zero failures): int8 LPIPS mean 0.146 / SSIM 0.937, fp8 LPIPS mean 0.116 / SSIM 0.946. Uploaded to unsloth/Lumina-Image-2.0-FP8 following the existing checkpoint repo conventions. * Add the HunyuanImage 2.1 family to the image backend The hunyuanvideo-community diffusers mirror carries the full stack in standard layout: a 17B dual-stream DiT (32.5 GB bf16), a Qwen2.5-VL text encoder, a ByT5 glyph encoder, the 32x HunyuanImage VAE, and guider/ocr_guider components (AdaptiveProjectedMixGuidance) that diffusers 0.39 loads natively, so the generic from_pretrained pipeline path covers everything with no per-component assembly. Family notes: - The call's guidance knob is distilled_guidance_scale (there is no guidance_scale kwarg), so cfg_kwarg routes the UI value there; real CFG runs inside the repo's guider at its baked scale. Defaults follow the card recipe: 50 steps, 3.25. - 2K-native: verified live at both 1024 and 2048. - Coexists with the HunyuanImage-3.0 structured exclusion (3.0 has no diffusers pipeline and stays excluded with its stated reason). - int8/fp8 dense quantization verified live (LPIPS 0.186 both vs same-seed bf16); a short prompt does not trip the int8 torch._int_mm minimum on this arch, so no family exclude entry is needed. - bf16 component table for the memory planner: (32.5, 16.3, 0.8) GB. * Surface HunyuanImage 2.1 in the image model catalog Catalog group with the open bf16 mirror pipeline (~50 GB resident, so a bare click on a consumer card routes to the QuantStack GGUF quants, which load and render through the generic GGUF path, verified live) plus the images page defaults (50 steps, guidance 3.25 feeding distilled_guidance_scale). * Add the HiDream-I1 family to the image backend A 17B MoE DiT (16 double + 32 single layers, 4 routed experts) with four text encoders, on HiDreamImagePipeline (diffusers 0.39). One family covers the open Full / Dev / Fast repos (same arch); per-variant generation defaults follow the upstream inference recipes (Full 50 steps at guidance 5, the distilled Dev 28 and Fast 16 guidance-free). The repos name a Llama-3.1-8B text_encoder_4 in their model_index but do not ship its weights; the official example passes the gated meta-llama repo in by hand. The loader instead assembles the component from the open unsloth mirror (byte-identical weights, already inside the non-GGUF trust gate), injected at the three pipeline from_pretrained sites, with output_hidden_states matching the official example. Memory planning counts the assembled TE4: 34.2 GB DiT + 28.8 GB encoders, ~63 GB bf16-resident. * Surface HiDream I1 in the image model catalog One catalog group with the three official bf16 pipelines (Full, plus the Dev and Fast distillations as labeled artifacts) at their ~63 GB resident size, so auto-routing keeps this a datacenter-GPU pick. city96's GGUF is deliberately not wired: the GGUF path would need the same Llama TE4 assembly for very small demand. Images-page defaults mirror the backend table with the variant keys ahead of the generic hidream key. * Pin the measured HiDream quant verdict in tests int8 and fp8 both engage and render cleanly on this family, including short prompts on int8: the routed MoE expert Linears only ever see the concatenated image+text stream (M far above the torch._int_mm minimum), so no deny entry and no family exclude tokens are warranted. Assert that so a future table edit cannot silently regress the measured behavior. * Wire the hosted HunyuanImage 2.1 int8/fp8 checkpoints Verified bit-identical to on-the-fly quantize: all 1264 state dict tensors (456 quantized) dequantize equal between the loaded checkpoint and a fresh quantize_ pass, so quality matches the runtime Dtype path exactly. Same-seed LPIPS suite means (0.35 int8 / 0.28 fp8) blend trajectory divergence with this family's own run-to-run nondeterminism (identical weights and seed reproduce a 17/255 mean pixel delta through the 50-step guider pipeline); per-case hard checks pass and the drift is compositional, reviewed visually. Uploaded to unsloth/HunyuanImage-2.1-FP8. * Fix silent LoRA drop and wasted transformer prefetch on GGUF quant loads Two live-test findings on the images load path: - transformer_quant with baked LoRAs, when the dense quantized build is declined for memory or fails: the load completed as a plain GGUF with the adapters silently dropped (HTTP success, supports_lora=false after the fact) -- wrong output with no signal. The load now fails with the recovery options (drop the adapters, free VRAM, or pick a smaller model). Weight-0 adapters still count as no bake request, and the plain no-LoRA decline keeps its silent GGUF fallback. - A fresh GGUF load on a small GPU prefetched the base repo's full bf16 transformer shards (~47 GB on Qwen-Image) because the dense-quant prefetch widening only checked scheme viability, not whether the device could ever hold the candidate resident. Gate the widening on total device capacity (reserve + 0.85 margin, the plan_fits_total_capacity bar) so a card that is certain to decline the dense build never pays the download; capable devices keep the prefetch. * Fix video progress under-reporting during load and generate Two live-test findings on the video progress endpoints: - load-progress downloaded_bytes froze mid-download: the counter used scan_cache_dir, which skips in-flight *.incomplete blobs, so it sat at the last completed blob for the whole multi-GB shard pull while the disk kept filling. Count the repo's cache directory directly (completed plus incomplete blobs, snapshot symlinks skipped so nothing is double-counted). - generate-progress reported total_steps=null / fraction=0 while step advanced: the video API only carried the native total field while the image API exposes total_steps and fraction, so one poller could not work against both. Derive the image-compatible aliases in generate_progress and declare them on the response model; the native total stays for back-compat. * Wire the hosted HiDream I1 int8/fp8 checkpoints Gate-validated: all 28 per-case pairs pass per scheme (LPIPS suite means 0.291 int8 / 0.278 fp8, in the 50-step trajectory-divergence band; CLIP delta means 0.007-0.008), and the int8 checkpoint is verified bit-identical to on-the-fly quantize across all 1615 state dict tensors (1073 quantized, max abs diff 0.0). Uploaded to unsloth/HiDream-I1-Full-FP8. * Add a pre-cast text-encoder loader for the layerwise fp8 scheme The runtime text_encoder_quant=fp8 path downloads the full bf16 text encoder and layerwise-casts it in place on every fresh load. For the heavyweight encoders (LTX's Gemma3-27B ~50 GB, FLUX.2-dev's Mistral-24B ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) that download dominates load time on a fresh machine. diffusion_te_prequant.py loads a pre-cast fp8-storage state dict instead: meta-init the encoder skeleton from the checkpoint's te_class, load_state_dict(assign=True), rebuild on CPU if non-persistent buffers stay on meta, then re-apply the same layerwise cast to install the upcast hooks. The cast is a deterministic storage transform, so the loaded encoder is bit-identical to dense-load-then-cast by construction. v1 hosts the layerwise fp8 storage scheme only: its state dict is plain tensors (torch.load(weights_only=True), no pickle execution). The dynamic-compute schemes (fp8_dynamic, int8, nvfp4) build torchao subclass wrappers at runtime and are deliberately not hosted. Checkpoints validate format, scheme, component and base_model_id before use and any problem falls back to the dense download and cast. Local path overrides reuse the DiT prequant allowlist env var. Families opt in via a new te_prequant_repos (scheme, component, repo_id) field on both DiffusionFamily and VideoFamily; the field defaults empty so nothing changes until a gate-validated artifact is wired. * Inject hosted pre-cast text encoders during pipeline assembly Wire te_prequant_pipe_kwargs into the three pipeline assembly sites: the diffusion full-pipeline branch, the diffusion transformer-only and GGUF branch (where the companion TE is the big remaining download), and the shared video assembly path before the pipeline/component split. Injection is gated exactly like the runtime cast (mode normalized to fp8, device supported, family not denied), so it can never engage where quantize_text_encoders would not; the later quantize_text_encoders call re-applies the cast idempotently and keeps status reporting truthful. With no hosted checkpoint configured the call returns {} and assembly loads the dense encoder as before. * Add the pre-cast text-encoder checkpoint builder Applies the runtime layerwise fp8 storage cast to a model's dense text encoder once and saves the cast state dict with baked metadata (format tag, base_model_id, family, scheme, component, te_class, versions) in the layout diffusion_te_prequant.py validates. Resolves the encoder class from the checkpoint's config.architectures so the recorded te_class matches what the pipeline instantiates. CPU-runnable: the cast touches storage dtypes only. * Test the pre-cast text-encoder load path Hermetic CPU coverage for diffusion_te_prequant: the checkpoint filename convention, family-table resolution by scheme and component with malformed entries skipped, resolution priority (path override, hosted repo, none) and the fp8-only scheme gate, the checkpoint validation matrix (wrong format, missing state_dict, wrong scheme, wrong component, wrong or missing base_model_id) with base case folding, the local-path allowlist refusal and missing-file fallback, and the assembly injection gating (mode, hosted entry, device support, family deny, load failure, successful injection). Also pins the te_prequant_repos field on both family dataclasses and that no family ships a hosted TE checkpoint until the campaign wires one. * Fix pre-cast TE checkpoint loading and engagement reporting Two bugs found while building the hosted checkpoints: - The builder recorded torch.__version__ (a TorchVersion object) in the checkpoint metadata, so torch.load(weights_only=True) rejected every artifact and the loader silently fell back to the dense download. Record plain strings. - Re-applying the layerwise fp8 cast to an injected pre-cast encoder raised on the duplicate hook registration, making quantize_text_encoders report the engaged cast as failed (status showed no TE quant while the encoder ran fp8). _cast_fp8 now returns early when the hooks are already installed. Also corrects the LTX TE size note: Gemma3-12B stored fp32 (~49 GB), not 27B. * Wire the hosted pre-cast fp8 text encoders qwen-image and flux.2-dev (diffusion) and ltx-2 (video) now resolve a hosted pre-cast fp8 text encoder from their unsloth -FP8 repos: - unsloth/Qwen-Image-FP8: Qwen2.5-VL-7B, 16.6 GB dense -> 8.8 GB - unsloth/FLUX.2-dev-FP8: Mistral-Small-24B, 48.0 GB dense -> 24.7 GB - unsloth/LTX-2-FP8: Gemma3-12B, 48.7 GB fp32 store -> 13.2 GB Every checkpoint verified bit-identical to dense-load-then-cast (729 / 585 / 1066 tensors, zero mismatches) and smoke-tested through the real backends with the repo engagement marker. Tests cover the wired entries, the resolver filenames, builder metadata weights_only survival, and the idempotent re-cast. * Report the compute dtype on fp8-cast encoders and inject the pre-cast TE on the dense fast path Two more findings from the hosted-TE GPU smokes: - Module.dtype reports the first floating parameter, which after the layerwise fp8 cast is the fp8 STORAGE dtype. Flux2 derives its prompt embed and latent dtypes from encoder.dtype and feeds them to randn_tensor, which has no fp8 kernel, so ANY flux.2 load with text_encoder_quant=fp8 crashed at generation (pre-existing, runtime cast included). The cast now swaps in a subclass whose dtype property reports the compute dtype; forward behaviour is unchanged. - The dense transformer_quant fast path assembles companions through _assemble_pipe, which never received the pre-cast TE injection, so the hosted encoder engaged on full-pipeline and GGUF builds but not on the fast path. Threaded through like the other two branches. Verified live on B200: qwen-image (full pipeline), flux.2-dev (GGUF picker with int8 DiT prequant), ltx-2 (video backend) all engage the hosted TE, render non-black, and report text_encoder_quant=fp8 truthfully. * Key the fp8 cast idempotency on an explicit completion marker Hook presence alone cannot distinguish a legitimately pre-cast text encoder from leftover hooks after a cast that failed mid-pass, so the early return now requires the completion marker _cast_fp8 sets once the hooks are fully installed. Leftover partial state keeps failing closed. Also tolerates non-Module encoder doubles in the hook probe and the dtype override. * Extend the fp8 TE quant to HiDream's Llama text_encoder_4 The generic quantize_text_encoders pass only covers text_encoder.._3, so HiDream's HEAVIEST encoder (Llama-3.1-8B TE4, 16.1 GB bf16) always stayed dense. TE4 is assembled separately (hidream_te4_kwargs), so the fp8 path now lives there: when the requested TE quant is layerwise fp8 and the device/family qualify, TE4 prefers the hosted pre-cast checkpoint (unsloth/HiDream-I1-Full-FP8, 8.6 GB) and falls back to dense-load-then- cast; a mid-pass cast failure reloads a fresh dense encoder instead of shipping partial state. The pre-cast loader and builder gain config_subfolder/config_overrides for standalone encoder repos whose config sits at the root and whose pipeline needs forward flags (output_hidden_states/attentions). Verified on B200: bit-identity 291 tensors (225 fp8, 0 mismatches), hosted checkpoint engages through the real backend (marker + status fp8), load 24.3 s vs 48.0 s dense, LPIPS 0.133 mean over 3 same-seed pairs vs the dense-TE render (gate 0.25), non-black frames. * Correct the ltx-2 resident TE estimate to the bf16 cast size The memory plan's bf16_components_gb held 50.4 GB for the LTX text encoder, which is the fp32 hub store of Gemma3-12B (~49 GB download), not what sits on device: the pipeline loads it torch_dtype=bf16, ~24.4 GB resident. The 26 GB over-estimate pushed the auto plan toward offload on cards that fit the real footprint. Comments and the size-table test now pin the resident semantics. * Host pre-cast fp8 text encoders for four more families Round 2 of the hosted TE set, each bit-identical to dense-load-then-cast and gated through the real backend (marker + status fp8 + same-seed LPIPS vs dense TEs): - FLUX.1 T5-XXL (text_encoder_2): 9.52 -> 5.90 GB, one artifact for schnell/dev/Krea-dev (T5 shards byte-identical across all three, verified sha256). 220 tensors, 144 fp8, LPIPS 0.109. - Lumina Gemma2-2B: fp32 hub store 10.46 -> 3.20 GB (3.3x download cut). 288 tensors, 182 fp8, LPIPS 0.041. - Z-Image Qwen3-4B: 8.04 -> 4.41 GB. 399 tensors, 252 fp8, LPIPS 0.112. NOT shared with flux.2-klein-4B: klein retrained layer 35's MLP (verified tensor diff, maxdiff 0.86), so klein hosts no entry. - Krea-2 Qwen3-VL-4B: 8.88 -> 4.83 GB. 713 tensors, 460 fp8, LPIPS 0.082. The constructor-assembled krea pipeline takes the encoder directly (load_krea2_pipeline text_encoder kwarg); the loader remaps 5.x rope_parameters and re-ties weights after assign so the rebuilt encoder matches the builder's structure. HunyuanImage 2.1 reuses the Qwen-Image artifact outright: its Qwen2.5-VL text encoder is byte-identical (every shard sha256, 16,584,414,544 bytes), recorded in the new component-level base-equivalence table the checkpoint validator consults. The injection loop now covers text_encoder.._3 so a family can host several components. Live check: LPIPS 0.123 vs dense. * Report the fp8-cast compute dtype without swapping the encoder class The dtype override swapped encoder.__class__ to a dynamic subclass, which breaks transformers' kwargs-based output recording: a fp8-cast Qwen3VLModel stopped returning hidden_states and every krea-2 generation with text_encoder_quant=fp8 crashed at encode_prompt (regression from the HiDream TE4 change; caught by the krea hosted-TE live smoke). The override is now a property shadowed on the ORIGINAL class that prefers a per-instance compute-dtype attribute, so class identity is preserved and uncast instances keep the stock behaviour. The idempotency test now pins exact class identity and the uncast-sibling fallback. * Pass the calibrated distilled sigma curve to LTX-2.3 8-step runs The 22B distilled DiT was trained against ltx_core's fixed DISTILLED_SIGMA_VALUES, but the diffusers scheduler derives 8-step spacing from resolution-shifted flow matching and lands far off at every reachable mu (second sigma 0.945-0.981 vs 0.99375, tail 0.37-0.61 -> 0.1 vs 0.725 -> 0.42 -> 0). At the distilled default step count the backend now passes the list verbatim, neutralising the scheduler's dynamic shift and terminal stretch for the call (they distort even explicit sigmas) and restoring them afterwards. Other step counts and the dev/base DiT keep the scheduler's own spacing. Live-verified on B200 through the video branch backend: the scheduler holds the exact curve after an 8-step distilled GGUF generation, config restored, healthy clip. Also reword the transformer_quant resolved reason to the measured reality: quant halves resident weights and hosted checkpoints cut load time, while per-step speed is roughly bf16 parity. * Pin the fp8 weight-quantize kernel against silent MSLK switching torchao's Float8Tensor KernelPreference defaults to AUTO, which switches the weight-quantize kernel to MSLK whenever an mslk package is importable on sm90+. Measured on B200: that changes fp8 scale rounding bitwise (8/8 FLUX matrices differ, scales ~55 percent of bytes), so a box that merely gains mslk would break the hosted-prequant bit-identity invariant; the mslk path is also slower under torch.compile (opaque extern call blocks inductor's quantize fusion, FLUX.1 fp8 e2e 1.149 to 1.624 s). Pin KernelPreference.TORCH explicitly, matching current no-mslk behaviour bit for bit; signature-gated for older torchao. GPU-smoked (finite, rel err 0.037) and pinned by test. * Shift Qwen-Image training sigmas to the inference distribution Qwen-Image's scheduler skips its static shift under use_dynamic_shifting, so the DiT trainer was drawing UNSHIFTED uniform-schedule sigmas for it (mean sigma 0.50) while inference always runs the exponential mu = log 3 shift plus the shift_terminal 0.02 stretch. Add a flow_shift config lever: "auto" (the new qwen-image default) rebuilds the training sigma table through the scheduler's own time_shift and stretch_shift_to_terminal so the draw matches the inference distribution exactly (mean sigma 0.72); a numeric value applies the standard linear shift s*u/(1+(s-1)*u); 1.0 keeps the historical identity behavior and stays the default for FLUX, Z-Image and Krea 2. The model timestep conditioning follows the shifted sigma, gathered in fp32 so bf16 rounding never skews it. Also wire two opt-in levers with off defaults: cfg_dropout (per-sample empty-prompt conditioning dropout, encoded alongside the captions before the text encoders are freed) and weighting_scheme="bell" (bsmntw-style mid-schedule Gaussian loss weighting normalized to mean 1). Verified with two 80-step rank-8 bf16 LoRA runs on Qwen/Qwen-Image (identity vs auto, same seed): both converge with finite decreasing loss and produce coherent same-seed previews. Unit tests cover the exact transform, the shifted sampling distribution, per-family defaults and config plumbing. * Add LoRA EMA, a persistent conditioning cache, and aspect bucketing helpers diffusion_train_extras hosts the opt-in training extras: LoRAEMA shadows only the trainable adapter params (warmup-ramped decay, default 0.99, exported as a second adapter under output_dir/ema), PersistentConditioningCache stores latent posterior stats and caption embeddings as safetensors keyed by content hash + family + resolution, and the aspect-ratio bucketing helpers group mixed-aspect datasets into same-area divisor-snapped shapes. The DiT trainer wires the first two behind config flags that default to the current behavior: ema_decay (0 disables) and cond_cache_dir (None disables). A fully warm cache skips loading the VAE and text encoders entirely; a cache hit is bit-identical to a fresh encode, including the per-channel qwen latent normalization. Also fixes the stale _gather_sigmas call in the perf test that still passed the scheduler instead of the sigma table. * Tighten torchao configs and note the FSDP2 design for the DiT trainer nf4 loads now enable double quantization (~0.4 bits/param off the frozen base scales at no fidelity cost), fp8 training uses the rowwise recipe when the torchao build ships it (per-row scaling confines the DiT activation outliers that a tensor-wide scale collapses), and the inference quant filter gains a per-scheme GEMM-tiling divisibility floor (16 for scaled_mm, 32 for MX blocks) so one ragged Linear cannot crash the first denoise after a clean quantize pass. plans/fsdp2_diffusion_design.md records the multi-GPU design: bf16/fp8 over FSDP2 with per-block units, LoRA attached before sharding, int8 out of scope (DTensor over the quantized subclass is undefined), per-family notes. * Batch diffusion inference with per-image seeds, an inference conditioning cache, and GGUF loader fixes Batched generation: /images/generate takes a prompts list (one image per prompt, txt2img only) or a seeds list (one prompt, one image per seed); the legacy batch_size path derives per-image seeds base..base+n-1 like the native engine. Every image gets its own torch.Generator so any batch member replays alone from its gallery recipe; the whole list runs as one forward by default with OOM backoff that halves a failed chunk, and an explicit batch_size caps images per forward. Validated 10-22x over serial engines on 32-image suites with LPIPS deltas within 0.002. Conditioning cache on the inference path: UNSLOTH_DIFFUSION_COND_CACHE_DIR (the inference sibling of the trainers' cond_cache_dir, same persistent store) wraps encode_prompt so repeated prompts skip the text-encoder forward entirely; verified bit-identical outputs. Bypassed while LoRA adapters are attached; tensor-argument calls pass through uncached. Compile cache: GGUF loads fingerprint their own bundles (quant=gguf, a different compiled graph than the dense family) and batched calls register every distinct (w, h, batch) chunk shape they ran, so the heavy GGUF batched warmups (~159 s at batch 32 on 12B-class, ~655 s on 20B CFG-batched) are paid once ever. GGUF loader: strip the sd.cpp model.diffusion_model. container prefix in the single-file converter; diffusers' FLUX.2 converter KeyErrors on it and the Qwen-Image identity mapping strands the model on meta. * Correct batched seed-replay docs to match measured behavior Same-seed images at the same batch shape are bit-identical; a solo regeneration with the recorded seed matches its batched rendition up to batch-size-dependent kernel numerics (mean abs pixel delta about 2.5/255, LPIPS delta under 0.002), not bit-exactly. The previous wording overclaimed bit-identity across batch shapes. * Note that batched bit-identity assumes a settled compiled graph The first generation issued while the deferred compile is still in flight can deviate transiently (observed once on a cold fp8 build: mean abs pixel delta 0.063/255); once the graph is settled, same-seed same-batch-shape images are bit-identical across runs. * Studio sidebar: Image03/FlimSlate icons, More flyout, Train row, New pills - Images uses Image03Icon and Video uses FlimSlateIcon. - New "More" row (MoreHorizontalIcon) opens a right-side flyout on click or hover holding Video, Recipes and Export; the close is delayed 180ms so the pointer can cross the gap. Its SidebarMenuButton deliberately takes `title` rather than `tooltip`: with `tooltip` the button returns a Tooltip root and DropdownMenuTrigger asChild would hand its ref to a non-DOM node. - Dropped the "Train" section heading; Train is now a top-level row between Images and More. data-tour="navbar" moves to the surviving nav group so the product tour keeps its anchor. - "New" pill beside Images and (inside the flyout) Video, via NavBadge. * Studio sidebar: match flyout rows and New pills to the existing scales - More flyout rows dropped their sidebar-row typography and size-icon override, which fought DropdownMenuItem's own scale (text-sm, gap-2.5, px-3 py-2 and size-4 icons) and rendered oversized glyphs and text next to the nav. - New pill reuses the brand "beta" badge recipe (nav-badge font, --ui-font-scale sizing, nav token colours) rather than hardcoded 9px values. - The More row's native title tooltip (an OS box on hover) is replaced by the app's Tooltip, wrapped around DropdownMenuTrigger so both triggers compose onto the same button, and shown only on the collapsed rail like other nav rows. * Settings: pin and reorder the sidebar navigation Adds a "Sidebar navigation" section to Settings -> Appearance, above the existing profile-menu customizer, with the same drag-to-reorder + switch UI. - New sidebarNav preference: one { id, pinned } entry per navigable row (projects, hub, images, train, video, recipes, export), array order = render order. Defaults match the shipped layout, so an untouched install is unchanged. - Unpinning moves a row into the More flyout rather than hiding it, so no page becomes unreachable. New chat and Search stay fixed as actions. - app-sidebar now renders from one navRows descriptor map, so a pinned row and its flyout counterpart cannot drift; the More row appears only when something is unpinned and highlights off whatever it actually holds. - Mirrored in the backend PersonalizationCustomization: without it the model's extra="ignore" would drop the field, and because sync replaces local state with the server's copy once customization is saved, the user's pin order would reset on the next sync. The validator dedupes and back-fills like sidebarMenu but preserves the client's order, since here order is meaningful. Frontend typecheck, i18n parity and catalog checks pass; 32 personalization tests pass, including a round-trip asserting a reordered list survives a save. * Sidebar customizer: drop the Search row, skip More for a lone item - Search is reached from the top bar, so it is no longer previewed as a fixed sidebar nav row; New chat stays. - More now appears only when it would hold two or more rows. A single unpinned row renders inline in its saved order position instead: a flyout wrapping one item costs a click and earns nothing. The customizer's More preview follows the same threshold. * Sidebar settings: hide a lone unpinned tab, match New chat icon, rename Profile menu - With exactly one tab unpinned, both More and that tab are dropped, so nothing is drawn for it (previously it rendered inline). The page stays reachable by URL. - The customizer's New chat preview uses PencilEdit02Icon, the icon the real row renders; Edit03Icon was a different glyph. - "Sidebar menu" is now "Profile menu", described as the shortcuts behind your name at the bottom of the sidebar, so it no longer reads as a second name for the navigation section above it. * Tighten comments in the new sidebar and delete-guard code * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio sidebar: keep the More row highlighted while its panel is open Moving the pointer into the flyout left the row unhighlighted while the panel stayed open. The row now carries data-menu-open, added to the nav hover selector list. Not data-state: the tooltip and menu triggers both write that attribute, so whichever lands last wins. * Images: use the shared pill toggle for Create/Train and pad the panels - Create/Train was the only segmented control on its own Tabs styling. It now uses PillTabs, the same control as the model picker and Hub toggles, pinned to the header row's 34px. PillTabs takes an icon per tab, so the inline-span workaround for TabsTrigger goes away. - pt-3 on both the Create and Train panels, which sat flush against the model selector row. * Images: make the workflow picker a dropdown instead of a 7-up strip Seven workflows in a 340px rail left ~48px each, so the labels crowded and the hints were only reachable as title tooltips. The strip is now a dropdown: the trigger shows the current workflow and its hint, and each row carries its own description. A row the loaded model can't run is disabled and shows the reason in place of the hint, so the gating explains itself. Adding a workflow no longer shrinks the others. * Images: workflow icons, hint under the trigger, more top room, unclipped Train cards - Each workflow carries an icon, shown on the closed trigger and on every row. - The trigger is one line (icon plus name). The selected workflow's description moved below it, where it reads like the Field hints further down the rail. - pt-6 instead of pt-3 on both Create and Train, so the cards clear the model selector row. - The Train right column scrolls while its cards use ring-1, which draws outside the box and was clipped at the scroll edges. p-px gives the ring room. * Images: one-line workflow rows, stronger trigger fill, roomier mode tabs - Dropdown rows are icon plus name only. The selected row's description already shows under the trigger, and a disabled row keeps its reason as a title. - Trigger fill moves to the bg-foreground/[0.07] dark:bg-foreground/[0.12] pair the hub cards use, so it reads against the card in both modes. - Description under the trigger goes from text-ui-10 to text-ui-11p5. - More horizontal padding on the Create / Train tabs. * Images: drop card borders for the composer shadow, keep scrollbars inside, use app controls in Train - Cards lose ring-1 for .panel-soft-surface: the composer's shadow in light, flat in dark, matching .chat-composer-surface and the menus. - Both rails now clip (overflow-hidden) with the scroller inside, so the scrollbar can't ride over the rounded corner. Same shape video-page already uses. - Train's 9 native selects become the app Select, so they no longer open an OS-native menu, and the native file input is hidden behind a Choose images button that reports the count. - Image previews use explicit 8-10px radii: this theme sets --radius to 1.1rem, so rounded-md was 15.6px and the thumbnails read as circles. * Images: one card for controls and preview, chat sliders, wider softer shadow - Controls and preview were two floating cards; they now share one card split by a divider. The Advanced dock stays separate since it toggles. - SliderField wraps Chat's ParamSlider, so the sliders match Chat (label row with the value, full-width neutral track) instead of a green track with a spin box. All 14 call sites keep their props. - panel-soft-surface goes from 0 2px 8px -2px /0.16 to 0 4px 22px -6px /0.10: lighter, spread wider. * Images: flat Create and Train panes, hover-only scrollbars, tidier Train dataset step Both Images tabs now sit on the page background like the Hub: no card, no shadow, no bounding box. A single rule divides the controls rail from the preview canvas (Create) and from the run area (Train), and the settings and previous-runs sections read as panes rather than nested cards. Also: - Scrollbars in these panes use the existing hover-scrollbar recipe, so the thumb only shows while the pane is hovered. - Workflow rows explain themselves with a tooltip after a short hover, which also works on disabled rows, and the descriptions are much shorter. - Training images rows are name plus image count; the license stays on the example card. - The upload step loses its dashed box, the buttons match the sizes around them, and Upload only appears once files are picked. - The empty preview uses the same icon as the Images nav item. * Images: full-height panes, wider settings rail, Create/Train offset from the selector The rule between the panes now runs the whole page height (the row drops its bottom padding and each pane pads its own content), the settings rail is wider on both Create and Train, and the Create/Train switch sits further right of the model selector. * Images: put both tabs on the Hub's centered measure Top bar and content now share mx-auto max-w-1100 with px-5 / sm:px-8, so Create and Train sit at the same width and position as the Hub instead of running edge to edge. * Images: restore the top bar position, drop the panes lower under it * Images: center the mode switch, flip the arrow with the orientation, app tooltips everywhere The Create/Train switch is centered on the page instead of trailing the model selector, with wider buttons. The flip control's arrows now rotate with the orientation and its label says which way the flip goes. Every native title tooltip on the page is now the app's tooltip, so they all get the rounded surface instead of the OS box. * Images Train: plainer field text, no green buttons, columns that stop colliding - The dataset name, trigger prompt, adapter name and custom base fields now say what they are in plain words instead of leaning on example values. - Import, Upload, Back, Back to settings and Train another are outline buttons, not green ones. - Example thumbnails are landscape tiles, so photos are not cropped to chunky squares. - Settings cells get min-w-0 and the select value truncates, so a long option like the nf4 label no longer widens its column into the next one. - The number stepper sits a little further in from the field edge. - Create and Train are wider. * Images Train: roomier example cards with Import on the thumbnail row * Video: same treatment as the Images tabs - No cards: the rail and the canvas sit on the page background, divided by a rule that runs the full page height, on the Hub's centered measure. - Wider rail, chat's sliders, hover-only scrollbars. - Every native title tooltip is now the app's tooltip, including the clip cards. - Reapply and Cancel are outline buttons, the empty state uses the Video nav icon, and the clip tiles are less rounded. * Images and Video: narrower generation rail, matching Train headings Create and Video rails go from 392px to 368px. Train a LoRA and Training settings are now the same size and both in the heading font: the h2 already picks it up from the base rule, so the settings header opts in with font-heading and the weight that rule pins. * Images Train: shorter copy throughout Family notes, example descriptions, precision labels and every helper line are trimmed so they stop wrapping to three lines and colliding with the next column. The nf4 label now fits its select without truncating. * Images Train: a little more spacing between field groups * Images and Video: tighten code comments * Fix training start NameError, the load-order guard test and CPU-only diffusion tests - start_training forwards resume_source_run_id to _start_training_impl, which reads it. Without it every start raised NameError. - Restore main's anchor in the load-marker order test: the file now has an earlier `if config.is_gguf:`, so indexing the first one compared the wrong branch. - The two diffusion tests that reach diffusers now skip when it is absent, matching the CPU repo-test env. - The UI smoke finds nav rows that live in the sidebar's More flyout. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Treat a null metadata caption as no caption str(None) stored the literal "None" as the caption, so a null row counted as captioned and would have trained on that text. Also drops an unused import. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix invalid-UTF-8 500s, the flat Canny map and the dropped DiT knobs read_text raises UnicodeDecodeError, which is not an OSError, so one bad caption sidecar or video sidecar 500d the info, upload and gallery routes. A flat image now yields the all-black edge map instead of its own luminance, and the four DiT loss knobs the trainer implements are declared so model_dump keeps them. * Use the ui font-size tokens instead of raw px text utilities text-[11px] and friends ignore the UI font size preference, which the repo's font-scale contract test enforces. Same rendered size at the default scale. * Fix diffusion dataset 500s, the dropout-1.0 no-op run and the reset base pick Four correctness fixes on the training side: - The labeling grid read caption sidecars under except OSError, but a non-UTF-8 sidecar raises UnicodeDecodeError (a ValueError), so one bad file 500d /diffusion/dataset/{name}/images and the grid could not be opened to repair it. Read it as no caption, matching the info summary. - An image past Pillow's own hard limit raises DecompressionBombError, which derives straight from Exception and so escaped the upload guard's (OSError, UnidentifiedImageError, ValueError) and returned 500 instead of the intended 400. - lora_dropout accepted 1.0, which makes PEFT build nn.Dropout(p=1.0): lora_A and lora_B receive no gradient and the run saves an untrained adapter while reporting normal progress. Bound it below 1.0, matching the LLM request schema. - The train panel re-seeded the base repo on every dataset refresh because the family object identity changes on each info fetch, so an upload or caption save silently replaced the user's chosen base and the run started on a different model. Track the pick and only re-seed on a real family change. * Show the retained failure when a video page mounts after a failed job Mount-time recovery handled only phase=completed, so reloading the page after a multi-minute generation failed left an idle view with no diagnosis: the backend keeps the terminal failed record only until the next job, and nothing else survives the reload. Surface it the same way the poll does, filtering the cancelled sentinel. * Fix batched generation crashes, cache keying and unreplayable recipes Four bugs in the batched inference path, all found by review: - A mixed-prompt batch sent a scalar negative prompt against a prompt list. Z-Image asserts on the length, and Qwen-Image, Krea 2 and FLUX true-CFG encode a batch-1 negative against batch-N latents and fail in the transformer's text/image concat. Broadcast it to match the batch. - The FBCache step-cache reset sat above the chunk loop. diffusers only resets that state at the end of a successful call, so a forward that raised (the OOM the backoff is meant to recover) left its own residual behind and the halved retry died on a shape mismatch. Reset before every forward instead. - The conditioning cache keyed on the checkpoint alone, but a GGUF or single-file load takes its text encoders from the companion base, so the same checkpoint against a different base reused the previous base's embeddings. Key the base too. - Gallery records stored the base seed and the requested batch size even when a prompts/seeds list drove the run, so restoring the second image of seeds=[5, 99] replayed seed 5. List-driven outputs now record as single-image recipes on their own seed. Also bound strength above 0: every img2img pipeline derives its step count from it, so 0 leaves zero denoising steps and either raises or, on SDXL, crashes on empty latents. * Fix quantized-load LoRA bake, prequant family exclusions and outpaint canvas Six review findings across the Images page and model scanning: - The quantized (int8/fp8) load path can only attach LoRA adapters before quantization, but the frontend load request had no loras field, so every generation after such a load was rejected and each reload repeated it. Send the selection with the load. - build_prequant_checkpoint passed no family to the scheme exclusions while recording the family in metadata, so a Qwen int8 artifact baked the short-M text-stream linears and was then rejected wholesale by the loader's family-keyed check. - Registering a bare single-file checkpoint directory produced no On Device row even though the images loader can load it; only its parent worked. Admit that shape when nothing else matched. - Unload left the Reapply target set, so the repair path was skipped and Reapply reloaded the ejected model. Clear it, as the video page does. - Both FLUX.2 bases were trusted for training but not inference, so Deploy to Create rejected every FLUX.2 adapter. - Outpaint allocated the grown canvas before downscaling, exceeding the browser canvas area cap on a large photo; an over-cap canvas is unusable, so Extend silently posted a fully transparent image and mask. Scale the source first. * Send the picked GGUF filename with the quant so diffusion loads fire The variant expander emitted only the quant label, and nothing else in the frontend set ggufFilename, so the Images and Video pages could never take their GGUF branch: both gate it on meta.ggufVariant and meta.ggufFilename, then fall through to the single-file path, which returns because the id is a repo id and not a .gguf name. Every quant pick was a silent dead click, with no load request reaching the backend. The filename was already on the variant row (the picker keys its list on it, and the variant validator requires a non-empty string), so thread it through the click handler. The chat path is unaffected: it reads ggufVariant and never needed the filename. * Version the conditioning cache key and reject non-finite flow_shift Two correctness fixes: - The cache keyed the checkpoint and its companion base by name only, so a Hub repo advancing to a new commit, or a local directory updated in place, kept returning embeddings from the previous text encoder. Pair both with a revision marker: the locally resolved commit sha for a Hub repo, config plus text-encoder file stats for a directory. Neither loads the encoders, so a warm run still keeps them off the GPU. - flow_shift only checked positivity, but JSON accepts 1e309, which floats to inf, and inf <= 0 is False while NaN fails every comparison. The sigma table then evaluates s * u / (1 + (s - 1) * u) as NaN, which poisons every sampled sigma and saves a corrupted adapter while progress looks normal. Require a finite value. * Keep curated models listed, guard the video companion repo, pin diffusers Three review findings: - The picker filtered every catalog member out of Recommended and Hub search on the way to canonical group rows, but nothing renders those rows yet (catalogGroupFitsDevice and groupMatchesQuery are imported and unused). A task-scoped picker's models list is catalogToModelOptions(), i.e. group members exclusively, so both lists came back empty and no curated model could be discovered or downloaded. Keep the artifacts listed until the grouped UI exists. - The video delete guard compared only repo_id, so deleting the companion base of a loaded GGUF video model was allowed even though it supplies the VAE and text encoders. Compare base_repo too, matching what the images guard already does for its companions. - diffusers was declared unversioned while the diffusion stack requires 0.39 (Krea2Pipeline, the cache_context child registries, the Flux2 and Z-Image pipelines), so an upgrade could keep an older release and selecting an advertised model failed until the user upgraded by hand. * Namespace the trainer conditioning cache per checkpoint, bound the learning rate - The trainer keyed its persistent conditioning cache on family and resolution only, while the keys themselves carry just the caption or image content and crop variant. One cache directory reused for two checkpoints, or for the same repo at a new revision, let a warm run skip loading its encoders and train on the other model's embeddings and latent statistics. Namespace on the base checkpoint and its resolved revision as well. The revision helper now lives beside the cache in diffusion_train_extras and the inference wrapper delegates to it, so the two cannot disagree about what counts as the same source. - The diffusion learning rate only checked positivity, but 1e309 floats to inf and satisfies gt, so the route evicted the resident models and started AdamW with an infinite rate: the first step destroys the adapter while progress looks normal and the result is saved. Bound it below 1.0, matching the LLM schema, which rejects inf for the same reason. * Fix GGUF image model picks doing nothing, and pick the train base in the top bar The quant rows never forwarded the .gguf filename, so every hub GGUF pick on Images/Video fell through to a silent return. On Train the top bar now picks the training base instead of a generation model, which is GGUF-only and untrainable. * Pin diffusion and video loads to the live HF cache root Both read huggingface_hub's import-time HF_HUB_CACHE, which changing the cache folder does not update: progress counted the old root while the download wrote to the new one, and from_pretrained could split one model across both. * Add the diffusion download plan endpoint Reports the repos and exact files a pick needs so the download manager can stage them with the loader's own file scope. A plain snapshot would add the packaged root single, transformer shards and fp16 twins the loader never opens. * Add a file-scoped flavour to the Hub download job Lets a consumer that reads a deliberate subset of a repo stage it through the normal download manager. Keyed as "@scope" so it never collides with a quant or with the repo's full snapshot, and the file list rides the registry so an XET to HTTP retry respawns the same scoped job. * Stage image and video downloads through the Hub download manager They downloaded inline inside the load, so they had none of the manager's disk preflight, manifest verification, resume or panel progress. Picks now stage as scoped jobs carrying the loader's own file list, then load from a warm cache. * Fetch staged GGUF checkpoints as scoped jobs, and stop calling diffusion models unsupported A GGUF entry went out as a full snapshot, whose ignore list drops *.gguf: the job finished at once having fetched only docs, and the repo landed on device unloadable. Every entry is scoped now. The Hub also no longer tags image/video models as unsupported (they run on their own pages), and those pickers name what they select. * Apply the picker task filter to local model sections LM Studio, ./models and custom-folder rows ignored it, so the Images picker listed chat GGUFs that 400 on a diffusion load. The backend already tags every local model with a task for this purpose. * Route a chat pick of a diffusion model to the Images or Video page Chat cannot load one, so it was either hidden or failed on load. The unfiltered picker now lists on-device diffusion models and navigates to the page that runs them, passing the repo and quant so that page loads it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Route the real GGUF filename, keep non-GGUF curated models, key scoped downloads by file set Five review findings, four of them ways a click did nothing or fetched the wrong thing: - A chat pick of a diffusion model routed ggufVariant (a label like Q4_K_M) in the search param the target page uses verbatim as the GGUF filename, so the load asked for a file that does not exist. Route ggufFilename; no filename means a curated non-GGUF pick, loaded as a pipeline. - The task-scoped pickers kept only GGUF repos, so the catalog's bf16, bnb-4bit and single-file fp8 artifacts could not be discovered or downloaded on the Images and Video pages even though loadSpecFor knows how to load them. Keep curated artifacts whatever their format, in Recommended and in Hub search. - Both pages deduplicated routed selections on the model alone, and they now stay mounted, so picking the same repo again -- another quant, or the same one after chat evicted it -- returned early without loading or clearing the query string. Key on model and quant. - Every scoped image download shared one @diffusion job key regardless of the requested files, so switching quant mid-download adopted the running job: the UI waited on the first file set, then loaded a file that was never fetched. Include a digest of the file set in the key. - A scoped plan silently dropped requested files missing from Hub metadata, and snapshot_download succeeds when an allow pattern matches nothing, so the job reported completion and triggered a load with required files absent. Fail the job instead. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the scoped download key derivable, and stop the hidden page hijacking a route Four review findings, the first a regression from my own last commit: - Keying scoped download jobs by a digest of the file set broke the download manager: it builds that key client-side (it polls and cancels before any response tells it a key), so it watched and cancelled a key no worker owned and never fired its ready callback. Keep the derivable "@scope" key and refuse the second request instead when a live job on the slot is fetching a different file set -- decided inside the registry claim, under the lock, so a concurrent claim cannot slip past it. The manager records the file set on the job as well, so a sibling quant's transfer is not adopted locally either. - Both diffusion pages read the route query through a loose useSearch and both stay mounted once visited, so the hidden one consumed the other's ?model=: it navigated back to its own route and tried to load, say, an image checkpoint as a video model. Only the visible page consumes it. - The staged download plan was built without the configured HF token or the Advanced values the load itself sends. The token matters most: the backend's Hub metadata lookup is best-effort, so a gated base silently planned no companion entry and the load pulled those multi-GB files inline, outside the manager. The memory/quant controls decide whether the base transformer/ shards are needed at all, and the route dropped memory_mode, cpu_offload, the prequant path and the LoRA selection before asking for the plan. - The video preview kept playing after leaving the page: the keep-alive layout only hides it, and display:none does not pause a media element, so a clip the user unmuted kept its audio going over the next page. Pause on the active transition and do not auto-replay while hidden. Also completes the hand-built request bodies in the hub download tests: the scoped-files field this branch added to the route read as an AttributeError against them, failing five tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Serialize the GPU handoffs, gate DiT training on a GPU, and keep 3.9 installable Six review findings, three of them evict-then-fail orderings: - The chat load reclaimed the GPU without telling the arbiter it existed. A chat load holds no llama-server process until its GGUF has downloaded, which is minutes, so a competing Images/Video acquire in that window found nothing to cancel, took the GPU, and the chat load then spawned onto the same device. It now registers an in-flight marker through acquire_for's register hook (under the arbiter lock, as the image and video loads do), the evictor cancels a marked load, and the route undoes itself if ownership moved while it loaded. - The Hub-download conflict check ran after that handoff, so a GGUF the download manager already owns destroyed the resident Images/Video pipeline and then 409'd, having loaded nothing. It moves above the handoff, together with the marker it handshakes with. - The image load released the engine router's transition lock before registering the load, so a second load choosing the other engine could unload the still-idle engine this one captured; the load then landed on a deactivated engine, where generate, status, unload and the arbiter's evictor can no longer reach it. Registration now happens under that lock and refuses if the engine changed. - Training a DiT family on a host with no GPU was accepted: nf4 is not a CPU fallback, its 4-bit load goes through bitsandbytes, which requires CUDA, XPU or MPS. The start unloaded the working Images pipeline, pulled the text encoders, and only then died in the child. Rejected before the teardown now, and /info stops advertising a precision that always 400s. SDXL keeps its documented fp32-on-CPU path. - Both diffusion pages kept the routed-pick marker forever, so re-picking the same checkpoint (after chat evicted it) neither loaded nor cleared the query string. The marker is released once the query is gone. The Images key also carried a stray NUL byte, which made the file read as binary to grep and other tooling. - diffusers dropped Python 3.9 in 0.38, so the unconditional >=0.39.0 pin left pip no candidate at all on 3.9 and made every install that composes the huggingface extras unresolvable there. The floor is conditional now. Also fixes tests that were already red on the branch: two hand-built request fakes had gone stale against fields this branch added, and the handoff-ordering test only failed on a host with fewer than two GPUs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the GGUF variant contract test against the merged handler signature The assertion pinned the exact single-line call handleVariantClick(v.quant, v.downloaded, expectedBytes, v.filename), but the handler takes (quant, filename, downloaded, sizeBytes) and prettier wraps the call across lines, so the mandatory repository test job failed on every push. Match the call structurally and assert the filename really is forwarded in the handler's argument order. * Stop a background page and a stale record taking the GPU or a download with them Five fixes from a review pass over the diffusion work. delete-finetuned rmtree'd a model the Images or Video engine was holding: every guard on that route is chat-only, and Images loads any local path, so deleting a local diffusion model under the storage root pulled the weights (and the companion VAE / text encoders sd.cpp re-reads each generation) out from under a live pipeline. The cached-model route already refuses this; the trained/exported one now does too, matching by path rather than repo id, and failing open on a chat-only install so it cannot block ordinary deletes. A staged download finishing while its page was hidden loaded the model and evicted whatever the user was actually using: both diffusion pages stay mounted behind the router and a load takes the GPU unconditionally. The pick is now held until its page is on screen again, which is also what chat does. A scoped download could report success having fetched nothing. With Hugging Face metadata unavailable no manifest is written, so verification is a no-op, and snapshot_download returns an existing snapshot folder without downloading when its own repo_info call fails. A repo already on disk from a full snapshot job (which ignores *.gguf) therefore completed with no weights and auto-loaded against them. The requested file list needs no network, so it is checked against the disk directly. The XET to HTTP retry reclaimed the job slot without the scoped file list, and that claim overwrites the stored record, so a later identical scoped start compared an empty list against the real one and 409'd instead of adopting the running download. The DiT accelerator gate probed torch.mps.is_available(), which only exists from torch 2.5 while the supported floor is 2.4. All three probes shared one try/except, so on torch 2.4 the AttributeError read as 'no block' and a CPU-only host still evicted the resident pipeline, downloaded the encoders and died in the child. Each accelerator is probed on its own now, through torch.backends.mps. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stage what an LTX-2.3 load reads, keep a routed file's load kind, drop an unbakeable LoRA Three from the latest review. The video download plan always asked for the wide base file list, so an LTX-2.3 pick staged the 2.0 base's VAEs, vocoder and connectors that the checkpoint supplies itself, while the companion files the 2.3 assembly does read were left out of the plan and pulled inline at load, outside the panel's progress, cancel and disk preflight. The plan now recognises a 2.3 pick by name (the load keeps the authoritative header probe, and under-guessing only falls back to the load-time pull), narrows the base list, and stages the extras in the same entry as the checkpoint so one repo stays one scoped job. A pick routed from the chat picker arrives as ?model= and ?quant= with no picker metadata, so a bare local .gguf or .safetensors was loaded as a pipeline: an explicit model_kind wins over the backend's filename sniffing, so it evicted the resident model and then failed on the missing model_index.json. Both pages now derive the load kind from the path, the same way their own picker handlers do. A torchao int8/fp8 build takes adapters only at load time. Switching artifact inside one family keeps the LoRA selection, since the family did not change, but the load did not bake it, so the next generation was rejected with 'reload the model with the adapter selection' while the picker still showed the adapter as active. The selection is now dropped once per resident build, with a message saying to pick and load again. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cancel an evicted safetensors load, spare the arbiter for CPU-only chat, fetch clips lazily Four fixes from the latest review round: - The GPU arbiter's chat evictor only cancelled the llama.cpp side. The orchestrator publishes active_model_name once its worker reports success, so an in-flight safetensors load was visible only as an entry in loading_models and finished onto the GPU after ownership had transferred. Cancel every pending load, and give the safetensors branch the post-load ownership recheck the GGUF branch already had. - A manual gpu_layers=0 GGUF load runs on the CPU with the GPUs hidden from the child, yet it took the arbiter unconditionally: it cancelled a running image or video generation for a model needing no VRAM, then held CHAT ownership so the next GPU workload unloaded it for nothing. Gate the acquire on the same predicate the launch-time CPU-only mask uses, as the image and video loaders gate on their resolved device. - The staged-download hook subscribes per repo, not per job, so another job on the same repo advanced the staged queue (starting a load whose scoped files were still downloading) or wiped a queue that was still running. Compare the variant each callback carries, like the chat page's auto-load does. - The video gallery fetched every record of a page into an object URL that lives until the page closes: 50 clips at tens to hundreds of MB each, for cards the user may never scroll to. Fetch a clip as its card nears the strip's edge, plus the selected one the player needs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim the comments across the diffusion backend Comment-only pass over the Python this PR touches: drop what the code already says, collapse multi-line explanations that still read on one line, and keep the reasoning that is not recoverable from the code. No code, docstring semantics or behaviour changes; verified with an AST comparison against the previous revision, and the backend suite is unchanged (same 37 environment failures as before: the API integration tests that need a live keyed server, the flash-attn install hooks, and the GPU memory fields). * Invalidate latents on a VAE swap, keep a cut-off generation, surface the EMA adapter - source_revision() scanned the checkpoint root plus text_encoder/tokenizer but not vae, so swapping or fine-tuning the VAE in place left the conditioning cache namespace unchanged and a warm run trained against latents from the old checkpoint. Include the vae directory, like any other component the cached tensors come from. - /images/generate answers only when the images are saved, and secure mode's tunnel caps an origin response near 100 seconds, which a native CPU or a high-step run passes routinely. The page reported failure while the work kept running, and a retry would duplicate it. A lost response (fetch rejection or a gateway status the origin never answered) is now told apart from a refusal: the page waits out generate-progress and reloads the gallery, so the run it started still lands. - The trainer emits the EMA adapter's path with the terminal event, but the state update dropped it, so neither the run history nor either response schema carried it and an enabled EMA left nothing discoverable. Keep it, and show it next to the primary adapter. - weighting_scheme advertised a choice of timestep sampling; sampling is always logit-normal and the flag only selects the bell loss weights. Describe what it does. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Hide unloadable cached rows, hold the dataset interlock, bound a GIF export - The cached-model listing tagged any repo with a model_index.json as text-to-image, so a community pipeline the image loader's trust rule refuses still got a row in the Images picker, and a detected-but-untrusted video repo fell through to that same tag. Gate the image tag on the load path's rule and hide an untrusted video repo outright. - A routed diffusion pick only carries a GGUF filename, which is all the chat picker has, so a curated single-file artifact arrived with no quant and was loaded as a pipeline: from_pretrained on a repo with no model_index.json. Pass the page's own catalog spec into the route pick, so a routed pick resolves to exactly what a direct pick on that page resolves to. - The dataset mutation endpoints checked is_active() and only then handed their filesystem work to a thread, so a start reserving in that gap changed captions or removed images underneath the preflight or the running trainer. The interlock is now registered for the whole request under the lock reserve() uses, and a start refuses while a mutation is open rather than waiting on it. - GIF export held every kept frame as a paletted image before encoding; a clip may be 2048x2048 for 1024 frames, and at the 12 fps target the step is 1, so one export click could allocate over 4 GB and take the backend down. Downscale past 720 px and widen the step to keep at most 300 frames. - seed accepted any Python int, so an out-of-range one passed every preflight, evicted the resident models, spawned the trainer and only then died in torch.manual_seed. Bound it to torch's 64-bit range in the request and config. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the accelerator probes in the DiT family-metadata tests Six tests read family_train_infos() (or a start preflight) without pinning the host probes, so they only held on a machine with a bf16 accelerator: on a GPU-less runner the DiT gate empties precision_modes, turns supports_compile off, and replaces any other preflight message with the no-accelerator note, and all six failed there. A conftest fixture pins both probes for exactly those tests, so they assert the family metadata they are about on every host. The gate's own CPU-only behaviour keeps its dedicated tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Carry the pipeline task into the hub inventory the pickers read - The task-scoped pickers filter On Device rows on a task, and the chat picker routes a diffusion pick by the same field, but those rows come from the /api/hub inventory, which never carried one: the Images and Video pickers listed nothing on device and the routing never fired. Both cached scans and the local listing now tag rows with the classifiers the models API already uses, the schemas and the frontend adapter carry it through, and a row the backend classified as a generation task is exempt from the chat-only guard that was also dropping it. - The local routing map was keyed by model_id while the row click passes id (a filesystem load id for a models_dir or LM Studio entry), so the lookup missed and the pick fell through to the chat loader. Key both. - A staged download whose start answered "error" left its head in place, where the effect never re-runs and onReady never fires, so the pick was stranded until the user reselected. Clear the queue and say so. - Every scoped pick in a repo shares the @diffusion variant, so the variant alone cannot tell two file sets apart: restaging while the first job finished let its completion pass for the new pick and load a checkpoint that had not downloaded. Bind the callbacks to the repo + file set they started, and to the staging generation. - A rejected generate POST does not say whether it reached the backend, so an immediately idle progress read was ambiguous and a submission that never landed looked like a finished image. Require evidence: progress seen active, or a gallery record that was not there before the POST. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the OpenAI image URL fetchable, keep WebM audio, stream example imports Four review items on the diffusion Studio work: - response_format=url returned the bearer-gated gallery route, which a standard image client downloads with no Authorization header, so the default response format was unusable. Mint a short-lived HMAC link instead (the shape RAG already uses for pdf.js) served by a signed route, and leave the gallery route itself bearer-only. - A manual gpu_layers=0 load carrying speculative_type="off" -- a value the UI persists and sends -- read as GPU-bearing, so it took the GPU arbiter and evicted a resident image/video pipeline even though the launcher hides the GPUs for it. Canonicalize the mode and exempt "off". - The curated example import prepared the whole split before the loop stopped at the 10-100 image cap; m1guelpf/nouns is 49,859 rows / 328 MB. Stream instead, with the prepared load kept as a fallback for a repo that cannot stream. - WebM export dropped the audio track an LTX-2 clip carries, silently, on the format offered for web embeds. Mux it as Opus through a resampler + FIFO, and keep exporting the video alone on a build without libopus. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not stub out triton on a GPU host when the Xet backend fails to import The lazy loader retries `import unsloth_zoo.hf_xet_fallback` under UNSLOTH_ZOO_DISABLE_GPU_INIT=1 whenever the first attempt raises. That flag makes unsloth_zoo take its MLX/CPU path, which injects triton and bitsandbytes STUBS into sys.modules for the rest of the process. On a working GPU box whose first import failed for an unrelated reason (a bitsandbytes/CUDA mismatch, say) the retry succeeds, so Studio boots looking healthy and then dies at the first CUDA-only kernel: a GGUF or compiled diffusion generation hits the stub and returns NotImplementedError: Unsloth: 'triton.tools.experimental_descriptor.enable_in_pytorch' was called on Apple Silicon / MLX, where triton is stubbed out. so every image generation 500s with an Apple-Silicon message on a Linux CUDA host, while the load reports success. Found by loading Z-Image-Turbo GGUF through the API on a box where bitsandbytes could not initialise. Gate the retry on the host genuinely having no accelerator. The Xet stall watchdog is optional and already degrades with a warning; a process whose triton is stubbed out is not recoverable. The warning now says why it did not retry. * Fix the lost-generation proof set, the settle timeout and the hub inventory's diffusion gates Seven fixes from the latest review round on the Images page and the hub cache inventory. Images page: - The lost-POST settle path built its "already seen" gallery id set inside the catch, after the request failed. By then the earlier runs of the same batch had already prepended their records, so run 2 could accept run 1's image as proof that its own request reached the backend. The set is now captured once before the first POST and grows with every record the batch produces. - settleLostGeneration fell out of its SETTLE_MAX_MS loop and returned normally, so a wedged generation was counted as done and the next run started against a busy backend. It now throws on timeout. - Restoring a recipe cleared the ControlNet selection but left the workflow tab and the init / mask / reference images pointing at whatever was loaded, so the next Generate conditioned on an unrelated image. It now clears all of them and returns to Create. - The download plan omitted the adapter selection the load itself bakes in. A baked LoRA forces the dense build path, so the plan described a different file set than the load that followed and the rest was pulled inline, outside the download manager. Both now derive the list from one helper. Hub cache inventory: - A download for a repo an Images or Video load is staging was allowed to start: only the llama.cpp loader was consulted. Both diffusion backends already expose loading_repo_ids for the delete guard, and the download guard now reads them too. - A companion-only prefetch (pipeline manifest plus VAE and text encoder, no transformer) passed the snapshot-partial check, since every file its manifest expected did arrive, and was advertised as on-device although from_pretrained cannot load it. - The single-file flag never reached the picker through the hub inventory path, so a checkpoint-only diffusion repo read as a full pipeline and failed after the handoff. The two pipeline-shape helpers now live in hub/utils/inventory_scan.py so /api/models/cached and the hub inventory classify the same repos the same way. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop adopting an unknown scoped download, leaking raced blobs and resurrecting deleted clips Three items from the latest review round. A scoped download job carries a deliberate file subset, and every file set of one repo rides the same "@scope" slot. A client that adopts a live job from the backend had no file list to compare against: the active-downloads response never carried one, so an adopted job's set was unknown and any later scoped request for the same repo read as "already started". Selecting a different checkpoint then waited on the wrong transfer and tried to load a file nobody fetched. The response now publishes the scoped file list, adoption records it, and an unknown set no longer satisfies a scoped request. A gallery record can be deleted while its blob is still downloading. The delete revokes the URL present at that moment, so the fetch that lands afterwards inserted a fresh object URL for a record no card renders and nothing can revoke: a full MP4, tens to hundreds of MB, pinned for the rest of the session, and once per raced fetch. Both galleries now discard a blob whose record went away, with an epoch covering the video page's Clear all. The video backend keeps the last completed job until the next one starts, and the Video page merges that record on mount to cover a job that finished after the gallery fetch. Deleting the clip left the record in place, so every reload prepended a ghost card whose file request 404s until another generation replaced it. Deleting the clip, or clearing the gallery, now clears the matching terminal record, and the page skips a record it deleted itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Resolve revisions from the live cache, serialize dataset imports, and stop pinning every gallery blob Five more items from the review round. The conditioning-cache revision marker read huggingface_hub's import-time HF_HUB_CACHE constant. Studio can move its cache during a session and loading follows the live setting, so after a move the marker went unresolved (or pointed into the previous root) and pulling a new revision of the same checkpoint no longer invalidated the cache: a warm run could reuse the old encoder's embeddings and the old VAE's latents. It now looks in the active Studio cache first and keeps the environment and the library constant as fallbacks, which the trainer subprocess still needs. The dataset interlock counts mutations rather than excluding them, so two imports of different examples into the same empty name both got past the emptiness check. The winner promoted its staging directory atomically; the loser found the folder non-empty, fell back to a per-file move, and merged its images and captions into the winner's dataset. Imports now take a per-folder lock, a second one is refused with 409, and the emptiness check is repeated under the lock. On Windows the sd.cpp asset resolver filtered only by accelerator token, so a Windows arm64 host matched an x64 zip, downloaded and installed it, and failed later when the binary would not run. It now filters by architecture the way the Darwin and Linux branches do. Every gallery page fetched every PNG up front and kept the object URL for the session, so scrolling a large gallery grew memory without bound for tiles the user may never look at. The Images strip now fetches a tile as it nears view, like the Video strip, and keeps the eager path only where IntersectionObserver is unavailable. A 503 carrying a JSON body comes from the application, not a proxy, so it is surfaced as the error it is instead of entering lost-response settlement and being reported as a request that never reached the server. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the class of a failed generation instead of a bare "Image generation failed." Found on a macOS runner: the native renderer aborts inside its own text encoder there, and the page showed only "Image generation failed." with the sd-server backtrace left in the server log, so nothing about the failure reached the user. The failure is now classified into fixed text, out of memory and native-process death, so the message says what happened and what to try. None of the engine's own output is echoed, since a native tail carries local paths and argv; that stays in the log, and an unrecognised failure keeps the original literal. * Treat an undecodable caption sidecar as the tombstone the trainer sees Uploads store .txt and .caption sidecars as raw bytes, so one can hold invalid UTF-8. The trainer treats any existing sidecar, decodable or not, as an empty tombstone and never falls back to the metadata row for that image. The labeling grid and the dataset summary read an undecodable sidecar as absent instead, so both showed a metadata caption that the run would silently replace with the instance prompt, and counted the image as captioned. Both now track sidecar presence separately, so what the user reviews is what the run trains on. * Keep the reason a native server died, not just its backtrace A ggml abort prints its cause first and then a stack trace, so reporting the last twenty captured lines gave twenty addresses and nothing about the failure: on the macOS runner the native server died on an unimplemented Metal op and the message carried only frame pointers. The captured tail now leads with the lines that name a cause and keeps recent context after them, for both the startup failure and the mid-request death. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop staging the dense text encoder for an fp8 video load Two halves of the same gap, found while measuring the LTX-2.3 download plan: - The video download plan and the scoped pre-download never saw text_encoder_quant. An fp8 request loads a hosted pre-cast encoder, so asking for one still staged and downloaded the base repo's dense Gemma3 (48.79 GB of Lightricks/LTX-2 on the 2.3 distilled pick) that the pipeline then never opened. The plan now drops those shards and stages the pre-cast checkpoint instead; their configs stay, since the pre-cast loader still meta-inits the encoder from the base repo's component config. - The LTX-2.3 assembly builds every component itself, so pipe_kwargs (which carries the pre-cast encoder for from_pretrained) never reached it and an fp8 request silently loaded the dense encoder anyway. It is passed across explicitly now. The dense skip is earned, not assumed: only a pre-cast checkpoint that resolves on the Hub lets the plan drop the dense shards, and only one already fetched to disk lets the pull drop them, so an unpublished or gated artifact leaves both exactly as they were. If injection still fails after that, the load tops the dense weights back up rather than handing from_pretrained a snapshot with no encoder in it. Measured against the real Hub on the 2.3 distilled Q4_K_M pick: 67.24 GB before, 18.92 GB with a 0.43 GB stand-in for the pre-cast artifact (the base entry drops from 24 files / 48.79 GB to 13 files / 0.04 GB). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Match the file's typing idiom for flow_shift models/training.py annotates with typing constructs throughout (105 Optional[...], no Union), and flow_shift was the one place using PEP 604. Union[] reads like the rest of the file, and it also drops the runtime evaluation that would raise on Python 3.9. * Bound the gallery blob cache, and three interlock fixes Four review findings, all reproduced first: - The gallery object-URL caches were unbounded. A clip runs from a few MB to a few hundred, both pages stay mounted after their first visit, and entries were only dropped on delete, so scrolling pinned everything for the session. Both pages now share a byte-budgeted LRU (512 MB video / 192 MB images) keyed off the visibility signal the near-viewport fetching already provides. On-screen media, the selected clip or image, and the item just fetched are never evicted, so eviction is invisible and a single item larger than the whole budget cannot evict itself into a refetch loop. - The image, video and chat load guards ran two independent training probes but returned early when the FIRST one raised, so an unreadable LLM backend disabled the diffusion interlock and a load could proceed straight into an active diffusion trainer on the same GPU. The probes are independent now. - An engine switch swallowed a failed teardown and published the new engine anyway, which is exactly the leak the unload exists to prevent: the arbiter's evictor, /images/unload and the next load all resolve through get_active_diffusion_engine(), so the still-resident pipeline (or a live sd-server) became unreachable and the next load allocated on top of it. The switch now fails and leaves the old engine published, so it stays reclaimable. - The native generation timeout was 30 minutes while the Images page waits up to 6 hours (SETTLE_MAX_MS), so slow-but-progressing CPU jobs died deterministically at the deadline. Measured on GPU-less runners, a 512x512 4-step Q2_K generation took 900 s on Linux and 1465 s on Windows, so larger images or step counts clear half an hour easily. The ceiling now matches the page's window and applies to the whole request: chunks of a split batch share one deadline instead of each getting a full budget. Cancellation is unchanged. Declined: gating the huggingfacenotorch extra off Python 3.9 over the conditional diffusers marker. The marker is deliberate and its comment says why: diffusers dropped 3.9 in 0.38, so pinning >=0.39 outright leaves pip no candidate and the whole extra unresolvable there. The pipelines it names live in studio/backend, which cannot install on 3.9 anyway (studio.txt pins matplotlib==3.10.9 and fastmcp>=3.0.2, both requires_python >=3.10), and the extra is the general core one, so the alternative drops 3.9 for library users who never touch Studio. * Close the load-versus-training-start race, and two picker fixes - The image and video load guards read is_active() and only then selected an engine, acquired the arbiter and registered the load. A /train/diffusion/start reserving inside that window freed residents the load had not registered yet, so the trainer came up beside a brand-new pipeline. The service already had exactly the right pattern for this in dataset_mutation, so gpu_load_admission mirrors it: reserve() refuses while an admission is open, an admission refuses once a start is reserved, both decided under the one lock. The span is only the registration, since begin_load returns as soon as the load is registered and _free_gpu_for_diffusion_training preempts an in-flight load from that point. Chat is deliberately not covered: its load spans an eviction plus a multi-minute GGUF load, and it admits models that fit beside training by design, which is a different contract from the diffusion pipeline's all-or-nothing one. - Hugging Face gives the LTX-2 family the image-to-video pipeline_tag (both Lightricks/LTX-2 and unsloth/LTX-2.3-GGUF report it), so a text-to-video-only filter dropped the flagship audio family out of Video Hub search while the rest of the app routed it to Video. - Task-scoped quant fit sized picks against the LARGEST visible device while resolve_diffusion_device_target returns a bare "cuda" and torch places on the current one. On a heterogeneous host that recommended a checkpoint sized for the bigger card and then loaded it onto the smaller one. Fit now uses the device the load actually lands on; identical on a homogeneous host. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Expose the persistent conditioning cache in the start schema DiffusionLoraConfig has carried cond_cache_dir for a while and the DiT trainer acts on it, but DiffusionTrainingStartRequest omitted the field, so Pydantic dropped it silently and every API-driven run fell back to the in-memory cache that is rebuilt from scratch each time. The warm path skips loading the VAE and the multi-GB text encoders on a rerun whose images, captions and resolution are unchanged, so this was a real capability that could not be reached. Contained like output_dir rather than left to the trainer subprocess's cwd, since it is another directory the trainer writes to. Blank or omitted still means the in-memory cache, so it must not resolve to the outputs root. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix diffusion policy and classification issues from review fp8 auto precision defaulted to precise accumulate on any non-consumer GPU, which made fp8 2.05x slower than int8 on RTX 6000 Ada and slower than not quantising at all. NVIDIA's professional whitepapers do publish equal FP8 rates for both accumulate modes there, so the hardware premise held, but the cost is in the cuBLAS path rather than the published rate. Default to fast accumulate: measured on B200 the flag is a no-op (4096^3 _scaled_mm at 3023.8 vs 3041.8 TFLOP/s, bitwise-identical output, 1.213 s vs 1.230 s end to end), so it is a large win where it bites and free where it does not. Precise accumulate stays available via transformer_quant_fast_accum. Z-Image's DiT is a Lumina2 derivative, so unsloth/Z-Image-GGUF and unsloth/Z-Image-Turbo-GGUF both declare general.architecture = "lumina2" and the whole line was tagged image-diffusion-unsupported and hidden from the Images "On Device" list, though validate_load_request loads them. Resolve shared archs from the repo/file name like bare "wan" already does, with a test asserting the picker and the loader agree for every family. The sage attention on-demand install ran an unpinned `pip install sageattention`, but PyPI's newest wheel is 1.0.6 and diffusers refuses anything below 2.1.1: the install always "succeeded", wrote an unusable version into the running venv, and was rejected on the next line. Carry the dispatcher's floor so pip resolves nothing instead. The dense-quant disk gate sized the download from the bf16-RESIDENT table. The fp32 families download twice that (Z-Image: 23,479 MiB against a 21,970 MiB gate), leaving a window where the check passed and the download filled the disk; Ideogram 4 ships fp8 and was overcharged the other way. Size the gate by published bytes, verified against HF sibling metadata for all 12 families. Patch installs went through unsloth_zoo, which refuses to import unless UNSLOTH_IS_PRESENT is set, and that is set by unsloth itself. The server imports unsloth at boot so it never showed there, but any other process ran silently unpatched with every install returning False, which is 13 test failures on a clean environment. Import unsloth and retry once, memoised per process. Also: the GGUF+LoRA refusal pointed at the native engine without saying a GPU host only selects it under UNSLOTH_DIFFUSION_ENGINE=sd_cpp, so the suggestion was unreachable; the gallery recipe recorded loras from the generate request alone, losing a load-time bake; load-progress claimed "40.07 GB downloaded" for a fully cached load; and pickers.tsx imported three catalog-group helpers it never used. Reported by oobabooga. * Keep the sd.cpp text encoder on CPU under Metal macos-14 loads FLUX.2-klein-4B Q2_K natively on mps and then dies on the first generation with exit code -6: ggml_metal_op_encode_impl: error: unsupported op 'RMS_NORM' -> ggml_abort LLMEmbedder::encode_prompt -> LLMRunner::compute -> GGMLRunner::compute ggml's Metal backend gates RMS_NORM on contiguous rows and aborts the process when that does not hold, with no per-op CPU fallback, so any LLM text encoder (Qwen3 for FLUX.2 and Z-Image, T5 for FLUX.1) takes sd-server down. The encoder runs once per prompt while the DiT runs every step, so pinning only the encoder keeps Metal for the part that matters. UNSLOTH_DIFFUSION_SD_CPP_METAL_TE_GPU=1 opts back in once ggml grows the kernel. * Gate the unsloth retry in the diffusion patch backend The retry added for the clean-environment patch failures is not free: importing unsloth pulls torch in behind it, which costs ~940 MB of RSS measured in a process that had neither, and on a host with no accelerator it fails anyway. A cross-platform CI job that had generated fine at ~900 s later died 19 s in with SIGTERM and every 'if: always()' step skipped, which is the runner being torn down rather than a step failing. Retry only when torch is already imported (true of the server and of anything patching a real module, and the condition that stops the retry from being what loads torch), unsloth is installed but not yet imported, and the first failure was the ImportError the sentinel guard raises. The clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only retry the unsloth import where it can succeed The gate still let the retry run on hosts unsloth does not support, which is where it is most harmful: a 7 GB macOS runner lost the Studio server 26 s into a load, and the Linux runner was torn down mid-generation. Neither MPS nor plain CPU can complete the import, so the retry there pays the cost and fails anyway. Require an accelerator unsloth actually supports (CUDA/ROCm via torch.cuda, or XPU), with UNSLOTH_ALLOW_CPU as the documented override, and hoist the predicate to module level so it is tested directly rather than through the import system. On a CPU-only host the retry no longer fires at all; on CUDA the clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Guard old diffusers, stream video exports, record conditioned recipes Three fixes from review. The 0.39-only pipeline classes (Flux2Klein, Z-Image, Krea 2, LTX-2, HunyuanImage) were resolved by getattr deep in the load, so on the older diffusers that packaging still allows on Python 3.9 -- diffusers dropped 3.9 in 0.38 and this project still supports it, so the 0.39 floor has to be conditional or the extra becomes unresolvable -- an advertised model failed with a bare AttributeError after its checkpoint had already been downloaded. Krea 2 already guarded itself this way; assert_pipeline_class_available now runs the same check for every image and video family from validation, before any fetch, and names the version and the fix. WebM export accumulated the whole VP9 output in a BytesIO and returned it as one bytes object that the response held again. The request caps allow 2048x2048 for 1024 frames, so an export runs to hundreds of MB and concurrent clicks could exhaust the process, while the MP4 route beside it already streamed from disk. transcode_to_file encodes to a temp file and the route returns a FileResponse with a background unlink, so nothing large is resident. A conditioned generation's recipe carried only the txt2img fields, so the gallery presented an inpaint or upscale result as a complete Create recipe and restoring it replayed an unrelated text-to-image request. The images themselves are still not persisted (user uploads with their own lifetime), but the workflow and its scalars are, restore reapplies them, and the toast now names the inputs that have to be supplied again instead of silently landing on Create. Reported by Codex. * Per-load video cancel event, family-gated image picker, cond cache refusal A cancelled video load could resume: begin_load cleared the shared cancel event, and unload() drops _loading without waiting for the worker, so the next load cleared the very object the cancelled worker was watching and its multi-gigabyte pull ran on alongside the replacement until the token check at the end. Each load now gets its own threading.Event, passed down through _fetch_te_prequant and _predownload_base, so a cancelled worker stays cancelled. A cached repo with a model_index.json was advertised as text-to-image on the trust rule alone, but validate_load_request also requires a detected image family, so a trusted pipeline of an unsupported class produced a picker row that deterministically 400s. The picker now applies both gates, mirroring the video branch. cond_cache_dir was accepted for sdxl and then ignored: only the DiT trainer reads it, while the SDXL trainer builds a per-run in-memory latent cache, so the promised cross-run reuse never happened. The route now refuses it with a 400 that names the families which do support it, checked against the resolved family so an omitted model_family with an SDXL base is caught too. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the frontend build broken by the gallery blob cache tsc -b failed on the branch head, so npm run build produced no dist and every platform job fell back to --api-only: blob-url-cache.ts(29,15): TS1294: This syntax is not allowed when 'erasableSyntaxOnly' is enabled dataset-labeling-grid.tsx / dataset-showcase.tsx: Argument of type '{ url: string; bytes: number; }' is not assignable to parameter of type 'string' The cache took its budget as a constructor parameter property, which the project's tsconfig forbids, and fetchGalleryObjectUrl now returns the blob size alongside the URL for that budget, which the two dataset thumbnail components still consumed as a bare string. Declare the field explicitly and destructure the URL at both call sites. tsc -b is clean and vite build emits dist again. * Recover from a ggml unsupported-op abort by restarting on the CPU backend ggml checks every node against the device's supports_op and calls GGML_ABORT when one is not implemented, because a single-backend graph has nowhere else to put it: there is no per-op CPU fallback. The whole sd-server dies with SIGABRT mid-generation and the user gets "the native image renderer stopped unexpectedly" with no way forward. Seen on macos-14 arm64 with FLUX.2-klein-4B Q2_K through the cross-platform CI: the text encoder is already pinned to CPU, and the abort moved into the denoise loop instead. ggml_metal_op_encode_impl: error: unsupported op 'MUL_MAT' -> ggml_abort StableDiffusionGGML::sample -> sample_k_diffusion A retry on the same backend would abort identically, so the load is restarted once with --backend cpu (the only flag that changes which backend executes the graph; --offload-to-cpu moves parameters, not compute) and the generation is re-submitted. The same checkpoint then renders slower rather than not at all. Strictly bounded: the signature must carry both the unsupported-op line and ggml_abort, the device must not already be CPU, and it happens once per load, so an OOM kill or a genuine crash still surfaces as itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix two tests that only fail in a full-suite run The 3.10 CI leg resolves PyAV 17, where av.container.OutputContainer is an immutable C type, so the no-libopus export test died on "cannot set 'add_stream' attribute of immutable type" before it asserted anything. Inject the refusal by wrapping the container av.open() returns instead; modules stay patchable on every build. Removing the injection makes the test fail again, so it still covers the branch it is named for. The Xet shim's degraded-path tests drop utils.hf_xet_fallback from sys.modules and import a throwaway copy. Restoring only the sys.modules entry left the utils package attribute bound to the throwaway, and the two disagreed for the rest of the process: a later monkeypatch of the dotted target patched one copy while the code under test imported the other, so the patch did nothing and test_fetch_te_prequant_only_reports_what_it_downloaded reached the real Hub and got a 401. Restore both bindings. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop an ignored-cancel sd-server, guard deletes during diffusion training, repair unusable managed binaries sd-server does not interrupt an in-flight job, so when it ignores a cancel the grace branch abandoned the poll and reported cancellation while the native job kept a core (or the GPU) busy to completion and held the server's job slot. The comment said the caller stops the server, but only unload does that immediately: a superseding load stops it after its multi-gigabyte download, and a load that then fails never gets there. Stop it here, as the deadline branch already does. DELETE /api/models/delete-finetuned checked only the LLM trainer, so it could rmtree the output directory a live diffusion LoRA run was about to write its adapter into. Consult the diffusion training service too, like the dataset mutation and model-load routes. find_sd_*_binary only checks is_file(), so an interrupted extraction (or a prebuilt for the wrong CPU) left a present-but-unrunnable binary the installer never retried: every load probed it, fell back to diffusers, and native inference stayed off until the directory was deleted by hand. Probe it and reinstall, but only for a copy under the installer-owned root -- SD_CLI_PATH, UNSLOTH_SD_CPP_PATH, an in-tree build and anything on PATH are the user's. * Plan the pre-cast text encoder, and make the cross-trainer GPU admission atomic An fp8 text-encoder request loads a hosted PRE-CAST checkpoint, but the image download plan never received text_encoder_quant, so the manager staged the base repo's dense encoder (FLUX.2-dev's Mistral-24B is ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) and the load then pulled the pre-cast file inline, outside the manager's progress and disk preflight. The plan now takes the field, resolves the hosted artifact with the same resolver the injection uses, stages that file, and drops only those components' dense weight shards. The load's own prefetch takes the same treatment, since it paid the same cost. Only a checkpoint that really resolves on the Hub earns the drop, so a gated or renamed artifact still stages the dense encoder the load will fall back to. The two trainers admitted each other with independent check-then-act guards: the diffusion route checks the LLM backend several network-bound preflights before it reserves, and the LLM route checks the diffusion service well before it spawns, so two near-simultaneous starts could both pass and train on one GPU. reserve() now re-tests the LLM backend under its own lock, and the LLM route holds the diffusion service's gpu_load_admission across its spawn, so exactly one of the two wins. Both halves fail open, so a chat-only install still trains. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the Advanced options a staged download planned against Staging does not set busy, so while a multi-gigabyte download runs the user can still change precision, memory mode, speed or the baked LoRA selection. The pending record held only the repo and artifact, and the completed download fired a load that read the CURRENT state: the staged file set could then be missing files that load needs (fetched inline, with no progress and no disk preflight) or hold gigabytes it no longer uses. One snapshot of every Advanced control is now taken when the plan is built, and it travels with the pending record into the load, so the load that runs is the one the download was planned for. * Do not advertise a family the installed diffusers cannot build The newer families (Z-Image, Krea 2, FLUX.2, LTX-2, HunyuanImage) exist only from diffusers 0.39, and 0.39 cannot be installed on Python 3.9 at all -- diffusers dropped 3.9 in 0.38, so the requirement is conditional or the whole extra becomes unresolvable. On such an environment the picker still offered those rows, every pick failed deterministically, and the error's advice to run pip install -U diffusers could not fix it without also upgrading Python. The cached-repo picker now applies the same availability check validate_load_request does, which is keyed on the pipeline class actually present rather than on the Python version, so it is also right for an intentionally pinned older diffusers on 3.10+. Fails open when diffusers cannot be imported at all: that is a different problem and the load path reports it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore the diffusion engine selection after each router test The active engine is module state, and several tests set it by plain assignment because what _activate does to it is the thing under test, so monkeypatch could not undo it. A leaked ENGINE_SD_CPP left get_active_diffusion_engine() handing back the sd.cpp backend for the rest of the process, and every later route that reads the active engine then saw an unloaded model: eight tests in test_openai_images_generations_route.py returned 503 in a full-suite run while passing on their own. The autouse fixture now snapshots and restores it. * Stream gallery clips, and close three races around them Four fixes from the latest review pass. The video gallery downloaded each clip into a blob before it could play, so playback waited on the whole file (tens to hundreds of MB), seeking was limited to what had arrived, and every viewed clip stayed pinned in the webview. The file route already streams and serves ranges; it just could not be a <video src> because it is bearer-gated. Mint a short-lived signed link instead (its own HMAC secret, 12 hour TTL, separate from the image links) and hand it to the element, which then fetches only the ranges it plays. That removes the blob budget, its LRU and every revoke on this page. The sd.cpp readiness probe accepted any process answering on the port, so a foreign server that grabbed the port between the bind check and the spawn was adopted as ours. Confirm the listener is our child before reporting ready, and stay best-effort (psutil missing, an unknown owner, or any probe error still passes) so the check can only reject a definitely foreign process. Dataset import held its lock for the extract but not for the upload path, so two concurrent uploads into the same folder interleaved; take the same lock and return 409. And reject Windows device names (CON, NUL, COM1..9, LPT1..9, with or without an extension) plus trailing periods in dataset names, which are unopenable on Windows. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Build the image download plan for the engine that will load /images/download-plan always asked the diffusers backend, while /images/load picks the engine per host: a GGUF pick on a machine with no usable GPU routes to native sd.cpp, which reads a single-file VAE plus text encoders and never opens the base repo's sharded components. Measured on unsloth/FLUX.2-klein-4B-GGUF (Q2_K): the plan staged 7.66 GB of FLUX.2-klein-4B components the native load discards, and the 7.80 GB sd-cli actually needs was then fetched inline by the loader, outside the download manager's progress and its disk preflight. Z-Image-Turbo is the same shape. The plan now asks whichever engine the load will select. predict_engine() applies the selection policy without any side effect: it activates nothing (staging a download must not unload the resident model) and only locates the binary rather than installing it, but still counts an installable binary as available, since that is what the load does on a fresh host. The native backend gains a download_plan built from the same _asset_specs the loader fetches, returning the same envelope, so the manager stages exactly the files sd-cli opens. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not let a queued generation outlive the model, and three scan fixes Five items from the latest review; four were real. An unload or arbiter eviction only cancels the generation holding _generate_lock. A second request queued behind it holds no cancel event yet, and Python locks are not FIFO, so it could take the lock the instant the active denoise released it, still see a loaded pipeline, and run a whole new denoise after the model was told to go away: the eviction then waits minutes for it and an image lands after the eject. Unload and a superseding load now raise a fence under _lock before they queue, and a generation that wins the lock while one is pending refuses instead. The cached-model scan judged pipeline completeness across every revision, so a repo holding an older complete snapshot plus a newer companion-only one read as complete while the snapshot from_pretrained actually opens has no transformer. Both scans now look at the revision the loader will open. Deleting a dataset image deleted its caption sidecar unconditionally, which for cat.jpg alongside cat.png removed the caption the survivor still resolves to. The sidecar now goes only with the last image of that stem, matching what the thumbnail cleanup beside it already did. Importing an example into a folder that holds no images but does hold files fell back to promoting the staging dir one file at a time, so an interruption left a partial dataset that the image_count check accepts as complete on retry. Those files are folded into the staging dir instead and the promotion stays a single atomic rename. The MPS generator report does not apply: torch.Generator(device="mps") has worked since PyTorch 2.0 (pytorch/pytorch#91348) and the studio installer pins torch>=2.4. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten diffusion comments Collapse the multi-line comment blocks across the image, video, sd.cpp and diffusion-training code to one or two lines each, and drop comments that only restate the statement below them. Comments only, no code or behaviour changes. * Tighten diffusion comments (second pass) Collapse the remaining multi-line comment blocks in the video page, training routes and service, sd.cpp server and installer, memory and speed planners, and the shared request models. Comments only, no code or behaviour changes. * Tighten diffusion comments (third pass) Collapse the remaining multi-line comment blocks in the attention, cache, LoRA, prequant, precision and compile-cache modules, the sd.cpp arg builder and engine, the video routes, the Ideogram 4 assembly, the model picker, and the diffusion test suites. Comments only, no code or behaviour changes. * Restore the dataset when an example import cannot be promoted Promotion folds the folder's pre-existing entries into the staging dir so the swap is one atomic rename. Every failure after that fold left the user with nothing: the 409 path and an os.replace error both fell through to 'finally: shutil.rmtree(staging)', which deleted the entries that had just been moved in there, while the response said 'Nothing was written'. A same-named entry was also unlinked outright before the promotion was known to succeed. Park superseded same-name entries in a rescue dir instead of deleting them, record every move, and restore all of them if any step of the promotion fails. The fold loop itself is covered too: renaming a non-writable directory raises EACCES on POSIX, which previously escaped as a 500 after the earlier entries had already been moved out. A failed rename now maps to the same retryable 409 as the rmdir conflict. Verified with the reported trigger (a non-empty mode-500 directory whose name collides with an imported file): the folder listing is now identical before and after the failed import. * Drain the teardown fence on a failing unload, give each load its own cancel event Two independent leaks on the image path, both already solved elsewhere in the same file. unload() incremented _teardown_waiters, ran _unload_locked() and decremented, with no try/finally, while the superseding-load path used a finally for the same pair. _unload_locked ends in clear_gpu_cache(), whose CUDA branch calls synchronize/empty_cache/ipc_collect unguarded, and a sticky CUDA fault makes those raise. The count then never drained, so every later generation was refused as cancelled for the life of the process, a fresh load included, since begin_load's own increment and decrement are symmetric. unload() is reached from the chat/video GPU handoff, the engine router and two training routes, so one fault during an ordinary handoff wedged image generation until restart. Release it in a finally. The image and native backends each cleared one shared cancel Event on a new load. unload() sets that event to cancel an in-flight multi-GB download and drops _loading in the same breath, so a replacement load is admitted while the cancelled worker is still inside the fetch, and its clear() re-enabled the very object that worker was watching: the cancelled download resumed and ran alongside the replacement. Take a fresh Event per load and thread it to the worker, as the video backend already does, and set it under the lock since begin_load now rebinds the attribute. * Name utf-8 on the diffusion text I/O and the sd.cpp subprocess pipes tests/test_text_io_encoding.py failed on five files this branch adds. Text I/O without an explicit encoding falls back to the Windows ANSI codepage, so a non-ASCII path or manifest value round-trips corrupted, and the three sd.cpp pipes decode the child's UTF-8 output as ANSI on Windows despite already passing errors = 'replace'. Eleven read_text() / write_text() sites across diffusion_compile_cache, diffusion_ideogram4 and diffusion_krea2, plus text = True on the sd-cli version probe, the sd-cli run and the sd-server pipe. * Record the load-time build on a gallery image's recipe A gallery record documents itself as the image's full generation recipe and is embedded in the PNG, but the only load-related field it carried was the repo id. A GGUF repo holds many quants, so that does not say which one made the pixels, and it says nothing about an adapter baked in at load time. The fallback meant to cover the baked case could never fire: with no loras on the request _adjust_baked_loras zeroes every baked adapter and _active_lora_pairs drops zero-weight entries, so active_loras was always empty. A baked-and-disabled build is not the same pipeline as a never-baked one, so the recipe could not reconstruct the image once the model was rebuilt. Persist model_kind, gguf_filename, transformer_quant and the baked adapter names, read off the load state rather than the request, and show them in the recipe popover. The new fields are optional with defaults, which matters because list_gallery_images drops any record that fails validation, so a required field would have emptied every existing gallery; a regression test pins that. * Drop eleven duplicated comment tails, restore the mxfp8 denial note The comment passes collapsed several wrapped blocks onto one line without deleting the last physical line of the original wrap, leaving the tail of each sentence repeated as its own comment underneath. Two of the eleven were re-worded rather than byte-identical, so a strict suffix match missed them. 6e16ad16f also dropped the line justifying the qwen-image mxfp8 denial while that rule stayed live in _FAMILY_SCHEME_DENY, under a header that then documented only fp8 and nvfp4. Restored. Comments only; verified with the AST gate. * Only let the dense-quant fallback use shards the prefetch actually staged The prefetch skips the base repo's transformer/ shards whenever a prequant checkpoint is expected, since that checkpoint replaces them. But a prequant fetch can fail for reasons the planner cannot see: an unpublished, gated or renamed artifact, a hub 5xx, a proxy, a checkpoint the validator rejects. The loader then fell through to from_pretrained(subfolder = 'transformer') and pulled those shards inside the load lock during 'finalizing', after the previous pipeline was already evicted, where the cancel event has no reach, load_progress has already reported bytes_downloaded == bytes_total, and the cache-disk gate had only reserved the small prequant checkpoint. That is verbatim the situation _dense_quant_prefetch_needed's own docstring exists to prevent. Gate the in-loader dense fallback on the shards being staged, read off the returned file list rather than the request so a failed size estimate closes it too, and let the GGUF build take over otherwise, which is what the prequant-sized replan already does one branch over. It is also the invariant the text-encoder path already enforces: only a component that really resolves may have its dense weights dropped from a plan or a prefetch. Adds the missing coverage for the gate's prequant arm, which the existing disk-gate tests never reached. * Refuse a training output_dir that resolves to the outputs root resolve_output_dir strips a leading 'outputs' and drops '' / '.' segments, so '.', './', './.', 'outputs', 'outputs/outputs' and ' . ' all clean away to nothing and land on the outputs root itself rather than a run directory under it. The DiT trainer then writes pytorch_lora_weights.safetensors flat into the root, where scan_trained_models and scan_checkpoints cannot see it (both filter is_dir()), and a second such run overwrites the first. The UI only checks the field is non-empty, so 'outputs' is one plausible run name away. Refuse it with a 400 that says what to do instead. cond_cache_dir collapses the same way but has an honest 'off' to fall back to, so a root-resolving value now means the in-memory cache, which is what the comment above it already promised: otherwise a run drops one flat safetensors per cached latent and caption into the directory trained models live in. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Repair three defects the main merge left behind Git merged all three files without a conflict, but the result was wrong in each case. The tests only started failing once the merged tree was run. routes/inference.py duplicated the GGUF load block: this branch had moved the gguf_load_in_flight marker and the _hub_download_blocks_gguf_load guard under "if config.is_gguf and config.gguf_hf_repo", and the conflict resolution re-added main's copy at the old position, so both ran. Dropped main's copy; the earlier placement is the deliberate one, so a 409 from the hub guard cannot tear down a resident Images or Video pipeline. test_gpu_selection.py still called _hf_offline_if_dns_dead, which main renamed to _hf_offline_if_unreachable_for (#7591). Disjoint edits, so no conflict, but four route-error tests referenced a function that no longer exists. test_gguf_load_cache_reuse.py anchored its ordering assertion with rindex over "if config.is_gguf:", taking the last one before the load marker. That only held while _resolve_inherited_extra_args sat above every such line; main has since hoisted it above the gpu_ids preflight, so the anchor landed between the call and the marker and the assertion compared against an unrelated later call site. The ordering it checks is a property of _load_model_impl as a whole, so it now anchors on the function. The ordering itself is intact: _resolve_inherited_extra_args, then the gguf_load_in_flight marker, then the hub guard, then the chat handoff, then unload_model. * Stop a real sd.cpp install from breaking its own discovery tests The five "nothing is installed" assertions in test_sd_cpp_engine.py cleared SD_CLI_PATH and UNSLOTH_SD_CPP_PATH and patched Path.home, which covers hops 1, 2 and the fallback half of hop 3. It leaves two hops live. Hop 3 goes through managed_install_root(), which honors UNSLOTH_STUDIO_HOME and STUDIO_HOME and resolves to the stable-diffusion.cpp directory beside the studio home. That is the documented way to run side-by-side Studios, so anyone who has one set gets a real sd-cli back from a finder the test expects to return None. Hop 4 is the in-tree developer build, which does the same for anyone who built sd.cpp inside the checkout. Isolated with an autouse fixture rather than another helper call, because reaching the failure needs no fixture: SdCppEngine(binary = None) runs the finder from its constructor, which is why test_generate_raises_when_binary_missing failed too. The fixture points the studio home and the in-tree root at an empty tmp tree, so every hop is answered by the test rather than by the box. The in-tree root moves into a named in_tree_install_root() so it can be pointed somewhere empty; behaviour is unchanged, including the OSError and IndexError guard on an unexpected layout. * Make the sd.cpp uninstall test actually extract the code it tests The two sed ranges anchored the production fragments at column 0, but both blocks sit inside the main removal function and are indented, so each range matched nothing and LOOP_FILE / DEFAULT_FILE came out empty. Sourcing an empty file is a no-op, so the suite reported 4 passed / 7 failed: the seven removal assertions failed because nothing ran, and the four that passed were 'kept' assertions that pass trivially when nothing runs. The shell job auto-discovers tests/sh/test_*.sh and runs each under set -e, and this file is not in its skip list, so it was a deterministic red. Anchor on optional leading whitespace and fail loudly on an empty or _remove_path-less fragment, so a future reshuffle of uninstall.sh cannot make the suite vacuous again. Now 11 passed, 0 failed, against the real removal loop and the real default-mode block. * Agree on what an engine can build, and on what the installer owns Two gates, each half-applied. The unbuildable-family gate had one caller, the image branch of the cached repo picker. The GGUF classifier, the local-model classifier and the video branch had none, so on a diffusers too old for a family the picker still offered its GGUF and the load then failed. Meanwhile the loader asserted the diffusers pipeline class before engine selection, so a GGUF this host routes to native sd.cpp, which instantiates no pipeline class at all, was refused with an upgrade instruction that could not help. Both are wrong one-sidedly: hide a family only when NEITHER engine can build it, and demand the diffusers class only when diffusers is what will load it. One predicate, family_buildable_here, now answers both, so the picker and the loader cannot disagree. The population is real on Python 3.9, whose diffusers ceiling is 0.36: Flux2KleinPipeline, Krea2Pipeline and LTX2Pipeline are all absent there, and FLUX.2-klein GGUF is a repo in this PR's title. That assertion also raised RuntimeError, which /images/load maps to 409, the status that otherwise means a load is already in progress, and which escaped /images/download-plan (it catches ValueError and FileNotFoundError) as a bare 500 with the message lost. It is an unloadable pick like every other, so it raises ValueError and both routes answer 400 with the text intact. The managed-binary repair used a path test for ownership while the installer uses a marker. On a managed root without the marker, which is any install predating it, the repair deleted sd-server and the reinstall was then refused, permanently, because the surviving sd-cli keeps the directory non-empty so the marker can never be claimed: the user went from an unrunnable binary to no binary and no way back. Require the marker before discarding, which is the same definition of ours that uninstall.sh already uses to keep a user's own stable-diffusion.cpp checkout. A genuinely interrupted extraction still self-heals, since install() writes the marker before it extracts. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add gguf and av to the studio extra so the wheel install matches studio.txt This branch added both to studio/backend/requirements/studio.txt, which is what install.sh uses, but never to the studio extra in pyproject.toml, which is what 'pip install unsloth[studio]' uses. Nothing else keeps the two in sync, so tests/studio/install/test_studio_extra_matches_requirements.py failed deterministically, and the CI job runs the whole tests/ tree. The drift is not only a red test: gguf backs diffusers' GGUFQuantizationConfig and av does the MP4 encode and audio mux, so a wheel install got a Studio that cannot read a GGUF or export a video, which is most of what this PR adds. * Fix two test-suite defects: the CPU-only patch gate and a popped module The patch backend reaches unsloth_zoo's helpers only through an ~940 MB 'import unsloth', which a CPU-only host cannot complete, so every patch install returned False and nine arch/eager/compile tests failed there -- exactly the runners the retry was narrowed to protect. unsloth_zoo only wants UNSLOTH_IS_PRESENT in the environment, which costs nothing and needs no accelerator, and the conftest already sets its sibling UNSLOTH_ALLOW_CPU. Set it there too, as run.py and main.py already do at module scope, so no real server ever took the expensive route. CPU-only goes from 9 failed to 24 passed, 1 skipped; the GPU run is unchanged at 29 passed. test_setup_cache_env_hf_home popped utils.hf_cache_settings to model a fresh process and never restored it, so a later import built a second module object and rebound it on the utils package. test_hf_cache_settings then wrote its setting into one object while core.inference.diffusion read the other, the same split-module failure the xet shim already had. Restore both bindings in teardown: 21 pass together, and each file still passes alone. * Restore the setup_fail assertions the main merge reverted A fourth instance of the class 9541cc535 fixed: the merge took main's studio/setup.sh, which #7644 changed to abort through the setup_fail helper so desktop mode still emits [TAURI:ERROR], but kept this branch's older copy of the test, which still asserted the literal 'exit 1'. setup.sh is byte-identical to main here and the only diff in this file was the reverted assertions, so take main's version. 60 passed. * Fence the video teardown, reject non-finite clipping knobs, stop a test installing sd.cpp The video backend never had the teardown fence the image backend documents. Its unload and its superseding-load path both signalled the active generation, then did 'with self._generate_lock: pass' and tore the state down with the lock free. A generation queued behind that barrier holds no cancel event yet, so the signal cannot reach it, and Python locks are not FIFO: it won the lock the instant the barrier released it, read a still-loaded state and denoised a whole clip against the pipeline being freed. Reproduced on both paths, where the queued generation returned a finished MP4. Mirror the image backend: count waiters, refuse a generation while one is pending, and tear down inside the barrier with the counter released in a finally. _teardown_state becomes _teardown_state_locked since the caller now holds both locks. max_grad_norm and snr_gamma were bounded on one side only, so 1e309 floated to inf and passed. clip_grad_norm_ then computes an infinite clip coefficient, clamps it to 1.0 and scales nothing, and min(snr, inf) / snr makes every min-SNR weight 1.0. The run starts, reports normal progress and trains with the requested knob silently disabled. Measured both. NaN already failed the bounds; allow_inf_nan makes that explicit. learning_rate and flow_shift already guard this exact vector. test_load_routes_to_sd_cpp_on_cpu stubbed ensure_sd_cpp_binary but not ensure_sd_server_binary, which select_and_activate_engine probes first with installs enabled, so the unit test downloaded and unpacked 108 MB into the developer's real ~/.unsloth when UNSLOTH_STUDIO_HOME was unset. Now 0 bytes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Key the video schedule reset on the resolved defaults, not the repo id A GGUF repo holds several variants, so another client swapping a distilled build for the base one from the same repo changes the steps and guidance the backend reports while repo_id stays put. modelChanged stayed false, the page kept the previous schedule, and later generations ran an 8-step distilled setting on a model that expects roughly 40 steps and CFG 4. Include the reported defaults in the key. They only move when the resident artifact does, so a manual steps change still survives a status poll. * Retire the release-lag pin now that 2026.7.6 carries the relaxed gates The virgin Windows container row that installs from PyPI on purpose was tolerated with continue-on-error, because a Server Core container has no Microsoft Store and so no winget, and the released studio/setup.ps1 hard-stopped on a winget-only git gate and reached for winget again for the VC++ runtime. #7549 relaxed both, but it landed on 2026-07-28 and the newest wheel was 2026.7.5 from the 23rd, so the row could not get past it. The pin carried its own tripwire for exactly this moment, and it has fired: unsloth 2026.7.6 shipped 2026-07-29 and the released wheel now installs end to end, so the step reported success and the assertion errored with "delete this pin and drop continue-on-error from the Install step so this row gates". Doing that. Both rows now gate unconditionally, and the assertion that enumerated the accepted failure signatures goes with the pin it protected, since there is no longer an accepted failure for it to describe. The other three release-lag mentions in this file are descriptive rather than tolerated-failure pins: they explain why the fedora and ubuntu-nonroot legs run with overlay: true, and both already refuse a triton failure as a regression rather than accepting it as lag. Left alone. * Pin the hosted pre-quant onto the plain-torch fp8 kernel A Z-Image GGUF pick at the DEFAULT speed mode, once the hosted fp8 pre-quant is actually reachable, dies at generate with an HTTP 500: torch._dynamo.exc.Unsupported: Operator does not support running with fake tensors. Developer debug context: unsupported operator: mslk.f8f8bf16_rowwise.default _fp8_config already pins KernelPreference.TORCH when it BUILDS a config, precisely because the default AUTO switches to the MSLK kernel wherever an mslk package is importable (sm90+). A hosted checkpoint escapes that pin completely: the preference is serialized per Float8Tensor, and all 239 weights in the published Z-Image-Turbo-FP8 checkpoint carry AUTO. Loading it re-arms the very kernel the pin exists to avoid, mslk.f8f8bf16_rowwise has no fake impl, and the first compiled generate therefore cannot be traced. Isolated away from the product to be sure of the mechanism: quantise one Linear three ways on this box and compile each. KernelPreference.TORCH eager ok, compiled ok KernelPreference.AUTO eager ok, compiled FAILS on mslk.f8f8bf16_rowwise library default eager ok, compiled FAILS the same way So the pin is correct and necessary, and the only gap is that the hosted path never got it. _validate_checkpoint checks scheme, granularity, base model, min_features, exclude tokens and fast_accum, but not this. Rewriting the preference on load is safe: it selects a matmul kernel, it is not weight data, so the tensors stay bit-identical and the checkpoint's own state_dict_sha256 still describes them. It is also the faster path compiled, since the opaque extern call blocks inductor quantize fusion. Why this went unnoticed: the pre-quant repos are private, so nothing that could not read them ever took this path. It becomes the default the moment they are readable. Verified end to end at 1024x1024, 8 steps, speed_mode default, on the published checkpoint: before HTTP 500 at generate, diffusion.generate_failed after pinned 239 weights to the plain-torch fp8 kernel, load 10.0 s, transformer_quant fp8 with compiled engaged, cold 7.9 s, warm 0.87 0.87 0.87 0.86 0.87 s That warm number also beats the 1.4 s recorded for this shape on 2026-07-26. 159 prequant and transformer-quant tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cover the fp8 kernel pin so the 500 cannot come back quietly The hosted pre-quant re-arming MSLK was only visible as an HTTP 500 on a compiled generate, which needs a GPU, a readable private repo and an importable mslk to reproduce. None of that is available in CI, so the pin would rot unnoticed. Three hermetic cases on _pin_kernel_preference instead: AUTO weights are rewritten and already-TORCH ones are left alone (counting only what changed), a weight that refuses the assignment does not sink the whole load, and with no torchao enum available the checkpoint is left exactly as saved. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: tighten comments across the image generation, video and training code Rewrite the comments added by this branch to be shorter and clearer: collapse multi-line explanations into one or two lines, drop comments that only restate the code, and keep the rationale that explains why a choice was made. No code changes. * Studio: let the pipeline-class guard survive a host with no diffusers Backend CI installs the CPU-only dependency set, which has no diffusers, and runs without a GPU. 13 tests failed there for three separate reasons, one of which is a real product bug rather than a test-environment gap. assert_pipeline_class_available did a bare "import diffusers", so on any host without it the call raised ModuleNotFoundError instead of the ValueError its own contract promises. /images/download-plan catches only ValueError and FileNotFoundError, so that escaped as a bare 500 with the message lost, which is the exact failure the guard exists to prevent. It is reachable in production: the native sd.cpp engine serves GGUF picks on a CPU or Apple host where diffusers is never installed. Absent diffusers now returns, since the check answers "is the installed diffusers new enough for this family" and there is no version to judge; a pick that genuinely needs diffusers still fails in the loader with its own message. The rest were tests asserting through gates unrelated to what they cover: - Three cond_cache_dir route tests drive a DiT family, which the start route refuses without an accelerator. They now take the existing dit_train_host fixture, written for this case, so they keep testing the schema on every host. - test_download_plan_forwards_the_load_time_controls stubs the diffusers planner but sends a GGUF pick, which routes to the native planner on a GPU-less host. It now pins the engine, so it tests kwarg forwarding rather than the hardware. Engine selection keeps its own tests. - The pipeline-class guard test needed a real diffusers only for its sweep over every shipped family. That half is split out and skips on its own; the stub-driven refusal needs no diffusers and still runs. Added a test for the absent-diffusers contract above. 409 pass on a GPU host with diffusers; 301 pass and 1 skips with diffusers blocked and no GPU. * Studio: cut the image generation comments down further Second pass over the comments this branch adds. Collapse the multi-line blocks that still read as paragraphs, rewrite the longest one-liners so they say the same thing in fewer words, and drop a stale comment that had drifted away from the constant it described. No code changes: only comments and whitespace. * Studio: workflow rail, sidebar Video row, and Train settings polish Sidebar - Pin Video under Images by default; persist store bumped to v3 so an untouched install adopts it while a custom arrangement is left alone. - Add "Customize sidebar" to the end of the More flyout, opening Settings > Appearance scrolled to the sidebar nav section. Images - Replace the workflow dropdown with an icon rail down the left edge: all seven workflows visible, keyboard nav, tooltips carry the labels. - Workflows stay selectable with no model loaded, so one can be set up before picking a model. Generate is already gated on a loaded model. - Video moves out of the Create/Train strip into a link at the far right, with a matching Images link on the Video page. - Wider gutters around the settings column, headings matched to Train. Train - Field guidance moves into "i" tooltips; only state that limits a control stays on the page. Steps, LoRA rank and the rest gain hints. - /info reports params, qlora_vram_gb, gated and note as fields, so the family note renders as chips. vram_note is rebuilt from them unchanged for older clients. Shared - Number steppers appear on hover or focus. - Lighter control border (#e9e9e9) on Images and Video, light mode only. * Studio: move Advanced inline on Images and Video - Replace the top-bar toggle and right-docked Advanced panel with a disclosure under Seed, so load-time tuning sits with the settings it affects and opening it no longer shrinks the preview canvas. - Video uses the bordered variant, since offload and memory decide whether a model fits at all. - Open state persists per page in localStorage. - Tighten the Steps unit trigger: 14px chevron, less right padding. * Studio: full width media pages and layout polish - Drop the 1100px cap on Images Create, Images Train and Video, so the preview canvas grows with the window instead of sitting in a band. - Align the workflow rail to the model selector label, widen the gap before its divider, and restore the divider itself. - Advanced: more room above and below, icon and label sized to the slider rows, and the same quiet row on Video as on Images. - Give Video a pane heading and description, matching Images. - More space above Seed on both pages. - More no longer takes the active style when the current page is one of its own rows. * Studio: fix the Extend side toggles The resting outline used a ring, and index.css blanks the ring on the button holding mouse focus, so the side you just clicked lost its border until focus moved. Use a border for the outline and leave the ring to focus-visible. Also separates the two states properly (the off state had no surface of its own), drops the border in dark mode as the inputs do, and adds the aria-pressed and focus-visible styling the buttons were missing. * Studio: drop the border on the Extend side toggles Light mode should carry no border either, so the fill alone marks the state: a muted surface when off, a primary tint when on. Matches the borderless treatment dark mode already had. * Studio: move the Images workflows into the sidebar The workflow switcher was a vertical icon rail on the page. It now lives in the sidebar under Images, so the page keeps that width for the canvas. - Workflows list under the Images row, with a chevron to fold them away. Hovering the row peeks them in a flyout on the standard menu surface. - Clicking Images while already there toggles the list instead of navigating. - The listed workflow carries the highlight, so the Images row drops it. - Create takes its own icon: it was sharing the New chat pencil. - Model hub moves above Projects. Persisted layouts bump to v4 and only adopt the new order where the stored one is still a shipped default. - Images and Video content both start at 32px, clear of the sidebar. * Studio: simplify the Images workflow submenu Dropping the hover flyout: two ways to reach the same seven workflows, one of them overlapping the list right below it, read as clutter. - The flyout is gone. The workflows are rows under Images and nothing else. - On the Images page they are always open, since they are that page's switcher. Elsewhere they stay folded, and the row's chevron opens them. - Create takes the sparkles icon. - Create and Train both pad their settings column to 40px a side, so the two tabs line up with each other and with the model selector. * Studio: dock the generate action and tidy the Create controls The primary action sat at the foot of a long scroll, so it was off screen until you scrolled for it. - Generate, Video's Generate and Train's Start training float at the bottom of their settings column. No bar behind them: hover lightens the fill rather than thinning it, and the disabled state is opaque, so the controls underneath never show through. - Aspect ratios read as names: Square (1:1), Widescreen (16:9) and so on. - Width and height replace their sliders with two compact boxes. Type a size or pick one from the menu; the value still snaps into range and still drives the locked ratio. - Negative prompt gets a quiet disclosure under the prompt. It was there before but only above guidance 0, which is not the default, so it never showed. - The info "i" is smaller across the settings UI. * Studio: reveal Images and Video field hints on hover The "i" next to every field label sat there permanently, which made a column of settings read as busier than it is. Scoped to the diffusion pages by CSS rather than per component: the hints come from the page's own Field and SelectRow, the train panel's FieldLabel, and the chat ParamSlider, which is shared and should not change for chat. Reveals when the pointer is inside the field, on focus-within for keyboard, and stays inert while hidden. InfoHint grows a data-slot so the rule has a stable hook, rather than keying off its aria-label. * Studio: one-line sliders, edge fades, shared negative prompt - Steps, Guidance and the rest put label, track and value on one row. ParamSlider is shared with chat, so this is an opt-in prop and chat keeps the stacked layout. - The settings columns fade at whichever edge they run past, as the sidebar and model picker do, instead of cutting off. New useScrollFades hook drives all three. - Negative prompt moves to a shared component and Video picks it up, so both pages collapse it the same way. - Right gutter sits closer to the rule, Steps takes a bigger break above it, and the Advanced rule sits between its neighbours rather than up against the field above. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: add the API monitor to the sidebar nav Sits under More, below Export. Points at the monitor page, not the API keys dialog the profile menu opens. No store version bump: an id missing from a stored layout is appended with its default pinned state, so existing sidebars gain the row at the end and keep their own order. * Studio: take main's copy of the sidebar spinner test The merge kept this branch's older version, which pins the Recents row to h-[33px]. Main already relaxed that to any height, since row density moves independently of the trailing column the test is about. * Studio: fix FLUX.2 klein size resolution, HunyuanVideo tier routing, and the personalization save Five fixes found while auditing the model registries against the live Hub. FLUX.2 klein 9B loaded against the 4B config. One family covers both klein sizes and defaults to 4B, relying on the base_model card tag for the real base, but the trust allowlist only had klein-4B, so the correct tag was discarded and a 9B GGUF (inner_dim 4096) was loaded against a 4B config (3072). That surfaced as a bare shape mismatch from inside the GGUF quantizer naming neither the file nor the repo. Adds the missing allowlist entries and a header-level size check that fails early with a legible message; the check is fail-open, so an unreadable file, a non-FLUX.2 family, or an unmapped base leaves the load exactly as it was. The sd.cpp text encoder rule matched the literal "klein-9b", so klein-base-9B was handed the 4B encoder. HunyuanVideo-1.5 720p checkpoints routed to the 480p family. Only the literal 720p_t2v path was aliased, so 720p_i2v and every GGUF repack fell through to the generic token and inherited the 480p base repo, which also supplies the VAE and text encoder. The tier is baked into the weights (transformer target_size 640 vs 960, scheduler shift 5.0 vs 9.0) and the two bucket lists are disjoint, so this ran the whole pipeline off-tier. Resolution presets that the checkpoints were never trained for. Wan2.2 TI2V-5B is 720P-only upstream (SUPPORTED_SIZES is exactly 704x1280 and 1280x704, asserted in generate.py), and the HunyuanVideo 480p square preset 624x624 is not a bucket of its tier; 640x640 is. Personalization could never be saved. The frontend ships eight sidebar nav ids and refills every missing one on each save, but the backend Literal had seven and no "api", so every PUT to /api/settings/personalization returned 422. Aligns the server defaults and the Literal with the shipped layout. A catalog row pointing at a file that does not exist. unsloth/Qwen-Image-2512-FP8 holds torch prequant .pt checkpoints, not qwen-image-2512-fp8.safetensors, and fp8 is denied for this family anyway because it renders black, yet the row was the auto-route target on any GPU above 40 GB. Also marks FLUX.1-schnell, Krea-2-Turbo and the two Ideogram repos gated: all four are gated on the Hub today, and schnell being Apache-2.0 does not make it ungated. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: stop a case-only duplicate upload destroying an image, and size the load against the right GPU Two fixes from the review round. A dataset upload containing two names that differ only by case silently loses one on Windows and macOS. The in-batch duplicate check is an exact string compare, so "Cat.png" and "cat.png" both pass, but on a case-folding filesystem they are one destination: the commit step moves the first staged part aside, writes the second over it, then deletes the backup, while the response still reports both as uploaded. Captions collapse the same way and never even reach the image branch. No single folder can hold such a pair, but the Windows and macOS open dialogs flatten search and Recents results across folders into one multi-selectable list, which is a normal way a LoRA dataset gets assembled. Rejecting the pair everywhere would regress Linux, where the two really are different files with their own sidecars, so the check probes the filesystem once per process instead of keying off sys.platform: macOS also ships case-sensitive APFS volumes, and a Linux host can keep its Studio home on an exFAT or NTFS mount. A failed probe answers "case-sensitive", which leaves today's behaviour untouched. The quantization fit budget was sized against the wrong card under a reordering CUDA_VISIBLE_DEVICES. A bare "cuda" load lands on visible ordinal 0, but the reducer took the lowest PHYSICAL index, which the nvidia-smi path reports as index_kind "physical". Under "3,1" that sizes against GPU 1 while the pipeline loads onto GPU 3. The hook's own interface doc already promised the lowest visible ordinal. Ranks by visible_ordinal, falling back to index only for an older backend that omits it. * Studio: rework the diffusion dataset panel and align the pane gutters Upload: - one Upload button beside the dataset name, with an upload icon - picking files uploads them, no second confirm click - Add button beside the Training images dropdown, so images can go into a set that already exists - fall back to the upload form when a selected dataset no longer resolves Labeling grid: - two columns at any width; sm:grid-cols-3 crushed the tiles in a fixed width column - pin the caption size so the Textarea's md:text-sm does not outsize a tile - name the tile hover group; a bare one revealed every tile's Remove at once - Remove is an icon button on the image, not a word over the artwork - drop the amber tile fill, keep the No caption label - match the header, toggle and status text to the section's type scale Gutters: - the run area and the preview canvas now sit 40px off the rule, the gutter the settings column has off the page edge * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: give the dataset preview per-image hover and remove, and wrap it onto rows - each thumbnail is its own button, so it can carry an X that removes just that image; the strip was one button and could not nest a second control - hover lightens the image instead of drawing a border - wraps onto more rows rather than scrolling sideways, which also keeps the bottom corners rounded - samples 12 thumbnails, up from 8, now that a second row is available - a delete refreshes the panel's dataset counts * Studio: show the workflow icon beside the Images and Video headings Images reads it off WORKFLOW_TABS, so it is the same icon the sidebar submenu shows and it follows the active workflow. Video's heading is also Create, so it takes the Create icon rather than the nav row's film slate. * Studio: reject a duplicate LoRA id on the diffusion load path too DiffusionGenerateRequest already refuses a repeated adapter id, with a comment naming the hazard: _resolve_lora_set suffixes colliding adapter names, so the same id resolves the SAME adapter twice and set_adapters stacks both copies past the per-adapter weight bound. DiffusionLoadRequest bounds only the list length, and it matters more there. Generation-time stacking spoils one image; on the load path the adapters are baked into the quantized build before compilation, so the unintended combination rides every image until a reload. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: teach the GGUF reuse test double about holds_no_vram The resident-GGUF fast path consults llama_backend.holds_no_vram before asserting CHAT ownership, but two tests hand it a SimpleNamespace built before that attribute existed, so they raised AttributeError and returned a 500. A real LlamaCppBackend exposes it as a property; the doubles now carry it too. Product code is unchanged: these pass on main and were the branch breaking its own test, not a defect in the guard. --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: michaelhan <michaelhan2050@gmail.com> Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: Michael Han <107991372+shimmyshimmer@users.noreply.github.com> | 1 个月前 | |
Studio: add image generation, editing workflows and LoRA training with Unsloth GGUFs (#6763) * Tighten comments in the image stack tests and scripts * Close video single-file, training reservation, and image mount-resume gaps Route on-device single-checkpoint video folders through the single_file loader: a bare local .safetensors directory (no model_index.json) is advertised as a pipeline with no filename, so validation rejected it before it could load. Reinterpret the pick as a single_file load of the sole checkpoint, mirroring the image load route. Treat a reserved-but-not-yet-spawned LLM training start as active in is_training_active() so /images/load, /video/load, and /diffusion/start cannot race the reserved run for VRAM during the pre-spawn free window. Mirrors the diffusion training service reservation. Resume an in-flight image generation on the Images page mount: probe generate-progress, re-enter the poll loop, and refresh the gallery on completion so a run started elsewhere is reflected and its saved image appears without a manual refresh. Seed resident image defaults from the resolved base_repo rather than a possibly path-shaped repo_id so the first resident generation uses the right recipe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Publish image generation active state before pre-denoise setup generate() assigned self._gen only at the pipe() call, after deferred compile, LoRA resolution/application, and ControlNet download/build had run. Across that setup window generate_progress() reported inactive even though _generate_lock was held, so a reloaded page's mount probe showed idle and let a second generate queue behind the first. Publish an active step-0 _GenState the moment the generation lock is acquired, before the setup work, and clear it in the outer finally so a setup-time error cannot leave the UI stuck active. Mirrors the video backend's queued phase and the training start guard. * Studio: fix diffusion install ownership, dataset upload atomicity, gallery pagination, and teardown races install_sd_cpp_prebuilt: only write the .unsloth-studio-owned marker when the install created the target directory or it was empty. Adopting a pre-existing, unowned, non-empty directory (a user's own stable-diffusion.cpp checkout) made it eligible for the uninstaller's recursive delete. routes/training upload: make the multi-file promotion transactional. Back up each displaced original and roll every destination back on any failure, so a mid-loop rename error can no longer partially overwrite the live dataset. routes/training _resolve_dataset_folder: reject a symlinked dataset directory and prove the resolved folder stays under the datasets root, so image read/caption/delete cannot escape the root through a link. routes/training delete: escape glob metacharacters in the thumbnail filename so deleting an image named like [ab].png removes only its own thumbnails. image_gallery / video_gallery listing: filter records against the response schema inside the pager via a valid callback, so offset/limit/has_more all count over accepted records. A leading schema-invalid record no longer returns an empty page with has_more=true and stalls infinite scroll at offset 0. image_gallery / video_gallery save: publish via a temp file plus atomic rename (the sidecar is the video pair's commit marker) and clean up on failure, so a partial write never surfaces a truncated PNG or strands an orphan MP4. diffusion_train_common discovery: treat an empty caption sidecar as a metadata tombstone that still falls through to the dreambooth instance prompt, so clearing every metadata caption no longer fails with no captioned images found. diffusion backend unload: wait for an in-flight denoise to exit before tearing down process-wide patches and state, mirroring the load path. diffusion_engine_router: serialize the whole check/unload/publish transition so a concurrent selection cannot return the engine being unloaded. uninstall.ps1: gate the default sd.cpp process stop on the owner marker so a user's own sd-server is not terminated for a directory we then keep. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reject native batch seeds outside the JSON-safe range * Refuse sd.cpp install into unowned non-empty target dir When the install target already exists, is non-empty and lacks the .unsloth-studio-owned marker (a user's own stable-diffusion.cpp checkout, or unrelated files beside a custom Studio root), install() previously still extracted the release into it. Skipping the ownership marker only stopped the uninstaller from deleting the directory; extraction still merged binaries into the user's working tree and could overwrite same-named files. Fail up front with a clear message pointing the user at a fresh/empty location before any download or extraction, leaving their directory untouched. Update the ownership test suite to assert the refusal. * Studio: gate dataset uploads on the symlink check and surface local video single-file checkpoints * Tighten comments and docstrings added by the image-generation fixes * Studio: close arbiter load-registration race and surface native progress + local pipeline folders Publish native sd.cpp generate progress (_gen) before LoRA resolution so a reload probe reads active during setup, matching the diffusers path. Register the diffusion/video GPU load under the arbiter lock (acquire_for now takes a register callback) so a competing acquire cannot evict an owner before its load is marked in-flight and let two loaders allocate VRAM at once. Admit local diffusers pipeline folders (root model_index.json, weights in component subdirs) in the local model scan so they reach task tagging and the On Device picker. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: include marker-owned custom sd.cpp roots in the uninstall stop scan; harden Pester install against the nuget.exe PSGallery bootstrap * Studio: surface local pipeline scan roots, tag single-file checkpoints by filename, mark companion-only pipelines partial - _scan_models_dir: admit a scan folder that is itself a diffusers pipeline (root model_index.json, weights in transformer/ vae/ subdirs). _is_model_directory rejects such a root, so the child scan would list the component subdirs as bogus models and hide the real pipeline; treat the root as one model via _local_pipeline_index. - _local_is_diffusers / _local_model_task: include the sole checkpoint filename in the family-detection needles (_local_family_needles, resolved via resolve_local_single_file). A generically named folder holding one loadable qwen-image-*.safetensors / ltx-*.safetensors identifies its family only from the filename; the load route already resolves that file, so tag it or the task-scoped picker (which rejects task=null) hides the on-device model. - list_cached_models: mark a companion-only base snapshot partial. A GGUF image load prefetches the base repo's VAE / text-encoder / model_index.json but skips the transformer (the GGUF supplies it); the snapshot has a pipeline manifest yet is not a loadable BF16 pipeline, and _cached_repo_partial misses it. _repo_pipeline_missing_denoiser flags a pipeline snapshot whose transformer/ or unet/ component carries no weight, so the picker drops it instead of advertising it as fully on-device. * Studio: preserve foreign gallery files, force safetensors on remote ControlNets, and close dataset/seed/GPU gaps Gallery clear/delete now scope to Studio-owned files: image_gallery and video_gallery skip PNGs / MP4s without a readable recipe (a hand-dropped or orphan file the listing already hides), so clear() and a guessed-id delete no longer destroy files the gallery never surfaced. Remote ControlNets now force use_safetensors: a bare owner/name reaches from_pretrained without the base trust gate, and the Hub scan fails open when unavailable, so requiring safetensors closes the pickle deserialization vector. POSIX uninstall now stops resident sd-server / sd-cli under an owned sd.cpp root before removing the tree (marker-gated), mirroring the Windows stop-before-delete scan; a live native server no longer survives unlinking its binary. Diffusion dataset containment: the training-start read path and the discovery picker route bare names through the protected resolver, so a symlinked dataset is rejected / not advertised like the caption/delete routes already do. Uploads gain the inference decode guard (oversized real images 400 before OOMing the trainer) and dataset upload/caption/delete/import are blocked with 409 while a diffusion run is active. JSONL readers (trainer + routes) tolerate non-object JSON and invalid UTF-8 instead of raising AttributeError / 500. LoRA family compatibility is enforced in the shared resolver, not only the picker, so a direct API client cannot apply a mismatched-family adapter. GPU arbiter gains release_if so the image/video unload idle-check and release are atomic against a concurrent same-owner load's registration. Native batch recipes persist the base batch_seed and restore replays from it, so a native batch_index>0 image no longer advances its seed twice. FLUX.2-klein selects its sd.cpp text encoder by variant (4B -> Qwen3-4B, 9B -> Qwen3-8B) instead of the single family default. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: track the loaded GGUF filename so native companion resolution reproduces the load identity * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: gate local-pipeline image tagging on a real family; validate video sidecars before delete/clear * Studio: tighten image-generation fix comments and docstrings * Studio: gate gallery serve/export on ownership; keep image progress active until persisted; reserve diffusion training before the dataset scan * Studio: restore Reapply target on async image/video load errors; recheck training state before dataset commit; reclaim partial sd.cpp installs on retry Images/Video: a background model load that fails AFTER starting (error/eviction during download) leaves the previous pipeline resident, but handleLoad had already overwritten lastLoad.current with the failed pick, so "Reapply to loaded model" reloaded the failed model. Carry the prior Reapply target into the poller and restore it on the async error/null paths, mirroring the quant rollback. Training: an in-flight diffusion dataset upload passed _require_diffusion_dataset_mutable() at entry but could still commit files after a concurrent /diffusion/start reserved the training slot, mutating the dataset underneath the trainer. Re-check the interlock immediately before the commit phase; a 409 there leaves the staged temps for the finally to clean. sd.cpp install: an interrupted extraction (disk full, killed process, a raising post-extract cudart fetch) left the target non-empty with no owner marker, so the next lazy install tripped the "not a Studio-managed directory" refusal and wedged native install. Write the ownership marker before the partial writes when the target is reclaimable, so a retry recognises the debris as ours and re-extracts. * fp8 DiT quant: floor the dynamic activation scale with activation_value_lb An all-zero activation token row makes the dynamic per-row fp8 scale 0, which turns the quantized data to NaN and the render to black frames on torchao's plain-torch kernel path. The fused fbgemm/mslk quantize kernels clamp zero rows internally, so the bug only reproduces on machines without them, which is most user environments. Zero rows are real inputs, not a corner case: Wan 2.2 zero-pads its text conditioning, and Hunyuan-1.5 and Qwen-Image regenerate zero rows inside their transformer blocks every step. Pass activation_value_lb=1e-12 to Float8DynamicActivationFloat8WeightConfig whenever the installed torchao supports the kwarg (Float8Tensor rework, 0.13+), checked via inspect.signature so older torchao keeps exactly the current behaviour; the existing Float8MMConfig fallback chain is unchanged. Verified on GPU: with the forced plain-torch kernel path a zero-row input NaNs without the floor and stays finite with it, and end to end on HunyuanVideo-1.5 fp8 goes from a solid black frame (LPIPS 1.00) to a normal render (LPIPS 0.225); on Wan the floor matches the condition_embedder exclusion (LPIPS 0.211 vs 0.206). Same-seed renders with fused kernels present are unaffected, and pre-quantized fp8 checkpoints stay valid since weight scales are untouched. * Wire hosted pre-quantized DiT checkpoints into the image families Point prequant_repos for flux.1, flux.2-klein, flux.2-dev, qwen-image (int8 only there; fp8 is family-denied), z-image and krea-2 at the unsloth/<Model>-FP8 Hub repos carrying gate-validated int8 and fp8 transformer checkpoints, so the fast quant path loads the small pre-quantized file instead of materialising the dense bf16 transformer and quantising on device. Measured on FLUX.2-dev int8: build peak drops from 60.7 GB (dense + quantize) to 30.7 GB (hosted prequant), identical 30.7 GB resident after either path since loading a checkpoint is bit-identical to on-the-fly quantisation. The hosted repos name files <Model>-<SCHEME>.pt, so resolve_prequant_source now derives that model-name filename from the repo id (scheme suffix stripped case-insensitively) and carries the legacy transformer_<scheme>.pt as a fallback the resolver tries when the primary 404s, keeping older repos loadable. Wiring a repo also exposed a fallback hazard: with a prequant source present, the dense-fit preflight used to be skipped entirely, so a failed prequant download would fall through to the dense bf16 load the memory plan never budgeted, OOMing after eviction. The preflight now always runs and gates an allow_dense_fallback flag through _load_dense_quant_pipeline: a dense misfit still skips the fast path when no prequant exists, but with one it proceeds and a prequant failure raises to the GGUF build instead of loading dense. The same flag is set when the auto-policy replans an offloaded GGUF against a prequant-sized transient. Tests updated to the new filename convention plus new coverage for the derivation and the legacy-name fallback; the prequant-skips-refit test now asserts the re-check runs and forbids the dense fallback. Verified end to end on GPU: z-image int8 resolves the hosted repo, downloads the model-name file and renders (6.8s load, 5.9 GB peak). * Route krea-2 through its per-component loader on the transformer-quant fast path _assemble_pipe used Pipeline.from_pretrained for every family, but the krea repo ships transformers-5.x configs and no top-level tokenizer files, so the tokenizer dies with vocab_file=None. The pre-quantized checkpoint loaded fine and then the assembly crashed, dropping the load to the GGUF build, which krea-2 cannot take (Krea2Transformer2DModel has no from_single_file). Assemble per-component via load_krea2_pipeline like the pipeline-kind and single-file paths already do. Verified live: Krea-2-Turbo int8 and fp8 hosted prequant loads now assemble and render through the Studio images tab. * Keep Qwen-Image's text-stream linears bf16 on int8 (short prompts break torch._int_mm) Qwen-Image's MMDiT runs every text-stream Linear at M = actual prompt tokens: the Qwen2.5-VL embeds are not padded to a fixed length like FLUX's 512-token T5. A short prompt (13 tokens) or the near-empty negative prompt drives torch._int_mm below its M > 16 floor and the first denoise step raises 'self.size(0) needs to be greater than 16, but got 13' (measured on B200 through the Studio images tab). Add per-family int8 exclusions (txt_in, add_q/k/v_proj, to_add_out, txt_mlp) for qwen-image and qwen-image-edit, threaded through exclude_tokens_for_scheme(scheme, family) and the prequant checkpoint validation, so a checkpoint baked under the old token list is rejected and re-quantised instead of loaded crashing. The text stream runs at M = tens vs the image stream's M ~ 4k, so the exclusion costs nothing; the rebuilt hosted checkpoint gates 28/28 PASS with LPIPS mean 0.057 (was 0.069). * Harden the diffusion memory plan against transient free-VRAM undercounts A cold FLUX.2-dev int8 load on an idle 183 GB B200 planned offload=model (companions exceed budget) and silently served the GGUF as-is; the identical retry went resident and engaged the hosted prequant. The plan arithmetic was byte-identical across both loads (required 90,228 MiB, resident needs free of about 124 GB); the only divergent input was torch.cuda.mem_get_info, which is device-wide and instantaneous: a transient foreign CUDA context briefly held about 100 GB at the first snapshot, and the planner trusted that single read. Three changes: - settled_snapshot_device_memory: on cuda, synchronize + empty_cache (best-effort) and take the MAX free over up to 3 spaced reads. A transient can only shrink free, so the max rejects transient undercounts while a persistent tenant still caps every read. _plan_memory now uses it. - plan_fits_total_capacity + one replan retry: when the dense/prequant candidate fits TOTAL device capacity under the standard reserve and the 0.85 resident margin, an offload verdict can only stem from the free reading, so the loader re-snapshots and replans once before declining the fast path. Explicit balanced/low_vram modes skip the retry (they offload by mode). - diffusion.transformer_quant_declined log line with required/budget/free and the plan reasons, so the next decline is diagnosable from the server log (previously silent). Verified: cold FLUX.2-dev int8 first load in a fresh server now engages the hosted prequant resident (offload=none). * Add FLUX.2 Klein and FLUX.2-dev DiT LoRA training Register flux.2-klein and flux.2-dev in the DiT trainer following the upstream DreamBooth references: latents train patchified and batch-norm normalized from the VAE posterior mode, the packed forward reuses step-invariant position ids, and the guidance vector (3.5) is gated on the variant's guidance_embeds config. Conditioning stacks load per variant (Mistral via Flux2Pipeline for dev, Qwen3 via Flux2KleinPipeline for Klein) and are encoded and freed before the transformer lands on the device. The fused single-stream to_qkv_mlp_proj joins the attention projections in the LoRA targets; the single-stream out projection stays dense because its to_out suffix would also match the double-stream ModuleList container. Wire both families through the training registry (family set, labels, VRAM notes, rank 16 / lr 1e-4 defaults, bf16-only preflight), mark them trainable with train base repos in the family registry, add FLUX.2-dev to the gated-repo token check, and trust both official bases for training downloads. Verified on B200: 30-step klein int8 (19.6s) and nf4 (20.9s) and dev int8 (52.0s) runs train with finite decreasing loss and the saved adapters apply on the bf16 base pipeline (weight 0 reproduces the base image exactly, weight 1 visibly restyles it). * Support LoRA adapters on torchao int8/fp8 quantized image pipelines Adapters are baked at load time: they attach to the dense transformer, then quantize_ converts only the frozen base linears (the lora_ side path is excluded by name), then the loader compiles. Post-quant PEFT injection is not possible on a manually quantized module, so the prequant shortcut is skipped for a baked load and the memory plan is sized for the dense build (force_dense on the quant candidate). At generation time the baked topology is frozen: weight tweaks and disabling (scale 0 reproduces the quantized base exactly) go through set_adapters, while adding or removing adapters returns a clean 400 telling the client to reload with the new selection. supports_lora now returns True for int8/fp8 diffusers loads (checked before the gguf-kind early return, since the quant fast path keeps the picker kind); nvfp4/mxfp8 and GGUF-via-diffusers stay blocked. The load request model takes an optional loras list, threaded through begin_load on both engines (native ignores it and keeps applying LoRA at generation). Verified end to end on GPU: Z-Image GGUF picker + int8 + trained adapter loads through the API, bake marker logged, weight 1.0 vs 0 renders differ visibly, weight 0.5 accepted live, unknown adapter rejected as 400. Affected suites: 296 passed. * Add FLUX.1 Krea dev to the image model catalog Krea's guidance-distilled FLUX.1-dev finetune keeps the exact dev layout, so it runs under the existing flux.1 family unchanged. Wire it up end to end: - Catalog group with the gated official bf16 pipeline and the open QuantStack GGUF quants; the gated artifact is skipped on auto-routing when undownloaded. - Trust the official repo for non-GGUF from_pretrained loads, next to the other black-forest-labs bases. - Generation defaults: 28 steps at guidance 4.5 per the model card. The generic "krea" defaults key (Krea-2-Turbo's 8-step no-CFG recipe) used to swallow the id, which would have produced garbage output; the new flux.1-krea key precedes it on both the backend table and the images page table. - The flux.1 prequant checkpoints are schnell-based; the loader's baked base_model_id validation refuses them for the Krea-dev base, so int8/fp8 requests dense-quantize instead (covered by existing prequant tests). * Resolve pre-quantized checkpoints per base variant One family entry covers several published variants whose weights differ (flux.1: schnell, dev, Krea-dev), but prequant resolution was keyed on (family, scheme) alone, so only the default base could ever be served: the loader's baked base_model_id validation correctly refused the schnell checkpoint for dev and Krea-dev bases and every such load paid the dense download plus on-the-fly quantise. Add an optional prequant_variant_repos table on DiffusionFamily as (base_repo, scheme, repo_id) triples and thread the resolved base repo through resolve_prequant_source / usable_prequant_source and their three call sites (load fast path, memory-plan probe, auto-policy candidate). A base without its own entry keeps returning the family default, preserving the existing refuse-then-dense behavior exactly. Wire the flux.1 variants: the gate-validated unsloth/FLUX.1-dev-FP8 checkpoints (built in the earlier campaign but never reachable) and the new unsloth/FLUX.1-Krea-dev-FP8. * Add the Lumina Image 2.0 family to the image catalog Alpha-VLLM/Lumina-Image-2.0 is a 2.6B single-stream DiT with a Gemma2-2B encoder and a standard 16-channel VAE, all transformers-4.x-compatible, so the generic from_pretrained pipeline path loads it as a new lumina-2 family: - Family entry (Lumina2Pipeline / Lumina2Transformer2DModel), aliased to lumina-image-2.0 / lumina-image-2 / lumina2. No bare lumina alias: Lumina-Next checkpoints are a different arch and must stay unknown rather than crash mid-load. bf16-only upstream, so the fp16 fallback stays off like z-image. - Trust the official repo for non-GGUF loads; bf16 component table entry (ships fp32, ~5.2 GB transformer + 5.2 GB encoder bf16-resident). - Generation defaults 50 steps / guidance 4.0 per the model card, and the generate call passes the card's cfg_trunc_ratio=0.25 itself (family-gated, signature-gated): the pipeline default (1.0) runs the CFG double-forward on every step and oversaturates output. - Catalog group with the single ungated bf16 pipeline artifact (11 GB resident) plus routing assertions; images page defaults row. - No GGUF artifact: none exists upstream (only finetune/LLM quants), so the dense transformer_quant fast path (GGUF-kind-only) stays unreachable for now. Offline probes of the future prequant campaign: int8 and fp8 both engage and render cleanly (fp8 LPIPS 0.11 vs bf16, int8 0.33 from 50-step trajectory drift with intact quality), so neither scheme is family-denied. * Wire the hosted Lumina Image 2.0 int8/fp8 checkpoints Gate-validated against same-seed bf16 renders (28/28 pairs per scheme, zero failures): int8 LPIPS mean 0.146 / SSIM 0.937, fp8 LPIPS mean 0.116 / SSIM 0.946. Uploaded to unsloth/Lumina-Image-2.0-FP8 following the existing checkpoint repo conventions. * Add the HunyuanImage 2.1 family to the image backend The hunyuanvideo-community diffusers mirror carries the full stack in standard layout: a 17B dual-stream DiT (32.5 GB bf16), a Qwen2.5-VL text encoder, a ByT5 glyph encoder, the 32x HunyuanImage VAE, and guider/ocr_guider components (AdaptiveProjectedMixGuidance) that diffusers 0.39 loads natively, so the generic from_pretrained pipeline path covers everything with no per-component assembly. Family notes: - The call's guidance knob is distilled_guidance_scale (there is no guidance_scale kwarg), so cfg_kwarg routes the UI value there; real CFG runs inside the repo's guider at its baked scale. Defaults follow the card recipe: 50 steps, 3.25. - 2K-native: verified live at both 1024 and 2048. - Coexists with the HunyuanImage-3.0 structured exclusion (3.0 has no diffusers pipeline and stays excluded with its stated reason). - int8/fp8 dense quantization verified live (LPIPS 0.186 both vs same-seed bf16); a short prompt does not trip the int8 torch._int_mm minimum on this arch, so no family exclude entry is needed. - bf16 component table for the memory planner: (32.5, 16.3, 0.8) GB. * Surface HunyuanImage 2.1 in the image model catalog Catalog group with the open bf16 mirror pipeline (~50 GB resident, so a bare click on a consumer card routes to the QuantStack GGUF quants, which load and render through the generic GGUF path, verified live) plus the images page defaults (50 steps, guidance 3.25 feeding distilled_guidance_scale). * Add the HiDream-I1 family to the image backend A 17B MoE DiT (16 double + 32 single layers, 4 routed experts) with four text encoders, on HiDreamImagePipeline (diffusers 0.39). One family covers the open Full / Dev / Fast repos (same arch); per-variant generation defaults follow the upstream inference recipes (Full 50 steps at guidance 5, the distilled Dev 28 and Fast 16 guidance-free). The repos name a Llama-3.1-8B text_encoder_4 in their model_index but do not ship its weights; the official example passes the gated meta-llama repo in by hand. The loader instead assembles the component from the open unsloth mirror (byte-identical weights, already inside the non-GGUF trust gate), injected at the three pipeline from_pretrained sites, with output_hidden_states matching the official example. Memory planning counts the assembled TE4: 34.2 GB DiT + 28.8 GB encoders, ~63 GB bf16-resident. * Surface HiDream I1 in the image model catalog One catalog group with the three official bf16 pipelines (Full, plus the Dev and Fast distillations as labeled artifacts) at their ~63 GB resident size, so auto-routing keeps this a datacenter-GPU pick. city96's GGUF is deliberately not wired: the GGUF path would need the same Llama TE4 assembly for very small demand. Images-page defaults mirror the backend table with the variant keys ahead of the generic hidream key. * Pin the measured HiDream quant verdict in tests int8 and fp8 both engage and render cleanly on this family, including short prompts on int8: the routed MoE expert Linears only ever see the concatenated image+text stream (M far above the torch._int_mm minimum), so no deny entry and no family exclude tokens are warranted. Assert that so a future table edit cannot silently regress the measured behavior. * Wire the hosted HunyuanImage 2.1 int8/fp8 checkpoints Verified bit-identical to on-the-fly quantize: all 1264 state dict tensors (456 quantized) dequantize equal between the loaded checkpoint and a fresh quantize_ pass, so quality matches the runtime Dtype path exactly. Same-seed LPIPS suite means (0.35 int8 / 0.28 fp8) blend trajectory divergence with this family's own run-to-run nondeterminism (identical weights and seed reproduce a 17/255 mean pixel delta through the 50-step guider pipeline); per-case hard checks pass and the drift is compositional, reviewed visually. Uploaded to unsloth/HunyuanImage-2.1-FP8. * Fix silent LoRA drop and wasted transformer prefetch on GGUF quant loads Two live-test findings on the images load path: - transformer_quant with baked LoRAs, when the dense quantized build is declined for memory or fails: the load completed as a plain GGUF with the adapters silently dropped (HTTP success, supports_lora=false after the fact) -- wrong output with no signal. The load now fails with the recovery options (drop the adapters, free VRAM, or pick a smaller model). Weight-0 adapters still count as no bake request, and the plain no-LoRA decline keeps its silent GGUF fallback. - A fresh GGUF load on a small GPU prefetched the base repo's full bf16 transformer shards (~47 GB on Qwen-Image) because the dense-quant prefetch widening only checked scheme viability, not whether the device could ever hold the candidate resident. Gate the widening on total device capacity (reserve + 0.85 margin, the plan_fits_total_capacity bar) so a card that is certain to decline the dense build never pays the download; capable devices keep the prefetch. * Fix video progress under-reporting during load and generate Two live-test findings on the video progress endpoints: - load-progress downloaded_bytes froze mid-download: the counter used scan_cache_dir, which skips in-flight *.incomplete blobs, so it sat at the last completed blob for the whole multi-GB shard pull while the disk kept filling. Count the repo's cache directory directly (completed plus incomplete blobs, snapshot symlinks skipped so nothing is double-counted). - generate-progress reported total_steps=null / fraction=0 while step advanced: the video API only carried the native total field while the image API exposes total_steps and fraction, so one poller could not work against both. Derive the image-compatible aliases in generate_progress and declare them on the response model; the native total stays for back-compat. * Wire the hosted HiDream I1 int8/fp8 checkpoints Gate-validated: all 28 per-case pairs pass per scheme (LPIPS suite means 0.291 int8 / 0.278 fp8, in the 50-step trajectory-divergence band; CLIP delta means 0.007-0.008), and the int8 checkpoint is verified bit-identical to on-the-fly quantize across all 1615 state dict tensors (1073 quantized, max abs diff 0.0). Uploaded to unsloth/HiDream-I1-Full-FP8. * Add a pre-cast text-encoder loader for the layerwise fp8 scheme The runtime text_encoder_quant=fp8 path downloads the full bf16 text encoder and layerwise-casts it in place on every fresh load. For the heavyweight encoders (LTX's Gemma3-27B ~50 GB, FLUX.2-dev's Mistral-24B ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) that download dominates load time on a fresh machine. diffusion_te_prequant.py loads a pre-cast fp8-storage state dict instead: meta-init the encoder skeleton from the checkpoint's te_class, load_state_dict(assign=True), rebuild on CPU if non-persistent buffers stay on meta, then re-apply the same layerwise cast to install the upcast hooks. The cast is a deterministic storage transform, so the loaded encoder is bit-identical to dense-load-then-cast by construction. v1 hosts the layerwise fp8 storage scheme only: its state dict is plain tensors (torch.load(weights_only=True), no pickle execution). The dynamic-compute schemes (fp8_dynamic, int8, nvfp4) build torchao subclass wrappers at runtime and are deliberately not hosted. Checkpoints validate format, scheme, component and base_model_id before use and any problem falls back to the dense download and cast. Local path overrides reuse the DiT prequant allowlist env var. Families opt in via a new te_prequant_repos (scheme, component, repo_id) field on both DiffusionFamily and VideoFamily; the field defaults empty so nothing changes until a gate-validated artifact is wired. * Inject hosted pre-cast text encoders during pipeline assembly Wire te_prequant_pipe_kwargs into the three pipeline assembly sites: the diffusion full-pipeline branch, the diffusion transformer-only and GGUF branch (where the companion TE is the big remaining download), and the shared video assembly path before the pipeline/component split. Injection is gated exactly like the runtime cast (mode normalized to fp8, device supported, family not denied), so it can never engage where quantize_text_encoders would not; the later quantize_text_encoders call re-applies the cast idempotently and keeps status reporting truthful. With no hosted checkpoint configured the call returns {} and assembly loads the dense encoder as before. * Add the pre-cast text-encoder checkpoint builder Applies the runtime layerwise fp8 storage cast to a model's dense text encoder once and saves the cast state dict with baked metadata (format tag, base_model_id, family, scheme, component, te_class, versions) in the layout diffusion_te_prequant.py validates. Resolves the encoder class from the checkpoint's config.architectures so the recorded te_class matches what the pipeline instantiates. CPU-runnable: the cast touches storage dtypes only. * Test the pre-cast text-encoder load path Hermetic CPU coverage for diffusion_te_prequant: the checkpoint filename convention, family-table resolution by scheme and component with malformed entries skipped, resolution priority (path override, hosted repo, none) and the fp8-only scheme gate, the checkpoint validation matrix (wrong format, missing state_dict, wrong scheme, wrong component, wrong or missing base_model_id) with base case folding, the local-path allowlist refusal and missing-file fallback, and the assembly injection gating (mode, hosted entry, device support, family deny, load failure, successful injection). Also pins the te_prequant_repos field on both family dataclasses and that no family ships a hosted TE checkpoint until the campaign wires one. * Fix pre-cast TE checkpoint loading and engagement reporting Two bugs found while building the hosted checkpoints: - The builder recorded torch.__version__ (a TorchVersion object) in the checkpoint metadata, so torch.load(weights_only=True) rejected every artifact and the loader silently fell back to the dense download. Record plain strings. - Re-applying the layerwise fp8 cast to an injected pre-cast encoder raised on the duplicate hook registration, making quantize_text_encoders report the engaged cast as failed (status showed no TE quant while the encoder ran fp8). _cast_fp8 now returns early when the hooks are already installed. Also corrects the LTX TE size note: Gemma3-12B stored fp32 (~49 GB), not 27B. * Wire the hosted pre-cast fp8 text encoders qwen-image and flux.2-dev (diffusion) and ltx-2 (video) now resolve a hosted pre-cast fp8 text encoder from their unsloth -FP8 repos: - unsloth/Qwen-Image-FP8: Qwen2.5-VL-7B, 16.6 GB dense -> 8.8 GB - unsloth/FLUX.2-dev-FP8: Mistral-Small-24B, 48.0 GB dense -> 24.7 GB - unsloth/LTX-2-FP8: Gemma3-12B, 48.7 GB fp32 store -> 13.2 GB Every checkpoint verified bit-identical to dense-load-then-cast (729 / 585 / 1066 tensors, zero mismatches) and smoke-tested through the real backends with the repo engagement marker. Tests cover the wired entries, the resolver filenames, builder metadata weights_only survival, and the idempotent re-cast. * Report the compute dtype on fp8-cast encoders and inject the pre-cast TE on the dense fast path Two more findings from the hosted-TE GPU smokes: - Module.dtype reports the first floating parameter, which after the layerwise fp8 cast is the fp8 STORAGE dtype. Flux2 derives its prompt embed and latent dtypes from encoder.dtype and feeds them to randn_tensor, which has no fp8 kernel, so ANY flux.2 load with text_encoder_quant=fp8 crashed at generation (pre-existing, runtime cast included). The cast now swaps in a subclass whose dtype property reports the compute dtype; forward behaviour is unchanged. - The dense transformer_quant fast path assembles companions through _assemble_pipe, which never received the pre-cast TE injection, so the hosted encoder engaged on full-pipeline and GGUF builds but not on the fast path. Threaded through like the other two branches. Verified live on B200: qwen-image (full pipeline), flux.2-dev (GGUF picker with int8 DiT prequant), ltx-2 (video backend) all engage the hosted TE, render non-black, and report text_encoder_quant=fp8 truthfully. * Key the fp8 cast idempotency on an explicit completion marker Hook presence alone cannot distinguish a legitimately pre-cast text encoder from leftover hooks after a cast that failed mid-pass, so the early return now requires the completion marker _cast_fp8 sets once the hooks are fully installed. Leftover partial state keeps failing closed. Also tolerates non-Module encoder doubles in the hook probe and the dtype override. * Extend the fp8 TE quant to HiDream's Llama text_encoder_4 The generic quantize_text_encoders pass only covers text_encoder.._3, so HiDream's HEAVIEST encoder (Llama-3.1-8B TE4, 16.1 GB bf16) always stayed dense. TE4 is assembled separately (hidream_te4_kwargs), so the fp8 path now lives there: when the requested TE quant is layerwise fp8 and the device/family qualify, TE4 prefers the hosted pre-cast checkpoint (unsloth/HiDream-I1-Full-FP8, 8.6 GB) and falls back to dense-load-then- cast; a mid-pass cast failure reloads a fresh dense encoder instead of shipping partial state. The pre-cast loader and builder gain config_subfolder/config_overrides for standalone encoder repos whose config sits at the root and whose pipeline needs forward flags (output_hidden_states/attentions). Verified on B200: bit-identity 291 tensors (225 fp8, 0 mismatches), hosted checkpoint engages through the real backend (marker + status fp8), load 24.3 s vs 48.0 s dense, LPIPS 0.133 mean over 3 same-seed pairs vs the dense-TE render (gate 0.25), non-black frames. * Correct the ltx-2 resident TE estimate to the bf16 cast size The memory plan's bf16_components_gb held 50.4 GB for the LTX text encoder, which is the fp32 hub store of Gemma3-12B (~49 GB download), not what sits on device: the pipeline loads it torch_dtype=bf16, ~24.4 GB resident. The 26 GB over-estimate pushed the auto plan toward offload on cards that fit the real footprint. Comments and the size-table test now pin the resident semantics. * Host pre-cast fp8 text encoders for four more families Round 2 of the hosted TE set, each bit-identical to dense-load-then-cast and gated through the real backend (marker + status fp8 + same-seed LPIPS vs dense TEs): - FLUX.1 T5-XXL (text_encoder_2): 9.52 -> 5.90 GB, one artifact for schnell/dev/Krea-dev (T5 shards byte-identical across all three, verified sha256). 220 tensors, 144 fp8, LPIPS 0.109. - Lumina Gemma2-2B: fp32 hub store 10.46 -> 3.20 GB (3.3x download cut). 288 tensors, 182 fp8, LPIPS 0.041. - Z-Image Qwen3-4B: 8.04 -> 4.41 GB. 399 tensors, 252 fp8, LPIPS 0.112. NOT shared with flux.2-klein-4B: klein retrained layer 35's MLP (verified tensor diff, maxdiff 0.86), so klein hosts no entry. - Krea-2 Qwen3-VL-4B: 8.88 -> 4.83 GB. 713 tensors, 460 fp8, LPIPS 0.082. The constructor-assembled krea pipeline takes the encoder directly (load_krea2_pipeline text_encoder kwarg); the loader remaps 5.x rope_parameters and re-ties weights after assign so the rebuilt encoder matches the builder's structure. HunyuanImage 2.1 reuses the Qwen-Image artifact outright: its Qwen2.5-VL text encoder is byte-identical (every shard sha256, 16,584,414,544 bytes), recorded in the new component-level base-equivalence table the checkpoint validator consults. The injection loop now covers text_encoder.._3 so a family can host several components. Live check: LPIPS 0.123 vs dense. * Report the fp8-cast compute dtype without swapping the encoder class The dtype override swapped encoder.__class__ to a dynamic subclass, which breaks transformers' kwargs-based output recording: a fp8-cast Qwen3VLModel stopped returning hidden_states and every krea-2 generation with text_encoder_quant=fp8 crashed at encode_prompt (regression from the HiDream TE4 change; caught by the krea hosted-TE live smoke). The override is now a property shadowed on the ORIGINAL class that prefers a per-instance compute-dtype attribute, so class identity is preserved and uncast instances keep the stock behaviour. The idempotency test now pins exact class identity and the uncast-sibling fallback. * Pass the calibrated distilled sigma curve to LTX-2.3 8-step runs The 22B distilled DiT was trained against ltx_core's fixed DISTILLED_SIGMA_VALUES, but the diffusers scheduler derives 8-step spacing from resolution-shifted flow matching and lands far off at every reachable mu (second sigma 0.945-0.981 vs 0.99375, tail 0.37-0.61 -> 0.1 vs 0.725 -> 0.42 -> 0). At the distilled default step count the backend now passes the list verbatim, neutralising the scheduler's dynamic shift and terminal stretch for the call (they distort even explicit sigmas) and restoring them afterwards. Other step counts and the dev/base DiT keep the scheduler's own spacing. Live-verified on B200 through the video branch backend: the scheduler holds the exact curve after an 8-step distilled GGUF generation, config restored, healthy clip. Also reword the transformer_quant resolved reason to the measured reality: quant halves resident weights and hosted checkpoints cut load time, while per-step speed is roughly bf16 parity. * Pin the fp8 weight-quantize kernel against silent MSLK switching torchao's Float8Tensor KernelPreference defaults to AUTO, which switches the weight-quantize kernel to MSLK whenever an mslk package is importable on sm90+. Measured on B200: that changes fp8 scale rounding bitwise (8/8 FLUX matrices differ, scales ~55 percent of bytes), so a box that merely gains mslk would break the hosted-prequant bit-identity invariant; the mslk path is also slower under torch.compile (opaque extern call blocks inductor's quantize fusion, FLUX.1 fp8 e2e 1.149 to 1.624 s). Pin KernelPreference.TORCH explicitly, matching current no-mslk behaviour bit for bit; signature-gated for older torchao. GPU-smoked (finite, rel err 0.037) and pinned by test. * Shift Qwen-Image training sigmas to the inference distribution Qwen-Image's scheduler skips its static shift under use_dynamic_shifting, so the DiT trainer was drawing UNSHIFTED uniform-schedule sigmas for it (mean sigma 0.50) while inference always runs the exponential mu = log 3 shift plus the shift_terminal 0.02 stretch. Add a flow_shift config lever: "auto" (the new qwen-image default) rebuilds the training sigma table through the scheduler's own time_shift and stretch_shift_to_terminal so the draw matches the inference distribution exactly (mean sigma 0.72); a numeric value applies the standard linear shift s*u/(1+(s-1)*u); 1.0 keeps the historical identity behavior and stays the default for FLUX, Z-Image and Krea 2. The model timestep conditioning follows the shifted sigma, gathered in fp32 so bf16 rounding never skews it. Also wire two opt-in levers with off defaults: cfg_dropout (per-sample empty-prompt conditioning dropout, encoded alongside the captions before the text encoders are freed) and weighting_scheme="bell" (bsmntw-style mid-schedule Gaussian loss weighting normalized to mean 1). Verified with two 80-step rank-8 bf16 LoRA runs on Qwen/Qwen-Image (identity vs auto, same seed): both converge with finite decreasing loss and produce coherent same-seed previews. Unit tests cover the exact transform, the shifted sampling distribution, per-family defaults and config plumbing. * Add LoRA EMA, a persistent conditioning cache, and aspect bucketing helpers diffusion_train_extras hosts the opt-in training extras: LoRAEMA shadows only the trainable adapter params (warmup-ramped decay, default 0.99, exported as a second adapter under output_dir/ema), PersistentConditioningCache stores latent posterior stats and caption embeddings as safetensors keyed by content hash + family + resolution, and the aspect-ratio bucketing helpers group mixed-aspect datasets into same-area divisor-snapped shapes. The DiT trainer wires the first two behind config flags that default to the current behavior: ema_decay (0 disables) and cond_cache_dir (None disables). A fully warm cache skips loading the VAE and text encoders entirely; a cache hit is bit-identical to a fresh encode, including the per-channel qwen latent normalization. Also fixes the stale _gather_sigmas call in the perf test that still passed the scheduler instead of the sigma table. * Tighten torchao configs and note the FSDP2 design for the DiT trainer nf4 loads now enable double quantization (~0.4 bits/param off the frozen base scales at no fidelity cost), fp8 training uses the rowwise recipe when the torchao build ships it (per-row scaling confines the DiT activation outliers that a tensor-wide scale collapses), and the inference quant filter gains a per-scheme GEMM-tiling divisibility floor (16 for scaled_mm, 32 for MX blocks) so one ragged Linear cannot crash the first denoise after a clean quantize pass. plans/fsdp2_diffusion_design.md records the multi-GPU design: bf16/fp8 over FSDP2 with per-block units, LoRA attached before sharding, int8 out of scope (DTensor over the quantized subclass is undefined), per-family notes. * Batch diffusion inference with per-image seeds, an inference conditioning cache, and GGUF loader fixes Batched generation: /images/generate takes a prompts list (one image per prompt, txt2img only) or a seeds list (one prompt, one image per seed); the legacy batch_size path derives per-image seeds base..base+n-1 like the native engine. Every image gets its own torch.Generator so any batch member replays alone from its gallery recipe; the whole list runs as one forward by default with OOM backoff that halves a failed chunk, and an explicit batch_size caps images per forward. Validated 10-22x over serial engines on 32-image suites with LPIPS deltas within 0.002. Conditioning cache on the inference path: UNSLOTH_DIFFUSION_COND_CACHE_DIR (the inference sibling of the trainers' cond_cache_dir, same persistent store) wraps encode_prompt so repeated prompts skip the text-encoder forward entirely; verified bit-identical outputs. Bypassed while LoRA adapters are attached; tensor-argument calls pass through uncached. Compile cache: GGUF loads fingerprint their own bundles (quant=gguf, a different compiled graph than the dense family) and batched calls register every distinct (w, h, batch) chunk shape they ran, so the heavy GGUF batched warmups (~159 s at batch 32 on 12B-class, ~655 s on 20B CFG-batched) are paid once ever. GGUF loader: strip the sd.cpp model.diffusion_model. container prefix in the single-file converter; diffusers' FLUX.2 converter KeyErrors on it and the Qwen-Image identity mapping strands the model on meta. * Correct batched seed-replay docs to match measured behavior Same-seed images at the same batch shape are bit-identical; a solo regeneration with the recorded seed matches its batched rendition up to batch-size-dependent kernel numerics (mean abs pixel delta about 2.5/255, LPIPS delta under 0.002), not bit-exactly. The previous wording overclaimed bit-identity across batch shapes. * Note that batched bit-identity assumes a settled compiled graph The first generation issued while the deferred compile is still in flight can deviate transiently (observed once on a cold fp8 build: mean abs pixel delta 0.063/255); once the graph is settled, same-seed same-batch-shape images are bit-identical across runs. * Studio sidebar: Image03/FlimSlate icons, More flyout, Train row, New pills - Images uses Image03Icon and Video uses FlimSlateIcon. - New "More" row (MoreHorizontalIcon) opens a right-side flyout on click or hover holding Video, Recipes and Export; the close is delayed 180ms so the pointer can cross the gap. Its SidebarMenuButton deliberately takes `title` rather than `tooltip`: with `tooltip` the button returns a Tooltip root and DropdownMenuTrigger asChild would hand its ref to a non-DOM node. - Dropped the "Train" section heading; Train is now a top-level row between Images and More. data-tour="navbar" moves to the surviving nav group so the product tour keeps its anchor. - "New" pill beside Images and (inside the flyout) Video, via NavBadge. * Studio sidebar: match flyout rows and New pills to the existing scales - More flyout rows dropped their sidebar-row typography and size-icon override, which fought DropdownMenuItem's own scale (text-sm, gap-2.5, px-3 py-2 and size-4 icons) and rendered oversized glyphs and text next to the nav. - New pill reuses the brand "beta" badge recipe (nav-badge font, --ui-font-scale sizing, nav token colours) rather than hardcoded 9px values. - The More row's native title tooltip (an OS box on hover) is replaced by the app's Tooltip, wrapped around DropdownMenuTrigger so both triggers compose onto the same button, and shown only on the collapsed rail like other nav rows. * Settings: pin and reorder the sidebar navigation Adds a "Sidebar navigation" section to Settings -> Appearance, above the existing profile-menu customizer, with the same drag-to-reorder + switch UI. - New sidebarNav preference: one { id, pinned } entry per navigable row (projects, hub, images, train, video, recipes, export), array order = render order. Defaults match the shipped layout, so an untouched install is unchanged. - Unpinning moves a row into the More flyout rather than hiding it, so no page becomes unreachable. New chat and Search stay fixed as actions. - app-sidebar now renders from one navRows descriptor map, so a pinned row and its flyout counterpart cannot drift; the More row appears only when something is unpinned and highlights off whatever it actually holds. - Mirrored in the backend PersonalizationCustomization: without it the model's extra="ignore" would drop the field, and because sync replaces local state with the server's copy once customization is saved, the user's pin order would reset on the next sync. The validator dedupes and back-fills like sidebarMenu but preserves the client's order, since here order is meaningful. Frontend typecheck, i18n parity and catalog checks pass; 32 personalization tests pass, including a round-trip asserting a reordered list survives a save. * Sidebar customizer: drop the Search row, skip More for a lone item - Search is reached from the top bar, so it is no longer previewed as a fixed sidebar nav row; New chat stays. - More now appears only when it would hold two or more rows. A single unpinned row renders inline in its saved order position instead: a flyout wrapping one item costs a click and earns nothing. The customizer's More preview follows the same threshold. * Sidebar settings: hide a lone unpinned tab, match New chat icon, rename Profile menu - With exactly one tab unpinned, both More and that tab are dropped, so nothing is drawn for it (previously it rendered inline). The page stays reachable by URL. - The customizer's New chat preview uses PencilEdit02Icon, the icon the real row renders; Edit03Icon was a different glyph. - "Sidebar menu" is now "Profile menu", described as the shortcuts behind your name at the bottom of the sidebar, so it no longer reads as a second name for the navigation section above it. * Tighten comments in the new sidebar and delete-guard code * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio sidebar: keep the More row highlighted while its panel is open Moving the pointer into the flyout left the row unhighlighted while the panel stayed open. The row now carries data-menu-open, added to the nav hover selector list. Not data-state: the tooltip and menu triggers both write that attribute, so whichever lands last wins. * Images: use the shared pill toggle for Create/Train and pad the panels - Create/Train was the only segmented control on its own Tabs styling. It now uses PillTabs, the same control as the model picker and Hub toggles, pinned to the header row's 34px. PillTabs takes an icon per tab, so the inline-span workaround for TabsTrigger goes away. - pt-3 on both the Create and Train panels, which sat flush against the model selector row. * Images: make the workflow picker a dropdown instead of a 7-up strip Seven workflows in a 340px rail left ~48px each, so the labels crowded and the hints were only reachable as title tooltips. The strip is now a dropdown: the trigger shows the current workflow and its hint, and each row carries its own description. A row the loaded model can't run is disabled and shows the reason in place of the hint, so the gating explains itself. Adding a workflow no longer shrinks the others. * Images: workflow icons, hint under the trigger, more top room, unclipped Train cards - Each workflow carries an icon, shown on the closed trigger and on every row. - The trigger is one line (icon plus name). The selected workflow's description moved below it, where it reads like the Field hints further down the rail. - pt-6 instead of pt-3 on both Create and Train, so the cards clear the model selector row. - The Train right column scrolls while its cards use ring-1, which draws outside the box and was clipped at the scroll edges. p-px gives the ring room. * Images: one-line workflow rows, stronger trigger fill, roomier mode tabs - Dropdown rows are icon plus name only. The selected row's description already shows under the trigger, and a disabled row keeps its reason as a title. - Trigger fill moves to the bg-foreground/[0.07] dark:bg-foreground/[0.12] pair the hub cards use, so it reads against the card in both modes. - Description under the trigger goes from text-ui-10 to text-ui-11p5. - More horizontal padding on the Create / Train tabs. * Images: drop card borders for the composer shadow, keep scrollbars inside, use app controls in Train - Cards lose ring-1 for .panel-soft-surface: the composer's shadow in light, flat in dark, matching .chat-composer-surface and the menus. - Both rails now clip (overflow-hidden) with the scroller inside, so the scrollbar can't ride over the rounded corner. Same shape video-page already uses. - Train's 9 native selects become the app Select, so they no longer open an OS-native menu, and the native file input is hidden behind a Choose images button that reports the count. - Image previews use explicit 8-10px radii: this theme sets --radius to 1.1rem, so rounded-md was 15.6px and the thumbnails read as circles. * Images: one card for controls and preview, chat sliders, wider softer shadow - Controls and preview were two floating cards; they now share one card split by a divider. The Advanced dock stays separate since it toggles. - SliderField wraps Chat's ParamSlider, so the sliders match Chat (label row with the value, full-width neutral track) instead of a green track with a spin box. All 14 call sites keep their props. - panel-soft-surface goes from 0 2px 8px -2px /0.16 to 0 4px 22px -6px /0.10: lighter, spread wider. * Images: flat Create and Train panes, hover-only scrollbars, tidier Train dataset step Both Images tabs now sit on the page background like the Hub: no card, no shadow, no bounding box. A single rule divides the controls rail from the preview canvas (Create) and from the run area (Train), and the settings and previous-runs sections read as panes rather than nested cards. Also: - Scrollbars in these panes use the existing hover-scrollbar recipe, so the thumb only shows while the pane is hovered. - Workflow rows explain themselves with a tooltip after a short hover, which also works on disabled rows, and the descriptions are much shorter. - Training images rows are name plus image count; the license stays on the example card. - The upload step loses its dashed box, the buttons match the sizes around them, and Upload only appears once files are picked. - The empty preview uses the same icon as the Images nav item. * Images: full-height panes, wider settings rail, Create/Train offset from the selector The rule between the panes now runs the whole page height (the row drops its bottom padding and each pane pads its own content), the settings rail is wider on both Create and Train, and the Create/Train switch sits further right of the model selector. * Images: put both tabs on the Hub's centered measure Top bar and content now share mx-auto max-w-1100 with px-5 / sm:px-8, so Create and Train sit at the same width and position as the Hub instead of running edge to edge. * Images: restore the top bar position, drop the panes lower under it * Images: center the mode switch, flip the arrow with the orientation, app tooltips everywhere The Create/Train switch is centered on the page instead of trailing the model selector, with wider buttons. The flip control's arrows now rotate with the orientation and its label says which way the flip goes. Every native title tooltip on the page is now the app's tooltip, so they all get the rounded surface instead of the OS box. * Images Train: plainer field text, no green buttons, columns that stop colliding - The dataset name, trigger prompt, adapter name and custom base fields now say what they are in plain words instead of leaning on example values. - Import, Upload, Back, Back to settings and Train another are outline buttons, not green ones. - Example thumbnails are landscape tiles, so photos are not cropped to chunky squares. - Settings cells get min-w-0 and the select value truncates, so a long option like the nf4 label no longer widens its column into the next one. - The number stepper sits a little further in from the field edge. - Create and Train are wider. * Images Train: roomier example cards with Import on the thumbnail row * Video: same treatment as the Images tabs - No cards: the rail and the canvas sit on the page background, divided by a rule that runs the full page height, on the Hub's centered measure. - Wider rail, chat's sliders, hover-only scrollbars. - Every native title tooltip is now the app's tooltip, including the clip cards. - Reapply and Cancel are outline buttons, the empty state uses the Video nav icon, and the clip tiles are less rounded. * Images and Video: narrower generation rail, matching Train headings Create and Video rails go from 392px to 368px. Train a LoRA and Training settings are now the same size and both in the heading font: the h2 already picks it up from the base rule, so the settings header opts in with font-heading and the weight that rule pins. * Images Train: shorter copy throughout Family notes, example descriptions, precision labels and every helper line are trimmed so they stop wrapping to three lines and colliding with the next column. The nf4 label now fits its select without truncating. * Images Train: a little more spacing between field groups * Images and Video: tighten code comments * Fix training start NameError, the load-order guard test and CPU-only diffusion tests - start_training forwards resume_source_run_id to _start_training_impl, which reads it. Without it every start raised NameError. - Restore main's anchor in the load-marker order test: the file now has an earlier `if config.is_gguf:`, so indexing the first one compared the wrong branch. - The two diffusion tests that reach diffusers now skip when it is absent, matching the CPU repo-test env. - The UI smoke finds nav rows that live in the sidebar's More flyout. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Treat a null metadata caption as no caption str(None) stored the literal "None" as the caption, so a null row counted as captioned and would have trained on that text. Also drops an unused import. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix invalid-UTF-8 500s, the flat Canny map and the dropped DiT knobs read_text raises UnicodeDecodeError, which is not an OSError, so one bad caption sidecar or video sidecar 500d the info, upload and gallery routes. A flat image now yields the all-black edge map instead of its own luminance, and the four DiT loss knobs the trainer implements are declared so model_dump keeps them. * Use the ui font-size tokens instead of raw px text utilities text-[11px] and friends ignore the UI font size preference, which the repo's font-scale contract test enforces. Same rendered size at the default scale. * Fix diffusion dataset 500s, the dropout-1.0 no-op run and the reset base pick Four correctness fixes on the training side: - The labeling grid read caption sidecars under except OSError, but a non-UTF-8 sidecar raises UnicodeDecodeError (a ValueError), so one bad file 500d /diffusion/dataset/{name}/images and the grid could not be opened to repair it. Read it as no caption, matching the info summary. - An image past Pillow's own hard limit raises DecompressionBombError, which derives straight from Exception and so escaped the upload guard's (OSError, UnidentifiedImageError, ValueError) and returned 500 instead of the intended 400. - lora_dropout accepted 1.0, which makes PEFT build nn.Dropout(p=1.0): lora_A and lora_B receive no gradient and the run saves an untrained adapter while reporting normal progress. Bound it below 1.0, matching the LLM request schema. - The train panel re-seeded the base repo on every dataset refresh because the family object identity changes on each info fetch, so an upload or caption save silently replaced the user's chosen base and the run started on a different model. Track the pick and only re-seed on a real family change. * Show the retained failure when a video page mounts after a failed job Mount-time recovery handled only phase=completed, so reloading the page after a multi-minute generation failed left an idle view with no diagnosis: the backend keeps the terminal failed record only until the next job, and nothing else survives the reload. Surface it the same way the poll does, filtering the cancelled sentinel. * Fix batched generation crashes, cache keying and unreplayable recipes Four bugs in the batched inference path, all found by review: - A mixed-prompt batch sent a scalar negative prompt against a prompt list. Z-Image asserts on the length, and Qwen-Image, Krea 2 and FLUX true-CFG encode a batch-1 negative against batch-N latents and fail in the transformer's text/image concat. Broadcast it to match the batch. - The FBCache step-cache reset sat above the chunk loop. diffusers only resets that state at the end of a successful call, so a forward that raised (the OOM the backoff is meant to recover) left its own residual behind and the halved retry died on a shape mismatch. Reset before every forward instead. - The conditioning cache keyed on the checkpoint alone, but a GGUF or single-file load takes its text encoders from the companion base, so the same checkpoint against a different base reused the previous base's embeddings. Key the base too. - Gallery records stored the base seed and the requested batch size even when a prompts/seeds list drove the run, so restoring the second image of seeds=[5, 99] replayed seed 5. List-driven outputs now record as single-image recipes on their own seed. Also bound strength above 0: every img2img pipeline derives its step count from it, so 0 leaves zero denoising steps and either raises or, on SDXL, crashes on empty latents. * Fix quantized-load LoRA bake, prequant family exclusions and outpaint canvas Six review findings across the Images page and model scanning: - The quantized (int8/fp8) load path can only attach LoRA adapters before quantization, but the frontend load request had no loras field, so every generation after such a load was rejected and each reload repeated it. Send the selection with the load. - build_prequant_checkpoint passed no family to the scheme exclusions while recording the family in metadata, so a Qwen int8 artifact baked the short-M text-stream linears and was then rejected wholesale by the loader's family-keyed check. - Registering a bare single-file checkpoint directory produced no On Device row even though the images loader can load it; only its parent worked. Admit that shape when nothing else matched. - Unload left the Reapply target set, so the repair path was skipped and Reapply reloaded the ejected model. Clear it, as the video page does. - Both FLUX.2 bases were trusted for training but not inference, so Deploy to Create rejected every FLUX.2 adapter. - Outpaint allocated the grown canvas before downscaling, exceeding the browser canvas area cap on a large photo; an over-cap canvas is unusable, so Extend silently posted a fully transparent image and mask. Scale the source first. * Send the picked GGUF filename with the quant so diffusion loads fire The variant expander emitted only the quant label, and nothing else in the frontend set ggufFilename, so the Images and Video pages could never take their GGUF branch: both gate it on meta.ggufVariant and meta.ggufFilename, then fall through to the single-file path, which returns because the id is a repo id and not a .gguf name. Every quant pick was a silent dead click, with no load request reaching the backend. The filename was already on the variant row (the picker keys its list on it, and the variant validator requires a non-empty string), so thread it through the click handler. The chat path is unaffected: it reads ggufVariant and never needed the filename. * Version the conditioning cache key and reject non-finite flow_shift Two correctness fixes: - The cache keyed the checkpoint and its companion base by name only, so a Hub repo advancing to a new commit, or a local directory updated in place, kept returning embeddings from the previous text encoder. Pair both with a revision marker: the locally resolved commit sha for a Hub repo, config plus text-encoder file stats for a directory. Neither loads the encoders, so a warm run still keeps them off the GPU. - flow_shift only checked positivity, but JSON accepts 1e309, which floats to inf, and inf <= 0 is False while NaN fails every comparison. The sigma table then evaluates s * u / (1 + (s - 1) * u) as NaN, which poisons every sampled sigma and saves a corrupted adapter while progress looks normal. Require a finite value. * Keep curated models listed, guard the video companion repo, pin diffusers Three review findings: - The picker filtered every catalog member out of Recommended and Hub search on the way to canonical group rows, but nothing renders those rows yet (catalogGroupFitsDevice and groupMatchesQuery are imported and unused). A task-scoped picker's models list is catalogToModelOptions(), i.e. group members exclusively, so both lists came back empty and no curated model could be discovered or downloaded. Keep the artifacts listed until the grouped UI exists. - The video delete guard compared only repo_id, so deleting the companion base of a loaded GGUF video model was allowed even though it supplies the VAE and text encoders. Compare base_repo too, matching what the images guard already does for its companions. - diffusers was declared unversioned while the diffusion stack requires 0.39 (Krea2Pipeline, the cache_context child registries, the Flux2 and Z-Image pipelines), so an upgrade could keep an older release and selecting an advertised model failed until the user upgraded by hand. * Namespace the trainer conditioning cache per checkpoint, bound the learning rate - The trainer keyed its persistent conditioning cache on family and resolution only, while the keys themselves carry just the caption or image content and crop variant. One cache directory reused for two checkpoints, or for the same repo at a new revision, let a warm run skip loading its encoders and train on the other model's embeddings and latent statistics. Namespace on the base checkpoint and its resolved revision as well. The revision helper now lives beside the cache in diffusion_train_extras and the inference wrapper delegates to it, so the two cannot disagree about what counts as the same source. - The diffusion learning rate only checked positivity, but 1e309 floats to inf and satisfies gt, so the route evicted the resident models and started AdamW with an infinite rate: the first step destroys the adapter while progress looks normal and the result is saved. Bound it below 1.0, matching the LLM schema, which rejects inf for the same reason. * Fix GGUF image model picks doing nothing, and pick the train base in the top bar The quant rows never forwarded the .gguf filename, so every hub GGUF pick on Images/Video fell through to a silent return. On Train the top bar now picks the training base instead of a generation model, which is GGUF-only and untrainable. * Pin diffusion and video loads to the live HF cache root Both read huggingface_hub's import-time HF_HUB_CACHE, which changing the cache folder does not update: progress counted the old root while the download wrote to the new one, and from_pretrained could split one model across both. * Add the diffusion download plan endpoint Reports the repos and exact files a pick needs so the download manager can stage them with the loader's own file scope. A plain snapshot would add the packaged root single, transformer shards and fp16 twins the loader never opens. * Add a file-scoped flavour to the Hub download job Lets a consumer that reads a deliberate subset of a repo stage it through the normal download manager. Keyed as "@scope" so it never collides with a quant or with the repo's full snapshot, and the file list rides the registry so an XET to HTTP retry respawns the same scoped job. * Stage image and video downloads through the Hub download manager They downloaded inline inside the load, so they had none of the manager's disk preflight, manifest verification, resume or panel progress. Picks now stage as scoped jobs carrying the loader's own file list, then load from a warm cache. * Fetch staged GGUF checkpoints as scoped jobs, and stop calling diffusion models unsupported A GGUF entry went out as a full snapshot, whose ignore list drops *.gguf: the job finished at once having fetched only docs, and the repo landed on device unloadable. Every entry is scoped now. The Hub also no longer tags image/video models as unsupported (they run on their own pages), and those pickers name what they select. * Apply the picker task filter to local model sections LM Studio, ./models and custom-folder rows ignored it, so the Images picker listed chat GGUFs that 400 on a diffusion load. The backend already tags every local model with a task for this purpose. * Route a chat pick of a diffusion model to the Images or Video page Chat cannot load one, so it was either hidden or failed on load. The unfiltered picker now lists on-device diffusion models and navigates to the page that runs them, passing the repo and quant so that page loads it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Route the real GGUF filename, keep non-GGUF curated models, key scoped downloads by file set Five review findings, four of them ways a click did nothing or fetched the wrong thing: - A chat pick of a diffusion model routed ggufVariant (a label like Q4_K_M) in the search param the target page uses verbatim as the GGUF filename, so the load asked for a file that does not exist. Route ggufFilename; no filename means a curated non-GGUF pick, loaded as a pipeline. - The task-scoped pickers kept only GGUF repos, so the catalog's bf16, bnb-4bit and single-file fp8 artifacts could not be discovered or downloaded on the Images and Video pages even though loadSpecFor knows how to load them. Keep curated artifacts whatever their format, in Recommended and in Hub search. - Both pages deduplicated routed selections on the model alone, and they now stay mounted, so picking the same repo again -- another quant, or the same one after chat evicted it -- returned early without loading or clearing the query string. Key on model and quant. - Every scoped image download shared one @diffusion job key regardless of the requested files, so switching quant mid-download adopted the running job: the UI waited on the first file set, then loaded a file that was never fetched. Include a digest of the file set in the key. - A scoped plan silently dropped requested files missing from Hub metadata, and snapshot_download succeeds when an allow pattern matches nothing, so the job reported completion and triggered a load with required files absent. Fail the job instead. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the scoped download key derivable, and stop the hidden page hijacking a route Four review findings, the first a regression from my own last commit: - Keying scoped download jobs by a digest of the file set broke the download manager: it builds that key client-side (it polls and cancels before any response tells it a key), so it watched and cancelled a key no worker owned and never fired its ready callback. Keep the derivable "@scope" key and refuse the second request instead when a live job on the slot is fetching a different file set -- decided inside the registry claim, under the lock, so a concurrent claim cannot slip past it. The manager records the file set on the job as well, so a sibling quant's transfer is not adopted locally either. - Both diffusion pages read the route query through a loose useSearch and both stay mounted once visited, so the hidden one consumed the other's ?model=: it navigated back to its own route and tried to load, say, an image checkpoint as a video model. Only the visible page consumes it. - The staged download plan was built without the configured HF token or the Advanced values the load itself sends. The token matters most: the backend's Hub metadata lookup is best-effort, so a gated base silently planned no companion entry and the load pulled those multi-GB files inline, outside the manager. The memory/quant controls decide whether the base transformer/ shards are needed at all, and the route dropped memory_mode, cpu_offload, the prequant path and the LoRA selection before asking for the plan. - The video preview kept playing after leaving the page: the keep-alive layout only hides it, and display:none does not pause a media element, so a clip the user unmuted kept its audio going over the next page. Pause on the active transition and do not auto-replay while hidden. Also completes the hand-built request bodies in the hub download tests: the scoped-files field this branch added to the route read as an AttributeError against them, failing five tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Serialize the GPU handoffs, gate DiT training on a GPU, and keep 3.9 installable Six review findings, three of them evict-then-fail orderings: - The chat load reclaimed the GPU without telling the arbiter it existed. A chat load holds no llama-server process until its GGUF has downloaded, which is minutes, so a competing Images/Video acquire in that window found nothing to cancel, took the GPU, and the chat load then spawned onto the same device. It now registers an in-flight marker through acquire_for's register hook (under the arbiter lock, as the image and video loads do), the evictor cancels a marked load, and the route undoes itself if ownership moved while it loaded. - The Hub-download conflict check ran after that handoff, so a GGUF the download manager already owns destroyed the resident Images/Video pipeline and then 409'd, having loaded nothing. It moves above the handoff, together with the marker it handshakes with. - The image load released the engine router's transition lock before registering the load, so a second load choosing the other engine could unload the still-idle engine this one captured; the load then landed on a deactivated engine, where generate, status, unload and the arbiter's evictor can no longer reach it. Registration now happens under that lock and refuses if the engine changed. - Training a DiT family on a host with no GPU was accepted: nf4 is not a CPU fallback, its 4-bit load goes through bitsandbytes, which requires CUDA, XPU or MPS. The start unloaded the working Images pipeline, pulled the text encoders, and only then died in the child. Rejected before the teardown now, and /info stops advertising a precision that always 400s. SDXL keeps its documented fp32-on-CPU path. - Both diffusion pages kept the routed-pick marker forever, so re-picking the same checkpoint (after chat evicted it) neither loaded nor cleared the query string. The marker is released once the query is gone. The Images key also carried a stray NUL byte, which made the file read as binary to grep and other tooling. - diffusers dropped Python 3.9 in 0.38, so the unconditional >=0.39.0 pin left pip no candidate at all on 3.9 and made every install that composes the huggingface extras unresolvable there. The floor is conditional now. Also fixes tests that were already red on the branch: two hand-built request fakes had gone stale against fields this branch added, and the handoff-ordering test only failed on a host with fewer than two GPUs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the GGUF variant contract test against the merged handler signature The assertion pinned the exact single-line call handleVariantClick(v.quant, v.downloaded, expectedBytes, v.filename), but the handler takes (quant, filename, downloaded, sizeBytes) and prettier wraps the call across lines, so the mandatory repository test job failed on every push. Match the call structurally and assert the filename really is forwarded in the handler's argument order. * Stop a background page and a stale record taking the GPU or a download with them Five fixes from a review pass over the diffusion work. delete-finetuned rmtree'd a model the Images or Video engine was holding: every guard on that route is chat-only, and Images loads any local path, so deleting a local diffusion model under the storage root pulled the weights (and the companion VAE / text encoders sd.cpp re-reads each generation) out from under a live pipeline. The cached-model route already refuses this; the trained/exported one now does too, matching by path rather than repo id, and failing open on a chat-only install so it cannot block ordinary deletes. A staged download finishing while its page was hidden loaded the model and evicted whatever the user was actually using: both diffusion pages stay mounted behind the router and a load takes the GPU unconditionally. The pick is now held until its page is on screen again, which is also what chat does. A scoped download could report success having fetched nothing. With Hugging Face metadata unavailable no manifest is written, so verification is a no-op, and snapshot_download returns an existing snapshot folder without downloading when its own repo_info call fails. A repo already on disk from a full snapshot job (which ignores *.gguf) therefore completed with no weights and auto-loaded against them. The requested file list needs no network, so it is checked against the disk directly. The XET to HTTP retry reclaimed the job slot without the scoped file list, and that claim overwrites the stored record, so a later identical scoped start compared an empty list against the real one and 409'd instead of adopting the running download. The DiT accelerator gate probed torch.mps.is_available(), which only exists from torch 2.5 while the supported floor is 2.4. All three probes shared one try/except, so on torch 2.4 the AttributeError read as 'no block' and a CPU-only host still evicted the resident pipeline, downloaded the encoders and died in the child. Each accelerator is probed on its own now, through torch.backends.mps. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stage what an LTX-2.3 load reads, keep a routed file's load kind, drop an unbakeable LoRA Three from the latest review. The video download plan always asked for the wide base file list, so an LTX-2.3 pick staged the 2.0 base's VAEs, vocoder and connectors that the checkpoint supplies itself, while the companion files the 2.3 assembly does read were left out of the plan and pulled inline at load, outside the panel's progress, cancel and disk preflight. The plan now recognises a 2.3 pick by name (the load keeps the authoritative header probe, and under-guessing only falls back to the load-time pull), narrows the base list, and stages the extras in the same entry as the checkpoint so one repo stays one scoped job. A pick routed from the chat picker arrives as ?model= and ?quant= with no picker metadata, so a bare local .gguf or .safetensors was loaded as a pipeline: an explicit model_kind wins over the backend's filename sniffing, so it evicted the resident model and then failed on the missing model_index.json. Both pages now derive the load kind from the path, the same way their own picker handlers do. A torchao int8/fp8 build takes adapters only at load time. Switching artifact inside one family keeps the LoRA selection, since the family did not change, but the load did not bake it, so the next generation was rejected with 'reload the model with the adapter selection' while the picker still showed the adapter as active. The selection is now dropped once per resident build, with a message saying to pick and load again. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cancel an evicted safetensors load, spare the arbiter for CPU-only chat, fetch clips lazily Four fixes from the latest review round: - The GPU arbiter's chat evictor only cancelled the llama.cpp side. The orchestrator publishes active_model_name once its worker reports success, so an in-flight safetensors load was visible only as an entry in loading_models and finished onto the GPU after ownership had transferred. Cancel every pending load, and give the safetensors branch the post-load ownership recheck the GGUF branch already had. - A manual gpu_layers=0 GGUF load runs on the CPU with the GPUs hidden from the child, yet it took the arbiter unconditionally: it cancelled a running image or video generation for a model needing no VRAM, then held CHAT ownership so the next GPU workload unloaded it for nothing. Gate the acquire on the same predicate the launch-time CPU-only mask uses, as the image and video loaders gate on their resolved device. - The staged-download hook subscribes per repo, not per job, so another job on the same repo advanced the staged queue (starting a load whose scoped files were still downloading) or wiped a queue that was still running. Compare the variant each callback carries, like the chat page's auto-load does. - The video gallery fetched every record of a page into an object URL that lives until the page closes: 50 clips at tens to hundreds of MB each, for cards the user may never scroll to. Fetch a clip as its card nears the strip's edge, plus the selected one the player needs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim the comments across the diffusion backend Comment-only pass over the Python this PR touches: drop what the code already says, collapse multi-line explanations that still read on one line, and keep the reasoning that is not recoverable from the code. No code, docstring semantics or behaviour changes; verified with an AST comparison against the previous revision, and the backend suite is unchanged (same 37 environment failures as before: the API integration tests that need a live keyed server, the flash-attn install hooks, and the GPU memory fields). * Invalidate latents on a VAE swap, keep a cut-off generation, surface the EMA adapter - source_revision() scanned the checkpoint root plus text_encoder/tokenizer but not vae, so swapping or fine-tuning the VAE in place left the conditioning cache namespace unchanged and a warm run trained against latents from the old checkpoint. Include the vae directory, like any other component the cached tensors come from. - /images/generate answers only when the images are saved, and secure mode's tunnel caps an origin response near 100 seconds, which a native CPU or a high-step run passes routinely. The page reported failure while the work kept running, and a retry would duplicate it. A lost response (fetch rejection or a gateway status the origin never answered) is now told apart from a refusal: the page waits out generate-progress and reloads the gallery, so the run it started still lands. - The trainer emits the EMA adapter's path with the terminal event, but the state update dropped it, so neither the run history nor either response schema carried it and an enabled EMA left nothing discoverable. Keep it, and show it next to the primary adapter. - weighting_scheme advertised a choice of timestep sampling; sampling is always logit-normal and the flag only selects the bell loss weights. Describe what it does. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Hide unloadable cached rows, hold the dataset interlock, bound a GIF export - The cached-model listing tagged any repo with a model_index.json as text-to-image, so a community pipeline the image loader's trust rule refuses still got a row in the Images picker, and a detected-but-untrusted video repo fell through to that same tag. Gate the image tag on the load path's rule and hide an untrusted video repo outright. - A routed diffusion pick only carries a GGUF filename, which is all the chat picker has, so a curated single-file artifact arrived with no quant and was loaded as a pipeline: from_pretrained on a repo with no model_index.json. Pass the page's own catalog spec into the route pick, so a routed pick resolves to exactly what a direct pick on that page resolves to. - The dataset mutation endpoints checked is_active() and only then handed their filesystem work to a thread, so a start reserving in that gap changed captions or removed images underneath the preflight or the running trainer. The interlock is now registered for the whole request under the lock reserve() uses, and a start refuses while a mutation is open rather than waiting on it. - GIF export held every kept frame as a paletted image before encoding; a clip may be 2048x2048 for 1024 frames, and at the 12 fps target the step is 1, so one export click could allocate over 4 GB and take the backend down. Downscale past 720 px and widen the step to keep at most 300 frames. - seed accepted any Python int, so an out-of-range one passed every preflight, evicted the resident models, spawned the trainer and only then died in torch.manual_seed. Bound it to torch's 64-bit range in the request and config. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the accelerator probes in the DiT family-metadata tests Six tests read family_train_infos() (or a start preflight) without pinning the host probes, so they only held on a machine with a bf16 accelerator: on a GPU-less runner the DiT gate empties precision_modes, turns supports_compile off, and replaces any other preflight message with the no-accelerator note, and all six failed there. A conftest fixture pins both probes for exactly those tests, so they assert the family metadata they are about on every host. The gate's own CPU-only behaviour keeps its dedicated tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Carry the pipeline task into the hub inventory the pickers read - The task-scoped pickers filter On Device rows on a task, and the chat picker routes a diffusion pick by the same field, but those rows come from the /api/hub inventory, which never carried one: the Images and Video pickers listed nothing on device and the routing never fired. Both cached scans and the local listing now tag rows with the classifiers the models API already uses, the schemas and the frontend adapter carry it through, and a row the backend classified as a generation task is exempt from the chat-only guard that was also dropping it. - The local routing map was keyed by model_id while the row click passes id (a filesystem load id for a models_dir or LM Studio entry), so the lookup missed and the pick fell through to the chat loader. Key both. - A staged download whose start answered "error" left its head in place, where the effect never re-runs and onReady never fires, so the pick was stranded until the user reselected. Clear the queue and say so. - Every scoped pick in a repo shares the @diffusion variant, so the variant alone cannot tell two file sets apart: restaging while the first job finished let its completion pass for the new pick and load a checkpoint that had not downloaded. Bind the callbacks to the repo + file set they started, and to the staging generation. - A rejected generate POST does not say whether it reached the backend, so an immediately idle progress read was ambiguous and a submission that never landed looked like a finished image. Require evidence: progress seen active, or a gallery record that was not there before the POST. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the OpenAI image URL fetchable, keep WebM audio, stream example imports Four review items on the diffusion Studio work: - response_format=url returned the bearer-gated gallery route, which a standard image client downloads with no Authorization header, so the default response format was unusable. Mint a short-lived HMAC link instead (the shape RAG already uses for pdf.js) served by a signed route, and leave the gallery route itself bearer-only. - A manual gpu_layers=0 load carrying speculative_type="off" -- a value the UI persists and sends -- read as GPU-bearing, so it took the GPU arbiter and evicted a resident image/video pipeline even though the launcher hides the GPUs for it. Canonicalize the mode and exempt "off". - The curated example import prepared the whole split before the loop stopped at the 10-100 image cap; m1guelpf/nouns is 49,859 rows / 328 MB. Stream instead, with the prepared load kept as a fallback for a repo that cannot stream. - WebM export dropped the audio track an LTX-2 clip carries, silently, on the format offered for web embeds. Mux it as Opus through a resampler + FIFO, and keep exporting the video alone on a build without libopus. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not stub out triton on a GPU host when the Xet backend fails to import The lazy loader retries `import unsloth_zoo.hf_xet_fallback` under UNSLOTH_ZOO_DISABLE_GPU_INIT=1 whenever the first attempt raises. That flag makes unsloth_zoo take its MLX/CPU path, which injects triton and bitsandbytes STUBS into sys.modules for the rest of the process. On a working GPU box whose first import failed for an unrelated reason (a bitsandbytes/CUDA mismatch, say) the retry succeeds, so Studio boots looking healthy and then dies at the first CUDA-only kernel: a GGUF or compiled diffusion generation hits the stub and returns NotImplementedError: Unsloth: 'triton.tools.experimental_descriptor.enable_in_pytorch' was called on Apple Silicon / MLX, where triton is stubbed out. so every image generation 500s with an Apple-Silicon message on a Linux CUDA host, while the load reports success. Found by loading Z-Image-Turbo GGUF through the API on a box where bitsandbytes could not initialise. Gate the retry on the host genuinely having no accelerator. The Xet stall watchdog is optional and already degrades with a warning; a process whose triton is stubbed out is not recoverable. The warning now says why it did not retry. * Fix the lost-generation proof set, the settle timeout and the hub inventory's diffusion gates Seven fixes from the latest review round on the Images page and the hub cache inventory. Images page: - The lost-POST settle path built its "already seen" gallery id set inside the catch, after the request failed. By then the earlier runs of the same batch had already prepended their records, so run 2 could accept run 1's image as proof that its own request reached the backend. The set is now captured once before the first POST and grows with every record the batch produces. - settleLostGeneration fell out of its SETTLE_MAX_MS loop and returned normally, so a wedged generation was counted as done and the next run started against a busy backend. It now throws on timeout. - Restoring a recipe cleared the ControlNet selection but left the workflow tab and the init / mask / reference images pointing at whatever was loaded, so the next Generate conditioned on an unrelated image. It now clears all of them and returns to Create. - The download plan omitted the adapter selection the load itself bakes in. A baked LoRA forces the dense build path, so the plan described a different file set than the load that followed and the rest was pulled inline, outside the download manager. Both now derive the list from one helper. Hub cache inventory: - A download for a repo an Images or Video load is staging was allowed to start: only the llama.cpp loader was consulted. Both diffusion backends already expose loading_repo_ids for the delete guard, and the download guard now reads them too. - A companion-only prefetch (pipeline manifest plus VAE and text encoder, no transformer) passed the snapshot-partial check, since every file its manifest expected did arrive, and was advertised as on-device although from_pretrained cannot load it. - The single-file flag never reached the picker through the hub inventory path, so a checkpoint-only diffusion repo read as a full pipeline and failed after the handoff. The two pipeline-shape helpers now live in hub/utils/inventory_scan.py so /api/models/cached and the hub inventory classify the same repos the same way. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop adopting an unknown scoped download, leaking raced blobs and resurrecting deleted clips Three items from the latest review round. A scoped download job carries a deliberate file subset, and every file set of one repo rides the same "@scope" slot. A client that adopts a live job from the backend had no file list to compare against: the active-downloads response never carried one, so an adopted job's set was unknown and any later scoped request for the same repo read as "already started". Selecting a different checkpoint then waited on the wrong transfer and tried to load a file nobody fetched. The response now publishes the scoped file list, adoption records it, and an unknown set no longer satisfies a scoped request. A gallery record can be deleted while its blob is still downloading. The delete revokes the URL present at that moment, so the fetch that lands afterwards inserted a fresh object URL for a record no card renders and nothing can revoke: a full MP4, tens to hundreds of MB, pinned for the rest of the session, and once per raced fetch. Both galleries now discard a blob whose record went away, with an epoch covering the video page's Clear all. The video backend keeps the last completed job until the next one starts, and the Video page merges that record on mount to cover a job that finished after the gallery fetch. Deleting the clip left the record in place, so every reload prepended a ghost card whose file request 404s until another generation replaced it. Deleting the clip, or clearing the gallery, now clears the matching terminal record, and the page skips a record it deleted itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Resolve revisions from the live cache, serialize dataset imports, and stop pinning every gallery blob Five more items from the review round. The conditioning-cache revision marker read huggingface_hub's import-time HF_HUB_CACHE constant. Studio can move its cache during a session and loading follows the live setting, so after a move the marker went unresolved (or pointed into the previous root) and pulling a new revision of the same checkpoint no longer invalidated the cache: a warm run could reuse the old encoder's embeddings and the old VAE's latents. It now looks in the active Studio cache first and keeps the environment and the library constant as fallbacks, which the trainer subprocess still needs. The dataset interlock counts mutations rather than excluding them, so two imports of different examples into the same empty name both got past the emptiness check. The winner promoted its staging directory atomically; the loser found the folder non-empty, fell back to a per-file move, and merged its images and captions into the winner's dataset. Imports now take a per-folder lock, a second one is refused with 409, and the emptiness check is repeated under the lock. On Windows the sd.cpp asset resolver filtered only by accelerator token, so a Windows arm64 host matched an x64 zip, downloaded and installed it, and failed later when the binary would not run. It now filters by architecture the way the Darwin and Linux branches do. Every gallery page fetched every PNG up front and kept the object URL for the session, so scrolling a large gallery grew memory without bound for tiles the user may never look at. The Images strip now fetches a tile as it nears view, like the Video strip, and keeps the eager path only where IntersectionObserver is unavailable. A 503 carrying a JSON body comes from the application, not a proxy, so it is surfaced as the error it is instead of entering lost-response settlement and being reported as a request that never reached the server. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the class of a failed generation instead of a bare "Image generation failed." Found on a macOS runner: the native renderer aborts inside its own text encoder there, and the page showed only "Image generation failed." with the sd-server backtrace left in the server log, so nothing about the failure reached the user. The failure is now classified into fixed text, out of memory and native-process death, so the message says what happened and what to try. None of the engine's own output is echoed, since a native tail carries local paths and argv; that stays in the log, and an unrecognised failure keeps the original literal. * Treat an undecodable caption sidecar as the tombstone the trainer sees Uploads store .txt and .caption sidecars as raw bytes, so one can hold invalid UTF-8. The trainer treats any existing sidecar, decodable or not, as an empty tombstone and never falls back to the metadata row for that image. The labeling grid and the dataset summary read an undecodable sidecar as absent instead, so both showed a metadata caption that the run would silently replace with the instance prompt, and counted the image as captioned. Both now track sidecar presence separately, so what the user reviews is what the run trains on. * Keep the reason a native server died, not just its backtrace A ggml abort prints its cause first and then a stack trace, so reporting the last twenty captured lines gave twenty addresses and nothing about the failure: on the macOS runner the native server died on an unimplemented Metal op and the message carried only frame pointers. The captured tail now leads with the lines that name a cause and keeps recent context after them, for both the startup failure and the mid-request death. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop staging the dense text encoder for an fp8 video load Two halves of the same gap, found while measuring the LTX-2.3 download plan: - The video download plan and the scoped pre-download never saw text_encoder_quant. An fp8 request loads a hosted pre-cast encoder, so asking for one still staged and downloaded the base repo's dense Gemma3 (48.79 GB of Lightricks/LTX-2 on the 2.3 distilled pick) that the pipeline then never opened. The plan now drops those shards and stages the pre-cast checkpoint instead; their configs stay, since the pre-cast loader still meta-inits the encoder from the base repo's component config. - The LTX-2.3 assembly builds every component itself, so pipe_kwargs (which carries the pre-cast encoder for from_pretrained) never reached it and an fp8 request silently loaded the dense encoder anyway. It is passed across explicitly now. The dense skip is earned, not assumed: only a pre-cast checkpoint that resolves on the Hub lets the plan drop the dense shards, and only one already fetched to disk lets the pull drop them, so an unpublished or gated artifact leaves both exactly as they were. If injection still fails after that, the load tops the dense weights back up rather than handing from_pretrained a snapshot with no encoder in it. Measured against the real Hub on the 2.3 distilled Q4_K_M pick: 67.24 GB before, 18.92 GB with a 0.43 GB stand-in for the pre-cast artifact (the base entry drops from 24 files / 48.79 GB to 13 files / 0.04 GB). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Match the file's typing idiom for flow_shift models/training.py annotates with typing constructs throughout (105 Optional[...], no Union), and flow_shift was the one place using PEP 604. Union[] reads like the rest of the file, and it also drops the runtime evaluation that would raise on Python 3.9. * Bound the gallery blob cache, and three interlock fixes Four review findings, all reproduced first: - The gallery object-URL caches were unbounded. A clip runs from a few MB to a few hundred, both pages stay mounted after their first visit, and entries were only dropped on delete, so scrolling pinned everything for the session. Both pages now share a byte-budgeted LRU (512 MB video / 192 MB images) keyed off the visibility signal the near-viewport fetching already provides. On-screen media, the selected clip or image, and the item just fetched are never evicted, so eviction is invisible and a single item larger than the whole budget cannot evict itself into a refetch loop. - The image, video and chat load guards ran two independent training probes but returned early when the FIRST one raised, so an unreadable LLM backend disabled the diffusion interlock and a load could proceed straight into an active diffusion trainer on the same GPU. The probes are independent now. - An engine switch swallowed a failed teardown and published the new engine anyway, which is exactly the leak the unload exists to prevent: the arbiter's evictor, /images/unload and the next load all resolve through get_active_diffusion_engine(), so the still-resident pipeline (or a live sd-server) became unreachable and the next load allocated on top of it. The switch now fails and leaves the old engine published, so it stays reclaimable. - The native generation timeout was 30 minutes while the Images page waits up to 6 hours (SETTLE_MAX_MS), so slow-but-progressing CPU jobs died deterministically at the deadline. Measured on GPU-less runners, a 512x512 4-step Q2_K generation took 900 s on Linux and 1465 s on Windows, so larger images or step counts clear half an hour easily. The ceiling now matches the page's window and applies to the whole request: chunks of a split batch share one deadline instead of each getting a full budget. Cancellation is unchanged. Declined: gating the huggingfacenotorch extra off Python 3.9 over the conditional diffusers marker. The marker is deliberate and its comment says why: diffusers dropped 3.9 in 0.38, so pinning >=0.39 outright leaves pip no candidate and the whole extra unresolvable there. The pipelines it names live in studio/backend, which cannot install on 3.9 anyway (studio.txt pins matplotlib==3.10.9 and fastmcp>=3.0.2, both requires_python >=3.10), and the extra is the general core one, so the alternative drops 3.9 for library users who never touch Studio. * Close the load-versus-training-start race, and two picker fixes - The image and video load guards read is_active() and only then selected an engine, acquired the arbiter and registered the load. A /train/diffusion/start reserving inside that window freed residents the load had not registered yet, so the trainer came up beside a brand-new pipeline. The service already had exactly the right pattern for this in dataset_mutation, so gpu_load_admission mirrors it: reserve() refuses while an admission is open, an admission refuses once a start is reserved, both decided under the one lock. The span is only the registration, since begin_load returns as soon as the load is registered and _free_gpu_for_diffusion_training preempts an in-flight load from that point. Chat is deliberately not covered: its load spans an eviction plus a multi-minute GGUF load, and it admits models that fit beside training by design, which is a different contract from the diffusion pipeline's all-or-nothing one. - Hugging Face gives the LTX-2 family the image-to-video pipeline_tag (both Lightricks/LTX-2 and unsloth/LTX-2.3-GGUF report it), so a text-to-video-only filter dropped the flagship audio family out of Video Hub search while the rest of the app routed it to Video. - Task-scoped quant fit sized picks against the LARGEST visible device while resolve_diffusion_device_target returns a bare "cuda" and torch places on the current one. On a heterogeneous host that recommended a checkpoint sized for the bigger card and then loaded it onto the smaller one. Fit now uses the device the load actually lands on; identical on a homogeneous host. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Expose the persistent conditioning cache in the start schema DiffusionLoraConfig has carried cond_cache_dir for a while and the DiT trainer acts on it, but DiffusionTrainingStartRequest omitted the field, so Pydantic dropped it silently and every API-driven run fell back to the in-memory cache that is rebuilt from scratch each time. The warm path skips loading the VAE and the multi-GB text encoders on a rerun whose images, captions and resolution are unchanged, so this was a real capability that could not be reached. Contained like output_dir rather than left to the trainer subprocess's cwd, since it is another directory the trainer writes to. Blank or omitted still means the in-memory cache, so it must not resolve to the outputs root. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix diffusion policy and classification issues from review fp8 auto precision defaulted to precise accumulate on any non-consumer GPU, which made fp8 2.05x slower than int8 on RTX 6000 Ada and slower than not quantising at all. NVIDIA's professional whitepapers do publish equal FP8 rates for both accumulate modes there, so the hardware premise held, but the cost is in the cuBLAS path rather than the published rate. Default to fast accumulate: measured on B200 the flag is a no-op (4096^3 _scaled_mm at 3023.8 vs 3041.8 TFLOP/s, bitwise-identical output, 1.213 s vs 1.230 s end to end), so it is a large win where it bites and free where it does not. Precise accumulate stays available via transformer_quant_fast_accum. Z-Image's DiT is a Lumina2 derivative, so unsloth/Z-Image-GGUF and unsloth/Z-Image-Turbo-GGUF both declare general.architecture = "lumina2" and the whole line was tagged image-diffusion-unsupported and hidden from the Images "On Device" list, though validate_load_request loads them. Resolve shared archs from the repo/file name like bare "wan" already does, with a test asserting the picker and the loader agree for every family. The sage attention on-demand install ran an unpinned `pip install sageattention`, but PyPI's newest wheel is 1.0.6 and diffusers refuses anything below 2.1.1: the install always "succeeded", wrote an unusable version into the running venv, and was rejected on the next line. Carry the dispatcher's floor so pip resolves nothing instead. The dense-quant disk gate sized the download from the bf16-RESIDENT table. The fp32 families download twice that (Z-Image: 23,479 MiB against a 21,970 MiB gate), leaving a window where the check passed and the download filled the disk; Ideogram 4 ships fp8 and was overcharged the other way. Size the gate by published bytes, verified against HF sibling metadata for all 12 families. Patch installs went through unsloth_zoo, which refuses to import unless UNSLOTH_IS_PRESENT is set, and that is set by unsloth itself. The server imports unsloth at boot so it never showed there, but any other process ran silently unpatched with every install returning False, which is 13 test failures on a clean environment. Import unsloth and retry once, memoised per process. Also: the GGUF+LoRA refusal pointed at the native engine without saying a GPU host only selects it under UNSLOTH_DIFFUSION_ENGINE=sd_cpp, so the suggestion was unreachable; the gallery recipe recorded loras from the generate request alone, losing a load-time bake; load-progress claimed "40.07 GB downloaded" for a fully cached load; and pickers.tsx imported three catalog-group helpers it never used. Reported by oobabooga. * Keep the sd.cpp text encoder on CPU under Metal macos-14 loads FLUX.2-klein-4B Q2_K natively on mps and then dies on the first generation with exit code -6: ggml_metal_op_encode_impl: error: unsupported op 'RMS_NORM' -> ggml_abort LLMEmbedder::encode_prompt -> LLMRunner::compute -> GGMLRunner::compute ggml's Metal backend gates RMS_NORM on contiguous rows and aborts the process when that does not hold, with no per-op CPU fallback, so any LLM text encoder (Qwen3 for FLUX.2 and Z-Image, T5 for FLUX.1) takes sd-server down. The encoder runs once per prompt while the DiT runs every step, so pinning only the encoder keeps Metal for the part that matters. UNSLOTH_DIFFUSION_SD_CPP_METAL_TE_GPU=1 opts back in once ggml grows the kernel. * Gate the unsloth retry in the diffusion patch backend The retry added for the clean-environment patch failures is not free: importing unsloth pulls torch in behind it, which costs ~940 MB of RSS measured in a process that had neither, and on a host with no accelerator it fails anyway. A cross-platform CI job that had generated fine at ~900 s later died 19 s in with SIGTERM and every 'if: always()' step skipped, which is the runner being torn down rather than a step failing. Retry only when torch is already imported (true of the server and of anything patching a real module, and the condition that stops the retry from being what loads torch), unsloth is installed but not yet imported, and the first failure was the ImportError the sentinel guard raises. The clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only retry the unsloth import where it can succeed The gate still let the retry run on hosts unsloth does not support, which is where it is most harmful: a 7 GB macOS runner lost the Studio server 26 s into a load, and the Linux runner was torn down mid-generation. Neither MPS nor plain CPU can complete the import, so the retry there pays the cost and fails anyway. Require an accelerator unsloth actually supports (CUDA/ROCm via torch.cuda, or XPU), with UNSLOTH_ALLOW_CPU as the documented override, and hoist the predicate to module level so it is tested directly rather than through the import system. On a CPU-only host the retry no longer fires at all; on CUDA the clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Guard old diffusers, stream video exports, record conditioned recipes Three fixes from review. The 0.39-only pipeline classes (Flux2Klein, Z-Image, Krea 2, LTX-2, HunyuanImage) were resolved by getattr deep in the load, so on the older diffusers that packaging still allows on Python 3.9 -- diffusers dropped 3.9 in 0.38 and this project still supports it, so the 0.39 floor has to be conditional or the extra becomes unresolvable -- an advertised model failed with a bare AttributeError after its checkpoint had already been downloaded. Krea 2 already guarded itself this way; assert_pipeline_class_available now runs the same check for every image and video family from validation, before any fetch, and names the version and the fix. WebM export accumulated the whole VP9 output in a BytesIO and returned it as one bytes object that the response held again. The request caps allow 2048x2048 for 1024 frames, so an export runs to hundreds of MB and concurrent clicks could exhaust the process, while the MP4 route beside it already streamed from disk. transcode_to_file encodes to a temp file and the route returns a FileResponse with a background unlink, so nothing large is resident. A conditioned generation's recipe carried only the txt2img fields, so the gallery presented an inpaint or upscale result as a complete Create recipe and restoring it replayed an unrelated text-to-image request. The images themselves are still not persisted (user uploads with their own lifetime), but the workflow and its scalars are, restore reapplies them, and the toast now names the inputs that have to be supplied again instead of silently landing on Create. Reported by Codex. * Per-load video cancel event, family-gated image picker, cond cache refusal A cancelled video load could resume: begin_load cleared the shared cancel event, and unload() drops _loading without waiting for the worker, so the next load cleared the very object the cancelled worker was watching and its multi-gigabyte pull ran on alongside the replacement until the token check at the end. Each load now gets its own threading.Event, passed down through _fetch_te_prequant and _predownload_base, so a cancelled worker stays cancelled. A cached repo with a model_index.json was advertised as text-to-image on the trust rule alone, but validate_load_request also requires a detected image family, so a trusted pipeline of an unsupported class produced a picker row that deterministically 400s. The picker now applies both gates, mirroring the video branch. cond_cache_dir was accepted for sdxl and then ignored: only the DiT trainer reads it, while the SDXL trainer builds a per-run in-memory latent cache, so the promised cross-run reuse never happened. The route now refuses it with a 400 that names the families which do support it, checked against the resolved family so an omitted model_family with an SDXL base is caught too. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the frontend build broken by the gallery blob cache tsc -b failed on the branch head, so npm run build produced no dist and every platform job fell back to --api-only: blob-url-cache.ts(29,15): TS1294: This syntax is not allowed when 'erasableSyntaxOnly' is enabled dataset-labeling-grid.tsx / dataset-showcase.tsx: Argument of type '{ url: string; bytes: number; }' is not assignable to parameter of type 'string' The cache took its budget as a constructor parameter property, which the project's tsconfig forbids, and fetchGalleryObjectUrl now returns the blob size alongside the URL for that budget, which the two dataset thumbnail components still consumed as a bare string. Declare the field explicitly and destructure the URL at both call sites. tsc -b is clean and vite build emits dist again. * Recover from a ggml unsupported-op abort by restarting on the CPU backend ggml checks every node against the device's supports_op and calls GGML_ABORT when one is not implemented, because a single-backend graph has nowhere else to put it: there is no per-op CPU fallback. The whole sd-server dies with SIGABRT mid-generation and the user gets "the native image renderer stopped unexpectedly" with no way forward. Seen on macos-14 arm64 with FLUX.2-klein-4B Q2_K through the cross-platform CI: the text encoder is already pinned to CPU, and the abort moved into the denoise loop instead. ggml_metal_op_encode_impl: error: unsupported op 'MUL_MAT' -> ggml_abort StableDiffusionGGML::sample -> sample_k_diffusion A retry on the same backend would abort identically, so the load is restarted once with --backend cpu (the only flag that changes which backend executes the graph; --offload-to-cpu moves parameters, not compute) and the generation is re-submitted. The same checkpoint then renders slower rather than not at all. Strictly bounded: the signature must carry both the unsupported-op line and ggml_abort, the device must not already be CPU, and it happens once per load, so an OOM kill or a genuine crash still surfaces as itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix two tests that only fail in a full-suite run The 3.10 CI leg resolves PyAV 17, where av.container.OutputContainer is an immutable C type, so the no-libopus export test died on "cannot set 'add_stream' attribute of immutable type" before it asserted anything. Inject the refusal by wrapping the container av.open() returns instead; modules stay patchable on every build. Removing the injection makes the test fail again, so it still covers the branch it is named for. The Xet shim's degraded-path tests drop utils.hf_xet_fallback from sys.modules and import a throwaway copy. Restoring only the sys.modules entry left the utils package attribute bound to the throwaway, and the two disagreed for the rest of the process: a later monkeypatch of the dotted target patched one copy while the code under test imported the other, so the patch did nothing and test_fetch_te_prequant_only_reports_what_it_downloaded reached the real Hub and got a 401. Restore both bindings. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop an ignored-cancel sd-server, guard deletes during diffusion training, repair unusable managed binaries sd-server does not interrupt an in-flight job, so when it ignores a cancel the grace branch abandoned the poll and reported cancellation while the native job kept a core (or the GPU) busy to completion and held the server's job slot. The comment said the caller stops the server, but only unload does that immediately: a superseding load stops it after its multi-gigabyte download, and a load that then fails never gets there. Stop it here, as the deadline branch already does. DELETE /api/models/delete-finetuned checked only the LLM trainer, so it could rmtree the output directory a live diffusion LoRA run was about to write its adapter into. Consult the diffusion training service too, like the dataset mutation and model-load routes. find_sd_*_binary only checks is_file(), so an interrupted extraction (or a prebuilt for the wrong CPU) left a present-but-unrunnable binary the installer never retried: every load probed it, fell back to diffusers, and native inference stayed off until the directory was deleted by hand. Probe it and reinstall, but only for a copy under the installer-owned root -- SD_CLI_PATH, UNSLOTH_SD_CPP_PATH, an in-tree build and anything on PATH are the user's. * Plan the pre-cast text encoder, and make the cross-trainer GPU admission atomic An fp8 text-encoder request loads a hosted PRE-CAST checkpoint, but the image download plan never received text_encoder_quant, so the manager staged the base repo's dense encoder (FLUX.2-dev's Mistral-24B is ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) and the load then pulled the pre-cast file inline, outside the manager's progress and disk preflight. The plan now takes the field, resolves the hosted artifact with the same resolver the injection uses, stages that file, and drops only those components' dense weight shards. The load's own prefetch takes the same treatment, since it paid the same cost. Only a checkpoint that really resolves on the Hub earns the drop, so a gated or renamed artifact still stages the dense encoder the load will fall back to. The two trainers admitted each other with independent check-then-act guards: the diffusion route checks the LLM backend several network-bound preflights before it reserves, and the LLM route checks the diffusion service well before it spawns, so two near-simultaneous starts could both pass and train on one GPU. reserve() now re-tests the LLM backend under its own lock, and the LLM route holds the diffusion service's gpu_load_admission across its spawn, so exactly one of the two wins. Both halves fail open, so a chat-only install still trains. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the Advanced options a staged download planned against Staging does not set busy, so while a multi-gigabyte download runs the user can still change precision, memory mode, speed or the baked LoRA selection. The pending record held only the repo and artifact, and the completed download fired a load that read the CURRENT state: the staged file set could then be missing files that load needs (fetched inline, with no progress and no disk preflight) or hold gigabytes it no longer uses. One snapshot of every Advanced control is now taken when the plan is built, and it travels with the pending record into the load, so the load that runs is the one the download was planned for. * Do not advertise a family the installed diffusers cannot build The newer families (Z-Image, Krea 2, FLUX.2, LTX-2, HunyuanImage) exist only from diffusers 0.39, and 0.39 cannot be installed on Python 3.9 at all -- diffusers dropped 3.9 in 0.38, so the requirement is conditional or the whole extra becomes unresolvable. On such an environment the picker still offered those rows, every pick failed deterministically, and the error's advice to run pip install -U diffusers could not fix it without also upgrading Python. The cached-repo picker now applies the same availability check validate_load_request does, which is keyed on the pipeline class actually present rather than on the Python version, so it is also right for an intentionally pinned older diffusers on 3.10+. Fails open when diffusers cannot be imported at all: that is a different problem and the load path reports it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore the diffusion engine selection after each router test The active engine is module state, and several tests set it by plain assignment because what _activate does to it is the thing under test, so monkeypatch could not undo it. A leaked ENGINE_SD_CPP left get_active_diffusion_engine() handing back the sd.cpp backend for the rest of the process, and every later route that reads the active engine then saw an unloaded model: eight tests in test_openai_images_generations_route.py returned 503 in a full-suite run while passing on their own. The autouse fixture now snapshots and restores it. * Stream gallery clips, and close three races around them Four fixes from the latest review pass. The video gallery downloaded each clip into a blob before it could play, so playback waited on the whole file (tens to hundreds of MB), seeking was limited to what had arrived, and every viewed clip stayed pinned in the webview. The file route already streams and serves ranges; it just could not be a <video src> because it is bearer-gated. Mint a short-lived signed link instead (its own HMAC secret, 12 hour TTL, separate from the image links) and hand it to the element, which then fetches only the ranges it plays. That removes the blob budget, its LRU and every revoke on this page. The sd.cpp readiness probe accepted any process answering on the port, so a foreign server that grabbed the port between the bind check and the spawn was adopted as ours. Confirm the listener is our child before reporting ready, and stay best-effort (psutil missing, an unknown owner, or any probe error still passes) so the check can only reject a definitely foreign process. Dataset import held its lock for the extract but not for the upload path, so two concurrent uploads into the same folder interleaved; take the same lock and return 409. And reject Windows device names (CON, NUL, COM1..9, LPT1..9, with or without an extension) plus trailing periods in dataset names, which are unopenable on Windows. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Build the image download plan for the engine that will load /images/download-plan always asked the diffusers backend, while /images/load picks the engine per host: a GGUF pick on a machine with no usable GPU routes to native sd.cpp, which reads a single-file VAE plus text encoders and never opens the base repo's sharded components. Measured on unsloth/FLUX.2-klein-4B-GGUF (Q2_K): the plan staged 7.66 GB of FLUX.2-klein-4B components the native load discards, and the 7.80 GB sd-cli actually needs was then fetched inline by the loader, outside the download manager's progress and its disk preflight. Z-Image-Turbo is the same shape. The plan now asks whichever engine the load will select. predict_engine() applies the selection policy without any side effect: it activates nothing (staging a download must not unload the resident model) and only locates the binary rather than installing it, but still counts an installable binary as available, since that is what the load does on a fresh host. The native backend gains a download_plan built from the same _asset_specs the loader fetches, returning the same envelope, so the manager stages exactly the files sd-cli opens. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not let a queued generation outlive the model, and three scan fixes Five items from the latest review; four were real. An unload or arbiter eviction only cancels the generation holding _generate_lock. A second request queued behind it holds no cancel event yet, and Python locks are not FIFO, so it could take the lock the instant the active denoise released it, still see a loaded pipeline, and run a whole new denoise after the model was told to go away: the eviction then waits minutes for it and an image lands after the eject. Unload and a superseding load now raise a fence under _lock before they queue, and a generation that wins the lock while one is pending refuses instead. The cached-model scan judged pipeline completeness across every revision, so a repo holding an older complete snapshot plus a newer companion-only one read as complete while the snapshot from_pretrained actually opens has no transformer. Both scans now look at the revision the loader will open. Deleting a dataset image deleted its caption sidecar unconditionally, which for cat.jpg alongside cat.png removed the caption the survivor still resolves to. The sidecar now goes only with the last image of that stem, matching what the thumbnail cleanup beside it already did. Importing an example into a folder that holds no images but does hold files fell back to promoting the staging dir one file at a time, so an interruption left a partial dataset that the image_count check accepts as complete on retry. Those files are folded into the staging dir instead and the promotion stays a single atomic rename. The MPS generator report does not apply: torch.Generator(device="mps") has worked since PyTorch 2.0 (pytorch/pytorch#91348) and the studio installer pins torch>=2.4. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten diffusion comments Collapse the multi-line comment blocks across the image, video, sd.cpp and diffusion-training code to one or two lines each, and drop comments that only restate the statement below them. Comments only, no code or behaviour changes. * Tighten diffusion comments (second pass) Collapse the remaining multi-line comment blocks in the video page, training routes and service, sd.cpp server and installer, memory and speed planners, and the shared request models. Comments only, no code or behaviour changes. * Tighten diffusion comments (third pass) Collapse the remaining multi-line comment blocks in the attention, cache, LoRA, prequant, precision and compile-cache modules, the sd.cpp arg builder and engine, the video routes, the Ideogram 4 assembly, the model picker, and the diffusion test suites. Comments only, no code or behaviour changes. * Restore the dataset when an example import cannot be promoted Promotion folds the folder's pre-existing entries into the staging dir so the swap is one atomic rename. Every failure after that fold left the user with nothing: the 409 path and an os.replace error both fell through to 'finally: shutil.rmtree(staging)', which deleted the entries that had just been moved in there, while the response said 'Nothing was written'. A same-named entry was also unlinked outright before the promotion was known to succeed. Park superseded same-name entries in a rescue dir instead of deleting them, record every move, and restore all of them if any step of the promotion fails. The fold loop itself is covered too: renaming a non-writable directory raises EACCES on POSIX, which previously escaped as a 500 after the earlier entries had already been moved out. A failed rename now maps to the same retryable 409 as the rmdir conflict. Verified with the reported trigger (a non-empty mode-500 directory whose name collides with an imported file): the folder listing is now identical before and after the failed import. * Drain the teardown fence on a failing unload, give each load its own cancel event Two independent leaks on the image path, both already solved elsewhere in the same file. unload() incremented _teardown_waiters, ran _unload_locked() and decremented, with no try/finally, while the superseding-load path used a finally for the same pair. _unload_locked ends in clear_gpu_cache(), whose CUDA branch calls synchronize/empty_cache/ipc_collect unguarded, and a sticky CUDA fault makes those raise. The count then never drained, so every later generation was refused as cancelled for the life of the process, a fresh load included, since begin_load's own increment and decrement are symmetric. unload() is reached from the chat/video GPU handoff, the engine router and two training routes, so one fault during an ordinary handoff wedged image generation until restart. Release it in a finally. The image and native backends each cleared one shared cancel Event on a new load. unload() sets that event to cancel an in-flight multi-GB download and drops _loading in the same breath, so a replacement load is admitted while the cancelled worker is still inside the fetch, and its clear() re-enabled the very object that worker was watching: the cancelled download resumed and ran alongside the replacement. Take a fresh Event per load and thread it to the worker, as the video backend already does, and set it under the lock since begin_load now rebinds the attribute. * Name utf-8 on the diffusion text I/O and the sd.cpp subprocess pipes tests/test_text_io_encoding.py failed on five files this branch adds. Text I/O without an explicit encoding falls back to the Windows ANSI codepage, so a non-ASCII path or manifest value round-trips corrupted, and the three sd.cpp pipes decode the child's UTF-8 output as ANSI on Windows despite already passing errors = 'replace'. Eleven read_text() / write_text() sites across diffusion_compile_cache, diffusion_ideogram4 and diffusion_krea2, plus text = True on the sd-cli version probe, the sd-cli run and the sd-server pipe. * Record the load-time build on a gallery image's recipe A gallery record documents itself as the image's full generation recipe and is embedded in the PNG, but the only load-related field it carried was the repo id. A GGUF repo holds many quants, so that does not say which one made the pixels, and it says nothing about an adapter baked in at load time. The fallback meant to cover the baked case could never fire: with no loras on the request _adjust_baked_loras zeroes every baked adapter and _active_lora_pairs drops zero-weight entries, so active_loras was always empty. A baked-and-disabled build is not the same pipeline as a never-baked one, so the recipe could not reconstruct the image once the model was rebuilt. Persist model_kind, gguf_filename, transformer_quant and the baked adapter names, read off the load state rather than the request, and show them in the recipe popover. The new fields are optional with defaults, which matters because list_gallery_images drops any record that fails validation, so a required field would have emptied every existing gallery; a regression test pins that. * Drop eleven duplicated comment tails, restore the mxfp8 denial note The comment passes collapsed several wrapped blocks onto one line without deleting the last physical line of the original wrap, leaving the tail of each sentence repeated as its own comment underneath. Two of the eleven were re-worded rather than byte-identical, so a strict suffix match missed them. 6e16ad16f also dropped the line justifying the qwen-image mxfp8 denial while that rule stayed live in _FAMILY_SCHEME_DENY, under a header that then documented only fp8 and nvfp4. Restored. Comments only; verified with the AST gate. * Only let the dense-quant fallback use shards the prefetch actually staged The prefetch skips the base repo's transformer/ shards whenever a prequant checkpoint is expected, since that checkpoint replaces them. But a prequant fetch can fail for reasons the planner cannot see: an unpublished, gated or renamed artifact, a hub 5xx, a proxy, a checkpoint the validator rejects. The loader then fell through to from_pretrained(subfolder = 'transformer') and pulled those shards inside the load lock during 'finalizing', after the previous pipeline was already evicted, where the cancel event has no reach, load_progress has already reported bytes_downloaded == bytes_total, and the cache-disk gate had only reserved the small prequant checkpoint. That is verbatim the situation _dense_quant_prefetch_needed's own docstring exists to prevent. Gate the in-loader dense fallback on the shards being staged, read off the returned file list rather than the request so a failed size estimate closes it too, and let the GGUF build take over otherwise, which is what the prequant-sized replan already does one branch over. It is also the invariant the text-encoder path already enforces: only a component that really resolves may have its dense weights dropped from a plan or a prefetch. Adds the missing coverage for the gate's prequant arm, which the existing disk-gate tests never reached. * Refuse a training output_dir that resolves to the outputs root resolve_output_dir strips a leading 'outputs' and drops '' / '.' segments, so '.', './', './.', 'outputs', 'outputs/outputs' and ' . ' all clean away to nothing and land on the outputs root itself rather than a run directory under it. The DiT trainer then writes pytorch_lora_weights.safetensors flat into the root, where scan_trained_models and scan_checkpoints cannot see it (both filter is_dir()), and a second such run overwrites the first. The UI only checks the field is non-empty, so 'outputs' is one plausible run name away. Refuse it with a 400 that says what to do instead. cond_cache_dir collapses the same way but has an honest 'off' to fall back to, so a root-resolving value now means the in-memory cache, which is what the comment above it already promised: otherwise a run drops one flat safetensors per cached latent and caption into the directory trained models live in. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Repair three defects the main merge left behind Git merged all three files without a conflict, but the result was wrong in each case. The tests only started failing once the merged tree was run. routes/inference.py duplicated the GGUF load block: this branch had moved the gguf_load_in_flight marker and the _hub_download_blocks_gguf_load guard under "if config.is_gguf and config.gguf_hf_repo", and the conflict resolution re-added main's copy at the old position, so both ran. Dropped main's copy; the earlier placement is the deliberate one, so a 409 from the hub guard cannot tear down a resident Images or Video pipeline. test_gpu_selection.py still called _hf_offline_if_dns_dead, which main renamed to _hf_offline_if_unreachable_for (#7591). Disjoint edits, so no conflict, but four route-error tests referenced a function that no longer exists. test_gguf_load_cache_reuse.py anchored its ordering assertion with rindex over "if config.is_gguf:", taking the last one before the load marker. That only held while _resolve_inherited_extra_args sat above every such line; main has since hoisted it above the gpu_ids preflight, so the anchor landed between the call and the marker and the assertion compared against an unrelated later call site. The ordering it checks is a property of _load_model_impl as a whole, so it now anchors on the function. The ordering itself is intact: _resolve_inherited_extra_args, then the gguf_load_in_flight marker, then the hub guard, then the chat handoff, then unload_model. * Stop a real sd.cpp install from breaking its own discovery tests The five "nothing is installed" assertions in test_sd_cpp_engine.py cleared SD_CLI_PATH and UNSLOTH_SD_CPP_PATH and patched Path.home, which covers hops 1, 2 and the fallback half of hop 3. It leaves two hops live. Hop 3 goes through managed_install_root(), which honors UNSLOTH_STUDIO_HOME and STUDIO_HOME and resolves to the stable-diffusion.cpp directory beside the studio home. That is the documented way to run side-by-side Studios, so anyone who has one set gets a real sd-cli back from a finder the test expects to return None. Hop 4 is the in-tree developer build, which does the same for anyone who built sd.cpp inside the checkout. Isolated with an autouse fixture rather than another helper call, because reaching the failure needs no fixture: SdCppEngine(binary = None) runs the finder from its constructor, which is why test_generate_raises_when_binary_missing failed too. The fixture points the studio home and the in-tree root at an empty tmp tree, so every hop is answered by the test rather than by the box. The in-tree root moves into a named in_tree_install_root() so it can be pointed somewhere empty; behaviour is unchanged, including the OSError and IndexError guard on an unexpected layout. * Make the sd.cpp uninstall test actually extract the code it tests The two sed ranges anchored the production fragments at column 0, but both blocks sit inside the main removal function and are indented, so each range matched nothing and LOOP_FILE / DEFAULT_FILE came out empty. Sourcing an empty file is a no-op, so the suite reported 4 passed / 7 failed: the seven removal assertions failed because nothing ran, and the four that passed were 'kept' assertions that pass trivially when nothing runs. The shell job auto-discovers tests/sh/test_*.sh and runs each under set -e, and this file is not in its skip list, so it was a deterministic red. Anchor on optional leading whitespace and fail loudly on an empty or _remove_path-less fragment, so a future reshuffle of uninstall.sh cannot make the suite vacuous again. Now 11 passed, 0 failed, against the real removal loop and the real default-mode block. * Agree on what an engine can build, and on what the installer owns Two gates, each half-applied. The unbuildable-family gate had one caller, the image branch of the cached repo picker. The GGUF classifier, the local-model classifier and the video branch had none, so on a diffusers too old for a family the picker still offered its GGUF and the load then failed. Meanwhile the loader asserted the diffusers pipeline class before engine selection, so a GGUF this host routes to native sd.cpp, which instantiates no pipeline class at all, was refused with an upgrade instruction that could not help. Both are wrong one-sidedly: hide a family only when NEITHER engine can build it, and demand the diffusers class only when diffusers is what will load it. One predicate, family_buildable_here, now answers both, so the picker and the loader cannot disagree. The population is real on Python 3.9, whose diffusers ceiling is 0.36: Flux2KleinPipeline, Krea2Pipeline and LTX2Pipeline are all absent there, and FLUX.2-klein GGUF is a repo in this PR's title. That assertion also raised RuntimeError, which /images/load maps to 409, the status that otherwise means a load is already in progress, and which escaped /images/download-plan (it catches ValueError and FileNotFoundError) as a bare 500 with the message lost. It is an unloadable pick like every other, so it raises ValueError and both routes answer 400 with the text intact. The managed-binary repair used a path test for ownership while the installer uses a marker. On a managed root without the marker, which is any install predating it, the repair deleted sd-server and the reinstall was then refused, permanently, because the surviving sd-cli keeps the directory non-empty so the marker can never be claimed: the user went from an unrunnable binary to no binary and no way back. Require the marker before discarding, which is the same definition of ours that uninstall.sh already uses to keep a user's own stable-diffusion.cpp checkout. A genuinely interrupted extraction still self-heals, since install() writes the marker before it extracts. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add gguf and av to the studio extra so the wheel install matches studio.txt This branch added both to studio/backend/requirements/studio.txt, which is what install.sh uses, but never to the studio extra in pyproject.toml, which is what 'pip install unsloth[studio]' uses. Nothing else keeps the two in sync, so tests/studio/install/test_studio_extra_matches_requirements.py failed deterministically, and the CI job runs the whole tests/ tree. The drift is not only a red test: gguf backs diffusers' GGUFQuantizationConfig and av does the MP4 encode and audio mux, so a wheel install got a Studio that cannot read a GGUF or export a video, which is most of what this PR adds. * Fix two test-suite defects: the CPU-only patch gate and a popped module The patch backend reaches unsloth_zoo's helpers only through an ~940 MB 'import unsloth', which a CPU-only host cannot complete, so every patch install returned False and nine arch/eager/compile tests failed there -- exactly the runners the retry was narrowed to protect. unsloth_zoo only wants UNSLOTH_IS_PRESENT in the environment, which costs nothing and needs no accelerator, and the conftest already sets its sibling UNSLOTH_ALLOW_CPU. Set it there too, as run.py and main.py already do at module scope, so no real server ever took the expensive route. CPU-only goes from 9 failed to 24 passed, 1 skipped; the GPU run is unchanged at 29 passed. test_setup_cache_env_hf_home popped utils.hf_cache_settings to model a fresh process and never restored it, so a later import built a second module object and rebound it on the utils package. test_hf_cache_settings then wrote its setting into one object while core.inference.diffusion read the other, the same split-module failure the xet shim already had. Restore both bindings in teardown: 21 pass together, and each file still passes alone. * Restore the setup_fail assertions the main merge reverted A fourth instance of the class 9541cc535 fixed: the merge took main's studio/setup.sh, which #7644 changed to abort through the setup_fail helper so desktop mode still emits [TAURI:ERROR], but kept this branch's older copy of the test, which still asserted the literal 'exit 1'. setup.sh is byte-identical to main here and the only diff in this file was the reverted assertions, so take main's version. 60 passed. * Fence the video teardown, reject non-finite clipping knobs, stop a test installing sd.cpp The video backend never had the teardown fence the image backend documents. Its unload and its superseding-load path both signalled the active generation, then did 'with self._generate_lock: pass' and tore the state down with the lock free. A generation queued behind that barrier holds no cancel event yet, so the signal cannot reach it, and Python locks are not FIFO: it won the lock the instant the barrier released it, read a still-loaded state and denoised a whole clip against the pipeline being freed. Reproduced on both paths, where the queued generation returned a finished MP4. Mirror the image backend: count waiters, refuse a generation while one is pending, and tear down inside the barrier with the counter released in a finally. _teardown_state becomes _teardown_state_locked since the caller now holds both locks. max_grad_norm and snr_gamma were bounded on one side only, so 1e309 floated to inf and passed. clip_grad_norm_ then computes an infinite clip coefficient, clamps it to 1.0 and scales nothing, and min(snr, inf) / snr makes every min-SNR weight 1.0. The run starts, reports normal progress and trains with the requested knob silently disabled. Measured both. NaN already failed the bounds; allow_inf_nan makes that explicit. learning_rate and flow_shift already guard this exact vector. test_load_routes_to_sd_cpp_on_cpu stubbed ensure_sd_cpp_binary but not ensure_sd_server_binary, which select_and_activate_engine probes first with installs enabled, so the unit test downloaded and unpacked 108 MB into the developer's real ~/.unsloth when UNSLOTH_STUDIO_HOME was unset. Now 0 bytes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Key the video schedule reset on the resolved defaults, not the repo id A GGUF repo holds several variants, so another client swapping a distilled build for the base one from the same repo changes the steps and guidance the backend reports while repo_id stays put. modelChanged stayed false, the page kept the previous schedule, and later generations ran an 8-step distilled setting on a model that expects roughly 40 steps and CFG 4. Include the reported defaults in the key. They only move when the resident artifact does, so a manual steps change still survives a status poll. * Retire the release-lag pin now that 2026.7.6 carries the relaxed gates The virgin Windows container row that installs from PyPI on purpose was tolerated with continue-on-error, because a Server Core container has no Microsoft Store and so no winget, and the released studio/setup.ps1 hard-stopped on a winget-only git gate and reached for winget again for the VC++ runtime. #7549 relaxed both, but it landed on 2026-07-28 and the newest wheel was 2026.7.5 from the 23rd, so the row could not get past it. The pin carried its own tripwire for exactly this moment, and it has fired: unsloth 2026.7.6 shipped 2026-07-29 and the released wheel now installs end to end, so the step reported success and the assertion errored with "delete this pin and drop continue-on-error from the Install step so this row gates". Doing that. Both rows now gate unconditionally, and the assertion that enumerated the accepted failure signatures goes with the pin it protected, since there is no longer an accepted failure for it to describe. The other three release-lag mentions in this file are descriptive rather than tolerated-failure pins: they explain why the fedora and ubuntu-nonroot legs run with overlay: true, and both already refuse a triton failure as a regression rather than accepting it as lag. Left alone. * Pin the hosted pre-quant onto the plain-torch fp8 kernel A Z-Image GGUF pick at the DEFAULT speed mode, once the hosted fp8 pre-quant is actually reachable, dies at generate with an HTTP 500: torch._dynamo.exc.Unsupported: Operator does not support running with fake tensors. Developer debug context: unsupported operator: mslk.f8f8bf16_rowwise.default _fp8_config already pins KernelPreference.TORCH when it BUILDS a config, precisely because the default AUTO switches to the MSLK kernel wherever an mslk package is importable (sm90+). A hosted checkpoint escapes that pin completely: the preference is serialized per Float8Tensor, and all 239 weights in the published Z-Image-Turbo-FP8 checkpoint carry AUTO. Loading it re-arms the very kernel the pin exists to avoid, mslk.f8f8bf16_rowwise has no fake impl, and the first compiled generate therefore cannot be traced. Isolated away from the product to be sure of the mechanism: quantise one Linear three ways on this box and compile each. KernelPreference.TORCH eager ok, compiled ok KernelPreference.AUTO eager ok, compiled FAILS on mslk.f8f8bf16_rowwise library default eager ok, compiled FAILS the same way So the pin is correct and necessary, and the only gap is that the hosted path never got it. _validate_checkpoint checks scheme, granularity, base model, min_features, exclude tokens and fast_accum, but not this. Rewriting the preference on load is safe: it selects a matmul kernel, it is not weight data, so the tensors stay bit-identical and the checkpoint's own state_dict_sha256 still describes them. It is also the faster path compiled, since the opaque extern call blocks inductor quantize fusion. Why this went unnoticed: the pre-quant repos are private, so nothing that could not read them ever took this path. It becomes the default the moment they are readable. Verified end to end at 1024x1024, 8 steps, speed_mode default, on the published checkpoint: before HTTP 500 at generate, diffusion.generate_failed after pinned 239 weights to the plain-torch fp8 kernel, load 10.0 s, transformer_quant fp8 with compiled engaged, cold 7.9 s, warm 0.87 0.87 0.87 0.86 0.87 s That warm number also beats the 1.4 s recorded for this shape on 2026-07-26. 159 prequant and transformer-quant tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cover the fp8 kernel pin so the 500 cannot come back quietly The hosted pre-quant re-arming MSLK was only visible as an HTTP 500 on a compiled generate, which needs a GPU, a readable private repo and an importable mslk to reproduce. None of that is available in CI, so the pin would rot unnoticed. Three hermetic cases on _pin_kernel_preference instead: AUTO weights are rewritten and already-TORCH ones are left alone (counting only what changed), a weight that refuses the assignment does not sink the whole load, and with no torchao enum available the checkpoint is left exactly as saved. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: tighten comments across the image generation, video and training code Rewrite the comments added by this branch to be shorter and clearer: collapse multi-line explanations into one or two lines, drop comments that only restate the code, and keep the rationale that explains why a choice was made. No code changes. * Studio: let the pipeline-class guard survive a host with no diffusers Backend CI installs the CPU-only dependency set, which has no diffusers, and runs without a GPU. 13 tests failed there for three separate reasons, one of which is a real product bug rather than a test-environment gap. assert_pipeline_class_available did a bare "import diffusers", so on any host without it the call raised ModuleNotFoundError instead of the ValueError its own contract promises. /images/download-plan catches only ValueError and FileNotFoundError, so that escaped as a bare 500 with the message lost, which is the exact failure the guard exists to prevent. It is reachable in production: the native sd.cpp engine serves GGUF picks on a CPU or Apple host where diffusers is never installed. Absent diffusers now returns, since the check answers "is the installed diffusers new enough for this family" and there is no version to judge; a pick that genuinely needs diffusers still fails in the loader with its own message. The rest were tests asserting through gates unrelated to what they cover: - Three cond_cache_dir route tests drive a DiT family, which the start route refuses without an accelerator. They now take the existing dit_train_host fixture, written for this case, so they keep testing the schema on every host. - test_download_plan_forwards_the_load_time_controls stubs the diffusers planner but sends a GGUF pick, which routes to the native planner on a GPU-less host. It now pins the engine, so it tests kwarg forwarding rather than the hardware. Engine selection keeps its own tests. - The pipeline-class guard test needed a real diffusers only for its sweep over every shipped family. That half is split out and skips on its own; the stub-driven refusal needs no diffusers and still runs. Added a test for the absent-diffusers contract above. 409 pass on a GPU host with diffusers; 301 pass and 1 skips with diffusers blocked and no GPU. * Studio: cut the image generation comments down further Second pass over the comments this branch adds. Collapse the multi-line blocks that still read as paragraphs, rewrite the longest one-liners so they say the same thing in fewer words, and drop a stale comment that had drifted away from the constant it described. No code changes: only comments and whitespace. * Studio: workflow rail, sidebar Video row, and Train settings polish Sidebar - Pin Video under Images by default; persist store bumped to v3 so an untouched install adopts it while a custom arrangement is left alone. - Add "Customize sidebar" to the end of the More flyout, opening Settings > Appearance scrolled to the sidebar nav section. Images - Replace the workflow dropdown with an icon rail down the left edge: all seven workflows visible, keyboard nav, tooltips carry the labels. - Workflows stay selectable with no model loaded, so one can be set up before picking a model. Generate is already gated on a loaded model. - Video moves out of the Create/Train strip into a link at the far right, with a matching Images link on the Video page. - Wider gutters around the settings column, headings matched to Train. Train - Field guidance moves into "i" tooltips; only state that limits a control stays on the page. Steps, LoRA rank and the rest gain hints. - /info reports params, qlora_vram_gb, gated and note as fields, so the family note renders as chips. vram_note is rebuilt from them unchanged for older clients. Shared - Number steppers appear on hover or focus. - Lighter control border (#e9e9e9) on Images and Video, light mode only. * Studio: move Advanced inline on Images and Video - Replace the top-bar toggle and right-docked Advanced panel with a disclosure under Seed, so load-time tuning sits with the settings it affects and opening it no longer shrinks the preview canvas. - Video uses the bordered variant, since offload and memory decide whether a model fits at all. - Open state persists per page in localStorage. - Tighten the Steps unit trigger: 14px chevron, less right padding. * Studio: full width media pages and layout polish - Drop the 1100px cap on Images Create, Images Train and Video, so the preview canvas grows with the window instead of sitting in a band. - Align the workflow rail to the model selector label, widen the gap before its divider, and restore the divider itself. - Advanced: more room above and below, icon and label sized to the slider rows, and the same quiet row on Video as on Images. - Give Video a pane heading and description, matching Images. - More space above Seed on both pages. - More no longer takes the active style when the current page is one of its own rows. * Studio: fix the Extend side toggles The resting outline used a ring, and index.css blanks the ring on the button holding mouse focus, so the side you just clicked lost its border until focus moved. Use a border for the outline and leave the ring to focus-visible. Also separates the two states properly (the off state had no surface of its own), drops the border in dark mode as the inputs do, and adds the aria-pressed and focus-visible styling the buttons were missing. * Studio: drop the border on the Extend side toggles Light mode should carry no border either, so the fill alone marks the state: a muted surface when off, a primary tint when on. Matches the borderless treatment dark mode already had. * Studio: move the Images workflows into the sidebar The workflow switcher was a vertical icon rail on the page. It now lives in the sidebar under Images, so the page keeps that width for the canvas. - Workflows list under the Images row, with a chevron to fold them away. Hovering the row peeks them in a flyout on the standard menu surface. - Clicking Images while already there toggles the list instead of navigating. - The listed workflow carries the highlight, so the Images row drops it. - Create takes its own icon: it was sharing the New chat pencil. - Model hub moves above Projects. Persisted layouts bump to v4 and only adopt the new order where the stored one is still a shipped default. - Images and Video content both start at 32px, clear of the sidebar. * Studio: simplify the Images workflow submenu Dropping the hover flyout: two ways to reach the same seven workflows, one of them overlapping the list right below it, read as clutter. - The flyout is gone. The workflows are rows under Images and nothing else. - On the Images page they are always open, since they are that page's switcher. Elsewhere they stay folded, and the row's chevron opens them. - Create takes the sparkles icon. - Create and Train both pad their settings column to 40px a side, so the two tabs line up with each other and with the model selector. * Studio: dock the generate action and tidy the Create controls The primary action sat at the foot of a long scroll, so it was off screen until you scrolled for it. - Generate, Video's Generate and Train's Start training float at the bottom of their settings column. No bar behind them: hover lightens the fill rather than thinning it, and the disabled state is opaque, so the controls underneath never show through. - Aspect ratios read as names: Square (1:1), Widescreen (16:9) and so on. - Width and height replace their sliders with two compact boxes. Type a size or pick one from the menu; the value still snaps into range and still drives the locked ratio. - Negative prompt gets a quiet disclosure under the prompt. It was there before but only above guidance 0, which is not the default, so it never showed. - The info "i" is smaller across the settings UI. * Studio: reveal Images and Video field hints on hover The "i" next to every field label sat there permanently, which made a column of settings read as busier than it is. Scoped to the diffusion pages by CSS rather than per component: the hints come from the page's own Field and SelectRow, the train panel's FieldLabel, and the chat ParamSlider, which is shared and should not change for chat. Reveals when the pointer is inside the field, on focus-within for keyboard, and stays inert while hidden. InfoHint grows a data-slot so the rule has a stable hook, rather than keying off its aria-label. * Studio: one-line sliders, edge fades, shared negative prompt - Steps, Guidance and the rest put label, track and value on one row. ParamSlider is shared with chat, so this is an opt-in prop and chat keeps the stacked layout. - The settings columns fade at whichever edge they run past, as the sidebar and model picker do, instead of cutting off. New useScrollFades hook drives all three. - Negative prompt moves to a shared component and Video picks it up, so both pages collapse it the same way. - Right gutter sits closer to the rule, Steps takes a bigger break above it, and the Advanced rule sits between its neighbours rather than up against the field above. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: add the API monitor to the sidebar nav Sits under More, below Export. Points at the monitor page, not the API keys dialog the profile menu opens. No store version bump: an id missing from a stored layout is appended with its default pinned state, so existing sidebars gain the row at the end and keep their own order. * Studio: take main's copy of the sidebar spinner test The merge kept this branch's older version, which pins the Recents row to h-[33px]. Main already relaxed that to any height, since row density moves independently of the trailing column the test is about. * Studio: fix FLUX.2 klein size resolution, HunyuanVideo tier routing, and the personalization save Five fixes found while auditing the model registries against the live Hub. FLUX.2 klein 9B loaded against the 4B config. One family covers both klein sizes and defaults to 4B, relying on the base_model card tag for the real base, but the trust allowlist only had klein-4B, so the correct tag was discarded and a 9B GGUF (inner_dim 4096) was loaded against a 4B config (3072). That surfaced as a bare shape mismatch from inside the GGUF quantizer naming neither the file nor the repo. Adds the missing allowlist entries and a header-level size check that fails early with a legible message; the check is fail-open, so an unreadable file, a non-FLUX.2 family, or an unmapped base leaves the load exactly as it was. The sd.cpp text encoder rule matched the literal "klein-9b", so klein-base-9B was handed the 4B encoder. HunyuanVideo-1.5 720p checkpoints routed to the 480p family. Only the literal 720p_t2v path was aliased, so 720p_i2v and every GGUF repack fell through to the generic token and inherited the 480p base repo, which also supplies the VAE and text encoder. The tier is baked into the weights (transformer target_size 640 vs 960, scheduler shift 5.0 vs 9.0) and the two bucket lists are disjoint, so this ran the whole pipeline off-tier. Resolution presets that the checkpoints were never trained for. Wan2.2 TI2V-5B is 720P-only upstream (SUPPORTED_SIZES is exactly 704x1280 and 1280x704, asserted in generate.py), and the HunyuanVideo 480p square preset 624x624 is not a bucket of its tier; 640x640 is. Personalization could never be saved. The frontend ships eight sidebar nav ids and refills every missing one on each save, but the backend Literal had seven and no "api", so every PUT to /api/settings/personalization returned 422. Aligns the server defaults and the Literal with the shipped layout. A catalog row pointing at a file that does not exist. unsloth/Qwen-Image-2512-FP8 holds torch prequant .pt checkpoints, not qwen-image-2512-fp8.safetensors, and fp8 is denied for this family anyway because it renders black, yet the row was the auto-route target on any GPU above 40 GB. Also marks FLUX.1-schnell, Krea-2-Turbo and the two Ideogram repos gated: all four are gated on the Hub today, and schnell being Apache-2.0 does not make it ungated. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: stop a case-only duplicate upload destroying an image, and size the load against the right GPU Two fixes from the review round. A dataset upload containing two names that differ only by case silently loses one on Windows and macOS. The in-batch duplicate check is an exact string compare, so "Cat.png" and "cat.png" both pass, but on a case-folding filesystem they are one destination: the commit step moves the first staged part aside, writes the second over it, then deletes the backup, while the response still reports both as uploaded. Captions collapse the same way and never even reach the image branch. No single folder can hold such a pair, but the Windows and macOS open dialogs flatten search and Recents results across folders into one multi-selectable list, which is a normal way a LoRA dataset gets assembled. Rejecting the pair everywhere would regress Linux, where the two really are different files with their own sidecars, so the check probes the filesystem once per process instead of keying off sys.platform: macOS also ships case-sensitive APFS volumes, and a Linux host can keep its Studio home on an exFAT or NTFS mount. A failed probe answers "case-sensitive", which leaves today's behaviour untouched. The quantization fit budget was sized against the wrong card under a reordering CUDA_VISIBLE_DEVICES. A bare "cuda" load lands on visible ordinal 0, but the reducer took the lowest PHYSICAL index, which the nvidia-smi path reports as index_kind "physical". Under "3,1" that sizes against GPU 1 while the pipeline loads onto GPU 3. The hook's own interface doc already promised the lowest visible ordinal. Ranks by visible_ordinal, falling back to index only for an older backend that omits it. * Studio: rework the diffusion dataset panel and align the pane gutters Upload: - one Upload button beside the dataset name, with an upload icon - picking files uploads them, no second confirm click - Add button beside the Training images dropdown, so images can go into a set that already exists - fall back to the upload form when a selected dataset no longer resolves Labeling grid: - two columns at any width; sm:grid-cols-3 crushed the tiles in a fixed width column - pin the caption size so the Textarea's md:text-sm does not outsize a tile - name the tile hover group; a bare one revealed every tile's Remove at once - Remove is an icon button on the image, not a word over the artwork - drop the amber tile fill, keep the No caption label - match the header, toggle and status text to the section's type scale Gutters: - the run area and the preview canvas now sit 40px off the rule, the gutter the settings column has off the page edge * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: give the dataset preview per-image hover and remove, and wrap it onto rows - each thumbnail is its own button, so it can carry an X that removes just that image; the strip was one button and could not nest a second control - hover lightens the image instead of drawing a border - wraps onto more rows rather than scrolling sideways, which also keeps the bottom corners rounded - samples 12 thumbnails, up from 8, now that a second row is available - a delete refreshes the panel's dataset counts * Studio: show the workflow icon beside the Images and Video headings Images reads it off WORKFLOW_TABS, so it is the same icon the sidebar submenu shows and it follows the active workflow. Video's heading is also Create, so it takes the Create icon rather than the nav row's film slate. * Studio: reject a duplicate LoRA id on the diffusion load path too DiffusionGenerateRequest already refuses a repeated adapter id, with a comment naming the hazard: _resolve_lora_set suffixes colliding adapter names, so the same id resolves the SAME adapter twice and set_adapters stacks both copies past the per-adapter weight bound. DiffusionLoadRequest bounds only the list length, and it matters more there. Generation-time stacking spoils one image; on the load path the adapters are baked into the quantized build before compilation, so the unintended combination rides every image until a reload. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: teach the GGUF reuse test double about holds_no_vram The resident-GGUF fast path consults llama_backend.holds_no_vram before asserting CHAT ownership, but two tests hand it a SimpleNamespace built before that attribute existed, so they raised AttributeError and returned a 500. A real LlamaCppBackend exposes it as a property; the doubles now carry it too. Product code is unchanged: these pass on main and were the branch breaking its own test, not a defect in the guard. --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: michaelhan <michaelhan2050@gmail.com> Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: Michael Han <107991372+shimmyshimmer@users.noreply.github.com> | 1 个月前 | |
One interpreter leg on a pull request, and a floor lint that reads more than syntax (#9100) * One interpreter leg on a pull request, and a floor lint that reads more than syntax A pull request ran 3.10 and 3.13. It now runs 3.13 only. Main still runs all four, so anything that needs a real run is caught at merge rather than never. The leg that goes is worth something, so this pays for it rather than dropping it. What a dropped floor leg actually stops catching is not syntax: it is reaching for a stdlib name that does not exist yet. core/research_runs.py already uses anext, which is 3.10, and that parses perfectly on every version and fails only when the line runs, so the existing ast.parse floor check would not have seen it. scripts/lint_backend_python_floor.py asks vermin instead, which reads syntax AND stdlib API availability, and takes its target from the workflow's own matrix rather than a number written in the script. Adding a call to itertools.batched, which is 3.12, fails it in seconds. It runs from workflow-trigger-lint.yml, which carries no paths filter, so it sees the pull requests that touch only backend source -- the ones that most need it now. The single leg has to be the NEWEST. Removals and deprecations land on the newest interpreter first and on the oldest never, so running only the oldest would be the wrong single choice; the guard asserts which end it is. What is genuinely given up, kept visible rather than deleted along with the old guard: the backend has version_info branches at 3.10, 3.11 and 3.12 boundaries, and a pull request no longer takes both sides of any of them. Nothing static covers that -- a parse reads both sides and runs neither. test_the_boundaries_the_subset_stops_executing_are_still_run_on_main lists them and fails if main ever stops running the full matrix, at which point this stops being a trade and becomes a straight loss. Mutation tested: making the single leg the floor fails one test, dropping the lint invocation fails another, removing vermin from the install fails it too, and taking Backend CI off push-to-main fails two. That vermin check needed a second pass: the first version looked for the string anywhere in the workflow and was satisfied by a comment mentioning it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scan the backend tree, not a list of packages I remembered The floor lint named core, utils and routes, and silently missed 116 shipped files: all of hub, plugins, models, storage, auth, picker and state, plus _platform_compat.py, which main.py imports directly. It also named "loggers.py", which is a directory, so that entry matched nothing at all. With the 3.10 leg dropped this lint is the only thing looking at the floor before a merge, and a check that covers most of a tree reads exactly like one that covers all of it. It now scans studio/backend and excludes only tests and vendored code, which takes it from 307 files to 422. Verified by putting an itertools.batched call, which is 3.12, into each of hub, auth, picker, state, storage, models, plugins and _platform_compat.py in turn: every one is caught now, and none of them was before. Widening it immediately found something real, which is the point: locale.getencoding is 3.11 and the floor is 3.10. It turns out to be correctly guarded, in a try/except AttributeError whose fallback is locale.getpreferredencoding(False), commented "Python < 3.11". vermin reads names rather than control flow, so a guarded attribute lookup is indistinguishable from an unguarded one. That file is exempt with its reason printed on every run, and an exemption naming a path that no longer exists fails the lint, so it cannot outlive the guard it was written for. The guard test counts what the lint would hand to vermin against what is on disk, so narrowing the input back to a package list fails rather than quietly shrinking coverage. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Suppress the guarded call, not the file it lives in Excluding state_store.py wholesale left everything else in it permanently unchecked, which is the package-allowlist mistake from the previous commit one level down: a new unguarded 3.12 call anywhere in that module would have passed the floor lint on a pull request that runs only 3.13. The suppression moves to the site. vermin honours a `# novermin` annotation, so comment parsing is on now and the one guarded call carries the annotation with a note saying the except below IS the guard and that vermin reads names rather than control flow. The file is back in the scan, which is 422 files again rather than 421, and adding an unguarded itertools.batched call elsewhere in it now fails. The coverage test no longer permits any file-level exemption at all, rather than permitting a recorded one, so reintroducing the exclusion fails it. Separately: the assertion that the lint step exists was only collected by Backend CI, whose paths cover its own YAML and studio/**, not .github/workflows/**. A pull request editing only workflow-trigger-lint.yml could therefore delete the step without failing anything, which is the one change the assertion exists to reject. It runs from that unfiltered workflow now, alongside the three guards already there for the same reason. * Scan unsloth_cli on the floor as well, since the matrix runs it studio-backend-ci lists unsloth_cli/** in its own paths filter and runs pytest unsloth_cli/tests as a step on every leg, so the 3.10 leg this replaces was executing shipped CLI code on the floor interpreter, not only backend code. A lint aimed at studio/backend alone covers part of that while reading like it covers all of it, which is the same shape as the package allowlist the previous round removed, one level up. ROOTS is now both trees and targets() walks each of them, 441 files rather than 422, and the run is still clean at 3.10. The guard asserts on what the lint would actually hand to vermin rather than on its source, and dropping unsloth_cli back out fails it. * Lint the test code at the floor too, since the matrix executes it The first version dropped tests on the theory that they are not shipped. Shipping is not the question, execution is: studio-backend-ci runs pytest tests/ from studio/backend on every leg, so a 3.11 API in a test file is executed by the 3.10 leg exactly as one in a shipped module is. With the pull request down to a single 3.13 leg, that leg and this lint would both pass and the failure would arrive on the push to main, which is the gap this exists to close. Only vendored code comes out now, pinned to its own support range. 1093 files rather than 441, still clean at 3.10, so this costs nothing today and closes the hole. Putting tests back into EXCLUDE_PARTS fails the new guard. * Run one interpreter and defend the floor statically The 3.10, 3.11 and 3.12 legs are gone from Backend CI, on pull requests and on main alike. Measured on one runner over the same tree, the four legs collected the same 26,320 tests and differed by exactly one: the >= 3.12 gate on test_demonstrates_the_underlying_stdlib_regression. 3.10 and 3.11 reported 26193 passed / 127 skipped, 3.12 and 3.13 reported 26194 / 126. That is 97 runner-minutes per push to run one identical suite four times and learn the value of a single skip marker, into a queue that has been observed 195 deep, and queue depth is wall-clock for every other workflow in the repo. What the older legs were really defending is that nothing reaches for a symbol newer than the floor, which is static. scripts/lint_backend_python_floor.py now checks exactly that, on every pull request, in seconds, across 1093 shipped and executed files, reading stdlib API availability rather than syntax alone. The floor is DECLARED, as PYTHON_FLOOR in the workflow, next to where the legs used to be. Deriving it from the matrix was right while the matrix ran several interpreters and becomes self-defeating with one: a 3.13-only matrix would move the floor to 3.13 and leave the lint asserting that code written for 3.13 runs on 3.13. 3.10 rather than the 3.9 pyproject.toml declares, because 3.9 is not true today. unsloth/models/_utils.py already uses dataclasses.dataclass(kw_only) and tempfile.TemporaryDirectory(ignore_cleanup_errors), both 3.10, so a 3.9 target fails on the tree as it stands. Either the declaration or those two call sites has to give, and that is worth its own change; this lint is what made the mismatch visible rather than what hides it. The cost is stated rather than buried. A static check does not run anything, so the sys.version_info branches in sitecustomize.py, native_path_leases.py, third_party_source.py and worker.py are now covered by reading and by the lint's view of the names they use, not by execution. The guard that used to assert main still ran them asserts instead that every file carrying such a branch is inside the lint's scan, since that is the only check left on them. * Keep executing the pre-3.12 branches, and pin the ceiling by name Two from review. The first is the honest objection to a 3.13-only matrix on push as well as on pull requests: a break in a supported older runtime path that uses no newer stdlib name passes the lint and is then executed nowhere. So the branches were counted rather than argued about. Seven backend files carry a sys.version_info comparison, at 3.10, 3.12 and 3.14. The 3.10 ones were never straddled even by the old matrix, whose oldest leg WAS 3.10, so every leg took the same side of them and dropping legs loses nothing there. 3.14 is above every leg there has ever been. What is genuinely lost is the pre-3.12 side of three files, and that is small enough to keep running: a second matrix entry on 3.11, the newest version that still takes that side, running those three files and nothing else. 57 tests in under seven seconds, beside the full leg rather than in front of it, so the critical path is the full leg either way. It is not a second copy of the suite, and the four legs it replaces are still gone. The second is that asserting the sole leg is merely above the floor let 3.11 or 3.12 satisfy it, which would give up the removals-and-deprecations coverage that is the entire reason the single leg is the newest one. The ceiling is now written down and compared by name, so moving it is a decision somebody makes and defends in the same change. Both new assertions fail when mutated: pointing the full leg at 3.12 fails the ceiling test, and pointing the spot-check leg at 3.13 fails the pre-3.12 test because it would then re-test what the full leg already covers. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> | 17 天前 | |
lint: refuse a name bound twice in one scope of a changed Python file (#9618) * lint: refuse a name bound twice in one scope of a changed Python file A bad merge resolution can leave two complete top-level copies of one def, or a name repeated inside a single 'from x import a, a'. Nothing in CI catches that today: compileall parses both copies, the ruff rule set here is E9 / F63 / F7 / F82 with no F811, and the two copies usually differ only in their comments so the diff reads as normal. Adds scripts/lint_duplicate_definitions.py (AST only, stdlib only, 18 self-test cases) and two Lint CI steps: the self-test, and a sweep over the Python files a PR changes. A finding fails only when it sits on a line the PR added, so a duplicate that was already in the file does not block an unrelated branch that edits it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * lint: decide a duplicate by comparing both revisions, not by which lines moved The added-line gate answered the wrong question. A merge that inserts the FIRST copy above an existing definition reports the finding at the second, unchanged line, so the branch that created the duplicate read as pre-existing and the check passed. Scan both revisions and compare findings by identity instead, resolving a rename to its old path so moving a file that already carries one of main's duplicates is not blamed on the mover. Also cover the shapes the rule was missing: typed module constants (AnnAssign), two aliases from one module landing on the same bound name, and imports inside a class body. On the workflow side, include renames in the changed-file sweep (an ACM filter drops a renamed-and-edited file entirely) and pin the head commit before diffing, refusing to continue if it is unreachable rather than silently checking nothing. * Close four false negatives in the duplicate-definition gate A bare @property is the FIRST binding of its name, so exempting it hid two complete copies of one property in a class. Only the accessor form (@v.setter / @v.deleter / @v.getter) legitimately rebinds, and that stays exempt. An ALL_CAPS tuple target is not one binding but several. B, H, N, D = 1, 16, 50345, 128 is a shape this repo declares module constants with in 32 places, and duplicating that line rebinds all four at once. The self-transform exemption is now decided per name, so A, B = B, A still exempts both while A, B = 1, 2 exempts neither. A plain import binds the ROOT of its dotted path, not the dotted path itself. Keying on the raw optional alias missed import urllib.parse followed by import urllib.parse as urllib, which silently repoints urllib from the package to the submodule. The emitted finding identity for a plain import dropped the module it came from, so two different duplicates both read as import:None:x and compare mode charged a newly introduced one against a pre-existing counter entry. Both changed-file sweeps now use -z. core.quotePath is on by default, so a non-ASCII path arrives as "caf\303\251.py", whose suffix is .py" rather than .py; the workflow would have checked zero files and reported success, and the rename map would have failed to match its own before side. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Close four more holes in the duplicate-definition gate Four wrong answers, three of them false negatives and one a false positive. The false positive is the one that matters most for a gate: it blocks correct code, which is how a gate gets switched off. Chained assignment. `N = K = 256` is two bindings in one statement, and this repo writes it (tests/test_grouped_gemm_optional_gather_indices.py:46). Requiring exactly one Assign target dropped the whole statement, so a duplicated copy rebound both names and the scan still reported clean. Overload through an import alias. The decorator was matched by its literal text, so `import typing as t` with `@t.overload`, or `from typing import overload as ov` with `@ov`, was not recognised and the signatures were reported as duplicate definitions. The alias is now resolved from the module's own imports, and only from typing / typing_extensions -- `import types as t` with `@t.overload` is still a finding. Explicit alias collisions. Imports are now keyed on the name they BIND. The carve-out is for names the author never chose: an implicit binding takes whatever its source is called, so `import urllib.parse` beside `import urllib.request`, and `from a import x` beside `from b import x`, stay legitimate. An explicit `as` alias is chosen, so a second binding of it is always dead -- `import urllib.parse as client` beside `import urllib.request as client` was invisible while the key carried the source. `import urllib.parse` beside `import urllib.parse as urllib` still reports, which the previous spelling was written to catch. Renames from a non-Python file. Following `mod.txt` to `mod.py` back to the text file consumed its duplicates as pre-existing, but the rename is precisely what makes those definitions active code, so the destination is now treated as an addition unless the before-side path was itself a .py file. Run over the whole repo the findings are unchanged, 33 before and 33 after, so none of this widens the gate on code already here. * Close five more holes in the duplicate-definition gate Two of these were introduced by the previous commit's own import rework. Every implicit source is now remembered, not just the first. Keeping one entry per bound name and skipping the legitimate different-source case left seen pointing at the first source forever, so in from a import x / from b import x / from b import x the third was compared with a, looked like the legitimate shape again, and the exact repeat of b went unreported. Implicit bindings are tracked per source; an explicit as alias, being a name the author chose, still collides with any earlier binding of that name at all. Typing aliases are collected at MODULE scope. The whole-tree walk let an alias bound inside a function exempt module-level decorators, so a local import typing as t could excuse @t.overload where module-level t is something else. Nested statements are still descended into, since try/except around the import is the conventional spelling; function and class bodies are what get skipped. The finding identity now delimits scope from source. Concatenated, a module-level from A.m import x and a from m import x inside class A both spelled import:A.m:x, so a branch swapping one for the other had its new finding charged to the old one's counter and passed. Source is decoded by its own declared encoding. text=True decoded git show with the runner's locale, so a valid # coding: cp1252 file holding a non-UTF-8 byte raised UnicodeDecodeError out of subprocess and took the whole run down, on a file the parser and compileall both accept. tokenize.detect_encoding is the parser's own reader for PEP 263, and the direct path now goes through it too so the two agree. The changed-file sweep adds T. A path that changes type, a symlink replaced by a regular .py file, is reported as neither M nor A, so the newly active source reached the verifier as no path at all and the step passed having checked nothing. Run over the whole repo the findings are still 33, unchanged from before this PR, so none of this widens the gate on code already here. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Scan each branch, each class scope, and each accessor kind Three more, two of them consequences of the previous commit. Accessors are exempt PER KIND, not wholesale. Discarding every accessor meant a getter followed by two copies of the same @value.setter scanned clean, with the later setter silently replacing the earlier one, which is the merge damage this gate exists for. Keying on the kind keeps a getter, a setter and a deleter apart while making two setters collide. @overload stays fully exempt, since any number of copies there is legitimate, and a bare @property is still not exempt at all. Overload aliases are collected per class scope as well as per module. Skipping nested scopes fixed the cross-scope leak but left a class that does its own import typing as t with its @t.overload methods unrecognised, so valid overloads were reported as duplicates and would block CI. A class now gets the module's aliases plus any it binds itself. Control-flow branches are scanned independently. Only direct children of a module or class body counted, so two copies of one def inside a single if ENABLED: branch were never looked at, and two identical imports inside one try body likewise. Each branch is now its own scan, so branches are still never compared with each other or with the body around them: one definition per branch of an if/else, and a fallback import in try/except, stay clean. That last one finds four real duplicates already on main, a repeated from unsloth import is_bfloat16_supported inside one if __name__ block in four of the saving tests, so the pre-existing tally goes from 33 to 37 and the workflow comment now says which four and why they were invisible. They are pre-existing, so compare mode prints them and blocks nobody. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Start a nested class from the module's overload aliases A class body is not in the scope chain of a class nested inside it, so an inner class resolves an unbound decorator name in the module namespace, not in the enclosing class. Passing the outer class's aliases down exempted @t.overload in the inner class where t is whatever the module bound, and two copies of one def scanned clean. Widening the exemption is the direction that hides merge damage, so start each class from the module-visible aliases and add only its own bindings. * Scan match case bodies as independent branches A match keeps its branches in cases[*].body, which _branch_bodies never enumerated, so two copies of one def inside a single case scanned clean while the same shape under an if was reported. The alias walk had the mirror blind spot: an overload alias bound inside a case went unseen, which would read a legitimate overload pair as a duplicate. Sibling cases are still never compared, one definition apiece being the conditional idiom. Self-test cases are guarded on 3.10, since match is a syntax error on the 3.9 this tool still supports and would otherwise report a parse finding. Findings over unsloth, tests, studio and scripts are unchanged at 35. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> | 10 天前 | |
Fail CI on a new exec/eval/compile of a value that is not written out (#9999) * Fail CI on a new exec/eval/compile of a value that is not written out The two holes fixed in unsloth-zoo#1108 and unsloth#9777 were the same shape: exec of an HTTPS response body, and exec of a string built from a downloaded config. CodeQL runs on this repository with Python enabled and raises py/path-injection here, but it has never raised py/code-injection, so the class is not covered by what is already running. The rule is the blunt one: the first argument to exec, eval or compile must be a written-out string. The 72 existing call sites are recorded in scripts/exec_literals_baseline.json, each with a reason, so the gate starts green and only new sites fail. An entry carries a digest of the call's own source rather than a line number, plus a count, so moving code does not churn the baseline and a new call cannot hide behind a removed one. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the encoding when reading the workflow file tests/test_source_read_encoding.py gates every read of a checked-in file on an explicit encoding, so the file does not break on Windows the day it gains a non-ASCII byte. --------- Co-authored-by: danielhanchen <elliegouldingstuff@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> | 5 天前 | |
Studio: stop MTP forcing llama-server to a single parallel slot (#8172) * Studio: stop MTP forcing llama-server to a single parallel slot #7717 clamped --parallel to 1 whenever MTP resolved, six days after #7455 raised the default slot count from 1 to 4. Anyone serving an MTP GGUF went from four concurrent slots to one, with a logger.warning as the only signal. Measured on b10310 with Qwen3.5-9B-MTP-GGUF Q4_K_M, 8 concurrent requests: four slots with MTP is 1.97x the batch throughput of one and 1.29x four slots without MTP, so the acceptance collapse the clamp was avoiding no longer outweighs the concurrency it cost. Removes both clamp sites and the two restore paths that existed only to undo them. The --kv-unified and diffusion downgrades stay: those are real limits, and they now carry an allow-slot-clamp marker. A new stdlib-ast lint plus a launch-argv test matrix keep a silent downgrade from coming back. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten the comments in the parallel-slot changes * Catch the annotated spelling of a slot clamp `n_parallel: int = 1` in a function body is an ast.AnnAssign, which the lint never visited, so the exact regression it exists to block could be reintroduced past a green gate. Handle both nodes, with self-test and pytest cases for the annotated form, its allow-slot-clamp opt-out, and a non-literal annotated assignment. * Record llama.cpp#26031 next to the slot decision It reports concurrent MTP garbling on hybrid architectures. It did not reproduce on qwen35 or qwen35moe here, so the slots stay, but a reader weighing this code should find the open report from it. * Re-price the MTP reserve for each slot-fit candidate The reserve is not slot-independent after all: compact SWA scales its window allowance by the slot count under kv_unified, and an MLA target with recurrent KDA layers charges per slot. _base_footprint carried it at the originally requested count, so every reduced candidate was over-charged and one that really fits could be rejected, leaving the load on --fit with layers offloaded to host. _slots_that_fit_on_gpu now takes mtp_bytes_for_slots and re-prices the reserve alongside the KV and compute buffer; the base footprint drops it. Passing None keeps the non-MTP path byte-identical. Also catch a clamp spelled as an expression: min(n_parallel, 1) and a conditional with a literal 1 branch both pinned the count past a green gate. A real bound like min(n, cap) and a conditional between two live counts stay clean. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Price the MTP reserve at the candidate micro-batch, and close two lint gaps The slot re-pricing added last commit only overrode the slot count. A reduced candidate also lowers the batch floor, so its micro-batch shrinks, and compact SWA adds one micro-batch to its window allowance. The reserve kept the ubatch captured for the original request, so it was still over-priced. The callback now takes (slots, ubatch) and the fit passes the candidate pair. The lint missed two shapes it should have caught from the start. Tuple targets were only tested at the outer ast.Tuple, so `gpu_indices, use_fit, n_parallel = gi, False, 1` -- the very style load_model uses for the VRAM fit -- passed a green gate. Targets are flattened and paired positionally with a tuple value now; an unpairable right-hand side says nothing and is still skipped. And the route resolves the request into `_n_parallel` before the load paths see it, so that alias counts as a slot variable too. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Track the server-wide slot alias in the clamp lint A request that names no count resolves to llama_parallel_slots, so a clamp on that name reduces the same user-facing slots the rule exists to protect. The existing bindings are keyword arguments rather than assignments, so the tree stays clean; self-test and pytest cover both the clamp and the handoff. * Tighten the comments added after the opening pass --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> | 26 天前 | |
studio: add higgs, moss, and minimax audio models * Studio: add Higgs, MOSS, and MiniMax audio models * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten comments and source headers * Restore audio PR scope * Fix native audio review findings * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix native audio platform edge cases * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix native audio review follow-ups * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Complete native audio cancellation guards * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix native audio inventory and cancellation follow-ups * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Secure MOSS codec selection and instructions * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reduce verbose comments * Fix native audio review findings * Fix Audio model loading and managed downloads * Keep cached Audio loads available offline * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix native audio placement and inventory * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix MOSS generation and audio history * Forward MOSS language through audio worker * Fix cached native audio handoff * Use MOSS model context for audio duration * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix cached native audio handoffs * Validate native audio prompt budgets * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix native audio placement and routing * Handle resident VRAM during audio model handoff * Apply repository Python formatting * Trim native audio test coverage * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix backend CI after main merge * Fix teardown padding CI stub * Fix native audio review findings * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fail closed on unknown llama gpu order * stage the higgs tts2 audio tokenizer * move the higgs tts2 codec to the model device * require remote code consent before audio staging * hide unsupported native audio adapters * load minimax components from the selected checkpoint * use safe attention backends for moss audio * fail closed on unavailable audio tokenizers * serialize audio generation and model swaps * use the released diffusers minimax support * neutralize moss tts prompt delimiters * honor speech request contracts * allow the full minimax music frame budget * expose native audio prompt controls * stabilize native audio generation * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * preserve frame-aligned minimax durations * resolve higgs codecs from checkpoint metadata * neutralize minimax music delimiters * load approved higgs3 codecs with auth * fail closed on oversized audio metadata * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * align native audio security preflight * invalidate stale audio mode work * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * reject unsupported audio plans before download --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@users.noreply.github.com> Co-authored-by: mahiatlinux <110882203+mahiatlinux@users.noreply.github.com> | 6 天前 | |
ci: harden lockfile audits and enforce pre-install checks (#5695) --------- Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> | 10 天前 | |
Start the bottom taper under the icon so the disc still reads round (#8321) Easing from the icon centre line took the halo down to 75 percent of its sideways value at 40pt and 47 percent at 60pt, which is inside the disc a viewer actually sees, so the glow read as flattened along the bottom. The taper now waits until 50pt, inside the icon's own lower half, and runs over 30pt: 99 percent of sideways at 40pt, 80 percent at 60pt, and the icon label sits on 9.3 percent mean tint against 10.0 before. Regenerated on macOS with tiffutil, so the asset keeps the same two-page structure and sRGB profile. | 25 天前 | |
Make the notebooks smoke job reach the check it exists for (#10125) | 3 天前 | |
Make the Colab oracle tripwire able to fire, and stop blaming start.py for a hung agent CLI (#7838) * Make the Colab oracle tripwire able to fire, and stop blaming start.py for a hung agent CLI * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep a connection-mode cap fatal, and do not read 137 as a timeout * Note that the opencode non-exit is intermittent * Close three gaps the waiver opened: turn-2 side effect, attribution-ab, and the guard tests * Bound a TERM-resistant CLI again, and judge a capped turn 2 on an artifact * Gate both timeout statuses on the clock, keep the cron lint reachable, and make refresh --all atomic * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> | 1 个月前 | |
Studio: add image generation, editing workflows and LoRA training with Unsloth GGUFs (#6763) * Tighten comments in the image stack tests and scripts * Close video single-file, training reservation, and image mount-resume gaps Route on-device single-checkpoint video folders through the single_file loader: a bare local .safetensors directory (no model_index.json) is advertised as a pipeline with no filename, so validation rejected it before it could load. Reinterpret the pick as a single_file load of the sole checkpoint, mirroring the image load route. Treat a reserved-but-not-yet-spawned LLM training start as active in is_training_active() so /images/load, /video/load, and /diffusion/start cannot race the reserved run for VRAM during the pre-spawn free window. Mirrors the diffusion training service reservation. Resume an in-flight image generation on the Images page mount: probe generate-progress, re-enter the poll loop, and refresh the gallery on completion so a run started elsewhere is reflected and its saved image appears without a manual refresh. Seed resident image defaults from the resolved base_repo rather than a possibly path-shaped repo_id so the first resident generation uses the right recipe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Publish image generation active state before pre-denoise setup generate() assigned self._gen only at the pipe() call, after deferred compile, LoRA resolution/application, and ControlNet download/build had run. Across that setup window generate_progress() reported inactive even though _generate_lock was held, so a reloaded page's mount probe showed idle and let a second generate queue behind the first. Publish an active step-0 _GenState the moment the generation lock is acquired, before the setup work, and clear it in the outer finally so a setup-time error cannot leave the UI stuck active. Mirrors the video backend's queued phase and the training start guard. * Studio: fix diffusion install ownership, dataset upload atomicity, gallery pagination, and teardown races install_sd_cpp_prebuilt: only write the .unsloth-studio-owned marker when the install created the target directory or it was empty. Adopting a pre-existing, unowned, non-empty directory (a user's own stable-diffusion.cpp checkout) made it eligible for the uninstaller's recursive delete. routes/training upload: make the multi-file promotion transactional. Back up each displaced original and roll every destination back on any failure, so a mid-loop rename error can no longer partially overwrite the live dataset. routes/training _resolve_dataset_folder: reject a symlinked dataset directory and prove the resolved folder stays under the datasets root, so image read/caption/delete cannot escape the root through a link. routes/training delete: escape glob metacharacters in the thumbnail filename so deleting an image named like [ab].png removes only its own thumbnails. image_gallery / video_gallery listing: filter records against the response schema inside the pager via a valid callback, so offset/limit/has_more all count over accepted records. A leading schema-invalid record no longer returns an empty page with has_more=true and stalls infinite scroll at offset 0. image_gallery / video_gallery save: publish via a temp file plus atomic rename (the sidecar is the video pair's commit marker) and clean up on failure, so a partial write never surfaces a truncated PNG or strands an orphan MP4. diffusion_train_common discovery: treat an empty caption sidecar as a metadata tombstone that still falls through to the dreambooth instance prompt, so clearing every metadata caption no longer fails with no captioned images found. diffusion backend unload: wait for an in-flight denoise to exit before tearing down process-wide patches and state, mirroring the load path. diffusion_engine_router: serialize the whole check/unload/publish transition so a concurrent selection cannot return the engine being unloaded. uninstall.ps1: gate the default sd.cpp process stop on the owner marker so a user's own sd-server is not terminated for a directory we then keep. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reject native batch seeds outside the JSON-safe range * Refuse sd.cpp install into unowned non-empty target dir When the install target already exists, is non-empty and lacks the .unsloth-studio-owned marker (a user's own stable-diffusion.cpp checkout, or unrelated files beside a custom Studio root), install() previously still extracted the release into it. Skipping the ownership marker only stopped the uninstaller from deleting the directory; extraction still merged binaries into the user's working tree and could overwrite same-named files. Fail up front with a clear message pointing the user at a fresh/empty location before any download or extraction, leaving their directory untouched. Update the ownership test suite to assert the refusal. * Studio: gate dataset uploads on the symlink check and surface local video single-file checkpoints * Tighten comments and docstrings added by the image-generation fixes * Studio: close arbiter load-registration race and surface native progress + local pipeline folders Publish native sd.cpp generate progress (_gen) before LoRA resolution so a reload probe reads active during setup, matching the diffusers path. Register the diffusion/video GPU load under the arbiter lock (acquire_for now takes a register callback) so a competing acquire cannot evict an owner before its load is marked in-flight and let two loaders allocate VRAM at once. Admit local diffusers pipeline folders (root model_index.json, weights in component subdirs) in the local model scan so they reach task tagging and the On Device picker. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: include marker-owned custom sd.cpp roots in the uninstall stop scan; harden Pester install against the nuget.exe PSGallery bootstrap * Studio: surface local pipeline scan roots, tag single-file checkpoints by filename, mark companion-only pipelines partial - _scan_models_dir: admit a scan folder that is itself a diffusers pipeline (root model_index.json, weights in transformer/ vae/ subdirs). _is_model_directory rejects such a root, so the child scan would list the component subdirs as bogus models and hide the real pipeline; treat the root as one model via _local_pipeline_index. - _local_is_diffusers / _local_model_task: include the sole checkpoint filename in the family-detection needles (_local_family_needles, resolved via resolve_local_single_file). A generically named folder holding one loadable qwen-image-*.safetensors / ltx-*.safetensors identifies its family only from the filename; the load route already resolves that file, so tag it or the task-scoped picker (which rejects task=null) hides the on-device model. - list_cached_models: mark a companion-only base snapshot partial. A GGUF image load prefetches the base repo's VAE / text-encoder / model_index.json but skips the transformer (the GGUF supplies it); the snapshot has a pipeline manifest yet is not a loadable BF16 pipeline, and _cached_repo_partial misses it. _repo_pipeline_missing_denoiser flags a pipeline snapshot whose transformer/ or unet/ component carries no weight, so the picker drops it instead of advertising it as fully on-device. * Studio: preserve foreign gallery files, force safetensors on remote ControlNets, and close dataset/seed/GPU gaps Gallery clear/delete now scope to Studio-owned files: image_gallery and video_gallery skip PNGs / MP4s without a readable recipe (a hand-dropped or orphan file the listing already hides), so clear() and a guessed-id delete no longer destroy files the gallery never surfaced. Remote ControlNets now force use_safetensors: a bare owner/name reaches from_pretrained without the base trust gate, and the Hub scan fails open when unavailable, so requiring safetensors closes the pickle deserialization vector. POSIX uninstall now stops resident sd-server / sd-cli under an owned sd.cpp root before removing the tree (marker-gated), mirroring the Windows stop-before-delete scan; a live native server no longer survives unlinking its binary. Diffusion dataset containment: the training-start read path and the discovery picker route bare names through the protected resolver, so a symlinked dataset is rejected / not advertised like the caption/delete routes already do. Uploads gain the inference decode guard (oversized real images 400 before OOMing the trainer) and dataset upload/caption/delete/import are blocked with 409 while a diffusion run is active. JSONL readers (trainer + routes) tolerate non-object JSON and invalid UTF-8 instead of raising AttributeError / 500. LoRA family compatibility is enforced in the shared resolver, not only the picker, so a direct API client cannot apply a mismatched-family adapter. GPU arbiter gains release_if so the image/video unload idle-check and release are atomic against a concurrent same-owner load's registration. Native batch recipes persist the base batch_seed and restore replays from it, so a native batch_index>0 image no longer advances its seed twice. FLUX.2-klein selects its sd.cpp text encoder by variant (4B -> Qwen3-4B, 9B -> Qwen3-8B) instead of the single family default. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: track the loaded GGUF filename so native companion resolution reproduces the load identity * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: gate local-pipeline image tagging on a real family; validate video sidecars before delete/clear * Studio: tighten image-generation fix comments and docstrings * Studio: gate gallery serve/export on ownership; keep image progress active until persisted; reserve diffusion training before the dataset scan * Studio: restore Reapply target on async image/video load errors; recheck training state before dataset commit; reclaim partial sd.cpp installs on retry Images/Video: a background model load that fails AFTER starting (error/eviction during download) leaves the previous pipeline resident, but handleLoad had already overwritten lastLoad.current with the failed pick, so "Reapply to loaded model" reloaded the failed model. Carry the prior Reapply target into the poller and restore it on the async error/null paths, mirroring the quant rollback. Training: an in-flight diffusion dataset upload passed _require_diffusion_dataset_mutable() at entry but could still commit files after a concurrent /diffusion/start reserved the training slot, mutating the dataset underneath the trainer. Re-check the interlock immediately before the commit phase; a 409 there leaves the staged temps for the finally to clean. sd.cpp install: an interrupted extraction (disk full, killed process, a raising post-extract cudart fetch) left the target non-empty with no owner marker, so the next lazy install tripped the "not a Studio-managed directory" refusal and wedged native install. Write the ownership marker before the partial writes when the target is reclaimable, so a retry recognises the debris as ours and re-extracts. * fp8 DiT quant: floor the dynamic activation scale with activation_value_lb An all-zero activation token row makes the dynamic per-row fp8 scale 0, which turns the quantized data to NaN and the render to black frames on torchao's plain-torch kernel path. The fused fbgemm/mslk quantize kernels clamp zero rows internally, so the bug only reproduces on machines without them, which is most user environments. Zero rows are real inputs, not a corner case: Wan 2.2 zero-pads its text conditioning, and Hunyuan-1.5 and Qwen-Image regenerate zero rows inside their transformer blocks every step. Pass activation_value_lb=1e-12 to Float8DynamicActivationFloat8WeightConfig whenever the installed torchao supports the kwarg (Float8Tensor rework, 0.13+), checked via inspect.signature so older torchao keeps exactly the current behaviour; the existing Float8MMConfig fallback chain is unchanged. Verified on GPU: with the forced plain-torch kernel path a zero-row input NaNs without the floor and stays finite with it, and end to end on HunyuanVideo-1.5 fp8 goes from a solid black frame (LPIPS 1.00) to a normal render (LPIPS 0.225); on Wan the floor matches the condition_embedder exclusion (LPIPS 0.211 vs 0.206). Same-seed renders with fused kernels present are unaffected, and pre-quantized fp8 checkpoints stay valid since weight scales are untouched. * Wire hosted pre-quantized DiT checkpoints into the image families Point prequant_repos for flux.1, flux.2-klein, flux.2-dev, qwen-image (int8 only there; fp8 is family-denied), z-image and krea-2 at the unsloth/<Model>-FP8 Hub repos carrying gate-validated int8 and fp8 transformer checkpoints, so the fast quant path loads the small pre-quantized file instead of materialising the dense bf16 transformer and quantising on device. Measured on FLUX.2-dev int8: build peak drops from 60.7 GB (dense + quantize) to 30.7 GB (hosted prequant), identical 30.7 GB resident after either path since loading a checkpoint is bit-identical to on-the-fly quantisation. The hosted repos name files <Model>-<SCHEME>.pt, so resolve_prequant_source now derives that model-name filename from the repo id (scheme suffix stripped case-insensitively) and carries the legacy transformer_<scheme>.pt as a fallback the resolver tries when the primary 404s, keeping older repos loadable. Wiring a repo also exposed a fallback hazard: with a prequant source present, the dense-fit preflight used to be skipped entirely, so a failed prequant download would fall through to the dense bf16 load the memory plan never budgeted, OOMing after eviction. The preflight now always runs and gates an allow_dense_fallback flag through _load_dense_quant_pipeline: a dense misfit still skips the fast path when no prequant exists, but with one it proceeds and a prequant failure raises to the GGUF build instead of loading dense. The same flag is set when the auto-policy replans an offloaded GGUF against a prequant-sized transient. Tests updated to the new filename convention plus new coverage for the derivation and the legacy-name fallback; the prequant-skips-refit test now asserts the re-check runs and forbids the dense fallback. Verified end to end on GPU: z-image int8 resolves the hosted repo, downloads the model-name file and renders (6.8s load, 5.9 GB peak). * Route krea-2 through its per-component loader on the transformer-quant fast path _assemble_pipe used Pipeline.from_pretrained for every family, but the krea repo ships transformers-5.x configs and no top-level tokenizer files, so the tokenizer dies with vocab_file=None. The pre-quantized checkpoint loaded fine and then the assembly crashed, dropping the load to the GGUF build, which krea-2 cannot take (Krea2Transformer2DModel has no from_single_file). Assemble per-component via load_krea2_pipeline like the pipeline-kind and single-file paths already do. Verified live: Krea-2-Turbo int8 and fp8 hosted prequant loads now assemble and render through the Studio images tab. * Keep Qwen-Image's text-stream linears bf16 on int8 (short prompts break torch._int_mm) Qwen-Image's MMDiT runs every text-stream Linear at M = actual prompt tokens: the Qwen2.5-VL embeds are not padded to a fixed length like FLUX's 512-token T5. A short prompt (13 tokens) or the near-empty negative prompt drives torch._int_mm below its M > 16 floor and the first denoise step raises 'self.size(0) needs to be greater than 16, but got 13' (measured on B200 through the Studio images tab). Add per-family int8 exclusions (txt_in, add_q/k/v_proj, to_add_out, txt_mlp) for qwen-image and qwen-image-edit, threaded through exclude_tokens_for_scheme(scheme, family) and the prequant checkpoint validation, so a checkpoint baked under the old token list is rejected and re-quantised instead of loaded crashing. The text stream runs at M = tens vs the image stream's M ~ 4k, so the exclusion costs nothing; the rebuilt hosted checkpoint gates 28/28 PASS with LPIPS mean 0.057 (was 0.069). * Harden the diffusion memory plan against transient free-VRAM undercounts A cold FLUX.2-dev int8 load on an idle 183 GB B200 planned offload=model (companions exceed budget) and silently served the GGUF as-is; the identical retry went resident and engaged the hosted prequant. The plan arithmetic was byte-identical across both loads (required 90,228 MiB, resident needs free of about 124 GB); the only divergent input was torch.cuda.mem_get_info, which is device-wide and instantaneous: a transient foreign CUDA context briefly held about 100 GB at the first snapshot, and the planner trusted that single read. Three changes: - settled_snapshot_device_memory: on cuda, synchronize + empty_cache (best-effort) and take the MAX free over up to 3 spaced reads. A transient can only shrink free, so the max rejects transient undercounts while a persistent tenant still caps every read. _plan_memory now uses it. - plan_fits_total_capacity + one replan retry: when the dense/prequant candidate fits TOTAL device capacity under the standard reserve and the 0.85 resident margin, an offload verdict can only stem from the free reading, so the loader re-snapshots and replans once before declining the fast path. Explicit balanced/low_vram modes skip the retry (they offload by mode). - diffusion.transformer_quant_declined log line with required/budget/free and the plan reasons, so the next decline is diagnosable from the server log (previously silent). Verified: cold FLUX.2-dev int8 first load in a fresh server now engages the hosted prequant resident (offload=none). * Add FLUX.2 Klein and FLUX.2-dev DiT LoRA training Register flux.2-klein and flux.2-dev in the DiT trainer following the upstream DreamBooth references: latents train patchified and batch-norm normalized from the VAE posterior mode, the packed forward reuses step-invariant position ids, and the guidance vector (3.5) is gated on the variant's guidance_embeds config. Conditioning stacks load per variant (Mistral via Flux2Pipeline for dev, Qwen3 via Flux2KleinPipeline for Klein) and are encoded and freed before the transformer lands on the device. The fused single-stream to_qkv_mlp_proj joins the attention projections in the LoRA targets; the single-stream out projection stays dense because its to_out suffix would also match the double-stream ModuleList container. Wire both families through the training registry (family set, labels, VRAM notes, rank 16 / lr 1e-4 defaults, bf16-only preflight), mark them trainable with train base repos in the family registry, add FLUX.2-dev to the gated-repo token check, and trust both official bases for training downloads. Verified on B200: 30-step klein int8 (19.6s) and nf4 (20.9s) and dev int8 (52.0s) runs train with finite decreasing loss and the saved adapters apply on the bf16 base pipeline (weight 0 reproduces the base image exactly, weight 1 visibly restyles it). * Support LoRA adapters on torchao int8/fp8 quantized image pipelines Adapters are baked at load time: they attach to the dense transformer, then quantize_ converts only the frozen base linears (the lora_ side path is excluded by name), then the loader compiles. Post-quant PEFT injection is not possible on a manually quantized module, so the prequant shortcut is skipped for a baked load and the memory plan is sized for the dense build (force_dense on the quant candidate). At generation time the baked topology is frozen: weight tweaks and disabling (scale 0 reproduces the quantized base exactly) go through set_adapters, while adding or removing adapters returns a clean 400 telling the client to reload with the new selection. supports_lora now returns True for int8/fp8 diffusers loads (checked before the gguf-kind early return, since the quant fast path keeps the picker kind); nvfp4/mxfp8 and GGUF-via-diffusers stay blocked. The load request model takes an optional loras list, threaded through begin_load on both engines (native ignores it and keeps applying LoRA at generation). Verified end to end on GPU: Z-Image GGUF picker + int8 + trained adapter loads through the API, bake marker logged, weight 1.0 vs 0 renders differ visibly, weight 0.5 accepted live, unknown adapter rejected as 400. Affected suites: 296 passed. * Add FLUX.1 Krea dev to the image model catalog Krea's guidance-distilled FLUX.1-dev finetune keeps the exact dev layout, so it runs under the existing flux.1 family unchanged. Wire it up end to end: - Catalog group with the gated official bf16 pipeline and the open QuantStack GGUF quants; the gated artifact is skipped on auto-routing when undownloaded. - Trust the official repo for non-GGUF from_pretrained loads, next to the other black-forest-labs bases. - Generation defaults: 28 steps at guidance 4.5 per the model card. The generic "krea" defaults key (Krea-2-Turbo's 8-step no-CFG recipe) used to swallow the id, which would have produced garbage output; the new flux.1-krea key precedes it on both the backend table and the images page table. - The flux.1 prequant checkpoints are schnell-based; the loader's baked base_model_id validation refuses them for the Krea-dev base, so int8/fp8 requests dense-quantize instead (covered by existing prequant tests). * Resolve pre-quantized checkpoints per base variant One family entry covers several published variants whose weights differ (flux.1: schnell, dev, Krea-dev), but prequant resolution was keyed on (family, scheme) alone, so only the default base could ever be served: the loader's baked base_model_id validation correctly refused the schnell checkpoint for dev and Krea-dev bases and every such load paid the dense download plus on-the-fly quantise. Add an optional prequant_variant_repos table on DiffusionFamily as (base_repo, scheme, repo_id) triples and thread the resolved base repo through resolve_prequant_source / usable_prequant_source and their three call sites (load fast path, memory-plan probe, auto-policy candidate). A base without its own entry keeps returning the family default, preserving the existing refuse-then-dense behavior exactly. Wire the flux.1 variants: the gate-validated unsloth/FLUX.1-dev-FP8 checkpoints (built in the earlier campaign but never reachable) and the new unsloth/FLUX.1-Krea-dev-FP8. * Add the Lumina Image 2.0 family to the image catalog Alpha-VLLM/Lumina-Image-2.0 is a 2.6B single-stream DiT with a Gemma2-2B encoder and a standard 16-channel VAE, all transformers-4.x-compatible, so the generic from_pretrained pipeline path loads it as a new lumina-2 family: - Family entry (Lumina2Pipeline / Lumina2Transformer2DModel), aliased to lumina-image-2.0 / lumina-image-2 / lumina2. No bare lumina alias: Lumina-Next checkpoints are a different arch and must stay unknown rather than crash mid-load. bf16-only upstream, so the fp16 fallback stays off like z-image. - Trust the official repo for non-GGUF loads; bf16 component table entry (ships fp32, ~5.2 GB transformer + 5.2 GB encoder bf16-resident). - Generation defaults 50 steps / guidance 4.0 per the model card, and the generate call passes the card's cfg_trunc_ratio=0.25 itself (family-gated, signature-gated): the pipeline default (1.0) runs the CFG double-forward on every step and oversaturates output. - Catalog group with the single ungated bf16 pipeline artifact (11 GB resident) plus routing assertions; images page defaults row. - No GGUF artifact: none exists upstream (only finetune/LLM quants), so the dense transformer_quant fast path (GGUF-kind-only) stays unreachable for now. Offline probes of the future prequant campaign: int8 and fp8 both engage and render cleanly (fp8 LPIPS 0.11 vs bf16, int8 0.33 from 50-step trajectory drift with intact quality), so neither scheme is family-denied. * Wire the hosted Lumina Image 2.0 int8/fp8 checkpoints Gate-validated against same-seed bf16 renders (28/28 pairs per scheme, zero failures): int8 LPIPS mean 0.146 / SSIM 0.937, fp8 LPIPS mean 0.116 / SSIM 0.946. Uploaded to unsloth/Lumina-Image-2.0-FP8 following the existing checkpoint repo conventions. * Add the HunyuanImage 2.1 family to the image backend The hunyuanvideo-community diffusers mirror carries the full stack in standard layout: a 17B dual-stream DiT (32.5 GB bf16), a Qwen2.5-VL text encoder, a ByT5 glyph encoder, the 32x HunyuanImage VAE, and guider/ocr_guider components (AdaptiveProjectedMixGuidance) that diffusers 0.39 loads natively, so the generic from_pretrained pipeline path covers everything with no per-component assembly. Family notes: - The call's guidance knob is distilled_guidance_scale (there is no guidance_scale kwarg), so cfg_kwarg routes the UI value there; real CFG runs inside the repo's guider at its baked scale. Defaults follow the card recipe: 50 steps, 3.25. - 2K-native: verified live at both 1024 and 2048. - Coexists with the HunyuanImage-3.0 structured exclusion (3.0 has no diffusers pipeline and stays excluded with its stated reason). - int8/fp8 dense quantization verified live (LPIPS 0.186 both vs same-seed bf16); a short prompt does not trip the int8 torch._int_mm minimum on this arch, so no family exclude entry is needed. - bf16 component table for the memory planner: (32.5, 16.3, 0.8) GB. * Surface HunyuanImage 2.1 in the image model catalog Catalog group with the open bf16 mirror pipeline (~50 GB resident, so a bare click on a consumer card routes to the QuantStack GGUF quants, which load and render through the generic GGUF path, verified live) plus the images page defaults (50 steps, guidance 3.25 feeding distilled_guidance_scale). * Add the HiDream-I1 family to the image backend A 17B MoE DiT (16 double + 32 single layers, 4 routed experts) with four text encoders, on HiDreamImagePipeline (diffusers 0.39). One family covers the open Full / Dev / Fast repos (same arch); per-variant generation defaults follow the upstream inference recipes (Full 50 steps at guidance 5, the distilled Dev 28 and Fast 16 guidance-free). The repos name a Llama-3.1-8B text_encoder_4 in their model_index but do not ship its weights; the official example passes the gated meta-llama repo in by hand. The loader instead assembles the component from the open unsloth mirror (byte-identical weights, already inside the non-GGUF trust gate), injected at the three pipeline from_pretrained sites, with output_hidden_states matching the official example. Memory planning counts the assembled TE4: 34.2 GB DiT + 28.8 GB encoders, ~63 GB bf16-resident. * Surface HiDream I1 in the image model catalog One catalog group with the three official bf16 pipelines (Full, plus the Dev and Fast distillations as labeled artifacts) at their ~63 GB resident size, so auto-routing keeps this a datacenter-GPU pick. city96's GGUF is deliberately not wired: the GGUF path would need the same Llama TE4 assembly for very small demand. Images-page defaults mirror the backend table with the variant keys ahead of the generic hidream key. * Pin the measured HiDream quant verdict in tests int8 and fp8 both engage and render cleanly on this family, including short prompts on int8: the routed MoE expert Linears only ever see the concatenated image+text stream (M far above the torch._int_mm minimum), so no deny entry and no family exclude tokens are warranted. Assert that so a future table edit cannot silently regress the measured behavior. * Wire the hosted HunyuanImage 2.1 int8/fp8 checkpoints Verified bit-identical to on-the-fly quantize: all 1264 state dict tensors (456 quantized) dequantize equal between the loaded checkpoint and a fresh quantize_ pass, so quality matches the runtime Dtype path exactly. Same-seed LPIPS suite means (0.35 int8 / 0.28 fp8) blend trajectory divergence with this family's own run-to-run nondeterminism (identical weights and seed reproduce a 17/255 mean pixel delta through the 50-step guider pipeline); per-case hard checks pass and the drift is compositional, reviewed visually. Uploaded to unsloth/HunyuanImage-2.1-FP8. * Fix silent LoRA drop and wasted transformer prefetch on GGUF quant loads Two live-test findings on the images load path: - transformer_quant with baked LoRAs, when the dense quantized build is declined for memory or fails: the load completed as a plain GGUF with the adapters silently dropped (HTTP success, supports_lora=false after the fact) -- wrong output with no signal. The load now fails with the recovery options (drop the adapters, free VRAM, or pick a smaller model). Weight-0 adapters still count as no bake request, and the plain no-LoRA decline keeps its silent GGUF fallback. - A fresh GGUF load on a small GPU prefetched the base repo's full bf16 transformer shards (~47 GB on Qwen-Image) because the dense-quant prefetch widening only checked scheme viability, not whether the device could ever hold the candidate resident. Gate the widening on total device capacity (reserve + 0.85 margin, the plan_fits_total_capacity bar) so a card that is certain to decline the dense build never pays the download; capable devices keep the prefetch. * Fix video progress under-reporting during load and generate Two live-test findings on the video progress endpoints: - load-progress downloaded_bytes froze mid-download: the counter used scan_cache_dir, which skips in-flight *.incomplete blobs, so it sat at the last completed blob for the whole multi-GB shard pull while the disk kept filling. Count the repo's cache directory directly (completed plus incomplete blobs, snapshot symlinks skipped so nothing is double-counted). - generate-progress reported total_steps=null / fraction=0 while step advanced: the video API only carried the native total field while the image API exposes total_steps and fraction, so one poller could not work against both. Derive the image-compatible aliases in generate_progress and declare them on the response model; the native total stays for back-compat. * Wire the hosted HiDream I1 int8/fp8 checkpoints Gate-validated: all 28 per-case pairs pass per scheme (LPIPS suite means 0.291 int8 / 0.278 fp8, in the 50-step trajectory-divergence band; CLIP delta means 0.007-0.008), and the int8 checkpoint is verified bit-identical to on-the-fly quantize across all 1615 state dict tensors (1073 quantized, max abs diff 0.0). Uploaded to unsloth/HiDream-I1-Full-FP8. * Add a pre-cast text-encoder loader for the layerwise fp8 scheme The runtime text_encoder_quant=fp8 path downloads the full bf16 text encoder and layerwise-casts it in place on every fresh load. For the heavyweight encoders (LTX's Gemma3-27B ~50 GB, FLUX.2-dev's Mistral-24B ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) that download dominates load time on a fresh machine. diffusion_te_prequant.py loads a pre-cast fp8-storage state dict instead: meta-init the encoder skeleton from the checkpoint's te_class, load_state_dict(assign=True), rebuild on CPU if non-persistent buffers stay on meta, then re-apply the same layerwise cast to install the upcast hooks. The cast is a deterministic storage transform, so the loaded encoder is bit-identical to dense-load-then-cast by construction. v1 hosts the layerwise fp8 storage scheme only: its state dict is plain tensors (torch.load(weights_only=True), no pickle execution). The dynamic-compute schemes (fp8_dynamic, int8, nvfp4) build torchao subclass wrappers at runtime and are deliberately not hosted. Checkpoints validate format, scheme, component and base_model_id before use and any problem falls back to the dense download and cast. Local path overrides reuse the DiT prequant allowlist env var. Families opt in via a new te_prequant_repos (scheme, component, repo_id) field on both DiffusionFamily and VideoFamily; the field defaults empty so nothing changes until a gate-validated artifact is wired. * Inject hosted pre-cast text encoders during pipeline assembly Wire te_prequant_pipe_kwargs into the three pipeline assembly sites: the diffusion full-pipeline branch, the diffusion transformer-only and GGUF branch (where the companion TE is the big remaining download), and the shared video assembly path before the pipeline/component split. Injection is gated exactly like the runtime cast (mode normalized to fp8, device supported, family not denied), so it can never engage where quantize_text_encoders would not; the later quantize_text_encoders call re-applies the cast idempotently and keeps status reporting truthful. With no hosted checkpoint configured the call returns {} and assembly loads the dense encoder as before. * Add the pre-cast text-encoder checkpoint builder Applies the runtime layerwise fp8 storage cast to a model's dense text encoder once and saves the cast state dict with baked metadata (format tag, base_model_id, family, scheme, component, te_class, versions) in the layout diffusion_te_prequant.py validates. Resolves the encoder class from the checkpoint's config.architectures so the recorded te_class matches what the pipeline instantiates. CPU-runnable: the cast touches storage dtypes only. * Test the pre-cast text-encoder load path Hermetic CPU coverage for diffusion_te_prequant: the checkpoint filename convention, family-table resolution by scheme and component with malformed entries skipped, resolution priority (path override, hosted repo, none) and the fp8-only scheme gate, the checkpoint validation matrix (wrong format, missing state_dict, wrong scheme, wrong component, wrong or missing base_model_id) with base case folding, the local-path allowlist refusal and missing-file fallback, and the assembly injection gating (mode, hosted entry, device support, family deny, load failure, successful injection). Also pins the te_prequant_repos field on both family dataclasses and that no family ships a hosted TE checkpoint until the campaign wires one. * Fix pre-cast TE checkpoint loading and engagement reporting Two bugs found while building the hosted checkpoints: - The builder recorded torch.__version__ (a TorchVersion object) in the checkpoint metadata, so torch.load(weights_only=True) rejected every artifact and the loader silently fell back to the dense download. Record plain strings. - Re-applying the layerwise fp8 cast to an injected pre-cast encoder raised on the duplicate hook registration, making quantize_text_encoders report the engaged cast as failed (status showed no TE quant while the encoder ran fp8). _cast_fp8 now returns early when the hooks are already installed. Also corrects the LTX TE size note: Gemma3-12B stored fp32 (~49 GB), not 27B. * Wire the hosted pre-cast fp8 text encoders qwen-image and flux.2-dev (diffusion) and ltx-2 (video) now resolve a hosted pre-cast fp8 text encoder from their unsloth -FP8 repos: - unsloth/Qwen-Image-FP8: Qwen2.5-VL-7B, 16.6 GB dense -> 8.8 GB - unsloth/FLUX.2-dev-FP8: Mistral-Small-24B, 48.0 GB dense -> 24.7 GB - unsloth/LTX-2-FP8: Gemma3-12B, 48.7 GB fp32 store -> 13.2 GB Every checkpoint verified bit-identical to dense-load-then-cast (729 / 585 / 1066 tensors, zero mismatches) and smoke-tested through the real backends with the repo engagement marker. Tests cover the wired entries, the resolver filenames, builder metadata weights_only survival, and the idempotent re-cast. * Report the compute dtype on fp8-cast encoders and inject the pre-cast TE on the dense fast path Two more findings from the hosted-TE GPU smokes: - Module.dtype reports the first floating parameter, which after the layerwise fp8 cast is the fp8 STORAGE dtype. Flux2 derives its prompt embed and latent dtypes from encoder.dtype and feeds them to randn_tensor, which has no fp8 kernel, so ANY flux.2 load with text_encoder_quant=fp8 crashed at generation (pre-existing, runtime cast included). The cast now swaps in a subclass whose dtype property reports the compute dtype; forward behaviour is unchanged. - The dense transformer_quant fast path assembles companions through _assemble_pipe, which never received the pre-cast TE injection, so the hosted encoder engaged on full-pipeline and GGUF builds but not on the fast path. Threaded through like the other two branches. Verified live on B200: qwen-image (full pipeline), flux.2-dev (GGUF picker with int8 DiT prequant), ltx-2 (video backend) all engage the hosted TE, render non-black, and report text_encoder_quant=fp8 truthfully. * Key the fp8 cast idempotency on an explicit completion marker Hook presence alone cannot distinguish a legitimately pre-cast text encoder from leftover hooks after a cast that failed mid-pass, so the early return now requires the completion marker _cast_fp8 sets once the hooks are fully installed. Leftover partial state keeps failing closed. Also tolerates non-Module encoder doubles in the hook probe and the dtype override. * Extend the fp8 TE quant to HiDream's Llama text_encoder_4 The generic quantize_text_encoders pass only covers text_encoder.._3, so HiDream's HEAVIEST encoder (Llama-3.1-8B TE4, 16.1 GB bf16) always stayed dense. TE4 is assembled separately (hidream_te4_kwargs), so the fp8 path now lives there: when the requested TE quant is layerwise fp8 and the device/family qualify, TE4 prefers the hosted pre-cast checkpoint (unsloth/HiDream-I1-Full-FP8, 8.6 GB) and falls back to dense-load-then- cast; a mid-pass cast failure reloads a fresh dense encoder instead of shipping partial state. The pre-cast loader and builder gain config_subfolder/config_overrides for standalone encoder repos whose config sits at the root and whose pipeline needs forward flags (output_hidden_states/attentions). Verified on B200: bit-identity 291 tensors (225 fp8, 0 mismatches), hosted checkpoint engages through the real backend (marker + status fp8), load 24.3 s vs 48.0 s dense, LPIPS 0.133 mean over 3 same-seed pairs vs the dense-TE render (gate 0.25), non-black frames. * Correct the ltx-2 resident TE estimate to the bf16 cast size The memory plan's bf16_components_gb held 50.4 GB for the LTX text encoder, which is the fp32 hub store of Gemma3-12B (~49 GB download), not what sits on device: the pipeline loads it torch_dtype=bf16, ~24.4 GB resident. The 26 GB over-estimate pushed the auto plan toward offload on cards that fit the real footprint. Comments and the size-table test now pin the resident semantics. * Host pre-cast fp8 text encoders for four more families Round 2 of the hosted TE set, each bit-identical to dense-load-then-cast and gated through the real backend (marker + status fp8 + same-seed LPIPS vs dense TEs): - FLUX.1 T5-XXL (text_encoder_2): 9.52 -> 5.90 GB, one artifact for schnell/dev/Krea-dev (T5 shards byte-identical across all three, verified sha256). 220 tensors, 144 fp8, LPIPS 0.109. - Lumina Gemma2-2B: fp32 hub store 10.46 -> 3.20 GB (3.3x download cut). 288 tensors, 182 fp8, LPIPS 0.041. - Z-Image Qwen3-4B: 8.04 -> 4.41 GB. 399 tensors, 252 fp8, LPIPS 0.112. NOT shared with flux.2-klein-4B: klein retrained layer 35's MLP (verified tensor diff, maxdiff 0.86), so klein hosts no entry. - Krea-2 Qwen3-VL-4B: 8.88 -> 4.83 GB. 713 tensors, 460 fp8, LPIPS 0.082. The constructor-assembled krea pipeline takes the encoder directly (load_krea2_pipeline text_encoder kwarg); the loader remaps 5.x rope_parameters and re-ties weights after assign so the rebuilt encoder matches the builder's structure. HunyuanImage 2.1 reuses the Qwen-Image artifact outright: its Qwen2.5-VL text encoder is byte-identical (every shard sha256, 16,584,414,544 bytes), recorded in the new component-level base-equivalence table the checkpoint validator consults. The injection loop now covers text_encoder.._3 so a family can host several components. Live check: LPIPS 0.123 vs dense. * Report the fp8-cast compute dtype without swapping the encoder class The dtype override swapped encoder.__class__ to a dynamic subclass, which breaks transformers' kwargs-based output recording: a fp8-cast Qwen3VLModel stopped returning hidden_states and every krea-2 generation with text_encoder_quant=fp8 crashed at encode_prompt (regression from the HiDream TE4 change; caught by the krea hosted-TE live smoke). The override is now a property shadowed on the ORIGINAL class that prefers a per-instance compute-dtype attribute, so class identity is preserved and uncast instances keep the stock behaviour. The idempotency test now pins exact class identity and the uncast-sibling fallback. * Pass the calibrated distilled sigma curve to LTX-2.3 8-step runs The 22B distilled DiT was trained against ltx_core's fixed DISTILLED_SIGMA_VALUES, but the diffusers scheduler derives 8-step spacing from resolution-shifted flow matching and lands far off at every reachable mu (second sigma 0.945-0.981 vs 0.99375, tail 0.37-0.61 -> 0.1 vs 0.725 -> 0.42 -> 0). At the distilled default step count the backend now passes the list verbatim, neutralising the scheduler's dynamic shift and terminal stretch for the call (they distort even explicit sigmas) and restoring them afterwards. Other step counts and the dev/base DiT keep the scheduler's own spacing. Live-verified on B200 through the video branch backend: the scheduler holds the exact curve after an 8-step distilled GGUF generation, config restored, healthy clip. Also reword the transformer_quant resolved reason to the measured reality: quant halves resident weights and hosted checkpoints cut load time, while per-step speed is roughly bf16 parity. * Pin the fp8 weight-quantize kernel against silent MSLK switching torchao's Float8Tensor KernelPreference defaults to AUTO, which switches the weight-quantize kernel to MSLK whenever an mslk package is importable on sm90+. Measured on B200: that changes fp8 scale rounding bitwise (8/8 FLUX matrices differ, scales ~55 percent of bytes), so a box that merely gains mslk would break the hosted-prequant bit-identity invariant; the mslk path is also slower under torch.compile (opaque extern call blocks inductor's quantize fusion, FLUX.1 fp8 e2e 1.149 to 1.624 s). Pin KernelPreference.TORCH explicitly, matching current no-mslk behaviour bit for bit; signature-gated for older torchao. GPU-smoked (finite, rel err 0.037) and pinned by test. * Shift Qwen-Image training sigmas to the inference distribution Qwen-Image's scheduler skips its static shift under use_dynamic_shifting, so the DiT trainer was drawing UNSHIFTED uniform-schedule sigmas for it (mean sigma 0.50) while inference always runs the exponential mu = log 3 shift plus the shift_terminal 0.02 stretch. Add a flow_shift config lever: "auto" (the new qwen-image default) rebuilds the training sigma table through the scheduler's own time_shift and stretch_shift_to_terminal so the draw matches the inference distribution exactly (mean sigma 0.72); a numeric value applies the standard linear shift s*u/(1+(s-1)*u); 1.0 keeps the historical identity behavior and stays the default for FLUX, Z-Image and Krea 2. The model timestep conditioning follows the shifted sigma, gathered in fp32 so bf16 rounding never skews it. Also wire two opt-in levers with off defaults: cfg_dropout (per-sample empty-prompt conditioning dropout, encoded alongside the captions before the text encoders are freed) and weighting_scheme="bell" (bsmntw-style mid-schedule Gaussian loss weighting normalized to mean 1). Verified with two 80-step rank-8 bf16 LoRA runs on Qwen/Qwen-Image (identity vs auto, same seed): both converge with finite decreasing loss and produce coherent same-seed previews. Unit tests cover the exact transform, the shifted sampling distribution, per-family defaults and config plumbing. * Add LoRA EMA, a persistent conditioning cache, and aspect bucketing helpers diffusion_train_extras hosts the opt-in training extras: LoRAEMA shadows only the trainable adapter params (warmup-ramped decay, default 0.99, exported as a second adapter under output_dir/ema), PersistentConditioningCache stores latent posterior stats and caption embeddings as safetensors keyed by content hash + family + resolution, and the aspect-ratio bucketing helpers group mixed-aspect datasets into same-area divisor-snapped shapes. The DiT trainer wires the first two behind config flags that default to the current behavior: ema_decay (0 disables) and cond_cache_dir (None disables). A fully warm cache skips loading the VAE and text encoders entirely; a cache hit is bit-identical to a fresh encode, including the per-channel qwen latent normalization. Also fixes the stale _gather_sigmas call in the perf test that still passed the scheduler instead of the sigma table. * Tighten torchao configs and note the FSDP2 design for the DiT trainer nf4 loads now enable double quantization (~0.4 bits/param off the frozen base scales at no fidelity cost), fp8 training uses the rowwise recipe when the torchao build ships it (per-row scaling confines the DiT activation outliers that a tensor-wide scale collapses), and the inference quant filter gains a per-scheme GEMM-tiling divisibility floor (16 for scaled_mm, 32 for MX blocks) so one ragged Linear cannot crash the first denoise after a clean quantize pass. plans/fsdp2_diffusion_design.md records the multi-GPU design: bf16/fp8 over FSDP2 with per-block units, LoRA attached before sharding, int8 out of scope (DTensor over the quantized subclass is undefined), per-family notes. * Batch diffusion inference with per-image seeds, an inference conditioning cache, and GGUF loader fixes Batched generation: /images/generate takes a prompts list (one image per prompt, txt2img only) or a seeds list (one prompt, one image per seed); the legacy batch_size path derives per-image seeds base..base+n-1 like the native engine. Every image gets its own torch.Generator so any batch member replays alone from its gallery recipe; the whole list runs as one forward by default with OOM backoff that halves a failed chunk, and an explicit batch_size caps images per forward. Validated 10-22x over serial engines on 32-image suites with LPIPS deltas within 0.002. Conditioning cache on the inference path: UNSLOTH_DIFFUSION_COND_CACHE_DIR (the inference sibling of the trainers' cond_cache_dir, same persistent store) wraps encode_prompt so repeated prompts skip the text-encoder forward entirely; verified bit-identical outputs. Bypassed while LoRA adapters are attached; tensor-argument calls pass through uncached. Compile cache: GGUF loads fingerprint their own bundles (quant=gguf, a different compiled graph than the dense family) and batched calls register every distinct (w, h, batch) chunk shape they ran, so the heavy GGUF batched warmups (~159 s at batch 32 on 12B-class, ~655 s on 20B CFG-batched) are paid once ever. GGUF loader: strip the sd.cpp model.diffusion_model. container prefix in the single-file converter; diffusers' FLUX.2 converter KeyErrors on it and the Qwen-Image identity mapping strands the model on meta. * Correct batched seed-replay docs to match measured behavior Same-seed images at the same batch shape are bit-identical; a solo regeneration with the recorded seed matches its batched rendition up to batch-size-dependent kernel numerics (mean abs pixel delta about 2.5/255, LPIPS delta under 0.002), not bit-exactly. The previous wording overclaimed bit-identity across batch shapes. * Note that batched bit-identity assumes a settled compiled graph The first generation issued while the deferred compile is still in flight can deviate transiently (observed once on a cold fp8 build: mean abs pixel delta 0.063/255); once the graph is settled, same-seed same-batch-shape images are bit-identical across runs. * Studio sidebar: Image03/FlimSlate icons, More flyout, Train row, New pills - Images uses Image03Icon and Video uses FlimSlateIcon. - New "More" row (MoreHorizontalIcon) opens a right-side flyout on click or hover holding Video, Recipes and Export; the close is delayed 180ms so the pointer can cross the gap. Its SidebarMenuButton deliberately takes `title` rather than `tooltip`: with `tooltip` the button returns a Tooltip root and DropdownMenuTrigger asChild would hand its ref to a non-DOM node. - Dropped the "Train" section heading; Train is now a top-level row between Images and More. data-tour="navbar" moves to the surviving nav group so the product tour keeps its anchor. - "New" pill beside Images and (inside the flyout) Video, via NavBadge. * Studio sidebar: match flyout rows and New pills to the existing scales - More flyout rows dropped their sidebar-row typography and size-icon override, which fought DropdownMenuItem's own scale (text-sm, gap-2.5, px-3 py-2 and size-4 icons) and rendered oversized glyphs and text next to the nav. - New pill reuses the brand "beta" badge recipe (nav-badge font, --ui-font-scale sizing, nav token colours) rather than hardcoded 9px values. - The More row's native title tooltip (an OS box on hover) is replaced by the app's Tooltip, wrapped around DropdownMenuTrigger so both triggers compose onto the same button, and shown only on the collapsed rail like other nav rows. * Settings: pin and reorder the sidebar navigation Adds a "Sidebar navigation" section to Settings -> Appearance, above the existing profile-menu customizer, with the same drag-to-reorder + switch UI. - New sidebarNav preference: one { id, pinned } entry per navigable row (projects, hub, images, train, video, recipes, export), array order = render order. Defaults match the shipped layout, so an untouched install is unchanged. - Unpinning moves a row into the More flyout rather than hiding it, so no page becomes unreachable. New chat and Search stay fixed as actions. - app-sidebar now renders from one navRows descriptor map, so a pinned row and its flyout counterpart cannot drift; the More row appears only when something is unpinned and highlights off whatever it actually holds. - Mirrored in the backend PersonalizationCustomization: without it the model's extra="ignore" would drop the field, and because sync replaces local state with the server's copy once customization is saved, the user's pin order would reset on the next sync. The validator dedupes and back-fills like sidebarMenu but preserves the client's order, since here order is meaningful. Frontend typecheck, i18n parity and catalog checks pass; 32 personalization tests pass, including a round-trip asserting a reordered list survives a save. * Sidebar customizer: drop the Search row, skip More for a lone item - Search is reached from the top bar, so it is no longer previewed as a fixed sidebar nav row; New chat stays. - More now appears only when it would hold two or more rows. A single unpinned row renders inline in its saved order position instead: a flyout wrapping one item costs a click and earns nothing. The customizer's More preview follows the same threshold. * Sidebar settings: hide a lone unpinned tab, match New chat icon, rename Profile menu - With exactly one tab unpinned, both More and that tab are dropped, so nothing is drawn for it (previously it rendered inline). The page stays reachable by URL. - The customizer's New chat preview uses PencilEdit02Icon, the icon the real row renders; Edit03Icon was a different glyph. - "Sidebar menu" is now "Profile menu", described as the shortcuts behind your name at the bottom of the sidebar, so it no longer reads as a second name for the navigation section above it. * Tighten comments in the new sidebar and delete-guard code * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio sidebar: keep the More row highlighted while its panel is open Moving the pointer into the flyout left the row unhighlighted while the panel stayed open. The row now carries data-menu-open, added to the nav hover selector list. Not data-state: the tooltip and menu triggers both write that attribute, so whichever lands last wins. * Images: use the shared pill toggle for Create/Train and pad the panels - Create/Train was the only segmented control on its own Tabs styling. It now uses PillTabs, the same control as the model picker and Hub toggles, pinned to the header row's 34px. PillTabs takes an icon per tab, so the inline-span workaround for TabsTrigger goes away. - pt-3 on both the Create and Train panels, which sat flush against the model selector row. * Images: make the workflow picker a dropdown instead of a 7-up strip Seven workflows in a 340px rail left ~48px each, so the labels crowded and the hints were only reachable as title tooltips. The strip is now a dropdown: the trigger shows the current workflow and its hint, and each row carries its own description. A row the loaded model can't run is disabled and shows the reason in place of the hint, so the gating explains itself. Adding a workflow no longer shrinks the others. * Images: workflow icons, hint under the trigger, more top room, unclipped Train cards - Each workflow carries an icon, shown on the closed trigger and on every row. - The trigger is one line (icon plus name). The selected workflow's description moved below it, where it reads like the Field hints further down the rail. - pt-6 instead of pt-3 on both Create and Train, so the cards clear the model selector row. - The Train right column scrolls while its cards use ring-1, which draws outside the box and was clipped at the scroll edges. p-px gives the ring room. * Images: one-line workflow rows, stronger trigger fill, roomier mode tabs - Dropdown rows are icon plus name only. The selected row's description already shows under the trigger, and a disabled row keeps its reason as a title. - Trigger fill moves to the bg-foreground/[0.07] dark:bg-foreground/[0.12] pair the hub cards use, so it reads against the card in both modes. - Description under the trigger goes from text-ui-10 to text-ui-11p5. - More horizontal padding on the Create / Train tabs. * Images: drop card borders for the composer shadow, keep scrollbars inside, use app controls in Train - Cards lose ring-1 for .panel-soft-surface: the composer's shadow in light, flat in dark, matching .chat-composer-surface and the menus. - Both rails now clip (overflow-hidden) with the scroller inside, so the scrollbar can't ride over the rounded corner. Same shape video-page already uses. - Train's 9 native selects become the app Select, so they no longer open an OS-native menu, and the native file input is hidden behind a Choose images button that reports the count. - Image previews use explicit 8-10px radii: this theme sets --radius to 1.1rem, so rounded-md was 15.6px and the thumbnails read as circles. * Images: one card for controls and preview, chat sliders, wider softer shadow - Controls and preview were two floating cards; they now share one card split by a divider. The Advanced dock stays separate since it toggles. - SliderField wraps Chat's ParamSlider, so the sliders match Chat (label row with the value, full-width neutral track) instead of a green track with a spin box. All 14 call sites keep their props. - panel-soft-surface goes from 0 2px 8px -2px /0.16 to 0 4px 22px -6px /0.10: lighter, spread wider. * Images: flat Create and Train panes, hover-only scrollbars, tidier Train dataset step Both Images tabs now sit on the page background like the Hub: no card, no shadow, no bounding box. A single rule divides the controls rail from the preview canvas (Create) and from the run area (Train), and the settings and previous-runs sections read as panes rather than nested cards. Also: - Scrollbars in these panes use the existing hover-scrollbar recipe, so the thumb only shows while the pane is hovered. - Workflow rows explain themselves with a tooltip after a short hover, which also works on disabled rows, and the descriptions are much shorter. - Training images rows are name plus image count; the license stays on the example card. - The upload step loses its dashed box, the buttons match the sizes around them, and Upload only appears once files are picked. - The empty preview uses the same icon as the Images nav item. * Images: full-height panes, wider settings rail, Create/Train offset from the selector The rule between the panes now runs the whole page height (the row drops its bottom padding and each pane pads its own content), the settings rail is wider on both Create and Train, and the Create/Train switch sits further right of the model selector. * Images: put both tabs on the Hub's centered measure Top bar and content now share mx-auto max-w-1100 with px-5 / sm:px-8, so Create and Train sit at the same width and position as the Hub instead of running edge to edge. * Images: restore the top bar position, drop the panes lower under it * Images: center the mode switch, flip the arrow with the orientation, app tooltips everywhere The Create/Train switch is centered on the page instead of trailing the model selector, with wider buttons. The flip control's arrows now rotate with the orientation and its label says which way the flip goes. Every native title tooltip on the page is now the app's tooltip, so they all get the rounded surface instead of the OS box. * Images Train: plainer field text, no green buttons, columns that stop colliding - The dataset name, trigger prompt, adapter name and custom base fields now say what they are in plain words instead of leaning on example values. - Import, Upload, Back, Back to settings and Train another are outline buttons, not green ones. - Example thumbnails are landscape tiles, so photos are not cropped to chunky squares. - Settings cells get min-w-0 and the select value truncates, so a long option like the nf4 label no longer widens its column into the next one. - The number stepper sits a little further in from the field edge. - Create and Train are wider. * Images Train: roomier example cards with Import on the thumbnail row * Video: same treatment as the Images tabs - No cards: the rail and the canvas sit on the page background, divided by a rule that runs the full page height, on the Hub's centered measure. - Wider rail, chat's sliders, hover-only scrollbars. - Every native title tooltip is now the app's tooltip, including the clip cards. - Reapply and Cancel are outline buttons, the empty state uses the Video nav icon, and the clip tiles are less rounded. * Images and Video: narrower generation rail, matching Train headings Create and Video rails go from 392px to 368px. Train a LoRA and Training settings are now the same size and both in the heading font: the h2 already picks it up from the base rule, so the settings header opts in with font-heading and the weight that rule pins. * Images Train: shorter copy throughout Family notes, example descriptions, precision labels and every helper line are trimmed so they stop wrapping to three lines and colliding with the next column. The nf4 label now fits its select without truncating. * Images Train: a little more spacing between field groups * Images and Video: tighten code comments * Fix training start NameError, the load-order guard test and CPU-only diffusion tests - start_training forwards resume_source_run_id to _start_training_impl, which reads it. Without it every start raised NameError. - Restore main's anchor in the load-marker order test: the file now has an earlier `if config.is_gguf:`, so indexing the first one compared the wrong branch. - The two diffusion tests that reach diffusers now skip when it is absent, matching the CPU repo-test env. - The UI smoke finds nav rows that live in the sidebar's More flyout. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Treat a null metadata caption as no caption str(None) stored the literal "None" as the caption, so a null row counted as captioned and would have trained on that text. Also drops an unused import. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix invalid-UTF-8 500s, the flat Canny map and the dropped DiT knobs read_text raises UnicodeDecodeError, which is not an OSError, so one bad caption sidecar or video sidecar 500d the info, upload and gallery routes. A flat image now yields the all-black edge map instead of its own luminance, and the four DiT loss knobs the trainer implements are declared so model_dump keeps them. * Use the ui font-size tokens instead of raw px text utilities text-[11px] and friends ignore the UI font size preference, which the repo's font-scale contract test enforces. Same rendered size at the default scale. * Fix diffusion dataset 500s, the dropout-1.0 no-op run and the reset base pick Four correctness fixes on the training side: - The labeling grid read caption sidecars under except OSError, but a non-UTF-8 sidecar raises UnicodeDecodeError (a ValueError), so one bad file 500d /diffusion/dataset/{name}/images and the grid could not be opened to repair it. Read it as no caption, matching the info summary. - An image past Pillow's own hard limit raises DecompressionBombError, which derives straight from Exception and so escaped the upload guard's (OSError, UnidentifiedImageError, ValueError) and returned 500 instead of the intended 400. - lora_dropout accepted 1.0, which makes PEFT build nn.Dropout(p=1.0): lora_A and lora_B receive no gradient and the run saves an untrained adapter while reporting normal progress. Bound it below 1.0, matching the LLM request schema. - The train panel re-seeded the base repo on every dataset refresh because the family object identity changes on each info fetch, so an upload or caption save silently replaced the user's chosen base and the run started on a different model. Track the pick and only re-seed on a real family change. * Show the retained failure when a video page mounts after a failed job Mount-time recovery handled only phase=completed, so reloading the page after a multi-minute generation failed left an idle view with no diagnosis: the backend keeps the terminal failed record only until the next job, and nothing else survives the reload. Surface it the same way the poll does, filtering the cancelled sentinel. * Fix batched generation crashes, cache keying and unreplayable recipes Four bugs in the batched inference path, all found by review: - A mixed-prompt batch sent a scalar negative prompt against a prompt list. Z-Image asserts on the length, and Qwen-Image, Krea 2 and FLUX true-CFG encode a batch-1 negative against batch-N latents and fail in the transformer's text/image concat. Broadcast it to match the batch. - The FBCache step-cache reset sat above the chunk loop. diffusers only resets that state at the end of a successful call, so a forward that raised (the OOM the backoff is meant to recover) left its own residual behind and the halved retry died on a shape mismatch. Reset before every forward instead. - The conditioning cache keyed on the checkpoint alone, but a GGUF or single-file load takes its text encoders from the companion base, so the same checkpoint against a different base reused the previous base's embeddings. Key the base too. - Gallery records stored the base seed and the requested batch size even when a prompts/seeds list drove the run, so restoring the second image of seeds=[5, 99] replayed seed 5. List-driven outputs now record as single-image recipes on their own seed. Also bound strength above 0: every img2img pipeline derives its step count from it, so 0 leaves zero denoising steps and either raises or, on SDXL, crashes on empty latents. * Fix quantized-load LoRA bake, prequant family exclusions and outpaint canvas Six review findings across the Images page and model scanning: - The quantized (int8/fp8) load path can only attach LoRA adapters before quantization, but the frontend load request had no loras field, so every generation after such a load was rejected and each reload repeated it. Send the selection with the load. - build_prequant_checkpoint passed no family to the scheme exclusions while recording the family in metadata, so a Qwen int8 artifact baked the short-M text-stream linears and was then rejected wholesale by the loader's family-keyed check. - Registering a bare single-file checkpoint directory produced no On Device row even though the images loader can load it; only its parent worked. Admit that shape when nothing else matched. - Unload left the Reapply target set, so the repair path was skipped and Reapply reloaded the ejected model. Clear it, as the video page does. - Both FLUX.2 bases were trusted for training but not inference, so Deploy to Create rejected every FLUX.2 adapter. - Outpaint allocated the grown canvas before downscaling, exceeding the browser canvas area cap on a large photo; an over-cap canvas is unusable, so Extend silently posted a fully transparent image and mask. Scale the source first. * Send the picked GGUF filename with the quant so diffusion loads fire The variant expander emitted only the quant label, and nothing else in the frontend set ggufFilename, so the Images and Video pages could never take their GGUF branch: both gate it on meta.ggufVariant and meta.ggufFilename, then fall through to the single-file path, which returns because the id is a repo id and not a .gguf name. Every quant pick was a silent dead click, with no load request reaching the backend. The filename was already on the variant row (the picker keys its list on it, and the variant validator requires a non-empty string), so thread it through the click handler. The chat path is unaffected: it reads ggufVariant and never needed the filename. * Version the conditioning cache key and reject non-finite flow_shift Two correctness fixes: - The cache keyed the checkpoint and its companion base by name only, so a Hub repo advancing to a new commit, or a local directory updated in place, kept returning embeddings from the previous text encoder. Pair both with a revision marker: the locally resolved commit sha for a Hub repo, config plus text-encoder file stats for a directory. Neither loads the encoders, so a warm run still keeps them off the GPU. - flow_shift only checked positivity, but JSON accepts 1e309, which floats to inf, and inf <= 0 is False while NaN fails every comparison. The sigma table then evaluates s * u / (1 + (s - 1) * u) as NaN, which poisons every sampled sigma and saves a corrupted adapter while progress looks normal. Require a finite value. * Keep curated models listed, guard the video companion repo, pin diffusers Three review findings: - The picker filtered every catalog member out of Recommended and Hub search on the way to canonical group rows, but nothing renders those rows yet (catalogGroupFitsDevice and groupMatchesQuery are imported and unused). A task-scoped picker's models list is catalogToModelOptions(), i.e. group members exclusively, so both lists came back empty and no curated model could be discovered or downloaded. Keep the artifacts listed until the grouped UI exists. - The video delete guard compared only repo_id, so deleting the companion base of a loaded GGUF video model was allowed even though it supplies the VAE and text encoders. Compare base_repo too, matching what the images guard already does for its companions. - diffusers was declared unversioned while the diffusion stack requires 0.39 (Krea2Pipeline, the cache_context child registries, the Flux2 and Z-Image pipelines), so an upgrade could keep an older release and selecting an advertised model failed until the user upgraded by hand. * Namespace the trainer conditioning cache per checkpoint, bound the learning rate - The trainer keyed its persistent conditioning cache on family and resolution only, while the keys themselves carry just the caption or image content and crop variant. One cache directory reused for two checkpoints, or for the same repo at a new revision, let a warm run skip loading its encoders and train on the other model's embeddings and latent statistics. Namespace on the base checkpoint and its resolved revision as well. The revision helper now lives beside the cache in diffusion_train_extras and the inference wrapper delegates to it, so the two cannot disagree about what counts as the same source. - The diffusion learning rate only checked positivity, but 1e309 floats to inf and satisfies gt, so the route evicted the resident models and started AdamW with an infinite rate: the first step destroys the adapter while progress looks normal and the result is saved. Bound it below 1.0, matching the LLM schema, which rejects inf for the same reason. * Fix GGUF image model picks doing nothing, and pick the train base in the top bar The quant rows never forwarded the .gguf filename, so every hub GGUF pick on Images/Video fell through to a silent return. On Train the top bar now picks the training base instead of a generation model, which is GGUF-only and untrainable. * Pin diffusion and video loads to the live HF cache root Both read huggingface_hub's import-time HF_HUB_CACHE, which changing the cache folder does not update: progress counted the old root while the download wrote to the new one, and from_pretrained could split one model across both. * Add the diffusion download plan endpoint Reports the repos and exact files a pick needs so the download manager can stage them with the loader's own file scope. A plain snapshot would add the packaged root single, transformer shards and fp16 twins the loader never opens. * Add a file-scoped flavour to the Hub download job Lets a consumer that reads a deliberate subset of a repo stage it through the normal download manager. Keyed as "@scope" so it never collides with a quant or with the repo's full snapshot, and the file list rides the registry so an XET to HTTP retry respawns the same scoped job. * Stage image and video downloads through the Hub download manager They downloaded inline inside the load, so they had none of the manager's disk preflight, manifest verification, resume or panel progress. Picks now stage as scoped jobs carrying the loader's own file list, then load from a warm cache. * Fetch staged GGUF checkpoints as scoped jobs, and stop calling diffusion models unsupported A GGUF entry went out as a full snapshot, whose ignore list drops *.gguf: the job finished at once having fetched only docs, and the repo landed on device unloadable. Every entry is scoped now. The Hub also no longer tags image/video models as unsupported (they run on their own pages), and those pickers name what they select. * Apply the picker task filter to local model sections LM Studio, ./models and custom-folder rows ignored it, so the Images picker listed chat GGUFs that 400 on a diffusion load. The backend already tags every local model with a task for this purpose. * Route a chat pick of a diffusion model to the Images or Video page Chat cannot load one, so it was either hidden or failed on load. The unfiltered picker now lists on-device diffusion models and navigates to the page that runs them, passing the repo and quant so that page loads it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Route the real GGUF filename, keep non-GGUF curated models, key scoped downloads by file set Five review findings, four of them ways a click did nothing or fetched the wrong thing: - A chat pick of a diffusion model routed ggufVariant (a label like Q4_K_M) in the search param the target page uses verbatim as the GGUF filename, so the load asked for a file that does not exist. Route ggufFilename; no filename means a curated non-GGUF pick, loaded as a pipeline. - The task-scoped pickers kept only GGUF repos, so the catalog's bf16, bnb-4bit and single-file fp8 artifacts could not be discovered or downloaded on the Images and Video pages even though loadSpecFor knows how to load them. Keep curated artifacts whatever their format, in Recommended and in Hub search. - Both pages deduplicated routed selections on the model alone, and they now stay mounted, so picking the same repo again -- another quant, or the same one after chat evicted it -- returned early without loading or clearing the query string. Key on model and quant. - Every scoped image download shared one @diffusion job key regardless of the requested files, so switching quant mid-download adopted the running job: the UI waited on the first file set, then loaded a file that was never fetched. Include a digest of the file set in the key. - A scoped plan silently dropped requested files missing from Hub metadata, and snapshot_download succeeds when an allow pattern matches nothing, so the job reported completion and triggered a load with required files absent. Fail the job instead. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the scoped download key derivable, and stop the hidden page hijacking a route Four review findings, the first a regression from my own last commit: - Keying scoped download jobs by a digest of the file set broke the download manager: it builds that key client-side (it polls and cancels before any response tells it a key), so it watched and cancelled a key no worker owned and never fired its ready callback. Keep the derivable "@scope" key and refuse the second request instead when a live job on the slot is fetching a different file set -- decided inside the registry claim, under the lock, so a concurrent claim cannot slip past it. The manager records the file set on the job as well, so a sibling quant's transfer is not adopted locally either. - Both diffusion pages read the route query through a loose useSearch and both stay mounted once visited, so the hidden one consumed the other's ?model=: it navigated back to its own route and tried to load, say, an image checkpoint as a video model. Only the visible page consumes it. - The staged download plan was built without the configured HF token or the Advanced values the load itself sends. The token matters most: the backend's Hub metadata lookup is best-effort, so a gated base silently planned no companion entry and the load pulled those multi-GB files inline, outside the manager. The memory/quant controls decide whether the base transformer/ shards are needed at all, and the route dropped memory_mode, cpu_offload, the prequant path and the LoRA selection before asking for the plan. - The video preview kept playing after leaving the page: the keep-alive layout only hides it, and display:none does not pause a media element, so a clip the user unmuted kept its audio going over the next page. Pause on the active transition and do not auto-replay while hidden. Also completes the hand-built request bodies in the hub download tests: the scoped-files field this branch added to the route read as an AttributeError against them, failing five tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Serialize the GPU handoffs, gate DiT training on a GPU, and keep 3.9 installable Six review findings, three of them evict-then-fail orderings: - The chat load reclaimed the GPU without telling the arbiter it existed. A chat load holds no llama-server process until its GGUF has downloaded, which is minutes, so a competing Images/Video acquire in that window found nothing to cancel, took the GPU, and the chat load then spawned onto the same device. It now registers an in-flight marker through acquire_for's register hook (under the arbiter lock, as the image and video loads do), the evictor cancels a marked load, and the route undoes itself if ownership moved while it loaded. - The Hub-download conflict check ran after that handoff, so a GGUF the download manager already owns destroyed the resident Images/Video pipeline and then 409'd, having loaded nothing. It moves above the handoff, together with the marker it handshakes with. - The image load released the engine router's transition lock before registering the load, so a second load choosing the other engine could unload the still-idle engine this one captured; the load then landed on a deactivated engine, where generate, status, unload and the arbiter's evictor can no longer reach it. Registration now happens under that lock and refuses if the engine changed. - Training a DiT family on a host with no GPU was accepted: nf4 is not a CPU fallback, its 4-bit load goes through bitsandbytes, which requires CUDA, XPU or MPS. The start unloaded the working Images pipeline, pulled the text encoders, and only then died in the child. Rejected before the teardown now, and /info stops advertising a precision that always 400s. SDXL keeps its documented fp32-on-CPU path. - Both diffusion pages kept the routed-pick marker forever, so re-picking the same checkpoint (after chat evicted it) neither loaded nor cleared the query string. The marker is released once the query is gone. The Images key also carried a stray NUL byte, which made the file read as binary to grep and other tooling. - diffusers dropped Python 3.9 in 0.38, so the unconditional >=0.39.0 pin left pip no candidate at all on 3.9 and made every install that composes the huggingface extras unresolvable there. The floor is conditional now. Also fixes tests that were already red on the branch: two hand-built request fakes had gone stale against fields this branch added, and the handoff-ordering test only failed on a host with fewer than two GPUs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the GGUF variant contract test against the merged handler signature The assertion pinned the exact single-line call handleVariantClick(v.quant, v.downloaded, expectedBytes, v.filename), but the handler takes (quant, filename, downloaded, sizeBytes) and prettier wraps the call across lines, so the mandatory repository test job failed on every push. Match the call structurally and assert the filename really is forwarded in the handler's argument order. * Stop a background page and a stale record taking the GPU or a download with them Five fixes from a review pass over the diffusion work. delete-finetuned rmtree'd a model the Images or Video engine was holding: every guard on that route is chat-only, and Images loads any local path, so deleting a local diffusion model under the storage root pulled the weights (and the companion VAE / text encoders sd.cpp re-reads each generation) out from under a live pipeline. The cached-model route already refuses this; the trained/exported one now does too, matching by path rather than repo id, and failing open on a chat-only install so it cannot block ordinary deletes. A staged download finishing while its page was hidden loaded the model and evicted whatever the user was actually using: both diffusion pages stay mounted behind the router and a load takes the GPU unconditionally. The pick is now held until its page is on screen again, which is also what chat does. A scoped download could report success having fetched nothing. With Hugging Face metadata unavailable no manifest is written, so verification is a no-op, and snapshot_download returns an existing snapshot folder without downloading when its own repo_info call fails. A repo already on disk from a full snapshot job (which ignores *.gguf) therefore completed with no weights and auto-loaded against them. The requested file list needs no network, so it is checked against the disk directly. The XET to HTTP retry reclaimed the job slot without the scoped file list, and that claim overwrites the stored record, so a later identical scoped start compared an empty list against the real one and 409'd instead of adopting the running download. The DiT accelerator gate probed torch.mps.is_available(), which only exists from torch 2.5 while the supported floor is 2.4. All three probes shared one try/except, so on torch 2.4 the AttributeError read as 'no block' and a CPU-only host still evicted the resident pipeline, downloaded the encoders and died in the child. Each accelerator is probed on its own now, through torch.backends.mps. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stage what an LTX-2.3 load reads, keep a routed file's load kind, drop an unbakeable LoRA Three from the latest review. The video download plan always asked for the wide base file list, so an LTX-2.3 pick staged the 2.0 base's VAEs, vocoder and connectors that the checkpoint supplies itself, while the companion files the 2.3 assembly does read were left out of the plan and pulled inline at load, outside the panel's progress, cancel and disk preflight. The plan now recognises a 2.3 pick by name (the load keeps the authoritative header probe, and under-guessing only falls back to the load-time pull), narrows the base list, and stages the extras in the same entry as the checkpoint so one repo stays one scoped job. A pick routed from the chat picker arrives as ?model= and ?quant= with no picker metadata, so a bare local .gguf or .safetensors was loaded as a pipeline: an explicit model_kind wins over the backend's filename sniffing, so it evicted the resident model and then failed on the missing model_index.json. Both pages now derive the load kind from the path, the same way their own picker handlers do. A torchao int8/fp8 build takes adapters only at load time. Switching artifact inside one family keeps the LoRA selection, since the family did not change, but the load did not bake it, so the next generation was rejected with 'reload the model with the adapter selection' while the picker still showed the adapter as active. The selection is now dropped once per resident build, with a message saying to pick and load again. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cancel an evicted safetensors load, spare the arbiter for CPU-only chat, fetch clips lazily Four fixes from the latest review round: - The GPU arbiter's chat evictor only cancelled the llama.cpp side. The orchestrator publishes active_model_name once its worker reports success, so an in-flight safetensors load was visible only as an entry in loading_models and finished onto the GPU after ownership had transferred. Cancel every pending load, and give the safetensors branch the post-load ownership recheck the GGUF branch already had. - A manual gpu_layers=0 GGUF load runs on the CPU with the GPUs hidden from the child, yet it took the arbiter unconditionally: it cancelled a running image or video generation for a model needing no VRAM, then held CHAT ownership so the next GPU workload unloaded it for nothing. Gate the acquire on the same predicate the launch-time CPU-only mask uses, as the image and video loaders gate on their resolved device. - The staged-download hook subscribes per repo, not per job, so another job on the same repo advanced the staged queue (starting a load whose scoped files were still downloading) or wiped a queue that was still running. Compare the variant each callback carries, like the chat page's auto-load does. - The video gallery fetched every record of a page into an object URL that lives until the page closes: 50 clips at tens to hundreds of MB each, for cards the user may never scroll to. Fetch a clip as its card nears the strip's edge, plus the selected one the player needs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim the comments across the diffusion backend Comment-only pass over the Python this PR touches: drop what the code already says, collapse multi-line explanations that still read on one line, and keep the reasoning that is not recoverable from the code. No code, docstring semantics or behaviour changes; verified with an AST comparison against the previous revision, and the backend suite is unchanged (same 37 environment failures as before: the API integration tests that need a live keyed server, the flash-attn install hooks, and the GPU memory fields). * Invalidate latents on a VAE swap, keep a cut-off generation, surface the EMA adapter - source_revision() scanned the checkpoint root plus text_encoder/tokenizer but not vae, so swapping or fine-tuning the VAE in place left the conditioning cache namespace unchanged and a warm run trained against latents from the old checkpoint. Include the vae directory, like any other component the cached tensors come from. - /images/generate answers only when the images are saved, and secure mode's tunnel caps an origin response near 100 seconds, which a native CPU or a high-step run passes routinely. The page reported failure while the work kept running, and a retry would duplicate it. A lost response (fetch rejection or a gateway status the origin never answered) is now told apart from a refusal: the page waits out generate-progress and reloads the gallery, so the run it started still lands. - The trainer emits the EMA adapter's path with the terminal event, but the state update dropped it, so neither the run history nor either response schema carried it and an enabled EMA left nothing discoverable. Keep it, and show it next to the primary adapter. - weighting_scheme advertised a choice of timestep sampling; sampling is always logit-normal and the flag only selects the bell loss weights. Describe what it does. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Hide unloadable cached rows, hold the dataset interlock, bound a GIF export - The cached-model listing tagged any repo with a model_index.json as text-to-image, so a community pipeline the image loader's trust rule refuses still got a row in the Images picker, and a detected-but-untrusted video repo fell through to that same tag. Gate the image tag on the load path's rule and hide an untrusted video repo outright. - A routed diffusion pick only carries a GGUF filename, which is all the chat picker has, so a curated single-file artifact arrived with no quant and was loaded as a pipeline: from_pretrained on a repo with no model_index.json. Pass the page's own catalog spec into the route pick, so a routed pick resolves to exactly what a direct pick on that page resolves to. - The dataset mutation endpoints checked is_active() and only then handed their filesystem work to a thread, so a start reserving in that gap changed captions or removed images underneath the preflight or the running trainer. The interlock is now registered for the whole request under the lock reserve() uses, and a start refuses while a mutation is open rather than waiting on it. - GIF export held every kept frame as a paletted image before encoding; a clip may be 2048x2048 for 1024 frames, and at the 12 fps target the step is 1, so one export click could allocate over 4 GB and take the backend down. Downscale past 720 px and widen the step to keep at most 300 frames. - seed accepted any Python int, so an out-of-range one passed every preflight, evicted the resident models, spawned the trainer and only then died in torch.manual_seed. Bound it to torch's 64-bit range in the request and config. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the accelerator probes in the DiT family-metadata tests Six tests read family_train_infos() (or a start preflight) without pinning the host probes, so they only held on a machine with a bf16 accelerator: on a GPU-less runner the DiT gate empties precision_modes, turns supports_compile off, and replaces any other preflight message with the no-accelerator note, and all six failed there. A conftest fixture pins both probes for exactly those tests, so they assert the family metadata they are about on every host. The gate's own CPU-only behaviour keeps its dedicated tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Carry the pipeline task into the hub inventory the pickers read - The task-scoped pickers filter On Device rows on a task, and the chat picker routes a diffusion pick by the same field, but those rows come from the /api/hub inventory, which never carried one: the Images and Video pickers listed nothing on device and the routing never fired. Both cached scans and the local listing now tag rows with the classifiers the models API already uses, the schemas and the frontend adapter carry it through, and a row the backend classified as a generation task is exempt from the chat-only guard that was also dropping it. - The local routing map was keyed by model_id while the row click passes id (a filesystem load id for a models_dir or LM Studio entry), so the lookup missed and the pick fell through to the chat loader. Key both. - A staged download whose start answered "error" left its head in place, where the effect never re-runs and onReady never fires, so the pick was stranded until the user reselected. Clear the queue and say so. - Every scoped pick in a repo shares the @diffusion variant, so the variant alone cannot tell two file sets apart: restaging while the first job finished let its completion pass for the new pick and load a checkpoint that had not downloaded. Bind the callbacks to the repo + file set they started, and to the staging generation. - A rejected generate POST does not say whether it reached the backend, so an immediately idle progress read was ambiguous and a submission that never landed looked like a finished image. Require evidence: progress seen active, or a gallery record that was not there before the POST. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the OpenAI image URL fetchable, keep WebM audio, stream example imports Four review items on the diffusion Studio work: - response_format=url returned the bearer-gated gallery route, which a standard image client downloads with no Authorization header, so the default response format was unusable. Mint a short-lived HMAC link instead (the shape RAG already uses for pdf.js) served by a signed route, and leave the gallery route itself bearer-only. - A manual gpu_layers=0 load carrying speculative_type="off" -- a value the UI persists and sends -- read as GPU-bearing, so it took the GPU arbiter and evicted a resident image/video pipeline even though the launcher hides the GPUs for it. Canonicalize the mode and exempt "off". - The curated example import prepared the whole split before the loop stopped at the 10-100 image cap; m1guelpf/nouns is 49,859 rows / 328 MB. Stream instead, with the prepared load kept as a fallback for a repo that cannot stream. - WebM export dropped the audio track an LTX-2 clip carries, silently, on the format offered for web embeds. Mux it as Opus through a resampler + FIFO, and keep exporting the video alone on a build without libopus. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not stub out triton on a GPU host when the Xet backend fails to import The lazy loader retries `import unsloth_zoo.hf_xet_fallback` under UNSLOTH_ZOO_DISABLE_GPU_INIT=1 whenever the first attempt raises. That flag makes unsloth_zoo take its MLX/CPU path, which injects triton and bitsandbytes STUBS into sys.modules for the rest of the process. On a working GPU box whose first import failed for an unrelated reason (a bitsandbytes/CUDA mismatch, say) the retry succeeds, so Studio boots looking healthy and then dies at the first CUDA-only kernel: a GGUF or compiled diffusion generation hits the stub and returns NotImplementedError: Unsloth: 'triton.tools.experimental_descriptor.enable_in_pytorch' was called on Apple Silicon / MLX, where triton is stubbed out. so every image generation 500s with an Apple-Silicon message on a Linux CUDA host, while the load reports success. Found by loading Z-Image-Turbo GGUF through the API on a box where bitsandbytes could not initialise. Gate the retry on the host genuinely having no accelerator. The Xet stall watchdog is optional and already degrades with a warning; a process whose triton is stubbed out is not recoverable. The warning now says why it did not retry. * Fix the lost-generation proof set, the settle timeout and the hub inventory's diffusion gates Seven fixes from the latest review round on the Images page and the hub cache inventory. Images page: - The lost-POST settle path built its "already seen" gallery id set inside the catch, after the request failed. By then the earlier runs of the same batch had already prepended their records, so run 2 could accept run 1's image as proof that its own request reached the backend. The set is now captured once before the first POST and grows with every record the batch produces. - settleLostGeneration fell out of its SETTLE_MAX_MS loop and returned normally, so a wedged generation was counted as done and the next run started against a busy backend. It now throws on timeout. - Restoring a recipe cleared the ControlNet selection but left the workflow tab and the init / mask / reference images pointing at whatever was loaded, so the next Generate conditioned on an unrelated image. It now clears all of them and returns to Create. - The download plan omitted the adapter selection the load itself bakes in. A baked LoRA forces the dense build path, so the plan described a different file set than the load that followed and the rest was pulled inline, outside the download manager. Both now derive the list from one helper. Hub cache inventory: - A download for a repo an Images or Video load is staging was allowed to start: only the llama.cpp loader was consulted. Both diffusion backends already expose loading_repo_ids for the delete guard, and the download guard now reads them too. - A companion-only prefetch (pipeline manifest plus VAE and text encoder, no transformer) passed the snapshot-partial check, since every file its manifest expected did arrive, and was advertised as on-device although from_pretrained cannot load it. - The single-file flag never reached the picker through the hub inventory path, so a checkpoint-only diffusion repo read as a full pipeline and failed after the handoff. The two pipeline-shape helpers now live in hub/utils/inventory_scan.py so /api/models/cached and the hub inventory classify the same repos the same way. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop adopting an unknown scoped download, leaking raced blobs and resurrecting deleted clips Three items from the latest review round. A scoped download job carries a deliberate file subset, and every file set of one repo rides the same "@scope" slot. A client that adopts a live job from the backend had no file list to compare against: the active-downloads response never carried one, so an adopted job's set was unknown and any later scoped request for the same repo read as "already started". Selecting a different checkpoint then waited on the wrong transfer and tried to load a file nobody fetched. The response now publishes the scoped file list, adoption records it, and an unknown set no longer satisfies a scoped request. A gallery record can be deleted while its blob is still downloading. The delete revokes the URL present at that moment, so the fetch that lands afterwards inserted a fresh object URL for a record no card renders and nothing can revoke: a full MP4, tens to hundreds of MB, pinned for the rest of the session, and once per raced fetch. Both galleries now discard a blob whose record went away, with an epoch covering the video page's Clear all. The video backend keeps the last completed job until the next one starts, and the Video page merges that record on mount to cover a job that finished after the gallery fetch. Deleting the clip left the record in place, so every reload prepended a ghost card whose file request 404s until another generation replaced it. Deleting the clip, or clearing the gallery, now clears the matching terminal record, and the page skips a record it deleted itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Resolve revisions from the live cache, serialize dataset imports, and stop pinning every gallery blob Five more items from the review round. The conditioning-cache revision marker read huggingface_hub's import-time HF_HUB_CACHE constant. Studio can move its cache during a session and loading follows the live setting, so after a move the marker went unresolved (or pointed into the previous root) and pulling a new revision of the same checkpoint no longer invalidated the cache: a warm run could reuse the old encoder's embeddings and the old VAE's latents. It now looks in the active Studio cache first and keeps the environment and the library constant as fallbacks, which the trainer subprocess still needs. The dataset interlock counts mutations rather than excluding them, so two imports of different examples into the same empty name both got past the emptiness check. The winner promoted its staging directory atomically; the loser found the folder non-empty, fell back to a per-file move, and merged its images and captions into the winner's dataset. Imports now take a per-folder lock, a second one is refused with 409, and the emptiness check is repeated under the lock. On Windows the sd.cpp asset resolver filtered only by accelerator token, so a Windows arm64 host matched an x64 zip, downloaded and installed it, and failed later when the binary would not run. It now filters by architecture the way the Darwin and Linux branches do. Every gallery page fetched every PNG up front and kept the object URL for the session, so scrolling a large gallery grew memory without bound for tiles the user may never look at. The Images strip now fetches a tile as it nears view, like the Video strip, and keeps the eager path only where IntersectionObserver is unavailable. A 503 carrying a JSON body comes from the application, not a proxy, so it is surfaced as the error it is instead of entering lost-response settlement and being reported as a request that never reached the server. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the class of a failed generation instead of a bare "Image generation failed." Found on a macOS runner: the native renderer aborts inside its own text encoder there, and the page showed only "Image generation failed." with the sd-server backtrace left in the server log, so nothing about the failure reached the user. The failure is now classified into fixed text, out of memory and native-process death, so the message says what happened and what to try. None of the engine's own output is echoed, since a native tail carries local paths and argv; that stays in the log, and an unrecognised failure keeps the original literal. * Treat an undecodable caption sidecar as the tombstone the trainer sees Uploads store .txt and .caption sidecars as raw bytes, so one can hold invalid UTF-8. The trainer treats any existing sidecar, decodable or not, as an empty tombstone and never falls back to the metadata row for that image. The labeling grid and the dataset summary read an undecodable sidecar as absent instead, so both showed a metadata caption that the run would silently replace with the instance prompt, and counted the image as captioned. Both now track sidecar presence separately, so what the user reviews is what the run trains on. * Keep the reason a native server died, not just its backtrace A ggml abort prints its cause first and then a stack trace, so reporting the last twenty captured lines gave twenty addresses and nothing about the failure: on the macOS runner the native server died on an unimplemented Metal op and the message carried only frame pointers. The captured tail now leads with the lines that name a cause and keeps recent context after them, for both the startup failure and the mid-request death. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop staging the dense text encoder for an fp8 video load Two halves of the same gap, found while measuring the LTX-2.3 download plan: - The video download plan and the scoped pre-download never saw text_encoder_quant. An fp8 request loads a hosted pre-cast encoder, so asking for one still staged and downloaded the base repo's dense Gemma3 (48.79 GB of Lightricks/LTX-2 on the 2.3 distilled pick) that the pipeline then never opened. The plan now drops those shards and stages the pre-cast checkpoint instead; their configs stay, since the pre-cast loader still meta-inits the encoder from the base repo's component config. - The LTX-2.3 assembly builds every component itself, so pipe_kwargs (which carries the pre-cast encoder for from_pretrained) never reached it and an fp8 request silently loaded the dense encoder anyway. It is passed across explicitly now. The dense skip is earned, not assumed: only a pre-cast checkpoint that resolves on the Hub lets the plan drop the dense shards, and only one already fetched to disk lets the pull drop them, so an unpublished or gated artifact leaves both exactly as they were. If injection still fails after that, the load tops the dense weights back up rather than handing from_pretrained a snapshot with no encoder in it. Measured against the real Hub on the 2.3 distilled Q4_K_M pick: 67.24 GB before, 18.92 GB with a 0.43 GB stand-in for the pre-cast artifact (the base entry drops from 24 files / 48.79 GB to 13 files / 0.04 GB). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Match the file's typing idiom for flow_shift models/training.py annotates with typing constructs throughout (105 Optional[...], no Union), and flow_shift was the one place using PEP 604. Union[] reads like the rest of the file, and it also drops the runtime evaluation that would raise on Python 3.9. * Bound the gallery blob cache, and three interlock fixes Four review findings, all reproduced first: - The gallery object-URL caches were unbounded. A clip runs from a few MB to a few hundred, both pages stay mounted after their first visit, and entries were only dropped on delete, so scrolling pinned everything for the session. Both pages now share a byte-budgeted LRU (512 MB video / 192 MB images) keyed off the visibility signal the near-viewport fetching already provides. On-screen media, the selected clip or image, and the item just fetched are never evicted, so eviction is invisible and a single item larger than the whole budget cannot evict itself into a refetch loop. - The image, video and chat load guards ran two independent training probes but returned early when the FIRST one raised, so an unreadable LLM backend disabled the diffusion interlock and a load could proceed straight into an active diffusion trainer on the same GPU. The probes are independent now. - An engine switch swallowed a failed teardown and published the new engine anyway, which is exactly the leak the unload exists to prevent: the arbiter's evictor, /images/unload and the next load all resolve through get_active_diffusion_engine(), so the still-resident pipeline (or a live sd-server) became unreachable and the next load allocated on top of it. The switch now fails and leaves the old engine published, so it stays reclaimable. - The native generation timeout was 30 minutes while the Images page waits up to 6 hours (SETTLE_MAX_MS), so slow-but-progressing CPU jobs died deterministically at the deadline. Measured on GPU-less runners, a 512x512 4-step Q2_K generation took 900 s on Linux and 1465 s on Windows, so larger images or step counts clear half an hour easily. The ceiling now matches the page's window and applies to the whole request: chunks of a split batch share one deadline instead of each getting a full budget. Cancellation is unchanged. Declined: gating the huggingfacenotorch extra off Python 3.9 over the conditional diffusers marker. The marker is deliberate and its comment says why: diffusers dropped 3.9 in 0.38, so pinning >=0.39 outright leaves pip no candidate and the whole extra unresolvable there. The pipelines it names live in studio/backend, which cannot install on 3.9 anyway (studio.txt pins matplotlib==3.10.9 and fastmcp>=3.0.2, both requires_python >=3.10), and the extra is the general core one, so the alternative drops 3.9 for library users who never touch Studio. * Close the load-versus-training-start race, and two picker fixes - The image and video load guards read is_active() and only then selected an engine, acquired the arbiter and registered the load. A /train/diffusion/start reserving inside that window freed residents the load had not registered yet, so the trainer came up beside a brand-new pipeline. The service already had exactly the right pattern for this in dataset_mutation, so gpu_load_admission mirrors it: reserve() refuses while an admission is open, an admission refuses once a start is reserved, both decided under the one lock. The span is only the registration, since begin_load returns as soon as the load is registered and _free_gpu_for_diffusion_training preempts an in-flight load from that point. Chat is deliberately not covered: its load spans an eviction plus a multi-minute GGUF load, and it admits models that fit beside training by design, which is a different contract from the diffusion pipeline's all-or-nothing one. - Hugging Face gives the LTX-2 family the image-to-video pipeline_tag (both Lightricks/LTX-2 and unsloth/LTX-2.3-GGUF report it), so a text-to-video-only filter dropped the flagship audio family out of Video Hub search while the rest of the app routed it to Video. - Task-scoped quant fit sized picks against the LARGEST visible device while resolve_diffusion_device_target returns a bare "cuda" and torch places on the current one. On a heterogeneous host that recommended a checkpoint sized for the bigger card and then loaded it onto the smaller one. Fit now uses the device the load actually lands on; identical on a homogeneous host. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Expose the persistent conditioning cache in the start schema DiffusionLoraConfig has carried cond_cache_dir for a while and the DiT trainer acts on it, but DiffusionTrainingStartRequest omitted the field, so Pydantic dropped it silently and every API-driven run fell back to the in-memory cache that is rebuilt from scratch each time. The warm path skips loading the VAE and the multi-GB text encoders on a rerun whose images, captions and resolution are unchanged, so this was a real capability that could not be reached. Contained like output_dir rather than left to the trainer subprocess's cwd, since it is another directory the trainer writes to. Blank or omitted still means the in-memory cache, so it must not resolve to the outputs root. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix diffusion policy and classification issues from review fp8 auto precision defaulted to precise accumulate on any non-consumer GPU, which made fp8 2.05x slower than int8 on RTX 6000 Ada and slower than not quantising at all. NVIDIA's professional whitepapers do publish equal FP8 rates for both accumulate modes there, so the hardware premise held, but the cost is in the cuBLAS path rather than the published rate. Default to fast accumulate: measured on B200 the flag is a no-op (4096^3 _scaled_mm at 3023.8 vs 3041.8 TFLOP/s, bitwise-identical output, 1.213 s vs 1.230 s end to end), so it is a large win where it bites and free where it does not. Precise accumulate stays available via transformer_quant_fast_accum. Z-Image's DiT is a Lumina2 derivative, so unsloth/Z-Image-GGUF and unsloth/Z-Image-Turbo-GGUF both declare general.architecture = "lumina2" and the whole line was tagged image-diffusion-unsupported and hidden from the Images "On Device" list, though validate_load_request loads them. Resolve shared archs from the repo/file name like bare "wan" already does, with a test asserting the picker and the loader agree for every family. The sage attention on-demand install ran an unpinned `pip install sageattention`, but PyPI's newest wheel is 1.0.6 and diffusers refuses anything below 2.1.1: the install always "succeeded", wrote an unusable version into the running venv, and was rejected on the next line. Carry the dispatcher's floor so pip resolves nothing instead. The dense-quant disk gate sized the download from the bf16-RESIDENT table. The fp32 families download twice that (Z-Image: 23,479 MiB against a 21,970 MiB gate), leaving a window where the check passed and the download filled the disk; Ideogram 4 ships fp8 and was overcharged the other way. Size the gate by published bytes, verified against HF sibling metadata for all 12 families. Patch installs went through unsloth_zoo, which refuses to import unless UNSLOTH_IS_PRESENT is set, and that is set by unsloth itself. The server imports unsloth at boot so it never showed there, but any other process ran silently unpatched with every install returning False, which is 13 test failures on a clean environment. Import unsloth and retry once, memoised per process. Also: the GGUF+LoRA refusal pointed at the native engine without saying a GPU host only selects it under UNSLOTH_DIFFUSION_ENGINE=sd_cpp, so the suggestion was unreachable; the gallery recipe recorded loras from the generate request alone, losing a load-time bake; load-progress claimed "40.07 GB downloaded" for a fully cached load; and pickers.tsx imported three catalog-group helpers it never used. Reported by oobabooga. * Keep the sd.cpp text encoder on CPU under Metal macos-14 loads FLUX.2-klein-4B Q2_K natively on mps and then dies on the first generation with exit code -6: ggml_metal_op_encode_impl: error: unsupported op 'RMS_NORM' -> ggml_abort LLMEmbedder::encode_prompt -> LLMRunner::compute -> GGMLRunner::compute ggml's Metal backend gates RMS_NORM on contiguous rows and aborts the process when that does not hold, with no per-op CPU fallback, so any LLM text encoder (Qwen3 for FLUX.2 and Z-Image, T5 for FLUX.1) takes sd-server down. The encoder runs once per prompt while the DiT runs every step, so pinning only the encoder keeps Metal for the part that matters. UNSLOTH_DIFFUSION_SD_CPP_METAL_TE_GPU=1 opts back in once ggml grows the kernel. * Gate the unsloth retry in the diffusion patch backend The retry added for the clean-environment patch failures is not free: importing unsloth pulls torch in behind it, which costs ~940 MB of RSS measured in a process that had neither, and on a host with no accelerator it fails anyway. A cross-platform CI job that had generated fine at ~900 s later died 19 s in with SIGTERM and every 'if: always()' step skipped, which is the runner being torn down rather than a step failing. Retry only when torch is already imported (true of the server and of anything patching a real module, and the condition that stops the retry from being what loads torch), unsloth is installed but not yet imported, and the first failure was the ImportError the sentinel guard raises. The clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only retry the unsloth import where it can succeed The gate still let the retry run on hosts unsloth does not support, which is where it is most harmful: a 7 GB macOS runner lost the Studio server 26 s into a load, and the Linux runner was torn down mid-generation. Neither MPS nor plain CPU can complete the import, so the retry there pays the cost and fails anyway. Require an accelerator unsloth actually supports (CUDA/ROCm via torch.cuda, or XPU), with UNSLOTH_ALLOW_CPU as the documented override, and hoist the predicate to module level so it is tested directly rather than through the import system. On a CPU-only host the retry no longer fires at all; on CUDA the clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Guard old diffusers, stream video exports, record conditioned recipes Three fixes from review. The 0.39-only pipeline classes (Flux2Klein, Z-Image, Krea 2, LTX-2, HunyuanImage) were resolved by getattr deep in the load, so on the older diffusers that packaging still allows on Python 3.9 -- diffusers dropped 3.9 in 0.38 and this project still supports it, so the 0.39 floor has to be conditional or the extra becomes unresolvable -- an advertised model failed with a bare AttributeError after its checkpoint had already been downloaded. Krea 2 already guarded itself this way; assert_pipeline_class_available now runs the same check for every image and video family from validation, before any fetch, and names the version and the fix. WebM export accumulated the whole VP9 output in a BytesIO and returned it as one bytes object that the response held again. The request caps allow 2048x2048 for 1024 frames, so an export runs to hundreds of MB and concurrent clicks could exhaust the process, while the MP4 route beside it already streamed from disk. transcode_to_file encodes to a temp file and the route returns a FileResponse with a background unlink, so nothing large is resident. A conditioned generation's recipe carried only the txt2img fields, so the gallery presented an inpaint or upscale result as a complete Create recipe and restoring it replayed an unrelated text-to-image request. The images themselves are still not persisted (user uploads with their own lifetime), but the workflow and its scalars are, restore reapplies them, and the toast now names the inputs that have to be supplied again instead of silently landing on Create. Reported by Codex. * Per-load video cancel event, family-gated image picker, cond cache refusal A cancelled video load could resume: begin_load cleared the shared cancel event, and unload() drops _loading without waiting for the worker, so the next load cleared the very object the cancelled worker was watching and its multi-gigabyte pull ran on alongside the replacement until the token check at the end. Each load now gets its own threading.Event, passed down through _fetch_te_prequant and _predownload_base, so a cancelled worker stays cancelled. A cached repo with a model_index.json was advertised as text-to-image on the trust rule alone, but validate_load_request also requires a detected image family, so a trusted pipeline of an unsupported class produced a picker row that deterministically 400s. The picker now applies both gates, mirroring the video branch. cond_cache_dir was accepted for sdxl and then ignored: only the DiT trainer reads it, while the SDXL trainer builds a per-run in-memory latent cache, so the promised cross-run reuse never happened. The route now refuses it with a 400 that names the families which do support it, checked against the resolved family so an omitted model_family with an SDXL base is caught too. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the frontend build broken by the gallery blob cache tsc -b failed on the branch head, so npm run build produced no dist and every platform job fell back to --api-only: blob-url-cache.ts(29,15): TS1294: This syntax is not allowed when 'erasableSyntaxOnly' is enabled dataset-labeling-grid.tsx / dataset-showcase.tsx: Argument of type '{ url: string; bytes: number; }' is not assignable to parameter of type 'string' The cache took its budget as a constructor parameter property, which the project's tsconfig forbids, and fetchGalleryObjectUrl now returns the blob size alongside the URL for that budget, which the two dataset thumbnail components still consumed as a bare string. Declare the field explicitly and destructure the URL at both call sites. tsc -b is clean and vite build emits dist again. * Recover from a ggml unsupported-op abort by restarting on the CPU backend ggml checks every node against the device's supports_op and calls GGML_ABORT when one is not implemented, because a single-backend graph has nowhere else to put it: there is no per-op CPU fallback. The whole sd-server dies with SIGABRT mid-generation and the user gets "the native image renderer stopped unexpectedly" with no way forward. Seen on macos-14 arm64 with FLUX.2-klein-4B Q2_K through the cross-platform CI: the text encoder is already pinned to CPU, and the abort moved into the denoise loop instead. ggml_metal_op_encode_impl: error: unsupported op 'MUL_MAT' -> ggml_abort StableDiffusionGGML::sample -> sample_k_diffusion A retry on the same backend would abort identically, so the load is restarted once with --backend cpu (the only flag that changes which backend executes the graph; --offload-to-cpu moves parameters, not compute) and the generation is re-submitted. The same checkpoint then renders slower rather than not at all. Strictly bounded: the signature must carry both the unsupported-op line and ggml_abort, the device must not already be CPU, and it happens once per load, so an OOM kill or a genuine crash still surfaces as itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix two tests that only fail in a full-suite run The 3.10 CI leg resolves PyAV 17, where av.container.OutputContainer is an immutable C type, so the no-libopus export test died on "cannot set 'add_stream' attribute of immutable type" before it asserted anything. Inject the refusal by wrapping the container av.open() returns instead; modules stay patchable on every build. Removing the injection makes the test fail again, so it still covers the branch it is named for. The Xet shim's degraded-path tests drop utils.hf_xet_fallback from sys.modules and import a throwaway copy. Restoring only the sys.modules entry left the utils package attribute bound to the throwaway, and the two disagreed for the rest of the process: a later monkeypatch of the dotted target patched one copy while the code under test imported the other, so the patch did nothing and test_fetch_te_prequant_only_reports_what_it_downloaded reached the real Hub and got a 401. Restore both bindings. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop an ignored-cancel sd-server, guard deletes during diffusion training, repair unusable managed binaries sd-server does not interrupt an in-flight job, so when it ignores a cancel the grace branch abandoned the poll and reported cancellation while the native job kept a core (or the GPU) busy to completion and held the server's job slot. The comment said the caller stops the server, but only unload does that immediately: a superseding load stops it after its multi-gigabyte download, and a load that then fails never gets there. Stop it here, as the deadline branch already does. DELETE /api/models/delete-finetuned checked only the LLM trainer, so it could rmtree the output directory a live diffusion LoRA run was about to write its adapter into. Consult the diffusion training service too, like the dataset mutation and model-load routes. find_sd_*_binary only checks is_file(), so an interrupted extraction (or a prebuilt for the wrong CPU) left a present-but-unrunnable binary the installer never retried: every load probed it, fell back to diffusers, and native inference stayed off until the directory was deleted by hand. Probe it and reinstall, but only for a copy under the installer-owned root -- SD_CLI_PATH, UNSLOTH_SD_CPP_PATH, an in-tree build and anything on PATH are the user's. * Plan the pre-cast text encoder, and make the cross-trainer GPU admission atomic An fp8 text-encoder request loads a hosted PRE-CAST checkpoint, but the image download plan never received text_encoder_quant, so the manager staged the base repo's dense encoder (FLUX.2-dev's Mistral-24B is ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) and the load then pulled the pre-cast file inline, outside the manager's progress and disk preflight. The plan now takes the field, resolves the hosted artifact with the same resolver the injection uses, stages that file, and drops only those components' dense weight shards. The load's own prefetch takes the same treatment, since it paid the same cost. Only a checkpoint that really resolves on the Hub earns the drop, so a gated or renamed artifact still stages the dense encoder the load will fall back to. The two trainers admitted each other with independent check-then-act guards: the diffusion route checks the LLM backend several network-bound preflights before it reserves, and the LLM route checks the diffusion service well before it spawns, so two near-simultaneous starts could both pass and train on one GPU. reserve() now re-tests the LLM backend under its own lock, and the LLM route holds the diffusion service's gpu_load_admission across its spawn, so exactly one of the two wins. Both halves fail open, so a chat-only install still trains. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the Advanced options a staged download planned against Staging does not set busy, so while a multi-gigabyte download runs the user can still change precision, memory mode, speed or the baked LoRA selection. The pending record held only the repo and artifact, and the completed download fired a load that read the CURRENT state: the staged file set could then be missing files that load needs (fetched inline, with no progress and no disk preflight) or hold gigabytes it no longer uses. One snapshot of every Advanced control is now taken when the plan is built, and it travels with the pending record into the load, so the load that runs is the one the download was planned for. * Do not advertise a family the installed diffusers cannot build The newer families (Z-Image, Krea 2, FLUX.2, LTX-2, HunyuanImage) exist only from diffusers 0.39, and 0.39 cannot be installed on Python 3.9 at all -- diffusers dropped 3.9 in 0.38, so the requirement is conditional or the whole extra becomes unresolvable. On such an environment the picker still offered those rows, every pick failed deterministically, and the error's advice to run pip install -U diffusers could not fix it without also upgrading Python. The cached-repo picker now applies the same availability check validate_load_request does, which is keyed on the pipeline class actually present rather than on the Python version, so it is also right for an intentionally pinned older diffusers on 3.10+. Fails open when diffusers cannot be imported at all: that is a different problem and the load path reports it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore the diffusion engine selection after each router test The active engine is module state, and several tests set it by plain assignment because what _activate does to it is the thing under test, so monkeypatch could not undo it. A leaked ENGINE_SD_CPP left get_active_diffusion_engine() handing back the sd.cpp backend for the rest of the process, and every later route that reads the active engine then saw an unloaded model: eight tests in test_openai_images_generations_route.py returned 503 in a full-suite run while passing on their own. The autouse fixture now snapshots and restores it. * Stream gallery clips, and close three races around them Four fixes from the latest review pass. The video gallery downloaded each clip into a blob before it could play, so playback waited on the whole file (tens to hundreds of MB), seeking was limited to what had arrived, and every viewed clip stayed pinned in the webview. The file route already streams and serves ranges; it just could not be a <video src> because it is bearer-gated. Mint a short-lived signed link instead (its own HMAC secret, 12 hour TTL, separate from the image links) and hand it to the element, which then fetches only the ranges it plays. That removes the blob budget, its LRU and every revoke on this page. The sd.cpp readiness probe accepted any process answering on the port, so a foreign server that grabbed the port between the bind check and the spawn was adopted as ours. Confirm the listener is our child before reporting ready, and stay best-effort (psutil missing, an unknown owner, or any probe error still passes) so the check can only reject a definitely foreign process. Dataset import held its lock for the extract but not for the upload path, so two concurrent uploads into the same folder interleaved; take the same lock and return 409. And reject Windows device names (CON, NUL, COM1..9, LPT1..9, with or without an extension) plus trailing periods in dataset names, which are unopenable on Windows. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Build the image download plan for the engine that will load /images/download-plan always asked the diffusers backend, while /images/load picks the engine per host: a GGUF pick on a machine with no usable GPU routes to native sd.cpp, which reads a single-file VAE plus text encoders and never opens the base repo's sharded components. Measured on unsloth/FLUX.2-klein-4B-GGUF (Q2_K): the plan staged 7.66 GB of FLUX.2-klein-4B components the native load discards, and the 7.80 GB sd-cli actually needs was then fetched inline by the loader, outside the download manager's progress and its disk preflight. Z-Image-Turbo is the same shape. The plan now asks whichever engine the load will select. predict_engine() applies the selection policy without any side effect: it activates nothing (staging a download must not unload the resident model) and only locates the binary rather than installing it, but still counts an installable binary as available, since that is what the load does on a fresh host. The native backend gains a download_plan built from the same _asset_specs the loader fetches, returning the same envelope, so the manager stages exactly the files sd-cli opens. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not let a queued generation outlive the model, and three scan fixes Five items from the latest review; four were real. An unload or arbiter eviction only cancels the generation holding _generate_lock. A second request queued behind it holds no cancel event yet, and Python locks are not FIFO, so it could take the lock the instant the active denoise released it, still see a loaded pipeline, and run a whole new denoise after the model was told to go away: the eviction then waits minutes for it and an image lands after the eject. Unload and a superseding load now raise a fence under _lock before they queue, and a generation that wins the lock while one is pending refuses instead. The cached-model scan judged pipeline completeness across every revision, so a repo holding an older complete snapshot plus a newer companion-only one read as complete while the snapshot from_pretrained actually opens has no transformer. Both scans now look at the revision the loader will open. Deleting a dataset image deleted its caption sidecar unconditionally, which for cat.jpg alongside cat.png removed the caption the survivor still resolves to. The sidecar now goes only with the last image of that stem, matching what the thumbnail cleanup beside it already did. Importing an example into a folder that holds no images but does hold files fell back to promoting the staging dir one file at a time, so an interruption left a partial dataset that the image_count check accepts as complete on retry. Those files are folded into the staging dir instead and the promotion stays a single atomic rename. The MPS generator report does not apply: torch.Generator(device="mps") has worked since PyTorch 2.0 (pytorch/pytorch#91348) and the studio installer pins torch>=2.4. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten diffusion comments Collapse the multi-line comment blocks across the image, video, sd.cpp and diffusion-training code to one or two lines each, and drop comments that only restate the statement below them. Comments only, no code or behaviour changes. * Tighten diffusion comments (second pass) Collapse the remaining multi-line comment blocks in the video page, training routes and service, sd.cpp server and installer, memory and speed planners, and the shared request models. Comments only, no code or behaviour changes. * Tighten diffusion comments (third pass) Collapse the remaining multi-line comment blocks in the attention, cache, LoRA, prequant, precision and compile-cache modules, the sd.cpp arg builder and engine, the video routes, the Ideogram 4 assembly, the model picker, and the diffusion test suites. Comments only, no code or behaviour changes. * Restore the dataset when an example import cannot be promoted Promotion folds the folder's pre-existing entries into the staging dir so the swap is one atomic rename. Every failure after that fold left the user with nothing: the 409 path and an os.replace error both fell through to 'finally: shutil.rmtree(staging)', which deleted the entries that had just been moved in there, while the response said 'Nothing was written'. A same-named entry was also unlinked outright before the promotion was known to succeed. Park superseded same-name entries in a rescue dir instead of deleting them, record every move, and restore all of them if any step of the promotion fails. The fold loop itself is covered too: renaming a non-writable directory raises EACCES on POSIX, which previously escaped as a 500 after the earlier entries had already been moved out. A failed rename now maps to the same retryable 409 as the rmdir conflict. Verified with the reported trigger (a non-empty mode-500 directory whose name collides with an imported file): the folder listing is now identical before and after the failed import. * Drain the teardown fence on a failing unload, give each load its own cancel event Two independent leaks on the image path, both already solved elsewhere in the same file. unload() incremented _teardown_waiters, ran _unload_locked() and decremented, with no try/finally, while the superseding-load path used a finally for the same pair. _unload_locked ends in clear_gpu_cache(), whose CUDA branch calls synchronize/empty_cache/ipc_collect unguarded, and a sticky CUDA fault makes those raise. The count then never drained, so every later generation was refused as cancelled for the life of the process, a fresh load included, since begin_load's own increment and decrement are symmetric. unload() is reached from the chat/video GPU handoff, the engine router and two training routes, so one fault during an ordinary handoff wedged image generation until restart. Release it in a finally. The image and native backends each cleared one shared cancel Event on a new load. unload() sets that event to cancel an in-flight multi-GB download and drops _loading in the same breath, so a replacement load is admitted while the cancelled worker is still inside the fetch, and its clear() re-enabled the very object that worker was watching: the cancelled download resumed and ran alongside the replacement. Take a fresh Event per load and thread it to the worker, as the video backend already does, and set it under the lock since begin_load now rebinds the attribute. * Name utf-8 on the diffusion text I/O and the sd.cpp subprocess pipes tests/test_text_io_encoding.py failed on five files this branch adds. Text I/O without an explicit encoding falls back to the Windows ANSI codepage, so a non-ASCII path or manifest value round-trips corrupted, and the three sd.cpp pipes decode the child's UTF-8 output as ANSI on Windows despite already passing errors = 'replace'. Eleven read_text() / write_text() sites across diffusion_compile_cache, diffusion_ideogram4 and diffusion_krea2, plus text = True on the sd-cli version probe, the sd-cli run and the sd-server pipe. * Record the load-time build on a gallery image's recipe A gallery record documents itself as the image's full generation recipe and is embedded in the PNG, but the only load-related field it carried was the repo id. A GGUF repo holds many quants, so that does not say which one made the pixels, and it says nothing about an adapter baked in at load time. The fallback meant to cover the baked case could never fire: with no loras on the request _adjust_baked_loras zeroes every baked adapter and _active_lora_pairs drops zero-weight entries, so active_loras was always empty. A baked-and-disabled build is not the same pipeline as a never-baked one, so the recipe could not reconstruct the image once the model was rebuilt. Persist model_kind, gguf_filename, transformer_quant and the baked adapter names, read off the load state rather than the request, and show them in the recipe popover. The new fields are optional with defaults, which matters because list_gallery_images drops any record that fails validation, so a required field would have emptied every existing gallery; a regression test pins that. * Drop eleven duplicated comment tails, restore the mxfp8 denial note The comment passes collapsed several wrapped blocks onto one line without deleting the last physical line of the original wrap, leaving the tail of each sentence repeated as its own comment underneath. Two of the eleven were re-worded rather than byte-identical, so a strict suffix match missed them. 6e16ad16f also dropped the line justifying the qwen-image mxfp8 denial while that rule stayed live in _FAMILY_SCHEME_DENY, under a header that then documented only fp8 and nvfp4. Restored. Comments only; verified with the AST gate. * Only let the dense-quant fallback use shards the prefetch actually staged The prefetch skips the base repo's transformer/ shards whenever a prequant checkpoint is expected, since that checkpoint replaces them. But a prequant fetch can fail for reasons the planner cannot see: an unpublished, gated or renamed artifact, a hub 5xx, a proxy, a checkpoint the validator rejects. The loader then fell through to from_pretrained(subfolder = 'transformer') and pulled those shards inside the load lock during 'finalizing', after the previous pipeline was already evicted, where the cancel event has no reach, load_progress has already reported bytes_downloaded == bytes_total, and the cache-disk gate had only reserved the small prequant checkpoint. That is verbatim the situation _dense_quant_prefetch_needed's own docstring exists to prevent. Gate the in-loader dense fallback on the shards being staged, read off the returned file list rather than the request so a failed size estimate closes it too, and let the GGUF build take over otherwise, which is what the prequant-sized replan already does one branch over. It is also the invariant the text-encoder path already enforces: only a component that really resolves may have its dense weights dropped from a plan or a prefetch. Adds the missing coverage for the gate's prequant arm, which the existing disk-gate tests never reached. * Refuse a training output_dir that resolves to the outputs root resolve_output_dir strips a leading 'outputs' and drops '' / '.' segments, so '.', './', './.', 'outputs', 'outputs/outputs' and ' . ' all clean away to nothing and land on the outputs root itself rather than a run directory under it. The DiT trainer then writes pytorch_lora_weights.safetensors flat into the root, where scan_trained_models and scan_checkpoints cannot see it (both filter is_dir()), and a second such run overwrites the first. The UI only checks the field is non-empty, so 'outputs' is one plausible run name away. Refuse it with a 400 that says what to do instead. cond_cache_dir collapses the same way but has an honest 'off' to fall back to, so a root-resolving value now means the in-memory cache, which is what the comment above it already promised: otherwise a run drops one flat safetensors per cached latent and caption into the directory trained models live in. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Repair three defects the main merge left behind Git merged all three files without a conflict, but the result was wrong in each case. The tests only started failing once the merged tree was run. routes/inference.py duplicated the GGUF load block: this branch had moved the gguf_load_in_flight marker and the _hub_download_blocks_gguf_load guard under "if config.is_gguf and config.gguf_hf_repo", and the conflict resolution re-added main's copy at the old position, so both ran. Dropped main's copy; the earlier placement is the deliberate one, so a 409 from the hub guard cannot tear down a resident Images or Video pipeline. test_gpu_selection.py still called _hf_offline_if_dns_dead, which main renamed to _hf_offline_if_unreachable_for (#7591). Disjoint edits, so no conflict, but four route-error tests referenced a function that no longer exists. test_gguf_load_cache_reuse.py anchored its ordering assertion with rindex over "if config.is_gguf:", taking the last one before the load marker. That only held while _resolve_inherited_extra_args sat above every such line; main has since hoisted it above the gpu_ids preflight, so the anchor landed between the call and the marker and the assertion compared against an unrelated later call site. The ordering it checks is a property of _load_model_impl as a whole, so it now anchors on the function. The ordering itself is intact: _resolve_inherited_extra_args, then the gguf_load_in_flight marker, then the hub guard, then the chat handoff, then unload_model. * Stop a real sd.cpp install from breaking its own discovery tests The five "nothing is installed" assertions in test_sd_cpp_engine.py cleared SD_CLI_PATH and UNSLOTH_SD_CPP_PATH and patched Path.home, which covers hops 1, 2 and the fallback half of hop 3. It leaves two hops live. Hop 3 goes through managed_install_root(), which honors UNSLOTH_STUDIO_HOME and STUDIO_HOME and resolves to the stable-diffusion.cpp directory beside the studio home. That is the documented way to run side-by-side Studios, so anyone who has one set gets a real sd-cli back from a finder the test expects to return None. Hop 4 is the in-tree developer build, which does the same for anyone who built sd.cpp inside the checkout. Isolated with an autouse fixture rather than another helper call, because reaching the failure needs no fixture: SdCppEngine(binary = None) runs the finder from its constructor, which is why test_generate_raises_when_binary_missing failed too. The fixture points the studio home and the in-tree root at an empty tmp tree, so every hop is answered by the test rather than by the box. The in-tree root moves into a named in_tree_install_root() so it can be pointed somewhere empty; behaviour is unchanged, including the OSError and IndexError guard on an unexpected layout. * Make the sd.cpp uninstall test actually extract the code it tests The two sed ranges anchored the production fragments at column 0, but both blocks sit inside the main removal function and are indented, so each range matched nothing and LOOP_FILE / DEFAULT_FILE came out empty. Sourcing an empty file is a no-op, so the suite reported 4 passed / 7 failed: the seven removal assertions failed because nothing ran, and the four that passed were 'kept' assertions that pass trivially when nothing runs. The shell job auto-discovers tests/sh/test_*.sh and runs each under set -e, and this file is not in its skip list, so it was a deterministic red. Anchor on optional leading whitespace and fail loudly on an empty or _remove_path-less fragment, so a future reshuffle of uninstall.sh cannot make the suite vacuous again. Now 11 passed, 0 failed, against the real removal loop and the real default-mode block. * Agree on what an engine can build, and on what the installer owns Two gates, each half-applied. The unbuildable-family gate had one caller, the image branch of the cached repo picker. The GGUF classifier, the local-model classifier and the video branch had none, so on a diffusers too old for a family the picker still offered its GGUF and the load then failed. Meanwhile the loader asserted the diffusers pipeline class before engine selection, so a GGUF this host routes to native sd.cpp, which instantiates no pipeline class at all, was refused with an upgrade instruction that could not help. Both are wrong one-sidedly: hide a family only when NEITHER engine can build it, and demand the diffusers class only when diffusers is what will load it. One predicate, family_buildable_here, now answers both, so the picker and the loader cannot disagree. The population is real on Python 3.9, whose diffusers ceiling is 0.36: Flux2KleinPipeline, Krea2Pipeline and LTX2Pipeline are all absent there, and FLUX.2-klein GGUF is a repo in this PR's title. That assertion also raised RuntimeError, which /images/load maps to 409, the status that otherwise means a load is already in progress, and which escaped /images/download-plan (it catches ValueError and FileNotFoundError) as a bare 500 with the message lost. It is an unloadable pick like every other, so it raises ValueError and both routes answer 400 with the text intact. The managed-binary repair used a path test for ownership while the installer uses a marker. On a managed root without the marker, which is any install predating it, the repair deleted sd-server and the reinstall was then refused, permanently, because the surviving sd-cli keeps the directory non-empty so the marker can never be claimed: the user went from an unrunnable binary to no binary and no way back. Require the marker before discarding, which is the same definition of ours that uninstall.sh already uses to keep a user's own stable-diffusion.cpp checkout. A genuinely interrupted extraction still self-heals, since install() writes the marker before it extracts. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add gguf and av to the studio extra so the wheel install matches studio.txt This branch added both to studio/backend/requirements/studio.txt, which is what install.sh uses, but never to the studio extra in pyproject.toml, which is what 'pip install unsloth[studio]' uses. Nothing else keeps the two in sync, so tests/studio/install/test_studio_extra_matches_requirements.py failed deterministically, and the CI job runs the whole tests/ tree. The drift is not only a red test: gguf backs diffusers' GGUFQuantizationConfig and av does the MP4 encode and audio mux, so a wheel install got a Studio that cannot read a GGUF or export a video, which is most of what this PR adds. * Fix two test-suite defects: the CPU-only patch gate and a popped module The patch backend reaches unsloth_zoo's helpers only through an ~940 MB 'import unsloth', which a CPU-only host cannot complete, so every patch install returned False and nine arch/eager/compile tests failed there -- exactly the runners the retry was narrowed to protect. unsloth_zoo only wants UNSLOTH_IS_PRESENT in the environment, which costs nothing and needs no accelerator, and the conftest already sets its sibling UNSLOTH_ALLOW_CPU. Set it there too, as run.py and main.py already do at module scope, so no real server ever took the expensive route. CPU-only goes from 9 failed to 24 passed, 1 skipped; the GPU run is unchanged at 29 passed. test_setup_cache_env_hf_home popped utils.hf_cache_settings to model a fresh process and never restored it, so a later import built a second module object and rebound it on the utils package. test_hf_cache_settings then wrote its setting into one object while core.inference.diffusion read the other, the same split-module failure the xet shim already had. Restore both bindings in teardown: 21 pass together, and each file still passes alone. * Restore the setup_fail assertions the main merge reverted A fourth instance of the class 9541cc535 fixed: the merge took main's studio/setup.sh, which #7644 changed to abort through the setup_fail helper so desktop mode still emits [TAURI:ERROR], but kept this branch's older copy of the test, which still asserted the literal 'exit 1'. setup.sh is byte-identical to main here and the only diff in this file was the reverted assertions, so take main's version. 60 passed. * Fence the video teardown, reject non-finite clipping knobs, stop a test installing sd.cpp The video backend never had the teardown fence the image backend documents. Its unload and its superseding-load path both signalled the active generation, then did 'with self._generate_lock: pass' and tore the state down with the lock free. A generation queued behind that barrier holds no cancel event yet, so the signal cannot reach it, and Python locks are not FIFO: it won the lock the instant the barrier released it, read a still-loaded state and denoised a whole clip against the pipeline being freed. Reproduced on both paths, where the queued generation returned a finished MP4. Mirror the image backend: count waiters, refuse a generation while one is pending, and tear down inside the barrier with the counter released in a finally. _teardown_state becomes _teardown_state_locked since the caller now holds both locks. max_grad_norm and snr_gamma were bounded on one side only, so 1e309 floated to inf and passed. clip_grad_norm_ then computes an infinite clip coefficient, clamps it to 1.0 and scales nothing, and min(snr, inf) / snr makes every min-SNR weight 1.0. The run starts, reports normal progress and trains with the requested knob silently disabled. Measured both. NaN already failed the bounds; allow_inf_nan makes that explicit. learning_rate and flow_shift already guard this exact vector. test_load_routes_to_sd_cpp_on_cpu stubbed ensure_sd_cpp_binary but not ensure_sd_server_binary, which select_and_activate_engine probes first with installs enabled, so the unit test downloaded and unpacked 108 MB into the developer's real ~/.unsloth when UNSLOTH_STUDIO_HOME was unset. Now 0 bytes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Key the video schedule reset on the resolved defaults, not the repo id A GGUF repo holds several variants, so another client swapping a distilled build for the base one from the same repo changes the steps and guidance the backend reports while repo_id stays put. modelChanged stayed false, the page kept the previous schedule, and later generations ran an 8-step distilled setting on a model that expects roughly 40 steps and CFG 4. Include the reported defaults in the key. They only move when the resident artifact does, so a manual steps change still survives a status poll. * Retire the release-lag pin now that 2026.7.6 carries the relaxed gates The virgin Windows container row that installs from PyPI on purpose was tolerated with continue-on-error, because a Server Core container has no Microsoft Store and so no winget, and the released studio/setup.ps1 hard-stopped on a winget-only git gate and reached for winget again for the VC++ runtime. #7549 relaxed both, but it landed on 2026-07-28 and the newest wheel was 2026.7.5 from the 23rd, so the row could not get past it. The pin carried its own tripwire for exactly this moment, and it has fired: unsloth 2026.7.6 shipped 2026-07-29 and the released wheel now installs end to end, so the step reported success and the assertion errored with "delete this pin and drop continue-on-error from the Install step so this row gates". Doing that. Both rows now gate unconditionally, and the assertion that enumerated the accepted failure signatures goes with the pin it protected, since there is no longer an accepted failure for it to describe. The other three release-lag mentions in this file are descriptive rather than tolerated-failure pins: they explain why the fedora and ubuntu-nonroot legs run with overlay: true, and both already refuse a triton failure as a regression rather than accepting it as lag. Left alone. * Pin the hosted pre-quant onto the plain-torch fp8 kernel A Z-Image GGUF pick at the DEFAULT speed mode, once the hosted fp8 pre-quant is actually reachable, dies at generate with an HTTP 500: torch._dynamo.exc.Unsupported: Operator does not support running with fake tensors. Developer debug context: unsupported operator: mslk.f8f8bf16_rowwise.default _fp8_config already pins KernelPreference.TORCH when it BUILDS a config, precisely because the default AUTO switches to the MSLK kernel wherever an mslk package is importable (sm90+). A hosted checkpoint escapes that pin completely: the preference is serialized per Float8Tensor, and all 239 weights in the published Z-Image-Turbo-FP8 checkpoint carry AUTO. Loading it re-arms the very kernel the pin exists to avoid, mslk.f8f8bf16_rowwise has no fake impl, and the first compiled generate therefore cannot be traced. Isolated away from the product to be sure of the mechanism: quantise one Linear three ways on this box and compile each. KernelPreference.TORCH eager ok, compiled ok KernelPreference.AUTO eager ok, compiled FAILS on mslk.f8f8bf16_rowwise library default eager ok, compiled FAILS the same way So the pin is correct and necessary, and the only gap is that the hosted path never got it. _validate_checkpoint checks scheme, granularity, base model, min_features, exclude tokens and fast_accum, but not this. Rewriting the preference on load is safe: it selects a matmul kernel, it is not weight data, so the tensors stay bit-identical and the checkpoint's own state_dict_sha256 still describes them. It is also the faster path compiled, since the opaque extern call blocks inductor quantize fusion. Why this went unnoticed: the pre-quant repos are private, so nothing that could not read them ever took this path. It becomes the default the moment they are readable. Verified end to end at 1024x1024, 8 steps, speed_mode default, on the published checkpoint: before HTTP 500 at generate, diffusion.generate_failed after pinned 239 weights to the plain-torch fp8 kernel, load 10.0 s, transformer_quant fp8 with compiled engaged, cold 7.9 s, warm 0.87 0.87 0.87 0.86 0.87 s That warm number also beats the 1.4 s recorded for this shape on 2026-07-26. 159 prequant and transformer-quant tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cover the fp8 kernel pin so the 500 cannot come back quietly The hosted pre-quant re-arming MSLK was only visible as an HTTP 500 on a compiled generate, which needs a GPU, a readable private repo and an importable mslk to reproduce. None of that is available in CI, so the pin would rot unnoticed. Three hermetic cases on _pin_kernel_preference instead: AUTO weights are rewritten and already-TORCH ones are left alone (counting only what changed), a weight that refuses the assignment does not sink the whole load, and with no torchao enum available the checkpoint is left exactly as saved. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: tighten comments across the image generation, video and training code Rewrite the comments added by this branch to be shorter and clearer: collapse multi-line explanations into one or two lines, drop comments that only restate the code, and keep the rationale that explains why a choice was made. No code changes. * Studio: let the pipeline-class guard survive a host with no diffusers Backend CI installs the CPU-only dependency set, which has no diffusers, and runs without a GPU. 13 tests failed there for three separate reasons, one of which is a real product bug rather than a test-environment gap. assert_pipeline_class_available did a bare "import diffusers", so on any host without it the call raised ModuleNotFoundError instead of the ValueError its own contract promises. /images/download-plan catches only ValueError and FileNotFoundError, so that escaped as a bare 500 with the message lost, which is the exact failure the guard exists to prevent. It is reachable in production: the native sd.cpp engine serves GGUF picks on a CPU or Apple host where diffusers is never installed. Absent diffusers now returns, since the check answers "is the installed diffusers new enough for this family" and there is no version to judge; a pick that genuinely needs diffusers still fails in the loader with its own message. The rest were tests asserting through gates unrelated to what they cover: - Three cond_cache_dir route tests drive a DiT family, which the start route refuses without an accelerator. They now take the existing dit_train_host fixture, written for this case, so they keep testing the schema on every host. - test_download_plan_forwards_the_load_time_controls stubs the diffusers planner but sends a GGUF pick, which routes to the native planner on a GPU-less host. It now pins the engine, so it tests kwarg forwarding rather than the hardware. Engine selection keeps its own tests. - The pipeline-class guard test needed a real diffusers only for its sweep over every shipped family. That half is split out and skips on its own; the stub-driven refusal needs no diffusers and still runs. Added a test for the absent-diffusers contract above. 409 pass on a GPU host with diffusers; 301 pass and 1 skips with diffusers blocked and no GPU. * Studio: cut the image generation comments down further Second pass over the comments this branch adds. Collapse the multi-line blocks that still read as paragraphs, rewrite the longest one-liners so they say the same thing in fewer words, and drop a stale comment that had drifted away from the constant it described. No code changes: only comments and whitespace. * Studio: workflow rail, sidebar Video row, and Train settings polish Sidebar - Pin Video under Images by default; persist store bumped to v3 so an untouched install adopts it while a custom arrangement is left alone. - Add "Customize sidebar" to the end of the More flyout, opening Settings > Appearance scrolled to the sidebar nav section. Images - Replace the workflow dropdown with an icon rail down the left edge: all seven workflows visible, keyboard nav, tooltips carry the labels. - Workflows stay selectable with no model loaded, so one can be set up before picking a model. Generate is already gated on a loaded model. - Video moves out of the Create/Train strip into a link at the far right, with a matching Images link on the Video page. - Wider gutters around the settings column, headings matched to Train. Train - Field guidance moves into "i" tooltips; only state that limits a control stays on the page. Steps, LoRA rank and the rest gain hints. - /info reports params, qlora_vram_gb, gated and note as fields, so the family note renders as chips. vram_note is rebuilt from them unchanged for older clients. Shared - Number steppers appear on hover or focus. - Lighter control border (#e9e9e9) on Images and Video, light mode only. * Studio: move Advanced inline on Images and Video - Replace the top-bar toggle and right-docked Advanced panel with a disclosure under Seed, so load-time tuning sits with the settings it affects and opening it no longer shrinks the preview canvas. - Video uses the bordered variant, since offload and memory decide whether a model fits at all. - Open state persists per page in localStorage. - Tighten the Steps unit trigger: 14px chevron, less right padding. * Studio: full width media pages and layout polish - Drop the 1100px cap on Images Create, Images Train and Video, so the preview canvas grows with the window instead of sitting in a band. - Align the workflow rail to the model selector label, widen the gap before its divider, and restore the divider itself. - Advanced: more room above and below, icon and label sized to the slider rows, and the same quiet row on Video as on Images. - Give Video a pane heading and description, matching Images. - More space above Seed on both pages. - More no longer takes the active style when the current page is one of its own rows. * Studio: fix the Extend side toggles The resting outline used a ring, and index.css blanks the ring on the button holding mouse focus, so the side you just clicked lost its border until focus moved. Use a border for the outline and leave the ring to focus-visible. Also separates the two states properly (the off state had no surface of its own), drops the border in dark mode as the inputs do, and adds the aria-pressed and focus-visible styling the buttons were missing. * Studio: drop the border on the Extend side toggles Light mode should carry no border either, so the fill alone marks the state: a muted surface when off, a primary tint when on. Matches the borderless treatment dark mode already had. * Studio: move the Images workflows into the sidebar The workflow switcher was a vertical icon rail on the page. It now lives in the sidebar under Images, so the page keeps that width for the canvas. - Workflows list under the Images row, with a chevron to fold them away. Hovering the row peeks them in a flyout on the standard menu surface. - Clicking Images while already there toggles the list instead of navigating. - The listed workflow carries the highlight, so the Images row drops it. - Create takes its own icon: it was sharing the New chat pencil. - Model hub moves above Projects. Persisted layouts bump to v4 and only adopt the new order where the stored one is still a shipped default. - Images and Video content both start at 32px, clear of the sidebar. * Studio: simplify the Images workflow submenu Dropping the hover flyout: two ways to reach the same seven workflows, one of them overlapping the list right below it, read as clutter. - The flyout is gone. The workflows are rows under Images and nothing else. - On the Images page they are always open, since they are that page's switcher. Elsewhere they stay folded, and the row's chevron opens them. - Create takes the sparkles icon. - Create and Train both pad their settings column to 40px a side, so the two tabs line up with each other and with the model selector. * Studio: dock the generate action and tidy the Create controls The primary action sat at the foot of a long scroll, so it was off screen until you scrolled for it. - Generate, Video's Generate and Train's Start training float at the bottom of their settings column. No bar behind them: hover lightens the fill rather than thinning it, and the disabled state is opaque, so the controls underneath never show through. - Aspect ratios read as names: Square (1:1), Widescreen (16:9) and so on. - Width and height replace their sliders with two compact boxes. Type a size or pick one from the menu; the value still snaps into range and still drives the locked ratio. - Negative prompt gets a quiet disclosure under the prompt. It was there before but only above guidance 0, which is not the default, so it never showed. - The info "i" is smaller across the settings UI. * Studio: reveal Images and Video field hints on hover The "i" next to every field label sat there permanently, which made a column of settings read as busier than it is. Scoped to the diffusion pages by CSS rather than per component: the hints come from the page's own Field and SelectRow, the train panel's FieldLabel, and the chat ParamSlider, which is shared and should not change for chat. Reveals when the pointer is inside the field, on focus-within for keyboard, and stays inert while hidden. InfoHint grows a data-slot so the rule has a stable hook, rather than keying off its aria-label. * Studio: one-line sliders, edge fades, shared negative prompt - Steps, Guidance and the rest put label, track and value on one row. ParamSlider is shared with chat, so this is an opt-in prop and chat keeps the stacked layout. - The settings columns fade at whichever edge they run past, as the sidebar and model picker do, instead of cutting off. New useScrollFades hook drives all three. - Negative prompt moves to a shared component and Video picks it up, so both pages collapse it the same way. - Right gutter sits closer to the rule, Steps takes a bigger break above it, and the Advanced rule sits between its neighbours rather than up against the field above. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: add the API monitor to the sidebar nav Sits under More, below Export. Points at the monitor page, not the API keys dialog the profile menu opens. No store version bump: an id missing from a stored layout is appended with its default pinned state, so existing sidebars gain the row at the end and keep their own order. * Studio: take main's copy of the sidebar spinner test The merge kept this branch's older version, which pins the Recents row to h-[33px]. Main already relaxed that to any height, since row density moves independently of the trailing column the test is about. * Studio: fix FLUX.2 klein size resolution, HunyuanVideo tier routing, and the personalization save Five fixes found while auditing the model registries against the live Hub. FLUX.2 klein 9B loaded against the 4B config. One family covers both klein sizes and defaults to 4B, relying on the base_model card tag for the real base, but the trust allowlist only had klein-4B, so the correct tag was discarded and a 9B GGUF (inner_dim 4096) was loaded against a 4B config (3072). That surfaced as a bare shape mismatch from inside the GGUF quantizer naming neither the file nor the repo. Adds the missing allowlist entries and a header-level size check that fails early with a legible message; the check is fail-open, so an unreadable file, a non-FLUX.2 family, or an unmapped base leaves the load exactly as it was. The sd.cpp text encoder rule matched the literal "klein-9b", so klein-base-9B was handed the 4B encoder. HunyuanVideo-1.5 720p checkpoints routed to the 480p family. Only the literal 720p_t2v path was aliased, so 720p_i2v and every GGUF repack fell through to the generic token and inherited the 480p base repo, which also supplies the VAE and text encoder. The tier is baked into the weights (transformer target_size 640 vs 960, scheduler shift 5.0 vs 9.0) and the two bucket lists are disjoint, so this ran the whole pipeline off-tier. Resolution presets that the checkpoints were never trained for. Wan2.2 TI2V-5B is 720P-only upstream (SUPPORTED_SIZES is exactly 704x1280 and 1280x704, asserted in generate.py), and the HunyuanVideo 480p square preset 624x624 is not a bucket of its tier; 640x640 is. Personalization could never be saved. The frontend ships eight sidebar nav ids and refills every missing one on each save, but the backend Literal had seven and no "api", so every PUT to /api/settings/personalization returned 422. Aligns the server defaults and the Literal with the shipped layout. A catalog row pointing at a file that does not exist. unsloth/Qwen-Image-2512-FP8 holds torch prequant .pt checkpoints, not qwen-image-2512-fp8.safetensors, and fp8 is denied for this family anyway because it renders black, yet the row was the auto-route target on any GPU above 40 GB. Also marks FLUX.1-schnell, Krea-2-Turbo and the two Ideogram repos gated: all four are gated on the Hub today, and schnell being Apache-2.0 does not make it ungated. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: stop a case-only duplicate upload destroying an image, and size the load against the right GPU Two fixes from the review round. A dataset upload containing two names that differ only by case silently loses one on Windows and macOS. The in-batch duplicate check is an exact string compare, so "Cat.png" and "cat.png" both pass, but on a case-folding filesystem they are one destination: the commit step moves the first staged part aside, writes the second over it, then deletes the backup, while the response still reports both as uploaded. Captions collapse the same way and never even reach the image branch. No single folder can hold such a pair, but the Windows and macOS open dialogs flatten search and Recents results across folders into one multi-selectable list, which is a normal way a LoRA dataset gets assembled. Rejecting the pair everywhere would regress Linux, where the two really are different files with their own sidecars, so the check probes the filesystem once per process instead of keying off sys.platform: macOS also ships case-sensitive APFS volumes, and a Linux host can keep its Studio home on an exFAT or NTFS mount. A failed probe answers "case-sensitive", which leaves today's behaviour untouched. The quantization fit budget was sized against the wrong card under a reordering CUDA_VISIBLE_DEVICES. A bare "cuda" load lands on visible ordinal 0, but the reducer took the lowest PHYSICAL index, which the nvidia-smi path reports as index_kind "physical". Under "3,1" that sizes against GPU 1 while the pipeline loads onto GPU 3. The hook's own interface doc already promised the lowest visible ordinal. Ranks by visible_ordinal, falling back to index only for an older backend that omits it. * Studio: rework the diffusion dataset panel and align the pane gutters Upload: - one Upload button beside the dataset name, with an upload icon - picking files uploads them, no second confirm click - Add button beside the Training images dropdown, so images can go into a set that already exists - fall back to the upload form when a selected dataset no longer resolves Labeling grid: - two columns at any width; sm:grid-cols-3 crushed the tiles in a fixed width column - pin the caption size so the Textarea's md:text-sm does not outsize a tile - name the tile hover group; a bare one revealed every tile's Remove at once - Remove is an icon button on the image, not a word over the artwork - drop the amber tile fill, keep the No caption label - match the header, toggle and status text to the section's type scale Gutters: - the run area and the preview canvas now sit 40px off the rule, the gutter the settings column has off the page edge * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: give the dataset preview per-image hover and remove, and wrap it onto rows - each thumbnail is its own button, so it can carry an X that removes just that image; the strip was one button and could not nest a second control - hover lightens the image instead of drawing a border - wraps onto more rows rather than scrolling sideways, which also keeps the bottom corners rounded - samples 12 thumbnails, up from 8, now that a second row is available - a delete refreshes the panel's dataset counts * Studio: show the workflow icon beside the Images and Video headings Images reads it off WORKFLOW_TABS, so it is the same icon the sidebar submenu shows and it follows the active workflow. Video's heading is also Create, so it takes the Create icon rather than the nav row's film slate. * Studio: reject a duplicate LoRA id on the diffusion load path too DiffusionGenerateRequest already refuses a repeated adapter id, with a comment naming the hazard: _resolve_lora_set suffixes colliding adapter names, so the same id resolves the SAME adapter twice and set_adapters stacks both copies past the per-adapter weight bound. DiffusionLoadRequest bounds only the list length, and it matters more there. Generation-time stacking spoils one image; on the load path the adapters are baked into the quantized build before compilation, so the unintended combination rides every image until a reload. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: teach the GGUF reuse test double about holds_no_vram The resident-GGUF fast path consults llama_backend.holds_no_vram before asserting CHAT ownership, but two tests hand it a SimpleNamespace built before that attribute existed, so they raised AttributeError and returned a 500. A real LlamaCppBackend exposes it as a property; the doubles now carry it too. Product code is unchanged: these pass on main and were the branch breaking its own test, not a defect in the guard. --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: michaelhan <michaelhan2050@gmail.com> Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: Michael Han <107991372+shimmyshimmer@users.noreply.github.com> | 1 个月前 | |
Studio: add image generation, editing workflows and LoRA training with Unsloth GGUFs (#6763) * Tighten comments in the image stack tests and scripts * Close video single-file, training reservation, and image mount-resume gaps Route on-device single-checkpoint video folders through the single_file loader: a bare local .safetensors directory (no model_index.json) is advertised as a pipeline with no filename, so validation rejected it before it could load. Reinterpret the pick as a single_file load of the sole checkpoint, mirroring the image load route. Treat a reserved-but-not-yet-spawned LLM training start as active in is_training_active() so /images/load, /video/load, and /diffusion/start cannot race the reserved run for VRAM during the pre-spawn free window. Mirrors the diffusion training service reservation. Resume an in-flight image generation on the Images page mount: probe generate-progress, re-enter the poll loop, and refresh the gallery on completion so a run started elsewhere is reflected and its saved image appears without a manual refresh. Seed resident image defaults from the resolved base_repo rather than a possibly path-shaped repo_id so the first resident generation uses the right recipe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Publish image generation active state before pre-denoise setup generate() assigned self._gen only at the pipe() call, after deferred compile, LoRA resolution/application, and ControlNet download/build had run. Across that setup window generate_progress() reported inactive even though _generate_lock was held, so a reloaded page's mount probe showed idle and let a second generate queue behind the first. Publish an active step-0 _GenState the moment the generation lock is acquired, before the setup work, and clear it in the outer finally so a setup-time error cannot leave the UI stuck active. Mirrors the video backend's queued phase and the training start guard. * Studio: fix diffusion install ownership, dataset upload atomicity, gallery pagination, and teardown races install_sd_cpp_prebuilt: only write the .unsloth-studio-owned marker when the install created the target directory or it was empty. Adopting a pre-existing, unowned, non-empty directory (a user's own stable-diffusion.cpp checkout) made it eligible for the uninstaller's recursive delete. routes/training upload: make the multi-file promotion transactional. Back up each displaced original and roll every destination back on any failure, so a mid-loop rename error can no longer partially overwrite the live dataset. routes/training _resolve_dataset_folder: reject a symlinked dataset directory and prove the resolved folder stays under the datasets root, so image read/caption/delete cannot escape the root through a link. routes/training delete: escape glob metacharacters in the thumbnail filename so deleting an image named like [ab].png removes only its own thumbnails. image_gallery / video_gallery listing: filter records against the response schema inside the pager via a valid callback, so offset/limit/has_more all count over accepted records. A leading schema-invalid record no longer returns an empty page with has_more=true and stalls infinite scroll at offset 0. image_gallery / video_gallery save: publish via a temp file plus atomic rename (the sidecar is the video pair's commit marker) and clean up on failure, so a partial write never surfaces a truncated PNG or strands an orphan MP4. diffusion_train_common discovery: treat an empty caption sidecar as a metadata tombstone that still falls through to the dreambooth instance prompt, so clearing every metadata caption no longer fails with no captioned images found. diffusion backend unload: wait for an in-flight denoise to exit before tearing down process-wide patches and state, mirroring the load path. diffusion_engine_router: serialize the whole check/unload/publish transition so a concurrent selection cannot return the engine being unloaded. uninstall.ps1: gate the default sd.cpp process stop on the owner marker so a user's own sd-server is not terminated for a directory we then keep. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reject native batch seeds outside the JSON-safe range * Refuse sd.cpp install into unowned non-empty target dir When the install target already exists, is non-empty and lacks the .unsloth-studio-owned marker (a user's own stable-diffusion.cpp checkout, or unrelated files beside a custom Studio root), install() previously still extracted the release into it. Skipping the ownership marker only stopped the uninstaller from deleting the directory; extraction still merged binaries into the user's working tree and could overwrite same-named files. Fail up front with a clear message pointing the user at a fresh/empty location before any download or extraction, leaving their directory untouched. Update the ownership test suite to assert the refusal. * Studio: gate dataset uploads on the symlink check and surface local video single-file checkpoints * Tighten comments and docstrings added by the image-generation fixes * Studio: close arbiter load-registration race and surface native progress + local pipeline folders Publish native sd.cpp generate progress (_gen) before LoRA resolution so a reload probe reads active during setup, matching the diffusers path. Register the diffusion/video GPU load under the arbiter lock (acquire_for now takes a register callback) so a competing acquire cannot evict an owner before its load is marked in-flight and let two loaders allocate VRAM at once. Admit local diffusers pipeline folders (root model_index.json, weights in component subdirs) in the local model scan so they reach task tagging and the On Device picker. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: include marker-owned custom sd.cpp roots in the uninstall stop scan; harden Pester install against the nuget.exe PSGallery bootstrap * Studio: surface local pipeline scan roots, tag single-file checkpoints by filename, mark companion-only pipelines partial - _scan_models_dir: admit a scan folder that is itself a diffusers pipeline (root model_index.json, weights in transformer/ vae/ subdirs). _is_model_directory rejects such a root, so the child scan would list the component subdirs as bogus models and hide the real pipeline; treat the root as one model via _local_pipeline_index. - _local_is_diffusers / _local_model_task: include the sole checkpoint filename in the family-detection needles (_local_family_needles, resolved via resolve_local_single_file). A generically named folder holding one loadable qwen-image-*.safetensors / ltx-*.safetensors identifies its family only from the filename; the load route already resolves that file, so tag it or the task-scoped picker (which rejects task=null) hides the on-device model. - list_cached_models: mark a companion-only base snapshot partial. A GGUF image load prefetches the base repo's VAE / text-encoder / model_index.json but skips the transformer (the GGUF supplies it); the snapshot has a pipeline manifest yet is not a loadable BF16 pipeline, and _cached_repo_partial misses it. _repo_pipeline_missing_denoiser flags a pipeline snapshot whose transformer/ or unet/ component carries no weight, so the picker drops it instead of advertising it as fully on-device. * Studio: preserve foreign gallery files, force safetensors on remote ControlNets, and close dataset/seed/GPU gaps Gallery clear/delete now scope to Studio-owned files: image_gallery and video_gallery skip PNGs / MP4s without a readable recipe (a hand-dropped or orphan file the listing already hides), so clear() and a guessed-id delete no longer destroy files the gallery never surfaced. Remote ControlNets now force use_safetensors: a bare owner/name reaches from_pretrained without the base trust gate, and the Hub scan fails open when unavailable, so requiring safetensors closes the pickle deserialization vector. POSIX uninstall now stops resident sd-server / sd-cli under an owned sd.cpp root before removing the tree (marker-gated), mirroring the Windows stop-before-delete scan; a live native server no longer survives unlinking its binary. Diffusion dataset containment: the training-start read path and the discovery picker route bare names through the protected resolver, so a symlinked dataset is rejected / not advertised like the caption/delete routes already do. Uploads gain the inference decode guard (oversized real images 400 before OOMing the trainer) and dataset upload/caption/delete/import are blocked with 409 while a diffusion run is active. JSONL readers (trainer + routes) tolerate non-object JSON and invalid UTF-8 instead of raising AttributeError / 500. LoRA family compatibility is enforced in the shared resolver, not only the picker, so a direct API client cannot apply a mismatched-family adapter. GPU arbiter gains release_if so the image/video unload idle-check and release are atomic against a concurrent same-owner load's registration. Native batch recipes persist the base batch_seed and restore replays from it, so a native batch_index>0 image no longer advances its seed twice. FLUX.2-klein selects its sd.cpp text encoder by variant (4B -> Qwen3-4B, 9B -> Qwen3-8B) instead of the single family default. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: track the loaded GGUF filename so native companion resolution reproduces the load identity * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: gate local-pipeline image tagging on a real family; validate video sidecars before delete/clear * Studio: tighten image-generation fix comments and docstrings * Studio: gate gallery serve/export on ownership; keep image progress active until persisted; reserve diffusion training before the dataset scan * Studio: restore Reapply target on async image/video load errors; recheck training state before dataset commit; reclaim partial sd.cpp installs on retry Images/Video: a background model load that fails AFTER starting (error/eviction during download) leaves the previous pipeline resident, but handleLoad had already overwritten lastLoad.current with the failed pick, so "Reapply to loaded model" reloaded the failed model. Carry the prior Reapply target into the poller and restore it on the async error/null paths, mirroring the quant rollback. Training: an in-flight diffusion dataset upload passed _require_diffusion_dataset_mutable() at entry but could still commit files after a concurrent /diffusion/start reserved the training slot, mutating the dataset underneath the trainer. Re-check the interlock immediately before the commit phase; a 409 there leaves the staged temps for the finally to clean. sd.cpp install: an interrupted extraction (disk full, killed process, a raising post-extract cudart fetch) left the target non-empty with no owner marker, so the next lazy install tripped the "not a Studio-managed directory" refusal and wedged native install. Write the ownership marker before the partial writes when the target is reclaimable, so a retry recognises the debris as ours and re-extracts. * fp8 DiT quant: floor the dynamic activation scale with activation_value_lb An all-zero activation token row makes the dynamic per-row fp8 scale 0, which turns the quantized data to NaN and the render to black frames on torchao's plain-torch kernel path. The fused fbgemm/mslk quantize kernels clamp zero rows internally, so the bug only reproduces on machines without them, which is most user environments. Zero rows are real inputs, not a corner case: Wan 2.2 zero-pads its text conditioning, and Hunyuan-1.5 and Qwen-Image regenerate zero rows inside their transformer blocks every step. Pass activation_value_lb=1e-12 to Float8DynamicActivationFloat8WeightConfig whenever the installed torchao supports the kwarg (Float8Tensor rework, 0.13+), checked via inspect.signature so older torchao keeps exactly the current behaviour; the existing Float8MMConfig fallback chain is unchanged. Verified on GPU: with the forced plain-torch kernel path a zero-row input NaNs without the floor and stays finite with it, and end to end on HunyuanVideo-1.5 fp8 goes from a solid black frame (LPIPS 1.00) to a normal render (LPIPS 0.225); on Wan the floor matches the condition_embedder exclusion (LPIPS 0.211 vs 0.206). Same-seed renders with fused kernels present are unaffected, and pre-quantized fp8 checkpoints stay valid since weight scales are untouched. * Wire hosted pre-quantized DiT checkpoints into the image families Point prequant_repos for flux.1, flux.2-klein, flux.2-dev, qwen-image (int8 only there; fp8 is family-denied), z-image and krea-2 at the unsloth/<Model>-FP8 Hub repos carrying gate-validated int8 and fp8 transformer checkpoints, so the fast quant path loads the small pre-quantized file instead of materialising the dense bf16 transformer and quantising on device. Measured on FLUX.2-dev int8: build peak drops from 60.7 GB (dense + quantize) to 30.7 GB (hosted prequant), identical 30.7 GB resident after either path since loading a checkpoint is bit-identical to on-the-fly quantisation. The hosted repos name files <Model>-<SCHEME>.pt, so resolve_prequant_source now derives that model-name filename from the repo id (scheme suffix stripped case-insensitively) and carries the legacy transformer_<scheme>.pt as a fallback the resolver tries when the primary 404s, keeping older repos loadable. Wiring a repo also exposed a fallback hazard: with a prequant source present, the dense-fit preflight used to be skipped entirely, so a failed prequant download would fall through to the dense bf16 load the memory plan never budgeted, OOMing after eviction. The preflight now always runs and gates an allow_dense_fallback flag through _load_dense_quant_pipeline: a dense misfit still skips the fast path when no prequant exists, but with one it proceeds and a prequant failure raises to the GGUF build instead of loading dense. The same flag is set when the auto-policy replans an offloaded GGUF against a prequant-sized transient. Tests updated to the new filename convention plus new coverage for the derivation and the legacy-name fallback; the prequant-skips-refit test now asserts the re-check runs and forbids the dense fallback. Verified end to end on GPU: z-image int8 resolves the hosted repo, downloads the model-name file and renders (6.8s load, 5.9 GB peak). * Route krea-2 through its per-component loader on the transformer-quant fast path _assemble_pipe used Pipeline.from_pretrained for every family, but the krea repo ships transformers-5.x configs and no top-level tokenizer files, so the tokenizer dies with vocab_file=None. The pre-quantized checkpoint loaded fine and then the assembly crashed, dropping the load to the GGUF build, which krea-2 cannot take (Krea2Transformer2DModel has no from_single_file). Assemble per-component via load_krea2_pipeline like the pipeline-kind and single-file paths already do. Verified live: Krea-2-Turbo int8 and fp8 hosted prequant loads now assemble and render through the Studio images tab. * Keep Qwen-Image's text-stream linears bf16 on int8 (short prompts break torch._int_mm) Qwen-Image's MMDiT runs every text-stream Linear at M = actual prompt tokens: the Qwen2.5-VL embeds are not padded to a fixed length like FLUX's 512-token T5. A short prompt (13 tokens) or the near-empty negative prompt drives torch._int_mm below its M > 16 floor and the first denoise step raises 'self.size(0) needs to be greater than 16, but got 13' (measured on B200 through the Studio images tab). Add per-family int8 exclusions (txt_in, add_q/k/v_proj, to_add_out, txt_mlp) for qwen-image and qwen-image-edit, threaded through exclude_tokens_for_scheme(scheme, family) and the prequant checkpoint validation, so a checkpoint baked under the old token list is rejected and re-quantised instead of loaded crashing. The text stream runs at M = tens vs the image stream's M ~ 4k, so the exclusion costs nothing; the rebuilt hosted checkpoint gates 28/28 PASS with LPIPS mean 0.057 (was 0.069). * Harden the diffusion memory plan against transient free-VRAM undercounts A cold FLUX.2-dev int8 load on an idle 183 GB B200 planned offload=model (companions exceed budget) and silently served the GGUF as-is; the identical retry went resident and engaged the hosted prequant. The plan arithmetic was byte-identical across both loads (required 90,228 MiB, resident needs free of about 124 GB); the only divergent input was torch.cuda.mem_get_info, which is device-wide and instantaneous: a transient foreign CUDA context briefly held about 100 GB at the first snapshot, and the planner trusted that single read. Three changes: - settled_snapshot_device_memory: on cuda, synchronize + empty_cache (best-effort) and take the MAX free over up to 3 spaced reads. A transient can only shrink free, so the max rejects transient undercounts while a persistent tenant still caps every read. _plan_memory now uses it. - plan_fits_total_capacity + one replan retry: when the dense/prequant candidate fits TOTAL device capacity under the standard reserve and the 0.85 resident margin, an offload verdict can only stem from the free reading, so the loader re-snapshots and replans once before declining the fast path. Explicit balanced/low_vram modes skip the retry (they offload by mode). - diffusion.transformer_quant_declined log line with required/budget/free and the plan reasons, so the next decline is diagnosable from the server log (previously silent). Verified: cold FLUX.2-dev int8 first load in a fresh server now engages the hosted prequant resident (offload=none). * Add FLUX.2 Klein and FLUX.2-dev DiT LoRA training Register flux.2-klein and flux.2-dev in the DiT trainer following the upstream DreamBooth references: latents train patchified and batch-norm normalized from the VAE posterior mode, the packed forward reuses step-invariant position ids, and the guidance vector (3.5) is gated on the variant's guidance_embeds config. Conditioning stacks load per variant (Mistral via Flux2Pipeline for dev, Qwen3 via Flux2KleinPipeline for Klein) and are encoded and freed before the transformer lands on the device. The fused single-stream to_qkv_mlp_proj joins the attention projections in the LoRA targets; the single-stream out projection stays dense because its to_out suffix would also match the double-stream ModuleList container. Wire both families through the training registry (family set, labels, VRAM notes, rank 16 / lr 1e-4 defaults, bf16-only preflight), mark them trainable with train base repos in the family registry, add FLUX.2-dev to the gated-repo token check, and trust both official bases for training downloads. Verified on B200: 30-step klein int8 (19.6s) and nf4 (20.9s) and dev int8 (52.0s) runs train with finite decreasing loss and the saved adapters apply on the bf16 base pipeline (weight 0 reproduces the base image exactly, weight 1 visibly restyles it). * Support LoRA adapters on torchao int8/fp8 quantized image pipelines Adapters are baked at load time: they attach to the dense transformer, then quantize_ converts only the frozen base linears (the lora_ side path is excluded by name), then the loader compiles. Post-quant PEFT injection is not possible on a manually quantized module, so the prequant shortcut is skipped for a baked load and the memory plan is sized for the dense build (force_dense on the quant candidate). At generation time the baked topology is frozen: weight tweaks and disabling (scale 0 reproduces the quantized base exactly) go through set_adapters, while adding or removing adapters returns a clean 400 telling the client to reload with the new selection. supports_lora now returns True for int8/fp8 diffusers loads (checked before the gguf-kind early return, since the quant fast path keeps the picker kind); nvfp4/mxfp8 and GGUF-via-diffusers stay blocked. The load request model takes an optional loras list, threaded through begin_load on both engines (native ignores it and keeps applying LoRA at generation). Verified end to end on GPU: Z-Image GGUF picker + int8 + trained adapter loads through the API, bake marker logged, weight 1.0 vs 0 renders differ visibly, weight 0.5 accepted live, unknown adapter rejected as 400. Affected suites: 296 passed. * Add FLUX.1 Krea dev to the image model catalog Krea's guidance-distilled FLUX.1-dev finetune keeps the exact dev layout, so it runs under the existing flux.1 family unchanged. Wire it up end to end: - Catalog group with the gated official bf16 pipeline and the open QuantStack GGUF quants; the gated artifact is skipped on auto-routing when undownloaded. - Trust the official repo for non-GGUF from_pretrained loads, next to the other black-forest-labs bases. - Generation defaults: 28 steps at guidance 4.5 per the model card. The generic "krea" defaults key (Krea-2-Turbo's 8-step no-CFG recipe) used to swallow the id, which would have produced garbage output; the new flux.1-krea key precedes it on both the backend table and the images page table. - The flux.1 prequant checkpoints are schnell-based; the loader's baked base_model_id validation refuses them for the Krea-dev base, so int8/fp8 requests dense-quantize instead (covered by existing prequant tests). * Resolve pre-quantized checkpoints per base variant One family entry covers several published variants whose weights differ (flux.1: schnell, dev, Krea-dev), but prequant resolution was keyed on (family, scheme) alone, so only the default base could ever be served: the loader's baked base_model_id validation correctly refused the schnell checkpoint for dev and Krea-dev bases and every such load paid the dense download plus on-the-fly quantise. Add an optional prequant_variant_repos table on DiffusionFamily as (base_repo, scheme, repo_id) triples and thread the resolved base repo through resolve_prequant_source / usable_prequant_source and their three call sites (load fast path, memory-plan probe, auto-policy candidate). A base without its own entry keeps returning the family default, preserving the existing refuse-then-dense behavior exactly. Wire the flux.1 variants: the gate-validated unsloth/FLUX.1-dev-FP8 checkpoints (built in the earlier campaign but never reachable) and the new unsloth/FLUX.1-Krea-dev-FP8. * Add the Lumina Image 2.0 family to the image catalog Alpha-VLLM/Lumina-Image-2.0 is a 2.6B single-stream DiT with a Gemma2-2B encoder and a standard 16-channel VAE, all transformers-4.x-compatible, so the generic from_pretrained pipeline path loads it as a new lumina-2 family: - Family entry (Lumina2Pipeline / Lumina2Transformer2DModel), aliased to lumina-image-2.0 / lumina-image-2 / lumina2. No bare lumina alias: Lumina-Next checkpoints are a different arch and must stay unknown rather than crash mid-load. bf16-only upstream, so the fp16 fallback stays off like z-image. - Trust the official repo for non-GGUF loads; bf16 component table entry (ships fp32, ~5.2 GB transformer + 5.2 GB encoder bf16-resident). - Generation defaults 50 steps / guidance 4.0 per the model card, and the generate call passes the card's cfg_trunc_ratio=0.25 itself (family-gated, signature-gated): the pipeline default (1.0) runs the CFG double-forward on every step and oversaturates output. - Catalog group with the single ungated bf16 pipeline artifact (11 GB resident) plus routing assertions; images page defaults row. - No GGUF artifact: none exists upstream (only finetune/LLM quants), so the dense transformer_quant fast path (GGUF-kind-only) stays unreachable for now. Offline probes of the future prequant campaign: int8 and fp8 both engage and render cleanly (fp8 LPIPS 0.11 vs bf16, int8 0.33 from 50-step trajectory drift with intact quality), so neither scheme is family-denied. * Wire the hosted Lumina Image 2.0 int8/fp8 checkpoints Gate-validated against same-seed bf16 renders (28/28 pairs per scheme, zero failures): int8 LPIPS mean 0.146 / SSIM 0.937, fp8 LPIPS mean 0.116 / SSIM 0.946. Uploaded to unsloth/Lumina-Image-2.0-FP8 following the existing checkpoint repo conventions. * Add the HunyuanImage 2.1 family to the image backend The hunyuanvideo-community diffusers mirror carries the full stack in standard layout: a 17B dual-stream DiT (32.5 GB bf16), a Qwen2.5-VL text encoder, a ByT5 glyph encoder, the 32x HunyuanImage VAE, and guider/ocr_guider components (AdaptiveProjectedMixGuidance) that diffusers 0.39 loads natively, so the generic from_pretrained pipeline path covers everything with no per-component assembly. Family notes: - The call's guidance knob is distilled_guidance_scale (there is no guidance_scale kwarg), so cfg_kwarg routes the UI value there; real CFG runs inside the repo's guider at its baked scale. Defaults follow the card recipe: 50 steps, 3.25. - 2K-native: verified live at both 1024 and 2048. - Coexists with the HunyuanImage-3.0 structured exclusion (3.0 has no diffusers pipeline and stays excluded with its stated reason). - int8/fp8 dense quantization verified live (LPIPS 0.186 both vs same-seed bf16); a short prompt does not trip the int8 torch._int_mm minimum on this arch, so no family exclude entry is needed. - bf16 component table for the memory planner: (32.5, 16.3, 0.8) GB. * Surface HunyuanImage 2.1 in the image model catalog Catalog group with the open bf16 mirror pipeline (~50 GB resident, so a bare click on a consumer card routes to the QuantStack GGUF quants, which load and render through the generic GGUF path, verified live) plus the images page defaults (50 steps, guidance 3.25 feeding distilled_guidance_scale). * Add the HiDream-I1 family to the image backend A 17B MoE DiT (16 double + 32 single layers, 4 routed experts) with four text encoders, on HiDreamImagePipeline (diffusers 0.39). One family covers the open Full / Dev / Fast repos (same arch); per-variant generation defaults follow the upstream inference recipes (Full 50 steps at guidance 5, the distilled Dev 28 and Fast 16 guidance-free). The repos name a Llama-3.1-8B text_encoder_4 in their model_index but do not ship its weights; the official example passes the gated meta-llama repo in by hand. The loader instead assembles the component from the open unsloth mirror (byte-identical weights, already inside the non-GGUF trust gate), injected at the three pipeline from_pretrained sites, with output_hidden_states matching the official example. Memory planning counts the assembled TE4: 34.2 GB DiT + 28.8 GB encoders, ~63 GB bf16-resident. * Surface HiDream I1 in the image model catalog One catalog group with the three official bf16 pipelines (Full, plus the Dev and Fast distillations as labeled artifacts) at their ~63 GB resident size, so auto-routing keeps this a datacenter-GPU pick. city96's GGUF is deliberately not wired: the GGUF path would need the same Llama TE4 assembly for very small demand. Images-page defaults mirror the backend table with the variant keys ahead of the generic hidream key. * Pin the measured HiDream quant verdict in tests int8 and fp8 both engage and render cleanly on this family, including short prompts on int8: the routed MoE expert Linears only ever see the concatenated image+text stream (M far above the torch._int_mm minimum), so no deny entry and no family exclude tokens are warranted. Assert that so a future table edit cannot silently regress the measured behavior. * Wire the hosted HunyuanImage 2.1 int8/fp8 checkpoints Verified bit-identical to on-the-fly quantize: all 1264 state dict tensors (456 quantized) dequantize equal between the loaded checkpoint and a fresh quantize_ pass, so quality matches the runtime Dtype path exactly. Same-seed LPIPS suite means (0.35 int8 / 0.28 fp8) blend trajectory divergence with this family's own run-to-run nondeterminism (identical weights and seed reproduce a 17/255 mean pixel delta through the 50-step guider pipeline); per-case hard checks pass and the drift is compositional, reviewed visually. Uploaded to unsloth/HunyuanImage-2.1-FP8. * Fix silent LoRA drop and wasted transformer prefetch on GGUF quant loads Two live-test findings on the images load path: - transformer_quant with baked LoRAs, when the dense quantized build is declined for memory or fails: the load completed as a plain GGUF with the adapters silently dropped (HTTP success, supports_lora=false after the fact) -- wrong output with no signal. The load now fails with the recovery options (drop the adapters, free VRAM, or pick a smaller model). Weight-0 adapters still count as no bake request, and the plain no-LoRA decline keeps its silent GGUF fallback. - A fresh GGUF load on a small GPU prefetched the base repo's full bf16 transformer shards (~47 GB on Qwen-Image) because the dense-quant prefetch widening only checked scheme viability, not whether the device could ever hold the candidate resident. Gate the widening on total device capacity (reserve + 0.85 margin, the plan_fits_total_capacity bar) so a card that is certain to decline the dense build never pays the download; capable devices keep the prefetch. * Fix video progress under-reporting during load and generate Two live-test findings on the video progress endpoints: - load-progress downloaded_bytes froze mid-download: the counter used scan_cache_dir, which skips in-flight *.incomplete blobs, so it sat at the last completed blob for the whole multi-GB shard pull while the disk kept filling. Count the repo's cache directory directly (completed plus incomplete blobs, snapshot symlinks skipped so nothing is double-counted). - generate-progress reported total_steps=null / fraction=0 while step advanced: the video API only carried the native total field while the image API exposes total_steps and fraction, so one poller could not work against both. Derive the image-compatible aliases in generate_progress and declare them on the response model; the native total stays for back-compat. * Wire the hosted HiDream I1 int8/fp8 checkpoints Gate-validated: all 28 per-case pairs pass per scheme (LPIPS suite means 0.291 int8 / 0.278 fp8, in the 50-step trajectory-divergence band; CLIP delta means 0.007-0.008), and the int8 checkpoint is verified bit-identical to on-the-fly quantize across all 1615 state dict tensors (1073 quantized, max abs diff 0.0). Uploaded to unsloth/HiDream-I1-Full-FP8. * Add a pre-cast text-encoder loader for the layerwise fp8 scheme The runtime text_encoder_quant=fp8 path downloads the full bf16 text encoder and layerwise-casts it in place on every fresh load. For the heavyweight encoders (LTX's Gemma3-27B ~50 GB, FLUX.2-dev's Mistral-24B ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) that download dominates load time on a fresh machine. diffusion_te_prequant.py loads a pre-cast fp8-storage state dict instead: meta-init the encoder skeleton from the checkpoint's te_class, load_state_dict(assign=True), rebuild on CPU if non-persistent buffers stay on meta, then re-apply the same layerwise cast to install the upcast hooks. The cast is a deterministic storage transform, so the loaded encoder is bit-identical to dense-load-then-cast by construction. v1 hosts the layerwise fp8 storage scheme only: its state dict is plain tensors (torch.load(weights_only=True), no pickle execution). The dynamic-compute schemes (fp8_dynamic, int8, nvfp4) build torchao subclass wrappers at runtime and are deliberately not hosted. Checkpoints validate format, scheme, component and base_model_id before use and any problem falls back to the dense download and cast. Local path overrides reuse the DiT prequant allowlist env var. Families opt in via a new te_prequant_repos (scheme, component, repo_id) field on both DiffusionFamily and VideoFamily; the field defaults empty so nothing changes until a gate-validated artifact is wired. * Inject hosted pre-cast text encoders during pipeline assembly Wire te_prequant_pipe_kwargs into the three pipeline assembly sites: the diffusion full-pipeline branch, the diffusion transformer-only and GGUF branch (where the companion TE is the big remaining download), and the shared video assembly path before the pipeline/component split. Injection is gated exactly like the runtime cast (mode normalized to fp8, device supported, family not denied), so it can never engage where quantize_text_encoders would not; the later quantize_text_encoders call re-applies the cast idempotently and keeps status reporting truthful. With no hosted checkpoint configured the call returns {} and assembly loads the dense encoder as before. * Add the pre-cast text-encoder checkpoint builder Applies the runtime layerwise fp8 storage cast to a model's dense text encoder once and saves the cast state dict with baked metadata (format tag, base_model_id, family, scheme, component, te_class, versions) in the layout diffusion_te_prequant.py validates. Resolves the encoder class from the checkpoint's config.architectures so the recorded te_class matches what the pipeline instantiates. CPU-runnable: the cast touches storage dtypes only. * Test the pre-cast text-encoder load path Hermetic CPU coverage for diffusion_te_prequant: the checkpoint filename convention, family-table resolution by scheme and component with malformed entries skipped, resolution priority (path override, hosted repo, none) and the fp8-only scheme gate, the checkpoint validation matrix (wrong format, missing state_dict, wrong scheme, wrong component, wrong or missing base_model_id) with base case folding, the local-path allowlist refusal and missing-file fallback, and the assembly injection gating (mode, hosted entry, device support, family deny, load failure, successful injection). Also pins the te_prequant_repos field on both family dataclasses and that no family ships a hosted TE checkpoint until the campaign wires one. * Fix pre-cast TE checkpoint loading and engagement reporting Two bugs found while building the hosted checkpoints: - The builder recorded torch.__version__ (a TorchVersion object) in the checkpoint metadata, so torch.load(weights_only=True) rejected every artifact and the loader silently fell back to the dense download. Record plain strings. - Re-applying the layerwise fp8 cast to an injected pre-cast encoder raised on the duplicate hook registration, making quantize_text_encoders report the engaged cast as failed (status showed no TE quant while the encoder ran fp8). _cast_fp8 now returns early when the hooks are already installed. Also corrects the LTX TE size note: Gemma3-12B stored fp32 (~49 GB), not 27B. * Wire the hosted pre-cast fp8 text encoders qwen-image and flux.2-dev (diffusion) and ltx-2 (video) now resolve a hosted pre-cast fp8 text encoder from their unsloth -FP8 repos: - unsloth/Qwen-Image-FP8: Qwen2.5-VL-7B, 16.6 GB dense -> 8.8 GB - unsloth/FLUX.2-dev-FP8: Mistral-Small-24B, 48.0 GB dense -> 24.7 GB - unsloth/LTX-2-FP8: Gemma3-12B, 48.7 GB fp32 store -> 13.2 GB Every checkpoint verified bit-identical to dense-load-then-cast (729 / 585 / 1066 tensors, zero mismatches) and smoke-tested through the real backends with the repo engagement marker. Tests cover the wired entries, the resolver filenames, builder metadata weights_only survival, and the idempotent re-cast. * Report the compute dtype on fp8-cast encoders and inject the pre-cast TE on the dense fast path Two more findings from the hosted-TE GPU smokes: - Module.dtype reports the first floating parameter, which after the layerwise fp8 cast is the fp8 STORAGE dtype. Flux2 derives its prompt embed and latent dtypes from encoder.dtype and feeds them to randn_tensor, which has no fp8 kernel, so ANY flux.2 load with text_encoder_quant=fp8 crashed at generation (pre-existing, runtime cast included). The cast now swaps in a subclass whose dtype property reports the compute dtype; forward behaviour is unchanged. - The dense transformer_quant fast path assembles companions through _assemble_pipe, which never received the pre-cast TE injection, so the hosted encoder engaged on full-pipeline and GGUF builds but not on the fast path. Threaded through like the other two branches. Verified live on B200: qwen-image (full pipeline), flux.2-dev (GGUF picker with int8 DiT prequant), ltx-2 (video backend) all engage the hosted TE, render non-black, and report text_encoder_quant=fp8 truthfully. * Key the fp8 cast idempotency on an explicit completion marker Hook presence alone cannot distinguish a legitimately pre-cast text encoder from leftover hooks after a cast that failed mid-pass, so the early return now requires the completion marker _cast_fp8 sets once the hooks are fully installed. Leftover partial state keeps failing closed. Also tolerates non-Module encoder doubles in the hook probe and the dtype override. * Extend the fp8 TE quant to HiDream's Llama text_encoder_4 The generic quantize_text_encoders pass only covers text_encoder.._3, so HiDream's HEAVIEST encoder (Llama-3.1-8B TE4, 16.1 GB bf16) always stayed dense. TE4 is assembled separately (hidream_te4_kwargs), so the fp8 path now lives there: when the requested TE quant is layerwise fp8 and the device/family qualify, TE4 prefers the hosted pre-cast checkpoint (unsloth/HiDream-I1-Full-FP8, 8.6 GB) and falls back to dense-load-then- cast; a mid-pass cast failure reloads a fresh dense encoder instead of shipping partial state. The pre-cast loader and builder gain config_subfolder/config_overrides for standalone encoder repos whose config sits at the root and whose pipeline needs forward flags (output_hidden_states/attentions). Verified on B200: bit-identity 291 tensors (225 fp8, 0 mismatches), hosted checkpoint engages through the real backend (marker + status fp8), load 24.3 s vs 48.0 s dense, LPIPS 0.133 mean over 3 same-seed pairs vs the dense-TE render (gate 0.25), non-black frames. * Correct the ltx-2 resident TE estimate to the bf16 cast size The memory plan's bf16_components_gb held 50.4 GB for the LTX text encoder, which is the fp32 hub store of Gemma3-12B (~49 GB download), not what sits on device: the pipeline loads it torch_dtype=bf16, ~24.4 GB resident. The 26 GB over-estimate pushed the auto plan toward offload on cards that fit the real footprint. Comments and the size-table test now pin the resident semantics. * Host pre-cast fp8 text encoders for four more families Round 2 of the hosted TE set, each bit-identical to dense-load-then-cast and gated through the real backend (marker + status fp8 + same-seed LPIPS vs dense TEs): - FLUX.1 T5-XXL (text_encoder_2): 9.52 -> 5.90 GB, one artifact for schnell/dev/Krea-dev (T5 shards byte-identical across all three, verified sha256). 220 tensors, 144 fp8, LPIPS 0.109. - Lumina Gemma2-2B: fp32 hub store 10.46 -> 3.20 GB (3.3x download cut). 288 tensors, 182 fp8, LPIPS 0.041. - Z-Image Qwen3-4B: 8.04 -> 4.41 GB. 399 tensors, 252 fp8, LPIPS 0.112. NOT shared with flux.2-klein-4B: klein retrained layer 35's MLP (verified tensor diff, maxdiff 0.86), so klein hosts no entry. - Krea-2 Qwen3-VL-4B: 8.88 -> 4.83 GB. 713 tensors, 460 fp8, LPIPS 0.082. The constructor-assembled krea pipeline takes the encoder directly (load_krea2_pipeline text_encoder kwarg); the loader remaps 5.x rope_parameters and re-ties weights after assign so the rebuilt encoder matches the builder's structure. HunyuanImage 2.1 reuses the Qwen-Image artifact outright: its Qwen2.5-VL text encoder is byte-identical (every shard sha256, 16,584,414,544 bytes), recorded in the new component-level base-equivalence table the checkpoint validator consults. The injection loop now covers text_encoder.._3 so a family can host several components. Live check: LPIPS 0.123 vs dense. * Report the fp8-cast compute dtype without swapping the encoder class The dtype override swapped encoder.__class__ to a dynamic subclass, which breaks transformers' kwargs-based output recording: a fp8-cast Qwen3VLModel stopped returning hidden_states and every krea-2 generation with text_encoder_quant=fp8 crashed at encode_prompt (regression from the HiDream TE4 change; caught by the krea hosted-TE live smoke). The override is now a property shadowed on the ORIGINAL class that prefers a per-instance compute-dtype attribute, so class identity is preserved and uncast instances keep the stock behaviour. The idempotency test now pins exact class identity and the uncast-sibling fallback. * Pass the calibrated distilled sigma curve to LTX-2.3 8-step runs The 22B distilled DiT was trained against ltx_core's fixed DISTILLED_SIGMA_VALUES, but the diffusers scheduler derives 8-step spacing from resolution-shifted flow matching and lands far off at every reachable mu (second sigma 0.945-0.981 vs 0.99375, tail 0.37-0.61 -> 0.1 vs 0.725 -> 0.42 -> 0). At the distilled default step count the backend now passes the list verbatim, neutralising the scheduler's dynamic shift and terminal stretch for the call (they distort even explicit sigmas) and restoring them afterwards. Other step counts and the dev/base DiT keep the scheduler's own spacing. Live-verified on B200 through the video branch backend: the scheduler holds the exact curve after an 8-step distilled GGUF generation, config restored, healthy clip. Also reword the transformer_quant resolved reason to the measured reality: quant halves resident weights and hosted checkpoints cut load time, while per-step speed is roughly bf16 parity. * Pin the fp8 weight-quantize kernel against silent MSLK switching torchao's Float8Tensor KernelPreference defaults to AUTO, which switches the weight-quantize kernel to MSLK whenever an mslk package is importable on sm90+. Measured on B200: that changes fp8 scale rounding bitwise (8/8 FLUX matrices differ, scales ~55 percent of bytes), so a box that merely gains mslk would break the hosted-prequant bit-identity invariant; the mslk path is also slower under torch.compile (opaque extern call blocks inductor's quantize fusion, FLUX.1 fp8 e2e 1.149 to 1.624 s). Pin KernelPreference.TORCH explicitly, matching current no-mslk behaviour bit for bit; signature-gated for older torchao. GPU-smoked (finite, rel err 0.037) and pinned by test. * Shift Qwen-Image training sigmas to the inference distribution Qwen-Image's scheduler skips its static shift under use_dynamic_shifting, so the DiT trainer was drawing UNSHIFTED uniform-schedule sigmas for it (mean sigma 0.50) while inference always runs the exponential mu = log 3 shift plus the shift_terminal 0.02 stretch. Add a flow_shift config lever: "auto" (the new qwen-image default) rebuilds the training sigma table through the scheduler's own time_shift and stretch_shift_to_terminal so the draw matches the inference distribution exactly (mean sigma 0.72); a numeric value applies the standard linear shift s*u/(1+(s-1)*u); 1.0 keeps the historical identity behavior and stays the default for FLUX, Z-Image and Krea 2. The model timestep conditioning follows the shifted sigma, gathered in fp32 so bf16 rounding never skews it. Also wire two opt-in levers with off defaults: cfg_dropout (per-sample empty-prompt conditioning dropout, encoded alongside the captions before the text encoders are freed) and weighting_scheme="bell" (bsmntw-style mid-schedule Gaussian loss weighting normalized to mean 1). Verified with two 80-step rank-8 bf16 LoRA runs on Qwen/Qwen-Image (identity vs auto, same seed): both converge with finite decreasing loss and produce coherent same-seed previews. Unit tests cover the exact transform, the shifted sampling distribution, per-family defaults and config plumbing. * Add LoRA EMA, a persistent conditioning cache, and aspect bucketing helpers diffusion_train_extras hosts the opt-in training extras: LoRAEMA shadows only the trainable adapter params (warmup-ramped decay, default 0.99, exported as a second adapter under output_dir/ema), PersistentConditioningCache stores latent posterior stats and caption embeddings as safetensors keyed by content hash + family + resolution, and the aspect-ratio bucketing helpers group mixed-aspect datasets into same-area divisor-snapped shapes. The DiT trainer wires the first two behind config flags that default to the current behavior: ema_decay (0 disables) and cond_cache_dir (None disables). A fully warm cache skips loading the VAE and text encoders entirely; a cache hit is bit-identical to a fresh encode, including the per-channel qwen latent normalization. Also fixes the stale _gather_sigmas call in the perf test that still passed the scheduler instead of the sigma table. * Tighten torchao configs and note the FSDP2 design for the DiT trainer nf4 loads now enable double quantization (~0.4 bits/param off the frozen base scales at no fidelity cost), fp8 training uses the rowwise recipe when the torchao build ships it (per-row scaling confines the DiT activation outliers that a tensor-wide scale collapses), and the inference quant filter gains a per-scheme GEMM-tiling divisibility floor (16 for scaled_mm, 32 for MX blocks) so one ragged Linear cannot crash the first denoise after a clean quantize pass. plans/fsdp2_diffusion_design.md records the multi-GPU design: bf16/fp8 over FSDP2 with per-block units, LoRA attached before sharding, int8 out of scope (DTensor over the quantized subclass is undefined), per-family notes. * Batch diffusion inference with per-image seeds, an inference conditioning cache, and GGUF loader fixes Batched generation: /images/generate takes a prompts list (one image per prompt, txt2img only) or a seeds list (one prompt, one image per seed); the legacy batch_size path derives per-image seeds base..base+n-1 like the native engine. Every image gets its own torch.Generator so any batch member replays alone from its gallery recipe; the whole list runs as one forward by default with OOM backoff that halves a failed chunk, and an explicit batch_size caps images per forward. Validated 10-22x over serial engines on 32-image suites with LPIPS deltas within 0.002. Conditioning cache on the inference path: UNSLOTH_DIFFUSION_COND_CACHE_DIR (the inference sibling of the trainers' cond_cache_dir, same persistent store) wraps encode_prompt so repeated prompts skip the text-encoder forward entirely; verified bit-identical outputs. Bypassed while LoRA adapters are attached; tensor-argument calls pass through uncached. Compile cache: GGUF loads fingerprint their own bundles (quant=gguf, a different compiled graph than the dense family) and batched calls register every distinct (w, h, batch) chunk shape they ran, so the heavy GGUF batched warmups (~159 s at batch 32 on 12B-class, ~655 s on 20B CFG-batched) are paid once ever. GGUF loader: strip the sd.cpp model.diffusion_model. container prefix in the single-file converter; diffusers' FLUX.2 converter KeyErrors on it and the Qwen-Image identity mapping strands the model on meta. * Correct batched seed-replay docs to match measured behavior Same-seed images at the same batch shape are bit-identical; a solo regeneration with the recorded seed matches its batched rendition up to batch-size-dependent kernel numerics (mean abs pixel delta about 2.5/255, LPIPS delta under 0.002), not bit-exactly. The previous wording overclaimed bit-identity across batch shapes. * Note that batched bit-identity assumes a settled compiled graph The first generation issued while the deferred compile is still in flight can deviate transiently (observed once on a cold fp8 build: mean abs pixel delta 0.063/255); once the graph is settled, same-seed same-batch-shape images are bit-identical across runs. * Studio sidebar: Image03/FlimSlate icons, More flyout, Train row, New pills - Images uses Image03Icon and Video uses FlimSlateIcon. - New "More" row (MoreHorizontalIcon) opens a right-side flyout on click or hover holding Video, Recipes and Export; the close is delayed 180ms so the pointer can cross the gap. Its SidebarMenuButton deliberately takes `title` rather than `tooltip`: with `tooltip` the button returns a Tooltip root and DropdownMenuTrigger asChild would hand its ref to a non-DOM node. - Dropped the "Train" section heading; Train is now a top-level row between Images and More. data-tour="navbar" moves to the surviving nav group so the product tour keeps its anchor. - "New" pill beside Images and (inside the flyout) Video, via NavBadge. * Studio sidebar: match flyout rows and New pills to the existing scales - More flyout rows dropped their sidebar-row typography and size-icon override, which fought DropdownMenuItem's own scale (text-sm, gap-2.5, px-3 py-2 and size-4 icons) and rendered oversized glyphs and text next to the nav. - New pill reuses the brand "beta" badge recipe (nav-badge font, --ui-font-scale sizing, nav token colours) rather than hardcoded 9px values. - The More row's native title tooltip (an OS box on hover) is replaced by the app's Tooltip, wrapped around DropdownMenuTrigger so both triggers compose onto the same button, and shown only on the collapsed rail like other nav rows. * Settings: pin and reorder the sidebar navigation Adds a "Sidebar navigation" section to Settings -> Appearance, above the existing profile-menu customizer, with the same drag-to-reorder + switch UI. - New sidebarNav preference: one { id, pinned } entry per navigable row (projects, hub, images, train, video, recipes, export), array order = render order. Defaults match the shipped layout, so an untouched install is unchanged. - Unpinning moves a row into the More flyout rather than hiding it, so no page becomes unreachable. New chat and Search stay fixed as actions. - app-sidebar now renders from one navRows descriptor map, so a pinned row and its flyout counterpart cannot drift; the More row appears only when something is unpinned and highlights off whatever it actually holds. - Mirrored in the backend PersonalizationCustomization: without it the model's extra="ignore" would drop the field, and because sync replaces local state with the server's copy once customization is saved, the user's pin order would reset on the next sync. The validator dedupes and back-fills like sidebarMenu but preserves the client's order, since here order is meaningful. Frontend typecheck, i18n parity and catalog checks pass; 32 personalization tests pass, including a round-trip asserting a reordered list survives a save. * Sidebar customizer: drop the Search row, skip More for a lone item - Search is reached from the top bar, so it is no longer previewed as a fixed sidebar nav row; New chat stays. - More now appears only when it would hold two or more rows. A single unpinned row renders inline in its saved order position instead: a flyout wrapping one item costs a click and earns nothing. The customizer's More preview follows the same threshold. * Sidebar settings: hide a lone unpinned tab, match New chat icon, rename Profile menu - With exactly one tab unpinned, both More and that tab are dropped, so nothing is drawn for it (previously it rendered inline). The page stays reachable by URL. - The customizer's New chat preview uses PencilEdit02Icon, the icon the real row renders; Edit03Icon was a different glyph. - "Sidebar menu" is now "Profile menu", described as the shortcuts behind your name at the bottom of the sidebar, so it no longer reads as a second name for the navigation section above it. * Tighten comments in the new sidebar and delete-guard code * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio sidebar: keep the More row highlighted while its panel is open Moving the pointer into the flyout left the row unhighlighted while the panel stayed open. The row now carries data-menu-open, added to the nav hover selector list. Not data-state: the tooltip and menu triggers both write that attribute, so whichever lands last wins. * Images: use the shared pill toggle for Create/Train and pad the panels - Create/Train was the only segmented control on its own Tabs styling. It now uses PillTabs, the same control as the model picker and Hub toggles, pinned to the header row's 34px. PillTabs takes an icon per tab, so the inline-span workaround for TabsTrigger goes away. - pt-3 on both the Create and Train panels, which sat flush against the model selector row. * Images: make the workflow picker a dropdown instead of a 7-up strip Seven workflows in a 340px rail left ~48px each, so the labels crowded and the hints were only reachable as title tooltips. The strip is now a dropdown: the trigger shows the current workflow and its hint, and each row carries its own description. A row the loaded model can't run is disabled and shows the reason in place of the hint, so the gating explains itself. Adding a workflow no longer shrinks the others. * Images: workflow icons, hint under the trigger, more top room, unclipped Train cards - Each workflow carries an icon, shown on the closed trigger and on every row. - The trigger is one line (icon plus name). The selected workflow's description moved below it, where it reads like the Field hints further down the rail. - pt-6 instead of pt-3 on both Create and Train, so the cards clear the model selector row. - The Train right column scrolls while its cards use ring-1, which draws outside the box and was clipped at the scroll edges. p-px gives the ring room. * Images: one-line workflow rows, stronger trigger fill, roomier mode tabs - Dropdown rows are icon plus name only. The selected row's description already shows under the trigger, and a disabled row keeps its reason as a title. - Trigger fill moves to the bg-foreground/[0.07] dark:bg-foreground/[0.12] pair the hub cards use, so it reads against the card in both modes. - Description under the trigger goes from text-ui-10 to text-ui-11p5. - More horizontal padding on the Create / Train tabs. * Images: drop card borders for the composer shadow, keep scrollbars inside, use app controls in Train - Cards lose ring-1 for .panel-soft-surface: the composer's shadow in light, flat in dark, matching .chat-composer-surface and the menus. - Both rails now clip (overflow-hidden) with the scroller inside, so the scrollbar can't ride over the rounded corner. Same shape video-page already uses. - Train's 9 native selects become the app Select, so they no longer open an OS-native menu, and the native file input is hidden behind a Choose images button that reports the count. - Image previews use explicit 8-10px radii: this theme sets --radius to 1.1rem, so rounded-md was 15.6px and the thumbnails read as circles. * Images: one card for controls and preview, chat sliders, wider softer shadow - Controls and preview were two floating cards; they now share one card split by a divider. The Advanced dock stays separate since it toggles. - SliderField wraps Chat's ParamSlider, so the sliders match Chat (label row with the value, full-width neutral track) instead of a green track with a spin box. All 14 call sites keep their props. - panel-soft-surface goes from 0 2px 8px -2px /0.16 to 0 4px 22px -6px /0.10: lighter, spread wider. * Images: flat Create and Train panes, hover-only scrollbars, tidier Train dataset step Both Images tabs now sit on the page background like the Hub: no card, no shadow, no bounding box. A single rule divides the controls rail from the preview canvas (Create) and from the run area (Train), and the settings and previous-runs sections read as panes rather than nested cards. Also: - Scrollbars in these panes use the existing hover-scrollbar recipe, so the thumb only shows while the pane is hovered. - Workflow rows explain themselves with a tooltip after a short hover, which also works on disabled rows, and the descriptions are much shorter. - Training images rows are name plus image count; the license stays on the example card. - The upload step loses its dashed box, the buttons match the sizes around them, and Upload only appears once files are picked. - The empty preview uses the same icon as the Images nav item. * Images: full-height panes, wider settings rail, Create/Train offset from the selector The rule between the panes now runs the whole page height (the row drops its bottom padding and each pane pads its own content), the settings rail is wider on both Create and Train, and the Create/Train switch sits further right of the model selector. * Images: put both tabs on the Hub's centered measure Top bar and content now share mx-auto max-w-1100 with px-5 / sm:px-8, so Create and Train sit at the same width and position as the Hub instead of running edge to edge. * Images: restore the top bar position, drop the panes lower under it * Images: center the mode switch, flip the arrow with the orientation, app tooltips everywhere The Create/Train switch is centered on the page instead of trailing the model selector, with wider buttons. The flip control's arrows now rotate with the orientation and its label says which way the flip goes. Every native title tooltip on the page is now the app's tooltip, so they all get the rounded surface instead of the OS box. * Images Train: plainer field text, no green buttons, columns that stop colliding - The dataset name, trigger prompt, adapter name and custom base fields now say what they are in plain words instead of leaning on example values. - Import, Upload, Back, Back to settings and Train another are outline buttons, not green ones. - Example thumbnails are landscape tiles, so photos are not cropped to chunky squares. - Settings cells get min-w-0 and the select value truncates, so a long option like the nf4 label no longer widens its column into the next one. - The number stepper sits a little further in from the field edge. - Create and Train are wider. * Images Train: roomier example cards with Import on the thumbnail row * Video: same treatment as the Images tabs - No cards: the rail and the canvas sit on the page background, divided by a rule that runs the full page height, on the Hub's centered measure. - Wider rail, chat's sliders, hover-only scrollbars. - Every native title tooltip is now the app's tooltip, including the clip cards. - Reapply and Cancel are outline buttons, the empty state uses the Video nav icon, and the clip tiles are less rounded. * Images and Video: narrower generation rail, matching Train headings Create and Video rails go from 392px to 368px. Train a LoRA and Training settings are now the same size and both in the heading font: the h2 already picks it up from the base rule, so the settings header opts in with font-heading and the weight that rule pins. * Images Train: shorter copy throughout Family notes, example descriptions, precision labels and every helper line are trimmed so they stop wrapping to three lines and colliding with the next column. The nf4 label now fits its select without truncating. * Images Train: a little more spacing between field groups * Images and Video: tighten code comments * Fix training start NameError, the load-order guard test and CPU-only diffusion tests - start_training forwards resume_source_run_id to _start_training_impl, which reads it. Without it every start raised NameError. - Restore main's anchor in the load-marker order test: the file now has an earlier `if config.is_gguf:`, so indexing the first one compared the wrong branch. - The two diffusion tests that reach diffusers now skip when it is absent, matching the CPU repo-test env. - The UI smoke finds nav rows that live in the sidebar's More flyout. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Treat a null metadata caption as no caption str(None) stored the literal "None" as the caption, so a null row counted as captioned and would have trained on that text. Also drops an unused import. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix invalid-UTF-8 500s, the flat Canny map and the dropped DiT knobs read_text raises UnicodeDecodeError, which is not an OSError, so one bad caption sidecar or video sidecar 500d the info, upload and gallery routes. A flat image now yields the all-black edge map instead of its own luminance, and the four DiT loss knobs the trainer implements are declared so model_dump keeps them. * Use the ui font-size tokens instead of raw px text utilities text-[11px] and friends ignore the UI font size preference, which the repo's font-scale contract test enforces. Same rendered size at the default scale. * Fix diffusion dataset 500s, the dropout-1.0 no-op run and the reset base pick Four correctness fixes on the training side: - The labeling grid read caption sidecars under except OSError, but a non-UTF-8 sidecar raises UnicodeDecodeError (a ValueError), so one bad file 500d /diffusion/dataset/{name}/images and the grid could not be opened to repair it. Read it as no caption, matching the info summary. - An image past Pillow's own hard limit raises DecompressionBombError, which derives straight from Exception and so escaped the upload guard's (OSError, UnidentifiedImageError, ValueError) and returned 500 instead of the intended 400. - lora_dropout accepted 1.0, which makes PEFT build nn.Dropout(p=1.0): lora_A and lora_B receive no gradient and the run saves an untrained adapter while reporting normal progress. Bound it below 1.0, matching the LLM request schema. - The train panel re-seeded the base repo on every dataset refresh because the family object identity changes on each info fetch, so an upload or caption save silently replaced the user's chosen base and the run started on a different model. Track the pick and only re-seed on a real family change. * Show the retained failure when a video page mounts after a failed job Mount-time recovery handled only phase=completed, so reloading the page after a multi-minute generation failed left an idle view with no diagnosis: the backend keeps the terminal failed record only until the next job, and nothing else survives the reload. Surface it the same way the poll does, filtering the cancelled sentinel. * Fix batched generation crashes, cache keying and unreplayable recipes Four bugs in the batched inference path, all found by review: - A mixed-prompt batch sent a scalar negative prompt against a prompt list. Z-Image asserts on the length, and Qwen-Image, Krea 2 and FLUX true-CFG encode a batch-1 negative against batch-N latents and fail in the transformer's text/image concat. Broadcast it to match the batch. - The FBCache step-cache reset sat above the chunk loop. diffusers only resets that state at the end of a successful call, so a forward that raised (the OOM the backoff is meant to recover) left its own residual behind and the halved retry died on a shape mismatch. Reset before every forward instead. - The conditioning cache keyed on the checkpoint alone, but a GGUF or single-file load takes its text encoders from the companion base, so the same checkpoint against a different base reused the previous base's embeddings. Key the base too. - Gallery records stored the base seed and the requested batch size even when a prompts/seeds list drove the run, so restoring the second image of seeds=[5, 99] replayed seed 5. List-driven outputs now record as single-image recipes on their own seed. Also bound strength above 0: every img2img pipeline derives its step count from it, so 0 leaves zero denoising steps and either raises or, on SDXL, crashes on empty latents. * Fix quantized-load LoRA bake, prequant family exclusions and outpaint canvas Six review findings across the Images page and model scanning: - The quantized (int8/fp8) load path can only attach LoRA adapters before quantization, but the frontend load request had no loras field, so every generation after such a load was rejected and each reload repeated it. Send the selection with the load. - build_prequant_checkpoint passed no family to the scheme exclusions while recording the family in metadata, so a Qwen int8 artifact baked the short-M text-stream linears and was then rejected wholesale by the loader's family-keyed check. - Registering a bare single-file checkpoint directory produced no On Device row even though the images loader can load it; only its parent worked. Admit that shape when nothing else matched. - Unload left the Reapply target set, so the repair path was skipped and Reapply reloaded the ejected model. Clear it, as the video page does. - Both FLUX.2 bases were trusted for training but not inference, so Deploy to Create rejected every FLUX.2 adapter. - Outpaint allocated the grown canvas before downscaling, exceeding the browser canvas area cap on a large photo; an over-cap canvas is unusable, so Extend silently posted a fully transparent image and mask. Scale the source first. * Send the picked GGUF filename with the quant so diffusion loads fire The variant expander emitted only the quant label, and nothing else in the frontend set ggufFilename, so the Images and Video pages could never take their GGUF branch: both gate it on meta.ggufVariant and meta.ggufFilename, then fall through to the single-file path, which returns because the id is a repo id and not a .gguf name. Every quant pick was a silent dead click, with no load request reaching the backend. The filename was already on the variant row (the picker keys its list on it, and the variant validator requires a non-empty string), so thread it through the click handler. The chat path is unaffected: it reads ggufVariant and never needed the filename. * Version the conditioning cache key and reject non-finite flow_shift Two correctness fixes: - The cache keyed the checkpoint and its companion base by name only, so a Hub repo advancing to a new commit, or a local directory updated in place, kept returning embeddings from the previous text encoder. Pair both with a revision marker: the locally resolved commit sha for a Hub repo, config plus text-encoder file stats for a directory. Neither loads the encoders, so a warm run still keeps them off the GPU. - flow_shift only checked positivity, but JSON accepts 1e309, which floats to inf, and inf <= 0 is False while NaN fails every comparison. The sigma table then evaluates s * u / (1 + (s - 1) * u) as NaN, which poisons every sampled sigma and saves a corrupted adapter while progress looks normal. Require a finite value. * Keep curated models listed, guard the video companion repo, pin diffusers Three review findings: - The picker filtered every catalog member out of Recommended and Hub search on the way to canonical group rows, but nothing renders those rows yet (catalogGroupFitsDevice and groupMatchesQuery are imported and unused). A task-scoped picker's models list is catalogToModelOptions(), i.e. group members exclusively, so both lists came back empty and no curated model could be discovered or downloaded. Keep the artifacts listed until the grouped UI exists. - The video delete guard compared only repo_id, so deleting the companion base of a loaded GGUF video model was allowed even though it supplies the VAE and text encoders. Compare base_repo too, matching what the images guard already does for its companions. - diffusers was declared unversioned while the diffusion stack requires 0.39 (Krea2Pipeline, the cache_context child registries, the Flux2 and Z-Image pipelines), so an upgrade could keep an older release and selecting an advertised model failed until the user upgraded by hand. * Namespace the trainer conditioning cache per checkpoint, bound the learning rate - The trainer keyed its persistent conditioning cache on family and resolution only, while the keys themselves carry just the caption or image content and crop variant. One cache directory reused for two checkpoints, or for the same repo at a new revision, let a warm run skip loading its encoders and train on the other model's embeddings and latent statistics. Namespace on the base checkpoint and its resolved revision as well. The revision helper now lives beside the cache in diffusion_train_extras and the inference wrapper delegates to it, so the two cannot disagree about what counts as the same source. - The diffusion learning rate only checked positivity, but 1e309 floats to inf and satisfies gt, so the route evicted the resident models and started AdamW with an infinite rate: the first step destroys the adapter while progress looks normal and the result is saved. Bound it below 1.0, matching the LLM schema, which rejects inf for the same reason. * Fix GGUF image model picks doing nothing, and pick the train base in the top bar The quant rows never forwarded the .gguf filename, so every hub GGUF pick on Images/Video fell through to a silent return. On Train the top bar now picks the training base instead of a generation model, which is GGUF-only and untrainable. * Pin diffusion and video loads to the live HF cache root Both read huggingface_hub's import-time HF_HUB_CACHE, which changing the cache folder does not update: progress counted the old root while the download wrote to the new one, and from_pretrained could split one model across both. * Add the diffusion download plan endpoint Reports the repos and exact files a pick needs so the download manager can stage them with the loader's own file scope. A plain snapshot would add the packaged root single, transformer shards and fp16 twins the loader never opens. * Add a file-scoped flavour to the Hub download job Lets a consumer that reads a deliberate subset of a repo stage it through the normal download manager. Keyed as "@scope" so it never collides with a quant or with the repo's full snapshot, and the file list rides the registry so an XET to HTTP retry respawns the same scoped job. * Stage image and video downloads through the Hub download manager They downloaded inline inside the load, so they had none of the manager's disk preflight, manifest verification, resume or panel progress. Picks now stage as scoped jobs carrying the loader's own file list, then load from a warm cache. * Fetch staged GGUF checkpoints as scoped jobs, and stop calling diffusion models unsupported A GGUF entry went out as a full snapshot, whose ignore list drops *.gguf: the job finished at once having fetched only docs, and the repo landed on device unloadable. Every entry is scoped now. The Hub also no longer tags image/video models as unsupported (they run on their own pages), and those pickers name what they select. * Apply the picker task filter to local model sections LM Studio, ./models and custom-folder rows ignored it, so the Images picker listed chat GGUFs that 400 on a diffusion load. The backend already tags every local model with a task for this purpose. * Route a chat pick of a diffusion model to the Images or Video page Chat cannot load one, so it was either hidden or failed on load. The unfiltered picker now lists on-device diffusion models and navigates to the page that runs them, passing the repo and quant so that page loads it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Route the real GGUF filename, keep non-GGUF curated models, key scoped downloads by file set Five review findings, four of them ways a click did nothing or fetched the wrong thing: - A chat pick of a diffusion model routed ggufVariant (a label like Q4_K_M) in the search param the target page uses verbatim as the GGUF filename, so the load asked for a file that does not exist. Route ggufFilename; no filename means a curated non-GGUF pick, loaded as a pipeline. - The task-scoped pickers kept only GGUF repos, so the catalog's bf16, bnb-4bit and single-file fp8 artifacts could not be discovered or downloaded on the Images and Video pages even though loadSpecFor knows how to load them. Keep curated artifacts whatever their format, in Recommended and in Hub search. - Both pages deduplicated routed selections on the model alone, and they now stay mounted, so picking the same repo again -- another quant, or the same one after chat evicted it -- returned early without loading or clearing the query string. Key on model and quant. - Every scoped image download shared one @diffusion job key regardless of the requested files, so switching quant mid-download adopted the running job: the UI waited on the first file set, then loaded a file that was never fetched. Include a digest of the file set in the key. - A scoped plan silently dropped requested files missing from Hub metadata, and snapshot_download succeeds when an allow pattern matches nothing, so the job reported completion and triggered a load with required files absent. Fail the job instead. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the scoped download key derivable, and stop the hidden page hijacking a route Four review findings, the first a regression from my own last commit: - Keying scoped download jobs by a digest of the file set broke the download manager: it builds that key client-side (it polls and cancels before any response tells it a key), so it watched and cancelled a key no worker owned and never fired its ready callback. Keep the derivable "@scope" key and refuse the second request instead when a live job on the slot is fetching a different file set -- decided inside the registry claim, under the lock, so a concurrent claim cannot slip past it. The manager records the file set on the job as well, so a sibling quant's transfer is not adopted locally either. - Both diffusion pages read the route query through a loose useSearch and both stay mounted once visited, so the hidden one consumed the other's ?model=: it navigated back to its own route and tried to load, say, an image checkpoint as a video model. Only the visible page consumes it. - The staged download plan was built without the configured HF token or the Advanced values the load itself sends. The token matters most: the backend's Hub metadata lookup is best-effort, so a gated base silently planned no companion entry and the load pulled those multi-GB files inline, outside the manager. The memory/quant controls decide whether the base transformer/ shards are needed at all, and the route dropped memory_mode, cpu_offload, the prequant path and the LoRA selection before asking for the plan. - The video preview kept playing after leaving the page: the keep-alive layout only hides it, and display:none does not pause a media element, so a clip the user unmuted kept its audio going over the next page. Pause on the active transition and do not auto-replay while hidden. Also completes the hand-built request bodies in the hub download tests: the scoped-files field this branch added to the route read as an AttributeError against them, failing five tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Serialize the GPU handoffs, gate DiT training on a GPU, and keep 3.9 installable Six review findings, three of them evict-then-fail orderings: - The chat load reclaimed the GPU without telling the arbiter it existed. A chat load holds no llama-server process until its GGUF has downloaded, which is minutes, so a competing Images/Video acquire in that window found nothing to cancel, took the GPU, and the chat load then spawned onto the same device. It now registers an in-flight marker through acquire_for's register hook (under the arbiter lock, as the image and video loads do), the evictor cancels a marked load, and the route undoes itself if ownership moved while it loaded. - The Hub-download conflict check ran after that handoff, so a GGUF the download manager already owns destroyed the resident Images/Video pipeline and then 409'd, having loaded nothing. It moves above the handoff, together with the marker it handshakes with. - The image load released the engine router's transition lock before registering the load, so a second load choosing the other engine could unload the still-idle engine this one captured; the load then landed on a deactivated engine, where generate, status, unload and the arbiter's evictor can no longer reach it. Registration now happens under that lock and refuses if the engine changed. - Training a DiT family on a host with no GPU was accepted: nf4 is not a CPU fallback, its 4-bit load goes through bitsandbytes, which requires CUDA, XPU or MPS. The start unloaded the working Images pipeline, pulled the text encoders, and only then died in the child. Rejected before the teardown now, and /info stops advertising a precision that always 400s. SDXL keeps its documented fp32-on-CPU path. - Both diffusion pages kept the routed-pick marker forever, so re-picking the same checkpoint (after chat evicted it) neither loaded nor cleared the query string. The marker is released once the query is gone. The Images key also carried a stray NUL byte, which made the file read as binary to grep and other tooling. - diffusers dropped Python 3.9 in 0.38, so the unconditional >=0.39.0 pin left pip no candidate at all on 3.9 and made every install that composes the huggingface extras unresolvable there. The floor is conditional now. Also fixes tests that were already red on the branch: two hand-built request fakes had gone stale against fields this branch added, and the handoff-ordering test only failed on a host with fewer than two GPUs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the GGUF variant contract test against the merged handler signature The assertion pinned the exact single-line call handleVariantClick(v.quant, v.downloaded, expectedBytes, v.filename), but the handler takes (quant, filename, downloaded, sizeBytes) and prettier wraps the call across lines, so the mandatory repository test job failed on every push. Match the call structurally and assert the filename really is forwarded in the handler's argument order. * Stop a background page and a stale record taking the GPU or a download with them Five fixes from a review pass over the diffusion work. delete-finetuned rmtree'd a model the Images or Video engine was holding: every guard on that route is chat-only, and Images loads any local path, so deleting a local diffusion model under the storage root pulled the weights (and the companion VAE / text encoders sd.cpp re-reads each generation) out from under a live pipeline. The cached-model route already refuses this; the trained/exported one now does too, matching by path rather than repo id, and failing open on a chat-only install so it cannot block ordinary deletes. A staged download finishing while its page was hidden loaded the model and evicted whatever the user was actually using: both diffusion pages stay mounted behind the router and a load takes the GPU unconditionally. The pick is now held until its page is on screen again, which is also what chat does. A scoped download could report success having fetched nothing. With Hugging Face metadata unavailable no manifest is written, so verification is a no-op, and snapshot_download returns an existing snapshot folder without downloading when its own repo_info call fails. A repo already on disk from a full snapshot job (which ignores *.gguf) therefore completed with no weights and auto-loaded against them. The requested file list needs no network, so it is checked against the disk directly. The XET to HTTP retry reclaimed the job slot without the scoped file list, and that claim overwrites the stored record, so a later identical scoped start compared an empty list against the real one and 409'd instead of adopting the running download. The DiT accelerator gate probed torch.mps.is_available(), which only exists from torch 2.5 while the supported floor is 2.4. All three probes shared one try/except, so on torch 2.4 the AttributeError read as 'no block' and a CPU-only host still evicted the resident pipeline, downloaded the encoders and died in the child. Each accelerator is probed on its own now, through torch.backends.mps. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stage what an LTX-2.3 load reads, keep a routed file's load kind, drop an unbakeable LoRA Three from the latest review. The video download plan always asked for the wide base file list, so an LTX-2.3 pick staged the 2.0 base's VAEs, vocoder and connectors that the checkpoint supplies itself, while the companion files the 2.3 assembly does read were left out of the plan and pulled inline at load, outside the panel's progress, cancel and disk preflight. The plan now recognises a 2.3 pick by name (the load keeps the authoritative header probe, and under-guessing only falls back to the load-time pull), narrows the base list, and stages the extras in the same entry as the checkpoint so one repo stays one scoped job. A pick routed from the chat picker arrives as ?model= and ?quant= with no picker metadata, so a bare local .gguf or .safetensors was loaded as a pipeline: an explicit model_kind wins over the backend's filename sniffing, so it evicted the resident model and then failed on the missing model_index.json. Both pages now derive the load kind from the path, the same way their own picker handlers do. A torchao int8/fp8 build takes adapters only at load time. Switching artifact inside one family keeps the LoRA selection, since the family did not change, but the load did not bake it, so the next generation was rejected with 'reload the model with the adapter selection' while the picker still showed the adapter as active. The selection is now dropped once per resident build, with a message saying to pick and load again. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cancel an evicted safetensors load, spare the arbiter for CPU-only chat, fetch clips lazily Four fixes from the latest review round: - The GPU arbiter's chat evictor only cancelled the llama.cpp side. The orchestrator publishes active_model_name once its worker reports success, so an in-flight safetensors load was visible only as an entry in loading_models and finished onto the GPU after ownership had transferred. Cancel every pending load, and give the safetensors branch the post-load ownership recheck the GGUF branch already had. - A manual gpu_layers=0 GGUF load runs on the CPU with the GPUs hidden from the child, yet it took the arbiter unconditionally: it cancelled a running image or video generation for a model needing no VRAM, then held CHAT ownership so the next GPU workload unloaded it for nothing. Gate the acquire on the same predicate the launch-time CPU-only mask uses, as the image and video loaders gate on their resolved device. - The staged-download hook subscribes per repo, not per job, so another job on the same repo advanced the staged queue (starting a load whose scoped files were still downloading) or wiped a queue that was still running. Compare the variant each callback carries, like the chat page's auto-load does. - The video gallery fetched every record of a page into an object URL that lives until the page closes: 50 clips at tens to hundreds of MB each, for cards the user may never scroll to. Fetch a clip as its card nears the strip's edge, plus the selected one the player needs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim the comments across the diffusion backend Comment-only pass over the Python this PR touches: drop what the code already says, collapse multi-line explanations that still read on one line, and keep the reasoning that is not recoverable from the code. No code, docstring semantics or behaviour changes; verified with an AST comparison against the previous revision, and the backend suite is unchanged (same 37 environment failures as before: the API integration tests that need a live keyed server, the flash-attn install hooks, and the GPU memory fields). * Invalidate latents on a VAE swap, keep a cut-off generation, surface the EMA adapter - source_revision() scanned the checkpoint root plus text_encoder/tokenizer but not vae, so swapping or fine-tuning the VAE in place left the conditioning cache namespace unchanged and a warm run trained against latents from the old checkpoint. Include the vae directory, like any other component the cached tensors come from. - /images/generate answers only when the images are saved, and secure mode's tunnel caps an origin response near 100 seconds, which a native CPU or a high-step run passes routinely. The page reported failure while the work kept running, and a retry would duplicate it. A lost response (fetch rejection or a gateway status the origin never answered) is now told apart from a refusal: the page waits out generate-progress and reloads the gallery, so the run it started still lands. - The trainer emits the EMA adapter's path with the terminal event, but the state update dropped it, so neither the run history nor either response schema carried it and an enabled EMA left nothing discoverable. Keep it, and show it next to the primary adapter. - weighting_scheme advertised a choice of timestep sampling; sampling is always logit-normal and the flag only selects the bell loss weights. Describe what it does. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Hide unloadable cached rows, hold the dataset interlock, bound a GIF export - The cached-model listing tagged any repo with a model_index.json as text-to-image, so a community pipeline the image loader's trust rule refuses still got a row in the Images picker, and a detected-but-untrusted video repo fell through to that same tag. Gate the image tag on the load path's rule and hide an untrusted video repo outright. - A routed diffusion pick only carries a GGUF filename, which is all the chat picker has, so a curated single-file artifact arrived with no quant and was loaded as a pipeline: from_pretrained on a repo with no model_index.json. Pass the page's own catalog spec into the route pick, so a routed pick resolves to exactly what a direct pick on that page resolves to. - The dataset mutation endpoints checked is_active() and only then handed their filesystem work to a thread, so a start reserving in that gap changed captions or removed images underneath the preflight or the running trainer. The interlock is now registered for the whole request under the lock reserve() uses, and a start refuses while a mutation is open rather than waiting on it. - GIF export held every kept frame as a paletted image before encoding; a clip may be 2048x2048 for 1024 frames, and at the 12 fps target the step is 1, so one export click could allocate over 4 GB and take the backend down. Downscale past 720 px and widen the step to keep at most 300 frames. - seed accepted any Python int, so an out-of-range one passed every preflight, evicted the resident models, spawned the trainer and only then died in torch.manual_seed. Bound it to torch's 64-bit range in the request and config. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the accelerator probes in the DiT family-metadata tests Six tests read family_train_infos() (or a start preflight) without pinning the host probes, so they only held on a machine with a bf16 accelerator: on a GPU-less runner the DiT gate empties precision_modes, turns supports_compile off, and replaces any other preflight message with the no-accelerator note, and all six failed there. A conftest fixture pins both probes for exactly those tests, so they assert the family metadata they are about on every host. The gate's own CPU-only behaviour keeps its dedicated tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Carry the pipeline task into the hub inventory the pickers read - The task-scoped pickers filter On Device rows on a task, and the chat picker routes a diffusion pick by the same field, but those rows come from the /api/hub inventory, which never carried one: the Images and Video pickers listed nothing on device and the routing never fired. Both cached scans and the local listing now tag rows with the classifiers the models API already uses, the schemas and the frontend adapter carry it through, and a row the backend classified as a generation task is exempt from the chat-only guard that was also dropping it. - The local routing map was keyed by model_id while the row click passes id (a filesystem load id for a models_dir or LM Studio entry), so the lookup missed and the pick fell through to the chat loader. Key both. - A staged download whose start answered "error" left its head in place, where the effect never re-runs and onReady never fires, so the pick was stranded until the user reselected. Clear the queue and say so. - Every scoped pick in a repo shares the @diffusion variant, so the variant alone cannot tell two file sets apart: restaging while the first job finished let its completion pass for the new pick and load a checkpoint that had not downloaded. Bind the callbacks to the repo + file set they started, and to the staging generation. - A rejected generate POST does not say whether it reached the backend, so an immediately idle progress read was ambiguous and a submission that never landed looked like a finished image. Require evidence: progress seen active, or a gallery record that was not there before the POST. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the OpenAI image URL fetchable, keep WebM audio, stream example imports Four review items on the diffusion Studio work: - response_format=url returned the bearer-gated gallery route, which a standard image client downloads with no Authorization header, so the default response format was unusable. Mint a short-lived HMAC link instead (the shape RAG already uses for pdf.js) served by a signed route, and leave the gallery route itself bearer-only. - A manual gpu_layers=0 load carrying speculative_type="off" -- a value the UI persists and sends -- read as GPU-bearing, so it took the GPU arbiter and evicted a resident image/video pipeline even though the launcher hides the GPUs for it. Canonicalize the mode and exempt "off". - The curated example import prepared the whole split before the loop stopped at the 10-100 image cap; m1guelpf/nouns is 49,859 rows / 328 MB. Stream instead, with the prepared load kept as a fallback for a repo that cannot stream. - WebM export dropped the audio track an LTX-2 clip carries, silently, on the format offered for web embeds. Mux it as Opus through a resampler + FIFO, and keep exporting the video alone on a build without libopus. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not stub out triton on a GPU host when the Xet backend fails to import The lazy loader retries `import unsloth_zoo.hf_xet_fallback` under UNSLOTH_ZOO_DISABLE_GPU_INIT=1 whenever the first attempt raises. That flag makes unsloth_zoo take its MLX/CPU path, which injects triton and bitsandbytes STUBS into sys.modules for the rest of the process. On a working GPU box whose first import failed for an unrelated reason (a bitsandbytes/CUDA mismatch, say) the retry succeeds, so Studio boots looking healthy and then dies at the first CUDA-only kernel: a GGUF or compiled diffusion generation hits the stub and returns NotImplementedError: Unsloth: 'triton.tools.experimental_descriptor.enable_in_pytorch' was called on Apple Silicon / MLX, where triton is stubbed out. so every image generation 500s with an Apple-Silicon message on a Linux CUDA host, while the load reports success. Found by loading Z-Image-Turbo GGUF through the API on a box where bitsandbytes could not initialise. Gate the retry on the host genuinely having no accelerator. The Xet stall watchdog is optional and already degrades with a warning; a process whose triton is stubbed out is not recoverable. The warning now says why it did not retry. * Fix the lost-generation proof set, the settle timeout and the hub inventory's diffusion gates Seven fixes from the latest review round on the Images page and the hub cache inventory. Images page: - The lost-POST settle path built its "already seen" gallery id set inside the catch, after the request failed. By then the earlier runs of the same batch had already prepended their records, so run 2 could accept run 1's image as proof that its own request reached the backend. The set is now captured once before the first POST and grows with every record the batch produces. - settleLostGeneration fell out of its SETTLE_MAX_MS loop and returned normally, so a wedged generation was counted as done and the next run started against a busy backend. It now throws on timeout. - Restoring a recipe cleared the ControlNet selection but left the workflow tab and the init / mask / reference images pointing at whatever was loaded, so the next Generate conditioned on an unrelated image. It now clears all of them and returns to Create. - The download plan omitted the adapter selection the load itself bakes in. A baked LoRA forces the dense build path, so the plan described a different file set than the load that followed and the rest was pulled inline, outside the download manager. Both now derive the list from one helper. Hub cache inventory: - A download for a repo an Images or Video load is staging was allowed to start: only the llama.cpp loader was consulted. Both diffusion backends already expose loading_repo_ids for the delete guard, and the download guard now reads them too. - A companion-only prefetch (pipeline manifest plus VAE and text encoder, no transformer) passed the snapshot-partial check, since every file its manifest expected did arrive, and was advertised as on-device although from_pretrained cannot load it. - The single-file flag never reached the picker through the hub inventory path, so a checkpoint-only diffusion repo read as a full pipeline and failed after the handoff. The two pipeline-shape helpers now live in hub/utils/inventory_scan.py so /api/models/cached and the hub inventory classify the same repos the same way. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop adopting an unknown scoped download, leaking raced blobs and resurrecting deleted clips Three items from the latest review round. A scoped download job carries a deliberate file subset, and every file set of one repo rides the same "@scope" slot. A client that adopts a live job from the backend had no file list to compare against: the active-downloads response never carried one, so an adopted job's set was unknown and any later scoped request for the same repo read as "already started". Selecting a different checkpoint then waited on the wrong transfer and tried to load a file nobody fetched. The response now publishes the scoped file list, adoption records it, and an unknown set no longer satisfies a scoped request. A gallery record can be deleted while its blob is still downloading. The delete revokes the URL present at that moment, so the fetch that lands afterwards inserted a fresh object URL for a record no card renders and nothing can revoke: a full MP4, tens to hundreds of MB, pinned for the rest of the session, and once per raced fetch. Both galleries now discard a blob whose record went away, with an epoch covering the video page's Clear all. The video backend keeps the last completed job until the next one starts, and the Video page merges that record on mount to cover a job that finished after the gallery fetch. Deleting the clip left the record in place, so every reload prepended a ghost card whose file request 404s until another generation replaced it. Deleting the clip, or clearing the gallery, now clears the matching terminal record, and the page skips a record it deleted itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Resolve revisions from the live cache, serialize dataset imports, and stop pinning every gallery blob Five more items from the review round. The conditioning-cache revision marker read huggingface_hub's import-time HF_HUB_CACHE constant. Studio can move its cache during a session and loading follows the live setting, so after a move the marker went unresolved (or pointed into the previous root) and pulling a new revision of the same checkpoint no longer invalidated the cache: a warm run could reuse the old encoder's embeddings and the old VAE's latents. It now looks in the active Studio cache first and keeps the environment and the library constant as fallbacks, which the trainer subprocess still needs. The dataset interlock counts mutations rather than excluding them, so two imports of different examples into the same empty name both got past the emptiness check. The winner promoted its staging directory atomically; the loser found the folder non-empty, fell back to a per-file move, and merged its images and captions into the winner's dataset. Imports now take a per-folder lock, a second one is refused with 409, and the emptiness check is repeated under the lock. On Windows the sd.cpp asset resolver filtered only by accelerator token, so a Windows arm64 host matched an x64 zip, downloaded and installed it, and failed later when the binary would not run. It now filters by architecture the way the Darwin and Linux branches do. Every gallery page fetched every PNG up front and kept the object URL for the session, so scrolling a large gallery grew memory without bound for tiles the user may never look at. The Images strip now fetches a tile as it nears view, like the Video strip, and keeps the eager path only where IntersectionObserver is unavailable. A 503 carrying a JSON body comes from the application, not a proxy, so it is surfaced as the error it is instead of entering lost-response settlement and being reported as a request that never reached the server. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the class of a failed generation instead of a bare "Image generation failed." Found on a macOS runner: the native renderer aborts inside its own text encoder there, and the page showed only "Image generation failed." with the sd-server backtrace left in the server log, so nothing about the failure reached the user. The failure is now classified into fixed text, out of memory and native-process death, so the message says what happened and what to try. None of the engine's own output is echoed, since a native tail carries local paths and argv; that stays in the log, and an unrecognised failure keeps the original literal. * Treat an undecodable caption sidecar as the tombstone the trainer sees Uploads store .txt and .caption sidecars as raw bytes, so one can hold invalid UTF-8. The trainer treats any existing sidecar, decodable or not, as an empty tombstone and never falls back to the metadata row for that image. The labeling grid and the dataset summary read an undecodable sidecar as absent instead, so both showed a metadata caption that the run would silently replace with the instance prompt, and counted the image as captioned. Both now track sidecar presence separately, so what the user reviews is what the run trains on. * Keep the reason a native server died, not just its backtrace A ggml abort prints its cause first and then a stack trace, so reporting the last twenty captured lines gave twenty addresses and nothing about the failure: on the macOS runner the native server died on an unimplemented Metal op and the message carried only frame pointers. The captured tail now leads with the lines that name a cause and keeps recent context after them, for both the startup failure and the mid-request death. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop staging the dense text encoder for an fp8 video load Two halves of the same gap, found while measuring the LTX-2.3 download plan: - The video download plan and the scoped pre-download never saw text_encoder_quant. An fp8 request loads a hosted pre-cast encoder, so asking for one still staged and downloaded the base repo's dense Gemma3 (48.79 GB of Lightricks/LTX-2 on the 2.3 distilled pick) that the pipeline then never opened. The plan now drops those shards and stages the pre-cast checkpoint instead; their configs stay, since the pre-cast loader still meta-inits the encoder from the base repo's component config. - The LTX-2.3 assembly builds every component itself, so pipe_kwargs (which carries the pre-cast encoder for from_pretrained) never reached it and an fp8 request silently loaded the dense encoder anyway. It is passed across explicitly now. The dense skip is earned, not assumed: only a pre-cast checkpoint that resolves on the Hub lets the plan drop the dense shards, and only one already fetched to disk lets the pull drop them, so an unpublished or gated artifact leaves both exactly as they were. If injection still fails after that, the load tops the dense weights back up rather than handing from_pretrained a snapshot with no encoder in it. Measured against the real Hub on the 2.3 distilled Q4_K_M pick: 67.24 GB before, 18.92 GB with a 0.43 GB stand-in for the pre-cast artifact (the base entry drops from 24 files / 48.79 GB to 13 files / 0.04 GB). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Match the file's typing idiom for flow_shift models/training.py annotates with typing constructs throughout (105 Optional[...], no Union), and flow_shift was the one place using PEP 604. Union[] reads like the rest of the file, and it also drops the runtime evaluation that would raise on Python 3.9. * Bound the gallery blob cache, and three interlock fixes Four review findings, all reproduced first: - The gallery object-URL caches were unbounded. A clip runs from a few MB to a few hundred, both pages stay mounted after their first visit, and entries were only dropped on delete, so scrolling pinned everything for the session. Both pages now share a byte-budgeted LRU (512 MB video / 192 MB images) keyed off the visibility signal the near-viewport fetching already provides. On-screen media, the selected clip or image, and the item just fetched are never evicted, so eviction is invisible and a single item larger than the whole budget cannot evict itself into a refetch loop. - The image, video and chat load guards ran two independent training probes but returned early when the FIRST one raised, so an unreadable LLM backend disabled the diffusion interlock and a load could proceed straight into an active diffusion trainer on the same GPU. The probes are independent now. - An engine switch swallowed a failed teardown and published the new engine anyway, which is exactly the leak the unload exists to prevent: the arbiter's evictor, /images/unload and the next load all resolve through get_active_diffusion_engine(), so the still-resident pipeline (or a live sd-server) became unreachable and the next load allocated on top of it. The switch now fails and leaves the old engine published, so it stays reclaimable. - The native generation timeout was 30 minutes while the Images page waits up to 6 hours (SETTLE_MAX_MS), so slow-but-progressing CPU jobs died deterministically at the deadline. Measured on GPU-less runners, a 512x512 4-step Q2_K generation took 900 s on Linux and 1465 s on Windows, so larger images or step counts clear half an hour easily. The ceiling now matches the page's window and applies to the whole request: chunks of a split batch share one deadline instead of each getting a full budget. Cancellation is unchanged. Declined: gating the huggingfacenotorch extra off Python 3.9 over the conditional diffusers marker. The marker is deliberate and its comment says why: diffusers dropped 3.9 in 0.38, so pinning >=0.39 outright leaves pip no candidate and the whole extra unresolvable there. The pipelines it names live in studio/backend, which cannot install on 3.9 anyway (studio.txt pins matplotlib==3.10.9 and fastmcp>=3.0.2, both requires_python >=3.10), and the extra is the general core one, so the alternative drops 3.9 for library users who never touch Studio. * Close the load-versus-training-start race, and two picker fixes - The image and video load guards read is_active() and only then selected an engine, acquired the arbiter and registered the load. A /train/diffusion/start reserving inside that window freed residents the load had not registered yet, so the trainer came up beside a brand-new pipeline. The service already had exactly the right pattern for this in dataset_mutation, so gpu_load_admission mirrors it: reserve() refuses while an admission is open, an admission refuses once a start is reserved, both decided under the one lock. The span is only the registration, since begin_load returns as soon as the load is registered and _free_gpu_for_diffusion_training preempts an in-flight load from that point. Chat is deliberately not covered: its load spans an eviction plus a multi-minute GGUF load, and it admits models that fit beside training by design, which is a different contract from the diffusion pipeline's all-or-nothing one. - Hugging Face gives the LTX-2 family the image-to-video pipeline_tag (both Lightricks/LTX-2 and unsloth/LTX-2.3-GGUF report it), so a text-to-video-only filter dropped the flagship audio family out of Video Hub search while the rest of the app routed it to Video. - Task-scoped quant fit sized picks against the LARGEST visible device while resolve_diffusion_device_target returns a bare "cuda" and torch places on the current one. On a heterogeneous host that recommended a checkpoint sized for the bigger card and then loaded it onto the smaller one. Fit now uses the device the load actually lands on; identical on a homogeneous host. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Expose the persistent conditioning cache in the start schema DiffusionLoraConfig has carried cond_cache_dir for a while and the DiT trainer acts on it, but DiffusionTrainingStartRequest omitted the field, so Pydantic dropped it silently and every API-driven run fell back to the in-memory cache that is rebuilt from scratch each time. The warm path skips loading the VAE and the multi-GB text encoders on a rerun whose images, captions and resolution are unchanged, so this was a real capability that could not be reached. Contained like output_dir rather than left to the trainer subprocess's cwd, since it is another directory the trainer writes to. Blank or omitted still means the in-memory cache, so it must not resolve to the outputs root. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix diffusion policy and classification issues from review fp8 auto precision defaulted to precise accumulate on any non-consumer GPU, which made fp8 2.05x slower than int8 on RTX 6000 Ada and slower than not quantising at all. NVIDIA's professional whitepapers do publish equal FP8 rates for both accumulate modes there, so the hardware premise held, but the cost is in the cuBLAS path rather than the published rate. Default to fast accumulate: measured on B200 the flag is a no-op (4096^3 _scaled_mm at 3023.8 vs 3041.8 TFLOP/s, bitwise-identical output, 1.213 s vs 1.230 s end to end), so it is a large win where it bites and free where it does not. Precise accumulate stays available via transformer_quant_fast_accum. Z-Image's DiT is a Lumina2 derivative, so unsloth/Z-Image-GGUF and unsloth/Z-Image-Turbo-GGUF both declare general.architecture = "lumina2" and the whole line was tagged image-diffusion-unsupported and hidden from the Images "On Device" list, though validate_load_request loads them. Resolve shared archs from the repo/file name like bare "wan" already does, with a test asserting the picker and the loader agree for every family. The sage attention on-demand install ran an unpinned `pip install sageattention`, but PyPI's newest wheel is 1.0.6 and diffusers refuses anything below 2.1.1: the install always "succeeded", wrote an unusable version into the running venv, and was rejected on the next line. Carry the dispatcher's floor so pip resolves nothing instead. The dense-quant disk gate sized the download from the bf16-RESIDENT table. The fp32 families download twice that (Z-Image: 23,479 MiB against a 21,970 MiB gate), leaving a window where the check passed and the download filled the disk; Ideogram 4 ships fp8 and was overcharged the other way. Size the gate by published bytes, verified against HF sibling metadata for all 12 families. Patch installs went through unsloth_zoo, which refuses to import unless UNSLOTH_IS_PRESENT is set, and that is set by unsloth itself. The server imports unsloth at boot so it never showed there, but any other process ran silently unpatched with every install returning False, which is 13 test failures on a clean environment. Import unsloth and retry once, memoised per process. Also: the GGUF+LoRA refusal pointed at the native engine without saying a GPU host only selects it under UNSLOTH_DIFFUSION_ENGINE=sd_cpp, so the suggestion was unreachable; the gallery recipe recorded loras from the generate request alone, losing a load-time bake; load-progress claimed "40.07 GB downloaded" for a fully cached load; and pickers.tsx imported three catalog-group helpers it never used. Reported by oobabooga. * Keep the sd.cpp text encoder on CPU under Metal macos-14 loads FLUX.2-klein-4B Q2_K natively on mps and then dies on the first generation with exit code -6: ggml_metal_op_encode_impl: error: unsupported op 'RMS_NORM' -> ggml_abort LLMEmbedder::encode_prompt -> LLMRunner::compute -> GGMLRunner::compute ggml's Metal backend gates RMS_NORM on contiguous rows and aborts the process when that does not hold, with no per-op CPU fallback, so any LLM text encoder (Qwen3 for FLUX.2 and Z-Image, T5 for FLUX.1) takes sd-server down. The encoder runs once per prompt while the DiT runs every step, so pinning only the encoder keeps Metal for the part that matters. UNSLOTH_DIFFUSION_SD_CPP_METAL_TE_GPU=1 opts back in once ggml grows the kernel. * Gate the unsloth retry in the diffusion patch backend The retry added for the clean-environment patch failures is not free: importing unsloth pulls torch in behind it, which costs ~940 MB of RSS measured in a process that had neither, and on a host with no accelerator it fails anyway. A cross-platform CI job that had generated fine at ~900 s later died 19 s in with SIGTERM and every 'if: always()' step skipped, which is the runner being torn down rather than a step failing. Retry only when torch is already imported (true of the server and of anything patching a real module, and the condition that stops the retry from being what loads torch), unsloth is installed but not yet imported, and the first failure was the ImportError the sentinel guard raises. The clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only retry the unsloth import where it can succeed The gate still let the retry run on hosts unsloth does not support, which is where it is most harmful: a 7 GB macOS runner lost the Studio server 26 s into a load, and the Linux runner was torn down mid-generation. Neither MPS nor plain CPU can complete the import, so the retry there pays the cost and fails anyway. Require an accelerator unsloth actually supports (CUDA/ROCm via torch.cuda, or XPU), with UNSLOTH_ALLOW_CPU as the documented override, and hoist the predicate to module level so it is tested directly rather than through the import system. On a CPU-only host the retry no longer fires at all; on CUDA the clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Guard old diffusers, stream video exports, record conditioned recipes Three fixes from review. The 0.39-only pipeline classes (Flux2Klein, Z-Image, Krea 2, LTX-2, HunyuanImage) were resolved by getattr deep in the load, so on the older diffusers that packaging still allows on Python 3.9 -- diffusers dropped 3.9 in 0.38 and this project still supports it, so the 0.39 floor has to be conditional or the extra becomes unresolvable -- an advertised model failed with a bare AttributeError after its checkpoint had already been downloaded. Krea 2 already guarded itself this way; assert_pipeline_class_available now runs the same check for every image and video family from validation, before any fetch, and names the version and the fix. WebM export accumulated the whole VP9 output in a BytesIO and returned it as one bytes object that the response held again. The request caps allow 2048x2048 for 1024 frames, so an export runs to hundreds of MB and concurrent clicks could exhaust the process, while the MP4 route beside it already streamed from disk. transcode_to_file encodes to a temp file and the route returns a FileResponse with a background unlink, so nothing large is resident. A conditioned generation's recipe carried only the txt2img fields, so the gallery presented an inpaint or upscale result as a complete Create recipe and restoring it replayed an unrelated text-to-image request. The images themselves are still not persisted (user uploads with their own lifetime), but the workflow and its scalars are, restore reapplies them, and the toast now names the inputs that have to be supplied again instead of silently landing on Create. Reported by Codex. * Per-load video cancel event, family-gated image picker, cond cache refusal A cancelled video load could resume: begin_load cleared the shared cancel event, and unload() drops _loading without waiting for the worker, so the next load cleared the very object the cancelled worker was watching and its multi-gigabyte pull ran on alongside the replacement until the token check at the end. Each load now gets its own threading.Event, passed down through _fetch_te_prequant and _predownload_base, so a cancelled worker stays cancelled. A cached repo with a model_index.json was advertised as text-to-image on the trust rule alone, but validate_load_request also requires a detected image family, so a trusted pipeline of an unsupported class produced a picker row that deterministically 400s. The picker now applies both gates, mirroring the video branch. cond_cache_dir was accepted for sdxl and then ignored: only the DiT trainer reads it, while the SDXL trainer builds a per-run in-memory latent cache, so the promised cross-run reuse never happened. The route now refuses it with a 400 that names the families which do support it, checked against the resolved family so an omitted model_family with an SDXL base is caught too. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the frontend build broken by the gallery blob cache tsc -b failed on the branch head, so npm run build produced no dist and every platform job fell back to --api-only: blob-url-cache.ts(29,15): TS1294: This syntax is not allowed when 'erasableSyntaxOnly' is enabled dataset-labeling-grid.tsx / dataset-showcase.tsx: Argument of type '{ url: string; bytes: number; }' is not assignable to parameter of type 'string' The cache took its budget as a constructor parameter property, which the project's tsconfig forbids, and fetchGalleryObjectUrl now returns the blob size alongside the URL for that budget, which the two dataset thumbnail components still consumed as a bare string. Declare the field explicitly and destructure the URL at both call sites. tsc -b is clean and vite build emits dist again. * Recover from a ggml unsupported-op abort by restarting on the CPU backend ggml checks every node against the device's supports_op and calls GGML_ABORT when one is not implemented, because a single-backend graph has nowhere else to put it: there is no per-op CPU fallback. The whole sd-server dies with SIGABRT mid-generation and the user gets "the native image renderer stopped unexpectedly" with no way forward. Seen on macos-14 arm64 with FLUX.2-klein-4B Q2_K through the cross-platform CI: the text encoder is already pinned to CPU, and the abort moved into the denoise loop instead. ggml_metal_op_encode_impl: error: unsupported op 'MUL_MAT' -> ggml_abort StableDiffusionGGML::sample -> sample_k_diffusion A retry on the same backend would abort identically, so the load is restarted once with --backend cpu (the only flag that changes which backend executes the graph; --offload-to-cpu moves parameters, not compute) and the generation is re-submitted. The same checkpoint then renders slower rather than not at all. Strictly bounded: the signature must carry both the unsupported-op line and ggml_abort, the device must not already be CPU, and it happens once per load, so an OOM kill or a genuine crash still surfaces as itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix two tests that only fail in a full-suite run The 3.10 CI leg resolves PyAV 17, where av.container.OutputContainer is an immutable C type, so the no-libopus export test died on "cannot set 'add_stream' attribute of immutable type" before it asserted anything. Inject the refusal by wrapping the container av.open() returns instead; modules stay patchable on every build. Removing the injection makes the test fail again, so it still covers the branch it is named for. The Xet shim's degraded-path tests drop utils.hf_xet_fallback from sys.modules and import a throwaway copy. Restoring only the sys.modules entry left the utils package attribute bound to the throwaway, and the two disagreed for the rest of the process: a later monkeypatch of the dotted target patched one copy while the code under test imported the other, so the patch did nothing and test_fetch_te_prequant_only_reports_what_it_downloaded reached the real Hub and got a 401. Restore both bindings. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop an ignored-cancel sd-server, guard deletes during diffusion training, repair unusable managed binaries sd-server does not interrupt an in-flight job, so when it ignores a cancel the grace branch abandoned the poll and reported cancellation while the native job kept a core (or the GPU) busy to completion and held the server's job slot. The comment said the caller stops the server, but only unload does that immediately: a superseding load stops it after its multi-gigabyte download, and a load that then fails never gets there. Stop it here, as the deadline branch already does. DELETE /api/models/delete-finetuned checked only the LLM trainer, so it could rmtree the output directory a live diffusion LoRA run was about to write its adapter into. Consult the diffusion training service too, like the dataset mutation and model-load routes. find_sd_*_binary only checks is_file(), so an interrupted extraction (or a prebuilt for the wrong CPU) left a present-but-unrunnable binary the installer never retried: every load probed it, fell back to diffusers, and native inference stayed off until the directory was deleted by hand. Probe it and reinstall, but only for a copy under the installer-owned root -- SD_CLI_PATH, UNSLOTH_SD_CPP_PATH, an in-tree build and anything on PATH are the user's. * Plan the pre-cast text encoder, and make the cross-trainer GPU admission atomic An fp8 text-encoder request loads a hosted PRE-CAST checkpoint, but the image download plan never received text_encoder_quant, so the manager staged the base repo's dense encoder (FLUX.2-dev's Mistral-24B is ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) and the load then pulled the pre-cast file inline, outside the manager's progress and disk preflight. The plan now takes the field, resolves the hosted artifact with the same resolver the injection uses, stages that file, and drops only those components' dense weight shards. The load's own prefetch takes the same treatment, since it paid the same cost. Only a checkpoint that really resolves on the Hub earns the drop, so a gated or renamed artifact still stages the dense encoder the load will fall back to. The two trainers admitted each other with independent check-then-act guards: the diffusion route checks the LLM backend several network-bound preflights before it reserves, and the LLM route checks the diffusion service well before it spawns, so two near-simultaneous starts could both pass and train on one GPU. reserve() now re-tests the LLM backend under its own lock, and the LLM route holds the diffusion service's gpu_load_admission across its spawn, so exactly one of the two wins. Both halves fail open, so a chat-only install still trains. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the Advanced options a staged download planned against Staging does not set busy, so while a multi-gigabyte download runs the user can still change precision, memory mode, speed or the baked LoRA selection. The pending record held only the repo and artifact, and the completed download fired a load that read the CURRENT state: the staged file set could then be missing files that load needs (fetched inline, with no progress and no disk preflight) or hold gigabytes it no longer uses. One snapshot of every Advanced control is now taken when the plan is built, and it travels with the pending record into the load, so the load that runs is the one the download was planned for. * Do not advertise a family the installed diffusers cannot build The newer families (Z-Image, Krea 2, FLUX.2, LTX-2, HunyuanImage) exist only from diffusers 0.39, and 0.39 cannot be installed on Python 3.9 at all -- diffusers dropped 3.9 in 0.38, so the requirement is conditional or the whole extra becomes unresolvable. On such an environment the picker still offered those rows, every pick failed deterministically, and the error's advice to run pip install -U diffusers could not fix it without also upgrading Python. The cached-repo picker now applies the same availability check validate_load_request does, which is keyed on the pipeline class actually present rather than on the Python version, so it is also right for an intentionally pinned older diffusers on 3.10+. Fails open when diffusers cannot be imported at all: that is a different problem and the load path reports it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore the diffusion engine selection after each router test The active engine is module state, and several tests set it by plain assignment because what _activate does to it is the thing under test, so monkeypatch could not undo it. A leaked ENGINE_SD_CPP left get_active_diffusion_engine() handing back the sd.cpp backend for the rest of the process, and every later route that reads the active engine then saw an unloaded model: eight tests in test_openai_images_generations_route.py returned 503 in a full-suite run while passing on their own. The autouse fixture now snapshots and restores it. * Stream gallery clips, and close three races around them Four fixes from the latest review pass. The video gallery downloaded each clip into a blob before it could play, so playback waited on the whole file (tens to hundreds of MB), seeking was limited to what had arrived, and every viewed clip stayed pinned in the webview. The file route already streams and serves ranges; it just could not be a <video src> because it is bearer-gated. Mint a short-lived signed link instead (its own HMAC secret, 12 hour TTL, separate from the image links) and hand it to the element, which then fetches only the ranges it plays. That removes the blob budget, its LRU and every revoke on this page. The sd.cpp readiness probe accepted any process answering on the port, so a foreign server that grabbed the port between the bind check and the spawn was adopted as ours. Confirm the listener is our child before reporting ready, and stay best-effort (psutil missing, an unknown owner, or any probe error still passes) so the check can only reject a definitely foreign process. Dataset import held its lock for the extract but not for the upload path, so two concurrent uploads into the same folder interleaved; take the same lock and return 409. And reject Windows device names (CON, NUL, COM1..9, LPT1..9, with or without an extension) plus trailing periods in dataset names, which are unopenable on Windows. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Build the image download plan for the engine that will load /images/download-plan always asked the diffusers backend, while /images/load picks the engine per host: a GGUF pick on a machine with no usable GPU routes to native sd.cpp, which reads a single-file VAE plus text encoders and never opens the base repo's sharded components. Measured on unsloth/FLUX.2-klein-4B-GGUF (Q2_K): the plan staged 7.66 GB of FLUX.2-klein-4B components the native load discards, and the 7.80 GB sd-cli actually needs was then fetched inline by the loader, outside the download manager's progress and its disk preflight. Z-Image-Turbo is the same shape. The plan now asks whichever engine the load will select. predict_engine() applies the selection policy without any side effect: it activates nothing (staging a download must not unload the resident model) and only locates the binary rather than installing it, but still counts an installable binary as available, since that is what the load does on a fresh host. The native backend gains a download_plan built from the same _asset_specs the loader fetches, returning the same envelope, so the manager stages exactly the files sd-cli opens. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not let a queued generation outlive the model, and three scan fixes Five items from the latest review; four were real. An unload or arbiter eviction only cancels the generation holding _generate_lock. A second request queued behind it holds no cancel event yet, and Python locks are not FIFO, so it could take the lock the instant the active denoise released it, still see a loaded pipeline, and run a whole new denoise after the model was told to go away: the eviction then waits minutes for it and an image lands after the eject. Unload and a superseding load now raise a fence under _lock before they queue, and a generation that wins the lock while one is pending refuses instead. The cached-model scan judged pipeline completeness across every revision, so a repo holding an older complete snapshot plus a newer companion-only one read as complete while the snapshot from_pretrained actually opens has no transformer. Both scans now look at the revision the loader will open. Deleting a dataset image deleted its caption sidecar unconditionally, which for cat.jpg alongside cat.png removed the caption the survivor still resolves to. The sidecar now goes only with the last image of that stem, matching what the thumbnail cleanup beside it already did. Importing an example into a folder that holds no images but does hold files fell back to promoting the staging dir one file at a time, so an interruption left a partial dataset that the image_count check accepts as complete on retry. Those files are folded into the staging dir instead and the promotion stays a single atomic rename. The MPS generator report does not apply: torch.Generator(device="mps") has worked since PyTorch 2.0 (pytorch/pytorch#91348) and the studio installer pins torch>=2.4. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten diffusion comments Collapse the multi-line comment blocks across the image, video, sd.cpp and diffusion-training code to one or two lines each, and drop comments that only restate the statement below them. Comments only, no code or behaviour changes. * Tighten diffusion comments (second pass) Collapse the remaining multi-line comment blocks in the video page, training routes and service, sd.cpp server and installer, memory and speed planners, and the shared request models. Comments only, no code or behaviour changes. * Tighten diffusion comments (third pass) Collapse the remaining multi-line comment blocks in the attention, cache, LoRA, prequant, precision and compile-cache modules, the sd.cpp arg builder and engine, the video routes, the Ideogram 4 assembly, the model picker, and the diffusion test suites. Comments only, no code or behaviour changes. * Restore the dataset when an example import cannot be promoted Promotion folds the folder's pre-existing entries into the staging dir so the swap is one atomic rename. Every failure after that fold left the user with nothing: the 409 path and an os.replace error both fell through to 'finally: shutil.rmtree(staging)', which deleted the entries that had just been moved in there, while the response said 'Nothing was written'. A same-named entry was also unlinked outright before the promotion was known to succeed. Park superseded same-name entries in a rescue dir instead of deleting them, record every move, and restore all of them if any step of the promotion fails. The fold loop itself is covered too: renaming a non-writable directory raises EACCES on POSIX, which previously escaped as a 500 after the earlier entries had already been moved out. A failed rename now maps to the same retryable 409 as the rmdir conflict. Verified with the reported trigger (a non-empty mode-500 directory whose name collides with an imported file): the folder listing is now identical before and after the failed import. * Drain the teardown fence on a failing unload, give each load its own cancel event Two independent leaks on the image path, both already solved elsewhere in the same file. unload() incremented _teardown_waiters, ran _unload_locked() and decremented, with no try/finally, while the superseding-load path used a finally for the same pair. _unload_locked ends in clear_gpu_cache(), whose CUDA branch calls synchronize/empty_cache/ipc_collect unguarded, and a sticky CUDA fault makes those raise. The count then never drained, so every later generation was refused as cancelled for the life of the process, a fresh load included, since begin_load's own increment and decrement are symmetric. unload() is reached from the chat/video GPU handoff, the engine router and two training routes, so one fault during an ordinary handoff wedged image generation until restart. Release it in a finally. The image and native backends each cleared one shared cancel Event on a new load. unload() sets that event to cancel an in-flight multi-GB download and drops _loading in the same breath, so a replacement load is admitted while the cancelled worker is still inside the fetch, and its clear() re-enabled the very object that worker was watching: the cancelled download resumed and ran alongside the replacement. Take a fresh Event per load and thread it to the worker, as the video backend already does, and set it under the lock since begin_load now rebinds the attribute. * Name utf-8 on the diffusion text I/O and the sd.cpp subprocess pipes tests/test_text_io_encoding.py failed on five files this branch adds. Text I/O without an explicit encoding falls back to the Windows ANSI codepage, so a non-ASCII path or manifest value round-trips corrupted, and the three sd.cpp pipes decode the child's UTF-8 output as ANSI on Windows despite already passing errors = 'replace'. Eleven read_text() / write_text() sites across diffusion_compile_cache, diffusion_ideogram4 and diffusion_krea2, plus text = True on the sd-cli version probe, the sd-cli run and the sd-server pipe. * Record the load-time build on a gallery image's recipe A gallery record documents itself as the image's full generation recipe and is embedded in the PNG, but the only load-related field it carried was the repo id. A GGUF repo holds many quants, so that does not say which one made the pixels, and it says nothing about an adapter baked in at load time. The fallback meant to cover the baked case could never fire: with no loras on the request _adjust_baked_loras zeroes every baked adapter and _active_lora_pairs drops zero-weight entries, so active_loras was always empty. A baked-and-disabled build is not the same pipeline as a never-baked one, so the recipe could not reconstruct the image once the model was rebuilt. Persist model_kind, gguf_filename, transformer_quant and the baked adapter names, read off the load state rather than the request, and show them in the recipe popover. The new fields are optional with defaults, which matters because list_gallery_images drops any record that fails validation, so a required field would have emptied every existing gallery; a regression test pins that. * Drop eleven duplicated comment tails, restore the mxfp8 denial note The comment passes collapsed several wrapped blocks onto one line without deleting the last physical line of the original wrap, leaving the tail of each sentence repeated as its own comment underneath. Two of the eleven were re-worded rather than byte-identical, so a strict suffix match missed them. 6e16ad16f also dropped the line justifying the qwen-image mxfp8 denial while that rule stayed live in _FAMILY_SCHEME_DENY, under a header that then documented only fp8 and nvfp4. Restored. Comments only; verified with the AST gate. * Only let the dense-quant fallback use shards the prefetch actually staged The prefetch skips the base repo's transformer/ shards whenever a prequant checkpoint is expected, since that checkpoint replaces them. But a prequant fetch can fail for reasons the planner cannot see: an unpublished, gated or renamed artifact, a hub 5xx, a proxy, a checkpoint the validator rejects. The loader then fell through to from_pretrained(subfolder = 'transformer') and pulled those shards inside the load lock during 'finalizing', after the previous pipeline was already evicted, where the cancel event has no reach, load_progress has already reported bytes_downloaded == bytes_total, and the cache-disk gate had only reserved the small prequant checkpoint. That is verbatim the situation _dense_quant_prefetch_needed's own docstring exists to prevent. Gate the in-loader dense fallback on the shards being staged, read off the returned file list rather than the request so a failed size estimate closes it too, and let the GGUF build take over otherwise, which is what the prequant-sized replan already does one branch over. It is also the invariant the text-encoder path already enforces: only a component that really resolves may have its dense weights dropped from a plan or a prefetch. Adds the missing coverage for the gate's prequant arm, which the existing disk-gate tests never reached. * Refuse a training output_dir that resolves to the outputs root resolve_output_dir strips a leading 'outputs' and drops '' / '.' segments, so '.', './', './.', 'outputs', 'outputs/outputs' and ' . ' all clean away to nothing and land on the outputs root itself rather than a run directory under it. The DiT trainer then writes pytorch_lora_weights.safetensors flat into the root, where scan_trained_models and scan_checkpoints cannot see it (both filter is_dir()), and a second such run overwrites the first. The UI only checks the field is non-empty, so 'outputs' is one plausible run name away. Refuse it with a 400 that says what to do instead. cond_cache_dir collapses the same way but has an honest 'off' to fall back to, so a root-resolving value now means the in-memory cache, which is what the comment above it already promised: otherwise a run drops one flat safetensors per cached latent and caption into the directory trained models live in. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Repair three defects the main merge left behind Git merged all three files without a conflict, but the result was wrong in each case. The tests only started failing once the merged tree was run. routes/inference.py duplicated the GGUF load block: this branch had moved the gguf_load_in_flight marker and the _hub_download_blocks_gguf_load guard under "if config.is_gguf and config.gguf_hf_repo", and the conflict resolution re-added main's copy at the old position, so both ran. Dropped main's copy; the earlier placement is the deliberate one, so a 409 from the hub guard cannot tear down a resident Images or Video pipeline. test_gpu_selection.py still called _hf_offline_if_dns_dead, which main renamed to _hf_offline_if_unreachable_for (#7591). Disjoint edits, so no conflict, but four route-error tests referenced a function that no longer exists. test_gguf_load_cache_reuse.py anchored its ordering assertion with rindex over "if config.is_gguf:", taking the last one before the load marker. That only held while _resolve_inherited_extra_args sat above every such line; main has since hoisted it above the gpu_ids preflight, so the anchor landed between the call and the marker and the assertion compared against an unrelated later call site. The ordering it checks is a property of _load_model_impl as a whole, so it now anchors on the function. The ordering itself is intact: _resolve_inherited_extra_args, then the gguf_load_in_flight marker, then the hub guard, then the chat handoff, then unload_model. * Stop a real sd.cpp install from breaking its own discovery tests The five "nothing is installed" assertions in test_sd_cpp_engine.py cleared SD_CLI_PATH and UNSLOTH_SD_CPP_PATH and patched Path.home, which covers hops 1, 2 and the fallback half of hop 3. It leaves two hops live. Hop 3 goes through managed_install_root(), which honors UNSLOTH_STUDIO_HOME and STUDIO_HOME and resolves to the stable-diffusion.cpp directory beside the studio home. That is the documented way to run side-by-side Studios, so anyone who has one set gets a real sd-cli back from a finder the test expects to return None. Hop 4 is the in-tree developer build, which does the same for anyone who built sd.cpp inside the checkout. Isolated with an autouse fixture rather than another helper call, because reaching the failure needs no fixture: SdCppEngine(binary = None) runs the finder from its constructor, which is why test_generate_raises_when_binary_missing failed too. The fixture points the studio home and the in-tree root at an empty tmp tree, so every hop is answered by the test rather than by the box. The in-tree root moves into a named in_tree_install_root() so it can be pointed somewhere empty; behaviour is unchanged, including the OSError and IndexError guard on an unexpected layout. * Make the sd.cpp uninstall test actually extract the code it tests The two sed ranges anchored the production fragments at column 0, but both blocks sit inside the main removal function and are indented, so each range matched nothing and LOOP_FILE / DEFAULT_FILE came out empty. Sourcing an empty file is a no-op, so the suite reported 4 passed / 7 failed: the seven removal assertions failed because nothing ran, and the four that passed were 'kept' assertions that pass trivially when nothing runs. The shell job auto-discovers tests/sh/test_*.sh and runs each under set -e, and this file is not in its skip list, so it was a deterministic red. Anchor on optional leading whitespace and fail loudly on an empty or _remove_path-less fragment, so a future reshuffle of uninstall.sh cannot make the suite vacuous again. Now 11 passed, 0 failed, against the real removal loop and the real default-mode block. * Agree on what an engine can build, and on what the installer owns Two gates, each half-applied. The unbuildable-family gate had one caller, the image branch of the cached repo picker. The GGUF classifier, the local-model classifier and the video branch had none, so on a diffusers too old for a family the picker still offered its GGUF and the load then failed. Meanwhile the loader asserted the diffusers pipeline class before engine selection, so a GGUF this host routes to native sd.cpp, which instantiates no pipeline class at all, was refused with an upgrade instruction that could not help. Both are wrong one-sidedly: hide a family only when NEITHER engine can build it, and demand the diffusers class only when diffusers is what will load it. One predicate, family_buildable_here, now answers both, so the picker and the loader cannot disagree. The population is real on Python 3.9, whose diffusers ceiling is 0.36: Flux2KleinPipeline, Krea2Pipeline and LTX2Pipeline are all absent there, and FLUX.2-klein GGUF is a repo in this PR's title. That assertion also raised RuntimeError, which /images/load maps to 409, the status that otherwise means a load is already in progress, and which escaped /images/download-plan (it catches ValueError and FileNotFoundError) as a bare 500 with the message lost. It is an unloadable pick like every other, so it raises ValueError and both routes answer 400 with the text intact. The managed-binary repair used a path test for ownership while the installer uses a marker. On a managed root without the marker, which is any install predating it, the repair deleted sd-server and the reinstall was then refused, permanently, because the surviving sd-cli keeps the directory non-empty so the marker can never be claimed: the user went from an unrunnable binary to no binary and no way back. Require the marker before discarding, which is the same definition of ours that uninstall.sh already uses to keep a user's own stable-diffusion.cpp checkout. A genuinely interrupted extraction still self-heals, since install() writes the marker before it extracts. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add gguf and av to the studio extra so the wheel install matches studio.txt This branch added both to studio/backend/requirements/studio.txt, which is what install.sh uses, but never to the studio extra in pyproject.toml, which is what 'pip install unsloth[studio]' uses. Nothing else keeps the two in sync, so tests/studio/install/test_studio_extra_matches_requirements.py failed deterministically, and the CI job runs the whole tests/ tree. The drift is not only a red test: gguf backs diffusers' GGUFQuantizationConfig and av does the MP4 encode and audio mux, so a wheel install got a Studio that cannot read a GGUF or export a video, which is most of what this PR adds. * Fix two test-suite defects: the CPU-only patch gate and a popped module The patch backend reaches unsloth_zoo's helpers only through an ~940 MB 'import unsloth', which a CPU-only host cannot complete, so every patch install returned False and nine arch/eager/compile tests failed there -- exactly the runners the retry was narrowed to protect. unsloth_zoo only wants UNSLOTH_IS_PRESENT in the environment, which costs nothing and needs no accelerator, and the conftest already sets its sibling UNSLOTH_ALLOW_CPU. Set it there too, as run.py and main.py already do at module scope, so no real server ever took the expensive route. CPU-only goes from 9 failed to 24 passed, 1 skipped; the GPU run is unchanged at 29 passed. test_setup_cache_env_hf_home popped utils.hf_cache_settings to model a fresh process and never restored it, so a later import built a second module object and rebound it on the utils package. test_hf_cache_settings then wrote its setting into one object while core.inference.diffusion read the other, the same split-module failure the xet shim already had. Restore both bindings in teardown: 21 pass together, and each file still passes alone. * Restore the setup_fail assertions the main merge reverted A fourth instance of the class 9541cc535 fixed: the merge took main's studio/setup.sh, which #7644 changed to abort through the setup_fail helper so desktop mode still emits [TAURI:ERROR], but kept this branch's older copy of the test, which still asserted the literal 'exit 1'. setup.sh is byte-identical to main here and the only diff in this file was the reverted assertions, so take main's version. 60 passed. * Fence the video teardown, reject non-finite clipping knobs, stop a test installing sd.cpp The video backend never had the teardown fence the image backend documents. Its unload and its superseding-load path both signalled the active generation, then did 'with self._generate_lock: pass' and tore the state down with the lock free. A generation queued behind that barrier holds no cancel event yet, so the signal cannot reach it, and Python locks are not FIFO: it won the lock the instant the barrier released it, read a still-loaded state and denoised a whole clip against the pipeline being freed. Reproduced on both paths, where the queued generation returned a finished MP4. Mirror the image backend: count waiters, refuse a generation while one is pending, and tear down inside the barrier with the counter released in a finally. _teardown_state becomes _teardown_state_locked since the caller now holds both locks. max_grad_norm and snr_gamma were bounded on one side only, so 1e309 floated to inf and passed. clip_grad_norm_ then computes an infinite clip coefficient, clamps it to 1.0 and scales nothing, and min(snr, inf) / snr makes every min-SNR weight 1.0. The run starts, reports normal progress and trains with the requested knob silently disabled. Measured both. NaN already failed the bounds; allow_inf_nan makes that explicit. learning_rate and flow_shift already guard this exact vector. test_load_routes_to_sd_cpp_on_cpu stubbed ensure_sd_cpp_binary but not ensure_sd_server_binary, which select_and_activate_engine probes first with installs enabled, so the unit test downloaded and unpacked 108 MB into the developer's real ~/.unsloth when UNSLOTH_STUDIO_HOME was unset. Now 0 bytes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Key the video schedule reset on the resolved defaults, not the repo id A GGUF repo holds several variants, so another client swapping a distilled build for the base one from the same repo changes the steps and guidance the backend reports while repo_id stays put. modelChanged stayed false, the page kept the previous schedule, and later generations ran an 8-step distilled setting on a model that expects roughly 40 steps and CFG 4. Include the reported defaults in the key. They only move when the resident artifact does, so a manual steps change still survives a status poll. * Retire the release-lag pin now that 2026.7.6 carries the relaxed gates The virgin Windows container row that installs from PyPI on purpose was tolerated with continue-on-error, because a Server Core container has no Microsoft Store and so no winget, and the released studio/setup.ps1 hard-stopped on a winget-only git gate and reached for winget again for the VC++ runtime. #7549 relaxed both, but it landed on 2026-07-28 and the newest wheel was 2026.7.5 from the 23rd, so the row could not get past it. The pin carried its own tripwire for exactly this moment, and it has fired: unsloth 2026.7.6 shipped 2026-07-29 and the released wheel now installs end to end, so the step reported success and the assertion errored with "delete this pin and drop continue-on-error from the Install step so this row gates". Doing that. Both rows now gate unconditionally, and the assertion that enumerated the accepted failure signatures goes with the pin it protected, since there is no longer an accepted failure for it to describe. The other three release-lag mentions in this file are descriptive rather than tolerated-failure pins: they explain why the fedora and ubuntu-nonroot legs run with overlay: true, and both already refuse a triton failure as a regression rather than accepting it as lag. Left alone. * Pin the hosted pre-quant onto the plain-torch fp8 kernel A Z-Image GGUF pick at the DEFAULT speed mode, once the hosted fp8 pre-quant is actually reachable, dies at generate with an HTTP 500: torch._dynamo.exc.Unsupported: Operator does not support running with fake tensors. Developer debug context: unsupported operator: mslk.f8f8bf16_rowwise.default _fp8_config already pins KernelPreference.TORCH when it BUILDS a config, precisely because the default AUTO switches to the MSLK kernel wherever an mslk package is importable (sm90+). A hosted checkpoint escapes that pin completely: the preference is serialized per Float8Tensor, and all 239 weights in the published Z-Image-Turbo-FP8 checkpoint carry AUTO. Loading it re-arms the very kernel the pin exists to avoid, mslk.f8f8bf16_rowwise has no fake impl, and the first compiled generate therefore cannot be traced. Isolated away from the product to be sure of the mechanism: quantise one Linear three ways on this box and compile each. KernelPreference.TORCH eager ok, compiled ok KernelPreference.AUTO eager ok, compiled FAILS on mslk.f8f8bf16_rowwise library default eager ok, compiled FAILS the same way So the pin is correct and necessary, and the only gap is that the hosted path never got it. _validate_checkpoint checks scheme, granularity, base model, min_features, exclude tokens and fast_accum, but not this. Rewriting the preference on load is safe: it selects a matmul kernel, it is not weight data, so the tensors stay bit-identical and the checkpoint's own state_dict_sha256 still describes them. It is also the faster path compiled, since the opaque extern call blocks inductor quantize fusion. Why this went unnoticed: the pre-quant repos are private, so nothing that could not read them ever took this path. It becomes the default the moment they are readable. Verified end to end at 1024x1024, 8 steps, speed_mode default, on the published checkpoint: before HTTP 500 at generate, diffusion.generate_failed after pinned 239 weights to the plain-torch fp8 kernel, load 10.0 s, transformer_quant fp8 with compiled engaged, cold 7.9 s, warm 0.87 0.87 0.87 0.86 0.87 s That warm number also beats the 1.4 s recorded for this shape on 2026-07-26. 159 prequant and transformer-quant tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cover the fp8 kernel pin so the 500 cannot come back quietly The hosted pre-quant re-arming MSLK was only visible as an HTTP 500 on a compiled generate, which needs a GPU, a readable private repo and an importable mslk to reproduce. None of that is available in CI, so the pin would rot unnoticed. Three hermetic cases on _pin_kernel_preference instead: AUTO weights are rewritten and already-TORCH ones are left alone (counting only what changed), a weight that refuses the assignment does not sink the whole load, and with no torchao enum available the checkpoint is left exactly as saved. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: tighten comments across the image generation, video and training code Rewrite the comments added by this branch to be shorter and clearer: collapse multi-line explanations into one or two lines, drop comments that only restate the code, and keep the rationale that explains why a choice was made. No code changes. * Studio: let the pipeline-class guard survive a host with no diffusers Backend CI installs the CPU-only dependency set, which has no diffusers, and runs without a GPU. 13 tests failed there for three separate reasons, one of which is a real product bug rather than a test-environment gap. assert_pipeline_class_available did a bare "import diffusers", so on any host without it the call raised ModuleNotFoundError instead of the ValueError its own contract promises. /images/download-plan catches only ValueError and FileNotFoundError, so that escaped as a bare 500 with the message lost, which is the exact failure the guard exists to prevent. It is reachable in production: the native sd.cpp engine serves GGUF picks on a CPU or Apple host where diffusers is never installed. Absent diffusers now returns, since the check answers "is the installed diffusers new enough for this family" and there is no version to judge; a pick that genuinely needs diffusers still fails in the loader with its own message. The rest were tests asserting through gates unrelated to what they cover: - Three cond_cache_dir route tests drive a DiT family, which the start route refuses without an accelerator. They now take the existing dit_train_host fixture, written for this case, so they keep testing the schema on every host. - test_download_plan_forwards_the_load_time_controls stubs the diffusers planner but sends a GGUF pick, which routes to the native planner on a GPU-less host. It now pins the engine, so it tests kwarg forwarding rather than the hardware. Engine selection keeps its own tests. - The pipeline-class guard test needed a real diffusers only for its sweep over every shipped family. That half is split out and skips on its own; the stub-driven refusal needs no diffusers and still runs. Added a test for the absent-diffusers contract above. 409 pass on a GPU host with diffusers; 301 pass and 1 skips with diffusers blocked and no GPU. * Studio: cut the image generation comments down further Second pass over the comments this branch adds. Collapse the multi-line blocks that still read as paragraphs, rewrite the longest one-liners so they say the same thing in fewer words, and drop a stale comment that had drifted away from the constant it described. No code changes: only comments and whitespace. * Studio: workflow rail, sidebar Video row, and Train settings polish Sidebar - Pin Video under Images by default; persist store bumped to v3 so an untouched install adopts it while a custom arrangement is left alone. - Add "Customize sidebar" to the end of the More flyout, opening Settings > Appearance scrolled to the sidebar nav section. Images - Replace the workflow dropdown with an icon rail down the left edge: all seven workflows visible, keyboard nav, tooltips carry the labels. - Workflows stay selectable with no model loaded, so one can be set up before picking a model. Generate is already gated on a loaded model. - Video moves out of the Create/Train strip into a link at the far right, with a matching Images link on the Video page. - Wider gutters around the settings column, headings matched to Train. Train - Field guidance moves into "i" tooltips; only state that limits a control stays on the page. Steps, LoRA rank and the rest gain hints. - /info reports params, qlora_vram_gb, gated and note as fields, so the family note renders as chips. vram_note is rebuilt from them unchanged for older clients. Shared - Number steppers appear on hover or focus. - Lighter control border (#e9e9e9) on Images and Video, light mode only. * Studio: move Advanced inline on Images and Video - Replace the top-bar toggle and right-docked Advanced panel with a disclosure under Seed, so load-time tuning sits with the settings it affects and opening it no longer shrinks the preview canvas. - Video uses the bordered variant, since offload and memory decide whether a model fits at all. - Open state persists per page in localStorage. - Tighten the Steps unit trigger: 14px chevron, less right padding. * Studio: full width media pages and layout polish - Drop the 1100px cap on Images Create, Images Train and Video, so the preview canvas grows with the window instead of sitting in a band. - Align the workflow rail to the model selector label, widen the gap before its divider, and restore the divider itself. - Advanced: more room above and below, icon and label sized to the slider rows, and the same quiet row on Video as on Images. - Give Video a pane heading and description, matching Images. - More space above Seed on both pages. - More no longer takes the active style when the current page is one of its own rows. * Studio: fix the Extend side toggles The resting outline used a ring, and index.css blanks the ring on the button holding mouse focus, so the side you just clicked lost its border until focus moved. Use a border for the outline and leave the ring to focus-visible. Also separates the two states properly (the off state had no surface of its own), drops the border in dark mode as the inputs do, and adds the aria-pressed and focus-visible styling the buttons were missing. * Studio: drop the border on the Extend side toggles Light mode should carry no border either, so the fill alone marks the state: a muted surface when off, a primary tint when on. Matches the borderless treatment dark mode already had. * Studio: move the Images workflows into the sidebar The workflow switcher was a vertical icon rail on the page. It now lives in the sidebar under Images, so the page keeps that width for the canvas. - Workflows list under the Images row, with a chevron to fold them away. Hovering the row peeks them in a flyout on the standard menu surface. - Clicking Images while already there toggles the list instead of navigating. - The listed workflow carries the highlight, so the Images row drops it. - Create takes its own icon: it was sharing the New chat pencil. - Model hub moves above Projects. Persisted layouts bump to v4 and only adopt the new order where the stored one is still a shipped default. - Images and Video content both start at 32px, clear of the sidebar. * Studio: simplify the Images workflow submenu Dropping the hover flyout: two ways to reach the same seven workflows, one of them overlapping the list right below it, read as clutter. - The flyout is gone. The workflows are rows under Images and nothing else. - On the Images page they are always open, since they are that page's switcher. Elsewhere they stay folded, and the row's chevron opens them. - Create takes the sparkles icon. - Create and Train both pad their settings column to 40px a side, so the two tabs line up with each other and with the model selector. * Studio: dock the generate action and tidy the Create controls The primary action sat at the foot of a long scroll, so it was off screen until you scrolled for it. - Generate, Video's Generate and Train's Start training float at the bottom of their settings column. No bar behind them: hover lightens the fill rather than thinning it, and the disabled state is opaque, so the controls underneath never show through. - Aspect ratios read as names: Square (1:1), Widescreen (16:9) and so on. - Width and height replace their sliders with two compact boxes. Type a size or pick one from the menu; the value still snaps into range and still drives the locked ratio. - Negative prompt gets a quiet disclosure under the prompt. It was there before but only above guidance 0, which is not the default, so it never showed. - The info "i" is smaller across the settings UI. * Studio: reveal Images and Video field hints on hover The "i" next to every field label sat there permanently, which made a column of settings read as busier than it is. Scoped to the diffusion pages by CSS rather than per component: the hints come from the page's own Field and SelectRow, the train panel's FieldLabel, and the chat ParamSlider, which is shared and should not change for chat. Reveals when the pointer is inside the field, on focus-within for keyboard, and stays inert while hidden. InfoHint grows a data-slot so the rule has a stable hook, rather than keying off its aria-label. * Studio: one-line sliders, edge fades, shared negative prompt - Steps, Guidance and the rest put label, track and value on one row. ParamSlider is shared with chat, so this is an opt-in prop and chat keeps the stacked layout. - The settings columns fade at whichever edge they run past, as the sidebar and model picker do, instead of cutting off. New useScrollFades hook drives all three. - Negative prompt moves to a shared component and Video picks it up, so both pages collapse it the same way. - Right gutter sits closer to the rule, Steps takes a bigger break above it, and the Advanced rule sits between its neighbours rather than up against the field above. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: add the API monitor to the sidebar nav Sits under More, below Export. Points at the monitor page, not the API keys dialog the profile menu opens. No store version bump: an id missing from a stored layout is appended with its default pinned state, so existing sidebars gain the row at the end and keep their own order. * Studio: take main's copy of the sidebar spinner test The merge kept this branch's older version, which pins the Recents row to h-[33px]. Main already relaxed that to any height, since row density moves independently of the trailing column the test is about. * Studio: fix FLUX.2 klein size resolution, HunyuanVideo tier routing, and the personalization save Five fixes found while auditing the model registries against the live Hub. FLUX.2 klein 9B loaded against the 4B config. One family covers both klein sizes and defaults to 4B, relying on the base_model card tag for the real base, but the trust allowlist only had klein-4B, so the correct tag was discarded and a 9B GGUF (inner_dim 4096) was loaded against a 4B config (3072). That surfaced as a bare shape mismatch from inside the GGUF quantizer naming neither the file nor the repo. Adds the missing allowlist entries and a header-level size check that fails early with a legible message; the check is fail-open, so an unreadable file, a non-FLUX.2 family, or an unmapped base leaves the load exactly as it was. The sd.cpp text encoder rule matched the literal "klein-9b", so klein-base-9B was handed the 4B encoder. HunyuanVideo-1.5 720p checkpoints routed to the 480p family. Only the literal 720p_t2v path was aliased, so 720p_i2v and every GGUF repack fell through to the generic token and inherited the 480p base repo, which also supplies the VAE and text encoder. The tier is baked into the weights (transformer target_size 640 vs 960, scheduler shift 5.0 vs 9.0) and the two bucket lists are disjoint, so this ran the whole pipeline off-tier. Resolution presets that the checkpoints were never trained for. Wan2.2 TI2V-5B is 720P-only upstream (SUPPORTED_SIZES is exactly 704x1280 and 1280x704, asserted in generate.py), and the HunyuanVideo 480p square preset 624x624 is not a bucket of its tier; 640x640 is. Personalization could never be saved. The frontend ships eight sidebar nav ids and refills every missing one on each save, but the backend Literal had seven and no "api", so every PUT to /api/settings/personalization returned 422. Aligns the server defaults and the Literal with the shipped layout. A catalog row pointing at a file that does not exist. unsloth/Qwen-Image-2512-FP8 holds torch prequant .pt checkpoints, not qwen-image-2512-fp8.safetensors, and fp8 is denied for this family anyway because it renders black, yet the row was the auto-route target on any GPU above 40 GB. Also marks FLUX.1-schnell, Krea-2-Turbo and the two Ideogram repos gated: all four are gated on the Hub today, and schnell being Apache-2.0 does not make it ungated. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: stop a case-only duplicate upload destroying an image, and size the load against the right GPU Two fixes from the review round. A dataset upload containing two names that differ only by case silently loses one on Windows and macOS. The in-batch duplicate check is an exact string compare, so "Cat.png" and "cat.png" both pass, but on a case-folding filesystem they are one destination: the commit step moves the first staged part aside, writes the second over it, then deletes the backup, while the response still reports both as uploaded. Captions collapse the same way and never even reach the image branch. No single folder can hold such a pair, but the Windows and macOS open dialogs flatten search and Recents results across folders into one multi-selectable list, which is a normal way a LoRA dataset gets assembled. Rejecting the pair everywhere would regress Linux, where the two really are different files with their own sidecars, so the check probes the filesystem once per process instead of keying off sys.platform: macOS also ships case-sensitive APFS volumes, and a Linux host can keep its Studio home on an exFAT or NTFS mount. A failed probe answers "case-sensitive", which leaves today's behaviour untouched. The quantization fit budget was sized against the wrong card under a reordering CUDA_VISIBLE_DEVICES. A bare "cuda" load lands on visible ordinal 0, but the reducer took the lowest PHYSICAL index, which the nvidia-smi path reports as index_kind "physical". Under "3,1" that sizes against GPU 1 while the pipeline loads onto GPU 3. The hook's own interface doc already promised the lowest visible ordinal. Ranks by visible_ordinal, falling back to index only for an older backend that omits it. * Studio: rework the diffusion dataset panel and align the pane gutters Upload: - one Upload button beside the dataset name, with an upload icon - picking files uploads them, no second confirm click - Add button beside the Training images dropdown, so images can go into a set that already exists - fall back to the upload form when a selected dataset no longer resolves Labeling grid: - two columns at any width; sm:grid-cols-3 crushed the tiles in a fixed width column - pin the caption size so the Textarea's md:text-sm does not outsize a tile - name the tile hover group; a bare one revealed every tile's Remove at once - Remove is an icon button on the image, not a word over the artwork - drop the amber tile fill, keep the No caption label - match the header, toggle and status text to the section's type scale Gutters: - the run area and the preview canvas now sit 40px off the rule, the gutter the settings column has off the page edge * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: give the dataset preview per-image hover and remove, and wrap it onto rows - each thumbnail is its own button, so it can carry an X that removes just that image; the strip was one button and could not nest a second control - hover lightens the image instead of drawing a border - wraps onto more rows rather than scrolling sideways, which also keeps the bottom corners rounded - samples 12 thumbnails, up from 8, now that a second row is available - a delete refreshes the panel's dataset counts * Studio: show the workflow icon beside the Images and Video headings Images reads it off WORKFLOW_TABS, so it is the same icon the sidebar submenu shows and it follows the active workflow. Video's heading is also Create, so it takes the Create icon rather than the nav row's film slate. * Studio: reject a duplicate LoRA id on the diffusion load path too DiffusionGenerateRequest already refuses a repeated adapter id, with a comment naming the hazard: _resolve_lora_set suffixes colliding adapter names, so the same id resolves the SAME adapter twice and set_adapters stacks both copies past the per-adapter weight bound. DiffusionLoadRequest bounds only the list length, and it matters more there. Generation-time stacking spoils one image; on the load path the adapters are baked into the quantized build before compilation, so the unintended combination rides every image until a reload. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: teach the GGUF reuse test double about holds_no_vram The resident-GGUF fast path consults llama_backend.holds_no_vram before asserting CHAT ownership, but two tests hand it a SimpleNamespace built before that attribute existed, so they raised AttributeError and returned a 500. A real LlamaCppBackend exposes it as a property; the doubles now carry it too. Product code is unchanged: these pass on main and were the branch breaking its own test, not a defect in the guard. --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: michaelhan <michaelhan2050@gmail.com> Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: Michael Han <107991372+shimmyshimmer@users.noreply.github.com> | 1 个月前 | |
Use Unsloth for bare Studio product references (#9732) * Studio: use Unsloth for product shorthand * Tests: restore case-folding fixture path * Complete Unsloth product reference rename * Tests: restore internal Studio references | 9 天前 | |
Studio: add image generation, editing workflows and LoRA training with Unsloth GGUFs (#6763) * Tighten comments in the image stack tests and scripts * Close video single-file, training reservation, and image mount-resume gaps Route on-device single-checkpoint video folders through the single_file loader: a bare local .safetensors directory (no model_index.json) is advertised as a pipeline with no filename, so validation rejected it before it could load. Reinterpret the pick as a single_file load of the sole checkpoint, mirroring the image load route. Treat a reserved-but-not-yet-spawned LLM training start as active in is_training_active() so /images/load, /video/load, and /diffusion/start cannot race the reserved run for VRAM during the pre-spawn free window. Mirrors the diffusion training service reservation. Resume an in-flight image generation on the Images page mount: probe generate-progress, re-enter the poll loop, and refresh the gallery on completion so a run started elsewhere is reflected and its saved image appears without a manual refresh. Seed resident image defaults from the resolved base_repo rather than a possibly path-shaped repo_id so the first resident generation uses the right recipe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Publish image generation active state before pre-denoise setup generate() assigned self._gen only at the pipe() call, after deferred compile, LoRA resolution/application, and ControlNet download/build had run. Across that setup window generate_progress() reported inactive even though _generate_lock was held, so a reloaded page's mount probe showed idle and let a second generate queue behind the first. Publish an active step-0 _GenState the moment the generation lock is acquired, before the setup work, and clear it in the outer finally so a setup-time error cannot leave the UI stuck active. Mirrors the video backend's queued phase and the training start guard. * Studio: fix diffusion install ownership, dataset upload atomicity, gallery pagination, and teardown races install_sd_cpp_prebuilt: only write the .unsloth-studio-owned marker when the install created the target directory or it was empty. Adopting a pre-existing, unowned, non-empty directory (a user's own stable-diffusion.cpp checkout) made it eligible for the uninstaller's recursive delete. routes/training upload: make the multi-file promotion transactional. Back up each displaced original and roll every destination back on any failure, so a mid-loop rename error can no longer partially overwrite the live dataset. routes/training _resolve_dataset_folder: reject a symlinked dataset directory and prove the resolved folder stays under the datasets root, so image read/caption/delete cannot escape the root through a link. routes/training delete: escape glob metacharacters in the thumbnail filename so deleting an image named like [ab].png removes only its own thumbnails. image_gallery / video_gallery listing: filter records against the response schema inside the pager via a valid callback, so offset/limit/has_more all count over accepted records. A leading schema-invalid record no longer returns an empty page with has_more=true and stalls infinite scroll at offset 0. image_gallery / video_gallery save: publish via a temp file plus atomic rename (the sidecar is the video pair's commit marker) and clean up on failure, so a partial write never surfaces a truncated PNG or strands an orphan MP4. diffusion_train_common discovery: treat an empty caption sidecar as a metadata tombstone that still falls through to the dreambooth instance prompt, so clearing every metadata caption no longer fails with no captioned images found. diffusion backend unload: wait for an in-flight denoise to exit before tearing down process-wide patches and state, mirroring the load path. diffusion_engine_router: serialize the whole check/unload/publish transition so a concurrent selection cannot return the engine being unloaded. uninstall.ps1: gate the default sd.cpp process stop on the owner marker so a user's own sd-server is not terminated for a directory we then keep. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reject native batch seeds outside the JSON-safe range * Refuse sd.cpp install into unowned non-empty target dir When the install target already exists, is non-empty and lacks the .unsloth-studio-owned marker (a user's own stable-diffusion.cpp checkout, or unrelated files beside a custom Studio root), install() previously still extracted the release into it. Skipping the ownership marker only stopped the uninstaller from deleting the directory; extraction still merged binaries into the user's working tree and could overwrite same-named files. Fail up front with a clear message pointing the user at a fresh/empty location before any download or extraction, leaving their directory untouched. Update the ownership test suite to assert the refusal. * Studio: gate dataset uploads on the symlink check and surface local video single-file checkpoints * Tighten comments and docstrings added by the image-generation fixes * Studio: close arbiter load-registration race and surface native progress + local pipeline folders Publish native sd.cpp generate progress (_gen) before LoRA resolution so a reload probe reads active during setup, matching the diffusers path. Register the diffusion/video GPU load under the arbiter lock (acquire_for now takes a register callback) so a competing acquire cannot evict an owner before its load is marked in-flight and let two loaders allocate VRAM at once. Admit local diffusers pipeline folders (root model_index.json, weights in component subdirs) in the local model scan so they reach task tagging and the On Device picker. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: include marker-owned custom sd.cpp roots in the uninstall stop scan; harden Pester install against the nuget.exe PSGallery bootstrap * Studio: surface local pipeline scan roots, tag single-file checkpoints by filename, mark companion-only pipelines partial - _scan_models_dir: admit a scan folder that is itself a diffusers pipeline (root model_index.json, weights in transformer/ vae/ subdirs). _is_model_directory rejects such a root, so the child scan would list the component subdirs as bogus models and hide the real pipeline; treat the root as one model via _local_pipeline_index. - _local_is_diffusers / _local_model_task: include the sole checkpoint filename in the family-detection needles (_local_family_needles, resolved via resolve_local_single_file). A generically named folder holding one loadable qwen-image-*.safetensors / ltx-*.safetensors identifies its family only from the filename; the load route already resolves that file, so tag it or the task-scoped picker (which rejects task=null) hides the on-device model. - list_cached_models: mark a companion-only base snapshot partial. A GGUF image load prefetches the base repo's VAE / text-encoder / model_index.json but skips the transformer (the GGUF supplies it); the snapshot has a pipeline manifest yet is not a loadable BF16 pipeline, and _cached_repo_partial misses it. _repo_pipeline_missing_denoiser flags a pipeline snapshot whose transformer/ or unet/ component carries no weight, so the picker drops it instead of advertising it as fully on-device. * Studio: preserve foreign gallery files, force safetensors on remote ControlNets, and close dataset/seed/GPU gaps Gallery clear/delete now scope to Studio-owned files: image_gallery and video_gallery skip PNGs / MP4s without a readable recipe (a hand-dropped or orphan file the listing already hides), so clear() and a guessed-id delete no longer destroy files the gallery never surfaced. Remote ControlNets now force use_safetensors: a bare owner/name reaches from_pretrained without the base trust gate, and the Hub scan fails open when unavailable, so requiring safetensors closes the pickle deserialization vector. POSIX uninstall now stops resident sd-server / sd-cli under an owned sd.cpp root before removing the tree (marker-gated), mirroring the Windows stop-before-delete scan; a live native server no longer survives unlinking its binary. Diffusion dataset containment: the training-start read path and the discovery picker route bare names through the protected resolver, so a symlinked dataset is rejected / not advertised like the caption/delete routes already do. Uploads gain the inference decode guard (oversized real images 400 before OOMing the trainer) and dataset upload/caption/delete/import are blocked with 409 while a diffusion run is active. JSONL readers (trainer + routes) tolerate non-object JSON and invalid UTF-8 instead of raising AttributeError / 500. LoRA family compatibility is enforced in the shared resolver, not only the picker, so a direct API client cannot apply a mismatched-family adapter. GPU arbiter gains release_if so the image/video unload idle-check and release are atomic against a concurrent same-owner load's registration. Native batch recipes persist the base batch_seed and restore replays from it, so a native batch_index>0 image no longer advances its seed twice. FLUX.2-klein selects its sd.cpp text encoder by variant (4B -> Qwen3-4B, 9B -> Qwen3-8B) instead of the single family default. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: track the loaded GGUF filename so native companion resolution reproduces the load identity * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: gate local-pipeline image tagging on a real family; validate video sidecars before delete/clear * Studio: tighten image-generation fix comments and docstrings * Studio: gate gallery serve/export on ownership; keep image progress active until persisted; reserve diffusion training before the dataset scan * Studio: restore Reapply target on async image/video load errors; recheck training state before dataset commit; reclaim partial sd.cpp installs on retry Images/Video: a background model load that fails AFTER starting (error/eviction during download) leaves the previous pipeline resident, but handleLoad had already overwritten lastLoad.current with the failed pick, so "Reapply to loaded model" reloaded the failed model. Carry the prior Reapply target into the poller and restore it on the async error/null paths, mirroring the quant rollback. Training: an in-flight diffusion dataset upload passed _require_diffusion_dataset_mutable() at entry but could still commit files after a concurrent /diffusion/start reserved the training slot, mutating the dataset underneath the trainer. Re-check the interlock immediately before the commit phase; a 409 there leaves the staged temps for the finally to clean. sd.cpp install: an interrupted extraction (disk full, killed process, a raising post-extract cudart fetch) left the target non-empty with no owner marker, so the next lazy install tripped the "not a Studio-managed directory" refusal and wedged native install. Write the ownership marker before the partial writes when the target is reclaimable, so a retry recognises the debris as ours and re-extracts. * fp8 DiT quant: floor the dynamic activation scale with activation_value_lb An all-zero activation token row makes the dynamic per-row fp8 scale 0, which turns the quantized data to NaN and the render to black frames on torchao's plain-torch kernel path. The fused fbgemm/mslk quantize kernels clamp zero rows internally, so the bug only reproduces on machines without them, which is most user environments. Zero rows are real inputs, not a corner case: Wan 2.2 zero-pads its text conditioning, and Hunyuan-1.5 and Qwen-Image regenerate zero rows inside their transformer blocks every step. Pass activation_value_lb=1e-12 to Float8DynamicActivationFloat8WeightConfig whenever the installed torchao supports the kwarg (Float8Tensor rework, 0.13+), checked via inspect.signature so older torchao keeps exactly the current behaviour; the existing Float8MMConfig fallback chain is unchanged. Verified on GPU: with the forced plain-torch kernel path a zero-row input NaNs without the floor and stays finite with it, and end to end on HunyuanVideo-1.5 fp8 goes from a solid black frame (LPIPS 1.00) to a normal render (LPIPS 0.225); on Wan the floor matches the condition_embedder exclusion (LPIPS 0.211 vs 0.206). Same-seed renders with fused kernels present are unaffected, and pre-quantized fp8 checkpoints stay valid since weight scales are untouched. * Wire hosted pre-quantized DiT checkpoints into the image families Point prequant_repos for flux.1, flux.2-klein, flux.2-dev, qwen-image (int8 only there; fp8 is family-denied), z-image and krea-2 at the unsloth/<Model>-FP8 Hub repos carrying gate-validated int8 and fp8 transformer checkpoints, so the fast quant path loads the small pre-quantized file instead of materialising the dense bf16 transformer and quantising on device. Measured on FLUX.2-dev int8: build peak drops from 60.7 GB (dense + quantize) to 30.7 GB (hosted prequant), identical 30.7 GB resident after either path since loading a checkpoint is bit-identical to on-the-fly quantisation. The hosted repos name files <Model>-<SCHEME>.pt, so resolve_prequant_source now derives that model-name filename from the repo id (scheme suffix stripped case-insensitively) and carries the legacy transformer_<scheme>.pt as a fallback the resolver tries when the primary 404s, keeping older repos loadable. Wiring a repo also exposed a fallback hazard: with a prequant source present, the dense-fit preflight used to be skipped entirely, so a failed prequant download would fall through to the dense bf16 load the memory plan never budgeted, OOMing after eviction. The preflight now always runs and gates an allow_dense_fallback flag through _load_dense_quant_pipeline: a dense misfit still skips the fast path when no prequant exists, but with one it proceeds and a prequant failure raises to the GGUF build instead of loading dense. The same flag is set when the auto-policy replans an offloaded GGUF against a prequant-sized transient. Tests updated to the new filename convention plus new coverage for the derivation and the legacy-name fallback; the prequant-skips-refit test now asserts the re-check runs and forbids the dense fallback. Verified end to end on GPU: z-image int8 resolves the hosted repo, downloads the model-name file and renders (6.8s load, 5.9 GB peak). * Route krea-2 through its per-component loader on the transformer-quant fast path _assemble_pipe used Pipeline.from_pretrained for every family, but the krea repo ships transformers-5.x configs and no top-level tokenizer files, so the tokenizer dies with vocab_file=None. The pre-quantized checkpoint loaded fine and then the assembly crashed, dropping the load to the GGUF build, which krea-2 cannot take (Krea2Transformer2DModel has no from_single_file). Assemble per-component via load_krea2_pipeline like the pipeline-kind and single-file paths already do. Verified live: Krea-2-Turbo int8 and fp8 hosted prequant loads now assemble and render through the Studio images tab. * Keep Qwen-Image's text-stream linears bf16 on int8 (short prompts break torch._int_mm) Qwen-Image's MMDiT runs every text-stream Linear at M = actual prompt tokens: the Qwen2.5-VL embeds are not padded to a fixed length like FLUX's 512-token T5. A short prompt (13 tokens) or the near-empty negative prompt drives torch._int_mm below its M > 16 floor and the first denoise step raises 'self.size(0) needs to be greater than 16, but got 13' (measured on B200 through the Studio images tab). Add per-family int8 exclusions (txt_in, add_q/k/v_proj, to_add_out, txt_mlp) for qwen-image and qwen-image-edit, threaded through exclude_tokens_for_scheme(scheme, family) and the prequant checkpoint validation, so a checkpoint baked under the old token list is rejected and re-quantised instead of loaded crashing. The text stream runs at M = tens vs the image stream's M ~ 4k, so the exclusion costs nothing; the rebuilt hosted checkpoint gates 28/28 PASS with LPIPS mean 0.057 (was 0.069). * Harden the diffusion memory plan against transient free-VRAM undercounts A cold FLUX.2-dev int8 load on an idle 183 GB B200 planned offload=model (companions exceed budget) and silently served the GGUF as-is; the identical retry went resident and engaged the hosted prequant. The plan arithmetic was byte-identical across both loads (required 90,228 MiB, resident needs free of about 124 GB); the only divergent input was torch.cuda.mem_get_info, which is device-wide and instantaneous: a transient foreign CUDA context briefly held about 100 GB at the first snapshot, and the planner trusted that single read. Three changes: - settled_snapshot_device_memory: on cuda, synchronize + empty_cache (best-effort) and take the MAX free over up to 3 spaced reads. A transient can only shrink free, so the max rejects transient undercounts while a persistent tenant still caps every read. _plan_memory now uses it. - plan_fits_total_capacity + one replan retry: when the dense/prequant candidate fits TOTAL device capacity under the standard reserve and the 0.85 resident margin, an offload verdict can only stem from the free reading, so the loader re-snapshots and replans once before declining the fast path. Explicit balanced/low_vram modes skip the retry (they offload by mode). - diffusion.transformer_quant_declined log line with required/budget/free and the plan reasons, so the next decline is diagnosable from the server log (previously silent). Verified: cold FLUX.2-dev int8 first load in a fresh server now engages the hosted prequant resident (offload=none). * Add FLUX.2 Klein and FLUX.2-dev DiT LoRA training Register flux.2-klein and flux.2-dev in the DiT trainer following the upstream DreamBooth references: latents train patchified and batch-norm normalized from the VAE posterior mode, the packed forward reuses step-invariant position ids, and the guidance vector (3.5) is gated on the variant's guidance_embeds config. Conditioning stacks load per variant (Mistral via Flux2Pipeline for dev, Qwen3 via Flux2KleinPipeline for Klein) and are encoded and freed before the transformer lands on the device. The fused single-stream to_qkv_mlp_proj joins the attention projections in the LoRA targets; the single-stream out projection stays dense because its to_out suffix would also match the double-stream ModuleList container. Wire both families through the training registry (family set, labels, VRAM notes, rank 16 / lr 1e-4 defaults, bf16-only preflight), mark them trainable with train base repos in the family registry, add FLUX.2-dev to the gated-repo token check, and trust both official bases for training downloads. Verified on B200: 30-step klein int8 (19.6s) and nf4 (20.9s) and dev int8 (52.0s) runs train with finite decreasing loss and the saved adapters apply on the bf16 base pipeline (weight 0 reproduces the base image exactly, weight 1 visibly restyles it). * Support LoRA adapters on torchao int8/fp8 quantized image pipelines Adapters are baked at load time: they attach to the dense transformer, then quantize_ converts only the frozen base linears (the lora_ side path is excluded by name), then the loader compiles. Post-quant PEFT injection is not possible on a manually quantized module, so the prequant shortcut is skipped for a baked load and the memory plan is sized for the dense build (force_dense on the quant candidate). At generation time the baked topology is frozen: weight tweaks and disabling (scale 0 reproduces the quantized base exactly) go through set_adapters, while adding or removing adapters returns a clean 400 telling the client to reload with the new selection. supports_lora now returns True for int8/fp8 diffusers loads (checked before the gguf-kind early return, since the quant fast path keeps the picker kind); nvfp4/mxfp8 and GGUF-via-diffusers stay blocked. The load request model takes an optional loras list, threaded through begin_load on both engines (native ignores it and keeps applying LoRA at generation). Verified end to end on GPU: Z-Image GGUF picker + int8 + trained adapter loads through the API, bake marker logged, weight 1.0 vs 0 renders differ visibly, weight 0.5 accepted live, unknown adapter rejected as 400. Affected suites: 296 passed. * Add FLUX.1 Krea dev to the image model catalog Krea's guidance-distilled FLUX.1-dev finetune keeps the exact dev layout, so it runs under the existing flux.1 family unchanged. Wire it up end to end: - Catalog group with the gated official bf16 pipeline and the open QuantStack GGUF quants; the gated artifact is skipped on auto-routing when undownloaded. - Trust the official repo for non-GGUF from_pretrained loads, next to the other black-forest-labs bases. - Generation defaults: 28 steps at guidance 4.5 per the model card. The generic "krea" defaults key (Krea-2-Turbo's 8-step no-CFG recipe) used to swallow the id, which would have produced garbage output; the new flux.1-krea key precedes it on both the backend table and the images page table. - The flux.1 prequant checkpoints are schnell-based; the loader's baked base_model_id validation refuses them for the Krea-dev base, so int8/fp8 requests dense-quantize instead (covered by existing prequant tests). * Resolve pre-quantized checkpoints per base variant One family entry covers several published variants whose weights differ (flux.1: schnell, dev, Krea-dev), but prequant resolution was keyed on (family, scheme) alone, so only the default base could ever be served: the loader's baked base_model_id validation correctly refused the schnell checkpoint for dev and Krea-dev bases and every such load paid the dense download plus on-the-fly quantise. Add an optional prequant_variant_repos table on DiffusionFamily as (base_repo, scheme, repo_id) triples and thread the resolved base repo through resolve_prequant_source / usable_prequant_source and their three call sites (load fast path, memory-plan probe, auto-policy candidate). A base without its own entry keeps returning the family default, preserving the existing refuse-then-dense behavior exactly. Wire the flux.1 variants: the gate-validated unsloth/FLUX.1-dev-FP8 checkpoints (built in the earlier campaign but never reachable) and the new unsloth/FLUX.1-Krea-dev-FP8. * Add the Lumina Image 2.0 family to the image catalog Alpha-VLLM/Lumina-Image-2.0 is a 2.6B single-stream DiT with a Gemma2-2B encoder and a standard 16-channel VAE, all transformers-4.x-compatible, so the generic from_pretrained pipeline path loads it as a new lumina-2 family: - Family entry (Lumina2Pipeline / Lumina2Transformer2DModel), aliased to lumina-image-2.0 / lumina-image-2 / lumina2. No bare lumina alias: Lumina-Next checkpoints are a different arch and must stay unknown rather than crash mid-load. bf16-only upstream, so the fp16 fallback stays off like z-image. - Trust the official repo for non-GGUF loads; bf16 component table entry (ships fp32, ~5.2 GB transformer + 5.2 GB encoder bf16-resident). - Generation defaults 50 steps / guidance 4.0 per the model card, and the generate call passes the card's cfg_trunc_ratio=0.25 itself (family-gated, signature-gated): the pipeline default (1.0) runs the CFG double-forward on every step and oversaturates output. - Catalog group with the single ungated bf16 pipeline artifact (11 GB resident) plus routing assertions; images page defaults row. - No GGUF artifact: none exists upstream (only finetune/LLM quants), so the dense transformer_quant fast path (GGUF-kind-only) stays unreachable for now. Offline probes of the future prequant campaign: int8 and fp8 both engage and render cleanly (fp8 LPIPS 0.11 vs bf16, int8 0.33 from 50-step trajectory drift with intact quality), so neither scheme is family-denied. * Wire the hosted Lumina Image 2.0 int8/fp8 checkpoints Gate-validated against same-seed bf16 renders (28/28 pairs per scheme, zero failures): int8 LPIPS mean 0.146 / SSIM 0.937, fp8 LPIPS mean 0.116 / SSIM 0.946. Uploaded to unsloth/Lumina-Image-2.0-FP8 following the existing checkpoint repo conventions. * Add the HunyuanImage 2.1 family to the image backend The hunyuanvideo-community diffusers mirror carries the full stack in standard layout: a 17B dual-stream DiT (32.5 GB bf16), a Qwen2.5-VL text encoder, a ByT5 glyph encoder, the 32x HunyuanImage VAE, and guider/ocr_guider components (AdaptiveProjectedMixGuidance) that diffusers 0.39 loads natively, so the generic from_pretrained pipeline path covers everything with no per-component assembly. Family notes: - The call's guidance knob is distilled_guidance_scale (there is no guidance_scale kwarg), so cfg_kwarg routes the UI value there; real CFG runs inside the repo's guider at its baked scale. Defaults follow the card recipe: 50 steps, 3.25. - 2K-native: verified live at both 1024 and 2048. - Coexists with the HunyuanImage-3.0 structured exclusion (3.0 has no diffusers pipeline and stays excluded with its stated reason). - int8/fp8 dense quantization verified live (LPIPS 0.186 both vs same-seed bf16); a short prompt does not trip the int8 torch._int_mm minimum on this arch, so no family exclude entry is needed. - bf16 component table for the memory planner: (32.5, 16.3, 0.8) GB. * Surface HunyuanImage 2.1 in the image model catalog Catalog group with the open bf16 mirror pipeline (~50 GB resident, so a bare click on a consumer card routes to the QuantStack GGUF quants, which load and render through the generic GGUF path, verified live) plus the images page defaults (50 steps, guidance 3.25 feeding distilled_guidance_scale). * Add the HiDream-I1 family to the image backend A 17B MoE DiT (16 double + 32 single layers, 4 routed experts) with four text encoders, on HiDreamImagePipeline (diffusers 0.39). One family covers the open Full / Dev / Fast repos (same arch); per-variant generation defaults follow the upstream inference recipes (Full 50 steps at guidance 5, the distilled Dev 28 and Fast 16 guidance-free). The repos name a Llama-3.1-8B text_encoder_4 in their model_index but do not ship its weights; the official example passes the gated meta-llama repo in by hand. The loader instead assembles the component from the open unsloth mirror (byte-identical weights, already inside the non-GGUF trust gate), injected at the three pipeline from_pretrained sites, with output_hidden_states matching the official example. Memory planning counts the assembled TE4: 34.2 GB DiT + 28.8 GB encoders, ~63 GB bf16-resident. * Surface HiDream I1 in the image model catalog One catalog group with the three official bf16 pipelines (Full, plus the Dev and Fast distillations as labeled artifacts) at their ~63 GB resident size, so auto-routing keeps this a datacenter-GPU pick. city96's GGUF is deliberately not wired: the GGUF path would need the same Llama TE4 assembly for very small demand. Images-page defaults mirror the backend table with the variant keys ahead of the generic hidream key. * Pin the measured HiDream quant verdict in tests int8 and fp8 both engage and render cleanly on this family, including short prompts on int8: the routed MoE expert Linears only ever see the concatenated image+text stream (M far above the torch._int_mm minimum), so no deny entry and no family exclude tokens are warranted. Assert that so a future table edit cannot silently regress the measured behavior. * Wire the hosted HunyuanImage 2.1 int8/fp8 checkpoints Verified bit-identical to on-the-fly quantize: all 1264 state dict tensors (456 quantized) dequantize equal between the loaded checkpoint and a fresh quantize_ pass, so quality matches the runtime Dtype path exactly. Same-seed LPIPS suite means (0.35 int8 / 0.28 fp8) blend trajectory divergence with this family's own run-to-run nondeterminism (identical weights and seed reproduce a 17/255 mean pixel delta through the 50-step guider pipeline); per-case hard checks pass and the drift is compositional, reviewed visually. Uploaded to unsloth/HunyuanImage-2.1-FP8. * Fix silent LoRA drop and wasted transformer prefetch on GGUF quant loads Two live-test findings on the images load path: - transformer_quant with baked LoRAs, when the dense quantized build is declined for memory or fails: the load completed as a plain GGUF with the adapters silently dropped (HTTP success, supports_lora=false after the fact) -- wrong output with no signal. The load now fails with the recovery options (drop the adapters, free VRAM, or pick a smaller model). Weight-0 adapters still count as no bake request, and the plain no-LoRA decline keeps its silent GGUF fallback. - A fresh GGUF load on a small GPU prefetched the base repo's full bf16 transformer shards (~47 GB on Qwen-Image) because the dense-quant prefetch widening only checked scheme viability, not whether the device could ever hold the candidate resident. Gate the widening on total device capacity (reserve + 0.85 margin, the plan_fits_total_capacity bar) so a card that is certain to decline the dense build never pays the download; capable devices keep the prefetch. * Fix video progress under-reporting during load and generate Two live-test findings on the video progress endpoints: - load-progress downloaded_bytes froze mid-download: the counter used scan_cache_dir, which skips in-flight *.incomplete blobs, so it sat at the last completed blob for the whole multi-GB shard pull while the disk kept filling. Count the repo's cache directory directly (completed plus incomplete blobs, snapshot symlinks skipped so nothing is double-counted). - generate-progress reported total_steps=null / fraction=0 while step advanced: the video API only carried the native total field while the image API exposes total_steps and fraction, so one poller could not work against both. Derive the image-compatible aliases in generate_progress and declare them on the response model; the native total stays for back-compat. * Wire the hosted HiDream I1 int8/fp8 checkpoints Gate-validated: all 28 per-case pairs pass per scheme (LPIPS suite means 0.291 int8 / 0.278 fp8, in the 50-step trajectory-divergence band; CLIP delta means 0.007-0.008), and the int8 checkpoint is verified bit-identical to on-the-fly quantize across all 1615 state dict tensors (1073 quantized, max abs diff 0.0). Uploaded to unsloth/HiDream-I1-Full-FP8. * Add a pre-cast text-encoder loader for the layerwise fp8 scheme The runtime text_encoder_quant=fp8 path downloads the full bf16 text encoder and layerwise-casts it in place on every fresh load. For the heavyweight encoders (LTX's Gemma3-27B ~50 GB, FLUX.2-dev's Mistral-24B ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) that download dominates load time on a fresh machine. diffusion_te_prequant.py loads a pre-cast fp8-storage state dict instead: meta-init the encoder skeleton from the checkpoint's te_class, load_state_dict(assign=True), rebuild on CPU if non-persistent buffers stay on meta, then re-apply the same layerwise cast to install the upcast hooks. The cast is a deterministic storage transform, so the loaded encoder is bit-identical to dense-load-then-cast by construction. v1 hosts the layerwise fp8 storage scheme only: its state dict is plain tensors (torch.load(weights_only=True), no pickle execution). The dynamic-compute schemes (fp8_dynamic, int8, nvfp4) build torchao subclass wrappers at runtime and are deliberately not hosted. Checkpoints validate format, scheme, component and base_model_id before use and any problem falls back to the dense download and cast. Local path overrides reuse the DiT prequant allowlist env var. Families opt in via a new te_prequant_repos (scheme, component, repo_id) field on both DiffusionFamily and VideoFamily; the field defaults empty so nothing changes until a gate-validated artifact is wired. * Inject hosted pre-cast text encoders during pipeline assembly Wire te_prequant_pipe_kwargs into the three pipeline assembly sites: the diffusion full-pipeline branch, the diffusion transformer-only and GGUF branch (where the companion TE is the big remaining download), and the shared video assembly path before the pipeline/component split. Injection is gated exactly like the runtime cast (mode normalized to fp8, device supported, family not denied), so it can never engage where quantize_text_encoders would not; the later quantize_text_encoders call re-applies the cast idempotently and keeps status reporting truthful. With no hosted checkpoint configured the call returns {} and assembly loads the dense encoder as before. * Add the pre-cast text-encoder checkpoint builder Applies the runtime layerwise fp8 storage cast to a model's dense text encoder once and saves the cast state dict with baked metadata (format tag, base_model_id, family, scheme, component, te_class, versions) in the layout diffusion_te_prequant.py validates. Resolves the encoder class from the checkpoint's config.architectures so the recorded te_class matches what the pipeline instantiates. CPU-runnable: the cast touches storage dtypes only. * Test the pre-cast text-encoder load path Hermetic CPU coverage for diffusion_te_prequant: the checkpoint filename convention, family-table resolution by scheme and component with malformed entries skipped, resolution priority (path override, hosted repo, none) and the fp8-only scheme gate, the checkpoint validation matrix (wrong format, missing state_dict, wrong scheme, wrong component, wrong or missing base_model_id) with base case folding, the local-path allowlist refusal and missing-file fallback, and the assembly injection gating (mode, hosted entry, device support, family deny, load failure, successful injection). Also pins the te_prequant_repos field on both family dataclasses and that no family ships a hosted TE checkpoint until the campaign wires one. * Fix pre-cast TE checkpoint loading and engagement reporting Two bugs found while building the hosted checkpoints: - The builder recorded torch.__version__ (a TorchVersion object) in the checkpoint metadata, so torch.load(weights_only=True) rejected every artifact and the loader silently fell back to the dense download. Record plain strings. - Re-applying the layerwise fp8 cast to an injected pre-cast encoder raised on the duplicate hook registration, making quantize_text_encoders report the engaged cast as failed (status showed no TE quant while the encoder ran fp8). _cast_fp8 now returns early when the hooks are already installed. Also corrects the LTX TE size note: Gemma3-12B stored fp32 (~49 GB), not 27B. * Wire the hosted pre-cast fp8 text encoders qwen-image and flux.2-dev (diffusion) and ltx-2 (video) now resolve a hosted pre-cast fp8 text encoder from their unsloth -FP8 repos: - unsloth/Qwen-Image-FP8: Qwen2.5-VL-7B, 16.6 GB dense -> 8.8 GB - unsloth/FLUX.2-dev-FP8: Mistral-Small-24B, 48.0 GB dense -> 24.7 GB - unsloth/LTX-2-FP8: Gemma3-12B, 48.7 GB fp32 store -> 13.2 GB Every checkpoint verified bit-identical to dense-load-then-cast (729 / 585 / 1066 tensors, zero mismatches) and smoke-tested through the real backends with the repo engagement marker. Tests cover the wired entries, the resolver filenames, builder metadata weights_only survival, and the idempotent re-cast. * Report the compute dtype on fp8-cast encoders and inject the pre-cast TE on the dense fast path Two more findings from the hosted-TE GPU smokes: - Module.dtype reports the first floating parameter, which after the layerwise fp8 cast is the fp8 STORAGE dtype. Flux2 derives its prompt embed and latent dtypes from encoder.dtype and feeds them to randn_tensor, which has no fp8 kernel, so ANY flux.2 load with text_encoder_quant=fp8 crashed at generation (pre-existing, runtime cast included). The cast now swaps in a subclass whose dtype property reports the compute dtype; forward behaviour is unchanged. - The dense transformer_quant fast path assembles companions through _assemble_pipe, which never received the pre-cast TE injection, so the hosted encoder engaged on full-pipeline and GGUF builds but not on the fast path. Threaded through like the other two branches. Verified live on B200: qwen-image (full pipeline), flux.2-dev (GGUF picker with int8 DiT prequant), ltx-2 (video backend) all engage the hosted TE, render non-black, and report text_encoder_quant=fp8 truthfully. * Key the fp8 cast idempotency on an explicit completion marker Hook presence alone cannot distinguish a legitimately pre-cast text encoder from leftover hooks after a cast that failed mid-pass, so the early return now requires the completion marker _cast_fp8 sets once the hooks are fully installed. Leftover partial state keeps failing closed. Also tolerates non-Module encoder doubles in the hook probe and the dtype override. * Extend the fp8 TE quant to HiDream's Llama text_encoder_4 The generic quantize_text_encoders pass only covers text_encoder.._3, so HiDream's HEAVIEST encoder (Llama-3.1-8B TE4, 16.1 GB bf16) always stayed dense. TE4 is assembled separately (hidream_te4_kwargs), so the fp8 path now lives there: when the requested TE quant is layerwise fp8 and the device/family qualify, TE4 prefers the hosted pre-cast checkpoint (unsloth/HiDream-I1-Full-FP8, 8.6 GB) and falls back to dense-load-then- cast; a mid-pass cast failure reloads a fresh dense encoder instead of shipping partial state. The pre-cast loader and builder gain config_subfolder/config_overrides for standalone encoder repos whose config sits at the root and whose pipeline needs forward flags (output_hidden_states/attentions). Verified on B200: bit-identity 291 tensors (225 fp8, 0 mismatches), hosted checkpoint engages through the real backend (marker + status fp8), load 24.3 s vs 48.0 s dense, LPIPS 0.133 mean over 3 same-seed pairs vs the dense-TE render (gate 0.25), non-black frames. * Correct the ltx-2 resident TE estimate to the bf16 cast size The memory plan's bf16_components_gb held 50.4 GB for the LTX text encoder, which is the fp32 hub store of Gemma3-12B (~49 GB download), not what sits on device: the pipeline loads it torch_dtype=bf16, ~24.4 GB resident. The 26 GB over-estimate pushed the auto plan toward offload on cards that fit the real footprint. Comments and the size-table test now pin the resident semantics. * Host pre-cast fp8 text encoders for four more families Round 2 of the hosted TE set, each bit-identical to dense-load-then-cast and gated through the real backend (marker + status fp8 + same-seed LPIPS vs dense TEs): - FLUX.1 T5-XXL (text_encoder_2): 9.52 -> 5.90 GB, one artifact for schnell/dev/Krea-dev (T5 shards byte-identical across all three, verified sha256). 220 tensors, 144 fp8, LPIPS 0.109. - Lumina Gemma2-2B: fp32 hub store 10.46 -> 3.20 GB (3.3x download cut). 288 tensors, 182 fp8, LPIPS 0.041. - Z-Image Qwen3-4B: 8.04 -> 4.41 GB. 399 tensors, 252 fp8, LPIPS 0.112. NOT shared with flux.2-klein-4B: klein retrained layer 35's MLP (verified tensor diff, maxdiff 0.86), so klein hosts no entry. - Krea-2 Qwen3-VL-4B: 8.88 -> 4.83 GB. 713 tensors, 460 fp8, LPIPS 0.082. The constructor-assembled krea pipeline takes the encoder directly (load_krea2_pipeline text_encoder kwarg); the loader remaps 5.x rope_parameters and re-ties weights after assign so the rebuilt encoder matches the builder's structure. HunyuanImage 2.1 reuses the Qwen-Image artifact outright: its Qwen2.5-VL text encoder is byte-identical (every shard sha256, 16,584,414,544 bytes), recorded in the new component-level base-equivalence table the checkpoint validator consults. The injection loop now covers text_encoder.._3 so a family can host several components. Live check: LPIPS 0.123 vs dense. * Report the fp8-cast compute dtype without swapping the encoder class The dtype override swapped encoder.__class__ to a dynamic subclass, which breaks transformers' kwargs-based output recording: a fp8-cast Qwen3VLModel stopped returning hidden_states and every krea-2 generation with text_encoder_quant=fp8 crashed at encode_prompt (regression from the HiDream TE4 change; caught by the krea hosted-TE live smoke). The override is now a property shadowed on the ORIGINAL class that prefers a per-instance compute-dtype attribute, so class identity is preserved and uncast instances keep the stock behaviour. The idempotency test now pins exact class identity and the uncast-sibling fallback. * Pass the calibrated distilled sigma curve to LTX-2.3 8-step runs The 22B distilled DiT was trained against ltx_core's fixed DISTILLED_SIGMA_VALUES, but the diffusers scheduler derives 8-step spacing from resolution-shifted flow matching and lands far off at every reachable mu (second sigma 0.945-0.981 vs 0.99375, tail 0.37-0.61 -> 0.1 vs 0.725 -> 0.42 -> 0). At the distilled default step count the backend now passes the list verbatim, neutralising the scheduler's dynamic shift and terminal stretch for the call (they distort even explicit sigmas) and restoring them afterwards. Other step counts and the dev/base DiT keep the scheduler's own spacing. Live-verified on B200 through the video branch backend: the scheduler holds the exact curve after an 8-step distilled GGUF generation, config restored, healthy clip. Also reword the transformer_quant resolved reason to the measured reality: quant halves resident weights and hosted checkpoints cut load time, while per-step speed is roughly bf16 parity. * Pin the fp8 weight-quantize kernel against silent MSLK switching torchao's Float8Tensor KernelPreference defaults to AUTO, which switches the weight-quantize kernel to MSLK whenever an mslk package is importable on sm90+. Measured on B200: that changes fp8 scale rounding bitwise (8/8 FLUX matrices differ, scales ~55 percent of bytes), so a box that merely gains mslk would break the hosted-prequant bit-identity invariant; the mslk path is also slower under torch.compile (opaque extern call blocks inductor's quantize fusion, FLUX.1 fp8 e2e 1.149 to 1.624 s). Pin KernelPreference.TORCH explicitly, matching current no-mslk behaviour bit for bit; signature-gated for older torchao. GPU-smoked (finite, rel err 0.037) and pinned by test. * Shift Qwen-Image training sigmas to the inference distribution Qwen-Image's scheduler skips its static shift under use_dynamic_shifting, so the DiT trainer was drawing UNSHIFTED uniform-schedule sigmas for it (mean sigma 0.50) while inference always runs the exponential mu = log 3 shift plus the shift_terminal 0.02 stretch. Add a flow_shift config lever: "auto" (the new qwen-image default) rebuilds the training sigma table through the scheduler's own time_shift and stretch_shift_to_terminal so the draw matches the inference distribution exactly (mean sigma 0.72); a numeric value applies the standard linear shift s*u/(1+(s-1)*u); 1.0 keeps the historical identity behavior and stays the default for FLUX, Z-Image and Krea 2. The model timestep conditioning follows the shifted sigma, gathered in fp32 so bf16 rounding never skews it. Also wire two opt-in levers with off defaults: cfg_dropout (per-sample empty-prompt conditioning dropout, encoded alongside the captions before the text encoders are freed) and weighting_scheme="bell" (bsmntw-style mid-schedule Gaussian loss weighting normalized to mean 1). Verified with two 80-step rank-8 bf16 LoRA runs on Qwen/Qwen-Image (identity vs auto, same seed): both converge with finite decreasing loss and produce coherent same-seed previews. Unit tests cover the exact transform, the shifted sampling distribution, per-family defaults and config plumbing. * Add LoRA EMA, a persistent conditioning cache, and aspect bucketing helpers diffusion_train_extras hosts the opt-in training extras: LoRAEMA shadows only the trainable adapter params (warmup-ramped decay, default 0.99, exported as a second adapter under output_dir/ema), PersistentConditioningCache stores latent posterior stats and caption embeddings as safetensors keyed by content hash + family + resolution, and the aspect-ratio bucketing helpers group mixed-aspect datasets into same-area divisor-snapped shapes. The DiT trainer wires the first two behind config flags that default to the current behavior: ema_decay (0 disables) and cond_cache_dir (None disables). A fully warm cache skips loading the VAE and text encoders entirely; a cache hit is bit-identical to a fresh encode, including the per-channel qwen latent normalization. Also fixes the stale _gather_sigmas call in the perf test that still passed the scheduler instead of the sigma table. * Tighten torchao configs and note the FSDP2 design for the DiT trainer nf4 loads now enable double quantization (~0.4 bits/param off the frozen base scales at no fidelity cost), fp8 training uses the rowwise recipe when the torchao build ships it (per-row scaling confines the DiT activation outliers that a tensor-wide scale collapses), and the inference quant filter gains a per-scheme GEMM-tiling divisibility floor (16 for scaled_mm, 32 for MX blocks) so one ragged Linear cannot crash the first denoise after a clean quantize pass. plans/fsdp2_diffusion_design.md records the multi-GPU design: bf16/fp8 over FSDP2 with per-block units, LoRA attached before sharding, int8 out of scope (DTensor over the quantized subclass is undefined), per-family notes. * Batch diffusion inference with per-image seeds, an inference conditioning cache, and GGUF loader fixes Batched generation: /images/generate takes a prompts list (one image per prompt, txt2img only) or a seeds list (one prompt, one image per seed); the legacy batch_size path derives per-image seeds base..base+n-1 like the native engine. Every image gets its own torch.Generator so any batch member replays alone from its gallery recipe; the whole list runs as one forward by default with OOM backoff that halves a failed chunk, and an explicit batch_size caps images per forward. Validated 10-22x over serial engines on 32-image suites with LPIPS deltas within 0.002. Conditioning cache on the inference path: UNSLOTH_DIFFUSION_COND_CACHE_DIR (the inference sibling of the trainers' cond_cache_dir, same persistent store) wraps encode_prompt so repeated prompts skip the text-encoder forward entirely; verified bit-identical outputs. Bypassed while LoRA adapters are attached; tensor-argument calls pass through uncached. Compile cache: GGUF loads fingerprint their own bundles (quant=gguf, a different compiled graph than the dense family) and batched calls register every distinct (w, h, batch) chunk shape they ran, so the heavy GGUF batched warmups (~159 s at batch 32 on 12B-class, ~655 s on 20B CFG-batched) are paid once ever. GGUF loader: strip the sd.cpp model.diffusion_model. container prefix in the single-file converter; diffusers' FLUX.2 converter KeyErrors on it and the Qwen-Image identity mapping strands the model on meta. * Correct batched seed-replay docs to match measured behavior Same-seed images at the same batch shape are bit-identical; a solo regeneration with the recorded seed matches its batched rendition up to batch-size-dependent kernel numerics (mean abs pixel delta about 2.5/255, LPIPS delta under 0.002), not bit-exactly. The previous wording overclaimed bit-identity across batch shapes. * Note that batched bit-identity assumes a settled compiled graph The first generation issued while the deferred compile is still in flight can deviate transiently (observed once on a cold fp8 build: mean abs pixel delta 0.063/255); once the graph is settled, same-seed same-batch-shape images are bit-identical across runs. * Studio sidebar: Image03/FlimSlate icons, More flyout, Train row, New pills - Images uses Image03Icon and Video uses FlimSlateIcon. - New "More" row (MoreHorizontalIcon) opens a right-side flyout on click or hover holding Video, Recipes and Export; the close is delayed 180ms so the pointer can cross the gap. Its SidebarMenuButton deliberately takes `title` rather than `tooltip`: with `tooltip` the button returns a Tooltip root and DropdownMenuTrigger asChild would hand its ref to a non-DOM node. - Dropped the "Train" section heading; Train is now a top-level row between Images and More. data-tour="navbar" moves to the surviving nav group so the product tour keeps its anchor. - "New" pill beside Images and (inside the flyout) Video, via NavBadge. * Studio sidebar: match flyout rows and New pills to the existing scales - More flyout rows dropped their sidebar-row typography and size-icon override, which fought DropdownMenuItem's own scale (text-sm, gap-2.5, px-3 py-2 and size-4 icons) and rendered oversized glyphs and text next to the nav. - New pill reuses the brand "beta" badge recipe (nav-badge font, --ui-font-scale sizing, nav token colours) rather than hardcoded 9px values. - The More row's native title tooltip (an OS box on hover) is replaced by the app's Tooltip, wrapped around DropdownMenuTrigger so both triggers compose onto the same button, and shown only on the collapsed rail like other nav rows. * Settings: pin and reorder the sidebar navigation Adds a "Sidebar navigation" section to Settings -> Appearance, above the existing profile-menu customizer, with the same drag-to-reorder + switch UI. - New sidebarNav preference: one { id, pinned } entry per navigable row (projects, hub, images, train, video, recipes, export), array order = render order. Defaults match the shipped layout, so an untouched install is unchanged. - Unpinning moves a row into the More flyout rather than hiding it, so no page becomes unreachable. New chat and Search stay fixed as actions. - app-sidebar now renders from one navRows descriptor map, so a pinned row and its flyout counterpart cannot drift; the More row appears only when something is unpinned and highlights off whatever it actually holds. - Mirrored in the backend PersonalizationCustomization: without it the model's extra="ignore" would drop the field, and because sync replaces local state with the server's copy once customization is saved, the user's pin order would reset on the next sync. The validator dedupes and back-fills like sidebarMenu but preserves the client's order, since here order is meaningful. Frontend typecheck, i18n parity and catalog checks pass; 32 personalization tests pass, including a round-trip asserting a reordered list survives a save. * Sidebar customizer: drop the Search row, skip More for a lone item - Search is reached from the top bar, so it is no longer previewed as a fixed sidebar nav row; New chat stays. - More now appears only when it would hold two or more rows. A single unpinned row renders inline in its saved order position instead: a flyout wrapping one item costs a click and earns nothing. The customizer's More preview follows the same threshold. * Sidebar settings: hide a lone unpinned tab, match New chat icon, rename Profile menu - With exactly one tab unpinned, both More and that tab are dropped, so nothing is drawn for it (previously it rendered inline). The page stays reachable by URL. - The customizer's New chat preview uses PencilEdit02Icon, the icon the real row renders; Edit03Icon was a different glyph. - "Sidebar menu" is now "Profile menu", described as the shortcuts behind your name at the bottom of the sidebar, so it no longer reads as a second name for the navigation section above it. * Tighten comments in the new sidebar and delete-guard code * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio sidebar: keep the More row highlighted while its panel is open Moving the pointer into the flyout left the row unhighlighted while the panel stayed open. The row now carries data-menu-open, added to the nav hover selector list. Not data-state: the tooltip and menu triggers both write that attribute, so whichever lands last wins. * Images: use the shared pill toggle for Create/Train and pad the panels - Create/Train was the only segmented control on its own Tabs styling. It now uses PillTabs, the same control as the model picker and Hub toggles, pinned to the header row's 34px. PillTabs takes an icon per tab, so the inline-span workaround for TabsTrigger goes away. - pt-3 on both the Create and Train panels, which sat flush against the model selector row. * Images: make the workflow picker a dropdown instead of a 7-up strip Seven workflows in a 340px rail left ~48px each, so the labels crowded and the hints were only reachable as title tooltips. The strip is now a dropdown: the trigger shows the current workflow and its hint, and each row carries its own description. A row the loaded model can't run is disabled and shows the reason in place of the hint, so the gating explains itself. Adding a workflow no longer shrinks the others. * Images: workflow icons, hint under the trigger, more top room, unclipped Train cards - Each workflow carries an icon, shown on the closed trigger and on every row. - The trigger is one line (icon plus name). The selected workflow's description moved below it, where it reads like the Field hints further down the rail. - pt-6 instead of pt-3 on both Create and Train, so the cards clear the model selector row. - The Train right column scrolls while its cards use ring-1, which draws outside the box and was clipped at the scroll edges. p-px gives the ring room. * Images: one-line workflow rows, stronger trigger fill, roomier mode tabs - Dropdown rows are icon plus name only. The selected row's description already shows under the trigger, and a disabled row keeps its reason as a title. - Trigger fill moves to the bg-foreground/[0.07] dark:bg-foreground/[0.12] pair the hub cards use, so it reads against the card in both modes. - Description under the trigger goes from text-ui-10 to text-ui-11p5. - More horizontal padding on the Create / Train tabs. * Images: drop card borders for the composer shadow, keep scrollbars inside, use app controls in Train - Cards lose ring-1 for .panel-soft-surface: the composer's shadow in light, flat in dark, matching .chat-composer-surface and the menus. - Both rails now clip (overflow-hidden) with the scroller inside, so the scrollbar can't ride over the rounded corner. Same shape video-page already uses. - Train's 9 native selects become the app Select, so they no longer open an OS-native menu, and the native file input is hidden behind a Choose images button that reports the count. - Image previews use explicit 8-10px radii: this theme sets --radius to 1.1rem, so rounded-md was 15.6px and the thumbnails read as circles. * Images: one card for controls and preview, chat sliders, wider softer shadow - Controls and preview were two floating cards; they now share one card split by a divider. The Advanced dock stays separate since it toggles. - SliderField wraps Chat's ParamSlider, so the sliders match Chat (label row with the value, full-width neutral track) instead of a green track with a spin box. All 14 call sites keep their props. - panel-soft-surface goes from 0 2px 8px -2px /0.16 to 0 4px 22px -6px /0.10: lighter, spread wider. * Images: flat Create and Train panes, hover-only scrollbars, tidier Train dataset step Both Images tabs now sit on the page background like the Hub: no card, no shadow, no bounding box. A single rule divides the controls rail from the preview canvas (Create) and from the run area (Train), and the settings and previous-runs sections read as panes rather than nested cards. Also: - Scrollbars in these panes use the existing hover-scrollbar recipe, so the thumb only shows while the pane is hovered. - Workflow rows explain themselves with a tooltip after a short hover, which also works on disabled rows, and the descriptions are much shorter. - Training images rows are name plus image count; the license stays on the example card. - The upload step loses its dashed box, the buttons match the sizes around them, and Upload only appears once files are picked. - The empty preview uses the same icon as the Images nav item. * Images: full-height panes, wider settings rail, Create/Train offset from the selector The rule between the panes now runs the whole page height (the row drops its bottom padding and each pane pads its own content), the settings rail is wider on both Create and Train, and the Create/Train switch sits further right of the model selector. * Images: put both tabs on the Hub's centered measure Top bar and content now share mx-auto max-w-1100 with px-5 / sm:px-8, so Create and Train sit at the same width and position as the Hub instead of running edge to edge. * Images: restore the top bar position, drop the panes lower under it * Images: center the mode switch, flip the arrow with the orientation, app tooltips everywhere The Create/Train switch is centered on the page instead of trailing the model selector, with wider buttons. The flip control's arrows now rotate with the orientation and its label says which way the flip goes. Every native title tooltip on the page is now the app's tooltip, so they all get the rounded surface instead of the OS box. * Images Train: plainer field text, no green buttons, columns that stop colliding - The dataset name, trigger prompt, adapter name and custom base fields now say what they are in plain words instead of leaning on example values. - Import, Upload, Back, Back to settings and Train another are outline buttons, not green ones. - Example thumbnails are landscape tiles, so photos are not cropped to chunky squares. - Settings cells get min-w-0 and the select value truncates, so a long option like the nf4 label no longer widens its column into the next one. - The number stepper sits a little further in from the field edge. - Create and Train are wider. * Images Train: roomier example cards with Import on the thumbnail row * Video: same treatment as the Images tabs - No cards: the rail and the canvas sit on the page background, divided by a rule that runs the full page height, on the Hub's centered measure. - Wider rail, chat's sliders, hover-only scrollbars. - Every native title tooltip is now the app's tooltip, including the clip cards. - Reapply and Cancel are outline buttons, the empty state uses the Video nav icon, and the clip tiles are less rounded. * Images and Video: narrower generation rail, matching Train headings Create and Video rails go from 392px to 368px. Train a LoRA and Training settings are now the same size and both in the heading font: the h2 already picks it up from the base rule, so the settings header opts in with font-heading and the weight that rule pins. * Images Train: shorter copy throughout Family notes, example descriptions, precision labels and every helper line are trimmed so they stop wrapping to three lines and colliding with the next column. The nf4 label now fits its select without truncating. * Images Train: a little more spacing between field groups * Images and Video: tighten code comments * Fix training start NameError, the load-order guard test and CPU-only diffusion tests - start_training forwards resume_source_run_id to _start_training_impl, which reads it. Without it every start raised NameError. - Restore main's anchor in the load-marker order test: the file now has an earlier `if config.is_gguf:`, so indexing the first one compared the wrong branch. - The two diffusion tests that reach diffusers now skip when it is absent, matching the CPU repo-test env. - The UI smoke finds nav rows that live in the sidebar's More flyout. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Treat a null metadata caption as no caption str(None) stored the literal "None" as the caption, so a null row counted as captioned and would have trained on that text. Also drops an unused import. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix invalid-UTF-8 500s, the flat Canny map and the dropped DiT knobs read_text raises UnicodeDecodeError, which is not an OSError, so one bad caption sidecar or video sidecar 500d the info, upload and gallery routes. A flat image now yields the all-black edge map instead of its own luminance, and the four DiT loss knobs the trainer implements are declared so model_dump keeps them. * Use the ui font-size tokens instead of raw px text utilities text-[11px] and friends ignore the UI font size preference, which the repo's font-scale contract test enforces. Same rendered size at the default scale. * Fix diffusion dataset 500s, the dropout-1.0 no-op run and the reset base pick Four correctness fixes on the training side: - The labeling grid read caption sidecars under except OSError, but a non-UTF-8 sidecar raises UnicodeDecodeError (a ValueError), so one bad file 500d /diffusion/dataset/{name}/images and the grid could not be opened to repair it. Read it as no caption, matching the info summary. - An image past Pillow's own hard limit raises DecompressionBombError, which derives straight from Exception and so escaped the upload guard's (OSError, UnidentifiedImageError, ValueError) and returned 500 instead of the intended 400. - lora_dropout accepted 1.0, which makes PEFT build nn.Dropout(p=1.0): lora_A and lora_B receive no gradient and the run saves an untrained adapter while reporting normal progress. Bound it below 1.0, matching the LLM request schema. - The train panel re-seeded the base repo on every dataset refresh because the family object identity changes on each info fetch, so an upload or caption save silently replaced the user's chosen base and the run started on a different model. Track the pick and only re-seed on a real family change. * Show the retained failure when a video page mounts after a failed job Mount-time recovery handled only phase=completed, so reloading the page after a multi-minute generation failed left an idle view with no diagnosis: the backend keeps the terminal failed record only until the next job, and nothing else survives the reload. Surface it the same way the poll does, filtering the cancelled sentinel. * Fix batched generation crashes, cache keying and unreplayable recipes Four bugs in the batched inference path, all found by review: - A mixed-prompt batch sent a scalar negative prompt against a prompt list. Z-Image asserts on the length, and Qwen-Image, Krea 2 and FLUX true-CFG encode a batch-1 negative against batch-N latents and fail in the transformer's text/image concat. Broadcast it to match the batch. - The FBCache step-cache reset sat above the chunk loop. diffusers only resets that state at the end of a successful call, so a forward that raised (the OOM the backoff is meant to recover) left its own residual behind and the halved retry died on a shape mismatch. Reset before every forward instead. - The conditioning cache keyed on the checkpoint alone, but a GGUF or single-file load takes its text encoders from the companion base, so the same checkpoint against a different base reused the previous base's embeddings. Key the base too. - Gallery records stored the base seed and the requested batch size even when a prompts/seeds list drove the run, so restoring the second image of seeds=[5, 99] replayed seed 5. List-driven outputs now record as single-image recipes on their own seed. Also bound strength above 0: every img2img pipeline derives its step count from it, so 0 leaves zero denoising steps and either raises or, on SDXL, crashes on empty latents. * Fix quantized-load LoRA bake, prequant family exclusions and outpaint canvas Six review findings across the Images page and model scanning: - The quantized (int8/fp8) load path can only attach LoRA adapters before quantization, but the frontend load request had no loras field, so every generation after such a load was rejected and each reload repeated it. Send the selection with the load. - build_prequant_checkpoint passed no family to the scheme exclusions while recording the family in metadata, so a Qwen int8 artifact baked the short-M text-stream linears and was then rejected wholesale by the loader's family-keyed check. - Registering a bare single-file checkpoint directory produced no On Device row even though the images loader can load it; only its parent worked. Admit that shape when nothing else matched. - Unload left the Reapply target set, so the repair path was skipped and Reapply reloaded the ejected model. Clear it, as the video page does. - Both FLUX.2 bases were trusted for training but not inference, so Deploy to Create rejected every FLUX.2 adapter. - Outpaint allocated the grown canvas before downscaling, exceeding the browser canvas area cap on a large photo; an over-cap canvas is unusable, so Extend silently posted a fully transparent image and mask. Scale the source first. * Send the picked GGUF filename with the quant so diffusion loads fire The variant expander emitted only the quant label, and nothing else in the frontend set ggufFilename, so the Images and Video pages could never take their GGUF branch: both gate it on meta.ggufVariant and meta.ggufFilename, then fall through to the single-file path, which returns because the id is a repo id and not a .gguf name. Every quant pick was a silent dead click, with no load request reaching the backend. The filename was already on the variant row (the picker keys its list on it, and the variant validator requires a non-empty string), so thread it through the click handler. The chat path is unaffected: it reads ggufVariant and never needed the filename. * Version the conditioning cache key and reject non-finite flow_shift Two correctness fixes: - The cache keyed the checkpoint and its companion base by name only, so a Hub repo advancing to a new commit, or a local directory updated in place, kept returning embeddings from the previous text encoder. Pair both with a revision marker: the locally resolved commit sha for a Hub repo, config plus text-encoder file stats for a directory. Neither loads the encoders, so a warm run still keeps them off the GPU. - flow_shift only checked positivity, but JSON accepts 1e309, which floats to inf, and inf <= 0 is False while NaN fails every comparison. The sigma table then evaluates s * u / (1 + (s - 1) * u) as NaN, which poisons every sampled sigma and saves a corrupted adapter while progress looks normal. Require a finite value. * Keep curated models listed, guard the video companion repo, pin diffusers Three review findings: - The picker filtered every catalog member out of Recommended and Hub search on the way to canonical group rows, but nothing renders those rows yet (catalogGroupFitsDevice and groupMatchesQuery are imported and unused). A task-scoped picker's models list is catalogToModelOptions(), i.e. group members exclusively, so both lists came back empty and no curated model could be discovered or downloaded. Keep the artifacts listed until the grouped UI exists. - The video delete guard compared only repo_id, so deleting the companion base of a loaded GGUF video model was allowed even though it supplies the VAE and text encoders. Compare base_repo too, matching what the images guard already does for its companions. - diffusers was declared unversioned while the diffusion stack requires 0.39 (Krea2Pipeline, the cache_context child registries, the Flux2 and Z-Image pipelines), so an upgrade could keep an older release and selecting an advertised model failed until the user upgraded by hand. * Namespace the trainer conditioning cache per checkpoint, bound the learning rate - The trainer keyed its persistent conditioning cache on family and resolution only, while the keys themselves carry just the caption or image content and crop variant. One cache directory reused for two checkpoints, or for the same repo at a new revision, let a warm run skip loading its encoders and train on the other model's embeddings and latent statistics. Namespace on the base checkpoint and its resolved revision as well. The revision helper now lives beside the cache in diffusion_train_extras and the inference wrapper delegates to it, so the two cannot disagree about what counts as the same source. - The diffusion learning rate only checked positivity, but 1e309 floats to inf and satisfies gt, so the route evicted the resident models and started AdamW with an infinite rate: the first step destroys the adapter while progress looks normal and the result is saved. Bound it below 1.0, matching the LLM schema, which rejects inf for the same reason. * Fix GGUF image model picks doing nothing, and pick the train base in the top bar The quant rows never forwarded the .gguf filename, so every hub GGUF pick on Images/Video fell through to a silent return. On Train the top bar now picks the training base instead of a generation model, which is GGUF-only and untrainable. * Pin diffusion and video loads to the live HF cache root Both read huggingface_hub's import-time HF_HUB_CACHE, which changing the cache folder does not update: progress counted the old root while the download wrote to the new one, and from_pretrained could split one model across both. * Add the diffusion download plan endpoint Reports the repos and exact files a pick needs so the download manager can stage them with the loader's own file scope. A plain snapshot would add the packaged root single, transformer shards and fp16 twins the loader never opens. * Add a file-scoped flavour to the Hub download job Lets a consumer that reads a deliberate subset of a repo stage it through the normal download manager. Keyed as "@scope" so it never collides with a quant or with the repo's full snapshot, and the file list rides the registry so an XET to HTTP retry respawns the same scoped job. * Stage image and video downloads through the Hub download manager They downloaded inline inside the load, so they had none of the manager's disk preflight, manifest verification, resume or panel progress. Picks now stage as scoped jobs carrying the loader's own file list, then load from a warm cache. * Fetch staged GGUF checkpoints as scoped jobs, and stop calling diffusion models unsupported A GGUF entry went out as a full snapshot, whose ignore list drops *.gguf: the job finished at once having fetched only docs, and the repo landed on device unloadable. Every entry is scoped now. The Hub also no longer tags image/video models as unsupported (they run on their own pages), and those pickers name what they select. * Apply the picker task filter to local model sections LM Studio, ./models and custom-folder rows ignored it, so the Images picker listed chat GGUFs that 400 on a diffusion load. The backend already tags every local model with a task for this purpose. * Route a chat pick of a diffusion model to the Images or Video page Chat cannot load one, so it was either hidden or failed on load. The unfiltered picker now lists on-device diffusion models and navigates to the page that runs them, passing the repo and quant so that page loads it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Route the real GGUF filename, keep non-GGUF curated models, key scoped downloads by file set Five review findings, four of them ways a click did nothing or fetched the wrong thing: - A chat pick of a diffusion model routed ggufVariant (a label like Q4_K_M) in the search param the target page uses verbatim as the GGUF filename, so the load asked for a file that does not exist. Route ggufFilename; no filename means a curated non-GGUF pick, loaded as a pipeline. - The task-scoped pickers kept only GGUF repos, so the catalog's bf16, bnb-4bit and single-file fp8 artifacts could not be discovered or downloaded on the Images and Video pages even though loadSpecFor knows how to load them. Keep curated artifacts whatever their format, in Recommended and in Hub search. - Both pages deduplicated routed selections on the model alone, and they now stay mounted, so picking the same repo again -- another quant, or the same one after chat evicted it -- returned early without loading or clearing the query string. Key on model and quant. - Every scoped image download shared one @diffusion job key regardless of the requested files, so switching quant mid-download adopted the running job: the UI waited on the first file set, then loaded a file that was never fetched. Include a digest of the file set in the key. - A scoped plan silently dropped requested files missing from Hub metadata, and snapshot_download succeeds when an allow pattern matches nothing, so the job reported completion and triggered a load with required files absent. Fail the job instead. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the scoped download key derivable, and stop the hidden page hijacking a route Four review findings, the first a regression from my own last commit: - Keying scoped download jobs by a digest of the file set broke the download manager: it builds that key client-side (it polls and cancels before any response tells it a key), so it watched and cancelled a key no worker owned and never fired its ready callback. Keep the derivable "@scope" key and refuse the second request instead when a live job on the slot is fetching a different file set -- decided inside the registry claim, under the lock, so a concurrent claim cannot slip past it. The manager records the file set on the job as well, so a sibling quant's transfer is not adopted locally either. - Both diffusion pages read the route query through a loose useSearch and both stay mounted once visited, so the hidden one consumed the other's ?model=: it navigated back to its own route and tried to load, say, an image checkpoint as a video model. Only the visible page consumes it. - The staged download plan was built without the configured HF token or the Advanced values the load itself sends. The token matters most: the backend's Hub metadata lookup is best-effort, so a gated base silently planned no companion entry and the load pulled those multi-GB files inline, outside the manager. The memory/quant controls decide whether the base transformer/ shards are needed at all, and the route dropped memory_mode, cpu_offload, the prequant path and the LoRA selection before asking for the plan. - The video preview kept playing after leaving the page: the keep-alive layout only hides it, and display:none does not pause a media element, so a clip the user unmuted kept its audio going over the next page. Pause on the active transition and do not auto-replay while hidden. Also completes the hand-built request bodies in the hub download tests: the scoped-files field this branch added to the route read as an AttributeError against them, failing five tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Serialize the GPU handoffs, gate DiT training on a GPU, and keep 3.9 installable Six review findings, three of them evict-then-fail orderings: - The chat load reclaimed the GPU without telling the arbiter it existed. A chat load holds no llama-server process until its GGUF has downloaded, which is minutes, so a competing Images/Video acquire in that window found nothing to cancel, took the GPU, and the chat load then spawned onto the same device. It now registers an in-flight marker through acquire_for's register hook (under the arbiter lock, as the image and video loads do), the evictor cancels a marked load, and the route undoes itself if ownership moved while it loaded. - The Hub-download conflict check ran after that handoff, so a GGUF the download manager already owns destroyed the resident Images/Video pipeline and then 409'd, having loaded nothing. It moves above the handoff, together with the marker it handshakes with. - The image load released the engine router's transition lock before registering the load, so a second load choosing the other engine could unload the still-idle engine this one captured; the load then landed on a deactivated engine, where generate, status, unload and the arbiter's evictor can no longer reach it. Registration now happens under that lock and refuses if the engine changed. - Training a DiT family on a host with no GPU was accepted: nf4 is not a CPU fallback, its 4-bit load goes through bitsandbytes, which requires CUDA, XPU or MPS. The start unloaded the working Images pipeline, pulled the text encoders, and only then died in the child. Rejected before the teardown now, and /info stops advertising a precision that always 400s. SDXL keeps its documented fp32-on-CPU path. - Both diffusion pages kept the routed-pick marker forever, so re-picking the same checkpoint (after chat evicted it) neither loaded nor cleared the query string. The marker is released once the query is gone. The Images key also carried a stray NUL byte, which made the file read as binary to grep and other tooling. - diffusers dropped Python 3.9 in 0.38, so the unconditional >=0.39.0 pin left pip no candidate at all on 3.9 and made every install that composes the huggingface extras unresolvable there. The floor is conditional now. Also fixes tests that were already red on the branch: two hand-built request fakes had gone stale against fields this branch added, and the handoff-ordering test only failed on a host with fewer than two GPUs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the GGUF variant contract test against the merged handler signature The assertion pinned the exact single-line call handleVariantClick(v.quant, v.downloaded, expectedBytes, v.filename), but the handler takes (quant, filename, downloaded, sizeBytes) and prettier wraps the call across lines, so the mandatory repository test job failed on every push. Match the call structurally and assert the filename really is forwarded in the handler's argument order. * Stop a background page and a stale record taking the GPU or a download with them Five fixes from a review pass over the diffusion work. delete-finetuned rmtree'd a model the Images or Video engine was holding: every guard on that route is chat-only, and Images loads any local path, so deleting a local diffusion model under the storage root pulled the weights (and the companion VAE / text encoders sd.cpp re-reads each generation) out from under a live pipeline. The cached-model route already refuses this; the trained/exported one now does too, matching by path rather than repo id, and failing open on a chat-only install so it cannot block ordinary deletes. A staged download finishing while its page was hidden loaded the model and evicted whatever the user was actually using: both diffusion pages stay mounted behind the router and a load takes the GPU unconditionally. The pick is now held until its page is on screen again, which is also what chat does. A scoped download could report success having fetched nothing. With Hugging Face metadata unavailable no manifest is written, so verification is a no-op, and snapshot_download returns an existing snapshot folder without downloading when its own repo_info call fails. A repo already on disk from a full snapshot job (which ignores *.gguf) therefore completed with no weights and auto-loaded against them. The requested file list needs no network, so it is checked against the disk directly. The XET to HTTP retry reclaimed the job slot without the scoped file list, and that claim overwrites the stored record, so a later identical scoped start compared an empty list against the real one and 409'd instead of adopting the running download. The DiT accelerator gate probed torch.mps.is_available(), which only exists from torch 2.5 while the supported floor is 2.4. All three probes shared one try/except, so on torch 2.4 the AttributeError read as 'no block' and a CPU-only host still evicted the resident pipeline, downloaded the encoders and died in the child. Each accelerator is probed on its own now, through torch.backends.mps. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stage what an LTX-2.3 load reads, keep a routed file's load kind, drop an unbakeable LoRA Three from the latest review. The video download plan always asked for the wide base file list, so an LTX-2.3 pick staged the 2.0 base's VAEs, vocoder and connectors that the checkpoint supplies itself, while the companion files the 2.3 assembly does read were left out of the plan and pulled inline at load, outside the panel's progress, cancel and disk preflight. The plan now recognises a 2.3 pick by name (the load keeps the authoritative header probe, and under-guessing only falls back to the load-time pull), narrows the base list, and stages the extras in the same entry as the checkpoint so one repo stays one scoped job. A pick routed from the chat picker arrives as ?model= and ?quant= with no picker metadata, so a bare local .gguf or .safetensors was loaded as a pipeline: an explicit model_kind wins over the backend's filename sniffing, so it evicted the resident model and then failed on the missing model_index.json. Both pages now derive the load kind from the path, the same way their own picker handlers do. A torchao int8/fp8 build takes adapters only at load time. Switching artifact inside one family keeps the LoRA selection, since the family did not change, but the load did not bake it, so the next generation was rejected with 'reload the model with the adapter selection' while the picker still showed the adapter as active. The selection is now dropped once per resident build, with a message saying to pick and load again. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cancel an evicted safetensors load, spare the arbiter for CPU-only chat, fetch clips lazily Four fixes from the latest review round: - The GPU arbiter's chat evictor only cancelled the llama.cpp side. The orchestrator publishes active_model_name once its worker reports success, so an in-flight safetensors load was visible only as an entry in loading_models and finished onto the GPU after ownership had transferred. Cancel every pending load, and give the safetensors branch the post-load ownership recheck the GGUF branch already had. - A manual gpu_layers=0 GGUF load runs on the CPU with the GPUs hidden from the child, yet it took the arbiter unconditionally: it cancelled a running image or video generation for a model needing no VRAM, then held CHAT ownership so the next GPU workload unloaded it for nothing. Gate the acquire on the same predicate the launch-time CPU-only mask uses, as the image and video loaders gate on their resolved device. - The staged-download hook subscribes per repo, not per job, so another job on the same repo advanced the staged queue (starting a load whose scoped files were still downloading) or wiped a queue that was still running. Compare the variant each callback carries, like the chat page's auto-load does. - The video gallery fetched every record of a page into an object URL that lives until the page closes: 50 clips at tens to hundreds of MB each, for cards the user may never scroll to. Fetch a clip as its card nears the strip's edge, plus the selected one the player needs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim the comments across the diffusion backend Comment-only pass over the Python this PR touches: drop what the code already says, collapse multi-line explanations that still read on one line, and keep the reasoning that is not recoverable from the code. No code, docstring semantics or behaviour changes; verified with an AST comparison against the previous revision, and the backend suite is unchanged (same 37 environment failures as before: the API integration tests that need a live keyed server, the flash-attn install hooks, and the GPU memory fields). * Invalidate latents on a VAE swap, keep a cut-off generation, surface the EMA adapter - source_revision() scanned the checkpoint root plus text_encoder/tokenizer but not vae, so swapping or fine-tuning the VAE in place left the conditioning cache namespace unchanged and a warm run trained against latents from the old checkpoint. Include the vae directory, like any other component the cached tensors come from. - /images/generate answers only when the images are saved, and secure mode's tunnel caps an origin response near 100 seconds, which a native CPU or a high-step run passes routinely. The page reported failure while the work kept running, and a retry would duplicate it. A lost response (fetch rejection or a gateway status the origin never answered) is now told apart from a refusal: the page waits out generate-progress and reloads the gallery, so the run it started still lands. - The trainer emits the EMA adapter's path with the terminal event, but the state update dropped it, so neither the run history nor either response schema carried it and an enabled EMA left nothing discoverable. Keep it, and show it next to the primary adapter. - weighting_scheme advertised a choice of timestep sampling; sampling is always logit-normal and the flag only selects the bell loss weights. Describe what it does. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Hide unloadable cached rows, hold the dataset interlock, bound a GIF export - The cached-model listing tagged any repo with a model_index.json as text-to-image, so a community pipeline the image loader's trust rule refuses still got a row in the Images picker, and a detected-but-untrusted video repo fell through to that same tag. Gate the image tag on the load path's rule and hide an untrusted video repo outright. - A routed diffusion pick only carries a GGUF filename, which is all the chat picker has, so a curated single-file artifact arrived with no quant and was loaded as a pipeline: from_pretrained on a repo with no model_index.json. Pass the page's own catalog spec into the route pick, so a routed pick resolves to exactly what a direct pick on that page resolves to. - The dataset mutation endpoints checked is_active() and only then handed their filesystem work to a thread, so a start reserving in that gap changed captions or removed images underneath the preflight or the running trainer. The interlock is now registered for the whole request under the lock reserve() uses, and a start refuses while a mutation is open rather than waiting on it. - GIF export held every kept frame as a paletted image before encoding; a clip may be 2048x2048 for 1024 frames, and at the 12 fps target the step is 1, so one export click could allocate over 4 GB and take the backend down. Downscale past 720 px and widen the step to keep at most 300 frames. - seed accepted any Python int, so an out-of-range one passed every preflight, evicted the resident models, spawned the trainer and only then died in torch.manual_seed. Bound it to torch's 64-bit range in the request and config. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the accelerator probes in the DiT family-metadata tests Six tests read family_train_infos() (or a start preflight) without pinning the host probes, so they only held on a machine with a bf16 accelerator: on a GPU-less runner the DiT gate empties precision_modes, turns supports_compile off, and replaces any other preflight message with the no-accelerator note, and all six failed there. A conftest fixture pins both probes for exactly those tests, so they assert the family metadata they are about on every host. The gate's own CPU-only behaviour keeps its dedicated tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Carry the pipeline task into the hub inventory the pickers read - The task-scoped pickers filter On Device rows on a task, and the chat picker routes a diffusion pick by the same field, but those rows come from the /api/hub inventory, which never carried one: the Images and Video pickers listed nothing on device and the routing never fired. Both cached scans and the local listing now tag rows with the classifiers the models API already uses, the schemas and the frontend adapter carry it through, and a row the backend classified as a generation task is exempt from the chat-only guard that was also dropping it. - The local routing map was keyed by model_id while the row click passes id (a filesystem load id for a models_dir or LM Studio entry), so the lookup missed and the pick fell through to the chat loader. Key both. - A staged download whose start answered "error" left its head in place, where the effect never re-runs and onReady never fires, so the pick was stranded until the user reselected. Clear the queue and say so. - Every scoped pick in a repo shares the @diffusion variant, so the variant alone cannot tell two file sets apart: restaging while the first job finished let its completion pass for the new pick and load a checkpoint that had not downloaded. Bind the callbacks to the repo + file set they started, and to the staging generation. - A rejected generate POST does not say whether it reached the backend, so an immediately idle progress read was ambiguous and a submission that never landed looked like a finished image. Require evidence: progress seen active, or a gallery record that was not there before the POST. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the OpenAI image URL fetchable, keep WebM audio, stream example imports Four review items on the diffusion Studio work: - response_format=url returned the bearer-gated gallery route, which a standard image client downloads with no Authorization header, so the default response format was unusable. Mint a short-lived HMAC link instead (the shape RAG already uses for pdf.js) served by a signed route, and leave the gallery route itself bearer-only. - A manual gpu_layers=0 load carrying speculative_type="off" -- a value the UI persists and sends -- read as GPU-bearing, so it took the GPU arbiter and evicted a resident image/video pipeline even though the launcher hides the GPUs for it. Canonicalize the mode and exempt "off". - The curated example import prepared the whole split before the loop stopped at the 10-100 image cap; m1guelpf/nouns is 49,859 rows / 328 MB. Stream instead, with the prepared load kept as a fallback for a repo that cannot stream. - WebM export dropped the audio track an LTX-2 clip carries, silently, on the format offered for web embeds. Mux it as Opus through a resampler + FIFO, and keep exporting the video alone on a build without libopus. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not stub out triton on a GPU host when the Xet backend fails to import The lazy loader retries `import unsloth_zoo.hf_xet_fallback` under UNSLOTH_ZOO_DISABLE_GPU_INIT=1 whenever the first attempt raises. That flag makes unsloth_zoo take its MLX/CPU path, which injects triton and bitsandbytes STUBS into sys.modules for the rest of the process. On a working GPU box whose first import failed for an unrelated reason (a bitsandbytes/CUDA mismatch, say) the retry succeeds, so Studio boots looking healthy and then dies at the first CUDA-only kernel: a GGUF or compiled diffusion generation hits the stub and returns NotImplementedError: Unsloth: 'triton.tools.experimental_descriptor.enable_in_pytorch' was called on Apple Silicon / MLX, where triton is stubbed out. so every image generation 500s with an Apple-Silicon message on a Linux CUDA host, while the load reports success. Found by loading Z-Image-Turbo GGUF through the API on a box where bitsandbytes could not initialise. Gate the retry on the host genuinely having no accelerator. The Xet stall watchdog is optional and already degrades with a warning; a process whose triton is stubbed out is not recoverable. The warning now says why it did not retry. * Fix the lost-generation proof set, the settle timeout and the hub inventory's diffusion gates Seven fixes from the latest review round on the Images page and the hub cache inventory. Images page: - The lost-POST settle path built its "already seen" gallery id set inside the catch, after the request failed. By then the earlier runs of the same batch had already prepended their records, so run 2 could accept run 1's image as proof that its own request reached the backend. The set is now captured once before the first POST and grows with every record the batch produces. - settleLostGeneration fell out of its SETTLE_MAX_MS loop and returned normally, so a wedged generation was counted as done and the next run started against a busy backend. It now throws on timeout. - Restoring a recipe cleared the ControlNet selection but left the workflow tab and the init / mask / reference images pointing at whatever was loaded, so the next Generate conditioned on an unrelated image. It now clears all of them and returns to Create. - The download plan omitted the adapter selection the load itself bakes in. A baked LoRA forces the dense build path, so the plan described a different file set than the load that followed and the rest was pulled inline, outside the download manager. Both now derive the list from one helper. Hub cache inventory: - A download for a repo an Images or Video load is staging was allowed to start: only the llama.cpp loader was consulted. Both diffusion backends already expose loading_repo_ids for the delete guard, and the download guard now reads them too. - A companion-only prefetch (pipeline manifest plus VAE and text encoder, no transformer) passed the snapshot-partial check, since every file its manifest expected did arrive, and was advertised as on-device although from_pretrained cannot load it. - The single-file flag never reached the picker through the hub inventory path, so a checkpoint-only diffusion repo read as a full pipeline and failed after the handoff. The two pipeline-shape helpers now live in hub/utils/inventory_scan.py so /api/models/cached and the hub inventory classify the same repos the same way. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop adopting an unknown scoped download, leaking raced blobs and resurrecting deleted clips Three items from the latest review round. A scoped download job carries a deliberate file subset, and every file set of one repo rides the same "@scope" slot. A client that adopts a live job from the backend had no file list to compare against: the active-downloads response never carried one, so an adopted job's set was unknown and any later scoped request for the same repo read as "already started". Selecting a different checkpoint then waited on the wrong transfer and tried to load a file nobody fetched. The response now publishes the scoped file list, adoption records it, and an unknown set no longer satisfies a scoped request. A gallery record can be deleted while its blob is still downloading. The delete revokes the URL present at that moment, so the fetch that lands afterwards inserted a fresh object URL for a record no card renders and nothing can revoke: a full MP4, tens to hundreds of MB, pinned for the rest of the session, and once per raced fetch. Both galleries now discard a blob whose record went away, with an epoch covering the video page's Clear all. The video backend keeps the last completed job until the next one starts, and the Video page merges that record on mount to cover a job that finished after the gallery fetch. Deleting the clip left the record in place, so every reload prepended a ghost card whose file request 404s until another generation replaced it. Deleting the clip, or clearing the gallery, now clears the matching terminal record, and the page skips a record it deleted itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Resolve revisions from the live cache, serialize dataset imports, and stop pinning every gallery blob Five more items from the review round. The conditioning-cache revision marker read huggingface_hub's import-time HF_HUB_CACHE constant. Studio can move its cache during a session and loading follows the live setting, so after a move the marker went unresolved (or pointed into the previous root) and pulling a new revision of the same checkpoint no longer invalidated the cache: a warm run could reuse the old encoder's embeddings and the old VAE's latents. It now looks in the active Studio cache first and keeps the environment and the library constant as fallbacks, which the trainer subprocess still needs. The dataset interlock counts mutations rather than excluding them, so two imports of different examples into the same empty name both got past the emptiness check. The winner promoted its staging directory atomically; the loser found the folder non-empty, fell back to a per-file move, and merged its images and captions into the winner's dataset. Imports now take a per-folder lock, a second one is refused with 409, and the emptiness check is repeated under the lock. On Windows the sd.cpp asset resolver filtered only by accelerator token, so a Windows arm64 host matched an x64 zip, downloaded and installed it, and failed later when the binary would not run. It now filters by architecture the way the Darwin and Linux branches do. Every gallery page fetched every PNG up front and kept the object URL for the session, so scrolling a large gallery grew memory without bound for tiles the user may never look at. The Images strip now fetches a tile as it nears view, like the Video strip, and keeps the eager path only where IntersectionObserver is unavailable. A 503 carrying a JSON body comes from the application, not a proxy, so it is surfaced as the error it is instead of entering lost-response settlement and being reported as a request that never reached the server. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the class of a failed generation instead of a bare "Image generation failed." Found on a macOS runner: the native renderer aborts inside its own text encoder there, and the page showed only "Image generation failed." with the sd-server backtrace left in the server log, so nothing about the failure reached the user. The failure is now classified into fixed text, out of memory and native-process death, so the message says what happened and what to try. None of the engine's own output is echoed, since a native tail carries local paths and argv; that stays in the log, and an unrecognised failure keeps the original literal. * Treat an undecodable caption sidecar as the tombstone the trainer sees Uploads store .txt and .caption sidecars as raw bytes, so one can hold invalid UTF-8. The trainer treats any existing sidecar, decodable or not, as an empty tombstone and never falls back to the metadata row for that image. The labeling grid and the dataset summary read an undecodable sidecar as absent instead, so both showed a metadata caption that the run would silently replace with the instance prompt, and counted the image as captioned. Both now track sidecar presence separately, so what the user reviews is what the run trains on. * Keep the reason a native server died, not just its backtrace A ggml abort prints its cause first and then a stack trace, so reporting the last twenty captured lines gave twenty addresses and nothing about the failure: on the macOS runner the native server died on an unimplemented Metal op and the message carried only frame pointers. The captured tail now leads with the lines that name a cause and keeps recent context after them, for both the startup failure and the mid-request death. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop staging the dense text encoder for an fp8 video load Two halves of the same gap, found while measuring the LTX-2.3 download plan: - The video download plan and the scoped pre-download never saw text_encoder_quant. An fp8 request loads a hosted pre-cast encoder, so asking for one still staged and downloaded the base repo's dense Gemma3 (48.79 GB of Lightricks/LTX-2 on the 2.3 distilled pick) that the pipeline then never opened. The plan now drops those shards and stages the pre-cast checkpoint instead; their configs stay, since the pre-cast loader still meta-inits the encoder from the base repo's component config. - The LTX-2.3 assembly builds every component itself, so pipe_kwargs (which carries the pre-cast encoder for from_pretrained) never reached it and an fp8 request silently loaded the dense encoder anyway. It is passed across explicitly now. The dense skip is earned, not assumed: only a pre-cast checkpoint that resolves on the Hub lets the plan drop the dense shards, and only one already fetched to disk lets the pull drop them, so an unpublished or gated artifact leaves both exactly as they were. If injection still fails after that, the load tops the dense weights back up rather than handing from_pretrained a snapshot with no encoder in it. Measured against the real Hub on the 2.3 distilled Q4_K_M pick: 67.24 GB before, 18.92 GB with a 0.43 GB stand-in for the pre-cast artifact (the base entry drops from 24 files / 48.79 GB to 13 files / 0.04 GB). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Match the file's typing idiom for flow_shift models/training.py annotates with typing constructs throughout (105 Optional[...], no Union), and flow_shift was the one place using PEP 604. Union[] reads like the rest of the file, and it also drops the runtime evaluation that would raise on Python 3.9. * Bound the gallery blob cache, and three interlock fixes Four review findings, all reproduced first: - The gallery object-URL caches were unbounded. A clip runs from a few MB to a few hundred, both pages stay mounted after their first visit, and entries were only dropped on delete, so scrolling pinned everything for the session. Both pages now share a byte-budgeted LRU (512 MB video / 192 MB images) keyed off the visibility signal the near-viewport fetching already provides. On-screen media, the selected clip or image, and the item just fetched are never evicted, so eviction is invisible and a single item larger than the whole budget cannot evict itself into a refetch loop. - The image, video and chat load guards ran two independent training probes but returned early when the FIRST one raised, so an unreadable LLM backend disabled the diffusion interlock and a load could proceed straight into an active diffusion trainer on the same GPU. The probes are independent now. - An engine switch swallowed a failed teardown and published the new engine anyway, which is exactly the leak the unload exists to prevent: the arbiter's evictor, /images/unload and the next load all resolve through get_active_diffusion_engine(), so the still-resident pipeline (or a live sd-server) became unreachable and the next load allocated on top of it. The switch now fails and leaves the old engine published, so it stays reclaimable. - The native generation timeout was 30 minutes while the Images page waits up to 6 hours (SETTLE_MAX_MS), so slow-but-progressing CPU jobs died deterministically at the deadline. Measured on GPU-less runners, a 512x512 4-step Q2_K generation took 900 s on Linux and 1465 s on Windows, so larger images or step counts clear half an hour easily. The ceiling now matches the page's window and applies to the whole request: chunks of a split batch share one deadline instead of each getting a full budget. Cancellation is unchanged. Declined: gating the huggingfacenotorch extra off Python 3.9 over the conditional diffusers marker. The marker is deliberate and its comment says why: diffusers dropped 3.9 in 0.38, so pinning >=0.39 outright leaves pip no candidate and the whole extra unresolvable there. The pipelines it names live in studio/backend, which cannot install on 3.9 anyway (studio.txt pins matplotlib==3.10.9 and fastmcp>=3.0.2, both requires_python >=3.10), and the extra is the general core one, so the alternative drops 3.9 for library users who never touch Studio. * Close the load-versus-training-start race, and two picker fixes - The image and video load guards read is_active() and only then selected an engine, acquired the arbiter and registered the load. A /train/diffusion/start reserving inside that window freed residents the load had not registered yet, so the trainer came up beside a brand-new pipeline. The service already had exactly the right pattern for this in dataset_mutation, so gpu_load_admission mirrors it: reserve() refuses while an admission is open, an admission refuses once a start is reserved, both decided under the one lock. The span is only the registration, since begin_load returns as soon as the load is registered and _free_gpu_for_diffusion_training preempts an in-flight load from that point. Chat is deliberately not covered: its load spans an eviction plus a multi-minute GGUF load, and it admits models that fit beside training by design, which is a different contract from the diffusion pipeline's all-or-nothing one. - Hugging Face gives the LTX-2 family the image-to-video pipeline_tag (both Lightricks/LTX-2 and unsloth/LTX-2.3-GGUF report it), so a text-to-video-only filter dropped the flagship audio family out of Video Hub search while the rest of the app routed it to Video. - Task-scoped quant fit sized picks against the LARGEST visible device while resolve_diffusion_device_target returns a bare "cuda" and torch places on the current one. On a heterogeneous host that recommended a checkpoint sized for the bigger card and then loaded it onto the smaller one. Fit now uses the device the load actually lands on; identical on a homogeneous host. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Expose the persistent conditioning cache in the start schema DiffusionLoraConfig has carried cond_cache_dir for a while and the DiT trainer acts on it, but DiffusionTrainingStartRequest omitted the field, so Pydantic dropped it silently and every API-driven run fell back to the in-memory cache that is rebuilt from scratch each time. The warm path skips loading the VAE and the multi-GB text encoders on a rerun whose images, captions and resolution are unchanged, so this was a real capability that could not be reached. Contained like output_dir rather than left to the trainer subprocess's cwd, since it is another directory the trainer writes to. Blank or omitted still means the in-memory cache, so it must not resolve to the outputs root. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix diffusion policy and classification issues from review fp8 auto precision defaulted to precise accumulate on any non-consumer GPU, which made fp8 2.05x slower than int8 on RTX 6000 Ada and slower than not quantising at all. NVIDIA's professional whitepapers do publish equal FP8 rates for both accumulate modes there, so the hardware premise held, but the cost is in the cuBLAS path rather than the published rate. Default to fast accumulate: measured on B200 the flag is a no-op (4096^3 _scaled_mm at 3023.8 vs 3041.8 TFLOP/s, bitwise-identical output, 1.213 s vs 1.230 s end to end), so it is a large win where it bites and free where it does not. Precise accumulate stays available via transformer_quant_fast_accum. Z-Image's DiT is a Lumina2 derivative, so unsloth/Z-Image-GGUF and unsloth/Z-Image-Turbo-GGUF both declare general.architecture = "lumina2" and the whole line was tagged image-diffusion-unsupported and hidden from the Images "On Device" list, though validate_load_request loads them. Resolve shared archs from the repo/file name like bare "wan" already does, with a test asserting the picker and the loader agree for every family. The sage attention on-demand install ran an unpinned `pip install sageattention`, but PyPI's newest wheel is 1.0.6 and diffusers refuses anything below 2.1.1: the install always "succeeded", wrote an unusable version into the running venv, and was rejected on the next line. Carry the dispatcher's floor so pip resolves nothing instead. The dense-quant disk gate sized the download from the bf16-RESIDENT table. The fp32 families download twice that (Z-Image: 23,479 MiB against a 21,970 MiB gate), leaving a window where the check passed and the download filled the disk; Ideogram 4 ships fp8 and was overcharged the other way. Size the gate by published bytes, verified against HF sibling metadata for all 12 families. Patch installs went through unsloth_zoo, which refuses to import unless UNSLOTH_IS_PRESENT is set, and that is set by unsloth itself. The server imports unsloth at boot so it never showed there, but any other process ran silently unpatched with every install returning False, which is 13 test failures on a clean environment. Import unsloth and retry once, memoised per process. Also: the GGUF+LoRA refusal pointed at the native engine without saying a GPU host only selects it under UNSLOTH_DIFFUSION_ENGINE=sd_cpp, so the suggestion was unreachable; the gallery recipe recorded loras from the generate request alone, losing a load-time bake; load-progress claimed "40.07 GB downloaded" for a fully cached load; and pickers.tsx imported three catalog-group helpers it never used. Reported by oobabooga. * Keep the sd.cpp text encoder on CPU under Metal macos-14 loads FLUX.2-klein-4B Q2_K natively on mps and then dies on the first generation with exit code -6: ggml_metal_op_encode_impl: error: unsupported op 'RMS_NORM' -> ggml_abort LLMEmbedder::encode_prompt -> LLMRunner::compute -> GGMLRunner::compute ggml's Metal backend gates RMS_NORM on contiguous rows and aborts the process when that does not hold, with no per-op CPU fallback, so any LLM text encoder (Qwen3 for FLUX.2 and Z-Image, T5 for FLUX.1) takes sd-server down. The encoder runs once per prompt while the DiT runs every step, so pinning only the encoder keeps Metal for the part that matters. UNSLOTH_DIFFUSION_SD_CPP_METAL_TE_GPU=1 opts back in once ggml grows the kernel. * Gate the unsloth retry in the diffusion patch backend The retry added for the clean-environment patch failures is not free: importing unsloth pulls torch in behind it, which costs ~940 MB of RSS measured in a process that had neither, and on a host with no accelerator it fails anyway. A cross-platform CI job that had generated fine at ~900 s later died 19 s in with SIGTERM and every 'if: always()' step skipped, which is the runner being torn down rather than a step failing. Retry only when torch is already imported (true of the server and of anything patching a real module, and the condition that stops the retry from being what loads torch), unsloth is installed but not yet imported, and the first failure was the ImportError the sentinel guard raises. The clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only retry the unsloth import where it can succeed The gate still let the retry run on hosts unsloth does not support, which is where it is most harmful: a 7 GB macOS runner lost the Studio server 26 s into a load, and the Linux runner was torn down mid-generation. Neither MPS nor plain CPU can complete the import, so the retry there pays the cost and fails anyway. Require an accelerator unsloth actually supports (CUDA/ROCm via torch.cuda, or XPU), with UNSLOTH_ALLOW_CPU as the documented override, and hoist the predicate to module level so it is tested directly rather than through the import system. On a CPU-only host the retry no longer fires at all; on CUDA the clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Guard old diffusers, stream video exports, record conditioned recipes Three fixes from review. The 0.39-only pipeline classes (Flux2Klein, Z-Image, Krea 2, LTX-2, HunyuanImage) were resolved by getattr deep in the load, so on the older diffusers that packaging still allows on Python 3.9 -- diffusers dropped 3.9 in 0.38 and this project still supports it, so the 0.39 floor has to be conditional or the extra becomes unresolvable -- an advertised model failed with a bare AttributeError after its checkpoint had already been downloaded. Krea 2 already guarded itself this way; assert_pipeline_class_available now runs the same check for every image and video family from validation, before any fetch, and names the version and the fix. WebM export accumulated the whole VP9 output in a BytesIO and returned it as one bytes object that the response held again. The request caps allow 2048x2048 for 1024 frames, so an export runs to hundreds of MB and concurrent clicks could exhaust the process, while the MP4 route beside it already streamed from disk. transcode_to_file encodes to a temp file and the route returns a FileResponse with a background unlink, so nothing large is resident. A conditioned generation's recipe carried only the txt2img fields, so the gallery presented an inpaint or upscale result as a complete Create recipe and restoring it replayed an unrelated text-to-image request. The images themselves are still not persisted (user uploads with their own lifetime), but the workflow and its scalars are, restore reapplies them, and the toast now names the inputs that have to be supplied again instead of silently landing on Create. Reported by Codex. * Per-load video cancel event, family-gated image picker, cond cache refusal A cancelled video load could resume: begin_load cleared the shared cancel event, and unload() drops _loading without waiting for the worker, so the next load cleared the very object the cancelled worker was watching and its multi-gigabyte pull ran on alongside the replacement until the token check at the end. Each load now gets its own threading.Event, passed down through _fetch_te_prequant and _predownload_base, so a cancelled worker stays cancelled. A cached repo with a model_index.json was advertised as text-to-image on the trust rule alone, but validate_load_request also requires a detected image family, so a trusted pipeline of an unsupported class produced a picker row that deterministically 400s. The picker now applies both gates, mirroring the video branch. cond_cache_dir was accepted for sdxl and then ignored: only the DiT trainer reads it, while the SDXL trainer builds a per-run in-memory latent cache, so the promised cross-run reuse never happened. The route now refuses it with a 400 that names the families which do support it, checked against the resolved family so an omitted model_family with an SDXL base is caught too. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the frontend build broken by the gallery blob cache tsc -b failed on the branch head, so npm run build produced no dist and every platform job fell back to --api-only: blob-url-cache.ts(29,15): TS1294: This syntax is not allowed when 'erasableSyntaxOnly' is enabled dataset-labeling-grid.tsx / dataset-showcase.tsx: Argument of type '{ url: string; bytes: number; }' is not assignable to parameter of type 'string' The cache took its budget as a constructor parameter property, which the project's tsconfig forbids, and fetchGalleryObjectUrl now returns the blob size alongside the URL for that budget, which the two dataset thumbnail components still consumed as a bare string. Declare the field explicitly and destructure the URL at both call sites. tsc -b is clean and vite build emits dist again. * Recover from a ggml unsupported-op abort by restarting on the CPU backend ggml checks every node against the device's supports_op and calls GGML_ABORT when one is not implemented, because a single-backend graph has nowhere else to put it: there is no per-op CPU fallback. The whole sd-server dies with SIGABRT mid-generation and the user gets "the native image renderer stopped unexpectedly" with no way forward. Seen on macos-14 arm64 with FLUX.2-klein-4B Q2_K through the cross-platform CI: the text encoder is already pinned to CPU, and the abort moved into the denoise loop instead. ggml_metal_op_encode_impl: error: unsupported op 'MUL_MAT' -> ggml_abort StableDiffusionGGML::sample -> sample_k_diffusion A retry on the same backend would abort identically, so the load is restarted once with --backend cpu (the only flag that changes which backend executes the graph; --offload-to-cpu moves parameters, not compute) and the generation is re-submitted. The same checkpoint then renders slower rather than not at all. Strictly bounded: the signature must carry both the unsupported-op line and ggml_abort, the device must not already be CPU, and it happens once per load, so an OOM kill or a genuine crash still surfaces as itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix two tests that only fail in a full-suite run The 3.10 CI leg resolves PyAV 17, where av.container.OutputContainer is an immutable C type, so the no-libopus export test died on "cannot set 'add_stream' attribute of immutable type" before it asserted anything. Inject the refusal by wrapping the container av.open() returns instead; modules stay patchable on every build. Removing the injection makes the test fail again, so it still covers the branch it is named for. The Xet shim's degraded-path tests drop utils.hf_xet_fallback from sys.modules and import a throwaway copy. Restoring only the sys.modules entry left the utils package attribute bound to the throwaway, and the two disagreed for the rest of the process: a later monkeypatch of the dotted target patched one copy while the code under test imported the other, so the patch did nothing and test_fetch_te_prequant_only_reports_what_it_downloaded reached the real Hub and got a 401. Restore both bindings. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop an ignored-cancel sd-server, guard deletes during diffusion training, repair unusable managed binaries sd-server does not interrupt an in-flight job, so when it ignores a cancel the grace branch abandoned the poll and reported cancellation while the native job kept a core (or the GPU) busy to completion and held the server's job slot. The comment said the caller stops the server, but only unload does that immediately: a superseding load stops it after its multi-gigabyte download, and a load that then fails never gets there. Stop it here, as the deadline branch already does. DELETE /api/models/delete-finetuned checked only the LLM trainer, so it could rmtree the output directory a live diffusion LoRA run was about to write its adapter into. Consult the diffusion training service too, like the dataset mutation and model-load routes. find_sd_*_binary only checks is_file(), so an interrupted extraction (or a prebuilt for the wrong CPU) left a present-but-unrunnable binary the installer never retried: every load probed it, fell back to diffusers, and native inference stayed off until the directory was deleted by hand. Probe it and reinstall, but only for a copy under the installer-owned root -- SD_CLI_PATH, UNSLOTH_SD_CPP_PATH, an in-tree build and anything on PATH are the user's. * Plan the pre-cast text encoder, and make the cross-trainer GPU admission atomic An fp8 text-encoder request loads a hosted PRE-CAST checkpoint, but the image download plan never received text_encoder_quant, so the manager staged the base repo's dense encoder (FLUX.2-dev's Mistral-24B is ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) and the load then pulled the pre-cast file inline, outside the manager's progress and disk preflight. The plan now takes the field, resolves the hosted artifact with the same resolver the injection uses, stages that file, and drops only those components' dense weight shards. The load's own prefetch takes the same treatment, since it paid the same cost. Only a checkpoint that really resolves on the Hub earns the drop, so a gated or renamed artifact still stages the dense encoder the load will fall back to. The two trainers admitted each other with independent check-then-act guards: the diffusion route checks the LLM backend several network-bound preflights before it reserves, and the LLM route checks the diffusion service well before it spawns, so two near-simultaneous starts could both pass and train on one GPU. reserve() now re-tests the LLM backend under its own lock, and the LLM route holds the diffusion service's gpu_load_admission across its spawn, so exactly one of the two wins. Both halves fail open, so a chat-only install still trains. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the Advanced options a staged download planned against Staging does not set busy, so while a multi-gigabyte download runs the user can still change precision, memory mode, speed or the baked LoRA selection. The pending record held only the repo and artifact, and the completed download fired a load that read the CURRENT state: the staged file set could then be missing files that load needs (fetched inline, with no progress and no disk preflight) or hold gigabytes it no longer uses. One snapshot of every Advanced control is now taken when the plan is built, and it travels with the pending record into the load, so the load that runs is the one the download was planned for. * Do not advertise a family the installed diffusers cannot build The newer families (Z-Image, Krea 2, FLUX.2, LTX-2, HunyuanImage) exist only from diffusers 0.39, and 0.39 cannot be installed on Python 3.9 at all -- diffusers dropped 3.9 in 0.38, so the requirement is conditional or the whole extra becomes unresolvable. On such an environment the picker still offered those rows, every pick failed deterministically, and the error's advice to run pip install -U diffusers could not fix it without also upgrading Python. The cached-repo picker now applies the same availability check validate_load_request does, which is keyed on the pipeline class actually present rather than on the Python version, so it is also right for an intentionally pinned older diffusers on 3.10+. Fails open when diffusers cannot be imported at all: that is a different problem and the load path reports it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore the diffusion engine selection after each router test The active engine is module state, and several tests set it by plain assignment because what _activate does to it is the thing under test, so monkeypatch could not undo it. A leaked ENGINE_SD_CPP left get_active_diffusion_engine() handing back the sd.cpp backend for the rest of the process, and every later route that reads the active engine then saw an unloaded model: eight tests in test_openai_images_generations_route.py returned 503 in a full-suite run while passing on their own. The autouse fixture now snapshots and restores it. * Stream gallery clips, and close three races around them Four fixes from the latest review pass. The video gallery downloaded each clip into a blob before it could play, so playback waited on the whole file (tens to hundreds of MB), seeking was limited to what had arrived, and every viewed clip stayed pinned in the webview. The file route already streams and serves ranges; it just could not be a <video src> because it is bearer-gated. Mint a short-lived signed link instead (its own HMAC secret, 12 hour TTL, separate from the image links) and hand it to the element, which then fetches only the ranges it plays. That removes the blob budget, its LRU and every revoke on this page. The sd.cpp readiness probe accepted any process answering on the port, so a foreign server that grabbed the port between the bind check and the spawn was adopted as ours. Confirm the listener is our child before reporting ready, and stay best-effort (psutil missing, an unknown owner, or any probe error still passes) so the check can only reject a definitely foreign process. Dataset import held its lock for the extract but not for the upload path, so two concurrent uploads into the same folder interleaved; take the same lock and return 409. And reject Windows device names (CON, NUL, COM1..9, LPT1..9, with or without an extension) plus trailing periods in dataset names, which are unopenable on Windows. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Build the image download plan for the engine that will load /images/download-plan always asked the diffusers backend, while /images/load picks the engine per host: a GGUF pick on a machine with no usable GPU routes to native sd.cpp, which reads a single-file VAE plus text encoders and never opens the base repo's sharded components. Measured on unsloth/FLUX.2-klein-4B-GGUF (Q2_K): the plan staged 7.66 GB of FLUX.2-klein-4B components the native load discards, and the 7.80 GB sd-cli actually needs was then fetched inline by the loader, outside the download manager's progress and its disk preflight. Z-Image-Turbo is the same shape. The plan now asks whichever engine the load will select. predict_engine() applies the selection policy without any side effect: it activates nothing (staging a download must not unload the resident model) and only locates the binary rather than installing it, but still counts an installable binary as available, since that is what the load does on a fresh host. The native backend gains a download_plan built from the same _asset_specs the loader fetches, returning the same envelope, so the manager stages exactly the files sd-cli opens. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not let a queued generation outlive the model, and three scan fixes Five items from the latest review; four were real. An unload or arbiter eviction only cancels the generation holding _generate_lock. A second request queued behind it holds no cancel event yet, and Python locks are not FIFO, so it could take the lock the instant the active denoise released it, still see a loaded pipeline, and run a whole new denoise after the model was told to go away: the eviction then waits minutes for it and an image lands after the eject. Unload and a superseding load now raise a fence under _lock before they queue, and a generation that wins the lock while one is pending refuses instead. The cached-model scan judged pipeline completeness across every revision, so a repo holding an older complete snapshot plus a newer companion-only one read as complete while the snapshot from_pretrained actually opens has no transformer. Both scans now look at the revision the loader will open. Deleting a dataset image deleted its caption sidecar unconditionally, which for cat.jpg alongside cat.png removed the caption the survivor still resolves to. The sidecar now goes only with the last image of that stem, matching what the thumbnail cleanup beside it already did. Importing an example into a folder that holds no images but does hold files fell back to promoting the staging dir one file at a time, so an interruption left a partial dataset that the image_count check accepts as complete on retry. Those files are folded into the staging dir instead and the promotion stays a single atomic rename. The MPS generator report does not apply: torch.Generator(device="mps") has worked since PyTorch 2.0 (pytorch/pytorch#91348) and the studio installer pins torch>=2.4. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten diffusion comments Collapse the multi-line comment blocks across the image, video, sd.cpp and diffusion-training code to one or two lines each, and drop comments that only restate the statement below them. Comments only, no code or behaviour changes. * Tighten diffusion comments (second pass) Collapse the remaining multi-line comment blocks in the video page, training routes and service, sd.cpp server and installer, memory and speed planners, and the shared request models. Comments only, no code or behaviour changes. * Tighten diffusion comments (third pass) Collapse the remaining multi-line comment blocks in the attention, cache, LoRA, prequant, precision and compile-cache modules, the sd.cpp arg builder and engine, the video routes, the Ideogram 4 assembly, the model picker, and the diffusion test suites. Comments only, no code or behaviour changes. * Restore the dataset when an example import cannot be promoted Promotion folds the folder's pre-existing entries into the staging dir so the swap is one atomic rename. Every failure after that fold left the user with nothing: the 409 path and an os.replace error both fell through to 'finally: shutil.rmtree(staging)', which deleted the entries that had just been moved in there, while the response said 'Nothing was written'. A same-named entry was also unlinked outright before the promotion was known to succeed. Park superseded same-name entries in a rescue dir instead of deleting them, record every move, and restore all of them if any step of the promotion fails. The fold loop itself is covered too: renaming a non-writable directory raises EACCES on POSIX, which previously escaped as a 500 after the earlier entries had already been moved out. A failed rename now maps to the same retryable 409 as the rmdir conflict. Verified with the reported trigger (a non-empty mode-500 directory whose name collides with an imported file): the folder listing is now identical before and after the failed import. * Drain the teardown fence on a failing unload, give each load its own cancel event Two independent leaks on the image path, both already solved elsewhere in the same file. unload() incremented _teardown_waiters, ran _unload_locked() and decremented, with no try/finally, while the superseding-load path used a finally for the same pair. _unload_locked ends in clear_gpu_cache(), whose CUDA branch calls synchronize/empty_cache/ipc_collect unguarded, and a sticky CUDA fault makes those raise. The count then never drained, so every later generation was refused as cancelled for the life of the process, a fresh load included, since begin_load's own increment and decrement are symmetric. unload() is reached from the chat/video GPU handoff, the engine router and two training routes, so one fault during an ordinary handoff wedged image generation until restart. Release it in a finally. The image and native backends each cleared one shared cancel Event on a new load. unload() sets that event to cancel an in-flight multi-GB download and drops _loading in the same breath, so a replacement load is admitted while the cancelled worker is still inside the fetch, and its clear() re-enabled the very object that worker was watching: the cancelled download resumed and ran alongside the replacement. Take a fresh Event per load and thread it to the worker, as the video backend already does, and set it under the lock since begin_load now rebinds the attribute. * Name utf-8 on the diffusion text I/O and the sd.cpp subprocess pipes tests/test_text_io_encoding.py failed on five files this branch adds. Text I/O without an explicit encoding falls back to the Windows ANSI codepage, so a non-ASCII path or manifest value round-trips corrupted, and the three sd.cpp pipes decode the child's UTF-8 output as ANSI on Windows despite already passing errors = 'replace'. Eleven read_text() / write_text() sites across diffusion_compile_cache, diffusion_ideogram4 and diffusion_krea2, plus text = True on the sd-cli version probe, the sd-cli run and the sd-server pipe. * Record the load-time build on a gallery image's recipe A gallery record documents itself as the image's full generation recipe and is embedded in the PNG, but the only load-related field it carried was the repo id. A GGUF repo holds many quants, so that does not say which one made the pixels, and it says nothing about an adapter baked in at load time. The fallback meant to cover the baked case could never fire: with no loras on the request _adjust_baked_loras zeroes every baked adapter and _active_lora_pairs drops zero-weight entries, so active_loras was always empty. A baked-and-disabled build is not the same pipeline as a never-baked one, so the recipe could not reconstruct the image once the model was rebuilt. Persist model_kind, gguf_filename, transformer_quant and the baked adapter names, read off the load state rather than the request, and show them in the recipe popover. The new fields are optional with defaults, which matters because list_gallery_images drops any record that fails validation, so a required field would have emptied every existing gallery; a regression test pins that. * Drop eleven duplicated comment tails, restore the mxfp8 denial note The comment passes collapsed several wrapped blocks onto one line without deleting the last physical line of the original wrap, leaving the tail of each sentence repeated as its own comment underneath. Two of the eleven were re-worded rather than byte-identical, so a strict suffix match missed them. 6e16ad16f also dropped the line justifying the qwen-image mxfp8 denial while that rule stayed live in _FAMILY_SCHEME_DENY, under a header that then documented only fp8 and nvfp4. Restored. Comments only; verified with the AST gate. * Only let the dense-quant fallback use shards the prefetch actually staged The prefetch skips the base repo's transformer/ shards whenever a prequant checkpoint is expected, since that checkpoint replaces them. But a prequant fetch can fail for reasons the planner cannot see: an unpublished, gated or renamed artifact, a hub 5xx, a proxy, a checkpoint the validator rejects. The loader then fell through to from_pretrained(subfolder = 'transformer') and pulled those shards inside the load lock during 'finalizing', after the previous pipeline was already evicted, where the cancel event has no reach, load_progress has already reported bytes_downloaded == bytes_total, and the cache-disk gate had only reserved the small prequant checkpoint. That is verbatim the situation _dense_quant_prefetch_needed's own docstring exists to prevent. Gate the in-loader dense fallback on the shards being staged, read off the returned file list rather than the request so a failed size estimate closes it too, and let the GGUF build take over otherwise, which is what the prequant-sized replan already does one branch over. It is also the invariant the text-encoder path already enforces: only a component that really resolves may have its dense weights dropped from a plan or a prefetch. Adds the missing coverage for the gate's prequant arm, which the existing disk-gate tests never reached. * Refuse a training output_dir that resolves to the outputs root resolve_output_dir strips a leading 'outputs' and drops '' / '.' segments, so '.', './', './.', 'outputs', 'outputs/outputs' and ' . ' all clean away to nothing and land on the outputs root itself rather than a run directory under it. The DiT trainer then writes pytorch_lora_weights.safetensors flat into the root, where scan_trained_models and scan_checkpoints cannot see it (both filter is_dir()), and a second such run overwrites the first. The UI only checks the field is non-empty, so 'outputs' is one plausible run name away. Refuse it with a 400 that says what to do instead. cond_cache_dir collapses the same way but has an honest 'off' to fall back to, so a root-resolving value now means the in-memory cache, which is what the comment above it already promised: otherwise a run drops one flat safetensors per cached latent and caption into the directory trained models live in. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Repair three defects the main merge left behind Git merged all three files without a conflict, but the result was wrong in each case. The tests only started failing once the merged tree was run. routes/inference.py duplicated the GGUF load block: this branch had moved the gguf_load_in_flight marker and the _hub_download_blocks_gguf_load guard under "if config.is_gguf and config.gguf_hf_repo", and the conflict resolution re-added main's copy at the old position, so both ran. Dropped main's copy; the earlier placement is the deliberate one, so a 409 from the hub guard cannot tear down a resident Images or Video pipeline. test_gpu_selection.py still called _hf_offline_if_dns_dead, which main renamed to _hf_offline_if_unreachable_for (#7591). Disjoint edits, so no conflict, but four route-error tests referenced a function that no longer exists. test_gguf_load_cache_reuse.py anchored its ordering assertion with rindex over "if config.is_gguf:", taking the last one before the load marker. That only held while _resolve_inherited_extra_args sat above every such line; main has since hoisted it above the gpu_ids preflight, so the anchor landed between the call and the marker and the assertion compared against an unrelated later call site. The ordering it checks is a property of _load_model_impl as a whole, so it now anchors on the function. The ordering itself is intact: _resolve_inherited_extra_args, then the gguf_load_in_flight marker, then the hub guard, then the chat handoff, then unload_model. * Stop a real sd.cpp install from breaking its own discovery tests The five "nothing is installed" assertions in test_sd_cpp_engine.py cleared SD_CLI_PATH and UNSLOTH_SD_CPP_PATH and patched Path.home, which covers hops 1, 2 and the fallback half of hop 3. It leaves two hops live. Hop 3 goes through managed_install_root(), which honors UNSLOTH_STUDIO_HOME and STUDIO_HOME and resolves to the stable-diffusion.cpp directory beside the studio home. That is the documented way to run side-by-side Studios, so anyone who has one set gets a real sd-cli back from a finder the test expects to return None. Hop 4 is the in-tree developer build, which does the same for anyone who built sd.cpp inside the checkout. Isolated with an autouse fixture rather than another helper call, because reaching the failure needs no fixture: SdCppEngine(binary = None) runs the finder from its constructor, which is why test_generate_raises_when_binary_missing failed too. The fixture points the studio home and the in-tree root at an empty tmp tree, so every hop is answered by the test rather than by the box. The in-tree root moves into a named in_tree_install_root() so it can be pointed somewhere empty; behaviour is unchanged, including the OSError and IndexError guard on an unexpected layout. * Make the sd.cpp uninstall test actually extract the code it tests The two sed ranges anchored the production fragments at column 0, but both blocks sit inside the main removal function and are indented, so each range matched nothing and LOOP_FILE / DEFAULT_FILE came out empty. Sourcing an empty file is a no-op, so the suite reported 4 passed / 7 failed: the seven removal assertions failed because nothing ran, and the four that passed were 'kept' assertions that pass trivially when nothing runs. The shell job auto-discovers tests/sh/test_*.sh and runs each under set -e, and this file is not in its skip list, so it was a deterministic red. Anchor on optional leading whitespace and fail loudly on an empty or _remove_path-less fragment, so a future reshuffle of uninstall.sh cannot make the suite vacuous again. Now 11 passed, 0 failed, against the real removal loop and the real default-mode block. * Agree on what an engine can build, and on what the installer owns Two gates, each half-applied. The unbuildable-family gate had one caller, the image branch of the cached repo picker. The GGUF classifier, the local-model classifier and the video branch had none, so on a diffusers too old for a family the picker still offered its GGUF and the load then failed. Meanwhile the loader asserted the diffusers pipeline class before engine selection, so a GGUF this host routes to native sd.cpp, which instantiates no pipeline class at all, was refused with an upgrade instruction that could not help. Both are wrong one-sidedly: hide a family only when NEITHER engine can build it, and demand the diffusers class only when diffusers is what will load it. One predicate, family_buildable_here, now answers both, so the picker and the loader cannot disagree. The population is real on Python 3.9, whose diffusers ceiling is 0.36: Flux2KleinPipeline, Krea2Pipeline and LTX2Pipeline are all absent there, and FLUX.2-klein GGUF is a repo in this PR's title. That assertion also raised RuntimeError, which /images/load maps to 409, the status that otherwise means a load is already in progress, and which escaped /images/download-plan (it catches ValueError and FileNotFoundError) as a bare 500 with the message lost. It is an unloadable pick like every other, so it raises ValueError and both routes answer 400 with the text intact. The managed-binary repair used a path test for ownership while the installer uses a marker. On a managed root without the marker, which is any install predating it, the repair deleted sd-server and the reinstall was then refused, permanently, because the surviving sd-cli keeps the directory non-empty so the marker can never be claimed: the user went from an unrunnable binary to no binary and no way back. Require the marker before discarding, which is the same definition of ours that uninstall.sh already uses to keep a user's own stable-diffusion.cpp checkout. A genuinely interrupted extraction still self-heals, since install() writes the marker before it extracts. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add gguf and av to the studio extra so the wheel install matches studio.txt This branch added both to studio/backend/requirements/studio.txt, which is what install.sh uses, but never to the studio extra in pyproject.toml, which is what 'pip install unsloth[studio]' uses. Nothing else keeps the two in sync, so tests/studio/install/test_studio_extra_matches_requirements.py failed deterministically, and the CI job runs the whole tests/ tree. The drift is not only a red test: gguf backs diffusers' GGUFQuantizationConfig and av does the MP4 encode and audio mux, so a wheel install got a Studio that cannot read a GGUF or export a video, which is most of what this PR adds. * Fix two test-suite defects: the CPU-only patch gate and a popped module The patch backend reaches unsloth_zoo's helpers only through an ~940 MB 'import unsloth', which a CPU-only host cannot complete, so every patch install returned False and nine arch/eager/compile tests failed there -- exactly the runners the retry was narrowed to protect. unsloth_zoo only wants UNSLOTH_IS_PRESENT in the environment, which costs nothing and needs no accelerator, and the conftest already sets its sibling UNSLOTH_ALLOW_CPU. Set it there too, as run.py and main.py already do at module scope, so no real server ever took the expensive route. CPU-only goes from 9 failed to 24 passed, 1 skipped; the GPU run is unchanged at 29 passed. test_setup_cache_env_hf_home popped utils.hf_cache_settings to model a fresh process and never restored it, so a later import built a second module object and rebound it on the utils package. test_hf_cache_settings then wrote its setting into one object while core.inference.diffusion read the other, the same split-module failure the xet shim already had. Restore both bindings in teardown: 21 pass together, and each file still passes alone. * Restore the setup_fail assertions the main merge reverted A fourth instance of the class 9541cc535 fixed: the merge took main's studio/setup.sh, which #7644 changed to abort through the setup_fail helper so desktop mode still emits [TAURI:ERROR], but kept this branch's older copy of the test, which still asserted the literal 'exit 1'. setup.sh is byte-identical to main here and the only diff in this file was the reverted assertions, so take main's version. 60 passed. * Fence the video teardown, reject non-finite clipping knobs, stop a test installing sd.cpp The video backend never had the teardown fence the image backend documents. Its unload and its superseding-load path both signalled the active generation, then did 'with self._generate_lock: pass' and tore the state down with the lock free. A generation queued behind that barrier holds no cancel event yet, so the signal cannot reach it, and Python locks are not FIFO: it won the lock the instant the barrier released it, read a still-loaded state and denoised a whole clip against the pipeline being freed. Reproduced on both paths, where the queued generation returned a finished MP4. Mirror the image backend: count waiters, refuse a generation while one is pending, and tear down inside the barrier with the counter released in a finally. _teardown_state becomes _teardown_state_locked since the caller now holds both locks. max_grad_norm and snr_gamma were bounded on one side only, so 1e309 floated to inf and passed. clip_grad_norm_ then computes an infinite clip coefficient, clamps it to 1.0 and scales nothing, and min(snr, inf) / snr makes every min-SNR weight 1.0. The run starts, reports normal progress and trains with the requested knob silently disabled. Measured both. NaN already failed the bounds; allow_inf_nan makes that explicit. learning_rate and flow_shift already guard this exact vector. test_load_routes_to_sd_cpp_on_cpu stubbed ensure_sd_cpp_binary but not ensure_sd_server_binary, which select_and_activate_engine probes first with installs enabled, so the unit test downloaded and unpacked 108 MB into the developer's real ~/.unsloth when UNSLOTH_STUDIO_HOME was unset. Now 0 bytes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Key the video schedule reset on the resolved defaults, not the repo id A GGUF repo holds several variants, so another client swapping a distilled build for the base one from the same repo changes the steps and guidance the backend reports while repo_id stays put. modelChanged stayed false, the page kept the previous schedule, and later generations ran an 8-step distilled setting on a model that expects roughly 40 steps and CFG 4. Include the reported defaults in the key. They only move when the resident artifact does, so a manual steps change still survives a status poll. * Retire the release-lag pin now that 2026.7.6 carries the relaxed gates The virgin Windows container row that installs from PyPI on purpose was tolerated with continue-on-error, because a Server Core container has no Microsoft Store and so no winget, and the released studio/setup.ps1 hard-stopped on a winget-only git gate and reached for winget again for the VC++ runtime. #7549 relaxed both, but it landed on 2026-07-28 and the newest wheel was 2026.7.5 from the 23rd, so the row could not get past it. The pin carried its own tripwire for exactly this moment, and it has fired: unsloth 2026.7.6 shipped 2026-07-29 and the released wheel now installs end to end, so the step reported success and the assertion errored with "delete this pin and drop continue-on-error from the Install step so this row gates". Doing that. Both rows now gate unconditionally, and the assertion that enumerated the accepted failure signatures goes with the pin it protected, since there is no longer an accepted failure for it to describe. The other three release-lag mentions in this file are descriptive rather than tolerated-failure pins: they explain why the fedora and ubuntu-nonroot legs run with overlay: true, and both already refuse a triton failure as a regression rather than accepting it as lag. Left alone. * Pin the hosted pre-quant onto the plain-torch fp8 kernel A Z-Image GGUF pick at the DEFAULT speed mode, once the hosted fp8 pre-quant is actually reachable, dies at generate with an HTTP 500: torch._dynamo.exc.Unsupported: Operator does not support running with fake tensors. Developer debug context: unsupported operator: mslk.f8f8bf16_rowwise.default _fp8_config already pins KernelPreference.TORCH when it BUILDS a config, precisely because the default AUTO switches to the MSLK kernel wherever an mslk package is importable (sm90+). A hosted checkpoint escapes that pin completely: the preference is serialized per Float8Tensor, and all 239 weights in the published Z-Image-Turbo-FP8 checkpoint carry AUTO. Loading it re-arms the very kernel the pin exists to avoid, mslk.f8f8bf16_rowwise has no fake impl, and the first compiled generate therefore cannot be traced. Isolated away from the product to be sure of the mechanism: quantise one Linear three ways on this box and compile each. KernelPreference.TORCH eager ok, compiled ok KernelPreference.AUTO eager ok, compiled FAILS on mslk.f8f8bf16_rowwise library default eager ok, compiled FAILS the same way So the pin is correct and necessary, and the only gap is that the hosted path never got it. _validate_checkpoint checks scheme, granularity, base model, min_features, exclude tokens and fast_accum, but not this. Rewriting the preference on load is safe: it selects a matmul kernel, it is not weight data, so the tensors stay bit-identical and the checkpoint's own state_dict_sha256 still describes them. It is also the faster path compiled, since the opaque extern call blocks inductor quantize fusion. Why this went unnoticed: the pre-quant repos are private, so nothing that could not read them ever took this path. It becomes the default the moment they are readable. Verified end to end at 1024x1024, 8 steps, speed_mode default, on the published checkpoint: before HTTP 500 at generate, diffusion.generate_failed after pinned 239 weights to the plain-torch fp8 kernel, load 10.0 s, transformer_quant fp8 with compiled engaged, cold 7.9 s, warm 0.87 0.87 0.87 0.86 0.87 s That warm number also beats the 1.4 s recorded for this shape on 2026-07-26. 159 prequant and transformer-quant tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cover the fp8 kernel pin so the 500 cannot come back quietly The hosted pre-quant re-arming MSLK was only visible as an HTTP 500 on a compiled generate, which needs a GPU, a readable private repo and an importable mslk to reproduce. None of that is available in CI, so the pin would rot unnoticed. Three hermetic cases on _pin_kernel_preference instead: AUTO weights are rewritten and already-TORCH ones are left alone (counting only what changed), a weight that refuses the assignment does not sink the whole load, and with no torchao enum available the checkpoint is left exactly as saved. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: tighten comments across the image generation, video and training code Rewrite the comments added by this branch to be shorter and clearer: collapse multi-line explanations into one or two lines, drop comments that only restate the code, and keep the rationale that explains why a choice was made. No code changes. * Studio: let the pipeline-class guard survive a host with no diffusers Backend CI installs the CPU-only dependency set, which has no diffusers, and runs without a GPU. 13 tests failed there for three separate reasons, one of which is a real product bug rather than a test-environment gap. assert_pipeline_class_available did a bare "import diffusers", so on any host without it the call raised ModuleNotFoundError instead of the ValueError its own contract promises. /images/download-plan catches only ValueError and FileNotFoundError, so that escaped as a bare 500 with the message lost, which is the exact failure the guard exists to prevent. It is reachable in production: the native sd.cpp engine serves GGUF picks on a CPU or Apple host where diffusers is never installed. Absent diffusers now returns, since the check answers "is the installed diffusers new enough for this family" and there is no version to judge; a pick that genuinely needs diffusers still fails in the loader with its own message. The rest were tests asserting through gates unrelated to what they cover: - Three cond_cache_dir route tests drive a DiT family, which the start route refuses without an accelerator. They now take the existing dit_train_host fixture, written for this case, so they keep testing the schema on every host. - test_download_plan_forwards_the_load_time_controls stubs the diffusers planner but sends a GGUF pick, which routes to the native planner on a GPU-less host. It now pins the engine, so it tests kwarg forwarding rather than the hardware. Engine selection keeps its own tests. - The pipeline-class guard test needed a real diffusers only for its sweep over every shipped family. That half is split out and skips on its own; the stub-driven refusal needs no diffusers and still runs. Added a test for the absent-diffusers contract above. 409 pass on a GPU host with diffusers; 301 pass and 1 skips with diffusers blocked and no GPU. * Studio: cut the image generation comments down further Second pass over the comments this branch adds. Collapse the multi-line blocks that still read as paragraphs, rewrite the longest one-liners so they say the same thing in fewer words, and drop a stale comment that had drifted away from the constant it described. No code changes: only comments and whitespace. * Studio: workflow rail, sidebar Video row, and Train settings polish Sidebar - Pin Video under Images by default; persist store bumped to v3 so an untouched install adopts it while a custom arrangement is left alone. - Add "Customize sidebar" to the end of the More flyout, opening Settings > Appearance scrolled to the sidebar nav section. Images - Replace the workflow dropdown with an icon rail down the left edge: all seven workflows visible, keyboard nav, tooltips carry the labels. - Workflows stay selectable with no model loaded, so one can be set up before picking a model. Generate is already gated on a loaded model. - Video moves out of the Create/Train strip into a link at the far right, with a matching Images link on the Video page. - Wider gutters around the settings column, headings matched to Train. Train - Field guidance moves into "i" tooltips; only state that limits a control stays on the page. Steps, LoRA rank and the rest gain hints. - /info reports params, qlora_vram_gb, gated and note as fields, so the family note renders as chips. vram_note is rebuilt from them unchanged for older clients. Shared - Number steppers appear on hover or focus. - Lighter control border (#e9e9e9) on Images and Video, light mode only. * Studio: move Advanced inline on Images and Video - Replace the top-bar toggle and right-docked Advanced panel with a disclosure under Seed, so load-time tuning sits with the settings it affects and opening it no longer shrinks the preview canvas. - Video uses the bordered variant, since offload and memory decide whether a model fits at all. - Open state persists per page in localStorage. - Tighten the Steps unit trigger: 14px chevron, less right padding. * Studio: full width media pages and layout polish - Drop the 1100px cap on Images Create, Images Train and Video, so the preview canvas grows with the window instead of sitting in a band. - Align the workflow rail to the model selector label, widen the gap before its divider, and restore the divider itself. - Advanced: more room above and below, icon and label sized to the slider rows, and the same quiet row on Video as on Images. - Give Video a pane heading and description, matching Images. - More space above Seed on both pages. - More no longer takes the active style when the current page is one of its own rows. * Studio: fix the Extend side toggles The resting outline used a ring, and index.css blanks the ring on the button holding mouse focus, so the side you just clicked lost its border until focus moved. Use a border for the outline and leave the ring to focus-visible. Also separates the two states properly (the off state had no surface of its own), drops the border in dark mode as the inputs do, and adds the aria-pressed and focus-visible styling the buttons were missing. * Studio: drop the border on the Extend side toggles Light mode should carry no border either, so the fill alone marks the state: a muted surface when off, a primary tint when on. Matches the borderless treatment dark mode already had. * Studio: move the Images workflows into the sidebar The workflow switcher was a vertical icon rail on the page. It now lives in the sidebar under Images, so the page keeps that width for the canvas. - Workflows list under the Images row, with a chevron to fold them away. Hovering the row peeks them in a flyout on the standard menu surface. - Clicking Images while already there toggles the list instead of navigating. - The listed workflow carries the highlight, so the Images row drops it. - Create takes its own icon: it was sharing the New chat pencil. - Model hub moves above Projects. Persisted layouts bump to v4 and only adopt the new order where the stored one is still a shipped default. - Images and Video content both start at 32px, clear of the sidebar. * Studio: simplify the Images workflow submenu Dropping the hover flyout: two ways to reach the same seven workflows, one of them overlapping the list right below it, read as clutter. - The flyout is gone. The workflows are rows under Images and nothing else. - On the Images page they are always open, since they are that page's switcher. Elsewhere they stay folded, and the row's chevron opens them. - Create takes the sparkles icon. - Create and Train both pad their settings column to 40px a side, so the two tabs line up with each other and with the model selector. * Studio: dock the generate action and tidy the Create controls The primary action sat at the foot of a long scroll, so it was off screen until you scrolled for it. - Generate, Video's Generate and Train's Start training float at the bottom of their settings column. No bar behind them: hover lightens the fill rather than thinning it, and the disabled state is opaque, so the controls underneath never show through. - Aspect ratios read as names: Square (1:1), Widescreen (16:9) and so on. - Width and height replace their sliders with two compact boxes. Type a size or pick one from the menu; the value still snaps into range and still drives the locked ratio. - Negative prompt gets a quiet disclosure under the prompt. It was there before but only above guidance 0, which is not the default, so it never showed. - The info "i" is smaller across the settings UI. * Studio: reveal Images and Video field hints on hover The "i" next to every field label sat there permanently, which made a column of settings read as busier than it is. Scoped to the diffusion pages by CSS rather than per component: the hints come from the page's own Field and SelectRow, the train panel's FieldLabel, and the chat ParamSlider, which is shared and should not change for chat. Reveals when the pointer is inside the field, on focus-within for keyboard, and stays inert while hidden. InfoHint grows a data-slot so the rule has a stable hook, rather than keying off its aria-label. * Studio: one-line sliders, edge fades, shared negative prompt - Steps, Guidance and the rest put label, track and value on one row. ParamSlider is shared with chat, so this is an opt-in prop and chat keeps the stacked layout. - The settings columns fade at whichever edge they run past, as the sidebar and model picker do, instead of cutting off. New useScrollFades hook drives all three. - Negative prompt moves to a shared component and Video picks it up, so both pages collapse it the same way. - Right gutter sits closer to the rule, Steps takes a bigger break above it, and the Advanced rule sits between its neighbours rather than up against the field above. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: add the API monitor to the sidebar nav Sits under More, below Export. Points at the monitor page, not the API keys dialog the profile menu opens. No store version bump: an id missing from a stored layout is appended with its default pinned state, so existing sidebars gain the row at the end and keep their own order. * Studio: take main's copy of the sidebar spinner test The merge kept this branch's older version, which pins the Recents row to h-[33px]. Main already relaxed that to any height, since row density moves independently of the trailing column the test is about. * Studio: fix FLUX.2 klein size resolution, HunyuanVideo tier routing, and the personalization save Five fixes found while auditing the model registries against the live Hub. FLUX.2 klein 9B loaded against the 4B config. One family covers both klein sizes and defaults to 4B, relying on the base_model card tag for the real base, but the trust allowlist only had klein-4B, so the correct tag was discarded and a 9B GGUF (inner_dim 4096) was loaded against a 4B config (3072). That surfaced as a bare shape mismatch from inside the GGUF quantizer naming neither the file nor the repo. Adds the missing allowlist entries and a header-level size check that fails early with a legible message; the check is fail-open, so an unreadable file, a non-FLUX.2 family, or an unmapped base leaves the load exactly as it was. The sd.cpp text encoder rule matched the literal "klein-9b", so klein-base-9B was handed the 4B encoder. HunyuanVideo-1.5 720p checkpoints routed to the 480p family. Only the literal 720p_t2v path was aliased, so 720p_i2v and every GGUF repack fell through to the generic token and inherited the 480p base repo, which also supplies the VAE and text encoder. The tier is baked into the weights (transformer target_size 640 vs 960, scheduler shift 5.0 vs 9.0) and the two bucket lists are disjoint, so this ran the whole pipeline off-tier. Resolution presets that the checkpoints were never trained for. Wan2.2 TI2V-5B is 720P-only upstream (SUPPORTED_SIZES is exactly 704x1280 and 1280x704, asserted in generate.py), and the HunyuanVideo 480p square preset 624x624 is not a bucket of its tier; 640x640 is. Personalization could never be saved. The frontend ships eight sidebar nav ids and refills every missing one on each save, but the backend Literal had seven and no "api", so every PUT to /api/settings/personalization returned 422. Aligns the server defaults and the Literal with the shipped layout. A catalog row pointing at a file that does not exist. unsloth/Qwen-Image-2512-FP8 holds torch prequant .pt checkpoints, not qwen-image-2512-fp8.safetensors, and fp8 is denied for this family anyway because it renders black, yet the row was the auto-route target on any GPU above 40 GB. Also marks FLUX.1-schnell, Krea-2-Turbo and the two Ideogram repos gated: all four are gated on the Hub today, and schnell being Apache-2.0 does not make it ungated. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: stop a case-only duplicate upload destroying an image, and size the load against the right GPU Two fixes from the review round. A dataset upload containing two names that differ only by case silently loses one on Windows and macOS. The in-batch duplicate check is an exact string compare, so "Cat.png" and "cat.png" both pass, but on a case-folding filesystem they are one destination: the commit step moves the first staged part aside, writes the second over it, then deletes the backup, while the response still reports both as uploaded. Captions collapse the same way and never even reach the image branch. No single folder can hold such a pair, but the Windows and macOS open dialogs flatten search and Recents results across folders into one multi-selectable list, which is a normal way a LoRA dataset gets assembled. Rejecting the pair everywhere would regress Linux, where the two really are different files with their own sidecars, so the check probes the filesystem once per process instead of keying off sys.platform: macOS also ships case-sensitive APFS volumes, and a Linux host can keep its Studio home on an exFAT or NTFS mount. A failed probe answers "case-sensitive", which leaves today's behaviour untouched. The quantization fit budget was sized against the wrong card under a reordering CUDA_VISIBLE_DEVICES. A bare "cuda" load lands on visible ordinal 0, but the reducer took the lowest PHYSICAL index, which the nvidia-smi path reports as index_kind "physical". Under "3,1" that sizes against GPU 1 while the pipeline loads onto GPU 3. The hook's own interface doc already promised the lowest visible ordinal. Ranks by visible_ordinal, falling back to index only for an older backend that omits it. * Studio: rework the diffusion dataset panel and align the pane gutters Upload: - one Upload button beside the dataset name, with an upload icon - picking files uploads them, no second confirm click - Add button beside the Training images dropdown, so images can go into a set that already exists - fall back to the upload form when a selected dataset no longer resolves Labeling grid: - two columns at any width; sm:grid-cols-3 crushed the tiles in a fixed width column - pin the caption size so the Textarea's md:text-sm does not outsize a tile - name the tile hover group; a bare one revealed every tile's Remove at once - Remove is an icon button on the image, not a word over the artwork - drop the amber tile fill, keep the No caption label - match the header, toggle and status text to the section's type scale Gutters: - the run area and the preview canvas now sit 40px off the rule, the gutter the settings column has off the page edge * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: give the dataset preview per-image hover and remove, and wrap it onto rows - each thumbnail is its own button, so it can carry an X that removes just that image; the strip was one button and could not nest a second control - hover lightens the image instead of drawing a border - wraps onto more rows rather than scrolling sideways, which also keeps the bottom corners rounded - samples 12 thumbnails, up from 8, now that a second row is available - a delete refreshes the panel's dataset counts * Studio: show the workflow icon beside the Images and Video headings Images reads it off WORKFLOW_TABS, so it is the same icon the sidebar submenu shows and it follows the active workflow. Video's heading is also Create, so it takes the Create icon rather than the nav row's film slate. * Studio: reject a duplicate LoRA id on the diffusion load path too DiffusionGenerateRequest already refuses a repeated adapter id, with a comment naming the hazard: _resolve_lora_set suffixes colliding adapter names, so the same id resolves the SAME adapter twice and set_adapters stacks both copies past the per-adapter weight bound. DiffusionLoadRequest bounds only the list length, and it matters more there. Generation-time stacking spoils one image; on the load path the adapters are baked into the quantized build before compilation, so the unintended combination rides every image until a reload. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: teach the GGUF reuse test double about holds_no_vram The resident-GGUF fast path consults llama_backend.holds_no_vram before asserting CHAT ownership, but two tests hand it a SimpleNamespace built before that attribute existed, so they raised AttributeError and returned a 500. A real LlamaCppBackend exposes it as a property; the doubles now carry it too. Product code is unchanged: these pass on main and were the branch breaking its own test, not a defect in the guard. --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: michaelhan <michaelhan2050@gmail.com> Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: Michael Han <107991372+shimmyshimmer@users.noreply.github.com> | 1 个月前 | |
Studio: add image generation, editing workflows and LoRA training with Unsloth GGUFs (#6763) * Tighten comments in the image stack tests and scripts * Close video single-file, training reservation, and image mount-resume gaps Route on-device single-checkpoint video folders through the single_file loader: a bare local .safetensors directory (no model_index.json) is advertised as a pipeline with no filename, so validation rejected it before it could load. Reinterpret the pick as a single_file load of the sole checkpoint, mirroring the image load route. Treat a reserved-but-not-yet-spawned LLM training start as active in is_training_active() so /images/load, /video/load, and /diffusion/start cannot race the reserved run for VRAM during the pre-spawn free window. Mirrors the diffusion training service reservation. Resume an in-flight image generation on the Images page mount: probe generate-progress, re-enter the poll loop, and refresh the gallery on completion so a run started elsewhere is reflected and its saved image appears without a manual refresh. Seed resident image defaults from the resolved base_repo rather than a possibly path-shaped repo_id so the first resident generation uses the right recipe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Publish image generation active state before pre-denoise setup generate() assigned self._gen only at the pipe() call, after deferred compile, LoRA resolution/application, and ControlNet download/build had run. Across that setup window generate_progress() reported inactive even though _generate_lock was held, so a reloaded page's mount probe showed idle and let a second generate queue behind the first. Publish an active step-0 _GenState the moment the generation lock is acquired, before the setup work, and clear it in the outer finally so a setup-time error cannot leave the UI stuck active. Mirrors the video backend's queued phase and the training start guard. * Studio: fix diffusion install ownership, dataset upload atomicity, gallery pagination, and teardown races install_sd_cpp_prebuilt: only write the .unsloth-studio-owned marker when the install created the target directory or it was empty. Adopting a pre-existing, unowned, non-empty directory (a user's own stable-diffusion.cpp checkout) made it eligible for the uninstaller's recursive delete. routes/training upload: make the multi-file promotion transactional. Back up each displaced original and roll every destination back on any failure, so a mid-loop rename error can no longer partially overwrite the live dataset. routes/training _resolve_dataset_folder: reject a symlinked dataset directory and prove the resolved folder stays under the datasets root, so image read/caption/delete cannot escape the root through a link. routes/training delete: escape glob metacharacters in the thumbnail filename so deleting an image named like [ab].png removes only its own thumbnails. image_gallery / video_gallery listing: filter records against the response schema inside the pager via a valid callback, so offset/limit/has_more all count over accepted records. A leading schema-invalid record no longer returns an empty page with has_more=true and stalls infinite scroll at offset 0. image_gallery / video_gallery save: publish via a temp file plus atomic rename (the sidecar is the video pair's commit marker) and clean up on failure, so a partial write never surfaces a truncated PNG or strands an orphan MP4. diffusion_train_common discovery: treat an empty caption sidecar as a metadata tombstone that still falls through to the dreambooth instance prompt, so clearing every metadata caption no longer fails with no captioned images found. diffusion backend unload: wait for an in-flight denoise to exit before tearing down process-wide patches and state, mirroring the load path. diffusion_engine_router: serialize the whole check/unload/publish transition so a concurrent selection cannot return the engine being unloaded. uninstall.ps1: gate the default sd.cpp process stop on the owner marker so a user's own sd-server is not terminated for a directory we then keep. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reject native batch seeds outside the JSON-safe range * Refuse sd.cpp install into unowned non-empty target dir When the install target already exists, is non-empty and lacks the .unsloth-studio-owned marker (a user's own stable-diffusion.cpp checkout, or unrelated files beside a custom Studio root), install() previously still extracted the release into it. Skipping the ownership marker only stopped the uninstaller from deleting the directory; extraction still merged binaries into the user's working tree and could overwrite same-named files. Fail up front with a clear message pointing the user at a fresh/empty location before any download or extraction, leaving their directory untouched. Update the ownership test suite to assert the refusal. * Studio: gate dataset uploads on the symlink check and surface local video single-file checkpoints * Tighten comments and docstrings added by the image-generation fixes * Studio: close arbiter load-registration race and surface native progress + local pipeline folders Publish native sd.cpp generate progress (_gen) before LoRA resolution so a reload probe reads active during setup, matching the diffusers path. Register the diffusion/video GPU load under the arbiter lock (acquire_for now takes a register callback) so a competing acquire cannot evict an owner before its load is marked in-flight and let two loaders allocate VRAM at once. Admit local diffusers pipeline folders (root model_index.json, weights in component subdirs) in the local model scan so they reach task tagging and the On Device picker. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: include marker-owned custom sd.cpp roots in the uninstall stop scan; harden Pester install against the nuget.exe PSGallery bootstrap * Studio: surface local pipeline scan roots, tag single-file checkpoints by filename, mark companion-only pipelines partial - _scan_models_dir: admit a scan folder that is itself a diffusers pipeline (root model_index.json, weights in transformer/ vae/ subdirs). _is_model_directory rejects such a root, so the child scan would list the component subdirs as bogus models and hide the real pipeline; treat the root as one model via _local_pipeline_index. - _local_is_diffusers / _local_model_task: include the sole checkpoint filename in the family-detection needles (_local_family_needles, resolved via resolve_local_single_file). A generically named folder holding one loadable qwen-image-*.safetensors / ltx-*.safetensors identifies its family only from the filename; the load route already resolves that file, so tag it or the task-scoped picker (which rejects task=null) hides the on-device model. - list_cached_models: mark a companion-only base snapshot partial. A GGUF image load prefetches the base repo's VAE / text-encoder / model_index.json but skips the transformer (the GGUF supplies it); the snapshot has a pipeline manifest yet is not a loadable BF16 pipeline, and _cached_repo_partial misses it. _repo_pipeline_missing_denoiser flags a pipeline snapshot whose transformer/ or unet/ component carries no weight, so the picker drops it instead of advertising it as fully on-device. * Studio: preserve foreign gallery files, force safetensors on remote ControlNets, and close dataset/seed/GPU gaps Gallery clear/delete now scope to Studio-owned files: image_gallery and video_gallery skip PNGs / MP4s without a readable recipe (a hand-dropped or orphan file the listing already hides), so clear() and a guessed-id delete no longer destroy files the gallery never surfaced. Remote ControlNets now force use_safetensors: a bare owner/name reaches from_pretrained without the base trust gate, and the Hub scan fails open when unavailable, so requiring safetensors closes the pickle deserialization vector. POSIX uninstall now stops resident sd-server / sd-cli under an owned sd.cpp root before removing the tree (marker-gated), mirroring the Windows stop-before-delete scan; a live native server no longer survives unlinking its binary. Diffusion dataset containment: the training-start read path and the discovery picker route bare names through the protected resolver, so a symlinked dataset is rejected / not advertised like the caption/delete routes already do. Uploads gain the inference decode guard (oversized real images 400 before OOMing the trainer) and dataset upload/caption/delete/import are blocked with 409 while a diffusion run is active. JSONL readers (trainer + routes) tolerate non-object JSON and invalid UTF-8 instead of raising AttributeError / 500. LoRA family compatibility is enforced in the shared resolver, not only the picker, so a direct API client cannot apply a mismatched-family adapter. GPU arbiter gains release_if so the image/video unload idle-check and release are atomic against a concurrent same-owner load's registration. Native batch recipes persist the base batch_seed and restore replays from it, so a native batch_index>0 image no longer advances its seed twice. FLUX.2-klein selects its sd.cpp text encoder by variant (4B -> Qwen3-4B, 9B -> Qwen3-8B) instead of the single family default. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: track the loaded GGUF filename so native companion resolution reproduces the load identity * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: gate local-pipeline image tagging on a real family; validate video sidecars before delete/clear * Studio: tighten image-generation fix comments and docstrings * Studio: gate gallery serve/export on ownership; keep image progress active until persisted; reserve diffusion training before the dataset scan * Studio: restore Reapply target on async image/video load errors; recheck training state before dataset commit; reclaim partial sd.cpp installs on retry Images/Video: a background model load that fails AFTER starting (error/eviction during download) leaves the previous pipeline resident, but handleLoad had already overwritten lastLoad.current with the failed pick, so "Reapply to loaded model" reloaded the failed model. Carry the prior Reapply target into the poller and restore it on the async error/null paths, mirroring the quant rollback. Training: an in-flight diffusion dataset upload passed _require_diffusion_dataset_mutable() at entry but could still commit files after a concurrent /diffusion/start reserved the training slot, mutating the dataset underneath the trainer. Re-check the interlock immediately before the commit phase; a 409 there leaves the staged temps for the finally to clean. sd.cpp install: an interrupted extraction (disk full, killed process, a raising post-extract cudart fetch) left the target non-empty with no owner marker, so the next lazy install tripped the "not a Studio-managed directory" refusal and wedged native install. Write the ownership marker before the partial writes when the target is reclaimable, so a retry recognises the debris as ours and re-extracts. * fp8 DiT quant: floor the dynamic activation scale with activation_value_lb An all-zero activation token row makes the dynamic per-row fp8 scale 0, which turns the quantized data to NaN and the render to black frames on torchao's plain-torch kernel path. The fused fbgemm/mslk quantize kernels clamp zero rows internally, so the bug only reproduces on machines without them, which is most user environments. Zero rows are real inputs, not a corner case: Wan 2.2 zero-pads its text conditioning, and Hunyuan-1.5 and Qwen-Image regenerate zero rows inside their transformer blocks every step. Pass activation_value_lb=1e-12 to Float8DynamicActivationFloat8WeightConfig whenever the installed torchao supports the kwarg (Float8Tensor rework, 0.13+), checked via inspect.signature so older torchao keeps exactly the current behaviour; the existing Float8MMConfig fallback chain is unchanged. Verified on GPU: with the forced plain-torch kernel path a zero-row input NaNs without the floor and stays finite with it, and end to end on HunyuanVideo-1.5 fp8 goes from a solid black frame (LPIPS 1.00) to a normal render (LPIPS 0.225); on Wan the floor matches the condition_embedder exclusion (LPIPS 0.211 vs 0.206). Same-seed renders with fused kernels present are unaffected, and pre-quantized fp8 checkpoints stay valid since weight scales are untouched. * Wire hosted pre-quantized DiT checkpoints into the image families Point prequant_repos for flux.1, flux.2-klein, flux.2-dev, qwen-image (int8 only there; fp8 is family-denied), z-image and krea-2 at the unsloth/<Model>-FP8 Hub repos carrying gate-validated int8 and fp8 transformer checkpoints, so the fast quant path loads the small pre-quantized file instead of materialising the dense bf16 transformer and quantising on device. Measured on FLUX.2-dev int8: build peak drops from 60.7 GB (dense + quantize) to 30.7 GB (hosted prequant), identical 30.7 GB resident after either path since loading a checkpoint is bit-identical to on-the-fly quantisation. The hosted repos name files <Model>-<SCHEME>.pt, so resolve_prequant_source now derives that model-name filename from the repo id (scheme suffix stripped case-insensitively) and carries the legacy transformer_<scheme>.pt as a fallback the resolver tries when the primary 404s, keeping older repos loadable. Wiring a repo also exposed a fallback hazard: with a prequant source present, the dense-fit preflight used to be skipped entirely, so a failed prequant download would fall through to the dense bf16 load the memory plan never budgeted, OOMing after eviction. The preflight now always runs and gates an allow_dense_fallback flag through _load_dense_quant_pipeline: a dense misfit still skips the fast path when no prequant exists, but with one it proceeds and a prequant failure raises to the GGUF build instead of loading dense. The same flag is set when the auto-policy replans an offloaded GGUF against a prequant-sized transient. Tests updated to the new filename convention plus new coverage for the derivation and the legacy-name fallback; the prequant-skips-refit test now asserts the re-check runs and forbids the dense fallback. Verified end to end on GPU: z-image int8 resolves the hosted repo, downloads the model-name file and renders (6.8s load, 5.9 GB peak). * Route krea-2 through its per-component loader on the transformer-quant fast path _assemble_pipe used Pipeline.from_pretrained for every family, but the krea repo ships transformers-5.x configs and no top-level tokenizer files, so the tokenizer dies with vocab_file=None. The pre-quantized checkpoint loaded fine and then the assembly crashed, dropping the load to the GGUF build, which krea-2 cannot take (Krea2Transformer2DModel has no from_single_file). Assemble per-component via load_krea2_pipeline like the pipeline-kind and single-file paths already do. Verified live: Krea-2-Turbo int8 and fp8 hosted prequant loads now assemble and render through the Studio images tab. * Keep Qwen-Image's text-stream linears bf16 on int8 (short prompts break torch._int_mm) Qwen-Image's MMDiT runs every text-stream Linear at M = actual prompt tokens: the Qwen2.5-VL embeds are not padded to a fixed length like FLUX's 512-token T5. A short prompt (13 tokens) or the near-empty negative prompt drives torch._int_mm below its M > 16 floor and the first denoise step raises 'self.size(0) needs to be greater than 16, but got 13' (measured on B200 through the Studio images tab). Add per-family int8 exclusions (txt_in, add_q/k/v_proj, to_add_out, txt_mlp) for qwen-image and qwen-image-edit, threaded through exclude_tokens_for_scheme(scheme, family) and the prequant checkpoint validation, so a checkpoint baked under the old token list is rejected and re-quantised instead of loaded crashing. The text stream runs at M = tens vs the image stream's M ~ 4k, so the exclusion costs nothing; the rebuilt hosted checkpoint gates 28/28 PASS with LPIPS mean 0.057 (was 0.069). * Harden the diffusion memory plan against transient free-VRAM undercounts A cold FLUX.2-dev int8 load on an idle 183 GB B200 planned offload=model (companions exceed budget) and silently served the GGUF as-is; the identical retry went resident and engaged the hosted prequant. The plan arithmetic was byte-identical across both loads (required 90,228 MiB, resident needs free of about 124 GB); the only divergent input was torch.cuda.mem_get_info, which is device-wide and instantaneous: a transient foreign CUDA context briefly held about 100 GB at the first snapshot, and the planner trusted that single read. Three changes: - settled_snapshot_device_memory: on cuda, synchronize + empty_cache (best-effort) and take the MAX free over up to 3 spaced reads. A transient can only shrink free, so the max rejects transient undercounts while a persistent tenant still caps every read. _plan_memory now uses it. - plan_fits_total_capacity + one replan retry: when the dense/prequant candidate fits TOTAL device capacity under the standard reserve and the 0.85 resident margin, an offload verdict can only stem from the free reading, so the loader re-snapshots and replans once before declining the fast path. Explicit balanced/low_vram modes skip the retry (they offload by mode). - diffusion.transformer_quant_declined log line with required/budget/free and the plan reasons, so the next decline is diagnosable from the server log (previously silent). Verified: cold FLUX.2-dev int8 first load in a fresh server now engages the hosted prequant resident (offload=none). * Add FLUX.2 Klein and FLUX.2-dev DiT LoRA training Register flux.2-klein and flux.2-dev in the DiT trainer following the upstream DreamBooth references: latents train patchified and batch-norm normalized from the VAE posterior mode, the packed forward reuses step-invariant position ids, and the guidance vector (3.5) is gated on the variant's guidance_embeds config. Conditioning stacks load per variant (Mistral via Flux2Pipeline for dev, Qwen3 via Flux2KleinPipeline for Klein) and are encoded and freed before the transformer lands on the device. The fused single-stream to_qkv_mlp_proj joins the attention projections in the LoRA targets; the single-stream out projection stays dense because its to_out suffix would also match the double-stream ModuleList container. Wire both families through the training registry (family set, labels, VRAM notes, rank 16 / lr 1e-4 defaults, bf16-only preflight), mark them trainable with train base repos in the family registry, add FLUX.2-dev to the gated-repo token check, and trust both official bases for training downloads. Verified on B200: 30-step klein int8 (19.6s) and nf4 (20.9s) and dev int8 (52.0s) runs train with finite decreasing loss and the saved adapters apply on the bf16 base pipeline (weight 0 reproduces the base image exactly, weight 1 visibly restyles it). * Support LoRA adapters on torchao int8/fp8 quantized image pipelines Adapters are baked at load time: they attach to the dense transformer, then quantize_ converts only the frozen base linears (the lora_ side path is excluded by name), then the loader compiles. Post-quant PEFT injection is not possible on a manually quantized module, so the prequant shortcut is skipped for a baked load and the memory plan is sized for the dense build (force_dense on the quant candidate). At generation time the baked topology is frozen: weight tweaks and disabling (scale 0 reproduces the quantized base exactly) go through set_adapters, while adding or removing adapters returns a clean 400 telling the client to reload with the new selection. supports_lora now returns True for int8/fp8 diffusers loads (checked before the gguf-kind early return, since the quant fast path keeps the picker kind); nvfp4/mxfp8 and GGUF-via-diffusers stay blocked. The load request model takes an optional loras list, threaded through begin_load on both engines (native ignores it and keeps applying LoRA at generation). Verified end to end on GPU: Z-Image GGUF picker + int8 + trained adapter loads through the API, bake marker logged, weight 1.0 vs 0 renders differ visibly, weight 0.5 accepted live, unknown adapter rejected as 400. Affected suites: 296 passed. * Add FLUX.1 Krea dev to the image model catalog Krea's guidance-distilled FLUX.1-dev finetune keeps the exact dev layout, so it runs under the existing flux.1 family unchanged. Wire it up end to end: - Catalog group with the gated official bf16 pipeline and the open QuantStack GGUF quants; the gated artifact is skipped on auto-routing when undownloaded. - Trust the official repo for non-GGUF from_pretrained loads, next to the other black-forest-labs bases. - Generation defaults: 28 steps at guidance 4.5 per the model card. The generic "krea" defaults key (Krea-2-Turbo's 8-step no-CFG recipe) used to swallow the id, which would have produced garbage output; the new flux.1-krea key precedes it on both the backend table and the images page table. - The flux.1 prequant checkpoints are schnell-based; the loader's baked base_model_id validation refuses them for the Krea-dev base, so int8/fp8 requests dense-quantize instead (covered by existing prequant tests). * Resolve pre-quantized checkpoints per base variant One family entry covers several published variants whose weights differ (flux.1: schnell, dev, Krea-dev), but prequant resolution was keyed on (family, scheme) alone, so only the default base could ever be served: the loader's baked base_model_id validation correctly refused the schnell checkpoint for dev and Krea-dev bases and every such load paid the dense download plus on-the-fly quantise. Add an optional prequant_variant_repos table on DiffusionFamily as (base_repo, scheme, repo_id) triples and thread the resolved base repo through resolve_prequant_source / usable_prequant_source and their three call sites (load fast path, memory-plan probe, auto-policy candidate). A base without its own entry keeps returning the family default, preserving the existing refuse-then-dense behavior exactly. Wire the flux.1 variants: the gate-validated unsloth/FLUX.1-dev-FP8 checkpoints (built in the earlier campaign but never reachable) and the new unsloth/FLUX.1-Krea-dev-FP8. * Add the Lumina Image 2.0 family to the image catalog Alpha-VLLM/Lumina-Image-2.0 is a 2.6B single-stream DiT with a Gemma2-2B encoder and a standard 16-channel VAE, all transformers-4.x-compatible, so the generic from_pretrained pipeline path loads it as a new lumina-2 family: - Family entry (Lumina2Pipeline / Lumina2Transformer2DModel), aliased to lumina-image-2.0 / lumina-image-2 / lumina2. No bare lumina alias: Lumina-Next checkpoints are a different arch and must stay unknown rather than crash mid-load. bf16-only upstream, so the fp16 fallback stays off like z-image. - Trust the official repo for non-GGUF loads; bf16 component table entry (ships fp32, ~5.2 GB transformer + 5.2 GB encoder bf16-resident). - Generation defaults 50 steps / guidance 4.0 per the model card, and the generate call passes the card's cfg_trunc_ratio=0.25 itself (family-gated, signature-gated): the pipeline default (1.0) runs the CFG double-forward on every step and oversaturates output. - Catalog group with the single ungated bf16 pipeline artifact (11 GB resident) plus routing assertions; images page defaults row. - No GGUF artifact: none exists upstream (only finetune/LLM quants), so the dense transformer_quant fast path (GGUF-kind-only) stays unreachable for now. Offline probes of the future prequant campaign: int8 and fp8 both engage and render cleanly (fp8 LPIPS 0.11 vs bf16, int8 0.33 from 50-step trajectory drift with intact quality), so neither scheme is family-denied. * Wire the hosted Lumina Image 2.0 int8/fp8 checkpoints Gate-validated against same-seed bf16 renders (28/28 pairs per scheme, zero failures): int8 LPIPS mean 0.146 / SSIM 0.937, fp8 LPIPS mean 0.116 / SSIM 0.946. Uploaded to unsloth/Lumina-Image-2.0-FP8 following the existing checkpoint repo conventions. * Add the HunyuanImage 2.1 family to the image backend The hunyuanvideo-community diffusers mirror carries the full stack in standard layout: a 17B dual-stream DiT (32.5 GB bf16), a Qwen2.5-VL text encoder, a ByT5 glyph encoder, the 32x HunyuanImage VAE, and guider/ocr_guider components (AdaptiveProjectedMixGuidance) that diffusers 0.39 loads natively, so the generic from_pretrained pipeline path covers everything with no per-component assembly. Family notes: - The call's guidance knob is distilled_guidance_scale (there is no guidance_scale kwarg), so cfg_kwarg routes the UI value there; real CFG runs inside the repo's guider at its baked scale. Defaults follow the card recipe: 50 steps, 3.25. - 2K-native: verified live at both 1024 and 2048. - Coexists with the HunyuanImage-3.0 structured exclusion (3.0 has no diffusers pipeline and stays excluded with its stated reason). - int8/fp8 dense quantization verified live (LPIPS 0.186 both vs same-seed bf16); a short prompt does not trip the int8 torch._int_mm minimum on this arch, so no family exclude entry is needed. - bf16 component table for the memory planner: (32.5, 16.3, 0.8) GB. * Surface HunyuanImage 2.1 in the image model catalog Catalog group with the open bf16 mirror pipeline (~50 GB resident, so a bare click on a consumer card routes to the QuantStack GGUF quants, which load and render through the generic GGUF path, verified live) plus the images page defaults (50 steps, guidance 3.25 feeding distilled_guidance_scale). * Add the HiDream-I1 family to the image backend A 17B MoE DiT (16 double + 32 single layers, 4 routed experts) with four text encoders, on HiDreamImagePipeline (diffusers 0.39). One family covers the open Full / Dev / Fast repos (same arch); per-variant generation defaults follow the upstream inference recipes (Full 50 steps at guidance 5, the distilled Dev 28 and Fast 16 guidance-free). The repos name a Llama-3.1-8B text_encoder_4 in their model_index but do not ship its weights; the official example passes the gated meta-llama repo in by hand. The loader instead assembles the component from the open unsloth mirror (byte-identical weights, already inside the non-GGUF trust gate), injected at the three pipeline from_pretrained sites, with output_hidden_states matching the official example. Memory planning counts the assembled TE4: 34.2 GB DiT + 28.8 GB encoders, ~63 GB bf16-resident. * Surface HiDream I1 in the image model catalog One catalog group with the three official bf16 pipelines (Full, plus the Dev and Fast distillations as labeled artifacts) at their ~63 GB resident size, so auto-routing keeps this a datacenter-GPU pick. city96's GGUF is deliberately not wired: the GGUF path would need the same Llama TE4 assembly for very small demand. Images-page defaults mirror the backend table with the variant keys ahead of the generic hidream key. * Pin the measured HiDream quant verdict in tests int8 and fp8 both engage and render cleanly on this family, including short prompts on int8: the routed MoE expert Linears only ever see the concatenated image+text stream (M far above the torch._int_mm minimum), so no deny entry and no family exclude tokens are warranted. Assert that so a future table edit cannot silently regress the measured behavior. * Wire the hosted HunyuanImage 2.1 int8/fp8 checkpoints Verified bit-identical to on-the-fly quantize: all 1264 state dict tensors (456 quantized) dequantize equal between the loaded checkpoint and a fresh quantize_ pass, so quality matches the runtime Dtype path exactly. Same-seed LPIPS suite means (0.35 int8 / 0.28 fp8) blend trajectory divergence with this family's own run-to-run nondeterminism (identical weights and seed reproduce a 17/255 mean pixel delta through the 50-step guider pipeline); per-case hard checks pass and the drift is compositional, reviewed visually. Uploaded to unsloth/HunyuanImage-2.1-FP8. * Fix silent LoRA drop and wasted transformer prefetch on GGUF quant loads Two live-test findings on the images load path: - transformer_quant with baked LoRAs, when the dense quantized build is declined for memory or fails: the load completed as a plain GGUF with the adapters silently dropped (HTTP success, supports_lora=false after the fact) -- wrong output with no signal. The load now fails with the recovery options (drop the adapters, free VRAM, or pick a smaller model). Weight-0 adapters still count as no bake request, and the plain no-LoRA decline keeps its silent GGUF fallback. - A fresh GGUF load on a small GPU prefetched the base repo's full bf16 transformer shards (~47 GB on Qwen-Image) because the dense-quant prefetch widening only checked scheme viability, not whether the device could ever hold the candidate resident. Gate the widening on total device capacity (reserve + 0.85 margin, the plan_fits_total_capacity bar) so a card that is certain to decline the dense build never pays the download; capable devices keep the prefetch. * Fix video progress under-reporting during load and generate Two live-test findings on the video progress endpoints: - load-progress downloaded_bytes froze mid-download: the counter used scan_cache_dir, which skips in-flight *.incomplete blobs, so it sat at the last completed blob for the whole multi-GB shard pull while the disk kept filling. Count the repo's cache directory directly (completed plus incomplete blobs, snapshot symlinks skipped so nothing is double-counted). - generate-progress reported total_steps=null / fraction=0 while step advanced: the video API only carried the native total field while the image API exposes total_steps and fraction, so one poller could not work against both. Derive the image-compatible aliases in generate_progress and declare them on the response model; the native total stays for back-compat. * Wire the hosted HiDream I1 int8/fp8 checkpoints Gate-validated: all 28 per-case pairs pass per scheme (LPIPS suite means 0.291 int8 / 0.278 fp8, in the 50-step trajectory-divergence band; CLIP delta means 0.007-0.008), and the int8 checkpoint is verified bit-identical to on-the-fly quantize across all 1615 state dict tensors (1073 quantized, max abs diff 0.0). Uploaded to unsloth/HiDream-I1-Full-FP8. * Add a pre-cast text-encoder loader for the layerwise fp8 scheme The runtime text_encoder_quant=fp8 path downloads the full bf16 text encoder and layerwise-casts it in place on every fresh load. For the heavyweight encoders (LTX's Gemma3-27B ~50 GB, FLUX.2-dev's Mistral-24B ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) that download dominates load time on a fresh machine. diffusion_te_prequant.py loads a pre-cast fp8-storage state dict instead: meta-init the encoder skeleton from the checkpoint's te_class, load_state_dict(assign=True), rebuild on CPU if non-persistent buffers stay on meta, then re-apply the same layerwise cast to install the upcast hooks. The cast is a deterministic storage transform, so the loaded encoder is bit-identical to dense-load-then-cast by construction. v1 hosts the layerwise fp8 storage scheme only: its state dict is plain tensors (torch.load(weights_only=True), no pickle execution). The dynamic-compute schemes (fp8_dynamic, int8, nvfp4) build torchao subclass wrappers at runtime and are deliberately not hosted. Checkpoints validate format, scheme, component and base_model_id before use and any problem falls back to the dense download and cast. Local path overrides reuse the DiT prequant allowlist env var. Families opt in via a new te_prequant_repos (scheme, component, repo_id) field on both DiffusionFamily and VideoFamily; the field defaults empty so nothing changes until a gate-validated artifact is wired. * Inject hosted pre-cast text encoders during pipeline assembly Wire te_prequant_pipe_kwargs into the three pipeline assembly sites: the diffusion full-pipeline branch, the diffusion transformer-only and GGUF branch (where the companion TE is the big remaining download), and the shared video assembly path before the pipeline/component split. Injection is gated exactly like the runtime cast (mode normalized to fp8, device supported, family not denied), so it can never engage where quantize_text_encoders would not; the later quantize_text_encoders call re-applies the cast idempotently and keeps status reporting truthful. With no hosted checkpoint configured the call returns {} and assembly loads the dense encoder as before. * Add the pre-cast text-encoder checkpoint builder Applies the runtime layerwise fp8 storage cast to a model's dense text encoder once and saves the cast state dict with baked metadata (format tag, base_model_id, family, scheme, component, te_class, versions) in the layout diffusion_te_prequant.py validates. Resolves the encoder class from the checkpoint's config.architectures so the recorded te_class matches what the pipeline instantiates. CPU-runnable: the cast touches storage dtypes only. * Test the pre-cast text-encoder load path Hermetic CPU coverage for diffusion_te_prequant: the checkpoint filename convention, family-table resolution by scheme and component with malformed entries skipped, resolution priority (path override, hosted repo, none) and the fp8-only scheme gate, the checkpoint validation matrix (wrong format, missing state_dict, wrong scheme, wrong component, wrong or missing base_model_id) with base case folding, the local-path allowlist refusal and missing-file fallback, and the assembly injection gating (mode, hosted entry, device support, family deny, load failure, successful injection). Also pins the te_prequant_repos field on both family dataclasses and that no family ships a hosted TE checkpoint until the campaign wires one. * Fix pre-cast TE checkpoint loading and engagement reporting Two bugs found while building the hosted checkpoints: - The builder recorded torch.__version__ (a TorchVersion object) in the checkpoint metadata, so torch.load(weights_only=True) rejected every artifact and the loader silently fell back to the dense download. Record plain strings. - Re-applying the layerwise fp8 cast to an injected pre-cast encoder raised on the duplicate hook registration, making quantize_text_encoders report the engaged cast as failed (status showed no TE quant while the encoder ran fp8). _cast_fp8 now returns early when the hooks are already installed. Also corrects the LTX TE size note: Gemma3-12B stored fp32 (~49 GB), not 27B. * Wire the hosted pre-cast fp8 text encoders qwen-image and flux.2-dev (diffusion) and ltx-2 (video) now resolve a hosted pre-cast fp8 text encoder from their unsloth -FP8 repos: - unsloth/Qwen-Image-FP8: Qwen2.5-VL-7B, 16.6 GB dense -> 8.8 GB - unsloth/FLUX.2-dev-FP8: Mistral-Small-24B, 48.0 GB dense -> 24.7 GB - unsloth/LTX-2-FP8: Gemma3-12B, 48.7 GB fp32 store -> 13.2 GB Every checkpoint verified bit-identical to dense-load-then-cast (729 / 585 / 1066 tensors, zero mismatches) and smoke-tested through the real backends with the repo engagement marker. Tests cover the wired entries, the resolver filenames, builder metadata weights_only survival, and the idempotent re-cast. * Report the compute dtype on fp8-cast encoders and inject the pre-cast TE on the dense fast path Two more findings from the hosted-TE GPU smokes: - Module.dtype reports the first floating parameter, which after the layerwise fp8 cast is the fp8 STORAGE dtype. Flux2 derives its prompt embed and latent dtypes from encoder.dtype and feeds them to randn_tensor, which has no fp8 kernel, so ANY flux.2 load with text_encoder_quant=fp8 crashed at generation (pre-existing, runtime cast included). The cast now swaps in a subclass whose dtype property reports the compute dtype; forward behaviour is unchanged. - The dense transformer_quant fast path assembles companions through _assemble_pipe, which never received the pre-cast TE injection, so the hosted encoder engaged on full-pipeline and GGUF builds but not on the fast path. Threaded through like the other two branches. Verified live on B200: qwen-image (full pipeline), flux.2-dev (GGUF picker with int8 DiT prequant), ltx-2 (video backend) all engage the hosted TE, render non-black, and report text_encoder_quant=fp8 truthfully. * Key the fp8 cast idempotency on an explicit completion marker Hook presence alone cannot distinguish a legitimately pre-cast text encoder from leftover hooks after a cast that failed mid-pass, so the early return now requires the completion marker _cast_fp8 sets once the hooks are fully installed. Leftover partial state keeps failing closed. Also tolerates non-Module encoder doubles in the hook probe and the dtype override. * Extend the fp8 TE quant to HiDream's Llama text_encoder_4 The generic quantize_text_encoders pass only covers text_encoder.._3, so HiDream's HEAVIEST encoder (Llama-3.1-8B TE4, 16.1 GB bf16) always stayed dense. TE4 is assembled separately (hidream_te4_kwargs), so the fp8 path now lives there: when the requested TE quant is layerwise fp8 and the device/family qualify, TE4 prefers the hosted pre-cast checkpoint (unsloth/HiDream-I1-Full-FP8, 8.6 GB) and falls back to dense-load-then- cast; a mid-pass cast failure reloads a fresh dense encoder instead of shipping partial state. The pre-cast loader and builder gain config_subfolder/config_overrides for standalone encoder repos whose config sits at the root and whose pipeline needs forward flags (output_hidden_states/attentions). Verified on B200: bit-identity 291 tensors (225 fp8, 0 mismatches), hosted checkpoint engages through the real backend (marker + status fp8), load 24.3 s vs 48.0 s dense, LPIPS 0.133 mean over 3 same-seed pairs vs the dense-TE render (gate 0.25), non-black frames. * Correct the ltx-2 resident TE estimate to the bf16 cast size The memory plan's bf16_components_gb held 50.4 GB for the LTX text encoder, which is the fp32 hub store of Gemma3-12B (~49 GB download), not what sits on device: the pipeline loads it torch_dtype=bf16, ~24.4 GB resident. The 26 GB over-estimate pushed the auto plan toward offload on cards that fit the real footprint. Comments and the size-table test now pin the resident semantics. * Host pre-cast fp8 text encoders for four more families Round 2 of the hosted TE set, each bit-identical to dense-load-then-cast and gated through the real backend (marker + status fp8 + same-seed LPIPS vs dense TEs): - FLUX.1 T5-XXL (text_encoder_2): 9.52 -> 5.90 GB, one artifact for schnell/dev/Krea-dev (T5 shards byte-identical across all three, verified sha256). 220 tensors, 144 fp8, LPIPS 0.109. - Lumina Gemma2-2B: fp32 hub store 10.46 -> 3.20 GB (3.3x download cut). 288 tensors, 182 fp8, LPIPS 0.041. - Z-Image Qwen3-4B: 8.04 -> 4.41 GB. 399 tensors, 252 fp8, LPIPS 0.112. NOT shared with flux.2-klein-4B: klein retrained layer 35's MLP (verified tensor diff, maxdiff 0.86), so klein hosts no entry. - Krea-2 Qwen3-VL-4B: 8.88 -> 4.83 GB. 713 tensors, 460 fp8, LPIPS 0.082. The constructor-assembled krea pipeline takes the encoder directly (load_krea2_pipeline text_encoder kwarg); the loader remaps 5.x rope_parameters and re-ties weights after assign so the rebuilt encoder matches the builder's structure. HunyuanImage 2.1 reuses the Qwen-Image artifact outright: its Qwen2.5-VL text encoder is byte-identical (every shard sha256, 16,584,414,544 bytes), recorded in the new component-level base-equivalence table the checkpoint validator consults. The injection loop now covers text_encoder.._3 so a family can host several components. Live check: LPIPS 0.123 vs dense. * Report the fp8-cast compute dtype without swapping the encoder class The dtype override swapped encoder.__class__ to a dynamic subclass, which breaks transformers' kwargs-based output recording: a fp8-cast Qwen3VLModel stopped returning hidden_states and every krea-2 generation with text_encoder_quant=fp8 crashed at encode_prompt (regression from the HiDream TE4 change; caught by the krea hosted-TE live smoke). The override is now a property shadowed on the ORIGINAL class that prefers a per-instance compute-dtype attribute, so class identity is preserved and uncast instances keep the stock behaviour. The idempotency test now pins exact class identity and the uncast-sibling fallback. * Pass the calibrated distilled sigma curve to LTX-2.3 8-step runs The 22B distilled DiT was trained against ltx_core's fixed DISTILLED_SIGMA_VALUES, but the diffusers scheduler derives 8-step spacing from resolution-shifted flow matching and lands far off at every reachable mu (second sigma 0.945-0.981 vs 0.99375, tail 0.37-0.61 -> 0.1 vs 0.725 -> 0.42 -> 0). At the distilled default step count the backend now passes the list verbatim, neutralising the scheduler's dynamic shift and terminal stretch for the call (they distort even explicit sigmas) and restoring them afterwards. Other step counts and the dev/base DiT keep the scheduler's own spacing. Live-verified on B200 through the video branch backend: the scheduler holds the exact curve after an 8-step distilled GGUF generation, config restored, healthy clip. Also reword the transformer_quant resolved reason to the measured reality: quant halves resident weights and hosted checkpoints cut load time, while per-step speed is roughly bf16 parity. * Pin the fp8 weight-quantize kernel against silent MSLK switching torchao's Float8Tensor KernelPreference defaults to AUTO, which switches the weight-quantize kernel to MSLK whenever an mslk package is importable on sm90+. Measured on B200: that changes fp8 scale rounding bitwise (8/8 FLUX matrices differ, scales ~55 percent of bytes), so a box that merely gains mslk would break the hosted-prequant bit-identity invariant; the mslk path is also slower under torch.compile (opaque extern call blocks inductor's quantize fusion, FLUX.1 fp8 e2e 1.149 to 1.624 s). Pin KernelPreference.TORCH explicitly, matching current no-mslk behaviour bit for bit; signature-gated for older torchao. GPU-smoked (finite, rel err 0.037) and pinned by test. * Shift Qwen-Image training sigmas to the inference distribution Qwen-Image's scheduler skips its static shift under use_dynamic_shifting, so the DiT trainer was drawing UNSHIFTED uniform-schedule sigmas for it (mean sigma 0.50) while inference always runs the exponential mu = log 3 shift plus the shift_terminal 0.02 stretch. Add a flow_shift config lever: "auto" (the new qwen-image default) rebuilds the training sigma table through the scheduler's own time_shift and stretch_shift_to_terminal so the draw matches the inference distribution exactly (mean sigma 0.72); a numeric value applies the standard linear shift s*u/(1+(s-1)*u); 1.0 keeps the historical identity behavior and stays the default for FLUX, Z-Image and Krea 2. The model timestep conditioning follows the shifted sigma, gathered in fp32 so bf16 rounding never skews it. Also wire two opt-in levers with off defaults: cfg_dropout (per-sample empty-prompt conditioning dropout, encoded alongside the captions before the text encoders are freed) and weighting_scheme="bell" (bsmntw-style mid-schedule Gaussian loss weighting normalized to mean 1). Verified with two 80-step rank-8 bf16 LoRA runs on Qwen/Qwen-Image (identity vs auto, same seed): both converge with finite decreasing loss and produce coherent same-seed previews. Unit tests cover the exact transform, the shifted sampling distribution, per-family defaults and config plumbing. * Add LoRA EMA, a persistent conditioning cache, and aspect bucketing helpers diffusion_train_extras hosts the opt-in training extras: LoRAEMA shadows only the trainable adapter params (warmup-ramped decay, default 0.99, exported as a second adapter under output_dir/ema), PersistentConditioningCache stores latent posterior stats and caption embeddings as safetensors keyed by content hash + family + resolution, and the aspect-ratio bucketing helpers group mixed-aspect datasets into same-area divisor-snapped shapes. The DiT trainer wires the first two behind config flags that default to the current behavior: ema_decay (0 disables) and cond_cache_dir (None disables). A fully warm cache skips loading the VAE and text encoders entirely; a cache hit is bit-identical to a fresh encode, including the per-channel qwen latent normalization. Also fixes the stale _gather_sigmas call in the perf test that still passed the scheduler instead of the sigma table. * Tighten torchao configs and note the FSDP2 design for the DiT trainer nf4 loads now enable double quantization (~0.4 bits/param off the frozen base scales at no fidelity cost), fp8 training uses the rowwise recipe when the torchao build ships it (per-row scaling confines the DiT activation outliers that a tensor-wide scale collapses), and the inference quant filter gains a per-scheme GEMM-tiling divisibility floor (16 for scaled_mm, 32 for MX blocks) so one ragged Linear cannot crash the first denoise after a clean quantize pass. plans/fsdp2_diffusion_design.md records the multi-GPU design: bf16/fp8 over FSDP2 with per-block units, LoRA attached before sharding, int8 out of scope (DTensor over the quantized subclass is undefined), per-family notes. * Batch diffusion inference with per-image seeds, an inference conditioning cache, and GGUF loader fixes Batched generation: /images/generate takes a prompts list (one image per prompt, txt2img only) or a seeds list (one prompt, one image per seed); the legacy batch_size path derives per-image seeds base..base+n-1 like the native engine. Every image gets its own torch.Generator so any batch member replays alone from its gallery recipe; the whole list runs as one forward by default with OOM backoff that halves a failed chunk, and an explicit batch_size caps images per forward. Validated 10-22x over serial engines on 32-image suites with LPIPS deltas within 0.002. Conditioning cache on the inference path: UNSLOTH_DIFFUSION_COND_CACHE_DIR (the inference sibling of the trainers' cond_cache_dir, same persistent store) wraps encode_prompt so repeated prompts skip the text-encoder forward entirely; verified bit-identical outputs. Bypassed while LoRA adapters are attached; tensor-argument calls pass through uncached. Compile cache: GGUF loads fingerprint their own bundles (quant=gguf, a different compiled graph than the dense family) and batched calls register every distinct (w, h, batch) chunk shape they ran, so the heavy GGUF batched warmups (~159 s at batch 32 on 12B-class, ~655 s on 20B CFG-batched) are paid once ever. GGUF loader: strip the sd.cpp model.diffusion_model. container prefix in the single-file converter; diffusers' FLUX.2 converter KeyErrors on it and the Qwen-Image identity mapping strands the model on meta. * Correct batched seed-replay docs to match measured behavior Same-seed images at the same batch shape are bit-identical; a solo regeneration with the recorded seed matches its batched rendition up to batch-size-dependent kernel numerics (mean abs pixel delta about 2.5/255, LPIPS delta under 0.002), not bit-exactly. The previous wording overclaimed bit-identity across batch shapes. * Note that batched bit-identity assumes a settled compiled graph The first generation issued while the deferred compile is still in flight can deviate transiently (observed once on a cold fp8 build: mean abs pixel delta 0.063/255); once the graph is settled, same-seed same-batch-shape images are bit-identical across runs. * Studio sidebar: Image03/FlimSlate icons, More flyout, Train row, New pills - Images uses Image03Icon and Video uses FlimSlateIcon. - New "More" row (MoreHorizontalIcon) opens a right-side flyout on click or hover holding Video, Recipes and Export; the close is delayed 180ms so the pointer can cross the gap. Its SidebarMenuButton deliberately takes `title` rather than `tooltip`: with `tooltip` the button returns a Tooltip root and DropdownMenuTrigger asChild would hand its ref to a non-DOM node. - Dropped the "Train" section heading; Train is now a top-level row between Images and More. data-tour="navbar" moves to the surviving nav group so the product tour keeps its anchor. - "New" pill beside Images and (inside the flyout) Video, via NavBadge. * Studio sidebar: match flyout rows and New pills to the existing scales - More flyout rows dropped their sidebar-row typography and size-icon override, which fought DropdownMenuItem's own scale (text-sm, gap-2.5, px-3 py-2 and size-4 icons) and rendered oversized glyphs and text next to the nav. - New pill reuses the brand "beta" badge recipe (nav-badge font, --ui-font-scale sizing, nav token colours) rather than hardcoded 9px values. - The More row's native title tooltip (an OS box on hover) is replaced by the app's Tooltip, wrapped around DropdownMenuTrigger so both triggers compose onto the same button, and shown only on the collapsed rail like other nav rows. * Settings: pin and reorder the sidebar navigation Adds a "Sidebar navigation" section to Settings -> Appearance, above the existing profile-menu customizer, with the same drag-to-reorder + switch UI. - New sidebarNav preference: one { id, pinned } entry per navigable row (projects, hub, images, train, video, recipes, export), array order = render order. Defaults match the shipped layout, so an untouched install is unchanged. - Unpinning moves a row into the More flyout rather than hiding it, so no page becomes unreachable. New chat and Search stay fixed as actions. - app-sidebar now renders from one navRows descriptor map, so a pinned row and its flyout counterpart cannot drift; the More row appears only when something is unpinned and highlights off whatever it actually holds. - Mirrored in the backend PersonalizationCustomization: without it the model's extra="ignore" would drop the field, and because sync replaces local state with the server's copy once customization is saved, the user's pin order would reset on the next sync. The validator dedupes and back-fills like sidebarMenu but preserves the client's order, since here order is meaningful. Frontend typecheck, i18n parity and catalog checks pass; 32 personalization tests pass, including a round-trip asserting a reordered list survives a save. * Sidebar customizer: drop the Search row, skip More for a lone item - Search is reached from the top bar, so it is no longer previewed as a fixed sidebar nav row; New chat stays. - More now appears only when it would hold two or more rows. A single unpinned row renders inline in its saved order position instead: a flyout wrapping one item costs a click and earns nothing. The customizer's More preview follows the same threshold. * Sidebar settings: hide a lone unpinned tab, match New chat icon, rename Profile menu - With exactly one tab unpinned, both More and that tab are dropped, so nothing is drawn for it (previously it rendered inline). The page stays reachable by URL. - The customizer's New chat preview uses PencilEdit02Icon, the icon the real row renders; Edit03Icon was a different glyph. - "Sidebar menu" is now "Profile menu", described as the shortcuts behind your name at the bottom of the sidebar, so it no longer reads as a second name for the navigation section above it. * Tighten comments in the new sidebar and delete-guard code * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio sidebar: keep the More row highlighted while its panel is open Moving the pointer into the flyout left the row unhighlighted while the panel stayed open. The row now carries data-menu-open, added to the nav hover selector list. Not data-state: the tooltip and menu triggers both write that attribute, so whichever lands last wins. * Images: use the shared pill toggle for Create/Train and pad the panels - Create/Train was the only segmented control on its own Tabs styling. It now uses PillTabs, the same control as the model picker and Hub toggles, pinned to the header row's 34px. PillTabs takes an icon per tab, so the inline-span workaround for TabsTrigger goes away. - pt-3 on both the Create and Train panels, which sat flush against the model selector row. * Images: make the workflow picker a dropdown instead of a 7-up strip Seven workflows in a 340px rail left ~48px each, so the labels crowded and the hints were only reachable as title tooltips. The strip is now a dropdown: the trigger shows the current workflow and its hint, and each row carries its own description. A row the loaded model can't run is disabled and shows the reason in place of the hint, so the gating explains itself. Adding a workflow no longer shrinks the others. * Images: workflow icons, hint under the trigger, more top room, unclipped Train cards - Each workflow carries an icon, shown on the closed trigger and on every row. - The trigger is one line (icon plus name). The selected workflow's description moved below it, where it reads like the Field hints further down the rail. - pt-6 instead of pt-3 on both Create and Train, so the cards clear the model selector row. - The Train right column scrolls while its cards use ring-1, which draws outside the box and was clipped at the scroll edges. p-px gives the ring room. * Images: one-line workflow rows, stronger trigger fill, roomier mode tabs - Dropdown rows are icon plus name only. The selected row's description already shows under the trigger, and a disabled row keeps its reason as a title. - Trigger fill moves to the bg-foreground/[0.07] dark:bg-foreground/[0.12] pair the hub cards use, so it reads against the card in both modes. - Description under the trigger goes from text-ui-10 to text-ui-11p5. - More horizontal padding on the Create / Train tabs. * Images: drop card borders for the composer shadow, keep scrollbars inside, use app controls in Train - Cards lose ring-1 for .panel-soft-surface: the composer's shadow in light, flat in dark, matching .chat-composer-surface and the menus. - Both rails now clip (overflow-hidden) with the scroller inside, so the scrollbar can't ride over the rounded corner. Same shape video-page already uses. - Train's 9 native selects become the app Select, so they no longer open an OS-native menu, and the native file input is hidden behind a Choose images button that reports the count. - Image previews use explicit 8-10px radii: this theme sets --radius to 1.1rem, so rounded-md was 15.6px and the thumbnails read as circles. * Images: one card for controls and preview, chat sliders, wider softer shadow - Controls and preview were two floating cards; they now share one card split by a divider. The Advanced dock stays separate since it toggles. - SliderField wraps Chat's ParamSlider, so the sliders match Chat (label row with the value, full-width neutral track) instead of a green track with a spin box. All 14 call sites keep their props. - panel-soft-surface goes from 0 2px 8px -2px /0.16 to 0 4px 22px -6px /0.10: lighter, spread wider. * Images: flat Create and Train panes, hover-only scrollbars, tidier Train dataset step Both Images tabs now sit on the page background like the Hub: no card, no shadow, no bounding box. A single rule divides the controls rail from the preview canvas (Create) and from the run area (Train), and the settings and previous-runs sections read as panes rather than nested cards. Also: - Scrollbars in these panes use the existing hover-scrollbar recipe, so the thumb only shows while the pane is hovered. - Workflow rows explain themselves with a tooltip after a short hover, which also works on disabled rows, and the descriptions are much shorter. - Training images rows are name plus image count; the license stays on the example card. - The upload step loses its dashed box, the buttons match the sizes around them, and Upload only appears once files are picked. - The empty preview uses the same icon as the Images nav item. * Images: full-height panes, wider settings rail, Create/Train offset from the selector The rule between the panes now runs the whole page height (the row drops its bottom padding and each pane pads its own content), the settings rail is wider on both Create and Train, and the Create/Train switch sits further right of the model selector. * Images: put both tabs on the Hub's centered measure Top bar and content now share mx-auto max-w-1100 with px-5 / sm:px-8, so Create and Train sit at the same width and position as the Hub instead of running edge to edge. * Images: restore the top bar position, drop the panes lower under it * Images: center the mode switch, flip the arrow with the orientation, app tooltips everywhere The Create/Train switch is centered on the page instead of trailing the model selector, with wider buttons. The flip control's arrows now rotate with the orientation and its label says which way the flip goes. Every native title tooltip on the page is now the app's tooltip, so they all get the rounded surface instead of the OS box. * Images Train: plainer field text, no green buttons, columns that stop colliding - The dataset name, trigger prompt, adapter name and custom base fields now say what they are in plain words instead of leaning on example values. - Import, Upload, Back, Back to settings and Train another are outline buttons, not green ones. - Example thumbnails are landscape tiles, so photos are not cropped to chunky squares. - Settings cells get min-w-0 and the select value truncates, so a long option like the nf4 label no longer widens its column into the next one. - The number stepper sits a little further in from the field edge. - Create and Train are wider. * Images Train: roomier example cards with Import on the thumbnail row * Video: same treatment as the Images tabs - No cards: the rail and the canvas sit on the page background, divided by a rule that runs the full page height, on the Hub's centered measure. - Wider rail, chat's sliders, hover-only scrollbars. - Every native title tooltip is now the app's tooltip, including the clip cards. - Reapply and Cancel are outline buttons, the empty state uses the Video nav icon, and the clip tiles are less rounded. * Images and Video: narrower generation rail, matching Train headings Create and Video rails go from 392px to 368px. Train a LoRA and Training settings are now the same size and both in the heading font: the h2 already picks it up from the base rule, so the settings header opts in with font-heading and the weight that rule pins. * Images Train: shorter copy throughout Family notes, example descriptions, precision labels and every helper line are trimmed so they stop wrapping to three lines and colliding with the next column. The nf4 label now fits its select without truncating. * Images Train: a little more spacing between field groups * Images and Video: tighten code comments * Fix training start NameError, the load-order guard test and CPU-only diffusion tests - start_training forwards resume_source_run_id to _start_training_impl, which reads it. Without it every start raised NameError. - Restore main's anchor in the load-marker order test: the file now has an earlier `if config.is_gguf:`, so indexing the first one compared the wrong branch. - The two diffusion tests that reach diffusers now skip when it is absent, matching the CPU repo-test env. - The UI smoke finds nav rows that live in the sidebar's More flyout. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Treat a null metadata caption as no caption str(None) stored the literal "None" as the caption, so a null row counted as captioned and would have trained on that text. Also drops an unused import. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix invalid-UTF-8 500s, the flat Canny map and the dropped DiT knobs read_text raises UnicodeDecodeError, which is not an OSError, so one bad caption sidecar or video sidecar 500d the info, upload and gallery routes. A flat image now yields the all-black edge map instead of its own luminance, and the four DiT loss knobs the trainer implements are declared so model_dump keeps them. * Use the ui font-size tokens instead of raw px text utilities text-[11px] and friends ignore the UI font size preference, which the repo's font-scale contract test enforces. Same rendered size at the default scale. * Fix diffusion dataset 500s, the dropout-1.0 no-op run and the reset base pick Four correctness fixes on the training side: - The labeling grid read caption sidecars under except OSError, but a non-UTF-8 sidecar raises UnicodeDecodeError (a ValueError), so one bad file 500d /diffusion/dataset/{name}/images and the grid could not be opened to repair it. Read it as no caption, matching the info summary. - An image past Pillow's own hard limit raises DecompressionBombError, which derives straight from Exception and so escaped the upload guard's (OSError, UnidentifiedImageError, ValueError) and returned 500 instead of the intended 400. - lora_dropout accepted 1.0, which makes PEFT build nn.Dropout(p=1.0): lora_A and lora_B receive no gradient and the run saves an untrained adapter while reporting normal progress. Bound it below 1.0, matching the LLM request schema. - The train panel re-seeded the base repo on every dataset refresh because the family object identity changes on each info fetch, so an upload or caption save silently replaced the user's chosen base and the run started on a different model. Track the pick and only re-seed on a real family change. * Show the retained failure when a video page mounts after a failed job Mount-time recovery handled only phase=completed, so reloading the page after a multi-minute generation failed left an idle view with no diagnosis: the backend keeps the terminal failed record only until the next job, and nothing else survives the reload. Surface it the same way the poll does, filtering the cancelled sentinel. * Fix batched generation crashes, cache keying and unreplayable recipes Four bugs in the batched inference path, all found by review: - A mixed-prompt batch sent a scalar negative prompt against a prompt list. Z-Image asserts on the length, and Qwen-Image, Krea 2 and FLUX true-CFG encode a batch-1 negative against batch-N latents and fail in the transformer's text/image concat. Broadcast it to match the batch. - The FBCache step-cache reset sat above the chunk loop. diffusers only resets that state at the end of a successful call, so a forward that raised (the OOM the backoff is meant to recover) left its own residual behind and the halved retry died on a shape mismatch. Reset before every forward instead. - The conditioning cache keyed on the checkpoint alone, but a GGUF or single-file load takes its text encoders from the companion base, so the same checkpoint against a different base reused the previous base's embeddings. Key the base too. - Gallery records stored the base seed and the requested batch size even when a prompts/seeds list drove the run, so restoring the second image of seeds=[5, 99] replayed seed 5. List-driven outputs now record as single-image recipes on their own seed. Also bound strength above 0: every img2img pipeline derives its step count from it, so 0 leaves zero denoising steps and either raises or, on SDXL, crashes on empty latents. * Fix quantized-load LoRA bake, prequant family exclusions and outpaint canvas Six review findings across the Images page and model scanning: - The quantized (int8/fp8) load path can only attach LoRA adapters before quantization, but the frontend load request had no loras field, so every generation after such a load was rejected and each reload repeated it. Send the selection with the load. - build_prequant_checkpoint passed no family to the scheme exclusions while recording the family in metadata, so a Qwen int8 artifact baked the short-M text-stream linears and was then rejected wholesale by the loader's family-keyed check. - Registering a bare single-file checkpoint directory produced no On Device row even though the images loader can load it; only its parent worked. Admit that shape when nothing else matched. - Unload left the Reapply target set, so the repair path was skipped and Reapply reloaded the ejected model. Clear it, as the video page does. - Both FLUX.2 bases were trusted for training but not inference, so Deploy to Create rejected every FLUX.2 adapter. - Outpaint allocated the grown canvas before downscaling, exceeding the browser canvas area cap on a large photo; an over-cap canvas is unusable, so Extend silently posted a fully transparent image and mask. Scale the source first. * Send the picked GGUF filename with the quant so diffusion loads fire The variant expander emitted only the quant label, and nothing else in the frontend set ggufFilename, so the Images and Video pages could never take their GGUF branch: both gate it on meta.ggufVariant and meta.ggufFilename, then fall through to the single-file path, which returns because the id is a repo id and not a .gguf name. Every quant pick was a silent dead click, with no load request reaching the backend. The filename was already on the variant row (the picker keys its list on it, and the variant validator requires a non-empty string), so thread it through the click handler. The chat path is unaffected: it reads ggufVariant and never needed the filename. * Version the conditioning cache key and reject non-finite flow_shift Two correctness fixes: - The cache keyed the checkpoint and its companion base by name only, so a Hub repo advancing to a new commit, or a local directory updated in place, kept returning embeddings from the previous text encoder. Pair both with a revision marker: the locally resolved commit sha for a Hub repo, config plus text-encoder file stats for a directory. Neither loads the encoders, so a warm run still keeps them off the GPU. - flow_shift only checked positivity, but JSON accepts 1e309, which floats to inf, and inf <= 0 is False while NaN fails every comparison. The sigma table then evaluates s * u / (1 + (s - 1) * u) as NaN, which poisons every sampled sigma and saves a corrupted adapter while progress looks normal. Require a finite value. * Keep curated models listed, guard the video companion repo, pin diffusers Three review findings: - The picker filtered every catalog member out of Recommended and Hub search on the way to canonical group rows, but nothing renders those rows yet (catalogGroupFitsDevice and groupMatchesQuery are imported and unused). A task-scoped picker's models list is catalogToModelOptions(), i.e. group members exclusively, so both lists came back empty and no curated model could be discovered or downloaded. Keep the artifacts listed until the grouped UI exists. - The video delete guard compared only repo_id, so deleting the companion base of a loaded GGUF video model was allowed even though it supplies the VAE and text encoders. Compare base_repo too, matching what the images guard already does for its companions. - diffusers was declared unversioned while the diffusion stack requires 0.39 (Krea2Pipeline, the cache_context child registries, the Flux2 and Z-Image pipelines), so an upgrade could keep an older release and selecting an advertised model failed until the user upgraded by hand. * Namespace the trainer conditioning cache per checkpoint, bound the learning rate - The trainer keyed its persistent conditioning cache on family and resolution only, while the keys themselves carry just the caption or image content and crop variant. One cache directory reused for two checkpoints, or for the same repo at a new revision, let a warm run skip loading its encoders and train on the other model's embeddings and latent statistics. Namespace on the base checkpoint and its resolved revision as well. The revision helper now lives beside the cache in diffusion_train_extras and the inference wrapper delegates to it, so the two cannot disagree about what counts as the same source. - The diffusion learning rate only checked positivity, but 1e309 floats to inf and satisfies gt, so the route evicted the resident models and started AdamW with an infinite rate: the first step destroys the adapter while progress looks normal and the result is saved. Bound it below 1.0, matching the LLM schema, which rejects inf for the same reason. * Fix GGUF image model picks doing nothing, and pick the train base in the top bar The quant rows never forwarded the .gguf filename, so every hub GGUF pick on Images/Video fell through to a silent return. On Train the top bar now picks the training base instead of a generation model, which is GGUF-only and untrainable. * Pin diffusion and video loads to the live HF cache root Both read huggingface_hub's import-time HF_HUB_CACHE, which changing the cache folder does not update: progress counted the old root while the download wrote to the new one, and from_pretrained could split one model across both. * Add the diffusion download plan endpoint Reports the repos and exact files a pick needs so the download manager can stage them with the loader's own file scope. A plain snapshot would add the packaged root single, transformer shards and fp16 twins the loader never opens. * Add a file-scoped flavour to the Hub download job Lets a consumer that reads a deliberate subset of a repo stage it through the normal download manager. Keyed as "@scope" so it never collides with a quant or with the repo's full snapshot, and the file list rides the registry so an XET to HTTP retry respawns the same scoped job. * Stage image and video downloads through the Hub download manager They downloaded inline inside the load, so they had none of the manager's disk preflight, manifest verification, resume or panel progress. Picks now stage as scoped jobs carrying the loader's own file list, then load from a warm cache. * Fetch staged GGUF checkpoints as scoped jobs, and stop calling diffusion models unsupported A GGUF entry went out as a full snapshot, whose ignore list drops *.gguf: the job finished at once having fetched only docs, and the repo landed on device unloadable. Every entry is scoped now. The Hub also no longer tags image/video models as unsupported (they run on their own pages), and those pickers name what they select. * Apply the picker task filter to local model sections LM Studio, ./models and custom-folder rows ignored it, so the Images picker listed chat GGUFs that 400 on a diffusion load. The backend already tags every local model with a task for this purpose. * Route a chat pick of a diffusion model to the Images or Video page Chat cannot load one, so it was either hidden or failed on load. The unfiltered picker now lists on-device diffusion models and navigates to the page that runs them, passing the repo and quant so that page loads it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Route the real GGUF filename, keep non-GGUF curated models, key scoped downloads by file set Five review findings, four of them ways a click did nothing or fetched the wrong thing: - A chat pick of a diffusion model routed ggufVariant (a label like Q4_K_M) in the search param the target page uses verbatim as the GGUF filename, so the load asked for a file that does not exist. Route ggufFilename; no filename means a curated non-GGUF pick, loaded as a pipeline. - The task-scoped pickers kept only GGUF repos, so the catalog's bf16, bnb-4bit and single-file fp8 artifacts could not be discovered or downloaded on the Images and Video pages even though loadSpecFor knows how to load them. Keep curated artifacts whatever their format, in Recommended and in Hub search. - Both pages deduplicated routed selections on the model alone, and they now stay mounted, so picking the same repo again -- another quant, or the same one after chat evicted it -- returned early without loading or clearing the query string. Key on model and quant. - Every scoped image download shared one @diffusion job key regardless of the requested files, so switching quant mid-download adopted the running job: the UI waited on the first file set, then loaded a file that was never fetched. Include a digest of the file set in the key. - A scoped plan silently dropped requested files missing from Hub metadata, and snapshot_download succeeds when an allow pattern matches nothing, so the job reported completion and triggered a load with required files absent. Fail the job instead. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the scoped download key derivable, and stop the hidden page hijacking a route Four review findings, the first a regression from my own last commit: - Keying scoped download jobs by a digest of the file set broke the download manager: it builds that key client-side (it polls and cancels before any response tells it a key), so it watched and cancelled a key no worker owned and never fired its ready callback. Keep the derivable "@scope" key and refuse the second request instead when a live job on the slot is fetching a different file set -- decided inside the registry claim, under the lock, so a concurrent claim cannot slip past it. The manager records the file set on the job as well, so a sibling quant's transfer is not adopted locally either. - Both diffusion pages read the route query through a loose useSearch and both stay mounted once visited, so the hidden one consumed the other's ?model=: it navigated back to its own route and tried to load, say, an image checkpoint as a video model. Only the visible page consumes it. - The staged download plan was built without the configured HF token or the Advanced values the load itself sends. The token matters most: the backend's Hub metadata lookup is best-effort, so a gated base silently planned no companion entry and the load pulled those multi-GB files inline, outside the manager. The memory/quant controls decide whether the base transformer/ shards are needed at all, and the route dropped memory_mode, cpu_offload, the prequant path and the LoRA selection before asking for the plan. - The video preview kept playing after leaving the page: the keep-alive layout only hides it, and display:none does not pause a media element, so a clip the user unmuted kept its audio going over the next page. Pause on the active transition and do not auto-replay while hidden. Also completes the hand-built request bodies in the hub download tests: the scoped-files field this branch added to the route read as an AttributeError against them, failing five tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Serialize the GPU handoffs, gate DiT training on a GPU, and keep 3.9 installable Six review findings, three of them evict-then-fail orderings: - The chat load reclaimed the GPU without telling the arbiter it existed. A chat load holds no llama-server process until its GGUF has downloaded, which is minutes, so a competing Images/Video acquire in that window found nothing to cancel, took the GPU, and the chat load then spawned onto the same device. It now registers an in-flight marker through acquire_for's register hook (under the arbiter lock, as the image and video loads do), the evictor cancels a marked load, and the route undoes itself if ownership moved while it loaded. - The Hub-download conflict check ran after that handoff, so a GGUF the download manager already owns destroyed the resident Images/Video pipeline and then 409'd, having loaded nothing. It moves above the handoff, together with the marker it handshakes with. - The image load released the engine router's transition lock before registering the load, so a second load choosing the other engine could unload the still-idle engine this one captured; the load then landed on a deactivated engine, where generate, status, unload and the arbiter's evictor can no longer reach it. Registration now happens under that lock and refuses if the engine changed. - Training a DiT family on a host with no GPU was accepted: nf4 is not a CPU fallback, its 4-bit load goes through bitsandbytes, which requires CUDA, XPU or MPS. The start unloaded the working Images pipeline, pulled the text encoders, and only then died in the child. Rejected before the teardown now, and /info stops advertising a precision that always 400s. SDXL keeps its documented fp32-on-CPU path. - Both diffusion pages kept the routed-pick marker forever, so re-picking the same checkpoint (after chat evicted it) neither loaded nor cleared the query string. The marker is released once the query is gone. The Images key also carried a stray NUL byte, which made the file read as binary to grep and other tooling. - diffusers dropped Python 3.9 in 0.38, so the unconditional >=0.39.0 pin left pip no candidate at all on 3.9 and made every install that composes the huggingface extras unresolvable there. The floor is conditional now. Also fixes tests that were already red on the branch: two hand-built request fakes had gone stale against fields this branch added, and the handoff-ordering test only failed on a host with fewer than two GPUs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the GGUF variant contract test against the merged handler signature The assertion pinned the exact single-line call handleVariantClick(v.quant, v.downloaded, expectedBytes, v.filename), but the handler takes (quant, filename, downloaded, sizeBytes) and prettier wraps the call across lines, so the mandatory repository test job failed on every push. Match the call structurally and assert the filename really is forwarded in the handler's argument order. * Stop a background page and a stale record taking the GPU or a download with them Five fixes from a review pass over the diffusion work. delete-finetuned rmtree'd a model the Images or Video engine was holding: every guard on that route is chat-only, and Images loads any local path, so deleting a local diffusion model under the storage root pulled the weights (and the companion VAE / text encoders sd.cpp re-reads each generation) out from under a live pipeline. The cached-model route already refuses this; the trained/exported one now does too, matching by path rather than repo id, and failing open on a chat-only install so it cannot block ordinary deletes. A staged download finishing while its page was hidden loaded the model and evicted whatever the user was actually using: both diffusion pages stay mounted behind the router and a load takes the GPU unconditionally. The pick is now held until its page is on screen again, which is also what chat does. A scoped download could report success having fetched nothing. With Hugging Face metadata unavailable no manifest is written, so verification is a no-op, and snapshot_download returns an existing snapshot folder without downloading when its own repo_info call fails. A repo already on disk from a full snapshot job (which ignores *.gguf) therefore completed with no weights and auto-loaded against them. The requested file list needs no network, so it is checked against the disk directly. The XET to HTTP retry reclaimed the job slot without the scoped file list, and that claim overwrites the stored record, so a later identical scoped start compared an empty list against the real one and 409'd instead of adopting the running download. The DiT accelerator gate probed torch.mps.is_available(), which only exists from torch 2.5 while the supported floor is 2.4. All three probes shared one try/except, so on torch 2.4 the AttributeError read as 'no block' and a CPU-only host still evicted the resident pipeline, downloaded the encoders and died in the child. Each accelerator is probed on its own now, through torch.backends.mps. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stage what an LTX-2.3 load reads, keep a routed file's load kind, drop an unbakeable LoRA Three from the latest review. The video download plan always asked for the wide base file list, so an LTX-2.3 pick staged the 2.0 base's VAEs, vocoder and connectors that the checkpoint supplies itself, while the companion files the 2.3 assembly does read were left out of the plan and pulled inline at load, outside the panel's progress, cancel and disk preflight. The plan now recognises a 2.3 pick by name (the load keeps the authoritative header probe, and under-guessing only falls back to the load-time pull), narrows the base list, and stages the extras in the same entry as the checkpoint so one repo stays one scoped job. A pick routed from the chat picker arrives as ?model= and ?quant= with no picker metadata, so a bare local .gguf or .safetensors was loaded as a pipeline: an explicit model_kind wins over the backend's filename sniffing, so it evicted the resident model and then failed on the missing model_index.json. Both pages now derive the load kind from the path, the same way their own picker handlers do. A torchao int8/fp8 build takes adapters only at load time. Switching artifact inside one family keeps the LoRA selection, since the family did not change, but the load did not bake it, so the next generation was rejected with 'reload the model with the adapter selection' while the picker still showed the adapter as active. The selection is now dropped once per resident build, with a message saying to pick and load again. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cancel an evicted safetensors load, spare the arbiter for CPU-only chat, fetch clips lazily Four fixes from the latest review round: - The GPU arbiter's chat evictor only cancelled the llama.cpp side. The orchestrator publishes active_model_name once its worker reports success, so an in-flight safetensors load was visible only as an entry in loading_models and finished onto the GPU after ownership had transferred. Cancel every pending load, and give the safetensors branch the post-load ownership recheck the GGUF branch already had. - A manual gpu_layers=0 GGUF load runs on the CPU with the GPUs hidden from the child, yet it took the arbiter unconditionally: it cancelled a running image or video generation for a model needing no VRAM, then held CHAT ownership so the next GPU workload unloaded it for nothing. Gate the acquire on the same predicate the launch-time CPU-only mask uses, as the image and video loaders gate on their resolved device. - The staged-download hook subscribes per repo, not per job, so another job on the same repo advanced the staged queue (starting a load whose scoped files were still downloading) or wiped a queue that was still running. Compare the variant each callback carries, like the chat page's auto-load does. - The video gallery fetched every record of a page into an object URL that lives until the page closes: 50 clips at tens to hundreds of MB each, for cards the user may never scroll to. Fetch a clip as its card nears the strip's edge, plus the selected one the player needs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim the comments across the diffusion backend Comment-only pass over the Python this PR touches: drop what the code already says, collapse multi-line explanations that still read on one line, and keep the reasoning that is not recoverable from the code. No code, docstring semantics or behaviour changes; verified with an AST comparison against the previous revision, and the backend suite is unchanged (same 37 environment failures as before: the API integration tests that need a live keyed server, the flash-attn install hooks, and the GPU memory fields). * Invalidate latents on a VAE swap, keep a cut-off generation, surface the EMA adapter - source_revision() scanned the checkpoint root plus text_encoder/tokenizer but not vae, so swapping or fine-tuning the VAE in place left the conditioning cache namespace unchanged and a warm run trained against latents from the old checkpoint. Include the vae directory, like any other component the cached tensors come from. - /images/generate answers only when the images are saved, and secure mode's tunnel caps an origin response near 100 seconds, which a native CPU or a high-step run passes routinely. The page reported failure while the work kept running, and a retry would duplicate it. A lost response (fetch rejection or a gateway status the origin never answered) is now told apart from a refusal: the page waits out generate-progress and reloads the gallery, so the run it started still lands. - The trainer emits the EMA adapter's path with the terminal event, but the state update dropped it, so neither the run history nor either response schema carried it and an enabled EMA left nothing discoverable. Keep it, and show it next to the primary adapter. - weighting_scheme advertised a choice of timestep sampling; sampling is always logit-normal and the flag only selects the bell loss weights. Describe what it does. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Hide unloadable cached rows, hold the dataset interlock, bound a GIF export - The cached-model listing tagged any repo with a model_index.json as text-to-image, so a community pipeline the image loader's trust rule refuses still got a row in the Images picker, and a detected-but-untrusted video repo fell through to that same tag. Gate the image tag on the load path's rule and hide an untrusted video repo outright. - A routed diffusion pick only carries a GGUF filename, which is all the chat picker has, so a curated single-file artifact arrived with no quant and was loaded as a pipeline: from_pretrained on a repo with no model_index.json. Pass the page's own catalog spec into the route pick, so a routed pick resolves to exactly what a direct pick on that page resolves to. - The dataset mutation endpoints checked is_active() and only then handed their filesystem work to a thread, so a start reserving in that gap changed captions or removed images underneath the preflight or the running trainer. The interlock is now registered for the whole request under the lock reserve() uses, and a start refuses while a mutation is open rather than waiting on it. - GIF export held every kept frame as a paletted image before encoding; a clip may be 2048x2048 for 1024 frames, and at the 12 fps target the step is 1, so one export click could allocate over 4 GB and take the backend down. Downscale past 720 px and widen the step to keep at most 300 frames. - seed accepted any Python int, so an out-of-range one passed every preflight, evicted the resident models, spawned the trainer and only then died in torch.manual_seed. Bound it to torch's 64-bit range in the request and config. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the accelerator probes in the DiT family-metadata tests Six tests read family_train_infos() (or a start preflight) without pinning the host probes, so they only held on a machine with a bf16 accelerator: on a GPU-less runner the DiT gate empties precision_modes, turns supports_compile off, and replaces any other preflight message with the no-accelerator note, and all six failed there. A conftest fixture pins both probes for exactly those tests, so they assert the family metadata they are about on every host. The gate's own CPU-only behaviour keeps its dedicated tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Carry the pipeline task into the hub inventory the pickers read - The task-scoped pickers filter On Device rows on a task, and the chat picker routes a diffusion pick by the same field, but those rows come from the /api/hub inventory, which never carried one: the Images and Video pickers listed nothing on device and the routing never fired. Both cached scans and the local listing now tag rows with the classifiers the models API already uses, the schemas and the frontend adapter carry it through, and a row the backend classified as a generation task is exempt from the chat-only guard that was also dropping it. - The local routing map was keyed by model_id while the row click passes id (a filesystem load id for a models_dir or LM Studio entry), so the lookup missed and the pick fell through to the chat loader. Key both. - A staged download whose start answered "error" left its head in place, where the effect never re-runs and onReady never fires, so the pick was stranded until the user reselected. Clear the queue and say so. - Every scoped pick in a repo shares the @diffusion variant, so the variant alone cannot tell two file sets apart: restaging while the first job finished let its completion pass for the new pick and load a checkpoint that had not downloaded. Bind the callbacks to the repo + file set they started, and to the staging generation. - A rejected generate POST does not say whether it reached the backend, so an immediately idle progress read was ambiguous and a submission that never landed looked like a finished image. Require evidence: progress seen active, or a gallery record that was not there before the POST. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the OpenAI image URL fetchable, keep WebM audio, stream example imports Four review items on the diffusion Studio work: - response_format=url returned the bearer-gated gallery route, which a standard image client downloads with no Authorization header, so the default response format was unusable. Mint a short-lived HMAC link instead (the shape RAG already uses for pdf.js) served by a signed route, and leave the gallery route itself bearer-only. - A manual gpu_layers=0 load carrying speculative_type="off" -- a value the UI persists and sends -- read as GPU-bearing, so it took the GPU arbiter and evicted a resident image/video pipeline even though the launcher hides the GPUs for it. Canonicalize the mode and exempt "off". - The curated example import prepared the whole split before the loop stopped at the 10-100 image cap; m1guelpf/nouns is 49,859 rows / 328 MB. Stream instead, with the prepared load kept as a fallback for a repo that cannot stream. - WebM export dropped the audio track an LTX-2 clip carries, silently, on the format offered for web embeds. Mux it as Opus through a resampler + FIFO, and keep exporting the video alone on a build without libopus. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not stub out triton on a GPU host when the Xet backend fails to import The lazy loader retries `import unsloth_zoo.hf_xet_fallback` under UNSLOTH_ZOO_DISABLE_GPU_INIT=1 whenever the first attempt raises. That flag makes unsloth_zoo take its MLX/CPU path, which injects triton and bitsandbytes STUBS into sys.modules for the rest of the process. On a working GPU box whose first import failed for an unrelated reason (a bitsandbytes/CUDA mismatch, say) the retry succeeds, so Studio boots looking healthy and then dies at the first CUDA-only kernel: a GGUF or compiled diffusion generation hits the stub and returns NotImplementedError: Unsloth: 'triton.tools.experimental_descriptor.enable_in_pytorch' was called on Apple Silicon / MLX, where triton is stubbed out. so every image generation 500s with an Apple-Silicon message on a Linux CUDA host, while the load reports success. Found by loading Z-Image-Turbo GGUF through the API on a box where bitsandbytes could not initialise. Gate the retry on the host genuinely having no accelerator. The Xet stall watchdog is optional and already degrades with a warning; a process whose triton is stubbed out is not recoverable. The warning now says why it did not retry. * Fix the lost-generation proof set, the settle timeout and the hub inventory's diffusion gates Seven fixes from the latest review round on the Images page and the hub cache inventory. Images page: - The lost-POST settle path built its "already seen" gallery id set inside the catch, after the request failed. By then the earlier runs of the same batch had already prepended their records, so run 2 could accept run 1's image as proof that its own request reached the backend. The set is now captured once before the first POST and grows with every record the batch produces. - settleLostGeneration fell out of its SETTLE_MAX_MS loop and returned normally, so a wedged generation was counted as done and the next run started against a busy backend. It now throws on timeout. - Restoring a recipe cleared the ControlNet selection but left the workflow tab and the init / mask / reference images pointing at whatever was loaded, so the next Generate conditioned on an unrelated image. It now clears all of them and returns to Create. - The download plan omitted the adapter selection the load itself bakes in. A baked LoRA forces the dense build path, so the plan described a different file set than the load that followed and the rest was pulled inline, outside the download manager. Both now derive the list from one helper. Hub cache inventory: - A download for a repo an Images or Video load is staging was allowed to start: only the llama.cpp loader was consulted. Both diffusion backends already expose loading_repo_ids for the delete guard, and the download guard now reads them too. - A companion-only prefetch (pipeline manifest plus VAE and text encoder, no transformer) passed the snapshot-partial check, since every file its manifest expected did arrive, and was advertised as on-device although from_pretrained cannot load it. - The single-file flag never reached the picker through the hub inventory path, so a checkpoint-only diffusion repo read as a full pipeline and failed after the handoff. The two pipeline-shape helpers now live in hub/utils/inventory_scan.py so /api/models/cached and the hub inventory classify the same repos the same way. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop adopting an unknown scoped download, leaking raced blobs and resurrecting deleted clips Three items from the latest review round. A scoped download job carries a deliberate file subset, and every file set of one repo rides the same "@scope" slot. A client that adopts a live job from the backend had no file list to compare against: the active-downloads response never carried one, so an adopted job's set was unknown and any later scoped request for the same repo read as "already started". Selecting a different checkpoint then waited on the wrong transfer and tried to load a file nobody fetched. The response now publishes the scoped file list, adoption records it, and an unknown set no longer satisfies a scoped request. A gallery record can be deleted while its blob is still downloading. The delete revokes the URL present at that moment, so the fetch that lands afterwards inserted a fresh object URL for a record no card renders and nothing can revoke: a full MP4, tens to hundreds of MB, pinned for the rest of the session, and once per raced fetch. Both galleries now discard a blob whose record went away, with an epoch covering the video page's Clear all. The video backend keeps the last completed job until the next one starts, and the Video page merges that record on mount to cover a job that finished after the gallery fetch. Deleting the clip left the record in place, so every reload prepended a ghost card whose file request 404s until another generation replaced it. Deleting the clip, or clearing the gallery, now clears the matching terminal record, and the page skips a record it deleted itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Resolve revisions from the live cache, serialize dataset imports, and stop pinning every gallery blob Five more items from the review round. The conditioning-cache revision marker read huggingface_hub's import-time HF_HUB_CACHE constant. Studio can move its cache during a session and loading follows the live setting, so after a move the marker went unresolved (or pointed into the previous root) and pulling a new revision of the same checkpoint no longer invalidated the cache: a warm run could reuse the old encoder's embeddings and the old VAE's latents. It now looks in the active Studio cache first and keeps the environment and the library constant as fallbacks, which the trainer subprocess still needs. The dataset interlock counts mutations rather than excluding them, so two imports of different examples into the same empty name both got past the emptiness check. The winner promoted its staging directory atomically; the loser found the folder non-empty, fell back to a per-file move, and merged its images and captions into the winner's dataset. Imports now take a per-folder lock, a second one is refused with 409, and the emptiness check is repeated under the lock. On Windows the sd.cpp asset resolver filtered only by accelerator token, so a Windows arm64 host matched an x64 zip, downloaded and installed it, and failed later when the binary would not run. It now filters by architecture the way the Darwin and Linux branches do. Every gallery page fetched every PNG up front and kept the object URL for the session, so scrolling a large gallery grew memory without bound for tiles the user may never look at. The Images strip now fetches a tile as it nears view, like the Video strip, and keeps the eager path only where IntersectionObserver is unavailable. A 503 carrying a JSON body comes from the application, not a proxy, so it is surfaced as the error it is instead of entering lost-response settlement and being reported as a request that never reached the server. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the class of a failed generation instead of a bare "Image generation failed." Found on a macOS runner: the native renderer aborts inside its own text encoder there, and the page showed only "Image generation failed." with the sd-server backtrace left in the server log, so nothing about the failure reached the user. The failure is now classified into fixed text, out of memory and native-process death, so the message says what happened and what to try. None of the engine's own output is echoed, since a native tail carries local paths and argv; that stays in the log, and an unrecognised failure keeps the original literal. * Treat an undecodable caption sidecar as the tombstone the trainer sees Uploads store .txt and .caption sidecars as raw bytes, so one can hold invalid UTF-8. The trainer treats any existing sidecar, decodable or not, as an empty tombstone and never falls back to the metadata row for that image. The labeling grid and the dataset summary read an undecodable sidecar as absent instead, so both showed a metadata caption that the run would silently replace with the instance prompt, and counted the image as captioned. Both now track sidecar presence separately, so what the user reviews is what the run trains on. * Keep the reason a native server died, not just its backtrace A ggml abort prints its cause first and then a stack trace, so reporting the last twenty captured lines gave twenty addresses and nothing about the failure: on the macOS runner the native server died on an unimplemented Metal op and the message carried only frame pointers. The captured tail now leads with the lines that name a cause and keeps recent context after them, for both the startup failure and the mid-request death. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop staging the dense text encoder for an fp8 video load Two halves of the same gap, found while measuring the LTX-2.3 download plan: - The video download plan and the scoped pre-download never saw text_encoder_quant. An fp8 request loads a hosted pre-cast encoder, so asking for one still staged and downloaded the base repo's dense Gemma3 (48.79 GB of Lightricks/LTX-2 on the 2.3 distilled pick) that the pipeline then never opened. The plan now drops those shards and stages the pre-cast checkpoint instead; their configs stay, since the pre-cast loader still meta-inits the encoder from the base repo's component config. - The LTX-2.3 assembly builds every component itself, so pipe_kwargs (which carries the pre-cast encoder for from_pretrained) never reached it and an fp8 request silently loaded the dense encoder anyway. It is passed across explicitly now. The dense skip is earned, not assumed: only a pre-cast checkpoint that resolves on the Hub lets the plan drop the dense shards, and only one already fetched to disk lets the pull drop them, so an unpublished or gated artifact leaves both exactly as they were. If injection still fails after that, the load tops the dense weights back up rather than handing from_pretrained a snapshot with no encoder in it. Measured against the real Hub on the 2.3 distilled Q4_K_M pick: 67.24 GB before, 18.92 GB with a 0.43 GB stand-in for the pre-cast artifact (the base entry drops from 24 files / 48.79 GB to 13 files / 0.04 GB). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Match the file's typing idiom for flow_shift models/training.py annotates with typing constructs throughout (105 Optional[...], no Union), and flow_shift was the one place using PEP 604. Union[] reads like the rest of the file, and it also drops the runtime evaluation that would raise on Python 3.9. * Bound the gallery blob cache, and three interlock fixes Four review findings, all reproduced first: - The gallery object-URL caches were unbounded. A clip runs from a few MB to a few hundred, both pages stay mounted after their first visit, and entries were only dropped on delete, so scrolling pinned everything for the session. Both pages now share a byte-budgeted LRU (512 MB video / 192 MB images) keyed off the visibility signal the near-viewport fetching already provides. On-screen media, the selected clip or image, and the item just fetched are never evicted, so eviction is invisible and a single item larger than the whole budget cannot evict itself into a refetch loop. - The image, video and chat load guards ran two independent training probes but returned early when the FIRST one raised, so an unreadable LLM backend disabled the diffusion interlock and a load could proceed straight into an active diffusion trainer on the same GPU. The probes are independent now. - An engine switch swallowed a failed teardown and published the new engine anyway, which is exactly the leak the unload exists to prevent: the arbiter's evictor, /images/unload and the next load all resolve through get_active_diffusion_engine(), so the still-resident pipeline (or a live sd-server) became unreachable and the next load allocated on top of it. The switch now fails and leaves the old engine published, so it stays reclaimable. - The native generation timeout was 30 minutes while the Images page waits up to 6 hours (SETTLE_MAX_MS), so slow-but-progressing CPU jobs died deterministically at the deadline. Measured on GPU-less runners, a 512x512 4-step Q2_K generation took 900 s on Linux and 1465 s on Windows, so larger images or step counts clear half an hour easily. The ceiling now matches the page's window and applies to the whole request: chunks of a split batch share one deadline instead of each getting a full budget. Cancellation is unchanged. Declined: gating the huggingfacenotorch extra off Python 3.9 over the conditional diffusers marker. The marker is deliberate and its comment says why: diffusers dropped 3.9 in 0.38, so pinning >=0.39 outright leaves pip no candidate and the whole extra unresolvable there. The pipelines it names live in studio/backend, which cannot install on 3.9 anyway (studio.txt pins matplotlib==3.10.9 and fastmcp>=3.0.2, both requires_python >=3.10), and the extra is the general core one, so the alternative drops 3.9 for library users who never touch Studio. * Close the load-versus-training-start race, and two picker fixes - The image and video load guards read is_active() and only then selected an engine, acquired the arbiter and registered the load. A /train/diffusion/start reserving inside that window freed residents the load had not registered yet, so the trainer came up beside a brand-new pipeline. The service already had exactly the right pattern for this in dataset_mutation, so gpu_load_admission mirrors it: reserve() refuses while an admission is open, an admission refuses once a start is reserved, both decided under the one lock. The span is only the registration, since begin_load returns as soon as the load is registered and _free_gpu_for_diffusion_training preempts an in-flight load from that point. Chat is deliberately not covered: its load spans an eviction plus a multi-minute GGUF load, and it admits models that fit beside training by design, which is a different contract from the diffusion pipeline's all-or-nothing one. - Hugging Face gives the LTX-2 family the image-to-video pipeline_tag (both Lightricks/LTX-2 and unsloth/LTX-2.3-GGUF report it), so a text-to-video-only filter dropped the flagship audio family out of Video Hub search while the rest of the app routed it to Video. - Task-scoped quant fit sized picks against the LARGEST visible device while resolve_diffusion_device_target returns a bare "cuda" and torch places on the current one. On a heterogeneous host that recommended a checkpoint sized for the bigger card and then loaded it onto the smaller one. Fit now uses the device the load actually lands on; identical on a homogeneous host. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Expose the persistent conditioning cache in the start schema DiffusionLoraConfig has carried cond_cache_dir for a while and the DiT trainer acts on it, but DiffusionTrainingStartRequest omitted the field, so Pydantic dropped it silently and every API-driven run fell back to the in-memory cache that is rebuilt from scratch each time. The warm path skips loading the VAE and the multi-GB text encoders on a rerun whose images, captions and resolution are unchanged, so this was a real capability that could not be reached. Contained like output_dir rather than left to the trainer subprocess's cwd, since it is another directory the trainer writes to. Blank or omitted still means the in-memory cache, so it must not resolve to the outputs root. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix diffusion policy and classification issues from review fp8 auto precision defaulted to precise accumulate on any non-consumer GPU, which made fp8 2.05x slower than int8 on RTX 6000 Ada and slower than not quantising at all. NVIDIA's professional whitepapers do publish equal FP8 rates for both accumulate modes there, so the hardware premise held, but the cost is in the cuBLAS path rather than the published rate. Default to fast accumulate: measured on B200 the flag is a no-op (4096^3 _scaled_mm at 3023.8 vs 3041.8 TFLOP/s, bitwise-identical output, 1.213 s vs 1.230 s end to end), so it is a large win where it bites and free where it does not. Precise accumulate stays available via transformer_quant_fast_accum. Z-Image's DiT is a Lumina2 derivative, so unsloth/Z-Image-GGUF and unsloth/Z-Image-Turbo-GGUF both declare general.architecture = "lumina2" and the whole line was tagged image-diffusion-unsupported and hidden from the Images "On Device" list, though validate_load_request loads them. Resolve shared archs from the repo/file name like bare "wan" already does, with a test asserting the picker and the loader agree for every family. The sage attention on-demand install ran an unpinned `pip install sageattention`, but PyPI's newest wheel is 1.0.6 and diffusers refuses anything below 2.1.1: the install always "succeeded", wrote an unusable version into the running venv, and was rejected on the next line. Carry the dispatcher's floor so pip resolves nothing instead. The dense-quant disk gate sized the download from the bf16-RESIDENT table. The fp32 families download twice that (Z-Image: 23,479 MiB against a 21,970 MiB gate), leaving a window where the check passed and the download filled the disk; Ideogram 4 ships fp8 and was overcharged the other way. Size the gate by published bytes, verified against HF sibling metadata for all 12 families. Patch installs went through unsloth_zoo, which refuses to import unless UNSLOTH_IS_PRESENT is set, and that is set by unsloth itself. The server imports unsloth at boot so it never showed there, but any other process ran silently unpatched with every install returning False, which is 13 test failures on a clean environment. Import unsloth and retry once, memoised per process. Also: the GGUF+LoRA refusal pointed at the native engine without saying a GPU host only selects it under UNSLOTH_DIFFUSION_ENGINE=sd_cpp, so the suggestion was unreachable; the gallery recipe recorded loras from the generate request alone, losing a load-time bake; load-progress claimed "40.07 GB downloaded" for a fully cached load; and pickers.tsx imported three catalog-group helpers it never used. Reported by oobabooga. * Keep the sd.cpp text encoder on CPU under Metal macos-14 loads FLUX.2-klein-4B Q2_K natively on mps and then dies on the first generation with exit code -6: ggml_metal_op_encode_impl: error: unsupported op 'RMS_NORM' -> ggml_abort LLMEmbedder::encode_prompt -> LLMRunner::compute -> GGMLRunner::compute ggml's Metal backend gates RMS_NORM on contiguous rows and aborts the process when that does not hold, with no per-op CPU fallback, so any LLM text encoder (Qwen3 for FLUX.2 and Z-Image, T5 for FLUX.1) takes sd-server down. The encoder runs once per prompt while the DiT runs every step, so pinning only the encoder keeps Metal for the part that matters. UNSLOTH_DIFFUSION_SD_CPP_METAL_TE_GPU=1 opts back in once ggml grows the kernel. * Gate the unsloth retry in the diffusion patch backend The retry added for the clean-environment patch failures is not free: importing unsloth pulls torch in behind it, which costs ~940 MB of RSS measured in a process that had neither, and on a host with no accelerator it fails anyway. A cross-platform CI job that had generated fine at ~900 s later died 19 s in with SIGTERM and every 'if: always()' step skipped, which is the runner being torn down rather than a step failing. Retry only when torch is already imported (true of the server and of anything patching a real module, and the condition that stops the retry from being what loads torch), unsloth is installed but not yet imported, and the first failure was the ImportError the sentinel guard raises. The clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only retry the unsloth import where it can succeed The gate still let the retry run on hosts unsloth does not support, which is where it is most harmful: a 7 GB macOS runner lost the Studio server 26 s into a load, and the Linux runner was torn down mid-generation. Neither MPS nor plain CPU can complete the import, so the retry there pays the cost and fails anyway. Require an accelerator unsloth actually supports (CUDA/ROCm via torch.cuda, or XPU), with UNSLOTH_ALLOW_CPU as the documented override, and hoist the predicate to module level so it is tested directly rather than through the import system. On a CPU-only host the retry no longer fires at all; on CUDA the clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Guard old diffusers, stream video exports, record conditioned recipes Three fixes from review. The 0.39-only pipeline classes (Flux2Klein, Z-Image, Krea 2, LTX-2, HunyuanImage) were resolved by getattr deep in the load, so on the older diffusers that packaging still allows on Python 3.9 -- diffusers dropped 3.9 in 0.38 and this project still supports it, so the 0.39 floor has to be conditional or the extra becomes unresolvable -- an advertised model failed with a bare AttributeError after its checkpoint had already been downloaded. Krea 2 already guarded itself this way; assert_pipeline_class_available now runs the same check for every image and video family from validation, before any fetch, and names the version and the fix. WebM export accumulated the whole VP9 output in a BytesIO and returned it as one bytes object that the response held again. The request caps allow 2048x2048 for 1024 frames, so an export runs to hundreds of MB and concurrent clicks could exhaust the process, while the MP4 route beside it already streamed from disk. transcode_to_file encodes to a temp file and the route returns a FileResponse with a background unlink, so nothing large is resident. A conditioned generation's recipe carried only the txt2img fields, so the gallery presented an inpaint or upscale result as a complete Create recipe and restoring it replayed an unrelated text-to-image request. The images themselves are still not persisted (user uploads with their own lifetime), but the workflow and its scalars are, restore reapplies them, and the toast now names the inputs that have to be supplied again instead of silently landing on Create. Reported by Codex. * Per-load video cancel event, family-gated image picker, cond cache refusal A cancelled video load could resume: begin_load cleared the shared cancel event, and unload() drops _loading without waiting for the worker, so the next load cleared the very object the cancelled worker was watching and its multi-gigabyte pull ran on alongside the replacement until the token check at the end. Each load now gets its own threading.Event, passed down through _fetch_te_prequant and _predownload_base, so a cancelled worker stays cancelled. A cached repo with a model_index.json was advertised as text-to-image on the trust rule alone, but validate_load_request also requires a detected image family, so a trusted pipeline of an unsupported class produced a picker row that deterministically 400s. The picker now applies both gates, mirroring the video branch. cond_cache_dir was accepted for sdxl and then ignored: only the DiT trainer reads it, while the SDXL trainer builds a per-run in-memory latent cache, so the promised cross-run reuse never happened. The route now refuses it with a 400 that names the families which do support it, checked against the resolved family so an omitted model_family with an SDXL base is caught too. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the frontend build broken by the gallery blob cache tsc -b failed on the branch head, so npm run build produced no dist and every platform job fell back to --api-only: blob-url-cache.ts(29,15): TS1294: This syntax is not allowed when 'erasableSyntaxOnly' is enabled dataset-labeling-grid.tsx / dataset-showcase.tsx: Argument of type '{ url: string; bytes: number; }' is not assignable to parameter of type 'string' The cache took its budget as a constructor parameter property, which the project's tsconfig forbids, and fetchGalleryObjectUrl now returns the blob size alongside the URL for that budget, which the two dataset thumbnail components still consumed as a bare string. Declare the field explicitly and destructure the URL at both call sites. tsc -b is clean and vite build emits dist again. * Recover from a ggml unsupported-op abort by restarting on the CPU backend ggml checks every node against the device's supports_op and calls GGML_ABORT when one is not implemented, because a single-backend graph has nowhere else to put it: there is no per-op CPU fallback. The whole sd-server dies with SIGABRT mid-generation and the user gets "the native image renderer stopped unexpectedly" with no way forward. Seen on macos-14 arm64 with FLUX.2-klein-4B Q2_K through the cross-platform CI: the text encoder is already pinned to CPU, and the abort moved into the denoise loop instead. ggml_metal_op_encode_impl: error: unsupported op 'MUL_MAT' -> ggml_abort StableDiffusionGGML::sample -> sample_k_diffusion A retry on the same backend would abort identically, so the load is restarted once with --backend cpu (the only flag that changes which backend executes the graph; --offload-to-cpu moves parameters, not compute) and the generation is re-submitted. The same checkpoint then renders slower rather than not at all. Strictly bounded: the signature must carry both the unsupported-op line and ggml_abort, the device must not already be CPU, and it happens once per load, so an OOM kill or a genuine crash still surfaces as itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix two tests that only fail in a full-suite run The 3.10 CI leg resolves PyAV 17, where av.container.OutputContainer is an immutable C type, so the no-libopus export test died on "cannot set 'add_stream' attribute of immutable type" before it asserted anything. Inject the refusal by wrapping the container av.open() returns instead; modules stay patchable on every build. Removing the injection makes the test fail again, so it still covers the branch it is named for. The Xet shim's degraded-path tests drop utils.hf_xet_fallback from sys.modules and import a throwaway copy. Restoring only the sys.modules entry left the utils package attribute bound to the throwaway, and the two disagreed for the rest of the process: a later monkeypatch of the dotted target patched one copy while the code under test imported the other, so the patch did nothing and test_fetch_te_prequant_only_reports_what_it_downloaded reached the real Hub and got a 401. Restore both bindings. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop an ignored-cancel sd-server, guard deletes during diffusion training, repair unusable managed binaries sd-server does not interrupt an in-flight job, so when it ignores a cancel the grace branch abandoned the poll and reported cancellation while the native job kept a core (or the GPU) busy to completion and held the server's job slot. The comment said the caller stops the server, but only unload does that immediately: a superseding load stops it after its multi-gigabyte download, and a load that then fails never gets there. Stop it here, as the deadline branch already does. DELETE /api/models/delete-finetuned checked only the LLM trainer, so it could rmtree the output directory a live diffusion LoRA run was about to write its adapter into. Consult the diffusion training service too, like the dataset mutation and model-load routes. find_sd_*_binary only checks is_file(), so an interrupted extraction (or a prebuilt for the wrong CPU) left a present-but-unrunnable binary the installer never retried: every load probed it, fell back to diffusers, and native inference stayed off until the directory was deleted by hand. Probe it and reinstall, but only for a copy under the installer-owned root -- SD_CLI_PATH, UNSLOTH_SD_CPP_PATH, an in-tree build and anything on PATH are the user's. * Plan the pre-cast text encoder, and make the cross-trainer GPU admission atomic An fp8 text-encoder request loads a hosted PRE-CAST checkpoint, but the image download plan never received text_encoder_quant, so the manager staged the base repo's dense encoder (FLUX.2-dev's Mistral-24B is ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) and the load then pulled the pre-cast file inline, outside the manager's progress and disk preflight. The plan now takes the field, resolves the hosted artifact with the same resolver the injection uses, stages that file, and drops only those components' dense weight shards. The load's own prefetch takes the same treatment, since it paid the same cost. Only a checkpoint that really resolves on the Hub earns the drop, so a gated or renamed artifact still stages the dense encoder the load will fall back to. The two trainers admitted each other with independent check-then-act guards: the diffusion route checks the LLM backend several network-bound preflights before it reserves, and the LLM route checks the diffusion service well before it spawns, so two near-simultaneous starts could both pass and train on one GPU. reserve() now re-tests the LLM backend under its own lock, and the LLM route holds the diffusion service's gpu_load_admission across its spawn, so exactly one of the two wins. Both halves fail open, so a chat-only install still trains. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the Advanced options a staged download planned against Staging does not set busy, so while a multi-gigabyte download runs the user can still change precision, memory mode, speed or the baked LoRA selection. The pending record held only the repo and artifact, and the completed download fired a load that read the CURRENT state: the staged file set could then be missing files that load needs (fetched inline, with no progress and no disk preflight) or hold gigabytes it no longer uses. One snapshot of every Advanced control is now taken when the plan is built, and it travels with the pending record into the load, so the load that runs is the one the download was planned for. * Do not advertise a family the installed diffusers cannot build The newer families (Z-Image, Krea 2, FLUX.2, LTX-2, HunyuanImage) exist only from diffusers 0.39, and 0.39 cannot be installed on Python 3.9 at all -- diffusers dropped 3.9 in 0.38, so the requirement is conditional or the whole extra becomes unresolvable. On such an environment the picker still offered those rows, every pick failed deterministically, and the error's advice to run pip install -U diffusers could not fix it without also upgrading Python. The cached-repo picker now applies the same availability check validate_load_request does, which is keyed on the pipeline class actually present rather than on the Python version, so it is also right for an intentionally pinned older diffusers on 3.10+. Fails open when diffusers cannot be imported at all: that is a different problem and the load path reports it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore the diffusion engine selection after each router test The active engine is module state, and several tests set it by plain assignment because what _activate does to it is the thing under test, so monkeypatch could not undo it. A leaked ENGINE_SD_CPP left get_active_diffusion_engine() handing back the sd.cpp backend for the rest of the process, and every later route that reads the active engine then saw an unloaded model: eight tests in test_openai_images_generations_route.py returned 503 in a full-suite run while passing on their own. The autouse fixture now snapshots and restores it. * Stream gallery clips, and close three races around them Four fixes from the latest review pass. The video gallery downloaded each clip into a blob before it could play, so playback waited on the whole file (tens to hundreds of MB), seeking was limited to what had arrived, and every viewed clip stayed pinned in the webview. The file route already streams and serves ranges; it just could not be a <video src> because it is bearer-gated. Mint a short-lived signed link instead (its own HMAC secret, 12 hour TTL, separate from the image links) and hand it to the element, which then fetches only the ranges it plays. That removes the blob budget, its LRU and every revoke on this page. The sd.cpp readiness probe accepted any process answering on the port, so a foreign server that grabbed the port between the bind check and the spawn was adopted as ours. Confirm the listener is our child before reporting ready, and stay best-effort (psutil missing, an unknown owner, or any probe error still passes) so the check can only reject a definitely foreign process. Dataset import held its lock for the extract but not for the upload path, so two concurrent uploads into the same folder interleaved; take the same lock and return 409. And reject Windows device names (CON, NUL, COM1..9, LPT1..9, with or without an extension) plus trailing periods in dataset names, which are unopenable on Windows. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Build the image download plan for the engine that will load /images/download-plan always asked the diffusers backend, while /images/load picks the engine per host: a GGUF pick on a machine with no usable GPU routes to native sd.cpp, which reads a single-file VAE plus text encoders and never opens the base repo's sharded components. Measured on unsloth/FLUX.2-klein-4B-GGUF (Q2_K): the plan staged 7.66 GB of FLUX.2-klein-4B components the native load discards, and the 7.80 GB sd-cli actually needs was then fetched inline by the loader, outside the download manager's progress and its disk preflight. Z-Image-Turbo is the same shape. The plan now asks whichever engine the load will select. predict_engine() applies the selection policy without any side effect: it activates nothing (staging a download must not unload the resident model) and only locates the binary rather than installing it, but still counts an installable binary as available, since that is what the load does on a fresh host. The native backend gains a download_plan built from the same _asset_specs the loader fetches, returning the same envelope, so the manager stages exactly the files sd-cli opens. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not let a queued generation outlive the model, and three scan fixes Five items from the latest review; four were real. An unload or arbiter eviction only cancels the generation holding _generate_lock. A second request queued behind it holds no cancel event yet, and Python locks are not FIFO, so it could take the lock the instant the active denoise released it, still see a loaded pipeline, and run a whole new denoise after the model was told to go away: the eviction then waits minutes for it and an image lands after the eject. Unload and a superseding load now raise a fence under _lock before they queue, and a generation that wins the lock while one is pending refuses instead. The cached-model scan judged pipeline completeness across every revision, so a repo holding an older complete snapshot plus a newer companion-only one read as complete while the snapshot from_pretrained actually opens has no transformer. Both scans now look at the revision the loader will open. Deleting a dataset image deleted its caption sidecar unconditionally, which for cat.jpg alongside cat.png removed the caption the survivor still resolves to. The sidecar now goes only with the last image of that stem, matching what the thumbnail cleanup beside it already did. Importing an example into a folder that holds no images but does hold files fell back to promoting the staging dir one file at a time, so an interruption left a partial dataset that the image_count check accepts as complete on retry. Those files are folded into the staging dir instead and the promotion stays a single atomic rename. The MPS generator report does not apply: torch.Generator(device="mps") has worked since PyTorch 2.0 (pytorch/pytorch#91348) and the studio installer pins torch>=2.4. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten diffusion comments Collapse the multi-line comment blocks across the image, video, sd.cpp and diffusion-training code to one or two lines each, and drop comments that only restate the statement below them. Comments only, no code or behaviour changes. * Tighten diffusion comments (second pass) Collapse the remaining multi-line comment blocks in the video page, training routes and service, sd.cpp server and installer, memory and speed planners, and the shared request models. Comments only, no code or behaviour changes. * Tighten diffusion comments (third pass) Collapse the remaining multi-line comment blocks in the attention, cache, LoRA, prequant, precision and compile-cache modules, the sd.cpp arg builder and engine, the video routes, the Ideogram 4 assembly, the model picker, and the diffusion test suites. Comments only, no code or behaviour changes. * Restore the dataset when an example import cannot be promoted Promotion folds the folder's pre-existing entries into the staging dir so the swap is one atomic rename. Every failure after that fold left the user with nothing: the 409 path and an os.replace error both fell through to 'finally: shutil.rmtree(staging)', which deleted the entries that had just been moved in there, while the response said 'Nothing was written'. A same-named entry was also unlinked outright before the promotion was known to succeed. Park superseded same-name entries in a rescue dir instead of deleting them, record every move, and restore all of them if any step of the promotion fails. The fold loop itself is covered too: renaming a non-writable directory raises EACCES on POSIX, which previously escaped as a 500 after the earlier entries had already been moved out. A failed rename now maps to the same retryable 409 as the rmdir conflict. Verified with the reported trigger (a non-empty mode-500 directory whose name collides with an imported file): the folder listing is now identical before and after the failed import. * Drain the teardown fence on a failing unload, give each load its own cancel event Two independent leaks on the image path, both already solved elsewhere in the same file. unload() incremented _teardown_waiters, ran _unload_locked() and decremented, with no try/finally, while the superseding-load path used a finally for the same pair. _unload_locked ends in clear_gpu_cache(), whose CUDA branch calls synchronize/empty_cache/ipc_collect unguarded, and a sticky CUDA fault makes those raise. The count then never drained, so every later generation was refused as cancelled for the life of the process, a fresh load included, since begin_load's own increment and decrement are symmetric. unload() is reached from the chat/video GPU handoff, the engine router and two training routes, so one fault during an ordinary handoff wedged image generation until restart. Release it in a finally. The image and native backends each cleared one shared cancel Event on a new load. unload() sets that event to cancel an in-flight multi-GB download and drops _loading in the same breath, so a replacement load is admitted while the cancelled worker is still inside the fetch, and its clear() re-enabled the very object that worker was watching: the cancelled download resumed and ran alongside the replacement. Take a fresh Event per load and thread it to the worker, as the video backend already does, and set it under the lock since begin_load now rebinds the attribute. * Name utf-8 on the diffusion text I/O and the sd.cpp subprocess pipes tests/test_text_io_encoding.py failed on five files this branch adds. Text I/O without an explicit encoding falls back to the Windows ANSI codepage, so a non-ASCII path or manifest value round-trips corrupted, and the three sd.cpp pipes decode the child's UTF-8 output as ANSI on Windows despite already passing errors = 'replace'. Eleven read_text() / write_text() sites across diffusion_compile_cache, diffusion_ideogram4 and diffusion_krea2, plus text = True on the sd-cli version probe, the sd-cli run and the sd-server pipe. * Record the load-time build on a gallery image's recipe A gallery record documents itself as the image's full generation recipe and is embedded in the PNG, but the only load-related field it carried was the repo id. A GGUF repo holds many quants, so that does not say which one made the pixels, and it says nothing about an adapter baked in at load time. The fallback meant to cover the baked case could never fire: with no loras on the request _adjust_baked_loras zeroes every baked adapter and _active_lora_pairs drops zero-weight entries, so active_loras was always empty. A baked-and-disabled build is not the same pipeline as a never-baked one, so the recipe could not reconstruct the image once the model was rebuilt. Persist model_kind, gguf_filename, transformer_quant and the baked adapter names, read off the load state rather than the request, and show them in the recipe popover. The new fields are optional with defaults, which matters because list_gallery_images drops any record that fails validation, so a required field would have emptied every existing gallery; a regression test pins that. * Drop eleven duplicated comment tails, restore the mxfp8 denial note The comment passes collapsed several wrapped blocks onto one line without deleting the last physical line of the original wrap, leaving the tail of each sentence repeated as its own comment underneath. Two of the eleven were re-worded rather than byte-identical, so a strict suffix match missed them. 6e16ad16f also dropped the line justifying the qwen-image mxfp8 denial while that rule stayed live in _FAMILY_SCHEME_DENY, under a header that then documented only fp8 and nvfp4. Restored. Comments only; verified with the AST gate. * Only let the dense-quant fallback use shards the prefetch actually staged The prefetch skips the base repo's transformer/ shards whenever a prequant checkpoint is expected, since that checkpoint replaces them. But a prequant fetch can fail for reasons the planner cannot see: an unpublished, gated or renamed artifact, a hub 5xx, a proxy, a checkpoint the validator rejects. The loader then fell through to from_pretrained(subfolder = 'transformer') and pulled those shards inside the load lock during 'finalizing', after the previous pipeline was already evicted, where the cancel event has no reach, load_progress has already reported bytes_downloaded == bytes_total, and the cache-disk gate had only reserved the small prequant checkpoint. That is verbatim the situation _dense_quant_prefetch_needed's own docstring exists to prevent. Gate the in-loader dense fallback on the shards being staged, read off the returned file list rather than the request so a failed size estimate closes it too, and let the GGUF build take over otherwise, which is what the prequant-sized replan already does one branch over. It is also the invariant the text-encoder path already enforces: only a component that really resolves may have its dense weights dropped from a plan or a prefetch. Adds the missing coverage for the gate's prequant arm, which the existing disk-gate tests never reached. * Refuse a training output_dir that resolves to the outputs root resolve_output_dir strips a leading 'outputs' and drops '' / '.' segments, so '.', './', './.', 'outputs', 'outputs/outputs' and ' . ' all clean away to nothing and land on the outputs root itself rather than a run directory under it. The DiT trainer then writes pytorch_lora_weights.safetensors flat into the root, where scan_trained_models and scan_checkpoints cannot see it (both filter is_dir()), and a second such run overwrites the first. The UI only checks the field is non-empty, so 'outputs' is one plausible run name away. Refuse it with a 400 that says what to do instead. cond_cache_dir collapses the same way but has an honest 'off' to fall back to, so a root-resolving value now means the in-memory cache, which is what the comment above it already promised: otherwise a run drops one flat safetensors per cached latent and caption into the directory trained models live in. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Repair three defects the main merge left behind Git merged all three files without a conflict, but the result was wrong in each case. The tests only started failing once the merged tree was run. routes/inference.py duplicated the GGUF load block: this branch had moved the gguf_load_in_flight marker and the _hub_download_blocks_gguf_load guard under "if config.is_gguf and config.gguf_hf_repo", and the conflict resolution re-added main's copy at the old position, so both ran. Dropped main's copy; the earlier placement is the deliberate one, so a 409 from the hub guard cannot tear down a resident Images or Video pipeline. test_gpu_selection.py still called _hf_offline_if_dns_dead, which main renamed to _hf_offline_if_unreachable_for (#7591). Disjoint edits, so no conflict, but four route-error tests referenced a function that no longer exists. test_gguf_load_cache_reuse.py anchored its ordering assertion with rindex over "if config.is_gguf:", taking the last one before the load marker. That only held while _resolve_inherited_extra_args sat above every such line; main has since hoisted it above the gpu_ids preflight, so the anchor landed between the call and the marker and the assertion compared against an unrelated later call site. The ordering it checks is a property of _load_model_impl as a whole, so it now anchors on the function. The ordering itself is intact: _resolve_inherited_extra_args, then the gguf_load_in_flight marker, then the hub guard, then the chat handoff, then unload_model. * Stop a real sd.cpp install from breaking its own discovery tests The five "nothing is installed" assertions in test_sd_cpp_engine.py cleared SD_CLI_PATH and UNSLOTH_SD_CPP_PATH and patched Path.home, which covers hops 1, 2 and the fallback half of hop 3. It leaves two hops live. Hop 3 goes through managed_install_root(), which honors UNSLOTH_STUDIO_HOME and STUDIO_HOME and resolves to the stable-diffusion.cpp directory beside the studio home. That is the documented way to run side-by-side Studios, so anyone who has one set gets a real sd-cli back from a finder the test expects to return None. Hop 4 is the in-tree developer build, which does the same for anyone who built sd.cpp inside the checkout. Isolated with an autouse fixture rather than another helper call, because reaching the failure needs no fixture: SdCppEngine(binary = None) runs the finder from its constructor, which is why test_generate_raises_when_binary_missing failed too. The fixture points the studio home and the in-tree root at an empty tmp tree, so every hop is answered by the test rather than by the box. The in-tree root moves into a named in_tree_install_root() so it can be pointed somewhere empty; behaviour is unchanged, including the OSError and IndexError guard on an unexpected layout. * Make the sd.cpp uninstall test actually extract the code it tests The two sed ranges anchored the production fragments at column 0, but both blocks sit inside the main removal function and are indented, so each range matched nothing and LOOP_FILE / DEFAULT_FILE came out empty. Sourcing an empty file is a no-op, so the suite reported 4 passed / 7 failed: the seven removal assertions failed because nothing ran, and the four that passed were 'kept' assertions that pass trivially when nothing runs. The shell job auto-discovers tests/sh/test_*.sh and runs each under set -e, and this file is not in its skip list, so it was a deterministic red. Anchor on optional leading whitespace and fail loudly on an empty or _remove_path-less fragment, so a future reshuffle of uninstall.sh cannot make the suite vacuous again. Now 11 passed, 0 failed, against the real removal loop and the real default-mode block. * Agree on what an engine can build, and on what the installer owns Two gates, each half-applied. The unbuildable-family gate had one caller, the image branch of the cached repo picker. The GGUF classifier, the local-model classifier and the video branch had none, so on a diffusers too old for a family the picker still offered its GGUF and the load then failed. Meanwhile the loader asserted the diffusers pipeline class before engine selection, so a GGUF this host routes to native sd.cpp, which instantiates no pipeline class at all, was refused with an upgrade instruction that could not help. Both are wrong one-sidedly: hide a family only when NEITHER engine can build it, and demand the diffusers class only when diffusers is what will load it. One predicate, family_buildable_here, now answers both, so the picker and the loader cannot disagree. The population is real on Python 3.9, whose diffusers ceiling is 0.36: Flux2KleinPipeline, Krea2Pipeline and LTX2Pipeline are all absent there, and FLUX.2-klein GGUF is a repo in this PR's title. That assertion also raised RuntimeError, which /images/load maps to 409, the status that otherwise means a load is already in progress, and which escaped /images/download-plan (it catches ValueError and FileNotFoundError) as a bare 500 with the message lost. It is an unloadable pick like every other, so it raises ValueError and both routes answer 400 with the text intact. The managed-binary repair used a path test for ownership while the installer uses a marker. On a managed root without the marker, which is any install predating it, the repair deleted sd-server and the reinstall was then refused, permanently, because the surviving sd-cli keeps the directory non-empty so the marker can never be claimed: the user went from an unrunnable binary to no binary and no way back. Require the marker before discarding, which is the same definition of ours that uninstall.sh already uses to keep a user's own stable-diffusion.cpp checkout. A genuinely interrupted extraction still self-heals, since install() writes the marker before it extracts. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add gguf and av to the studio extra so the wheel install matches studio.txt This branch added both to studio/backend/requirements/studio.txt, which is what install.sh uses, but never to the studio extra in pyproject.toml, which is what 'pip install unsloth[studio]' uses. Nothing else keeps the two in sync, so tests/studio/install/test_studio_extra_matches_requirements.py failed deterministically, and the CI job runs the whole tests/ tree. The drift is not only a red test: gguf backs diffusers' GGUFQuantizationConfig and av does the MP4 encode and audio mux, so a wheel install got a Studio that cannot read a GGUF or export a video, which is most of what this PR adds. * Fix two test-suite defects: the CPU-only patch gate and a popped module The patch backend reaches unsloth_zoo's helpers only through an ~940 MB 'import unsloth', which a CPU-only host cannot complete, so every patch install returned False and nine arch/eager/compile tests failed there -- exactly the runners the retry was narrowed to protect. unsloth_zoo only wants UNSLOTH_IS_PRESENT in the environment, which costs nothing and needs no accelerator, and the conftest already sets its sibling UNSLOTH_ALLOW_CPU. Set it there too, as run.py and main.py already do at module scope, so no real server ever took the expensive route. CPU-only goes from 9 failed to 24 passed, 1 skipped; the GPU run is unchanged at 29 passed. test_setup_cache_env_hf_home popped utils.hf_cache_settings to model a fresh process and never restored it, so a later import built a second module object and rebound it on the utils package. test_hf_cache_settings then wrote its setting into one object while core.inference.diffusion read the other, the same split-module failure the xet shim already had. Restore both bindings in teardown: 21 pass together, and each file still passes alone. * Restore the setup_fail assertions the main merge reverted A fourth instance of the class 9541cc535 fixed: the merge took main's studio/setup.sh, which #7644 changed to abort through the setup_fail helper so desktop mode still emits [TAURI:ERROR], but kept this branch's older copy of the test, which still asserted the literal 'exit 1'. setup.sh is byte-identical to main here and the only diff in this file was the reverted assertions, so take main's version. 60 passed. * Fence the video teardown, reject non-finite clipping knobs, stop a test installing sd.cpp The video backend never had the teardown fence the image backend documents. Its unload and its superseding-load path both signalled the active generation, then did 'with self._generate_lock: pass' and tore the state down with the lock free. A generation queued behind that barrier holds no cancel event yet, so the signal cannot reach it, and Python locks are not FIFO: it won the lock the instant the barrier released it, read a still-loaded state and denoised a whole clip against the pipeline being freed. Reproduced on both paths, where the queued generation returned a finished MP4. Mirror the image backend: count waiters, refuse a generation while one is pending, and tear down inside the barrier with the counter released in a finally. _teardown_state becomes _teardown_state_locked since the caller now holds both locks. max_grad_norm and snr_gamma were bounded on one side only, so 1e309 floated to inf and passed. clip_grad_norm_ then computes an infinite clip coefficient, clamps it to 1.0 and scales nothing, and min(snr, inf) / snr makes every min-SNR weight 1.0. The run starts, reports normal progress and trains with the requested knob silently disabled. Measured both. NaN already failed the bounds; allow_inf_nan makes that explicit. learning_rate and flow_shift already guard this exact vector. test_load_routes_to_sd_cpp_on_cpu stubbed ensure_sd_cpp_binary but not ensure_sd_server_binary, which select_and_activate_engine probes first with installs enabled, so the unit test downloaded and unpacked 108 MB into the developer's real ~/.unsloth when UNSLOTH_STUDIO_HOME was unset. Now 0 bytes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Key the video schedule reset on the resolved defaults, not the repo id A GGUF repo holds several variants, so another client swapping a distilled build for the base one from the same repo changes the steps and guidance the backend reports while repo_id stays put. modelChanged stayed false, the page kept the previous schedule, and later generations ran an 8-step distilled setting on a model that expects roughly 40 steps and CFG 4. Include the reported defaults in the key. They only move when the resident artifact does, so a manual steps change still survives a status poll. * Retire the release-lag pin now that 2026.7.6 carries the relaxed gates The virgin Windows container row that installs from PyPI on purpose was tolerated with continue-on-error, because a Server Core container has no Microsoft Store and so no winget, and the released studio/setup.ps1 hard-stopped on a winget-only git gate and reached for winget again for the VC++ runtime. #7549 relaxed both, but it landed on 2026-07-28 and the newest wheel was 2026.7.5 from the 23rd, so the row could not get past it. The pin carried its own tripwire for exactly this moment, and it has fired: unsloth 2026.7.6 shipped 2026-07-29 and the released wheel now installs end to end, so the step reported success and the assertion errored with "delete this pin and drop continue-on-error from the Install step so this row gates". Doing that. Both rows now gate unconditionally, and the assertion that enumerated the accepted failure signatures goes with the pin it protected, since there is no longer an accepted failure for it to describe. The other three release-lag mentions in this file are descriptive rather than tolerated-failure pins: they explain why the fedora and ubuntu-nonroot legs run with overlay: true, and both already refuse a triton failure as a regression rather than accepting it as lag. Left alone. * Pin the hosted pre-quant onto the plain-torch fp8 kernel A Z-Image GGUF pick at the DEFAULT speed mode, once the hosted fp8 pre-quant is actually reachable, dies at generate with an HTTP 500: torch._dynamo.exc.Unsupported: Operator does not support running with fake tensors. Developer debug context: unsupported operator: mslk.f8f8bf16_rowwise.default _fp8_config already pins KernelPreference.TORCH when it BUILDS a config, precisely because the default AUTO switches to the MSLK kernel wherever an mslk package is importable (sm90+). A hosted checkpoint escapes that pin completely: the preference is serialized per Float8Tensor, and all 239 weights in the published Z-Image-Turbo-FP8 checkpoint carry AUTO. Loading it re-arms the very kernel the pin exists to avoid, mslk.f8f8bf16_rowwise has no fake impl, and the first compiled generate therefore cannot be traced. Isolated away from the product to be sure of the mechanism: quantise one Linear three ways on this box and compile each. KernelPreference.TORCH eager ok, compiled ok KernelPreference.AUTO eager ok, compiled FAILS on mslk.f8f8bf16_rowwise library default eager ok, compiled FAILS the same way So the pin is correct and necessary, and the only gap is that the hosted path never got it. _validate_checkpoint checks scheme, granularity, base model, min_features, exclude tokens and fast_accum, but not this. Rewriting the preference on load is safe: it selects a matmul kernel, it is not weight data, so the tensors stay bit-identical and the checkpoint's own state_dict_sha256 still describes them. It is also the faster path compiled, since the opaque extern call blocks inductor quantize fusion. Why this went unnoticed: the pre-quant repos are private, so nothing that could not read them ever took this path. It becomes the default the moment they are readable. Verified end to end at 1024x1024, 8 steps, speed_mode default, on the published checkpoint: before HTTP 500 at generate, diffusion.generate_failed after pinned 239 weights to the plain-torch fp8 kernel, load 10.0 s, transformer_quant fp8 with compiled engaged, cold 7.9 s, warm 0.87 0.87 0.87 0.86 0.87 s That warm number also beats the 1.4 s recorded for this shape on 2026-07-26. 159 prequant and transformer-quant tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cover the fp8 kernel pin so the 500 cannot come back quietly The hosted pre-quant re-arming MSLK was only visible as an HTTP 500 on a compiled generate, which needs a GPU, a readable private repo and an importable mslk to reproduce. None of that is available in CI, so the pin would rot unnoticed. Three hermetic cases on _pin_kernel_preference instead: AUTO weights are rewritten and already-TORCH ones are left alone (counting only what changed), a weight that refuses the assignment does not sink the whole load, and with no torchao enum available the checkpoint is left exactly as saved. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: tighten comments across the image generation, video and training code Rewrite the comments added by this branch to be shorter and clearer: collapse multi-line explanations into one or two lines, drop comments that only restate the code, and keep the rationale that explains why a choice was made. No code changes. * Studio: let the pipeline-class guard survive a host with no diffusers Backend CI installs the CPU-only dependency set, which has no diffusers, and runs without a GPU. 13 tests failed there for three separate reasons, one of which is a real product bug rather than a test-environment gap. assert_pipeline_class_available did a bare "import diffusers", so on any host without it the call raised ModuleNotFoundError instead of the ValueError its own contract promises. /images/download-plan catches only ValueError and FileNotFoundError, so that escaped as a bare 500 with the message lost, which is the exact failure the guard exists to prevent. It is reachable in production: the native sd.cpp engine serves GGUF picks on a CPU or Apple host where diffusers is never installed. Absent diffusers now returns, since the check answers "is the installed diffusers new enough for this family" and there is no version to judge; a pick that genuinely needs diffusers still fails in the loader with its own message. The rest were tests asserting through gates unrelated to what they cover: - Three cond_cache_dir route tests drive a DiT family, which the start route refuses without an accelerator. They now take the existing dit_train_host fixture, written for this case, so they keep testing the schema on every host. - test_download_plan_forwards_the_load_time_controls stubs the diffusers planner but sends a GGUF pick, which routes to the native planner on a GPU-less host. It now pins the engine, so it tests kwarg forwarding rather than the hardware. Engine selection keeps its own tests. - The pipeline-class guard test needed a real diffusers only for its sweep over every shipped family. That half is split out and skips on its own; the stub-driven refusal needs no diffusers and still runs. Added a test for the absent-diffusers contract above. 409 pass on a GPU host with diffusers; 301 pass and 1 skips with diffusers blocked and no GPU. * Studio: cut the image generation comments down further Second pass over the comments this branch adds. Collapse the multi-line blocks that still read as paragraphs, rewrite the longest one-liners so they say the same thing in fewer words, and drop a stale comment that had drifted away from the constant it described. No code changes: only comments and whitespace. * Studio: workflow rail, sidebar Video row, and Train settings polish Sidebar - Pin Video under Images by default; persist store bumped to v3 so an untouched install adopts it while a custom arrangement is left alone. - Add "Customize sidebar" to the end of the More flyout, opening Settings > Appearance scrolled to the sidebar nav section. Images - Replace the workflow dropdown with an icon rail down the left edge: all seven workflows visible, keyboard nav, tooltips carry the labels. - Workflows stay selectable with no model loaded, so one can be set up before picking a model. Generate is already gated on a loaded model. - Video moves out of the Create/Train strip into a link at the far right, with a matching Images link on the Video page. - Wider gutters around the settings column, headings matched to Train. Train - Field guidance moves into "i" tooltips; only state that limits a control stays on the page. Steps, LoRA rank and the rest gain hints. - /info reports params, qlora_vram_gb, gated and note as fields, so the family note renders as chips. vram_note is rebuilt from them unchanged for older clients. Shared - Number steppers appear on hover or focus. - Lighter control border (#e9e9e9) on Images and Video, light mode only. * Studio: move Advanced inline on Images and Video - Replace the top-bar toggle and right-docked Advanced panel with a disclosure under Seed, so load-time tuning sits with the settings it affects and opening it no longer shrinks the preview canvas. - Video uses the bordered variant, since offload and memory decide whether a model fits at all. - Open state persists per page in localStorage. - Tighten the Steps unit trigger: 14px chevron, less right padding. * Studio: full width media pages and layout polish - Drop the 1100px cap on Images Create, Images Train and Video, so the preview canvas grows with the window instead of sitting in a band. - Align the workflow rail to the model selector label, widen the gap before its divider, and restore the divider itself. - Advanced: more room above and below, icon and label sized to the slider rows, and the same quiet row on Video as on Images. - Give Video a pane heading and description, matching Images. - More space above Seed on both pages. - More no longer takes the active style when the current page is one of its own rows. * Studio: fix the Extend side toggles The resting outline used a ring, and index.css blanks the ring on the button holding mouse focus, so the side you just clicked lost its border until focus moved. Use a border for the outline and leave the ring to focus-visible. Also separates the two states properly (the off state had no surface of its own), drops the border in dark mode as the inputs do, and adds the aria-pressed and focus-visible styling the buttons were missing. * Studio: drop the border on the Extend side toggles Light mode should carry no border either, so the fill alone marks the state: a muted surface when off, a primary tint when on. Matches the borderless treatment dark mode already had. * Studio: move the Images workflows into the sidebar The workflow switcher was a vertical icon rail on the page. It now lives in the sidebar under Images, so the page keeps that width for the canvas. - Workflows list under the Images row, with a chevron to fold them away. Hovering the row peeks them in a flyout on the standard menu surface. - Clicking Images while already there toggles the list instead of navigating. - The listed workflow carries the highlight, so the Images row drops it. - Create takes its own icon: it was sharing the New chat pencil. - Model hub moves above Projects. Persisted layouts bump to v4 and only adopt the new order where the stored one is still a shipped default. - Images and Video content both start at 32px, clear of the sidebar. * Studio: simplify the Images workflow submenu Dropping the hover flyout: two ways to reach the same seven workflows, one of them overlapping the list right below it, read as clutter. - The flyout is gone. The workflows are rows under Images and nothing else. - On the Images page they are always open, since they are that page's switcher. Elsewhere they stay folded, and the row's chevron opens them. - Create takes the sparkles icon. - Create and Train both pad their settings column to 40px a side, so the two tabs line up with each other and with the model selector. * Studio: dock the generate action and tidy the Create controls The primary action sat at the foot of a long scroll, so it was off screen until you scrolled for it. - Generate, Video's Generate and Train's Start training float at the bottom of their settings column. No bar behind them: hover lightens the fill rather than thinning it, and the disabled state is opaque, so the controls underneath never show through. - Aspect ratios read as names: Square (1:1), Widescreen (16:9) and so on. - Width and height replace their sliders with two compact boxes. Type a size or pick one from the menu; the value still snaps into range and still drives the locked ratio. - Negative prompt gets a quiet disclosure under the prompt. It was there before but only above guidance 0, which is not the default, so it never showed. - The info "i" is smaller across the settings UI. * Studio: reveal Images and Video field hints on hover The "i" next to every field label sat there permanently, which made a column of settings read as busier than it is. Scoped to the diffusion pages by CSS rather than per component: the hints come from the page's own Field and SelectRow, the train panel's FieldLabel, and the chat ParamSlider, which is shared and should not change for chat. Reveals when the pointer is inside the field, on focus-within for keyboard, and stays inert while hidden. InfoHint grows a data-slot so the rule has a stable hook, rather than keying off its aria-label. * Studio: one-line sliders, edge fades, shared negative prompt - Steps, Guidance and the rest put label, track and value on one row. ParamSlider is shared with chat, so this is an opt-in prop and chat keeps the stacked layout. - The settings columns fade at whichever edge they run past, as the sidebar and model picker do, instead of cutting off. New useScrollFades hook drives all three. - Negative prompt moves to a shared component and Video picks it up, so both pages collapse it the same way. - Right gutter sits closer to the rule, Steps takes a bigger break above it, and the Advanced rule sits between its neighbours rather than up against the field above. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: add the API monitor to the sidebar nav Sits under More, below Export. Points at the monitor page, not the API keys dialog the profile menu opens. No store version bump: an id missing from a stored layout is appended with its default pinned state, so existing sidebars gain the row at the end and keep their own order. * Studio: take main's copy of the sidebar spinner test The merge kept this branch's older version, which pins the Recents row to h-[33px]. Main already relaxed that to any height, since row density moves independently of the trailing column the test is about. * Studio: fix FLUX.2 klein size resolution, HunyuanVideo tier routing, and the personalization save Five fixes found while auditing the model registries against the live Hub. FLUX.2 klein 9B loaded against the 4B config. One family covers both klein sizes and defaults to 4B, relying on the base_model card tag for the real base, but the trust allowlist only had klein-4B, so the correct tag was discarded and a 9B GGUF (inner_dim 4096) was loaded against a 4B config (3072). That surfaced as a bare shape mismatch from inside the GGUF quantizer naming neither the file nor the repo. Adds the missing allowlist entries and a header-level size check that fails early with a legible message; the check is fail-open, so an unreadable file, a non-FLUX.2 family, or an unmapped base leaves the load exactly as it was. The sd.cpp text encoder rule matched the literal "klein-9b", so klein-base-9B was handed the 4B encoder. HunyuanVideo-1.5 720p checkpoints routed to the 480p family. Only the literal 720p_t2v path was aliased, so 720p_i2v and every GGUF repack fell through to the generic token and inherited the 480p base repo, which also supplies the VAE and text encoder. The tier is baked into the weights (transformer target_size 640 vs 960, scheduler shift 5.0 vs 9.0) and the two bucket lists are disjoint, so this ran the whole pipeline off-tier. Resolution presets that the checkpoints were never trained for. Wan2.2 TI2V-5B is 720P-only upstream (SUPPORTED_SIZES is exactly 704x1280 and 1280x704, asserted in generate.py), and the HunyuanVideo 480p square preset 624x624 is not a bucket of its tier; 640x640 is. Personalization could never be saved. The frontend ships eight sidebar nav ids and refills every missing one on each save, but the backend Literal had seven and no "api", so every PUT to /api/settings/personalization returned 422. Aligns the server defaults and the Literal with the shipped layout. A catalog row pointing at a file that does not exist. unsloth/Qwen-Image-2512-FP8 holds torch prequant .pt checkpoints, not qwen-image-2512-fp8.safetensors, and fp8 is denied for this family anyway because it renders black, yet the row was the auto-route target on any GPU above 40 GB. Also marks FLUX.1-schnell, Krea-2-Turbo and the two Ideogram repos gated: all four are gated on the Hub today, and schnell being Apache-2.0 does not make it ungated. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: stop a case-only duplicate upload destroying an image, and size the load against the right GPU Two fixes from the review round. A dataset upload containing two names that differ only by case silently loses one on Windows and macOS. The in-batch duplicate check is an exact string compare, so "Cat.png" and "cat.png" both pass, but on a case-folding filesystem they are one destination: the commit step moves the first staged part aside, writes the second over it, then deletes the backup, while the response still reports both as uploaded. Captions collapse the same way and never even reach the image branch. No single folder can hold such a pair, but the Windows and macOS open dialogs flatten search and Recents results across folders into one multi-selectable list, which is a normal way a LoRA dataset gets assembled. Rejecting the pair everywhere would regress Linux, where the two really are different files with their own sidecars, so the check probes the filesystem once per process instead of keying off sys.platform: macOS also ships case-sensitive APFS volumes, and a Linux host can keep its Studio home on an exFAT or NTFS mount. A failed probe answers "case-sensitive", which leaves today's behaviour untouched. The quantization fit budget was sized against the wrong card under a reordering CUDA_VISIBLE_DEVICES. A bare "cuda" load lands on visible ordinal 0, but the reducer took the lowest PHYSICAL index, which the nvidia-smi path reports as index_kind "physical". Under "3,1" that sizes against GPU 1 while the pipeline loads onto GPU 3. The hook's own interface doc already promised the lowest visible ordinal. Ranks by visible_ordinal, falling back to index only for an older backend that omits it. * Studio: rework the diffusion dataset panel and align the pane gutters Upload: - one Upload button beside the dataset name, with an upload icon - picking files uploads them, no second confirm click - Add button beside the Training images dropdown, so images can go into a set that already exists - fall back to the upload form when a selected dataset no longer resolves Labeling grid: - two columns at any width; sm:grid-cols-3 crushed the tiles in a fixed width column - pin the caption size so the Textarea's md:text-sm does not outsize a tile - name the tile hover group; a bare one revealed every tile's Remove at once - Remove is an icon button on the image, not a word over the artwork - drop the amber tile fill, keep the No caption label - match the header, toggle and status text to the section's type scale Gutters: - the run area and the preview canvas now sit 40px off the rule, the gutter the settings column has off the page edge * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: give the dataset preview per-image hover and remove, and wrap it onto rows - each thumbnail is its own button, so it can carry an X that removes just that image; the strip was one button and could not nest a second control - hover lightens the image instead of drawing a border - wraps onto more rows rather than scrolling sideways, which also keeps the bottom corners rounded - samples 12 thumbnails, up from 8, now that a second row is available - a delete refreshes the panel's dataset counts * Studio: show the workflow icon beside the Images and Video headings Images reads it off WORKFLOW_TABS, so it is the same icon the sidebar submenu shows and it follows the active workflow. Video's heading is also Create, so it takes the Create icon rather than the nav row's film slate. * Studio: reject a duplicate LoRA id on the diffusion load path too DiffusionGenerateRequest already refuses a repeated adapter id, with a comment naming the hazard: _resolve_lora_set suffixes colliding adapter names, so the same id resolves the SAME adapter twice and set_adapters stacks both copies past the per-adapter weight bound. DiffusionLoadRequest bounds only the list length, and it matters more there. Generation-time stacking spoils one image; on the load path the adapters are baked into the quantized build before compilation, so the unintended combination rides every image until a reload. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: teach the GGUF reuse test double about holds_no_vram The resident-GGUF fast path consults llama_backend.holds_no_vram before asserting CHAT ownership, but two tests hand it a SimpleNamespace built before that attribute existed, so they raised AttributeError and returned a 500. A real LlamaCppBackend exposes it as a property; the doubles now carry it too. Product code is unchanged: these pass on main and were the branch breaking its own test, not a defect in the guard. --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: michaelhan <michaelhan2050@gmail.com> Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: Michael Han <107991372+shimmyshimmer@users.noreply.github.com> | 1 个月前 | |
Studio: add image generation, editing workflows and LoRA training with Unsloth GGUFs (#6763) * Tighten comments in the image stack tests and scripts * Close video single-file, training reservation, and image mount-resume gaps Route on-device single-checkpoint video folders through the single_file loader: a bare local .safetensors directory (no model_index.json) is advertised as a pipeline with no filename, so validation rejected it before it could load. Reinterpret the pick as a single_file load of the sole checkpoint, mirroring the image load route. Treat a reserved-but-not-yet-spawned LLM training start as active in is_training_active() so /images/load, /video/load, and /diffusion/start cannot race the reserved run for VRAM during the pre-spawn free window. Mirrors the diffusion training service reservation. Resume an in-flight image generation on the Images page mount: probe generate-progress, re-enter the poll loop, and refresh the gallery on completion so a run started elsewhere is reflected and its saved image appears without a manual refresh. Seed resident image defaults from the resolved base_repo rather than a possibly path-shaped repo_id so the first resident generation uses the right recipe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Publish image generation active state before pre-denoise setup generate() assigned self._gen only at the pipe() call, after deferred compile, LoRA resolution/application, and ControlNet download/build had run. Across that setup window generate_progress() reported inactive even though _generate_lock was held, so a reloaded page's mount probe showed idle and let a second generate queue behind the first. Publish an active step-0 _GenState the moment the generation lock is acquired, before the setup work, and clear it in the outer finally so a setup-time error cannot leave the UI stuck active. Mirrors the video backend's queued phase and the training start guard. * Studio: fix diffusion install ownership, dataset upload atomicity, gallery pagination, and teardown races install_sd_cpp_prebuilt: only write the .unsloth-studio-owned marker when the install created the target directory or it was empty. Adopting a pre-existing, unowned, non-empty directory (a user's own stable-diffusion.cpp checkout) made it eligible for the uninstaller's recursive delete. routes/training upload: make the multi-file promotion transactional. Back up each displaced original and roll every destination back on any failure, so a mid-loop rename error can no longer partially overwrite the live dataset. routes/training _resolve_dataset_folder: reject a symlinked dataset directory and prove the resolved folder stays under the datasets root, so image read/caption/delete cannot escape the root through a link. routes/training delete: escape glob metacharacters in the thumbnail filename so deleting an image named like [ab].png removes only its own thumbnails. image_gallery / video_gallery listing: filter records against the response schema inside the pager via a valid callback, so offset/limit/has_more all count over accepted records. A leading schema-invalid record no longer returns an empty page with has_more=true and stalls infinite scroll at offset 0. image_gallery / video_gallery save: publish via a temp file plus atomic rename (the sidecar is the video pair's commit marker) and clean up on failure, so a partial write never surfaces a truncated PNG or strands an orphan MP4. diffusion_train_common discovery: treat an empty caption sidecar as a metadata tombstone that still falls through to the dreambooth instance prompt, so clearing every metadata caption no longer fails with no captioned images found. diffusion backend unload: wait for an in-flight denoise to exit before tearing down process-wide patches and state, mirroring the load path. diffusion_engine_router: serialize the whole check/unload/publish transition so a concurrent selection cannot return the engine being unloaded. uninstall.ps1: gate the default sd.cpp process stop on the owner marker so a user's own sd-server is not terminated for a directory we then keep. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reject native batch seeds outside the JSON-safe range * Refuse sd.cpp install into unowned non-empty target dir When the install target already exists, is non-empty and lacks the .unsloth-studio-owned marker (a user's own stable-diffusion.cpp checkout, or unrelated files beside a custom Studio root), install() previously still extracted the release into it. Skipping the ownership marker only stopped the uninstaller from deleting the directory; extraction still merged binaries into the user's working tree and could overwrite same-named files. Fail up front with a clear message pointing the user at a fresh/empty location before any download or extraction, leaving their directory untouched. Update the ownership test suite to assert the refusal. * Studio: gate dataset uploads on the symlink check and surface local video single-file checkpoints * Tighten comments and docstrings added by the image-generation fixes * Studio: close arbiter load-registration race and surface native progress + local pipeline folders Publish native sd.cpp generate progress (_gen) before LoRA resolution so a reload probe reads active during setup, matching the diffusers path. Register the diffusion/video GPU load under the arbiter lock (acquire_for now takes a register callback) so a competing acquire cannot evict an owner before its load is marked in-flight and let two loaders allocate VRAM at once. Admit local diffusers pipeline folders (root model_index.json, weights in component subdirs) in the local model scan so they reach task tagging and the On Device picker. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: include marker-owned custom sd.cpp roots in the uninstall stop scan; harden Pester install against the nuget.exe PSGallery bootstrap * Studio: surface local pipeline scan roots, tag single-file checkpoints by filename, mark companion-only pipelines partial - _scan_models_dir: admit a scan folder that is itself a diffusers pipeline (root model_index.json, weights in transformer/ vae/ subdirs). _is_model_directory rejects such a root, so the child scan would list the component subdirs as bogus models and hide the real pipeline; treat the root as one model via _local_pipeline_index. - _local_is_diffusers / _local_model_task: include the sole checkpoint filename in the family-detection needles (_local_family_needles, resolved via resolve_local_single_file). A generically named folder holding one loadable qwen-image-*.safetensors / ltx-*.safetensors identifies its family only from the filename; the load route already resolves that file, so tag it or the task-scoped picker (which rejects task=null) hides the on-device model. - list_cached_models: mark a companion-only base snapshot partial. A GGUF image load prefetches the base repo's VAE / text-encoder / model_index.json but skips the transformer (the GGUF supplies it); the snapshot has a pipeline manifest yet is not a loadable BF16 pipeline, and _cached_repo_partial misses it. _repo_pipeline_missing_denoiser flags a pipeline snapshot whose transformer/ or unet/ component carries no weight, so the picker drops it instead of advertising it as fully on-device. * Studio: preserve foreign gallery files, force safetensors on remote ControlNets, and close dataset/seed/GPU gaps Gallery clear/delete now scope to Studio-owned files: image_gallery and video_gallery skip PNGs / MP4s without a readable recipe (a hand-dropped or orphan file the listing already hides), so clear() and a guessed-id delete no longer destroy files the gallery never surfaced. Remote ControlNets now force use_safetensors: a bare owner/name reaches from_pretrained without the base trust gate, and the Hub scan fails open when unavailable, so requiring safetensors closes the pickle deserialization vector. POSIX uninstall now stops resident sd-server / sd-cli under an owned sd.cpp root before removing the tree (marker-gated), mirroring the Windows stop-before-delete scan; a live native server no longer survives unlinking its binary. Diffusion dataset containment: the training-start read path and the discovery picker route bare names through the protected resolver, so a symlinked dataset is rejected / not advertised like the caption/delete routes already do. Uploads gain the inference decode guard (oversized real images 400 before OOMing the trainer) and dataset upload/caption/delete/import are blocked with 409 while a diffusion run is active. JSONL readers (trainer + routes) tolerate non-object JSON and invalid UTF-8 instead of raising AttributeError / 500. LoRA family compatibility is enforced in the shared resolver, not only the picker, so a direct API client cannot apply a mismatched-family adapter. GPU arbiter gains release_if so the image/video unload idle-check and release are atomic against a concurrent same-owner load's registration. Native batch recipes persist the base batch_seed and restore replays from it, so a native batch_index>0 image no longer advances its seed twice. FLUX.2-klein selects its sd.cpp text encoder by variant (4B -> Qwen3-4B, 9B -> Qwen3-8B) instead of the single family default. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: track the loaded GGUF filename so native companion resolution reproduces the load identity * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: gate local-pipeline image tagging on a real family; validate video sidecars before delete/clear * Studio: tighten image-generation fix comments and docstrings * Studio: gate gallery serve/export on ownership; keep image progress active until persisted; reserve diffusion training before the dataset scan * Studio: restore Reapply target on async image/video load errors; recheck training state before dataset commit; reclaim partial sd.cpp installs on retry Images/Video: a background model load that fails AFTER starting (error/eviction during download) leaves the previous pipeline resident, but handleLoad had already overwritten lastLoad.current with the failed pick, so "Reapply to loaded model" reloaded the failed model. Carry the prior Reapply target into the poller and restore it on the async error/null paths, mirroring the quant rollback. Training: an in-flight diffusion dataset upload passed _require_diffusion_dataset_mutable() at entry but could still commit files after a concurrent /diffusion/start reserved the training slot, mutating the dataset underneath the trainer. Re-check the interlock immediately before the commit phase; a 409 there leaves the staged temps for the finally to clean. sd.cpp install: an interrupted extraction (disk full, killed process, a raising post-extract cudart fetch) left the target non-empty with no owner marker, so the next lazy install tripped the "not a Studio-managed directory" refusal and wedged native install. Write the ownership marker before the partial writes when the target is reclaimable, so a retry recognises the debris as ours and re-extracts. * fp8 DiT quant: floor the dynamic activation scale with activation_value_lb An all-zero activation token row makes the dynamic per-row fp8 scale 0, which turns the quantized data to NaN and the render to black frames on torchao's plain-torch kernel path. The fused fbgemm/mslk quantize kernels clamp zero rows internally, so the bug only reproduces on machines without them, which is most user environments. Zero rows are real inputs, not a corner case: Wan 2.2 zero-pads its text conditioning, and Hunyuan-1.5 and Qwen-Image regenerate zero rows inside their transformer blocks every step. Pass activation_value_lb=1e-12 to Float8DynamicActivationFloat8WeightConfig whenever the installed torchao supports the kwarg (Float8Tensor rework, 0.13+), checked via inspect.signature so older torchao keeps exactly the current behaviour; the existing Float8MMConfig fallback chain is unchanged. Verified on GPU: with the forced plain-torch kernel path a zero-row input NaNs without the floor and stays finite with it, and end to end on HunyuanVideo-1.5 fp8 goes from a solid black frame (LPIPS 1.00) to a normal render (LPIPS 0.225); on Wan the floor matches the condition_embedder exclusion (LPIPS 0.211 vs 0.206). Same-seed renders with fused kernels present are unaffected, and pre-quantized fp8 checkpoints stay valid since weight scales are untouched. * Wire hosted pre-quantized DiT checkpoints into the image families Point prequant_repos for flux.1, flux.2-klein, flux.2-dev, qwen-image (int8 only there; fp8 is family-denied), z-image and krea-2 at the unsloth/<Model>-FP8 Hub repos carrying gate-validated int8 and fp8 transformer checkpoints, so the fast quant path loads the small pre-quantized file instead of materialising the dense bf16 transformer and quantising on device. Measured on FLUX.2-dev int8: build peak drops from 60.7 GB (dense + quantize) to 30.7 GB (hosted prequant), identical 30.7 GB resident after either path since loading a checkpoint is bit-identical to on-the-fly quantisation. The hosted repos name files <Model>-<SCHEME>.pt, so resolve_prequant_source now derives that model-name filename from the repo id (scheme suffix stripped case-insensitively) and carries the legacy transformer_<scheme>.pt as a fallback the resolver tries when the primary 404s, keeping older repos loadable. Wiring a repo also exposed a fallback hazard: with a prequant source present, the dense-fit preflight used to be skipped entirely, so a failed prequant download would fall through to the dense bf16 load the memory plan never budgeted, OOMing after eviction. The preflight now always runs and gates an allow_dense_fallback flag through _load_dense_quant_pipeline: a dense misfit still skips the fast path when no prequant exists, but with one it proceeds and a prequant failure raises to the GGUF build instead of loading dense. The same flag is set when the auto-policy replans an offloaded GGUF against a prequant-sized transient. Tests updated to the new filename convention plus new coverage for the derivation and the legacy-name fallback; the prequant-skips-refit test now asserts the re-check runs and forbids the dense fallback. Verified end to end on GPU: z-image int8 resolves the hosted repo, downloads the model-name file and renders (6.8s load, 5.9 GB peak). * Route krea-2 through its per-component loader on the transformer-quant fast path _assemble_pipe used Pipeline.from_pretrained for every family, but the krea repo ships transformers-5.x configs and no top-level tokenizer files, so the tokenizer dies with vocab_file=None. The pre-quantized checkpoint loaded fine and then the assembly crashed, dropping the load to the GGUF build, which krea-2 cannot take (Krea2Transformer2DModel has no from_single_file). Assemble per-component via load_krea2_pipeline like the pipeline-kind and single-file paths already do. Verified live: Krea-2-Turbo int8 and fp8 hosted prequant loads now assemble and render through the Studio images tab. * Keep Qwen-Image's text-stream linears bf16 on int8 (short prompts break torch._int_mm) Qwen-Image's MMDiT runs every text-stream Linear at M = actual prompt tokens: the Qwen2.5-VL embeds are not padded to a fixed length like FLUX's 512-token T5. A short prompt (13 tokens) or the near-empty negative prompt drives torch._int_mm below its M > 16 floor and the first denoise step raises 'self.size(0) needs to be greater than 16, but got 13' (measured on B200 through the Studio images tab). Add per-family int8 exclusions (txt_in, add_q/k/v_proj, to_add_out, txt_mlp) for qwen-image and qwen-image-edit, threaded through exclude_tokens_for_scheme(scheme, family) and the prequant checkpoint validation, so a checkpoint baked under the old token list is rejected and re-quantised instead of loaded crashing. The text stream runs at M = tens vs the image stream's M ~ 4k, so the exclusion costs nothing; the rebuilt hosted checkpoint gates 28/28 PASS with LPIPS mean 0.057 (was 0.069). * Harden the diffusion memory plan against transient free-VRAM undercounts A cold FLUX.2-dev int8 load on an idle 183 GB B200 planned offload=model (companions exceed budget) and silently served the GGUF as-is; the identical retry went resident and engaged the hosted prequant. The plan arithmetic was byte-identical across both loads (required 90,228 MiB, resident needs free of about 124 GB); the only divergent input was torch.cuda.mem_get_info, which is device-wide and instantaneous: a transient foreign CUDA context briefly held about 100 GB at the first snapshot, and the planner trusted that single read. Three changes: - settled_snapshot_device_memory: on cuda, synchronize + empty_cache (best-effort) and take the MAX free over up to 3 spaced reads. A transient can only shrink free, so the max rejects transient undercounts while a persistent tenant still caps every read. _plan_memory now uses it. - plan_fits_total_capacity + one replan retry: when the dense/prequant candidate fits TOTAL device capacity under the standard reserve and the 0.85 resident margin, an offload verdict can only stem from the free reading, so the loader re-snapshots and replans once before declining the fast path. Explicit balanced/low_vram modes skip the retry (they offload by mode). - diffusion.transformer_quant_declined log line with required/budget/free and the plan reasons, so the next decline is diagnosable from the server log (previously silent). Verified: cold FLUX.2-dev int8 first load in a fresh server now engages the hosted prequant resident (offload=none). * Add FLUX.2 Klein and FLUX.2-dev DiT LoRA training Register flux.2-klein and flux.2-dev in the DiT trainer following the upstream DreamBooth references: latents train patchified and batch-norm normalized from the VAE posterior mode, the packed forward reuses step-invariant position ids, and the guidance vector (3.5) is gated on the variant's guidance_embeds config. Conditioning stacks load per variant (Mistral via Flux2Pipeline for dev, Qwen3 via Flux2KleinPipeline for Klein) and are encoded and freed before the transformer lands on the device. The fused single-stream to_qkv_mlp_proj joins the attention projections in the LoRA targets; the single-stream out projection stays dense because its to_out suffix would also match the double-stream ModuleList container. Wire both families through the training registry (family set, labels, VRAM notes, rank 16 / lr 1e-4 defaults, bf16-only preflight), mark them trainable with train base repos in the family registry, add FLUX.2-dev to the gated-repo token check, and trust both official bases for training downloads. Verified on B200: 30-step klein int8 (19.6s) and nf4 (20.9s) and dev int8 (52.0s) runs train with finite decreasing loss and the saved adapters apply on the bf16 base pipeline (weight 0 reproduces the base image exactly, weight 1 visibly restyles it). * Support LoRA adapters on torchao int8/fp8 quantized image pipelines Adapters are baked at load time: they attach to the dense transformer, then quantize_ converts only the frozen base linears (the lora_ side path is excluded by name), then the loader compiles. Post-quant PEFT injection is not possible on a manually quantized module, so the prequant shortcut is skipped for a baked load and the memory plan is sized for the dense build (force_dense on the quant candidate). At generation time the baked topology is frozen: weight tweaks and disabling (scale 0 reproduces the quantized base exactly) go through set_adapters, while adding or removing adapters returns a clean 400 telling the client to reload with the new selection. supports_lora now returns True for int8/fp8 diffusers loads (checked before the gguf-kind early return, since the quant fast path keeps the picker kind); nvfp4/mxfp8 and GGUF-via-diffusers stay blocked. The load request model takes an optional loras list, threaded through begin_load on both engines (native ignores it and keeps applying LoRA at generation). Verified end to end on GPU: Z-Image GGUF picker + int8 + trained adapter loads through the API, bake marker logged, weight 1.0 vs 0 renders differ visibly, weight 0.5 accepted live, unknown adapter rejected as 400. Affected suites: 296 passed. * Add FLUX.1 Krea dev to the image model catalog Krea's guidance-distilled FLUX.1-dev finetune keeps the exact dev layout, so it runs under the existing flux.1 family unchanged. Wire it up end to end: - Catalog group with the gated official bf16 pipeline and the open QuantStack GGUF quants; the gated artifact is skipped on auto-routing when undownloaded. - Trust the official repo for non-GGUF from_pretrained loads, next to the other black-forest-labs bases. - Generation defaults: 28 steps at guidance 4.5 per the model card. The generic "krea" defaults key (Krea-2-Turbo's 8-step no-CFG recipe) used to swallow the id, which would have produced garbage output; the new flux.1-krea key precedes it on both the backend table and the images page table. - The flux.1 prequant checkpoints are schnell-based; the loader's baked base_model_id validation refuses them for the Krea-dev base, so int8/fp8 requests dense-quantize instead (covered by existing prequant tests). * Resolve pre-quantized checkpoints per base variant One family entry covers several published variants whose weights differ (flux.1: schnell, dev, Krea-dev), but prequant resolution was keyed on (family, scheme) alone, so only the default base could ever be served: the loader's baked base_model_id validation correctly refused the schnell checkpoint for dev and Krea-dev bases and every such load paid the dense download plus on-the-fly quantise. Add an optional prequant_variant_repos table on DiffusionFamily as (base_repo, scheme, repo_id) triples and thread the resolved base repo through resolve_prequant_source / usable_prequant_source and their three call sites (load fast path, memory-plan probe, auto-policy candidate). A base without its own entry keeps returning the family default, preserving the existing refuse-then-dense behavior exactly. Wire the flux.1 variants: the gate-validated unsloth/FLUX.1-dev-FP8 checkpoints (built in the earlier campaign but never reachable) and the new unsloth/FLUX.1-Krea-dev-FP8. * Add the Lumina Image 2.0 family to the image catalog Alpha-VLLM/Lumina-Image-2.0 is a 2.6B single-stream DiT with a Gemma2-2B encoder and a standard 16-channel VAE, all transformers-4.x-compatible, so the generic from_pretrained pipeline path loads it as a new lumina-2 family: - Family entry (Lumina2Pipeline / Lumina2Transformer2DModel), aliased to lumina-image-2.0 / lumina-image-2 / lumina2. No bare lumina alias: Lumina-Next checkpoints are a different arch and must stay unknown rather than crash mid-load. bf16-only upstream, so the fp16 fallback stays off like z-image. - Trust the official repo for non-GGUF loads; bf16 component table entry (ships fp32, ~5.2 GB transformer + 5.2 GB encoder bf16-resident). - Generation defaults 50 steps / guidance 4.0 per the model card, and the generate call passes the card's cfg_trunc_ratio=0.25 itself (family-gated, signature-gated): the pipeline default (1.0) runs the CFG double-forward on every step and oversaturates output. - Catalog group with the single ungated bf16 pipeline artifact (11 GB resident) plus routing assertions; images page defaults row. - No GGUF artifact: none exists upstream (only finetune/LLM quants), so the dense transformer_quant fast path (GGUF-kind-only) stays unreachable for now. Offline probes of the future prequant campaign: int8 and fp8 both engage and render cleanly (fp8 LPIPS 0.11 vs bf16, int8 0.33 from 50-step trajectory drift with intact quality), so neither scheme is family-denied. * Wire the hosted Lumina Image 2.0 int8/fp8 checkpoints Gate-validated against same-seed bf16 renders (28/28 pairs per scheme, zero failures): int8 LPIPS mean 0.146 / SSIM 0.937, fp8 LPIPS mean 0.116 / SSIM 0.946. Uploaded to unsloth/Lumina-Image-2.0-FP8 following the existing checkpoint repo conventions. * Add the HunyuanImage 2.1 family to the image backend The hunyuanvideo-community diffusers mirror carries the full stack in standard layout: a 17B dual-stream DiT (32.5 GB bf16), a Qwen2.5-VL text encoder, a ByT5 glyph encoder, the 32x HunyuanImage VAE, and guider/ocr_guider components (AdaptiveProjectedMixGuidance) that diffusers 0.39 loads natively, so the generic from_pretrained pipeline path covers everything with no per-component assembly. Family notes: - The call's guidance knob is distilled_guidance_scale (there is no guidance_scale kwarg), so cfg_kwarg routes the UI value there; real CFG runs inside the repo's guider at its baked scale. Defaults follow the card recipe: 50 steps, 3.25. - 2K-native: verified live at both 1024 and 2048. - Coexists with the HunyuanImage-3.0 structured exclusion (3.0 has no diffusers pipeline and stays excluded with its stated reason). - int8/fp8 dense quantization verified live (LPIPS 0.186 both vs same-seed bf16); a short prompt does not trip the int8 torch._int_mm minimum on this arch, so no family exclude entry is needed. - bf16 component table for the memory planner: (32.5, 16.3, 0.8) GB. * Surface HunyuanImage 2.1 in the image model catalog Catalog group with the open bf16 mirror pipeline (~50 GB resident, so a bare click on a consumer card routes to the QuantStack GGUF quants, which load and render through the generic GGUF path, verified live) plus the images page defaults (50 steps, guidance 3.25 feeding distilled_guidance_scale). * Add the HiDream-I1 family to the image backend A 17B MoE DiT (16 double + 32 single layers, 4 routed experts) with four text encoders, on HiDreamImagePipeline (diffusers 0.39). One family covers the open Full / Dev / Fast repos (same arch); per-variant generation defaults follow the upstream inference recipes (Full 50 steps at guidance 5, the distilled Dev 28 and Fast 16 guidance-free). The repos name a Llama-3.1-8B text_encoder_4 in their model_index but do not ship its weights; the official example passes the gated meta-llama repo in by hand. The loader instead assembles the component from the open unsloth mirror (byte-identical weights, already inside the non-GGUF trust gate), injected at the three pipeline from_pretrained sites, with output_hidden_states matching the official example. Memory planning counts the assembled TE4: 34.2 GB DiT + 28.8 GB encoders, ~63 GB bf16-resident. * Surface HiDream I1 in the image model catalog One catalog group with the three official bf16 pipelines (Full, plus the Dev and Fast distillations as labeled artifacts) at their ~63 GB resident size, so auto-routing keeps this a datacenter-GPU pick. city96's GGUF is deliberately not wired: the GGUF path would need the same Llama TE4 assembly for very small demand. Images-page defaults mirror the backend table with the variant keys ahead of the generic hidream key. * Pin the measured HiDream quant verdict in tests int8 and fp8 both engage and render cleanly on this family, including short prompts on int8: the routed MoE expert Linears only ever see the concatenated image+text stream (M far above the torch._int_mm minimum), so no deny entry and no family exclude tokens are warranted. Assert that so a future table edit cannot silently regress the measured behavior. * Wire the hosted HunyuanImage 2.1 int8/fp8 checkpoints Verified bit-identical to on-the-fly quantize: all 1264 state dict tensors (456 quantized) dequantize equal between the loaded checkpoint and a fresh quantize_ pass, so quality matches the runtime Dtype path exactly. Same-seed LPIPS suite means (0.35 int8 / 0.28 fp8) blend trajectory divergence with this family's own run-to-run nondeterminism (identical weights and seed reproduce a 17/255 mean pixel delta through the 50-step guider pipeline); per-case hard checks pass and the drift is compositional, reviewed visually. Uploaded to unsloth/HunyuanImage-2.1-FP8. * Fix silent LoRA drop and wasted transformer prefetch on GGUF quant loads Two live-test findings on the images load path: - transformer_quant with baked LoRAs, when the dense quantized build is declined for memory or fails: the load completed as a plain GGUF with the adapters silently dropped (HTTP success, supports_lora=false after the fact) -- wrong output with no signal. The load now fails with the recovery options (drop the adapters, free VRAM, or pick a smaller model). Weight-0 adapters still count as no bake request, and the plain no-LoRA decline keeps its silent GGUF fallback. - A fresh GGUF load on a small GPU prefetched the base repo's full bf16 transformer shards (~47 GB on Qwen-Image) because the dense-quant prefetch widening only checked scheme viability, not whether the device could ever hold the candidate resident. Gate the widening on total device capacity (reserve + 0.85 margin, the plan_fits_total_capacity bar) so a card that is certain to decline the dense build never pays the download; capable devices keep the prefetch. * Fix video progress under-reporting during load and generate Two live-test findings on the video progress endpoints: - load-progress downloaded_bytes froze mid-download: the counter used scan_cache_dir, which skips in-flight *.incomplete blobs, so it sat at the last completed blob for the whole multi-GB shard pull while the disk kept filling. Count the repo's cache directory directly (completed plus incomplete blobs, snapshot symlinks skipped so nothing is double-counted). - generate-progress reported total_steps=null / fraction=0 while step advanced: the video API only carried the native total field while the image API exposes total_steps and fraction, so one poller could not work against both. Derive the image-compatible aliases in generate_progress and declare them on the response model; the native total stays for back-compat. * Wire the hosted HiDream I1 int8/fp8 checkpoints Gate-validated: all 28 per-case pairs pass per scheme (LPIPS suite means 0.291 int8 / 0.278 fp8, in the 50-step trajectory-divergence band; CLIP delta means 0.007-0.008), and the int8 checkpoint is verified bit-identical to on-the-fly quantize across all 1615 state dict tensors (1073 quantized, max abs diff 0.0). Uploaded to unsloth/HiDream-I1-Full-FP8. * Add a pre-cast text-encoder loader for the layerwise fp8 scheme The runtime text_encoder_quant=fp8 path downloads the full bf16 text encoder and layerwise-casts it in place on every fresh load. For the heavyweight encoders (LTX's Gemma3-27B ~50 GB, FLUX.2-dev's Mistral-24B ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) that download dominates load time on a fresh machine. diffusion_te_prequant.py loads a pre-cast fp8-storage state dict instead: meta-init the encoder skeleton from the checkpoint's te_class, load_state_dict(assign=True), rebuild on CPU if non-persistent buffers stay on meta, then re-apply the same layerwise cast to install the upcast hooks. The cast is a deterministic storage transform, so the loaded encoder is bit-identical to dense-load-then-cast by construction. v1 hosts the layerwise fp8 storage scheme only: its state dict is plain tensors (torch.load(weights_only=True), no pickle execution). The dynamic-compute schemes (fp8_dynamic, int8, nvfp4) build torchao subclass wrappers at runtime and are deliberately not hosted. Checkpoints validate format, scheme, component and base_model_id before use and any problem falls back to the dense download and cast. Local path overrides reuse the DiT prequant allowlist env var. Families opt in via a new te_prequant_repos (scheme, component, repo_id) field on both DiffusionFamily and VideoFamily; the field defaults empty so nothing changes until a gate-validated artifact is wired. * Inject hosted pre-cast text encoders during pipeline assembly Wire te_prequant_pipe_kwargs into the three pipeline assembly sites: the diffusion full-pipeline branch, the diffusion transformer-only and GGUF branch (where the companion TE is the big remaining download), and the shared video assembly path before the pipeline/component split. Injection is gated exactly like the runtime cast (mode normalized to fp8, device supported, family not denied), so it can never engage where quantize_text_encoders would not; the later quantize_text_encoders call re-applies the cast idempotently and keeps status reporting truthful. With no hosted checkpoint configured the call returns {} and assembly loads the dense encoder as before. * Add the pre-cast text-encoder checkpoint builder Applies the runtime layerwise fp8 storage cast to a model's dense text encoder once and saves the cast state dict with baked metadata (format tag, base_model_id, family, scheme, component, te_class, versions) in the layout diffusion_te_prequant.py validates. Resolves the encoder class from the checkpoint's config.architectures so the recorded te_class matches what the pipeline instantiates. CPU-runnable: the cast touches storage dtypes only. * Test the pre-cast text-encoder load path Hermetic CPU coverage for diffusion_te_prequant: the checkpoint filename convention, family-table resolution by scheme and component with malformed entries skipped, resolution priority (path override, hosted repo, none) and the fp8-only scheme gate, the checkpoint validation matrix (wrong format, missing state_dict, wrong scheme, wrong component, wrong or missing base_model_id) with base case folding, the local-path allowlist refusal and missing-file fallback, and the assembly injection gating (mode, hosted entry, device support, family deny, load failure, successful injection). Also pins the te_prequant_repos field on both family dataclasses and that no family ships a hosted TE checkpoint until the campaign wires one. * Fix pre-cast TE checkpoint loading and engagement reporting Two bugs found while building the hosted checkpoints: - The builder recorded torch.__version__ (a TorchVersion object) in the checkpoint metadata, so torch.load(weights_only=True) rejected every artifact and the loader silently fell back to the dense download. Record plain strings. - Re-applying the layerwise fp8 cast to an injected pre-cast encoder raised on the duplicate hook registration, making quantize_text_encoders report the engaged cast as failed (status showed no TE quant while the encoder ran fp8). _cast_fp8 now returns early when the hooks are already installed. Also corrects the LTX TE size note: Gemma3-12B stored fp32 (~49 GB), not 27B. * Wire the hosted pre-cast fp8 text encoders qwen-image and flux.2-dev (diffusion) and ltx-2 (video) now resolve a hosted pre-cast fp8 text encoder from their unsloth -FP8 repos: - unsloth/Qwen-Image-FP8: Qwen2.5-VL-7B, 16.6 GB dense -> 8.8 GB - unsloth/FLUX.2-dev-FP8: Mistral-Small-24B, 48.0 GB dense -> 24.7 GB - unsloth/LTX-2-FP8: Gemma3-12B, 48.7 GB fp32 store -> 13.2 GB Every checkpoint verified bit-identical to dense-load-then-cast (729 / 585 / 1066 tensors, zero mismatches) and smoke-tested through the real backends with the repo engagement marker. Tests cover the wired entries, the resolver filenames, builder metadata weights_only survival, and the idempotent re-cast. * Report the compute dtype on fp8-cast encoders and inject the pre-cast TE on the dense fast path Two more findings from the hosted-TE GPU smokes: - Module.dtype reports the first floating parameter, which after the layerwise fp8 cast is the fp8 STORAGE dtype. Flux2 derives its prompt embed and latent dtypes from encoder.dtype and feeds them to randn_tensor, which has no fp8 kernel, so ANY flux.2 load with text_encoder_quant=fp8 crashed at generation (pre-existing, runtime cast included). The cast now swaps in a subclass whose dtype property reports the compute dtype; forward behaviour is unchanged. - The dense transformer_quant fast path assembles companions through _assemble_pipe, which never received the pre-cast TE injection, so the hosted encoder engaged on full-pipeline and GGUF builds but not on the fast path. Threaded through like the other two branches. Verified live on B200: qwen-image (full pipeline), flux.2-dev (GGUF picker with int8 DiT prequant), ltx-2 (video backend) all engage the hosted TE, render non-black, and report text_encoder_quant=fp8 truthfully. * Key the fp8 cast idempotency on an explicit completion marker Hook presence alone cannot distinguish a legitimately pre-cast text encoder from leftover hooks after a cast that failed mid-pass, so the early return now requires the completion marker _cast_fp8 sets once the hooks are fully installed. Leftover partial state keeps failing closed. Also tolerates non-Module encoder doubles in the hook probe and the dtype override. * Extend the fp8 TE quant to HiDream's Llama text_encoder_4 The generic quantize_text_encoders pass only covers text_encoder.._3, so HiDream's HEAVIEST encoder (Llama-3.1-8B TE4, 16.1 GB bf16) always stayed dense. TE4 is assembled separately (hidream_te4_kwargs), so the fp8 path now lives there: when the requested TE quant is layerwise fp8 and the device/family qualify, TE4 prefers the hosted pre-cast checkpoint (unsloth/HiDream-I1-Full-FP8, 8.6 GB) and falls back to dense-load-then- cast; a mid-pass cast failure reloads a fresh dense encoder instead of shipping partial state. The pre-cast loader and builder gain config_subfolder/config_overrides for standalone encoder repos whose config sits at the root and whose pipeline needs forward flags (output_hidden_states/attentions). Verified on B200: bit-identity 291 tensors (225 fp8, 0 mismatches), hosted checkpoint engages through the real backend (marker + status fp8), load 24.3 s vs 48.0 s dense, LPIPS 0.133 mean over 3 same-seed pairs vs the dense-TE render (gate 0.25), non-black frames. * Correct the ltx-2 resident TE estimate to the bf16 cast size The memory plan's bf16_components_gb held 50.4 GB for the LTX text encoder, which is the fp32 hub store of Gemma3-12B (~49 GB download), not what sits on device: the pipeline loads it torch_dtype=bf16, ~24.4 GB resident. The 26 GB over-estimate pushed the auto plan toward offload on cards that fit the real footprint. Comments and the size-table test now pin the resident semantics. * Host pre-cast fp8 text encoders for four more families Round 2 of the hosted TE set, each bit-identical to dense-load-then-cast and gated through the real backend (marker + status fp8 + same-seed LPIPS vs dense TEs): - FLUX.1 T5-XXL (text_encoder_2): 9.52 -> 5.90 GB, one artifact for schnell/dev/Krea-dev (T5 shards byte-identical across all three, verified sha256). 220 tensors, 144 fp8, LPIPS 0.109. - Lumina Gemma2-2B: fp32 hub store 10.46 -> 3.20 GB (3.3x download cut). 288 tensors, 182 fp8, LPIPS 0.041. - Z-Image Qwen3-4B: 8.04 -> 4.41 GB. 399 tensors, 252 fp8, LPIPS 0.112. NOT shared with flux.2-klein-4B: klein retrained layer 35's MLP (verified tensor diff, maxdiff 0.86), so klein hosts no entry. - Krea-2 Qwen3-VL-4B: 8.88 -> 4.83 GB. 713 tensors, 460 fp8, LPIPS 0.082. The constructor-assembled krea pipeline takes the encoder directly (load_krea2_pipeline text_encoder kwarg); the loader remaps 5.x rope_parameters and re-ties weights after assign so the rebuilt encoder matches the builder's structure. HunyuanImage 2.1 reuses the Qwen-Image artifact outright: its Qwen2.5-VL text encoder is byte-identical (every shard sha256, 16,584,414,544 bytes), recorded in the new component-level base-equivalence table the checkpoint validator consults. The injection loop now covers text_encoder.._3 so a family can host several components. Live check: LPIPS 0.123 vs dense. * Report the fp8-cast compute dtype without swapping the encoder class The dtype override swapped encoder.__class__ to a dynamic subclass, which breaks transformers' kwargs-based output recording: a fp8-cast Qwen3VLModel stopped returning hidden_states and every krea-2 generation with text_encoder_quant=fp8 crashed at encode_prompt (regression from the HiDream TE4 change; caught by the krea hosted-TE live smoke). The override is now a property shadowed on the ORIGINAL class that prefers a per-instance compute-dtype attribute, so class identity is preserved and uncast instances keep the stock behaviour. The idempotency test now pins exact class identity and the uncast-sibling fallback. * Pass the calibrated distilled sigma curve to LTX-2.3 8-step runs The 22B distilled DiT was trained against ltx_core's fixed DISTILLED_SIGMA_VALUES, but the diffusers scheduler derives 8-step spacing from resolution-shifted flow matching and lands far off at every reachable mu (second sigma 0.945-0.981 vs 0.99375, tail 0.37-0.61 -> 0.1 vs 0.725 -> 0.42 -> 0). At the distilled default step count the backend now passes the list verbatim, neutralising the scheduler's dynamic shift and terminal stretch for the call (they distort even explicit sigmas) and restoring them afterwards. Other step counts and the dev/base DiT keep the scheduler's own spacing. Live-verified on B200 through the video branch backend: the scheduler holds the exact curve after an 8-step distilled GGUF generation, config restored, healthy clip. Also reword the transformer_quant resolved reason to the measured reality: quant halves resident weights and hosted checkpoints cut load time, while per-step speed is roughly bf16 parity. * Pin the fp8 weight-quantize kernel against silent MSLK switching torchao's Float8Tensor KernelPreference defaults to AUTO, which switches the weight-quantize kernel to MSLK whenever an mslk package is importable on sm90+. Measured on B200: that changes fp8 scale rounding bitwise (8/8 FLUX matrices differ, scales ~55 percent of bytes), so a box that merely gains mslk would break the hosted-prequant bit-identity invariant; the mslk path is also slower under torch.compile (opaque extern call blocks inductor's quantize fusion, FLUX.1 fp8 e2e 1.149 to 1.624 s). Pin KernelPreference.TORCH explicitly, matching current no-mslk behaviour bit for bit; signature-gated for older torchao. GPU-smoked (finite, rel err 0.037) and pinned by test. * Shift Qwen-Image training sigmas to the inference distribution Qwen-Image's scheduler skips its static shift under use_dynamic_shifting, so the DiT trainer was drawing UNSHIFTED uniform-schedule sigmas for it (mean sigma 0.50) while inference always runs the exponential mu = log 3 shift plus the shift_terminal 0.02 stretch. Add a flow_shift config lever: "auto" (the new qwen-image default) rebuilds the training sigma table through the scheduler's own time_shift and stretch_shift_to_terminal so the draw matches the inference distribution exactly (mean sigma 0.72); a numeric value applies the standard linear shift s*u/(1+(s-1)*u); 1.0 keeps the historical identity behavior and stays the default for FLUX, Z-Image and Krea 2. The model timestep conditioning follows the shifted sigma, gathered in fp32 so bf16 rounding never skews it. Also wire two opt-in levers with off defaults: cfg_dropout (per-sample empty-prompt conditioning dropout, encoded alongside the captions before the text encoders are freed) and weighting_scheme="bell" (bsmntw-style mid-schedule Gaussian loss weighting normalized to mean 1). Verified with two 80-step rank-8 bf16 LoRA runs on Qwen/Qwen-Image (identity vs auto, same seed): both converge with finite decreasing loss and produce coherent same-seed previews. Unit tests cover the exact transform, the shifted sampling distribution, per-family defaults and config plumbing. * Add LoRA EMA, a persistent conditioning cache, and aspect bucketing helpers diffusion_train_extras hosts the opt-in training extras: LoRAEMA shadows only the trainable adapter params (warmup-ramped decay, default 0.99, exported as a second adapter under output_dir/ema), PersistentConditioningCache stores latent posterior stats and caption embeddings as safetensors keyed by content hash + family + resolution, and the aspect-ratio bucketing helpers group mixed-aspect datasets into same-area divisor-snapped shapes. The DiT trainer wires the first two behind config flags that default to the current behavior: ema_decay (0 disables) and cond_cache_dir (None disables). A fully warm cache skips loading the VAE and text encoders entirely; a cache hit is bit-identical to a fresh encode, including the per-channel qwen latent normalization. Also fixes the stale _gather_sigmas call in the perf test that still passed the scheduler instead of the sigma table. * Tighten torchao configs and note the FSDP2 design for the DiT trainer nf4 loads now enable double quantization (~0.4 bits/param off the frozen base scales at no fidelity cost), fp8 training uses the rowwise recipe when the torchao build ships it (per-row scaling confines the DiT activation outliers that a tensor-wide scale collapses), and the inference quant filter gains a per-scheme GEMM-tiling divisibility floor (16 for scaled_mm, 32 for MX blocks) so one ragged Linear cannot crash the first denoise after a clean quantize pass. plans/fsdp2_diffusion_design.md records the multi-GPU design: bf16/fp8 over FSDP2 with per-block units, LoRA attached before sharding, int8 out of scope (DTensor over the quantized subclass is undefined), per-family notes. * Batch diffusion inference with per-image seeds, an inference conditioning cache, and GGUF loader fixes Batched generation: /images/generate takes a prompts list (one image per prompt, txt2img only) or a seeds list (one prompt, one image per seed); the legacy batch_size path derives per-image seeds base..base+n-1 like the native engine. Every image gets its own torch.Generator so any batch member replays alone from its gallery recipe; the whole list runs as one forward by default with OOM backoff that halves a failed chunk, and an explicit batch_size caps images per forward. Validated 10-22x over serial engines on 32-image suites with LPIPS deltas within 0.002. Conditioning cache on the inference path: UNSLOTH_DIFFUSION_COND_CACHE_DIR (the inference sibling of the trainers' cond_cache_dir, same persistent store) wraps encode_prompt so repeated prompts skip the text-encoder forward entirely; verified bit-identical outputs. Bypassed while LoRA adapters are attached; tensor-argument calls pass through uncached. Compile cache: GGUF loads fingerprint their own bundles (quant=gguf, a different compiled graph than the dense family) and batched calls register every distinct (w, h, batch) chunk shape they ran, so the heavy GGUF batched warmups (~159 s at batch 32 on 12B-class, ~655 s on 20B CFG-batched) are paid once ever. GGUF loader: strip the sd.cpp model.diffusion_model. container prefix in the single-file converter; diffusers' FLUX.2 converter KeyErrors on it and the Qwen-Image identity mapping strands the model on meta. * Correct batched seed-replay docs to match measured behavior Same-seed images at the same batch shape are bit-identical; a solo regeneration with the recorded seed matches its batched rendition up to batch-size-dependent kernel numerics (mean abs pixel delta about 2.5/255, LPIPS delta under 0.002), not bit-exactly. The previous wording overclaimed bit-identity across batch shapes. * Note that batched bit-identity assumes a settled compiled graph The first generation issued while the deferred compile is still in flight can deviate transiently (observed once on a cold fp8 build: mean abs pixel delta 0.063/255); once the graph is settled, same-seed same-batch-shape images are bit-identical across runs. * Studio sidebar: Image03/FlimSlate icons, More flyout, Train row, New pills - Images uses Image03Icon and Video uses FlimSlateIcon. - New "More" row (MoreHorizontalIcon) opens a right-side flyout on click or hover holding Video, Recipes and Export; the close is delayed 180ms so the pointer can cross the gap. Its SidebarMenuButton deliberately takes `title` rather than `tooltip`: with `tooltip` the button returns a Tooltip root and DropdownMenuTrigger asChild would hand its ref to a non-DOM node. - Dropped the "Train" section heading; Train is now a top-level row between Images and More. data-tour="navbar" moves to the surviving nav group so the product tour keeps its anchor. - "New" pill beside Images and (inside the flyout) Video, via NavBadge. * Studio sidebar: match flyout rows and New pills to the existing scales - More flyout rows dropped their sidebar-row typography and size-icon override, which fought DropdownMenuItem's own scale (text-sm, gap-2.5, px-3 py-2 and size-4 icons) and rendered oversized glyphs and text next to the nav. - New pill reuses the brand "beta" badge recipe (nav-badge font, --ui-font-scale sizing, nav token colours) rather than hardcoded 9px values. - The More row's native title tooltip (an OS box on hover) is replaced by the app's Tooltip, wrapped around DropdownMenuTrigger so both triggers compose onto the same button, and shown only on the collapsed rail like other nav rows. * Settings: pin and reorder the sidebar navigation Adds a "Sidebar navigation" section to Settings -> Appearance, above the existing profile-menu customizer, with the same drag-to-reorder + switch UI. - New sidebarNav preference: one { id, pinned } entry per navigable row (projects, hub, images, train, video, recipes, export), array order = render order. Defaults match the shipped layout, so an untouched install is unchanged. - Unpinning moves a row into the More flyout rather than hiding it, so no page becomes unreachable. New chat and Search stay fixed as actions. - app-sidebar now renders from one navRows descriptor map, so a pinned row and its flyout counterpart cannot drift; the More row appears only when something is unpinned and highlights off whatever it actually holds. - Mirrored in the backend PersonalizationCustomization: without it the model's extra="ignore" would drop the field, and because sync replaces local state with the server's copy once customization is saved, the user's pin order would reset on the next sync. The validator dedupes and back-fills like sidebarMenu but preserves the client's order, since here order is meaningful. Frontend typecheck, i18n parity and catalog checks pass; 32 personalization tests pass, including a round-trip asserting a reordered list survives a save. * Sidebar customizer: drop the Search row, skip More for a lone item - Search is reached from the top bar, so it is no longer previewed as a fixed sidebar nav row; New chat stays. - More now appears only when it would hold two or more rows. A single unpinned row renders inline in its saved order position instead: a flyout wrapping one item costs a click and earns nothing. The customizer's More preview follows the same threshold. * Sidebar settings: hide a lone unpinned tab, match New chat icon, rename Profile menu - With exactly one tab unpinned, both More and that tab are dropped, so nothing is drawn for it (previously it rendered inline). The page stays reachable by URL. - The customizer's New chat preview uses PencilEdit02Icon, the icon the real row renders; Edit03Icon was a different glyph. - "Sidebar menu" is now "Profile menu", described as the shortcuts behind your name at the bottom of the sidebar, so it no longer reads as a second name for the navigation section above it. * Tighten comments in the new sidebar and delete-guard code * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio sidebar: keep the More row highlighted while its panel is open Moving the pointer into the flyout left the row unhighlighted while the panel stayed open. The row now carries data-menu-open, added to the nav hover selector list. Not data-state: the tooltip and menu triggers both write that attribute, so whichever lands last wins. * Images: use the shared pill toggle for Create/Train and pad the panels - Create/Train was the only segmented control on its own Tabs styling. It now uses PillTabs, the same control as the model picker and Hub toggles, pinned to the header row's 34px. PillTabs takes an icon per tab, so the inline-span workaround for TabsTrigger goes away. - pt-3 on both the Create and Train panels, which sat flush against the model selector row. * Images: make the workflow picker a dropdown instead of a 7-up strip Seven workflows in a 340px rail left ~48px each, so the labels crowded and the hints were only reachable as title tooltips. The strip is now a dropdown: the trigger shows the current workflow and its hint, and each row carries its own description. A row the loaded model can't run is disabled and shows the reason in place of the hint, so the gating explains itself. Adding a workflow no longer shrinks the others. * Images: workflow icons, hint under the trigger, more top room, unclipped Train cards - Each workflow carries an icon, shown on the closed trigger and on every row. - The trigger is one line (icon plus name). The selected workflow's description moved below it, where it reads like the Field hints further down the rail. - pt-6 instead of pt-3 on both Create and Train, so the cards clear the model selector row. - The Train right column scrolls while its cards use ring-1, which draws outside the box and was clipped at the scroll edges. p-px gives the ring room. * Images: one-line workflow rows, stronger trigger fill, roomier mode tabs - Dropdown rows are icon plus name only. The selected row's description already shows under the trigger, and a disabled row keeps its reason as a title. - Trigger fill moves to the bg-foreground/[0.07] dark:bg-foreground/[0.12] pair the hub cards use, so it reads against the card in both modes. - Description under the trigger goes from text-ui-10 to text-ui-11p5. - More horizontal padding on the Create / Train tabs. * Images: drop card borders for the composer shadow, keep scrollbars inside, use app controls in Train - Cards lose ring-1 for .panel-soft-surface: the composer's shadow in light, flat in dark, matching .chat-composer-surface and the menus. - Both rails now clip (overflow-hidden) with the scroller inside, so the scrollbar can't ride over the rounded corner. Same shape video-page already uses. - Train's 9 native selects become the app Select, so they no longer open an OS-native menu, and the native file input is hidden behind a Choose images button that reports the count. - Image previews use explicit 8-10px radii: this theme sets --radius to 1.1rem, so rounded-md was 15.6px and the thumbnails read as circles. * Images: one card for controls and preview, chat sliders, wider softer shadow - Controls and preview were two floating cards; they now share one card split by a divider. The Advanced dock stays separate since it toggles. - SliderField wraps Chat's ParamSlider, so the sliders match Chat (label row with the value, full-width neutral track) instead of a green track with a spin box. All 14 call sites keep their props. - panel-soft-surface goes from 0 2px 8px -2px /0.16 to 0 4px 22px -6px /0.10: lighter, spread wider. * Images: flat Create and Train panes, hover-only scrollbars, tidier Train dataset step Both Images tabs now sit on the page background like the Hub: no card, no shadow, no bounding box. A single rule divides the controls rail from the preview canvas (Create) and from the run area (Train), and the settings and previous-runs sections read as panes rather than nested cards. Also: - Scrollbars in these panes use the existing hover-scrollbar recipe, so the thumb only shows while the pane is hovered. - Workflow rows explain themselves with a tooltip after a short hover, which also works on disabled rows, and the descriptions are much shorter. - Training images rows are name plus image count; the license stays on the example card. - The upload step loses its dashed box, the buttons match the sizes around them, and Upload only appears once files are picked. - The empty preview uses the same icon as the Images nav item. * Images: full-height panes, wider settings rail, Create/Train offset from the selector The rule between the panes now runs the whole page height (the row drops its bottom padding and each pane pads its own content), the settings rail is wider on both Create and Train, and the Create/Train switch sits further right of the model selector. * Images: put both tabs on the Hub's centered measure Top bar and content now share mx-auto max-w-1100 with px-5 / sm:px-8, so Create and Train sit at the same width and position as the Hub instead of running edge to edge. * Images: restore the top bar position, drop the panes lower under it * Images: center the mode switch, flip the arrow with the orientation, app tooltips everywhere The Create/Train switch is centered on the page instead of trailing the model selector, with wider buttons. The flip control's arrows now rotate with the orientation and its label says which way the flip goes. Every native title tooltip on the page is now the app's tooltip, so they all get the rounded surface instead of the OS box. * Images Train: plainer field text, no green buttons, columns that stop colliding - The dataset name, trigger prompt, adapter name and custom base fields now say what they are in plain words instead of leaning on example values. - Import, Upload, Back, Back to settings and Train another are outline buttons, not green ones. - Example thumbnails are landscape tiles, so photos are not cropped to chunky squares. - Settings cells get min-w-0 and the select value truncates, so a long option like the nf4 label no longer widens its column into the next one. - The number stepper sits a little further in from the field edge. - Create and Train are wider. * Images Train: roomier example cards with Import on the thumbnail row * Video: same treatment as the Images tabs - No cards: the rail and the canvas sit on the page background, divided by a rule that runs the full page height, on the Hub's centered measure. - Wider rail, chat's sliders, hover-only scrollbars. - Every native title tooltip is now the app's tooltip, including the clip cards. - Reapply and Cancel are outline buttons, the empty state uses the Video nav icon, and the clip tiles are less rounded. * Images and Video: narrower generation rail, matching Train headings Create and Video rails go from 392px to 368px. Train a LoRA and Training settings are now the same size and both in the heading font: the h2 already picks it up from the base rule, so the settings header opts in with font-heading and the weight that rule pins. * Images Train: shorter copy throughout Family notes, example descriptions, precision labels and every helper line are trimmed so they stop wrapping to three lines and colliding with the next column. The nf4 label now fits its select without truncating. * Images Train: a little more spacing between field groups * Images and Video: tighten code comments * Fix training start NameError, the load-order guard test and CPU-only diffusion tests - start_training forwards resume_source_run_id to _start_training_impl, which reads it. Without it every start raised NameError. - Restore main's anchor in the load-marker order test: the file now has an earlier `if config.is_gguf:`, so indexing the first one compared the wrong branch. - The two diffusion tests that reach diffusers now skip when it is absent, matching the CPU repo-test env. - The UI smoke finds nav rows that live in the sidebar's More flyout. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Treat a null metadata caption as no caption str(None) stored the literal "None" as the caption, so a null row counted as captioned and would have trained on that text. Also drops an unused import. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix invalid-UTF-8 500s, the flat Canny map and the dropped DiT knobs read_text raises UnicodeDecodeError, which is not an OSError, so one bad caption sidecar or video sidecar 500d the info, upload and gallery routes. A flat image now yields the all-black edge map instead of its own luminance, and the four DiT loss knobs the trainer implements are declared so model_dump keeps them. * Use the ui font-size tokens instead of raw px text utilities text-[11px] and friends ignore the UI font size preference, which the repo's font-scale contract test enforces. Same rendered size at the default scale. * Fix diffusion dataset 500s, the dropout-1.0 no-op run and the reset base pick Four correctness fixes on the training side: - The labeling grid read caption sidecars under except OSError, but a non-UTF-8 sidecar raises UnicodeDecodeError (a ValueError), so one bad file 500d /diffusion/dataset/{name}/images and the grid could not be opened to repair it. Read it as no caption, matching the info summary. - An image past Pillow's own hard limit raises DecompressionBombError, which derives straight from Exception and so escaped the upload guard's (OSError, UnidentifiedImageError, ValueError) and returned 500 instead of the intended 400. - lora_dropout accepted 1.0, which makes PEFT build nn.Dropout(p=1.0): lora_A and lora_B receive no gradient and the run saves an untrained adapter while reporting normal progress. Bound it below 1.0, matching the LLM request schema. - The train panel re-seeded the base repo on every dataset refresh because the family object identity changes on each info fetch, so an upload or caption save silently replaced the user's chosen base and the run started on a different model. Track the pick and only re-seed on a real family change. * Show the retained failure when a video page mounts after a failed job Mount-time recovery handled only phase=completed, so reloading the page after a multi-minute generation failed left an idle view with no diagnosis: the backend keeps the terminal failed record only until the next job, and nothing else survives the reload. Surface it the same way the poll does, filtering the cancelled sentinel. * Fix batched generation crashes, cache keying and unreplayable recipes Four bugs in the batched inference path, all found by review: - A mixed-prompt batch sent a scalar negative prompt against a prompt list. Z-Image asserts on the length, and Qwen-Image, Krea 2 and FLUX true-CFG encode a batch-1 negative against batch-N latents and fail in the transformer's text/image concat. Broadcast it to match the batch. - The FBCache step-cache reset sat above the chunk loop. diffusers only resets that state at the end of a successful call, so a forward that raised (the OOM the backoff is meant to recover) left its own residual behind and the halved retry died on a shape mismatch. Reset before every forward instead. - The conditioning cache keyed on the checkpoint alone, but a GGUF or single-file load takes its text encoders from the companion base, so the same checkpoint against a different base reused the previous base's embeddings. Key the base too. - Gallery records stored the base seed and the requested batch size even when a prompts/seeds list drove the run, so restoring the second image of seeds=[5, 99] replayed seed 5. List-driven outputs now record as single-image recipes on their own seed. Also bound strength above 0: every img2img pipeline derives its step count from it, so 0 leaves zero denoising steps and either raises or, on SDXL, crashes on empty latents. * Fix quantized-load LoRA bake, prequant family exclusions and outpaint canvas Six review findings across the Images page and model scanning: - The quantized (int8/fp8) load path can only attach LoRA adapters before quantization, but the frontend load request had no loras field, so every generation after such a load was rejected and each reload repeated it. Send the selection with the load. - build_prequant_checkpoint passed no family to the scheme exclusions while recording the family in metadata, so a Qwen int8 artifact baked the short-M text-stream linears and was then rejected wholesale by the loader's family-keyed check. - Registering a bare single-file checkpoint directory produced no On Device row even though the images loader can load it; only its parent worked. Admit that shape when nothing else matched. - Unload left the Reapply target set, so the repair path was skipped and Reapply reloaded the ejected model. Clear it, as the video page does. - Both FLUX.2 bases were trusted for training but not inference, so Deploy to Create rejected every FLUX.2 adapter. - Outpaint allocated the grown canvas before downscaling, exceeding the browser canvas area cap on a large photo; an over-cap canvas is unusable, so Extend silently posted a fully transparent image and mask. Scale the source first. * Send the picked GGUF filename with the quant so diffusion loads fire The variant expander emitted only the quant label, and nothing else in the frontend set ggufFilename, so the Images and Video pages could never take their GGUF branch: both gate it on meta.ggufVariant and meta.ggufFilename, then fall through to the single-file path, which returns because the id is a repo id and not a .gguf name. Every quant pick was a silent dead click, with no load request reaching the backend. The filename was already on the variant row (the picker keys its list on it, and the variant validator requires a non-empty string), so thread it through the click handler. The chat path is unaffected: it reads ggufVariant and never needed the filename. * Version the conditioning cache key and reject non-finite flow_shift Two correctness fixes: - The cache keyed the checkpoint and its companion base by name only, so a Hub repo advancing to a new commit, or a local directory updated in place, kept returning embeddings from the previous text encoder. Pair both with a revision marker: the locally resolved commit sha for a Hub repo, config plus text-encoder file stats for a directory. Neither loads the encoders, so a warm run still keeps them off the GPU. - flow_shift only checked positivity, but JSON accepts 1e309, which floats to inf, and inf <= 0 is False while NaN fails every comparison. The sigma table then evaluates s * u / (1 + (s - 1) * u) as NaN, which poisons every sampled sigma and saves a corrupted adapter while progress looks normal. Require a finite value. * Keep curated models listed, guard the video companion repo, pin diffusers Three review findings: - The picker filtered every catalog member out of Recommended and Hub search on the way to canonical group rows, but nothing renders those rows yet (catalogGroupFitsDevice and groupMatchesQuery are imported and unused). A task-scoped picker's models list is catalogToModelOptions(), i.e. group members exclusively, so both lists came back empty and no curated model could be discovered or downloaded. Keep the artifacts listed until the grouped UI exists. - The video delete guard compared only repo_id, so deleting the companion base of a loaded GGUF video model was allowed even though it supplies the VAE and text encoders. Compare base_repo too, matching what the images guard already does for its companions. - diffusers was declared unversioned while the diffusion stack requires 0.39 (Krea2Pipeline, the cache_context child registries, the Flux2 and Z-Image pipelines), so an upgrade could keep an older release and selecting an advertised model failed until the user upgraded by hand. * Namespace the trainer conditioning cache per checkpoint, bound the learning rate - The trainer keyed its persistent conditioning cache on family and resolution only, while the keys themselves carry just the caption or image content and crop variant. One cache directory reused for two checkpoints, or for the same repo at a new revision, let a warm run skip loading its encoders and train on the other model's embeddings and latent statistics. Namespace on the base checkpoint and its resolved revision as well. The revision helper now lives beside the cache in diffusion_train_extras and the inference wrapper delegates to it, so the two cannot disagree about what counts as the same source. - The diffusion learning rate only checked positivity, but 1e309 floats to inf and satisfies gt, so the route evicted the resident models and started AdamW with an infinite rate: the first step destroys the adapter while progress looks normal and the result is saved. Bound it below 1.0, matching the LLM schema, which rejects inf for the same reason. * Fix GGUF image model picks doing nothing, and pick the train base in the top bar The quant rows never forwarded the .gguf filename, so every hub GGUF pick on Images/Video fell through to a silent return. On Train the top bar now picks the training base instead of a generation model, which is GGUF-only and untrainable. * Pin diffusion and video loads to the live HF cache root Both read huggingface_hub's import-time HF_HUB_CACHE, which changing the cache folder does not update: progress counted the old root while the download wrote to the new one, and from_pretrained could split one model across both. * Add the diffusion download plan endpoint Reports the repos and exact files a pick needs so the download manager can stage them with the loader's own file scope. A plain snapshot would add the packaged root single, transformer shards and fp16 twins the loader never opens. * Add a file-scoped flavour to the Hub download job Lets a consumer that reads a deliberate subset of a repo stage it through the normal download manager. Keyed as "@scope" so it never collides with a quant or with the repo's full snapshot, and the file list rides the registry so an XET to HTTP retry respawns the same scoped job. * Stage image and video downloads through the Hub download manager They downloaded inline inside the load, so they had none of the manager's disk preflight, manifest verification, resume or panel progress. Picks now stage as scoped jobs carrying the loader's own file list, then load from a warm cache. * Fetch staged GGUF checkpoints as scoped jobs, and stop calling diffusion models unsupported A GGUF entry went out as a full snapshot, whose ignore list drops *.gguf: the job finished at once having fetched only docs, and the repo landed on device unloadable. Every entry is scoped now. The Hub also no longer tags image/video models as unsupported (they run on their own pages), and those pickers name what they select. * Apply the picker task filter to local model sections LM Studio, ./models and custom-folder rows ignored it, so the Images picker listed chat GGUFs that 400 on a diffusion load. The backend already tags every local model with a task for this purpose. * Route a chat pick of a diffusion model to the Images or Video page Chat cannot load one, so it was either hidden or failed on load. The unfiltered picker now lists on-device diffusion models and navigates to the page that runs them, passing the repo and quant so that page loads it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Route the real GGUF filename, keep non-GGUF curated models, key scoped downloads by file set Five review findings, four of them ways a click did nothing or fetched the wrong thing: - A chat pick of a diffusion model routed ggufVariant (a label like Q4_K_M) in the search param the target page uses verbatim as the GGUF filename, so the load asked for a file that does not exist. Route ggufFilename; no filename means a curated non-GGUF pick, loaded as a pipeline. - The task-scoped pickers kept only GGUF repos, so the catalog's bf16, bnb-4bit and single-file fp8 artifacts could not be discovered or downloaded on the Images and Video pages even though loadSpecFor knows how to load them. Keep curated artifacts whatever their format, in Recommended and in Hub search. - Both pages deduplicated routed selections on the model alone, and they now stay mounted, so picking the same repo again -- another quant, or the same one after chat evicted it -- returned early without loading or clearing the query string. Key on model and quant. - Every scoped image download shared one @diffusion job key regardless of the requested files, so switching quant mid-download adopted the running job: the UI waited on the first file set, then loaded a file that was never fetched. Include a digest of the file set in the key. - A scoped plan silently dropped requested files missing from Hub metadata, and snapshot_download succeeds when an allow pattern matches nothing, so the job reported completion and triggered a load with required files absent. Fail the job instead. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the scoped download key derivable, and stop the hidden page hijacking a route Four review findings, the first a regression from my own last commit: - Keying scoped download jobs by a digest of the file set broke the download manager: it builds that key client-side (it polls and cancels before any response tells it a key), so it watched and cancelled a key no worker owned and never fired its ready callback. Keep the derivable "@scope" key and refuse the second request instead when a live job on the slot is fetching a different file set -- decided inside the registry claim, under the lock, so a concurrent claim cannot slip past it. The manager records the file set on the job as well, so a sibling quant's transfer is not adopted locally either. - Both diffusion pages read the route query through a loose useSearch and both stay mounted once visited, so the hidden one consumed the other's ?model=: it navigated back to its own route and tried to load, say, an image checkpoint as a video model. Only the visible page consumes it. - The staged download plan was built without the configured HF token or the Advanced values the load itself sends. The token matters most: the backend's Hub metadata lookup is best-effort, so a gated base silently planned no companion entry and the load pulled those multi-GB files inline, outside the manager. The memory/quant controls decide whether the base transformer/ shards are needed at all, and the route dropped memory_mode, cpu_offload, the prequant path and the LoRA selection before asking for the plan. - The video preview kept playing after leaving the page: the keep-alive layout only hides it, and display:none does not pause a media element, so a clip the user unmuted kept its audio going over the next page. Pause on the active transition and do not auto-replay while hidden. Also completes the hand-built request bodies in the hub download tests: the scoped-files field this branch added to the route read as an AttributeError against them, failing five tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Serialize the GPU handoffs, gate DiT training on a GPU, and keep 3.9 installable Six review findings, three of them evict-then-fail orderings: - The chat load reclaimed the GPU without telling the arbiter it existed. A chat load holds no llama-server process until its GGUF has downloaded, which is minutes, so a competing Images/Video acquire in that window found nothing to cancel, took the GPU, and the chat load then spawned onto the same device. It now registers an in-flight marker through acquire_for's register hook (under the arbiter lock, as the image and video loads do), the evictor cancels a marked load, and the route undoes itself if ownership moved while it loaded. - The Hub-download conflict check ran after that handoff, so a GGUF the download manager already owns destroyed the resident Images/Video pipeline and then 409'd, having loaded nothing. It moves above the handoff, together with the marker it handshakes with. - The image load released the engine router's transition lock before registering the load, so a second load choosing the other engine could unload the still-idle engine this one captured; the load then landed on a deactivated engine, where generate, status, unload and the arbiter's evictor can no longer reach it. Registration now happens under that lock and refuses if the engine changed. - Training a DiT family on a host with no GPU was accepted: nf4 is not a CPU fallback, its 4-bit load goes through bitsandbytes, which requires CUDA, XPU or MPS. The start unloaded the working Images pipeline, pulled the text encoders, and only then died in the child. Rejected before the teardown now, and /info stops advertising a precision that always 400s. SDXL keeps its documented fp32-on-CPU path. - Both diffusion pages kept the routed-pick marker forever, so re-picking the same checkpoint (after chat evicted it) neither loaded nor cleared the query string. The marker is released once the query is gone. The Images key also carried a stray NUL byte, which made the file read as binary to grep and other tooling. - diffusers dropped Python 3.9 in 0.38, so the unconditional >=0.39.0 pin left pip no candidate at all on 3.9 and made every install that composes the huggingface extras unresolvable there. The floor is conditional now. Also fixes tests that were already red on the branch: two hand-built request fakes had gone stale against fields this branch added, and the handoff-ordering test only failed on a host with fewer than two GPUs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the GGUF variant contract test against the merged handler signature The assertion pinned the exact single-line call handleVariantClick(v.quant, v.downloaded, expectedBytes, v.filename), but the handler takes (quant, filename, downloaded, sizeBytes) and prettier wraps the call across lines, so the mandatory repository test job failed on every push. Match the call structurally and assert the filename really is forwarded in the handler's argument order. * Stop a background page and a stale record taking the GPU or a download with them Five fixes from a review pass over the diffusion work. delete-finetuned rmtree'd a model the Images or Video engine was holding: every guard on that route is chat-only, and Images loads any local path, so deleting a local diffusion model under the storage root pulled the weights (and the companion VAE / text encoders sd.cpp re-reads each generation) out from under a live pipeline. The cached-model route already refuses this; the trained/exported one now does too, matching by path rather than repo id, and failing open on a chat-only install so it cannot block ordinary deletes. A staged download finishing while its page was hidden loaded the model and evicted whatever the user was actually using: both diffusion pages stay mounted behind the router and a load takes the GPU unconditionally. The pick is now held until its page is on screen again, which is also what chat does. A scoped download could report success having fetched nothing. With Hugging Face metadata unavailable no manifest is written, so verification is a no-op, and snapshot_download returns an existing snapshot folder without downloading when its own repo_info call fails. A repo already on disk from a full snapshot job (which ignores *.gguf) therefore completed with no weights and auto-loaded against them. The requested file list needs no network, so it is checked against the disk directly. The XET to HTTP retry reclaimed the job slot without the scoped file list, and that claim overwrites the stored record, so a later identical scoped start compared an empty list against the real one and 409'd instead of adopting the running download. The DiT accelerator gate probed torch.mps.is_available(), which only exists from torch 2.5 while the supported floor is 2.4. All three probes shared one try/except, so on torch 2.4 the AttributeError read as 'no block' and a CPU-only host still evicted the resident pipeline, downloaded the encoders and died in the child. Each accelerator is probed on its own now, through torch.backends.mps. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stage what an LTX-2.3 load reads, keep a routed file's load kind, drop an unbakeable LoRA Three from the latest review. The video download plan always asked for the wide base file list, so an LTX-2.3 pick staged the 2.0 base's VAEs, vocoder and connectors that the checkpoint supplies itself, while the companion files the 2.3 assembly does read were left out of the plan and pulled inline at load, outside the panel's progress, cancel and disk preflight. The plan now recognises a 2.3 pick by name (the load keeps the authoritative header probe, and under-guessing only falls back to the load-time pull), narrows the base list, and stages the extras in the same entry as the checkpoint so one repo stays one scoped job. A pick routed from the chat picker arrives as ?model= and ?quant= with no picker metadata, so a bare local .gguf or .safetensors was loaded as a pipeline: an explicit model_kind wins over the backend's filename sniffing, so it evicted the resident model and then failed on the missing model_index.json. Both pages now derive the load kind from the path, the same way their own picker handlers do. A torchao int8/fp8 build takes adapters only at load time. Switching artifact inside one family keeps the LoRA selection, since the family did not change, but the load did not bake it, so the next generation was rejected with 'reload the model with the adapter selection' while the picker still showed the adapter as active. The selection is now dropped once per resident build, with a message saying to pick and load again. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cancel an evicted safetensors load, spare the arbiter for CPU-only chat, fetch clips lazily Four fixes from the latest review round: - The GPU arbiter's chat evictor only cancelled the llama.cpp side. The orchestrator publishes active_model_name once its worker reports success, so an in-flight safetensors load was visible only as an entry in loading_models and finished onto the GPU after ownership had transferred. Cancel every pending load, and give the safetensors branch the post-load ownership recheck the GGUF branch already had. - A manual gpu_layers=0 GGUF load runs on the CPU with the GPUs hidden from the child, yet it took the arbiter unconditionally: it cancelled a running image or video generation for a model needing no VRAM, then held CHAT ownership so the next GPU workload unloaded it for nothing. Gate the acquire on the same predicate the launch-time CPU-only mask uses, as the image and video loaders gate on their resolved device. - The staged-download hook subscribes per repo, not per job, so another job on the same repo advanced the staged queue (starting a load whose scoped files were still downloading) or wiped a queue that was still running. Compare the variant each callback carries, like the chat page's auto-load does. - The video gallery fetched every record of a page into an object URL that lives until the page closes: 50 clips at tens to hundreds of MB each, for cards the user may never scroll to. Fetch a clip as its card nears the strip's edge, plus the selected one the player needs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim the comments across the diffusion backend Comment-only pass over the Python this PR touches: drop what the code already says, collapse multi-line explanations that still read on one line, and keep the reasoning that is not recoverable from the code. No code, docstring semantics or behaviour changes; verified with an AST comparison against the previous revision, and the backend suite is unchanged (same 37 environment failures as before: the API integration tests that need a live keyed server, the flash-attn install hooks, and the GPU memory fields). * Invalidate latents on a VAE swap, keep a cut-off generation, surface the EMA adapter - source_revision() scanned the checkpoint root plus text_encoder/tokenizer but not vae, so swapping or fine-tuning the VAE in place left the conditioning cache namespace unchanged and a warm run trained against latents from the old checkpoint. Include the vae directory, like any other component the cached tensors come from. - /images/generate answers only when the images are saved, and secure mode's tunnel caps an origin response near 100 seconds, which a native CPU or a high-step run passes routinely. The page reported failure while the work kept running, and a retry would duplicate it. A lost response (fetch rejection or a gateway status the origin never answered) is now told apart from a refusal: the page waits out generate-progress and reloads the gallery, so the run it started still lands. - The trainer emits the EMA adapter's path with the terminal event, but the state update dropped it, so neither the run history nor either response schema carried it and an enabled EMA left nothing discoverable. Keep it, and show it next to the primary adapter. - weighting_scheme advertised a choice of timestep sampling; sampling is always logit-normal and the flag only selects the bell loss weights. Describe what it does. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Hide unloadable cached rows, hold the dataset interlock, bound a GIF export - The cached-model listing tagged any repo with a model_index.json as text-to-image, so a community pipeline the image loader's trust rule refuses still got a row in the Images picker, and a detected-but-untrusted video repo fell through to that same tag. Gate the image tag on the load path's rule and hide an untrusted video repo outright. - A routed diffusion pick only carries a GGUF filename, which is all the chat picker has, so a curated single-file artifact arrived with no quant and was loaded as a pipeline: from_pretrained on a repo with no model_index.json. Pass the page's own catalog spec into the route pick, so a routed pick resolves to exactly what a direct pick on that page resolves to. - The dataset mutation endpoints checked is_active() and only then handed their filesystem work to a thread, so a start reserving in that gap changed captions or removed images underneath the preflight or the running trainer. The interlock is now registered for the whole request under the lock reserve() uses, and a start refuses while a mutation is open rather than waiting on it. - GIF export held every kept frame as a paletted image before encoding; a clip may be 2048x2048 for 1024 frames, and at the 12 fps target the step is 1, so one export click could allocate over 4 GB and take the backend down. Downscale past 720 px and widen the step to keep at most 300 frames. - seed accepted any Python int, so an out-of-range one passed every preflight, evicted the resident models, spawned the trainer and only then died in torch.manual_seed. Bound it to torch's 64-bit range in the request and config. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the accelerator probes in the DiT family-metadata tests Six tests read family_train_infos() (or a start preflight) without pinning the host probes, so they only held on a machine with a bf16 accelerator: on a GPU-less runner the DiT gate empties precision_modes, turns supports_compile off, and replaces any other preflight message with the no-accelerator note, and all six failed there. A conftest fixture pins both probes for exactly those tests, so they assert the family metadata they are about on every host. The gate's own CPU-only behaviour keeps its dedicated tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Carry the pipeline task into the hub inventory the pickers read - The task-scoped pickers filter On Device rows on a task, and the chat picker routes a diffusion pick by the same field, but those rows come from the /api/hub inventory, which never carried one: the Images and Video pickers listed nothing on device and the routing never fired. Both cached scans and the local listing now tag rows with the classifiers the models API already uses, the schemas and the frontend adapter carry it through, and a row the backend classified as a generation task is exempt from the chat-only guard that was also dropping it. - The local routing map was keyed by model_id while the row click passes id (a filesystem load id for a models_dir or LM Studio entry), so the lookup missed and the pick fell through to the chat loader. Key both. - A staged download whose start answered "error" left its head in place, where the effect never re-runs and onReady never fires, so the pick was stranded until the user reselected. Clear the queue and say so. - Every scoped pick in a repo shares the @diffusion variant, so the variant alone cannot tell two file sets apart: restaging while the first job finished let its completion pass for the new pick and load a checkpoint that had not downloaded. Bind the callbacks to the repo + file set they started, and to the staging generation. - A rejected generate POST does not say whether it reached the backend, so an immediately idle progress read was ambiguous and a submission that never landed looked like a finished image. Require evidence: progress seen active, or a gallery record that was not there before the POST. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the OpenAI image URL fetchable, keep WebM audio, stream example imports Four review items on the diffusion Studio work: - response_format=url returned the bearer-gated gallery route, which a standard image client downloads with no Authorization header, so the default response format was unusable. Mint a short-lived HMAC link instead (the shape RAG already uses for pdf.js) served by a signed route, and leave the gallery route itself bearer-only. - A manual gpu_layers=0 load carrying speculative_type="off" -- a value the UI persists and sends -- read as GPU-bearing, so it took the GPU arbiter and evicted a resident image/video pipeline even though the launcher hides the GPUs for it. Canonicalize the mode and exempt "off". - The curated example import prepared the whole split before the loop stopped at the 10-100 image cap; m1guelpf/nouns is 49,859 rows / 328 MB. Stream instead, with the prepared load kept as a fallback for a repo that cannot stream. - WebM export dropped the audio track an LTX-2 clip carries, silently, on the format offered for web embeds. Mux it as Opus through a resampler + FIFO, and keep exporting the video alone on a build without libopus. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not stub out triton on a GPU host when the Xet backend fails to import The lazy loader retries `import unsloth_zoo.hf_xet_fallback` under UNSLOTH_ZOO_DISABLE_GPU_INIT=1 whenever the first attempt raises. That flag makes unsloth_zoo take its MLX/CPU path, which injects triton and bitsandbytes STUBS into sys.modules for the rest of the process. On a working GPU box whose first import failed for an unrelated reason (a bitsandbytes/CUDA mismatch, say) the retry succeeds, so Studio boots looking healthy and then dies at the first CUDA-only kernel: a GGUF or compiled diffusion generation hits the stub and returns NotImplementedError: Unsloth: 'triton.tools.experimental_descriptor.enable_in_pytorch' was called on Apple Silicon / MLX, where triton is stubbed out. so every image generation 500s with an Apple-Silicon message on a Linux CUDA host, while the load reports success. Found by loading Z-Image-Turbo GGUF through the API on a box where bitsandbytes could not initialise. Gate the retry on the host genuinely having no accelerator. The Xet stall watchdog is optional and already degrades with a warning; a process whose triton is stubbed out is not recoverable. The warning now says why it did not retry. * Fix the lost-generation proof set, the settle timeout and the hub inventory's diffusion gates Seven fixes from the latest review round on the Images page and the hub cache inventory. Images page: - The lost-POST settle path built its "already seen" gallery id set inside the catch, after the request failed. By then the earlier runs of the same batch had already prepended their records, so run 2 could accept run 1's image as proof that its own request reached the backend. The set is now captured once before the first POST and grows with every record the batch produces. - settleLostGeneration fell out of its SETTLE_MAX_MS loop and returned normally, so a wedged generation was counted as done and the next run started against a busy backend. It now throws on timeout. - Restoring a recipe cleared the ControlNet selection but left the workflow tab and the init / mask / reference images pointing at whatever was loaded, so the next Generate conditioned on an unrelated image. It now clears all of them and returns to Create. - The download plan omitted the adapter selection the load itself bakes in. A baked LoRA forces the dense build path, so the plan described a different file set than the load that followed and the rest was pulled inline, outside the download manager. Both now derive the list from one helper. Hub cache inventory: - A download for a repo an Images or Video load is staging was allowed to start: only the llama.cpp loader was consulted. Both diffusion backends already expose loading_repo_ids for the delete guard, and the download guard now reads them too. - A companion-only prefetch (pipeline manifest plus VAE and text encoder, no transformer) passed the snapshot-partial check, since every file its manifest expected did arrive, and was advertised as on-device although from_pretrained cannot load it. - The single-file flag never reached the picker through the hub inventory path, so a checkpoint-only diffusion repo read as a full pipeline and failed after the handoff. The two pipeline-shape helpers now live in hub/utils/inventory_scan.py so /api/models/cached and the hub inventory classify the same repos the same way. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop adopting an unknown scoped download, leaking raced blobs and resurrecting deleted clips Three items from the latest review round. A scoped download job carries a deliberate file subset, and every file set of one repo rides the same "@scope" slot. A client that adopts a live job from the backend had no file list to compare against: the active-downloads response never carried one, so an adopted job's set was unknown and any later scoped request for the same repo read as "already started". Selecting a different checkpoint then waited on the wrong transfer and tried to load a file nobody fetched. The response now publishes the scoped file list, adoption records it, and an unknown set no longer satisfies a scoped request. A gallery record can be deleted while its blob is still downloading. The delete revokes the URL present at that moment, so the fetch that lands afterwards inserted a fresh object URL for a record no card renders and nothing can revoke: a full MP4, tens to hundreds of MB, pinned for the rest of the session, and once per raced fetch. Both galleries now discard a blob whose record went away, with an epoch covering the video page's Clear all. The video backend keeps the last completed job until the next one starts, and the Video page merges that record on mount to cover a job that finished after the gallery fetch. Deleting the clip left the record in place, so every reload prepended a ghost card whose file request 404s until another generation replaced it. Deleting the clip, or clearing the gallery, now clears the matching terminal record, and the page skips a record it deleted itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Resolve revisions from the live cache, serialize dataset imports, and stop pinning every gallery blob Five more items from the review round. The conditioning-cache revision marker read huggingface_hub's import-time HF_HUB_CACHE constant. Studio can move its cache during a session and loading follows the live setting, so after a move the marker went unresolved (or pointed into the previous root) and pulling a new revision of the same checkpoint no longer invalidated the cache: a warm run could reuse the old encoder's embeddings and the old VAE's latents. It now looks in the active Studio cache first and keeps the environment and the library constant as fallbacks, which the trainer subprocess still needs. The dataset interlock counts mutations rather than excluding them, so two imports of different examples into the same empty name both got past the emptiness check. The winner promoted its staging directory atomically; the loser found the folder non-empty, fell back to a per-file move, and merged its images and captions into the winner's dataset. Imports now take a per-folder lock, a second one is refused with 409, and the emptiness check is repeated under the lock. On Windows the sd.cpp asset resolver filtered only by accelerator token, so a Windows arm64 host matched an x64 zip, downloaded and installed it, and failed later when the binary would not run. It now filters by architecture the way the Darwin and Linux branches do. Every gallery page fetched every PNG up front and kept the object URL for the session, so scrolling a large gallery grew memory without bound for tiles the user may never look at. The Images strip now fetches a tile as it nears view, like the Video strip, and keeps the eager path only where IntersectionObserver is unavailable. A 503 carrying a JSON body comes from the application, not a proxy, so it is surfaced as the error it is instead of entering lost-response settlement and being reported as a request that never reached the server. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the class of a failed generation instead of a bare "Image generation failed." Found on a macOS runner: the native renderer aborts inside its own text encoder there, and the page showed only "Image generation failed." with the sd-server backtrace left in the server log, so nothing about the failure reached the user. The failure is now classified into fixed text, out of memory and native-process death, so the message says what happened and what to try. None of the engine's own output is echoed, since a native tail carries local paths and argv; that stays in the log, and an unrecognised failure keeps the original literal. * Treat an undecodable caption sidecar as the tombstone the trainer sees Uploads store .txt and .caption sidecars as raw bytes, so one can hold invalid UTF-8. The trainer treats any existing sidecar, decodable or not, as an empty tombstone and never falls back to the metadata row for that image. The labeling grid and the dataset summary read an undecodable sidecar as absent instead, so both showed a metadata caption that the run would silently replace with the instance prompt, and counted the image as captioned. Both now track sidecar presence separately, so what the user reviews is what the run trains on. * Keep the reason a native server died, not just its backtrace A ggml abort prints its cause first and then a stack trace, so reporting the last twenty captured lines gave twenty addresses and nothing about the failure: on the macOS runner the native server died on an unimplemented Metal op and the message carried only frame pointers. The captured tail now leads with the lines that name a cause and keeps recent context after them, for both the startup failure and the mid-request death. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop staging the dense text encoder for an fp8 video load Two halves of the same gap, found while measuring the LTX-2.3 download plan: - The video download plan and the scoped pre-download never saw text_encoder_quant. An fp8 request loads a hosted pre-cast encoder, so asking for one still staged and downloaded the base repo's dense Gemma3 (48.79 GB of Lightricks/LTX-2 on the 2.3 distilled pick) that the pipeline then never opened. The plan now drops those shards and stages the pre-cast checkpoint instead; their configs stay, since the pre-cast loader still meta-inits the encoder from the base repo's component config. - The LTX-2.3 assembly builds every component itself, so pipe_kwargs (which carries the pre-cast encoder for from_pretrained) never reached it and an fp8 request silently loaded the dense encoder anyway. It is passed across explicitly now. The dense skip is earned, not assumed: only a pre-cast checkpoint that resolves on the Hub lets the plan drop the dense shards, and only one already fetched to disk lets the pull drop them, so an unpublished or gated artifact leaves both exactly as they were. If injection still fails after that, the load tops the dense weights back up rather than handing from_pretrained a snapshot with no encoder in it. Measured against the real Hub on the 2.3 distilled Q4_K_M pick: 67.24 GB before, 18.92 GB with a 0.43 GB stand-in for the pre-cast artifact (the base entry drops from 24 files / 48.79 GB to 13 files / 0.04 GB). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Match the file's typing idiom for flow_shift models/training.py annotates with typing constructs throughout (105 Optional[...], no Union), and flow_shift was the one place using PEP 604. Union[] reads like the rest of the file, and it also drops the runtime evaluation that would raise on Python 3.9. * Bound the gallery blob cache, and three interlock fixes Four review findings, all reproduced first: - The gallery object-URL caches were unbounded. A clip runs from a few MB to a few hundred, both pages stay mounted after their first visit, and entries were only dropped on delete, so scrolling pinned everything for the session. Both pages now share a byte-budgeted LRU (512 MB video / 192 MB images) keyed off the visibility signal the near-viewport fetching already provides. On-screen media, the selected clip or image, and the item just fetched are never evicted, so eviction is invisible and a single item larger than the whole budget cannot evict itself into a refetch loop. - The image, video and chat load guards ran two independent training probes but returned early when the FIRST one raised, so an unreadable LLM backend disabled the diffusion interlock and a load could proceed straight into an active diffusion trainer on the same GPU. The probes are independent now. - An engine switch swallowed a failed teardown and published the new engine anyway, which is exactly the leak the unload exists to prevent: the arbiter's evictor, /images/unload and the next load all resolve through get_active_diffusion_engine(), so the still-resident pipeline (or a live sd-server) became unreachable and the next load allocated on top of it. The switch now fails and leaves the old engine published, so it stays reclaimable. - The native generation timeout was 30 minutes while the Images page waits up to 6 hours (SETTLE_MAX_MS), so slow-but-progressing CPU jobs died deterministically at the deadline. Measured on GPU-less runners, a 512x512 4-step Q2_K generation took 900 s on Linux and 1465 s on Windows, so larger images or step counts clear half an hour easily. The ceiling now matches the page's window and applies to the whole request: chunks of a split batch share one deadline instead of each getting a full budget. Cancellation is unchanged. Declined: gating the huggingfacenotorch extra off Python 3.9 over the conditional diffusers marker. The marker is deliberate and its comment says why: diffusers dropped 3.9 in 0.38, so pinning >=0.39 outright leaves pip no candidate and the whole extra unresolvable there. The pipelines it names live in studio/backend, which cannot install on 3.9 anyway (studio.txt pins matplotlib==3.10.9 and fastmcp>=3.0.2, both requires_python >=3.10), and the extra is the general core one, so the alternative drops 3.9 for library users who never touch Studio. * Close the load-versus-training-start race, and two picker fixes - The image and video load guards read is_active() and only then selected an engine, acquired the arbiter and registered the load. A /train/diffusion/start reserving inside that window freed residents the load had not registered yet, so the trainer came up beside a brand-new pipeline. The service already had exactly the right pattern for this in dataset_mutation, so gpu_load_admission mirrors it: reserve() refuses while an admission is open, an admission refuses once a start is reserved, both decided under the one lock. The span is only the registration, since begin_load returns as soon as the load is registered and _free_gpu_for_diffusion_training preempts an in-flight load from that point. Chat is deliberately not covered: its load spans an eviction plus a multi-minute GGUF load, and it admits models that fit beside training by design, which is a different contract from the diffusion pipeline's all-or-nothing one. - Hugging Face gives the LTX-2 family the image-to-video pipeline_tag (both Lightricks/LTX-2 and unsloth/LTX-2.3-GGUF report it), so a text-to-video-only filter dropped the flagship audio family out of Video Hub search while the rest of the app routed it to Video. - Task-scoped quant fit sized picks against the LARGEST visible device while resolve_diffusion_device_target returns a bare "cuda" and torch places on the current one. On a heterogeneous host that recommended a checkpoint sized for the bigger card and then loaded it onto the smaller one. Fit now uses the device the load actually lands on; identical on a homogeneous host. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Expose the persistent conditioning cache in the start schema DiffusionLoraConfig has carried cond_cache_dir for a while and the DiT trainer acts on it, but DiffusionTrainingStartRequest omitted the field, so Pydantic dropped it silently and every API-driven run fell back to the in-memory cache that is rebuilt from scratch each time. The warm path skips loading the VAE and the multi-GB text encoders on a rerun whose images, captions and resolution are unchanged, so this was a real capability that could not be reached. Contained like output_dir rather than left to the trainer subprocess's cwd, since it is another directory the trainer writes to. Blank or omitted still means the in-memory cache, so it must not resolve to the outputs root. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix diffusion policy and classification issues from review fp8 auto precision defaulted to precise accumulate on any non-consumer GPU, which made fp8 2.05x slower than int8 on RTX 6000 Ada and slower than not quantising at all. NVIDIA's professional whitepapers do publish equal FP8 rates for both accumulate modes there, so the hardware premise held, but the cost is in the cuBLAS path rather than the published rate. Default to fast accumulate: measured on B200 the flag is a no-op (4096^3 _scaled_mm at 3023.8 vs 3041.8 TFLOP/s, bitwise-identical output, 1.213 s vs 1.230 s end to end), so it is a large win where it bites and free where it does not. Precise accumulate stays available via transformer_quant_fast_accum. Z-Image's DiT is a Lumina2 derivative, so unsloth/Z-Image-GGUF and unsloth/Z-Image-Turbo-GGUF both declare general.architecture = "lumina2" and the whole line was tagged image-diffusion-unsupported and hidden from the Images "On Device" list, though validate_load_request loads them. Resolve shared archs from the repo/file name like bare "wan" already does, with a test asserting the picker and the loader agree for every family. The sage attention on-demand install ran an unpinned `pip install sageattention`, but PyPI's newest wheel is 1.0.6 and diffusers refuses anything below 2.1.1: the install always "succeeded", wrote an unusable version into the running venv, and was rejected on the next line. Carry the dispatcher's floor so pip resolves nothing instead. The dense-quant disk gate sized the download from the bf16-RESIDENT table. The fp32 families download twice that (Z-Image: 23,479 MiB against a 21,970 MiB gate), leaving a window where the check passed and the download filled the disk; Ideogram 4 ships fp8 and was overcharged the other way. Size the gate by published bytes, verified against HF sibling metadata for all 12 families. Patch installs went through unsloth_zoo, which refuses to import unless UNSLOTH_IS_PRESENT is set, and that is set by unsloth itself. The server imports unsloth at boot so it never showed there, but any other process ran silently unpatched with every install returning False, which is 13 test failures on a clean environment. Import unsloth and retry once, memoised per process. Also: the GGUF+LoRA refusal pointed at the native engine without saying a GPU host only selects it under UNSLOTH_DIFFUSION_ENGINE=sd_cpp, so the suggestion was unreachable; the gallery recipe recorded loras from the generate request alone, losing a load-time bake; load-progress claimed "40.07 GB downloaded" for a fully cached load; and pickers.tsx imported three catalog-group helpers it never used. Reported by oobabooga. * Keep the sd.cpp text encoder on CPU under Metal macos-14 loads FLUX.2-klein-4B Q2_K natively on mps and then dies on the first generation with exit code -6: ggml_metal_op_encode_impl: error: unsupported op 'RMS_NORM' -> ggml_abort LLMEmbedder::encode_prompt -> LLMRunner::compute -> GGMLRunner::compute ggml's Metal backend gates RMS_NORM on contiguous rows and aborts the process when that does not hold, with no per-op CPU fallback, so any LLM text encoder (Qwen3 for FLUX.2 and Z-Image, T5 for FLUX.1) takes sd-server down. The encoder runs once per prompt while the DiT runs every step, so pinning only the encoder keeps Metal for the part that matters. UNSLOTH_DIFFUSION_SD_CPP_METAL_TE_GPU=1 opts back in once ggml grows the kernel. * Gate the unsloth retry in the diffusion patch backend The retry added for the clean-environment patch failures is not free: importing unsloth pulls torch in behind it, which costs ~940 MB of RSS measured in a process that had neither, and on a host with no accelerator it fails anyway. A cross-platform CI job that had generated fine at ~900 s later died 19 s in with SIGTERM and every 'if: always()' step skipped, which is the runner being torn down rather than a step failing. Retry only when torch is already imported (true of the server and of anything patching a real module, and the condition that stops the retry from being what loads torch), unsloth is installed but not yet imported, and the first failure was the ImportError the sentinel guard raises. The clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only retry the unsloth import where it can succeed The gate still let the retry run on hosts unsloth does not support, which is where it is most harmful: a 7 GB macOS runner lost the Studio server 26 s into a load, and the Linux runner was torn down mid-generation. Neither MPS nor plain CPU can complete the import, so the retry there pays the cost and fails anyway. Require an accelerator unsloth actually supports (CUDA/ROCm via torch.cuda, or XPU), with UNSLOTH_ALLOW_CPU as the documented override, and hoist the predicate to module level so it is tested directly rather than through the import system. On a CPU-only host the retry no longer fires at all; on CUDA the clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Guard old diffusers, stream video exports, record conditioned recipes Three fixes from review. The 0.39-only pipeline classes (Flux2Klein, Z-Image, Krea 2, LTX-2, HunyuanImage) were resolved by getattr deep in the load, so on the older diffusers that packaging still allows on Python 3.9 -- diffusers dropped 3.9 in 0.38 and this project still supports it, so the 0.39 floor has to be conditional or the extra becomes unresolvable -- an advertised model failed with a bare AttributeError after its checkpoint had already been downloaded. Krea 2 already guarded itself this way; assert_pipeline_class_available now runs the same check for every image and video family from validation, before any fetch, and names the version and the fix. WebM export accumulated the whole VP9 output in a BytesIO and returned it as one bytes object that the response held again. The request caps allow 2048x2048 for 1024 frames, so an export runs to hundreds of MB and concurrent clicks could exhaust the process, while the MP4 route beside it already streamed from disk. transcode_to_file encodes to a temp file and the route returns a FileResponse with a background unlink, so nothing large is resident. A conditioned generation's recipe carried only the txt2img fields, so the gallery presented an inpaint or upscale result as a complete Create recipe and restoring it replayed an unrelated text-to-image request. The images themselves are still not persisted (user uploads with their own lifetime), but the workflow and its scalars are, restore reapplies them, and the toast now names the inputs that have to be supplied again instead of silently landing on Create. Reported by Codex. * Per-load video cancel event, family-gated image picker, cond cache refusal A cancelled video load could resume: begin_load cleared the shared cancel event, and unload() drops _loading without waiting for the worker, so the next load cleared the very object the cancelled worker was watching and its multi-gigabyte pull ran on alongside the replacement until the token check at the end. Each load now gets its own threading.Event, passed down through _fetch_te_prequant and _predownload_base, so a cancelled worker stays cancelled. A cached repo with a model_index.json was advertised as text-to-image on the trust rule alone, but validate_load_request also requires a detected image family, so a trusted pipeline of an unsupported class produced a picker row that deterministically 400s. The picker now applies both gates, mirroring the video branch. cond_cache_dir was accepted for sdxl and then ignored: only the DiT trainer reads it, while the SDXL trainer builds a per-run in-memory latent cache, so the promised cross-run reuse never happened. The route now refuses it with a 400 that names the families which do support it, checked against the resolved family so an omitted model_family with an SDXL base is caught too. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the frontend build broken by the gallery blob cache tsc -b failed on the branch head, so npm run build produced no dist and every platform job fell back to --api-only: blob-url-cache.ts(29,15): TS1294: This syntax is not allowed when 'erasableSyntaxOnly' is enabled dataset-labeling-grid.tsx / dataset-showcase.tsx: Argument of type '{ url: string; bytes: number; }' is not assignable to parameter of type 'string' The cache took its budget as a constructor parameter property, which the project's tsconfig forbids, and fetchGalleryObjectUrl now returns the blob size alongside the URL for that budget, which the two dataset thumbnail components still consumed as a bare string. Declare the field explicitly and destructure the URL at both call sites. tsc -b is clean and vite build emits dist again. * Recover from a ggml unsupported-op abort by restarting on the CPU backend ggml checks every node against the device's supports_op and calls GGML_ABORT when one is not implemented, because a single-backend graph has nowhere else to put it: there is no per-op CPU fallback. The whole sd-server dies with SIGABRT mid-generation and the user gets "the native image renderer stopped unexpectedly" with no way forward. Seen on macos-14 arm64 with FLUX.2-klein-4B Q2_K through the cross-platform CI: the text encoder is already pinned to CPU, and the abort moved into the denoise loop instead. ggml_metal_op_encode_impl: error: unsupported op 'MUL_MAT' -> ggml_abort StableDiffusionGGML::sample -> sample_k_diffusion A retry on the same backend would abort identically, so the load is restarted once with --backend cpu (the only flag that changes which backend executes the graph; --offload-to-cpu moves parameters, not compute) and the generation is re-submitted. The same checkpoint then renders slower rather than not at all. Strictly bounded: the signature must carry both the unsupported-op line and ggml_abort, the device must not already be CPU, and it happens once per load, so an OOM kill or a genuine crash still surfaces as itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix two tests that only fail in a full-suite run The 3.10 CI leg resolves PyAV 17, where av.container.OutputContainer is an immutable C type, so the no-libopus export test died on "cannot set 'add_stream' attribute of immutable type" before it asserted anything. Inject the refusal by wrapping the container av.open() returns instead; modules stay patchable on every build. Removing the injection makes the test fail again, so it still covers the branch it is named for. The Xet shim's degraded-path tests drop utils.hf_xet_fallback from sys.modules and import a throwaway copy. Restoring only the sys.modules entry left the utils package attribute bound to the throwaway, and the two disagreed for the rest of the process: a later monkeypatch of the dotted target patched one copy while the code under test imported the other, so the patch did nothing and test_fetch_te_prequant_only_reports_what_it_downloaded reached the real Hub and got a 401. Restore both bindings. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop an ignored-cancel sd-server, guard deletes during diffusion training, repair unusable managed binaries sd-server does not interrupt an in-flight job, so when it ignores a cancel the grace branch abandoned the poll and reported cancellation while the native job kept a core (or the GPU) busy to completion and held the server's job slot. The comment said the caller stops the server, but only unload does that immediately: a superseding load stops it after its multi-gigabyte download, and a load that then fails never gets there. Stop it here, as the deadline branch already does. DELETE /api/models/delete-finetuned checked only the LLM trainer, so it could rmtree the output directory a live diffusion LoRA run was about to write its adapter into. Consult the diffusion training service too, like the dataset mutation and model-load routes. find_sd_*_binary only checks is_file(), so an interrupted extraction (or a prebuilt for the wrong CPU) left a present-but-unrunnable binary the installer never retried: every load probed it, fell back to diffusers, and native inference stayed off until the directory was deleted by hand. Probe it and reinstall, but only for a copy under the installer-owned root -- SD_CLI_PATH, UNSLOTH_SD_CPP_PATH, an in-tree build and anything on PATH are the user's. * Plan the pre-cast text encoder, and make the cross-trainer GPU admission atomic An fp8 text-encoder request loads a hosted PRE-CAST checkpoint, but the image download plan never received text_encoder_quant, so the manager staged the base repo's dense encoder (FLUX.2-dev's Mistral-24B is ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) and the load then pulled the pre-cast file inline, outside the manager's progress and disk preflight. The plan now takes the field, resolves the hosted artifact with the same resolver the injection uses, stages that file, and drops only those components' dense weight shards. The load's own prefetch takes the same treatment, since it paid the same cost. Only a checkpoint that really resolves on the Hub earns the drop, so a gated or renamed artifact still stages the dense encoder the load will fall back to. The two trainers admitted each other with independent check-then-act guards: the diffusion route checks the LLM backend several network-bound preflights before it reserves, and the LLM route checks the diffusion service well before it spawns, so two near-simultaneous starts could both pass and train on one GPU. reserve() now re-tests the LLM backend under its own lock, and the LLM route holds the diffusion service's gpu_load_admission across its spawn, so exactly one of the two wins. Both halves fail open, so a chat-only install still trains. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the Advanced options a staged download planned against Staging does not set busy, so while a multi-gigabyte download runs the user can still change precision, memory mode, speed or the baked LoRA selection. The pending record held only the repo and artifact, and the completed download fired a load that read the CURRENT state: the staged file set could then be missing files that load needs (fetched inline, with no progress and no disk preflight) or hold gigabytes it no longer uses. One snapshot of every Advanced control is now taken when the plan is built, and it travels with the pending record into the load, so the load that runs is the one the download was planned for. * Do not advertise a family the installed diffusers cannot build The newer families (Z-Image, Krea 2, FLUX.2, LTX-2, HunyuanImage) exist only from diffusers 0.39, and 0.39 cannot be installed on Python 3.9 at all -- diffusers dropped 3.9 in 0.38, so the requirement is conditional or the whole extra becomes unresolvable. On such an environment the picker still offered those rows, every pick failed deterministically, and the error's advice to run pip install -U diffusers could not fix it without also upgrading Python. The cached-repo picker now applies the same availability check validate_load_request does, which is keyed on the pipeline class actually present rather than on the Python version, so it is also right for an intentionally pinned older diffusers on 3.10+. Fails open when diffusers cannot be imported at all: that is a different problem and the load path reports it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore the diffusion engine selection after each router test The active engine is module state, and several tests set it by plain assignment because what _activate does to it is the thing under test, so monkeypatch could not undo it. A leaked ENGINE_SD_CPP left get_active_diffusion_engine() handing back the sd.cpp backend for the rest of the process, and every later route that reads the active engine then saw an unloaded model: eight tests in test_openai_images_generations_route.py returned 503 in a full-suite run while passing on their own. The autouse fixture now snapshots and restores it. * Stream gallery clips, and close three races around them Four fixes from the latest review pass. The video gallery downloaded each clip into a blob before it could play, so playback waited on the whole file (tens to hundreds of MB), seeking was limited to what had arrived, and every viewed clip stayed pinned in the webview. The file route already streams and serves ranges; it just could not be a <video src> because it is bearer-gated. Mint a short-lived signed link instead (its own HMAC secret, 12 hour TTL, separate from the image links) and hand it to the element, which then fetches only the ranges it plays. That removes the blob budget, its LRU and every revoke on this page. The sd.cpp readiness probe accepted any process answering on the port, so a foreign server that grabbed the port between the bind check and the spawn was adopted as ours. Confirm the listener is our child before reporting ready, and stay best-effort (psutil missing, an unknown owner, or any probe error still passes) so the check can only reject a definitely foreign process. Dataset import held its lock for the extract but not for the upload path, so two concurrent uploads into the same folder interleaved; take the same lock and return 409. And reject Windows device names (CON, NUL, COM1..9, LPT1..9, with or without an extension) plus trailing periods in dataset names, which are unopenable on Windows. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Build the image download plan for the engine that will load /images/download-plan always asked the diffusers backend, while /images/load picks the engine per host: a GGUF pick on a machine with no usable GPU routes to native sd.cpp, which reads a single-file VAE plus text encoders and never opens the base repo's sharded components. Measured on unsloth/FLUX.2-klein-4B-GGUF (Q2_K): the plan staged 7.66 GB of FLUX.2-klein-4B components the native load discards, and the 7.80 GB sd-cli actually needs was then fetched inline by the loader, outside the download manager's progress and its disk preflight. Z-Image-Turbo is the same shape. The plan now asks whichever engine the load will select. predict_engine() applies the selection policy without any side effect: it activates nothing (staging a download must not unload the resident model) and only locates the binary rather than installing it, but still counts an installable binary as available, since that is what the load does on a fresh host. The native backend gains a download_plan built from the same _asset_specs the loader fetches, returning the same envelope, so the manager stages exactly the files sd-cli opens. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not let a queued generation outlive the model, and three scan fixes Five items from the latest review; four were real. An unload or arbiter eviction only cancels the generation holding _generate_lock. A second request queued behind it holds no cancel event yet, and Python locks are not FIFO, so it could take the lock the instant the active denoise released it, still see a loaded pipeline, and run a whole new denoise after the model was told to go away: the eviction then waits minutes for it and an image lands after the eject. Unload and a superseding load now raise a fence under _lock before they queue, and a generation that wins the lock while one is pending refuses instead. The cached-model scan judged pipeline completeness across every revision, so a repo holding an older complete snapshot plus a newer companion-only one read as complete while the snapshot from_pretrained actually opens has no transformer. Both scans now look at the revision the loader will open. Deleting a dataset image deleted its caption sidecar unconditionally, which for cat.jpg alongside cat.png removed the caption the survivor still resolves to. The sidecar now goes only with the last image of that stem, matching what the thumbnail cleanup beside it already did. Importing an example into a folder that holds no images but does hold files fell back to promoting the staging dir one file at a time, so an interruption left a partial dataset that the image_count check accepts as complete on retry. Those files are folded into the staging dir instead and the promotion stays a single atomic rename. The MPS generator report does not apply: torch.Generator(device="mps") has worked since PyTorch 2.0 (pytorch/pytorch#91348) and the studio installer pins torch>=2.4. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten diffusion comments Collapse the multi-line comment blocks across the image, video, sd.cpp and diffusion-training code to one or two lines each, and drop comments that only restate the statement below them. Comments only, no code or behaviour changes. * Tighten diffusion comments (second pass) Collapse the remaining multi-line comment blocks in the video page, training routes and service, sd.cpp server and installer, memory and speed planners, and the shared request models. Comments only, no code or behaviour changes. * Tighten diffusion comments (third pass) Collapse the remaining multi-line comment blocks in the attention, cache, LoRA, prequant, precision and compile-cache modules, the sd.cpp arg builder and engine, the video routes, the Ideogram 4 assembly, the model picker, and the diffusion test suites. Comments only, no code or behaviour changes. * Restore the dataset when an example import cannot be promoted Promotion folds the folder's pre-existing entries into the staging dir so the swap is one atomic rename. Every failure after that fold left the user with nothing: the 409 path and an os.replace error both fell through to 'finally: shutil.rmtree(staging)', which deleted the entries that had just been moved in there, while the response said 'Nothing was written'. A same-named entry was also unlinked outright before the promotion was known to succeed. Park superseded same-name entries in a rescue dir instead of deleting them, record every move, and restore all of them if any step of the promotion fails. The fold loop itself is covered too: renaming a non-writable directory raises EACCES on POSIX, which previously escaped as a 500 after the earlier entries had already been moved out. A failed rename now maps to the same retryable 409 as the rmdir conflict. Verified with the reported trigger (a non-empty mode-500 directory whose name collides with an imported file): the folder listing is now identical before and after the failed import. * Drain the teardown fence on a failing unload, give each load its own cancel event Two independent leaks on the image path, both already solved elsewhere in the same file. unload() incremented _teardown_waiters, ran _unload_locked() and decremented, with no try/finally, while the superseding-load path used a finally for the same pair. _unload_locked ends in clear_gpu_cache(), whose CUDA branch calls synchronize/empty_cache/ipc_collect unguarded, and a sticky CUDA fault makes those raise. The count then never drained, so every later generation was refused as cancelled for the life of the process, a fresh load included, since begin_load's own increment and decrement are symmetric. unload() is reached from the chat/video GPU handoff, the engine router and two training routes, so one fault during an ordinary handoff wedged image generation until restart. Release it in a finally. The image and native backends each cleared one shared cancel Event on a new load. unload() sets that event to cancel an in-flight multi-GB download and drops _loading in the same breath, so a replacement load is admitted while the cancelled worker is still inside the fetch, and its clear() re-enabled the very object that worker was watching: the cancelled download resumed and ran alongside the replacement. Take a fresh Event per load and thread it to the worker, as the video backend already does, and set it under the lock since begin_load now rebinds the attribute. * Name utf-8 on the diffusion text I/O and the sd.cpp subprocess pipes tests/test_text_io_encoding.py failed on five files this branch adds. Text I/O without an explicit encoding falls back to the Windows ANSI codepage, so a non-ASCII path or manifest value round-trips corrupted, and the three sd.cpp pipes decode the child's UTF-8 output as ANSI on Windows despite already passing errors = 'replace'. Eleven read_text() / write_text() sites across diffusion_compile_cache, diffusion_ideogram4 and diffusion_krea2, plus text = True on the sd-cli version probe, the sd-cli run and the sd-server pipe. * Record the load-time build on a gallery image's recipe A gallery record documents itself as the image's full generation recipe and is embedded in the PNG, but the only load-related field it carried was the repo id. A GGUF repo holds many quants, so that does not say which one made the pixels, and it says nothing about an adapter baked in at load time. The fallback meant to cover the baked case could never fire: with no loras on the request _adjust_baked_loras zeroes every baked adapter and _active_lora_pairs drops zero-weight entries, so active_loras was always empty. A baked-and-disabled build is not the same pipeline as a never-baked one, so the recipe could not reconstruct the image once the model was rebuilt. Persist model_kind, gguf_filename, transformer_quant and the baked adapter names, read off the load state rather than the request, and show them in the recipe popover. The new fields are optional with defaults, which matters because list_gallery_images drops any record that fails validation, so a required field would have emptied every existing gallery; a regression test pins that. * Drop eleven duplicated comment tails, restore the mxfp8 denial note The comment passes collapsed several wrapped blocks onto one line without deleting the last physical line of the original wrap, leaving the tail of each sentence repeated as its own comment underneath. Two of the eleven were re-worded rather than byte-identical, so a strict suffix match missed them. 6e16ad16f also dropped the line justifying the qwen-image mxfp8 denial while that rule stayed live in _FAMILY_SCHEME_DENY, under a header that then documented only fp8 and nvfp4. Restored. Comments only; verified with the AST gate. * Only let the dense-quant fallback use shards the prefetch actually staged The prefetch skips the base repo's transformer/ shards whenever a prequant checkpoint is expected, since that checkpoint replaces them. But a prequant fetch can fail for reasons the planner cannot see: an unpublished, gated or renamed artifact, a hub 5xx, a proxy, a checkpoint the validator rejects. The loader then fell through to from_pretrained(subfolder = 'transformer') and pulled those shards inside the load lock during 'finalizing', after the previous pipeline was already evicted, where the cancel event has no reach, load_progress has already reported bytes_downloaded == bytes_total, and the cache-disk gate had only reserved the small prequant checkpoint. That is verbatim the situation _dense_quant_prefetch_needed's own docstring exists to prevent. Gate the in-loader dense fallback on the shards being staged, read off the returned file list rather than the request so a failed size estimate closes it too, and let the GGUF build take over otherwise, which is what the prequant-sized replan already does one branch over. It is also the invariant the text-encoder path already enforces: only a component that really resolves may have its dense weights dropped from a plan or a prefetch. Adds the missing coverage for the gate's prequant arm, which the existing disk-gate tests never reached. * Refuse a training output_dir that resolves to the outputs root resolve_output_dir strips a leading 'outputs' and drops '' / '.' segments, so '.', './', './.', 'outputs', 'outputs/outputs' and ' . ' all clean away to nothing and land on the outputs root itself rather than a run directory under it. The DiT trainer then writes pytorch_lora_weights.safetensors flat into the root, where scan_trained_models and scan_checkpoints cannot see it (both filter is_dir()), and a second such run overwrites the first. The UI only checks the field is non-empty, so 'outputs' is one plausible run name away. Refuse it with a 400 that says what to do instead. cond_cache_dir collapses the same way but has an honest 'off' to fall back to, so a root-resolving value now means the in-memory cache, which is what the comment above it already promised: otherwise a run drops one flat safetensors per cached latent and caption into the directory trained models live in. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Repair three defects the main merge left behind Git merged all three files without a conflict, but the result was wrong in each case. The tests only started failing once the merged tree was run. routes/inference.py duplicated the GGUF load block: this branch had moved the gguf_load_in_flight marker and the _hub_download_blocks_gguf_load guard under "if config.is_gguf and config.gguf_hf_repo", and the conflict resolution re-added main's copy at the old position, so both ran. Dropped main's copy; the earlier placement is the deliberate one, so a 409 from the hub guard cannot tear down a resident Images or Video pipeline. test_gpu_selection.py still called _hf_offline_if_dns_dead, which main renamed to _hf_offline_if_unreachable_for (#7591). Disjoint edits, so no conflict, but four route-error tests referenced a function that no longer exists. test_gguf_load_cache_reuse.py anchored its ordering assertion with rindex over "if config.is_gguf:", taking the last one before the load marker. That only held while _resolve_inherited_extra_args sat above every such line; main has since hoisted it above the gpu_ids preflight, so the anchor landed between the call and the marker and the assertion compared against an unrelated later call site. The ordering it checks is a property of _load_model_impl as a whole, so it now anchors on the function. The ordering itself is intact: _resolve_inherited_extra_args, then the gguf_load_in_flight marker, then the hub guard, then the chat handoff, then unload_model. * Stop a real sd.cpp install from breaking its own discovery tests The five "nothing is installed" assertions in test_sd_cpp_engine.py cleared SD_CLI_PATH and UNSLOTH_SD_CPP_PATH and patched Path.home, which covers hops 1, 2 and the fallback half of hop 3. It leaves two hops live. Hop 3 goes through managed_install_root(), which honors UNSLOTH_STUDIO_HOME and STUDIO_HOME and resolves to the stable-diffusion.cpp directory beside the studio home. That is the documented way to run side-by-side Studios, so anyone who has one set gets a real sd-cli back from a finder the test expects to return None. Hop 4 is the in-tree developer build, which does the same for anyone who built sd.cpp inside the checkout. Isolated with an autouse fixture rather than another helper call, because reaching the failure needs no fixture: SdCppEngine(binary = None) runs the finder from its constructor, which is why test_generate_raises_when_binary_missing failed too. The fixture points the studio home and the in-tree root at an empty tmp tree, so every hop is answered by the test rather than by the box. The in-tree root moves into a named in_tree_install_root() so it can be pointed somewhere empty; behaviour is unchanged, including the OSError and IndexError guard on an unexpected layout. * Make the sd.cpp uninstall test actually extract the code it tests The two sed ranges anchored the production fragments at column 0, but both blocks sit inside the main removal function and are indented, so each range matched nothing and LOOP_FILE / DEFAULT_FILE came out empty. Sourcing an empty file is a no-op, so the suite reported 4 passed / 7 failed: the seven removal assertions failed because nothing ran, and the four that passed were 'kept' assertions that pass trivially when nothing runs. The shell job auto-discovers tests/sh/test_*.sh and runs each under set -e, and this file is not in its skip list, so it was a deterministic red. Anchor on optional leading whitespace and fail loudly on an empty or _remove_path-less fragment, so a future reshuffle of uninstall.sh cannot make the suite vacuous again. Now 11 passed, 0 failed, against the real removal loop and the real default-mode block. * Agree on what an engine can build, and on what the installer owns Two gates, each half-applied. The unbuildable-family gate had one caller, the image branch of the cached repo picker. The GGUF classifier, the local-model classifier and the video branch had none, so on a diffusers too old for a family the picker still offered its GGUF and the load then failed. Meanwhile the loader asserted the diffusers pipeline class before engine selection, so a GGUF this host routes to native sd.cpp, which instantiates no pipeline class at all, was refused with an upgrade instruction that could not help. Both are wrong one-sidedly: hide a family only when NEITHER engine can build it, and demand the diffusers class only when diffusers is what will load it. One predicate, family_buildable_here, now answers both, so the picker and the loader cannot disagree. The population is real on Python 3.9, whose diffusers ceiling is 0.36: Flux2KleinPipeline, Krea2Pipeline and LTX2Pipeline are all absent there, and FLUX.2-klein GGUF is a repo in this PR's title. That assertion also raised RuntimeError, which /images/load maps to 409, the status that otherwise means a load is already in progress, and which escaped /images/download-plan (it catches ValueError and FileNotFoundError) as a bare 500 with the message lost. It is an unloadable pick like every other, so it raises ValueError and both routes answer 400 with the text intact. The managed-binary repair used a path test for ownership while the installer uses a marker. On a managed root without the marker, which is any install predating it, the repair deleted sd-server and the reinstall was then refused, permanently, because the surviving sd-cli keeps the directory non-empty so the marker can never be claimed: the user went from an unrunnable binary to no binary and no way back. Require the marker before discarding, which is the same definition of ours that uninstall.sh already uses to keep a user's own stable-diffusion.cpp checkout. A genuinely interrupted extraction still self-heals, since install() writes the marker before it extracts. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add gguf and av to the studio extra so the wheel install matches studio.txt This branch added both to studio/backend/requirements/studio.txt, which is what install.sh uses, but never to the studio extra in pyproject.toml, which is what 'pip install unsloth[studio]' uses. Nothing else keeps the two in sync, so tests/studio/install/test_studio_extra_matches_requirements.py failed deterministically, and the CI job runs the whole tests/ tree. The drift is not only a red test: gguf backs diffusers' GGUFQuantizationConfig and av does the MP4 encode and audio mux, so a wheel install got a Studio that cannot read a GGUF or export a video, which is most of what this PR adds. * Fix two test-suite defects: the CPU-only patch gate and a popped module The patch backend reaches unsloth_zoo's helpers only through an ~940 MB 'import unsloth', which a CPU-only host cannot complete, so every patch install returned False and nine arch/eager/compile tests failed there -- exactly the runners the retry was narrowed to protect. unsloth_zoo only wants UNSLOTH_IS_PRESENT in the environment, which costs nothing and needs no accelerator, and the conftest already sets its sibling UNSLOTH_ALLOW_CPU. Set it there too, as run.py and main.py already do at module scope, so no real server ever took the expensive route. CPU-only goes from 9 failed to 24 passed, 1 skipped; the GPU run is unchanged at 29 passed. test_setup_cache_env_hf_home popped utils.hf_cache_settings to model a fresh process and never restored it, so a later import built a second module object and rebound it on the utils package. test_hf_cache_settings then wrote its setting into one object while core.inference.diffusion read the other, the same split-module failure the xet shim already had. Restore both bindings in teardown: 21 pass together, and each file still passes alone. * Restore the setup_fail assertions the main merge reverted A fourth instance of the class 9541cc535 fixed: the merge took main's studio/setup.sh, which #7644 changed to abort through the setup_fail helper so desktop mode still emits [TAURI:ERROR], but kept this branch's older copy of the test, which still asserted the literal 'exit 1'. setup.sh is byte-identical to main here and the only diff in this file was the reverted assertions, so take main's version. 60 passed. * Fence the video teardown, reject non-finite clipping knobs, stop a test installing sd.cpp The video backend never had the teardown fence the image backend documents. Its unload and its superseding-load path both signalled the active generation, then did 'with self._generate_lock: pass' and tore the state down with the lock free. A generation queued behind that barrier holds no cancel event yet, so the signal cannot reach it, and Python locks are not FIFO: it won the lock the instant the barrier released it, read a still-loaded state and denoised a whole clip against the pipeline being freed. Reproduced on both paths, where the queued generation returned a finished MP4. Mirror the image backend: count waiters, refuse a generation while one is pending, and tear down inside the barrier with the counter released in a finally. _teardown_state becomes _teardown_state_locked since the caller now holds both locks. max_grad_norm and snr_gamma were bounded on one side only, so 1e309 floated to inf and passed. clip_grad_norm_ then computes an infinite clip coefficient, clamps it to 1.0 and scales nothing, and min(snr, inf) / snr makes every min-SNR weight 1.0. The run starts, reports normal progress and trains with the requested knob silently disabled. Measured both. NaN already failed the bounds; allow_inf_nan makes that explicit. learning_rate and flow_shift already guard this exact vector. test_load_routes_to_sd_cpp_on_cpu stubbed ensure_sd_cpp_binary but not ensure_sd_server_binary, which select_and_activate_engine probes first with installs enabled, so the unit test downloaded and unpacked 108 MB into the developer's real ~/.unsloth when UNSLOTH_STUDIO_HOME was unset. Now 0 bytes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Key the video schedule reset on the resolved defaults, not the repo id A GGUF repo holds several variants, so another client swapping a distilled build for the base one from the same repo changes the steps and guidance the backend reports while repo_id stays put. modelChanged stayed false, the page kept the previous schedule, and later generations ran an 8-step distilled setting on a model that expects roughly 40 steps and CFG 4. Include the reported defaults in the key. They only move when the resident artifact does, so a manual steps change still survives a status poll. * Retire the release-lag pin now that 2026.7.6 carries the relaxed gates The virgin Windows container row that installs from PyPI on purpose was tolerated with continue-on-error, because a Server Core container has no Microsoft Store and so no winget, and the released studio/setup.ps1 hard-stopped on a winget-only git gate and reached for winget again for the VC++ runtime. #7549 relaxed both, but it landed on 2026-07-28 and the newest wheel was 2026.7.5 from the 23rd, so the row could not get past it. The pin carried its own tripwire for exactly this moment, and it has fired: unsloth 2026.7.6 shipped 2026-07-29 and the released wheel now installs end to end, so the step reported success and the assertion errored with "delete this pin and drop continue-on-error from the Install step so this row gates". Doing that. Both rows now gate unconditionally, and the assertion that enumerated the accepted failure signatures goes with the pin it protected, since there is no longer an accepted failure for it to describe. The other three release-lag mentions in this file are descriptive rather than tolerated-failure pins: they explain why the fedora and ubuntu-nonroot legs run with overlay: true, and both already refuse a triton failure as a regression rather than accepting it as lag. Left alone. * Pin the hosted pre-quant onto the plain-torch fp8 kernel A Z-Image GGUF pick at the DEFAULT speed mode, once the hosted fp8 pre-quant is actually reachable, dies at generate with an HTTP 500: torch._dynamo.exc.Unsupported: Operator does not support running with fake tensors. Developer debug context: unsupported operator: mslk.f8f8bf16_rowwise.default _fp8_config already pins KernelPreference.TORCH when it BUILDS a config, precisely because the default AUTO switches to the MSLK kernel wherever an mslk package is importable (sm90+). A hosted checkpoint escapes that pin completely: the preference is serialized per Float8Tensor, and all 239 weights in the published Z-Image-Turbo-FP8 checkpoint carry AUTO. Loading it re-arms the very kernel the pin exists to avoid, mslk.f8f8bf16_rowwise has no fake impl, and the first compiled generate therefore cannot be traced. Isolated away from the product to be sure of the mechanism: quantise one Linear three ways on this box and compile each. KernelPreference.TORCH eager ok, compiled ok KernelPreference.AUTO eager ok, compiled FAILS on mslk.f8f8bf16_rowwise library default eager ok, compiled FAILS the same way So the pin is correct and necessary, and the only gap is that the hosted path never got it. _validate_checkpoint checks scheme, granularity, base model, min_features, exclude tokens and fast_accum, but not this. Rewriting the preference on load is safe: it selects a matmul kernel, it is not weight data, so the tensors stay bit-identical and the checkpoint's own state_dict_sha256 still describes them. It is also the faster path compiled, since the opaque extern call blocks inductor quantize fusion. Why this went unnoticed: the pre-quant repos are private, so nothing that could not read them ever took this path. It becomes the default the moment they are readable. Verified end to end at 1024x1024, 8 steps, speed_mode default, on the published checkpoint: before HTTP 500 at generate, diffusion.generate_failed after pinned 239 weights to the plain-torch fp8 kernel, load 10.0 s, transformer_quant fp8 with compiled engaged, cold 7.9 s, warm 0.87 0.87 0.87 0.86 0.87 s That warm number also beats the 1.4 s recorded for this shape on 2026-07-26. 159 prequant and transformer-quant tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cover the fp8 kernel pin so the 500 cannot come back quietly The hosted pre-quant re-arming MSLK was only visible as an HTTP 500 on a compiled generate, which needs a GPU, a readable private repo and an importable mslk to reproduce. None of that is available in CI, so the pin would rot unnoticed. Three hermetic cases on _pin_kernel_preference instead: AUTO weights are rewritten and already-TORCH ones are left alone (counting only what changed), a weight that refuses the assignment does not sink the whole load, and with no torchao enum available the checkpoint is left exactly as saved. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: tighten comments across the image generation, video and training code Rewrite the comments added by this branch to be shorter and clearer: collapse multi-line explanations into one or two lines, drop comments that only restate the code, and keep the rationale that explains why a choice was made. No code changes. * Studio: let the pipeline-class guard survive a host with no diffusers Backend CI installs the CPU-only dependency set, which has no diffusers, and runs without a GPU. 13 tests failed there for three separate reasons, one of which is a real product bug rather than a test-environment gap. assert_pipeline_class_available did a bare "import diffusers", so on any host without it the call raised ModuleNotFoundError instead of the ValueError its own contract promises. /images/download-plan catches only ValueError and FileNotFoundError, so that escaped as a bare 500 with the message lost, which is the exact failure the guard exists to prevent. It is reachable in production: the native sd.cpp engine serves GGUF picks on a CPU or Apple host where diffusers is never installed. Absent diffusers now returns, since the check answers "is the installed diffusers new enough for this family" and there is no version to judge; a pick that genuinely needs diffusers still fails in the loader with its own message. The rest were tests asserting through gates unrelated to what they cover: - Three cond_cache_dir route tests drive a DiT family, which the start route refuses without an accelerator. They now take the existing dit_train_host fixture, written for this case, so they keep testing the schema on every host. - test_download_plan_forwards_the_load_time_controls stubs the diffusers planner but sends a GGUF pick, which routes to the native planner on a GPU-less host. It now pins the engine, so it tests kwarg forwarding rather than the hardware. Engine selection keeps its own tests. - The pipeline-class guard test needed a real diffusers only for its sweep over every shipped family. That half is split out and skips on its own; the stub-driven refusal needs no diffusers and still runs. Added a test for the absent-diffusers contract above. 409 pass on a GPU host with diffusers; 301 pass and 1 skips with diffusers blocked and no GPU. * Studio: cut the image generation comments down further Second pass over the comments this branch adds. Collapse the multi-line blocks that still read as paragraphs, rewrite the longest one-liners so they say the same thing in fewer words, and drop a stale comment that had drifted away from the constant it described. No code changes: only comments and whitespace. * Studio: workflow rail, sidebar Video row, and Train settings polish Sidebar - Pin Video under Images by default; persist store bumped to v3 so an untouched install adopts it while a custom arrangement is left alone. - Add "Customize sidebar" to the end of the More flyout, opening Settings > Appearance scrolled to the sidebar nav section. Images - Replace the workflow dropdown with an icon rail down the left edge: all seven workflows visible, keyboard nav, tooltips carry the labels. - Workflows stay selectable with no model loaded, so one can be set up before picking a model. Generate is already gated on a loaded model. - Video moves out of the Create/Train strip into a link at the far right, with a matching Images link on the Video page. - Wider gutters around the settings column, headings matched to Train. Train - Field guidance moves into "i" tooltips; only state that limits a control stays on the page. Steps, LoRA rank and the rest gain hints. - /info reports params, qlora_vram_gb, gated and note as fields, so the family note renders as chips. vram_note is rebuilt from them unchanged for older clients. Shared - Number steppers appear on hover or focus. - Lighter control border (#e9e9e9) on Images and Video, light mode only. * Studio: move Advanced inline on Images and Video - Replace the top-bar toggle and right-docked Advanced panel with a disclosure under Seed, so load-time tuning sits with the settings it affects and opening it no longer shrinks the preview canvas. - Video uses the bordered variant, since offload and memory decide whether a model fits at all. - Open state persists per page in localStorage. - Tighten the Steps unit trigger: 14px chevron, less right padding. * Studio: full width media pages and layout polish - Drop the 1100px cap on Images Create, Images Train and Video, so the preview canvas grows with the window instead of sitting in a band. - Align the workflow rail to the model selector label, widen the gap before its divider, and restore the divider itself. - Advanced: more room above and below, icon and label sized to the slider rows, and the same quiet row on Video as on Images. - Give Video a pane heading and description, matching Images. - More space above Seed on both pages. - More no longer takes the active style when the current page is one of its own rows. * Studio: fix the Extend side toggles The resting outline used a ring, and index.css blanks the ring on the button holding mouse focus, so the side you just clicked lost its border until focus moved. Use a border for the outline and leave the ring to focus-visible. Also separates the two states properly (the off state had no surface of its own), drops the border in dark mode as the inputs do, and adds the aria-pressed and focus-visible styling the buttons were missing. * Studio: drop the border on the Extend side toggles Light mode should carry no border either, so the fill alone marks the state: a muted surface when off, a primary tint when on. Matches the borderless treatment dark mode already had. * Studio: move the Images workflows into the sidebar The workflow switcher was a vertical icon rail on the page. It now lives in the sidebar under Images, so the page keeps that width for the canvas. - Workflows list under the Images row, with a chevron to fold them away. Hovering the row peeks them in a flyout on the standard menu surface. - Clicking Images while already there toggles the list instead of navigating. - The listed workflow carries the highlight, so the Images row drops it. - Create takes its own icon: it was sharing the New chat pencil. - Model hub moves above Projects. Persisted layouts bump to v4 and only adopt the new order where the stored one is still a shipped default. - Images and Video content both start at 32px, clear of the sidebar. * Studio: simplify the Images workflow submenu Dropping the hover flyout: two ways to reach the same seven workflows, one of them overlapping the list right below it, read as clutter. - The flyout is gone. The workflows are rows under Images and nothing else. - On the Images page they are always open, since they are that page's switcher. Elsewhere they stay folded, and the row's chevron opens them. - Create takes the sparkles icon. - Create and Train both pad their settings column to 40px a side, so the two tabs line up with each other and with the model selector. * Studio: dock the generate action and tidy the Create controls The primary action sat at the foot of a long scroll, so it was off screen until you scrolled for it. - Generate, Video's Generate and Train's Start training float at the bottom of their settings column. No bar behind them: hover lightens the fill rather than thinning it, and the disabled state is opaque, so the controls underneath never show through. - Aspect ratios read as names: Square (1:1), Widescreen (16:9) and so on. - Width and height replace their sliders with two compact boxes. Type a size or pick one from the menu; the value still snaps into range and still drives the locked ratio. - Negative prompt gets a quiet disclosure under the prompt. It was there before but only above guidance 0, which is not the default, so it never showed. - The info "i" is smaller across the settings UI. * Studio: reveal Images and Video field hints on hover The "i" next to every field label sat there permanently, which made a column of settings read as busier than it is. Scoped to the diffusion pages by CSS rather than per component: the hints come from the page's own Field and SelectRow, the train panel's FieldLabel, and the chat ParamSlider, which is shared and should not change for chat. Reveals when the pointer is inside the field, on focus-within for keyboard, and stays inert while hidden. InfoHint grows a data-slot so the rule has a stable hook, rather than keying off its aria-label. * Studio: one-line sliders, edge fades, shared negative prompt - Steps, Guidance and the rest put label, track and value on one row. ParamSlider is shared with chat, so this is an opt-in prop and chat keeps the stacked layout. - The settings columns fade at whichever edge they run past, as the sidebar and model picker do, instead of cutting off. New useScrollFades hook drives all three. - Negative prompt moves to a shared component and Video picks it up, so both pages collapse it the same way. - Right gutter sits closer to the rule, Steps takes a bigger break above it, and the Advanced rule sits between its neighbours rather than up against the field above. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: add the API monitor to the sidebar nav Sits under More, below Export. Points at the monitor page, not the API keys dialog the profile menu opens. No store version bump: an id missing from a stored layout is appended with its default pinned state, so existing sidebars gain the row at the end and keep their own order. * Studio: take main's copy of the sidebar spinner test The merge kept this branch's older version, which pins the Recents row to h-[33px]. Main already relaxed that to any height, since row density moves independently of the trailing column the test is about. * Studio: fix FLUX.2 klein size resolution, HunyuanVideo tier routing, and the personalization save Five fixes found while auditing the model registries against the live Hub. FLUX.2 klein 9B loaded against the 4B config. One family covers both klein sizes and defaults to 4B, relying on the base_model card tag for the real base, but the trust allowlist only had klein-4B, so the correct tag was discarded and a 9B GGUF (inner_dim 4096) was loaded against a 4B config (3072). That surfaced as a bare shape mismatch from inside the GGUF quantizer naming neither the file nor the repo. Adds the missing allowlist entries and a header-level size check that fails early with a legible message; the check is fail-open, so an unreadable file, a non-FLUX.2 family, or an unmapped base leaves the load exactly as it was. The sd.cpp text encoder rule matched the literal "klein-9b", so klein-base-9B was handed the 4B encoder. HunyuanVideo-1.5 720p checkpoints routed to the 480p family. Only the literal 720p_t2v path was aliased, so 720p_i2v and every GGUF repack fell through to the generic token and inherited the 480p base repo, which also supplies the VAE and text encoder. The tier is baked into the weights (transformer target_size 640 vs 960, scheduler shift 5.0 vs 9.0) and the two bucket lists are disjoint, so this ran the whole pipeline off-tier. Resolution presets that the checkpoints were never trained for. Wan2.2 TI2V-5B is 720P-only upstream (SUPPORTED_SIZES is exactly 704x1280 and 1280x704, asserted in generate.py), and the HunyuanVideo 480p square preset 624x624 is not a bucket of its tier; 640x640 is. Personalization could never be saved. The frontend ships eight sidebar nav ids and refills every missing one on each save, but the backend Literal had seven and no "api", so every PUT to /api/settings/personalization returned 422. Aligns the server defaults and the Literal with the shipped layout. A catalog row pointing at a file that does not exist. unsloth/Qwen-Image-2512-FP8 holds torch prequant .pt checkpoints, not qwen-image-2512-fp8.safetensors, and fp8 is denied for this family anyway because it renders black, yet the row was the auto-route target on any GPU above 40 GB. Also marks FLUX.1-schnell, Krea-2-Turbo and the two Ideogram repos gated: all four are gated on the Hub today, and schnell being Apache-2.0 does not make it ungated. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: stop a case-only duplicate upload destroying an image, and size the load against the right GPU Two fixes from the review round. A dataset upload containing two names that differ only by case silently loses one on Windows and macOS. The in-batch duplicate check is an exact string compare, so "Cat.png" and "cat.png" both pass, but on a case-folding filesystem they are one destination: the commit step moves the first staged part aside, writes the second over it, then deletes the backup, while the response still reports both as uploaded. Captions collapse the same way and never even reach the image branch. No single folder can hold such a pair, but the Windows and macOS open dialogs flatten search and Recents results across folders into one multi-selectable list, which is a normal way a LoRA dataset gets assembled. Rejecting the pair everywhere would regress Linux, where the two really are different files with their own sidecars, so the check probes the filesystem once per process instead of keying off sys.platform: macOS also ships case-sensitive APFS volumes, and a Linux host can keep its Studio home on an exFAT or NTFS mount. A failed probe answers "case-sensitive", which leaves today's behaviour untouched. The quantization fit budget was sized against the wrong card under a reordering CUDA_VISIBLE_DEVICES. A bare "cuda" load lands on visible ordinal 0, but the reducer took the lowest PHYSICAL index, which the nvidia-smi path reports as index_kind "physical". Under "3,1" that sizes against GPU 1 while the pipeline loads onto GPU 3. The hook's own interface doc already promised the lowest visible ordinal. Ranks by visible_ordinal, falling back to index only for an older backend that omits it. * Studio: rework the diffusion dataset panel and align the pane gutters Upload: - one Upload button beside the dataset name, with an upload icon - picking files uploads them, no second confirm click - Add button beside the Training images dropdown, so images can go into a set that already exists - fall back to the upload form when a selected dataset no longer resolves Labeling grid: - two columns at any width; sm:grid-cols-3 crushed the tiles in a fixed width column - pin the caption size so the Textarea's md:text-sm does not outsize a tile - name the tile hover group; a bare one revealed every tile's Remove at once - Remove is an icon button on the image, not a word over the artwork - drop the amber tile fill, keep the No caption label - match the header, toggle and status text to the section's type scale Gutters: - the run area and the preview canvas now sit 40px off the rule, the gutter the settings column has off the page edge * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: give the dataset preview per-image hover and remove, and wrap it onto rows - each thumbnail is its own button, so it can carry an X that removes just that image; the strip was one button and could not nest a second control - hover lightens the image instead of drawing a border - wraps onto more rows rather than scrolling sideways, which also keeps the bottom corners rounded - samples 12 thumbnails, up from 8, now that a second row is available - a delete refreshes the panel's dataset counts * Studio: show the workflow icon beside the Images and Video headings Images reads it off WORKFLOW_TABS, so it is the same icon the sidebar submenu shows and it follows the active workflow. Video's heading is also Create, so it takes the Create icon rather than the nav row's film slate. * Studio: reject a duplicate LoRA id on the diffusion load path too DiffusionGenerateRequest already refuses a repeated adapter id, with a comment naming the hazard: _resolve_lora_set suffixes colliding adapter names, so the same id resolves the SAME adapter twice and set_adapters stacks both copies past the per-adapter weight bound. DiffusionLoadRequest bounds only the list length, and it matters more there. Generation-time stacking spoils one image; on the load path the adapters are baked into the quantized build before compilation, so the unintended combination rides every image until a reload. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: teach the GGUF reuse test double about holds_no_vram The resident-GGUF fast path consults llama_backend.holds_no_vram before asserting CHAT ownership, but two tests hand it a SimpleNamespace built before that attribute existed, so they raised AttributeError and returned a 500. A real LlamaCppBackend exposes it as a property; the doubles now carry it too. Product code is unchanged: these pass on main and were the branch breaking its own test, not a defect in the guard. --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: michaelhan <michaelhan2050@gmail.com> Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: Michael Han <107991372+shimmyshimmer@users.noreply.github.com> | 1 个月前 | |
Use Unsloth for bare Studio product references (#9732) * Studio: use Unsloth for product shorthand * Tests: restore case-folding fixture path * Complete Unsloth product reference rename * Tests: restore internal Studio references | 9 天前 | |
Studio: add image generation, editing workflows and LoRA training with Unsloth GGUFs (#6763) * Tighten comments in the image stack tests and scripts * Close video single-file, training reservation, and image mount-resume gaps Route on-device single-checkpoint video folders through the single_file loader: a bare local .safetensors directory (no model_index.json) is advertised as a pipeline with no filename, so validation rejected it before it could load. Reinterpret the pick as a single_file load of the sole checkpoint, mirroring the image load route. Treat a reserved-but-not-yet-spawned LLM training start as active in is_training_active() so /images/load, /video/load, and /diffusion/start cannot race the reserved run for VRAM during the pre-spawn free window. Mirrors the diffusion training service reservation. Resume an in-flight image generation on the Images page mount: probe generate-progress, re-enter the poll loop, and refresh the gallery on completion so a run started elsewhere is reflected and its saved image appears without a manual refresh. Seed resident image defaults from the resolved base_repo rather than a possibly path-shaped repo_id so the first resident generation uses the right recipe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Publish image generation active state before pre-denoise setup generate() assigned self._gen only at the pipe() call, after deferred compile, LoRA resolution/application, and ControlNet download/build had run. Across that setup window generate_progress() reported inactive even though _generate_lock was held, so a reloaded page's mount probe showed idle and let a second generate queue behind the first. Publish an active step-0 _GenState the moment the generation lock is acquired, before the setup work, and clear it in the outer finally so a setup-time error cannot leave the UI stuck active. Mirrors the video backend's queued phase and the training start guard. * Studio: fix diffusion install ownership, dataset upload atomicity, gallery pagination, and teardown races install_sd_cpp_prebuilt: only write the .unsloth-studio-owned marker when the install created the target directory or it was empty. Adopting a pre-existing, unowned, non-empty directory (a user's own stable-diffusion.cpp checkout) made it eligible for the uninstaller's recursive delete. routes/training upload: make the multi-file promotion transactional. Back up each displaced original and roll every destination back on any failure, so a mid-loop rename error can no longer partially overwrite the live dataset. routes/training _resolve_dataset_folder: reject a symlinked dataset directory and prove the resolved folder stays under the datasets root, so image read/caption/delete cannot escape the root through a link. routes/training delete: escape glob metacharacters in the thumbnail filename so deleting an image named like [ab].png removes only its own thumbnails. image_gallery / video_gallery listing: filter records against the response schema inside the pager via a valid callback, so offset/limit/has_more all count over accepted records. A leading schema-invalid record no longer returns an empty page with has_more=true and stalls infinite scroll at offset 0. image_gallery / video_gallery save: publish via a temp file plus atomic rename (the sidecar is the video pair's commit marker) and clean up on failure, so a partial write never surfaces a truncated PNG or strands an orphan MP4. diffusion_train_common discovery: treat an empty caption sidecar as a metadata tombstone that still falls through to the dreambooth instance prompt, so clearing every metadata caption no longer fails with no captioned images found. diffusion backend unload: wait for an in-flight denoise to exit before tearing down process-wide patches and state, mirroring the load path. diffusion_engine_router: serialize the whole check/unload/publish transition so a concurrent selection cannot return the engine being unloaded. uninstall.ps1: gate the default sd.cpp process stop on the owner marker so a user's own sd-server is not terminated for a directory we then keep. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reject native batch seeds outside the JSON-safe range * Refuse sd.cpp install into unowned non-empty target dir When the install target already exists, is non-empty and lacks the .unsloth-studio-owned marker (a user's own stable-diffusion.cpp checkout, or unrelated files beside a custom Studio root), install() previously still extracted the release into it. Skipping the ownership marker only stopped the uninstaller from deleting the directory; extraction still merged binaries into the user's working tree and could overwrite same-named files. Fail up front with a clear message pointing the user at a fresh/empty location before any download or extraction, leaving their directory untouched. Update the ownership test suite to assert the refusal. * Studio: gate dataset uploads on the symlink check and surface local video single-file checkpoints * Tighten comments and docstrings added by the image-generation fixes * Studio: close arbiter load-registration race and surface native progress + local pipeline folders Publish native sd.cpp generate progress (_gen) before LoRA resolution so a reload probe reads active during setup, matching the diffusers path. Register the diffusion/video GPU load under the arbiter lock (acquire_for now takes a register callback) so a competing acquire cannot evict an owner before its load is marked in-flight and let two loaders allocate VRAM at once. Admit local diffusers pipeline folders (root model_index.json, weights in component subdirs) in the local model scan so they reach task tagging and the On Device picker. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: include marker-owned custom sd.cpp roots in the uninstall stop scan; harden Pester install against the nuget.exe PSGallery bootstrap * Studio: surface local pipeline scan roots, tag single-file checkpoints by filename, mark companion-only pipelines partial - _scan_models_dir: admit a scan folder that is itself a diffusers pipeline (root model_index.json, weights in transformer/ vae/ subdirs). _is_model_directory rejects such a root, so the child scan would list the component subdirs as bogus models and hide the real pipeline; treat the root as one model via _local_pipeline_index. - _local_is_diffusers / _local_model_task: include the sole checkpoint filename in the family-detection needles (_local_family_needles, resolved via resolve_local_single_file). A generically named folder holding one loadable qwen-image-*.safetensors / ltx-*.safetensors identifies its family only from the filename; the load route already resolves that file, so tag it or the task-scoped picker (which rejects task=null) hides the on-device model. - list_cached_models: mark a companion-only base snapshot partial. A GGUF image load prefetches the base repo's VAE / text-encoder / model_index.json but skips the transformer (the GGUF supplies it); the snapshot has a pipeline manifest yet is not a loadable BF16 pipeline, and _cached_repo_partial misses it. _repo_pipeline_missing_denoiser flags a pipeline snapshot whose transformer/ or unet/ component carries no weight, so the picker drops it instead of advertising it as fully on-device. * Studio: preserve foreign gallery files, force safetensors on remote ControlNets, and close dataset/seed/GPU gaps Gallery clear/delete now scope to Studio-owned files: image_gallery and video_gallery skip PNGs / MP4s without a readable recipe (a hand-dropped or orphan file the listing already hides), so clear() and a guessed-id delete no longer destroy files the gallery never surfaced. Remote ControlNets now force use_safetensors: a bare owner/name reaches from_pretrained without the base trust gate, and the Hub scan fails open when unavailable, so requiring safetensors closes the pickle deserialization vector. POSIX uninstall now stops resident sd-server / sd-cli under an owned sd.cpp root before removing the tree (marker-gated), mirroring the Windows stop-before-delete scan; a live native server no longer survives unlinking its binary. Diffusion dataset containment: the training-start read path and the discovery picker route bare names through the protected resolver, so a symlinked dataset is rejected / not advertised like the caption/delete routes already do. Uploads gain the inference decode guard (oversized real images 400 before OOMing the trainer) and dataset upload/caption/delete/import are blocked with 409 while a diffusion run is active. JSONL readers (trainer + routes) tolerate non-object JSON and invalid UTF-8 instead of raising AttributeError / 500. LoRA family compatibility is enforced in the shared resolver, not only the picker, so a direct API client cannot apply a mismatched-family adapter. GPU arbiter gains release_if so the image/video unload idle-check and release are atomic against a concurrent same-owner load's registration. Native batch recipes persist the base batch_seed and restore replays from it, so a native batch_index>0 image no longer advances its seed twice. FLUX.2-klein selects its sd.cpp text encoder by variant (4B -> Qwen3-4B, 9B -> Qwen3-8B) instead of the single family default. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: track the loaded GGUF filename so native companion resolution reproduces the load identity * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: gate local-pipeline image tagging on a real family; validate video sidecars before delete/clear * Studio: tighten image-generation fix comments and docstrings * Studio: gate gallery serve/export on ownership; keep image progress active until persisted; reserve diffusion training before the dataset scan * Studio: restore Reapply target on async image/video load errors; recheck training state before dataset commit; reclaim partial sd.cpp installs on retry Images/Video: a background model load that fails AFTER starting (error/eviction during download) leaves the previous pipeline resident, but handleLoad had already overwritten lastLoad.current with the failed pick, so "Reapply to loaded model" reloaded the failed model. Carry the prior Reapply target into the poller and restore it on the async error/null paths, mirroring the quant rollback. Training: an in-flight diffusion dataset upload passed _require_diffusion_dataset_mutable() at entry but could still commit files after a concurrent /diffusion/start reserved the training slot, mutating the dataset underneath the trainer. Re-check the interlock immediately before the commit phase; a 409 there leaves the staged temps for the finally to clean. sd.cpp install: an interrupted extraction (disk full, killed process, a raising post-extract cudart fetch) left the target non-empty with no owner marker, so the next lazy install tripped the "not a Studio-managed directory" refusal and wedged native install. Write the ownership marker before the partial writes when the target is reclaimable, so a retry recognises the debris as ours and re-extracts. * fp8 DiT quant: floor the dynamic activation scale with activation_value_lb An all-zero activation token row makes the dynamic per-row fp8 scale 0, which turns the quantized data to NaN and the render to black frames on torchao's plain-torch kernel path. The fused fbgemm/mslk quantize kernels clamp zero rows internally, so the bug only reproduces on machines without them, which is most user environments. Zero rows are real inputs, not a corner case: Wan 2.2 zero-pads its text conditioning, and Hunyuan-1.5 and Qwen-Image regenerate zero rows inside their transformer blocks every step. Pass activation_value_lb=1e-12 to Float8DynamicActivationFloat8WeightConfig whenever the installed torchao supports the kwarg (Float8Tensor rework, 0.13+), checked via inspect.signature so older torchao keeps exactly the current behaviour; the existing Float8MMConfig fallback chain is unchanged. Verified on GPU: with the forced plain-torch kernel path a zero-row input NaNs without the floor and stays finite with it, and end to end on HunyuanVideo-1.5 fp8 goes from a solid black frame (LPIPS 1.00) to a normal render (LPIPS 0.225); on Wan the floor matches the condition_embedder exclusion (LPIPS 0.211 vs 0.206). Same-seed renders with fused kernels present are unaffected, and pre-quantized fp8 checkpoints stay valid since weight scales are untouched. * Wire hosted pre-quantized DiT checkpoints into the image families Point prequant_repos for flux.1, flux.2-klein, flux.2-dev, qwen-image (int8 only there; fp8 is family-denied), z-image and krea-2 at the unsloth/<Model>-FP8 Hub repos carrying gate-validated int8 and fp8 transformer checkpoints, so the fast quant path loads the small pre-quantized file instead of materialising the dense bf16 transformer and quantising on device. Measured on FLUX.2-dev int8: build peak drops from 60.7 GB (dense + quantize) to 30.7 GB (hosted prequant), identical 30.7 GB resident after either path since loading a checkpoint is bit-identical to on-the-fly quantisation. The hosted repos name files <Model>-<SCHEME>.pt, so resolve_prequant_source now derives that model-name filename from the repo id (scheme suffix stripped case-insensitively) and carries the legacy transformer_<scheme>.pt as a fallback the resolver tries when the primary 404s, keeping older repos loadable. Wiring a repo also exposed a fallback hazard: with a prequant source present, the dense-fit preflight used to be skipped entirely, so a failed prequant download would fall through to the dense bf16 load the memory plan never budgeted, OOMing after eviction. The preflight now always runs and gates an allow_dense_fallback flag through _load_dense_quant_pipeline: a dense misfit still skips the fast path when no prequant exists, but with one it proceeds and a prequant failure raises to the GGUF build instead of loading dense. The same flag is set when the auto-policy replans an offloaded GGUF against a prequant-sized transient. Tests updated to the new filename convention plus new coverage for the derivation and the legacy-name fallback; the prequant-skips-refit test now asserts the re-check runs and forbids the dense fallback. Verified end to end on GPU: z-image int8 resolves the hosted repo, downloads the model-name file and renders (6.8s load, 5.9 GB peak). * Route krea-2 through its per-component loader on the transformer-quant fast path _assemble_pipe used Pipeline.from_pretrained for every family, but the krea repo ships transformers-5.x configs and no top-level tokenizer files, so the tokenizer dies with vocab_file=None. The pre-quantized checkpoint loaded fine and then the assembly crashed, dropping the load to the GGUF build, which krea-2 cannot take (Krea2Transformer2DModel has no from_single_file). Assemble per-component via load_krea2_pipeline like the pipeline-kind and single-file paths already do. Verified live: Krea-2-Turbo int8 and fp8 hosted prequant loads now assemble and render through the Studio images tab. * Keep Qwen-Image's text-stream linears bf16 on int8 (short prompts break torch._int_mm) Qwen-Image's MMDiT runs every text-stream Linear at M = actual prompt tokens: the Qwen2.5-VL embeds are not padded to a fixed length like FLUX's 512-token T5. A short prompt (13 tokens) or the near-empty negative prompt drives torch._int_mm below its M > 16 floor and the first denoise step raises 'self.size(0) needs to be greater than 16, but got 13' (measured on B200 through the Studio images tab). Add per-family int8 exclusions (txt_in, add_q/k/v_proj, to_add_out, txt_mlp) for qwen-image and qwen-image-edit, threaded through exclude_tokens_for_scheme(scheme, family) and the prequant checkpoint validation, so a checkpoint baked under the old token list is rejected and re-quantised instead of loaded crashing. The text stream runs at M = tens vs the image stream's M ~ 4k, so the exclusion costs nothing; the rebuilt hosted checkpoint gates 28/28 PASS with LPIPS mean 0.057 (was 0.069). * Harden the diffusion memory plan against transient free-VRAM undercounts A cold FLUX.2-dev int8 load on an idle 183 GB B200 planned offload=model (companions exceed budget) and silently served the GGUF as-is; the identical retry went resident and engaged the hosted prequant. The plan arithmetic was byte-identical across both loads (required 90,228 MiB, resident needs free of about 124 GB); the only divergent input was torch.cuda.mem_get_info, which is device-wide and instantaneous: a transient foreign CUDA context briefly held about 100 GB at the first snapshot, and the planner trusted that single read. Three changes: - settled_snapshot_device_memory: on cuda, synchronize + empty_cache (best-effort) and take the MAX free over up to 3 spaced reads. A transient can only shrink free, so the max rejects transient undercounts while a persistent tenant still caps every read. _plan_memory now uses it. - plan_fits_total_capacity + one replan retry: when the dense/prequant candidate fits TOTAL device capacity under the standard reserve and the 0.85 resident margin, an offload verdict can only stem from the free reading, so the loader re-snapshots and replans once before declining the fast path. Explicit balanced/low_vram modes skip the retry (they offload by mode). - diffusion.transformer_quant_declined log line with required/budget/free and the plan reasons, so the next decline is diagnosable from the server log (previously silent). Verified: cold FLUX.2-dev int8 first load in a fresh server now engages the hosted prequant resident (offload=none). * Add FLUX.2 Klein and FLUX.2-dev DiT LoRA training Register flux.2-klein and flux.2-dev in the DiT trainer following the upstream DreamBooth references: latents train patchified and batch-norm normalized from the VAE posterior mode, the packed forward reuses step-invariant position ids, and the guidance vector (3.5) is gated on the variant's guidance_embeds config. Conditioning stacks load per variant (Mistral via Flux2Pipeline for dev, Qwen3 via Flux2KleinPipeline for Klein) and are encoded and freed before the transformer lands on the device. The fused single-stream to_qkv_mlp_proj joins the attention projections in the LoRA targets; the single-stream out projection stays dense because its to_out suffix would also match the double-stream ModuleList container. Wire both families through the training registry (family set, labels, VRAM notes, rank 16 / lr 1e-4 defaults, bf16-only preflight), mark them trainable with train base repos in the family registry, add FLUX.2-dev to the gated-repo token check, and trust both official bases for training downloads. Verified on B200: 30-step klein int8 (19.6s) and nf4 (20.9s) and dev int8 (52.0s) runs train with finite decreasing loss and the saved adapters apply on the bf16 base pipeline (weight 0 reproduces the base image exactly, weight 1 visibly restyles it). * Support LoRA adapters on torchao int8/fp8 quantized image pipelines Adapters are baked at load time: they attach to the dense transformer, then quantize_ converts only the frozen base linears (the lora_ side path is excluded by name), then the loader compiles. Post-quant PEFT injection is not possible on a manually quantized module, so the prequant shortcut is skipped for a baked load and the memory plan is sized for the dense build (force_dense on the quant candidate). At generation time the baked topology is frozen: weight tweaks and disabling (scale 0 reproduces the quantized base exactly) go through set_adapters, while adding or removing adapters returns a clean 400 telling the client to reload with the new selection. supports_lora now returns True for int8/fp8 diffusers loads (checked before the gguf-kind early return, since the quant fast path keeps the picker kind); nvfp4/mxfp8 and GGUF-via-diffusers stay blocked. The load request model takes an optional loras list, threaded through begin_load on both engines (native ignores it and keeps applying LoRA at generation). Verified end to end on GPU: Z-Image GGUF picker + int8 + trained adapter loads through the API, bake marker logged, weight 1.0 vs 0 renders differ visibly, weight 0.5 accepted live, unknown adapter rejected as 400. Affected suites: 296 passed. * Add FLUX.1 Krea dev to the image model catalog Krea's guidance-distilled FLUX.1-dev finetune keeps the exact dev layout, so it runs under the existing flux.1 family unchanged. Wire it up end to end: - Catalog group with the gated official bf16 pipeline and the open QuantStack GGUF quants; the gated artifact is skipped on auto-routing when undownloaded. - Trust the official repo for non-GGUF from_pretrained loads, next to the other black-forest-labs bases. - Generation defaults: 28 steps at guidance 4.5 per the model card. The generic "krea" defaults key (Krea-2-Turbo's 8-step no-CFG recipe) used to swallow the id, which would have produced garbage output; the new flux.1-krea key precedes it on both the backend table and the images page table. - The flux.1 prequant checkpoints are schnell-based; the loader's baked base_model_id validation refuses them for the Krea-dev base, so int8/fp8 requests dense-quantize instead (covered by existing prequant tests). * Resolve pre-quantized checkpoints per base variant One family entry covers several published variants whose weights differ (flux.1: schnell, dev, Krea-dev), but prequant resolution was keyed on (family, scheme) alone, so only the default base could ever be served: the loader's baked base_model_id validation correctly refused the schnell checkpoint for dev and Krea-dev bases and every such load paid the dense download plus on-the-fly quantise. Add an optional prequant_variant_repos table on DiffusionFamily as (base_repo, scheme, repo_id) triples and thread the resolved base repo through resolve_prequant_source / usable_prequant_source and their three call sites (load fast path, memory-plan probe, auto-policy candidate). A base without its own entry keeps returning the family default, preserving the existing refuse-then-dense behavior exactly. Wire the flux.1 variants: the gate-validated unsloth/FLUX.1-dev-FP8 checkpoints (built in the earlier campaign but never reachable) and the new unsloth/FLUX.1-Krea-dev-FP8. * Add the Lumina Image 2.0 family to the image catalog Alpha-VLLM/Lumina-Image-2.0 is a 2.6B single-stream DiT with a Gemma2-2B encoder and a standard 16-channel VAE, all transformers-4.x-compatible, so the generic from_pretrained pipeline path loads it as a new lumina-2 family: - Family entry (Lumina2Pipeline / Lumina2Transformer2DModel), aliased to lumina-image-2.0 / lumina-image-2 / lumina2. No bare lumina alias: Lumina-Next checkpoints are a different arch and must stay unknown rather than crash mid-load. bf16-only upstream, so the fp16 fallback stays off like z-image. - Trust the official repo for non-GGUF loads; bf16 component table entry (ships fp32, ~5.2 GB transformer + 5.2 GB encoder bf16-resident). - Generation defaults 50 steps / guidance 4.0 per the model card, and the generate call passes the card's cfg_trunc_ratio=0.25 itself (family-gated, signature-gated): the pipeline default (1.0) runs the CFG double-forward on every step and oversaturates output. - Catalog group with the single ungated bf16 pipeline artifact (11 GB resident) plus routing assertions; images page defaults row. - No GGUF artifact: none exists upstream (only finetune/LLM quants), so the dense transformer_quant fast path (GGUF-kind-only) stays unreachable for now. Offline probes of the future prequant campaign: int8 and fp8 both engage and render cleanly (fp8 LPIPS 0.11 vs bf16, int8 0.33 from 50-step trajectory drift with intact quality), so neither scheme is family-denied. * Wire the hosted Lumina Image 2.0 int8/fp8 checkpoints Gate-validated against same-seed bf16 renders (28/28 pairs per scheme, zero failures): int8 LPIPS mean 0.146 / SSIM 0.937, fp8 LPIPS mean 0.116 / SSIM 0.946. Uploaded to unsloth/Lumina-Image-2.0-FP8 following the existing checkpoint repo conventions. * Add the HunyuanImage 2.1 family to the image backend The hunyuanvideo-community diffusers mirror carries the full stack in standard layout: a 17B dual-stream DiT (32.5 GB bf16), a Qwen2.5-VL text encoder, a ByT5 glyph encoder, the 32x HunyuanImage VAE, and guider/ocr_guider components (AdaptiveProjectedMixGuidance) that diffusers 0.39 loads natively, so the generic from_pretrained pipeline path covers everything with no per-component assembly. Family notes: - The call's guidance knob is distilled_guidance_scale (there is no guidance_scale kwarg), so cfg_kwarg routes the UI value there; real CFG runs inside the repo's guider at its baked scale. Defaults follow the card recipe: 50 steps, 3.25. - 2K-native: verified live at both 1024 and 2048. - Coexists with the HunyuanImage-3.0 structured exclusion (3.0 has no diffusers pipeline and stays excluded with its stated reason). - int8/fp8 dense quantization verified live (LPIPS 0.186 both vs same-seed bf16); a short prompt does not trip the int8 torch._int_mm minimum on this arch, so no family exclude entry is needed. - bf16 component table for the memory planner: (32.5, 16.3, 0.8) GB. * Surface HunyuanImage 2.1 in the image model catalog Catalog group with the open bf16 mirror pipeline (~50 GB resident, so a bare click on a consumer card routes to the QuantStack GGUF quants, which load and render through the generic GGUF path, verified live) plus the images page defaults (50 steps, guidance 3.25 feeding distilled_guidance_scale). * Add the HiDream-I1 family to the image backend A 17B MoE DiT (16 double + 32 single layers, 4 routed experts) with four text encoders, on HiDreamImagePipeline (diffusers 0.39). One family covers the open Full / Dev / Fast repos (same arch); per-variant generation defaults follow the upstream inference recipes (Full 50 steps at guidance 5, the distilled Dev 28 and Fast 16 guidance-free). The repos name a Llama-3.1-8B text_encoder_4 in their model_index but do not ship its weights; the official example passes the gated meta-llama repo in by hand. The loader instead assembles the component from the open unsloth mirror (byte-identical weights, already inside the non-GGUF trust gate), injected at the three pipeline from_pretrained sites, with output_hidden_states matching the official example. Memory planning counts the assembled TE4: 34.2 GB DiT + 28.8 GB encoders, ~63 GB bf16-resident. * Surface HiDream I1 in the image model catalog One catalog group with the three official bf16 pipelines (Full, plus the Dev and Fast distillations as labeled artifacts) at their ~63 GB resident size, so auto-routing keeps this a datacenter-GPU pick. city96's GGUF is deliberately not wired: the GGUF path would need the same Llama TE4 assembly for very small demand. Images-page defaults mirror the backend table with the variant keys ahead of the generic hidream key. * Pin the measured HiDream quant verdict in tests int8 and fp8 both engage and render cleanly on this family, including short prompts on int8: the routed MoE expert Linears only ever see the concatenated image+text stream (M far above the torch._int_mm minimum), so no deny entry and no family exclude tokens are warranted. Assert that so a future table edit cannot silently regress the measured behavior. * Wire the hosted HunyuanImage 2.1 int8/fp8 checkpoints Verified bit-identical to on-the-fly quantize: all 1264 state dict tensors (456 quantized) dequantize equal between the loaded checkpoint and a fresh quantize_ pass, so quality matches the runtime Dtype path exactly. Same-seed LPIPS suite means (0.35 int8 / 0.28 fp8) blend trajectory divergence with this family's own run-to-run nondeterminism (identical weights and seed reproduce a 17/255 mean pixel delta through the 50-step guider pipeline); per-case hard checks pass and the drift is compositional, reviewed visually. Uploaded to unsloth/HunyuanImage-2.1-FP8. * Fix silent LoRA drop and wasted transformer prefetch on GGUF quant loads Two live-test findings on the images load path: - transformer_quant with baked LoRAs, when the dense quantized build is declined for memory or fails: the load completed as a plain GGUF with the adapters silently dropped (HTTP success, supports_lora=false after the fact) -- wrong output with no signal. The load now fails with the recovery options (drop the adapters, free VRAM, or pick a smaller model). Weight-0 adapters still count as no bake request, and the plain no-LoRA decline keeps its silent GGUF fallback. - A fresh GGUF load on a small GPU prefetched the base repo's full bf16 transformer shards (~47 GB on Qwen-Image) because the dense-quant prefetch widening only checked scheme viability, not whether the device could ever hold the candidate resident. Gate the widening on total device capacity (reserve + 0.85 margin, the plan_fits_total_capacity bar) so a card that is certain to decline the dense build never pays the download; capable devices keep the prefetch. * Fix video progress under-reporting during load and generate Two live-test findings on the video progress endpoints: - load-progress downloaded_bytes froze mid-download: the counter used scan_cache_dir, which skips in-flight *.incomplete blobs, so it sat at the last completed blob for the whole multi-GB shard pull while the disk kept filling. Count the repo's cache directory directly (completed plus incomplete blobs, snapshot symlinks skipped so nothing is double-counted). - generate-progress reported total_steps=null / fraction=0 while step advanced: the video API only carried the native total field while the image API exposes total_steps and fraction, so one poller could not work against both. Derive the image-compatible aliases in generate_progress and declare them on the response model; the native total stays for back-compat. * Wire the hosted HiDream I1 int8/fp8 checkpoints Gate-validated: all 28 per-case pairs pass per scheme (LPIPS suite means 0.291 int8 / 0.278 fp8, in the 50-step trajectory-divergence band; CLIP delta means 0.007-0.008), and the int8 checkpoint is verified bit-identical to on-the-fly quantize across all 1615 state dict tensors (1073 quantized, max abs diff 0.0). Uploaded to unsloth/HiDream-I1-Full-FP8. * Add a pre-cast text-encoder loader for the layerwise fp8 scheme The runtime text_encoder_quant=fp8 path downloads the full bf16 text encoder and layerwise-casts it in place on every fresh load. For the heavyweight encoders (LTX's Gemma3-27B ~50 GB, FLUX.2-dev's Mistral-24B ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) that download dominates load time on a fresh machine. diffusion_te_prequant.py loads a pre-cast fp8-storage state dict instead: meta-init the encoder skeleton from the checkpoint's te_class, load_state_dict(assign=True), rebuild on CPU if non-persistent buffers stay on meta, then re-apply the same layerwise cast to install the upcast hooks. The cast is a deterministic storage transform, so the loaded encoder is bit-identical to dense-load-then-cast by construction. v1 hosts the layerwise fp8 storage scheme only: its state dict is plain tensors (torch.load(weights_only=True), no pickle execution). The dynamic-compute schemes (fp8_dynamic, int8, nvfp4) build torchao subclass wrappers at runtime and are deliberately not hosted. Checkpoints validate format, scheme, component and base_model_id before use and any problem falls back to the dense download and cast. Local path overrides reuse the DiT prequant allowlist env var. Families opt in via a new te_prequant_repos (scheme, component, repo_id) field on both DiffusionFamily and VideoFamily; the field defaults empty so nothing changes until a gate-validated artifact is wired. * Inject hosted pre-cast text encoders during pipeline assembly Wire te_prequant_pipe_kwargs into the three pipeline assembly sites: the diffusion full-pipeline branch, the diffusion transformer-only and GGUF branch (where the companion TE is the big remaining download), and the shared video assembly path before the pipeline/component split. Injection is gated exactly like the runtime cast (mode normalized to fp8, device supported, family not denied), so it can never engage where quantize_text_encoders would not; the later quantize_text_encoders call re-applies the cast idempotently and keeps status reporting truthful. With no hosted checkpoint configured the call returns {} and assembly loads the dense encoder as before. * Add the pre-cast text-encoder checkpoint builder Applies the runtime layerwise fp8 storage cast to a model's dense text encoder once and saves the cast state dict with baked metadata (format tag, base_model_id, family, scheme, component, te_class, versions) in the layout diffusion_te_prequant.py validates. Resolves the encoder class from the checkpoint's config.architectures so the recorded te_class matches what the pipeline instantiates. CPU-runnable: the cast touches storage dtypes only. * Test the pre-cast text-encoder load path Hermetic CPU coverage for diffusion_te_prequant: the checkpoint filename convention, family-table resolution by scheme and component with malformed entries skipped, resolution priority (path override, hosted repo, none) and the fp8-only scheme gate, the checkpoint validation matrix (wrong format, missing state_dict, wrong scheme, wrong component, wrong or missing base_model_id) with base case folding, the local-path allowlist refusal and missing-file fallback, and the assembly injection gating (mode, hosted entry, device support, family deny, load failure, successful injection). Also pins the te_prequant_repos field on both family dataclasses and that no family ships a hosted TE checkpoint until the campaign wires one. * Fix pre-cast TE checkpoint loading and engagement reporting Two bugs found while building the hosted checkpoints: - The builder recorded torch.__version__ (a TorchVersion object) in the checkpoint metadata, so torch.load(weights_only=True) rejected every artifact and the loader silently fell back to the dense download. Record plain strings. - Re-applying the layerwise fp8 cast to an injected pre-cast encoder raised on the duplicate hook registration, making quantize_text_encoders report the engaged cast as failed (status showed no TE quant while the encoder ran fp8). _cast_fp8 now returns early when the hooks are already installed. Also corrects the LTX TE size note: Gemma3-12B stored fp32 (~49 GB), not 27B. * Wire the hosted pre-cast fp8 text encoders qwen-image and flux.2-dev (diffusion) and ltx-2 (video) now resolve a hosted pre-cast fp8 text encoder from their unsloth -FP8 repos: - unsloth/Qwen-Image-FP8: Qwen2.5-VL-7B, 16.6 GB dense -> 8.8 GB - unsloth/FLUX.2-dev-FP8: Mistral-Small-24B, 48.0 GB dense -> 24.7 GB - unsloth/LTX-2-FP8: Gemma3-12B, 48.7 GB fp32 store -> 13.2 GB Every checkpoint verified bit-identical to dense-load-then-cast (729 / 585 / 1066 tensors, zero mismatches) and smoke-tested through the real backends with the repo engagement marker. Tests cover the wired entries, the resolver filenames, builder metadata weights_only survival, and the idempotent re-cast. * Report the compute dtype on fp8-cast encoders and inject the pre-cast TE on the dense fast path Two more findings from the hosted-TE GPU smokes: - Module.dtype reports the first floating parameter, which after the layerwise fp8 cast is the fp8 STORAGE dtype. Flux2 derives its prompt embed and latent dtypes from encoder.dtype and feeds them to randn_tensor, which has no fp8 kernel, so ANY flux.2 load with text_encoder_quant=fp8 crashed at generation (pre-existing, runtime cast included). The cast now swaps in a subclass whose dtype property reports the compute dtype; forward behaviour is unchanged. - The dense transformer_quant fast path assembles companions through _assemble_pipe, which never received the pre-cast TE injection, so the hosted encoder engaged on full-pipeline and GGUF builds but not on the fast path. Threaded through like the other two branches. Verified live on B200: qwen-image (full pipeline), flux.2-dev (GGUF picker with int8 DiT prequant), ltx-2 (video backend) all engage the hosted TE, render non-black, and report text_encoder_quant=fp8 truthfully. * Key the fp8 cast idempotency on an explicit completion marker Hook presence alone cannot distinguish a legitimately pre-cast text encoder from leftover hooks after a cast that failed mid-pass, so the early return now requires the completion marker _cast_fp8 sets once the hooks are fully installed. Leftover partial state keeps failing closed. Also tolerates non-Module encoder doubles in the hook probe and the dtype override. * Extend the fp8 TE quant to HiDream's Llama text_encoder_4 The generic quantize_text_encoders pass only covers text_encoder.._3, so HiDream's HEAVIEST encoder (Llama-3.1-8B TE4, 16.1 GB bf16) always stayed dense. TE4 is assembled separately (hidream_te4_kwargs), so the fp8 path now lives there: when the requested TE quant is layerwise fp8 and the device/family qualify, TE4 prefers the hosted pre-cast checkpoint (unsloth/HiDream-I1-Full-FP8, 8.6 GB) and falls back to dense-load-then- cast; a mid-pass cast failure reloads a fresh dense encoder instead of shipping partial state. The pre-cast loader and builder gain config_subfolder/config_overrides for standalone encoder repos whose config sits at the root and whose pipeline needs forward flags (output_hidden_states/attentions). Verified on B200: bit-identity 291 tensors (225 fp8, 0 mismatches), hosted checkpoint engages through the real backend (marker + status fp8), load 24.3 s vs 48.0 s dense, LPIPS 0.133 mean over 3 same-seed pairs vs the dense-TE render (gate 0.25), non-black frames. * Correct the ltx-2 resident TE estimate to the bf16 cast size The memory plan's bf16_components_gb held 50.4 GB for the LTX text encoder, which is the fp32 hub store of Gemma3-12B (~49 GB download), not what sits on device: the pipeline loads it torch_dtype=bf16, ~24.4 GB resident. The 26 GB over-estimate pushed the auto plan toward offload on cards that fit the real footprint. Comments and the size-table test now pin the resident semantics. * Host pre-cast fp8 text encoders for four more families Round 2 of the hosted TE set, each bit-identical to dense-load-then-cast and gated through the real backend (marker + status fp8 + same-seed LPIPS vs dense TEs): - FLUX.1 T5-XXL (text_encoder_2): 9.52 -> 5.90 GB, one artifact for schnell/dev/Krea-dev (T5 shards byte-identical across all three, verified sha256). 220 tensors, 144 fp8, LPIPS 0.109. - Lumina Gemma2-2B: fp32 hub store 10.46 -> 3.20 GB (3.3x download cut). 288 tensors, 182 fp8, LPIPS 0.041. - Z-Image Qwen3-4B: 8.04 -> 4.41 GB. 399 tensors, 252 fp8, LPIPS 0.112. NOT shared with flux.2-klein-4B: klein retrained layer 35's MLP (verified tensor diff, maxdiff 0.86), so klein hosts no entry. - Krea-2 Qwen3-VL-4B: 8.88 -> 4.83 GB. 713 tensors, 460 fp8, LPIPS 0.082. The constructor-assembled krea pipeline takes the encoder directly (load_krea2_pipeline text_encoder kwarg); the loader remaps 5.x rope_parameters and re-ties weights after assign so the rebuilt encoder matches the builder's structure. HunyuanImage 2.1 reuses the Qwen-Image artifact outright: its Qwen2.5-VL text encoder is byte-identical (every shard sha256, 16,584,414,544 bytes), recorded in the new component-level base-equivalence table the checkpoint validator consults. The injection loop now covers text_encoder.._3 so a family can host several components. Live check: LPIPS 0.123 vs dense. * Report the fp8-cast compute dtype without swapping the encoder class The dtype override swapped encoder.__class__ to a dynamic subclass, which breaks transformers' kwargs-based output recording: a fp8-cast Qwen3VLModel stopped returning hidden_states and every krea-2 generation with text_encoder_quant=fp8 crashed at encode_prompt (regression from the HiDream TE4 change; caught by the krea hosted-TE live smoke). The override is now a property shadowed on the ORIGINAL class that prefers a per-instance compute-dtype attribute, so class identity is preserved and uncast instances keep the stock behaviour. The idempotency test now pins exact class identity and the uncast-sibling fallback. * Pass the calibrated distilled sigma curve to LTX-2.3 8-step runs The 22B distilled DiT was trained against ltx_core's fixed DISTILLED_SIGMA_VALUES, but the diffusers scheduler derives 8-step spacing from resolution-shifted flow matching and lands far off at every reachable mu (second sigma 0.945-0.981 vs 0.99375, tail 0.37-0.61 -> 0.1 vs 0.725 -> 0.42 -> 0). At the distilled default step count the backend now passes the list verbatim, neutralising the scheduler's dynamic shift and terminal stretch for the call (they distort even explicit sigmas) and restoring them afterwards. Other step counts and the dev/base DiT keep the scheduler's own spacing. Live-verified on B200 through the video branch backend: the scheduler holds the exact curve after an 8-step distilled GGUF generation, config restored, healthy clip. Also reword the transformer_quant resolved reason to the measured reality: quant halves resident weights and hosted checkpoints cut load time, while per-step speed is roughly bf16 parity. * Pin the fp8 weight-quantize kernel against silent MSLK switching torchao's Float8Tensor KernelPreference defaults to AUTO, which switches the weight-quantize kernel to MSLK whenever an mslk package is importable on sm90+. Measured on B200: that changes fp8 scale rounding bitwise (8/8 FLUX matrices differ, scales ~55 percent of bytes), so a box that merely gains mslk would break the hosted-prequant bit-identity invariant; the mslk path is also slower under torch.compile (opaque extern call blocks inductor's quantize fusion, FLUX.1 fp8 e2e 1.149 to 1.624 s). Pin KernelPreference.TORCH explicitly, matching current no-mslk behaviour bit for bit; signature-gated for older torchao. GPU-smoked (finite, rel err 0.037) and pinned by test. * Shift Qwen-Image training sigmas to the inference distribution Qwen-Image's scheduler skips its static shift under use_dynamic_shifting, so the DiT trainer was drawing UNSHIFTED uniform-schedule sigmas for it (mean sigma 0.50) while inference always runs the exponential mu = log 3 shift plus the shift_terminal 0.02 stretch. Add a flow_shift config lever: "auto" (the new qwen-image default) rebuilds the training sigma table through the scheduler's own time_shift and stretch_shift_to_terminal so the draw matches the inference distribution exactly (mean sigma 0.72); a numeric value applies the standard linear shift s*u/(1+(s-1)*u); 1.0 keeps the historical identity behavior and stays the default for FLUX, Z-Image and Krea 2. The model timestep conditioning follows the shifted sigma, gathered in fp32 so bf16 rounding never skews it. Also wire two opt-in levers with off defaults: cfg_dropout (per-sample empty-prompt conditioning dropout, encoded alongside the captions before the text encoders are freed) and weighting_scheme="bell" (bsmntw-style mid-schedule Gaussian loss weighting normalized to mean 1). Verified with two 80-step rank-8 bf16 LoRA runs on Qwen/Qwen-Image (identity vs auto, same seed): both converge with finite decreasing loss and produce coherent same-seed previews. Unit tests cover the exact transform, the shifted sampling distribution, per-family defaults and config plumbing. * Add LoRA EMA, a persistent conditioning cache, and aspect bucketing helpers diffusion_train_extras hosts the opt-in training extras: LoRAEMA shadows only the trainable adapter params (warmup-ramped decay, default 0.99, exported as a second adapter under output_dir/ema), PersistentConditioningCache stores latent posterior stats and caption embeddings as safetensors keyed by content hash + family + resolution, and the aspect-ratio bucketing helpers group mixed-aspect datasets into same-area divisor-snapped shapes. The DiT trainer wires the first two behind config flags that default to the current behavior: ema_decay (0 disables) and cond_cache_dir (None disables). A fully warm cache skips loading the VAE and text encoders entirely; a cache hit is bit-identical to a fresh encode, including the per-channel qwen latent normalization. Also fixes the stale _gather_sigmas call in the perf test that still passed the scheduler instead of the sigma table. * Tighten torchao configs and note the FSDP2 design for the DiT trainer nf4 loads now enable double quantization (~0.4 bits/param off the frozen base scales at no fidelity cost), fp8 training uses the rowwise recipe when the torchao build ships it (per-row scaling confines the DiT activation outliers that a tensor-wide scale collapses), and the inference quant filter gains a per-scheme GEMM-tiling divisibility floor (16 for scaled_mm, 32 for MX blocks) so one ragged Linear cannot crash the first denoise after a clean quantize pass. plans/fsdp2_diffusion_design.md records the multi-GPU design: bf16/fp8 over FSDP2 with per-block units, LoRA attached before sharding, int8 out of scope (DTensor over the quantized subclass is undefined), per-family notes. * Batch diffusion inference with per-image seeds, an inference conditioning cache, and GGUF loader fixes Batched generation: /images/generate takes a prompts list (one image per prompt, txt2img only) or a seeds list (one prompt, one image per seed); the legacy batch_size path derives per-image seeds base..base+n-1 like the native engine. Every image gets its own torch.Generator so any batch member replays alone from its gallery recipe; the whole list runs as one forward by default with OOM backoff that halves a failed chunk, and an explicit batch_size caps images per forward. Validated 10-22x over serial engines on 32-image suites with LPIPS deltas within 0.002. Conditioning cache on the inference path: UNSLOTH_DIFFUSION_COND_CACHE_DIR (the inference sibling of the trainers' cond_cache_dir, same persistent store) wraps encode_prompt so repeated prompts skip the text-encoder forward entirely; verified bit-identical outputs. Bypassed while LoRA adapters are attached; tensor-argument calls pass through uncached. Compile cache: GGUF loads fingerprint their own bundles (quant=gguf, a different compiled graph than the dense family) and batched calls register every distinct (w, h, batch) chunk shape they ran, so the heavy GGUF batched warmups (~159 s at batch 32 on 12B-class, ~655 s on 20B CFG-batched) are paid once ever. GGUF loader: strip the sd.cpp model.diffusion_model. container prefix in the single-file converter; diffusers' FLUX.2 converter KeyErrors on it and the Qwen-Image identity mapping strands the model on meta. * Correct batched seed-replay docs to match measured behavior Same-seed images at the same batch shape are bit-identical; a solo regeneration with the recorded seed matches its batched rendition up to batch-size-dependent kernel numerics (mean abs pixel delta about 2.5/255, LPIPS delta under 0.002), not bit-exactly. The previous wording overclaimed bit-identity across batch shapes. * Note that batched bit-identity assumes a settled compiled graph The first generation issued while the deferred compile is still in flight can deviate transiently (observed once on a cold fp8 build: mean abs pixel delta 0.063/255); once the graph is settled, same-seed same-batch-shape images are bit-identical across runs. * Studio sidebar: Image03/FlimSlate icons, More flyout, Train row, New pills - Images uses Image03Icon and Video uses FlimSlateIcon. - New "More" row (MoreHorizontalIcon) opens a right-side flyout on click or hover holding Video, Recipes and Export; the close is delayed 180ms so the pointer can cross the gap. Its SidebarMenuButton deliberately takes `title` rather than `tooltip`: with `tooltip` the button returns a Tooltip root and DropdownMenuTrigger asChild would hand its ref to a non-DOM node. - Dropped the "Train" section heading; Train is now a top-level row between Images and More. data-tour="navbar" moves to the surviving nav group so the product tour keeps its anchor. - "New" pill beside Images and (inside the flyout) Video, via NavBadge. * Studio sidebar: match flyout rows and New pills to the existing scales - More flyout rows dropped their sidebar-row typography and size-icon override, which fought DropdownMenuItem's own scale (text-sm, gap-2.5, px-3 py-2 and size-4 icons) and rendered oversized glyphs and text next to the nav. - New pill reuses the brand "beta" badge recipe (nav-badge font, --ui-font-scale sizing, nav token colours) rather than hardcoded 9px values. - The More row's native title tooltip (an OS box on hover) is replaced by the app's Tooltip, wrapped around DropdownMenuTrigger so both triggers compose onto the same button, and shown only on the collapsed rail like other nav rows. * Settings: pin and reorder the sidebar navigation Adds a "Sidebar navigation" section to Settings -> Appearance, above the existing profile-menu customizer, with the same drag-to-reorder + switch UI. - New sidebarNav preference: one { id, pinned } entry per navigable row (projects, hub, images, train, video, recipes, export), array order = render order. Defaults match the shipped layout, so an untouched install is unchanged. - Unpinning moves a row into the More flyout rather than hiding it, so no page becomes unreachable. New chat and Search stay fixed as actions. - app-sidebar now renders from one navRows descriptor map, so a pinned row and its flyout counterpart cannot drift; the More row appears only when something is unpinned and highlights off whatever it actually holds. - Mirrored in the backend PersonalizationCustomization: without it the model's extra="ignore" would drop the field, and because sync replaces local state with the server's copy once customization is saved, the user's pin order would reset on the next sync. The validator dedupes and back-fills like sidebarMenu but preserves the client's order, since here order is meaningful. Frontend typecheck, i18n parity and catalog checks pass; 32 personalization tests pass, including a round-trip asserting a reordered list survives a save. * Sidebar customizer: drop the Search row, skip More for a lone item - Search is reached from the top bar, so it is no longer previewed as a fixed sidebar nav row; New chat stays. - More now appears only when it would hold two or more rows. A single unpinned row renders inline in its saved order position instead: a flyout wrapping one item costs a click and earns nothing. The customizer's More preview follows the same threshold. * Sidebar settings: hide a lone unpinned tab, match New chat icon, rename Profile menu - With exactly one tab unpinned, both More and that tab are dropped, so nothing is drawn for it (previously it rendered inline). The page stays reachable by URL. - The customizer's New chat preview uses PencilEdit02Icon, the icon the real row renders; Edit03Icon was a different glyph. - "Sidebar menu" is now "Profile menu", described as the shortcuts behind your name at the bottom of the sidebar, so it no longer reads as a second name for the navigation section above it. * Tighten comments in the new sidebar and delete-guard code * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio sidebar: keep the More row highlighted while its panel is open Moving the pointer into the flyout left the row unhighlighted while the panel stayed open. The row now carries data-menu-open, added to the nav hover selector list. Not data-state: the tooltip and menu triggers both write that attribute, so whichever lands last wins. * Images: use the shared pill toggle for Create/Train and pad the panels - Create/Train was the only segmented control on its own Tabs styling. It now uses PillTabs, the same control as the model picker and Hub toggles, pinned to the header row's 34px. PillTabs takes an icon per tab, so the inline-span workaround for TabsTrigger goes away. - pt-3 on both the Create and Train panels, which sat flush against the model selector row. * Images: make the workflow picker a dropdown instead of a 7-up strip Seven workflows in a 340px rail left ~48px each, so the labels crowded and the hints were only reachable as title tooltips. The strip is now a dropdown: the trigger shows the current workflow and its hint, and each row carries its own description. A row the loaded model can't run is disabled and shows the reason in place of the hint, so the gating explains itself. Adding a workflow no longer shrinks the others. * Images: workflow icons, hint under the trigger, more top room, unclipped Train cards - Each workflow carries an icon, shown on the closed trigger and on every row. - The trigger is one line (icon plus name). The selected workflow's description moved below it, where it reads like the Field hints further down the rail. - pt-6 instead of pt-3 on both Create and Train, so the cards clear the model selector row. - The Train right column scrolls while its cards use ring-1, which draws outside the box and was clipped at the scroll edges. p-px gives the ring room. * Images: one-line workflow rows, stronger trigger fill, roomier mode tabs - Dropdown rows are icon plus name only. The selected row's description already shows under the trigger, and a disabled row keeps its reason as a title. - Trigger fill moves to the bg-foreground/[0.07] dark:bg-foreground/[0.12] pair the hub cards use, so it reads against the card in both modes. - Description under the trigger goes from text-ui-10 to text-ui-11p5. - More horizontal padding on the Create / Train tabs. * Images: drop card borders for the composer shadow, keep scrollbars inside, use app controls in Train - Cards lose ring-1 for .panel-soft-surface: the composer's shadow in light, flat in dark, matching .chat-composer-surface and the menus. - Both rails now clip (overflow-hidden) with the scroller inside, so the scrollbar can't ride over the rounded corner. Same shape video-page already uses. - Train's 9 native selects become the app Select, so they no longer open an OS-native menu, and the native file input is hidden behind a Choose images button that reports the count. - Image previews use explicit 8-10px radii: this theme sets --radius to 1.1rem, so rounded-md was 15.6px and the thumbnails read as circles. * Images: one card for controls and preview, chat sliders, wider softer shadow - Controls and preview were two floating cards; they now share one card split by a divider. The Advanced dock stays separate since it toggles. - SliderField wraps Chat's ParamSlider, so the sliders match Chat (label row with the value, full-width neutral track) instead of a green track with a spin box. All 14 call sites keep their props. - panel-soft-surface goes from 0 2px 8px -2px /0.16 to 0 4px 22px -6px /0.10: lighter, spread wider. * Images: flat Create and Train panes, hover-only scrollbars, tidier Train dataset step Both Images tabs now sit on the page background like the Hub: no card, no shadow, no bounding box. A single rule divides the controls rail from the preview canvas (Create) and from the run area (Train), and the settings and previous-runs sections read as panes rather than nested cards. Also: - Scrollbars in these panes use the existing hover-scrollbar recipe, so the thumb only shows while the pane is hovered. - Workflow rows explain themselves with a tooltip after a short hover, which also works on disabled rows, and the descriptions are much shorter. - Training images rows are name plus image count; the license stays on the example card. - The upload step loses its dashed box, the buttons match the sizes around them, and Upload only appears once files are picked. - The empty preview uses the same icon as the Images nav item. * Images: full-height panes, wider settings rail, Create/Train offset from the selector The rule between the panes now runs the whole page height (the row drops its bottom padding and each pane pads its own content), the settings rail is wider on both Create and Train, and the Create/Train switch sits further right of the model selector. * Images: put both tabs on the Hub's centered measure Top bar and content now share mx-auto max-w-1100 with px-5 / sm:px-8, so Create and Train sit at the same width and position as the Hub instead of running edge to edge. * Images: restore the top bar position, drop the panes lower under it * Images: center the mode switch, flip the arrow with the orientation, app tooltips everywhere The Create/Train switch is centered on the page instead of trailing the model selector, with wider buttons. The flip control's arrows now rotate with the orientation and its label says which way the flip goes. Every native title tooltip on the page is now the app's tooltip, so they all get the rounded surface instead of the OS box. * Images Train: plainer field text, no green buttons, columns that stop colliding - The dataset name, trigger prompt, adapter name and custom base fields now say what they are in plain words instead of leaning on example values. - Import, Upload, Back, Back to settings and Train another are outline buttons, not green ones. - Example thumbnails are landscape tiles, so photos are not cropped to chunky squares. - Settings cells get min-w-0 and the select value truncates, so a long option like the nf4 label no longer widens its column into the next one. - The number stepper sits a little further in from the field edge. - Create and Train are wider. * Images Train: roomier example cards with Import on the thumbnail row * Video: same treatment as the Images tabs - No cards: the rail and the canvas sit on the page background, divided by a rule that runs the full page height, on the Hub's centered measure. - Wider rail, chat's sliders, hover-only scrollbars. - Every native title tooltip is now the app's tooltip, including the clip cards. - Reapply and Cancel are outline buttons, the empty state uses the Video nav icon, and the clip tiles are less rounded. * Images and Video: narrower generation rail, matching Train headings Create and Video rails go from 392px to 368px. Train a LoRA and Training settings are now the same size and both in the heading font: the h2 already picks it up from the base rule, so the settings header opts in with font-heading and the weight that rule pins. * Images Train: shorter copy throughout Family notes, example descriptions, precision labels and every helper line are trimmed so they stop wrapping to three lines and colliding with the next column. The nf4 label now fits its select without truncating. * Images Train: a little more spacing between field groups * Images and Video: tighten code comments * Fix training start NameError, the load-order guard test and CPU-only diffusion tests - start_training forwards resume_source_run_id to _start_training_impl, which reads it. Without it every start raised NameError. - Restore main's anchor in the load-marker order test: the file now has an earlier `if config.is_gguf:`, so indexing the first one compared the wrong branch. - The two diffusion tests that reach diffusers now skip when it is absent, matching the CPU repo-test env. - The UI smoke finds nav rows that live in the sidebar's More flyout. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Treat a null metadata caption as no caption str(None) stored the literal "None" as the caption, so a null row counted as captioned and would have trained on that text. Also drops an unused import. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix invalid-UTF-8 500s, the flat Canny map and the dropped DiT knobs read_text raises UnicodeDecodeError, which is not an OSError, so one bad caption sidecar or video sidecar 500d the info, upload and gallery routes. A flat image now yields the all-black edge map instead of its own luminance, and the four DiT loss knobs the trainer implements are declared so model_dump keeps them. * Use the ui font-size tokens instead of raw px text utilities text-[11px] and friends ignore the UI font size preference, which the repo's font-scale contract test enforces. Same rendered size at the default scale. * Fix diffusion dataset 500s, the dropout-1.0 no-op run and the reset base pick Four correctness fixes on the training side: - The labeling grid read caption sidecars under except OSError, but a non-UTF-8 sidecar raises UnicodeDecodeError (a ValueError), so one bad file 500d /diffusion/dataset/{name}/images and the grid could not be opened to repair it. Read it as no caption, matching the info summary. - An image past Pillow's own hard limit raises DecompressionBombError, which derives straight from Exception and so escaped the upload guard's (OSError, UnidentifiedImageError, ValueError) and returned 500 instead of the intended 400. - lora_dropout accepted 1.0, which makes PEFT build nn.Dropout(p=1.0): lora_A and lora_B receive no gradient and the run saves an untrained adapter while reporting normal progress. Bound it below 1.0, matching the LLM request schema. - The train panel re-seeded the base repo on every dataset refresh because the family object identity changes on each info fetch, so an upload or caption save silently replaced the user's chosen base and the run started on a different model. Track the pick and only re-seed on a real family change. * Show the retained failure when a video page mounts after a failed job Mount-time recovery handled only phase=completed, so reloading the page after a multi-minute generation failed left an idle view with no diagnosis: the backend keeps the terminal failed record only until the next job, and nothing else survives the reload. Surface it the same way the poll does, filtering the cancelled sentinel. * Fix batched generation crashes, cache keying and unreplayable recipes Four bugs in the batched inference path, all found by review: - A mixed-prompt batch sent a scalar negative prompt against a prompt list. Z-Image asserts on the length, and Qwen-Image, Krea 2 and FLUX true-CFG encode a batch-1 negative against batch-N latents and fail in the transformer's text/image concat. Broadcast it to match the batch. - The FBCache step-cache reset sat above the chunk loop. diffusers only resets that state at the end of a successful call, so a forward that raised (the OOM the backoff is meant to recover) left its own residual behind and the halved retry died on a shape mismatch. Reset before every forward instead. - The conditioning cache keyed on the checkpoint alone, but a GGUF or single-file load takes its text encoders from the companion base, so the same checkpoint against a different base reused the previous base's embeddings. Key the base too. - Gallery records stored the base seed and the requested batch size even when a prompts/seeds list drove the run, so restoring the second image of seeds=[5, 99] replayed seed 5. List-driven outputs now record as single-image recipes on their own seed. Also bound strength above 0: every img2img pipeline derives its step count from it, so 0 leaves zero denoising steps and either raises or, on SDXL, crashes on empty latents. * Fix quantized-load LoRA bake, prequant family exclusions and outpaint canvas Six review findings across the Images page and model scanning: - The quantized (int8/fp8) load path can only attach LoRA adapters before quantization, but the frontend load request had no loras field, so every generation after such a load was rejected and each reload repeated it. Send the selection with the load. - build_prequant_checkpoint passed no family to the scheme exclusions while recording the family in metadata, so a Qwen int8 artifact baked the short-M text-stream linears and was then rejected wholesale by the loader's family-keyed check. - Registering a bare single-file checkpoint directory produced no On Device row even though the images loader can load it; only its parent worked. Admit that shape when nothing else matched. - Unload left the Reapply target set, so the repair path was skipped and Reapply reloaded the ejected model. Clear it, as the video page does. - Both FLUX.2 bases were trusted for training but not inference, so Deploy to Create rejected every FLUX.2 adapter. - Outpaint allocated the grown canvas before downscaling, exceeding the browser canvas area cap on a large photo; an over-cap canvas is unusable, so Extend silently posted a fully transparent image and mask. Scale the source first. * Send the picked GGUF filename with the quant so diffusion loads fire The variant expander emitted only the quant label, and nothing else in the frontend set ggufFilename, so the Images and Video pages could never take their GGUF branch: both gate it on meta.ggufVariant and meta.ggufFilename, then fall through to the single-file path, which returns because the id is a repo id and not a .gguf name. Every quant pick was a silent dead click, with no load request reaching the backend. The filename was already on the variant row (the picker keys its list on it, and the variant validator requires a non-empty string), so thread it through the click handler. The chat path is unaffected: it reads ggufVariant and never needed the filename. * Version the conditioning cache key and reject non-finite flow_shift Two correctness fixes: - The cache keyed the checkpoint and its companion base by name only, so a Hub repo advancing to a new commit, or a local directory updated in place, kept returning embeddings from the previous text encoder. Pair both with a revision marker: the locally resolved commit sha for a Hub repo, config plus text-encoder file stats for a directory. Neither loads the encoders, so a warm run still keeps them off the GPU. - flow_shift only checked positivity, but JSON accepts 1e309, which floats to inf, and inf <= 0 is False while NaN fails every comparison. The sigma table then evaluates s * u / (1 + (s - 1) * u) as NaN, which poisons every sampled sigma and saves a corrupted adapter while progress looks normal. Require a finite value. * Keep curated models listed, guard the video companion repo, pin diffusers Three review findings: - The picker filtered every catalog member out of Recommended and Hub search on the way to canonical group rows, but nothing renders those rows yet (catalogGroupFitsDevice and groupMatchesQuery are imported and unused). A task-scoped picker's models list is catalogToModelOptions(), i.e. group members exclusively, so both lists came back empty and no curated model could be discovered or downloaded. Keep the artifacts listed until the grouped UI exists. - The video delete guard compared only repo_id, so deleting the companion base of a loaded GGUF video model was allowed even though it supplies the VAE and text encoders. Compare base_repo too, matching what the images guard already does for its companions. - diffusers was declared unversioned while the diffusion stack requires 0.39 (Krea2Pipeline, the cache_context child registries, the Flux2 and Z-Image pipelines), so an upgrade could keep an older release and selecting an advertised model failed until the user upgraded by hand. * Namespace the trainer conditioning cache per checkpoint, bound the learning rate - The trainer keyed its persistent conditioning cache on family and resolution only, while the keys themselves carry just the caption or image content and crop variant. One cache directory reused for two checkpoints, or for the same repo at a new revision, let a warm run skip loading its encoders and train on the other model's embeddings and latent statistics. Namespace on the base checkpoint and its resolved revision as well. The revision helper now lives beside the cache in diffusion_train_extras and the inference wrapper delegates to it, so the two cannot disagree about what counts as the same source. - The diffusion learning rate only checked positivity, but 1e309 floats to inf and satisfies gt, so the route evicted the resident models and started AdamW with an infinite rate: the first step destroys the adapter while progress looks normal and the result is saved. Bound it below 1.0, matching the LLM schema, which rejects inf for the same reason. * Fix GGUF image model picks doing nothing, and pick the train base in the top bar The quant rows never forwarded the .gguf filename, so every hub GGUF pick on Images/Video fell through to a silent return. On Train the top bar now picks the training base instead of a generation model, which is GGUF-only and untrainable. * Pin diffusion and video loads to the live HF cache root Both read huggingface_hub's import-time HF_HUB_CACHE, which changing the cache folder does not update: progress counted the old root while the download wrote to the new one, and from_pretrained could split one model across both. * Add the diffusion download plan endpoint Reports the repos and exact files a pick needs so the download manager can stage them with the loader's own file scope. A plain snapshot would add the packaged root single, transformer shards and fp16 twins the loader never opens. * Add a file-scoped flavour to the Hub download job Lets a consumer that reads a deliberate subset of a repo stage it through the normal download manager. Keyed as "@scope" so it never collides with a quant or with the repo's full snapshot, and the file list rides the registry so an XET to HTTP retry respawns the same scoped job. * Stage image and video downloads through the Hub download manager They downloaded inline inside the load, so they had none of the manager's disk preflight, manifest verification, resume or panel progress. Picks now stage as scoped jobs carrying the loader's own file list, then load from a warm cache. * Fetch staged GGUF checkpoints as scoped jobs, and stop calling diffusion models unsupported A GGUF entry went out as a full snapshot, whose ignore list drops *.gguf: the job finished at once having fetched only docs, and the repo landed on device unloadable. Every entry is scoped now. The Hub also no longer tags image/video models as unsupported (they run on their own pages), and those pickers name what they select. * Apply the picker task filter to local model sections LM Studio, ./models and custom-folder rows ignored it, so the Images picker listed chat GGUFs that 400 on a diffusion load. The backend already tags every local model with a task for this purpose. * Route a chat pick of a diffusion model to the Images or Video page Chat cannot load one, so it was either hidden or failed on load. The unfiltered picker now lists on-device diffusion models and navigates to the page that runs them, passing the repo and quant so that page loads it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Route the real GGUF filename, keep non-GGUF curated models, key scoped downloads by file set Five review findings, four of them ways a click did nothing or fetched the wrong thing: - A chat pick of a diffusion model routed ggufVariant (a label like Q4_K_M) in the search param the target page uses verbatim as the GGUF filename, so the load asked for a file that does not exist. Route ggufFilename; no filename means a curated non-GGUF pick, loaded as a pipeline. - The task-scoped pickers kept only GGUF repos, so the catalog's bf16, bnb-4bit and single-file fp8 artifacts could not be discovered or downloaded on the Images and Video pages even though loadSpecFor knows how to load them. Keep curated artifacts whatever their format, in Recommended and in Hub search. - Both pages deduplicated routed selections on the model alone, and they now stay mounted, so picking the same repo again -- another quant, or the same one after chat evicted it -- returned early without loading or clearing the query string. Key on model and quant. - Every scoped image download shared one @diffusion job key regardless of the requested files, so switching quant mid-download adopted the running job: the UI waited on the first file set, then loaded a file that was never fetched. Include a digest of the file set in the key. - A scoped plan silently dropped requested files missing from Hub metadata, and snapshot_download succeeds when an allow pattern matches nothing, so the job reported completion and triggered a load with required files absent. Fail the job instead. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the scoped download key derivable, and stop the hidden page hijacking a route Four review findings, the first a regression from my own last commit: - Keying scoped download jobs by a digest of the file set broke the download manager: it builds that key client-side (it polls and cancels before any response tells it a key), so it watched and cancelled a key no worker owned and never fired its ready callback. Keep the derivable "@scope" key and refuse the second request instead when a live job on the slot is fetching a different file set -- decided inside the registry claim, under the lock, so a concurrent claim cannot slip past it. The manager records the file set on the job as well, so a sibling quant's transfer is not adopted locally either. - Both diffusion pages read the route query through a loose useSearch and both stay mounted once visited, so the hidden one consumed the other's ?model=: it navigated back to its own route and tried to load, say, an image checkpoint as a video model. Only the visible page consumes it. - The staged download plan was built without the configured HF token or the Advanced values the load itself sends. The token matters most: the backend's Hub metadata lookup is best-effort, so a gated base silently planned no companion entry and the load pulled those multi-GB files inline, outside the manager. The memory/quant controls decide whether the base transformer/ shards are needed at all, and the route dropped memory_mode, cpu_offload, the prequant path and the LoRA selection before asking for the plan. - The video preview kept playing after leaving the page: the keep-alive layout only hides it, and display:none does not pause a media element, so a clip the user unmuted kept its audio going over the next page. Pause on the active transition and do not auto-replay while hidden. Also completes the hand-built request bodies in the hub download tests: the scoped-files field this branch added to the route read as an AttributeError against them, failing five tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Serialize the GPU handoffs, gate DiT training on a GPU, and keep 3.9 installable Six review findings, three of them evict-then-fail orderings: - The chat load reclaimed the GPU without telling the arbiter it existed. A chat load holds no llama-server process until its GGUF has downloaded, which is minutes, so a competing Images/Video acquire in that window found nothing to cancel, took the GPU, and the chat load then spawned onto the same device. It now registers an in-flight marker through acquire_for's register hook (under the arbiter lock, as the image and video loads do), the evictor cancels a marked load, and the route undoes itself if ownership moved while it loaded. - The Hub-download conflict check ran after that handoff, so a GGUF the download manager already owns destroyed the resident Images/Video pipeline and then 409'd, having loaded nothing. It moves above the handoff, together with the marker it handshakes with. - The image load released the engine router's transition lock before registering the load, so a second load choosing the other engine could unload the still-idle engine this one captured; the load then landed on a deactivated engine, where generate, status, unload and the arbiter's evictor can no longer reach it. Registration now happens under that lock and refuses if the engine changed. - Training a DiT family on a host with no GPU was accepted: nf4 is not a CPU fallback, its 4-bit load goes through bitsandbytes, which requires CUDA, XPU or MPS. The start unloaded the working Images pipeline, pulled the text encoders, and only then died in the child. Rejected before the teardown now, and /info stops advertising a precision that always 400s. SDXL keeps its documented fp32-on-CPU path. - Both diffusion pages kept the routed-pick marker forever, so re-picking the same checkpoint (after chat evicted it) neither loaded nor cleared the query string. The marker is released once the query is gone. The Images key also carried a stray NUL byte, which made the file read as binary to grep and other tooling. - diffusers dropped Python 3.9 in 0.38, so the unconditional >=0.39.0 pin left pip no candidate at all on 3.9 and made every install that composes the huggingface extras unresolvable there. The floor is conditional now. Also fixes tests that were already red on the branch: two hand-built request fakes had gone stale against fields this branch added, and the handoff-ordering test only failed on a host with fewer than two GPUs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the GGUF variant contract test against the merged handler signature The assertion pinned the exact single-line call handleVariantClick(v.quant, v.downloaded, expectedBytes, v.filename), but the handler takes (quant, filename, downloaded, sizeBytes) and prettier wraps the call across lines, so the mandatory repository test job failed on every push. Match the call structurally and assert the filename really is forwarded in the handler's argument order. * Stop a background page and a stale record taking the GPU or a download with them Five fixes from a review pass over the diffusion work. delete-finetuned rmtree'd a model the Images or Video engine was holding: every guard on that route is chat-only, and Images loads any local path, so deleting a local diffusion model under the storage root pulled the weights (and the companion VAE / text encoders sd.cpp re-reads each generation) out from under a live pipeline. The cached-model route already refuses this; the trained/exported one now does too, matching by path rather than repo id, and failing open on a chat-only install so it cannot block ordinary deletes. A staged download finishing while its page was hidden loaded the model and evicted whatever the user was actually using: both diffusion pages stay mounted behind the router and a load takes the GPU unconditionally. The pick is now held until its page is on screen again, which is also what chat does. A scoped download could report success having fetched nothing. With Hugging Face metadata unavailable no manifest is written, so verification is a no-op, and snapshot_download returns an existing snapshot folder without downloading when its own repo_info call fails. A repo already on disk from a full snapshot job (which ignores *.gguf) therefore completed with no weights and auto-loaded against them. The requested file list needs no network, so it is checked against the disk directly. The XET to HTTP retry reclaimed the job slot without the scoped file list, and that claim overwrites the stored record, so a later identical scoped start compared an empty list against the real one and 409'd instead of adopting the running download. The DiT accelerator gate probed torch.mps.is_available(), which only exists from torch 2.5 while the supported floor is 2.4. All three probes shared one try/except, so on torch 2.4 the AttributeError read as 'no block' and a CPU-only host still evicted the resident pipeline, downloaded the encoders and died in the child. Each accelerator is probed on its own now, through torch.backends.mps. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stage what an LTX-2.3 load reads, keep a routed file's load kind, drop an unbakeable LoRA Three from the latest review. The video download plan always asked for the wide base file list, so an LTX-2.3 pick staged the 2.0 base's VAEs, vocoder and connectors that the checkpoint supplies itself, while the companion files the 2.3 assembly does read were left out of the plan and pulled inline at load, outside the panel's progress, cancel and disk preflight. The plan now recognises a 2.3 pick by name (the load keeps the authoritative header probe, and under-guessing only falls back to the load-time pull), narrows the base list, and stages the extras in the same entry as the checkpoint so one repo stays one scoped job. A pick routed from the chat picker arrives as ?model= and ?quant= with no picker metadata, so a bare local .gguf or .safetensors was loaded as a pipeline: an explicit model_kind wins over the backend's filename sniffing, so it evicted the resident model and then failed on the missing model_index.json. Both pages now derive the load kind from the path, the same way their own picker handlers do. A torchao int8/fp8 build takes adapters only at load time. Switching artifact inside one family keeps the LoRA selection, since the family did not change, but the load did not bake it, so the next generation was rejected with 'reload the model with the adapter selection' while the picker still showed the adapter as active. The selection is now dropped once per resident build, with a message saying to pick and load again. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cancel an evicted safetensors load, spare the arbiter for CPU-only chat, fetch clips lazily Four fixes from the latest review round: - The GPU arbiter's chat evictor only cancelled the llama.cpp side. The orchestrator publishes active_model_name once its worker reports success, so an in-flight safetensors load was visible only as an entry in loading_models and finished onto the GPU after ownership had transferred. Cancel every pending load, and give the safetensors branch the post-load ownership recheck the GGUF branch already had. - A manual gpu_layers=0 GGUF load runs on the CPU with the GPUs hidden from the child, yet it took the arbiter unconditionally: it cancelled a running image or video generation for a model needing no VRAM, then held CHAT ownership so the next GPU workload unloaded it for nothing. Gate the acquire on the same predicate the launch-time CPU-only mask uses, as the image and video loaders gate on their resolved device. - The staged-download hook subscribes per repo, not per job, so another job on the same repo advanced the staged queue (starting a load whose scoped files were still downloading) or wiped a queue that was still running. Compare the variant each callback carries, like the chat page's auto-load does. - The video gallery fetched every record of a page into an object URL that lives until the page closes: 50 clips at tens to hundreds of MB each, for cards the user may never scroll to. Fetch a clip as its card nears the strip's edge, plus the selected one the player needs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim the comments across the diffusion backend Comment-only pass over the Python this PR touches: drop what the code already says, collapse multi-line explanations that still read on one line, and keep the reasoning that is not recoverable from the code. No code, docstring semantics or behaviour changes; verified with an AST comparison against the previous revision, and the backend suite is unchanged (same 37 environment failures as before: the API integration tests that need a live keyed server, the flash-attn install hooks, and the GPU memory fields). * Invalidate latents on a VAE swap, keep a cut-off generation, surface the EMA adapter - source_revision() scanned the checkpoint root plus text_encoder/tokenizer but not vae, so swapping or fine-tuning the VAE in place left the conditioning cache namespace unchanged and a warm run trained against latents from the old checkpoint. Include the vae directory, like any other component the cached tensors come from. - /images/generate answers only when the images are saved, and secure mode's tunnel caps an origin response near 100 seconds, which a native CPU or a high-step run passes routinely. The page reported failure while the work kept running, and a retry would duplicate it. A lost response (fetch rejection or a gateway status the origin never answered) is now told apart from a refusal: the page waits out generate-progress and reloads the gallery, so the run it started still lands. - The trainer emits the EMA adapter's path with the terminal event, but the state update dropped it, so neither the run history nor either response schema carried it and an enabled EMA left nothing discoverable. Keep it, and show it next to the primary adapter. - weighting_scheme advertised a choice of timestep sampling; sampling is always logit-normal and the flag only selects the bell loss weights. Describe what it does. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Hide unloadable cached rows, hold the dataset interlock, bound a GIF export - The cached-model listing tagged any repo with a model_index.json as text-to-image, so a community pipeline the image loader's trust rule refuses still got a row in the Images picker, and a detected-but-untrusted video repo fell through to that same tag. Gate the image tag on the load path's rule and hide an untrusted video repo outright. - A routed diffusion pick only carries a GGUF filename, which is all the chat picker has, so a curated single-file artifact arrived with no quant and was loaded as a pipeline: from_pretrained on a repo with no model_index.json. Pass the page's own catalog spec into the route pick, so a routed pick resolves to exactly what a direct pick on that page resolves to. - The dataset mutation endpoints checked is_active() and only then handed their filesystem work to a thread, so a start reserving in that gap changed captions or removed images underneath the preflight or the running trainer. The interlock is now registered for the whole request under the lock reserve() uses, and a start refuses while a mutation is open rather than waiting on it. - GIF export held every kept frame as a paletted image before encoding; a clip may be 2048x2048 for 1024 frames, and at the 12 fps target the step is 1, so one export click could allocate over 4 GB and take the backend down. Downscale past 720 px and widen the step to keep at most 300 frames. - seed accepted any Python int, so an out-of-range one passed every preflight, evicted the resident models, spawned the trainer and only then died in torch.manual_seed. Bound it to torch's 64-bit range in the request and config. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the accelerator probes in the DiT family-metadata tests Six tests read family_train_infos() (or a start preflight) without pinning the host probes, so they only held on a machine with a bf16 accelerator: on a GPU-less runner the DiT gate empties precision_modes, turns supports_compile off, and replaces any other preflight message with the no-accelerator note, and all six failed there. A conftest fixture pins both probes for exactly those tests, so they assert the family metadata they are about on every host. The gate's own CPU-only behaviour keeps its dedicated tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Carry the pipeline task into the hub inventory the pickers read - The task-scoped pickers filter On Device rows on a task, and the chat picker routes a diffusion pick by the same field, but those rows come from the /api/hub inventory, which never carried one: the Images and Video pickers listed nothing on device and the routing never fired. Both cached scans and the local listing now tag rows with the classifiers the models API already uses, the schemas and the frontend adapter carry it through, and a row the backend classified as a generation task is exempt from the chat-only guard that was also dropping it. - The local routing map was keyed by model_id while the row click passes id (a filesystem load id for a models_dir or LM Studio entry), so the lookup missed and the pick fell through to the chat loader. Key both. - A staged download whose start answered "error" left its head in place, where the effect never re-runs and onReady never fires, so the pick was stranded until the user reselected. Clear the queue and say so. - Every scoped pick in a repo shares the @diffusion variant, so the variant alone cannot tell two file sets apart: restaging while the first job finished let its completion pass for the new pick and load a checkpoint that had not downloaded. Bind the callbacks to the repo + file set they started, and to the staging generation. - A rejected generate POST does not say whether it reached the backend, so an immediately idle progress read was ambiguous and a submission that never landed looked like a finished image. Require evidence: progress seen active, or a gallery record that was not there before the POST. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the OpenAI image URL fetchable, keep WebM audio, stream example imports Four review items on the diffusion Studio work: - response_format=url returned the bearer-gated gallery route, which a standard image client downloads with no Authorization header, so the default response format was unusable. Mint a short-lived HMAC link instead (the shape RAG already uses for pdf.js) served by a signed route, and leave the gallery route itself bearer-only. - A manual gpu_layers=0 load carrying speculative_type="off" -- a value the UI persists and sends -- read as GPU-bearing, so it took the GPU arbiter and evicted a resident image/video pipeline even though the launcher hides the GPUs for it. Canonicalize the mode and exempt "off". - The curated example import prepared the whole split before the loop stopped at the 10-100 image cap; m1guelpf/nouns is 49,859 rows / 328 MB. Stream instead, with the prepared load kept as a fallback for a repo that cannot stream. - WebM export dropped the audio track an LTX-2 clip carries, silently, on the format offered for web embeds. Mux it as Opus through a resampler + FIFO, and keep exporting the video alone on a build without libopus. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not stub out triton on a GPU host when the Xet backend fails to import The lazy loader retries `import unsloth_zoo.hf_xet_fallback` under UNSLOTH_ZOO_DISABLE_GPU_INIT=1 whenever the first attempt raises. That flag makes unsloth_zoo take its MLX/CPU path, which injects triton and bitsandbytes STUBS into sys.modules for the rest of the process. On a working GPU box whose first import failed for an unrelated reason (a bitsandbytes/CUDA mismatch, say) the retry succeeds, so Studio boots looking healthy and then dies at the first CUDA-only kernel: a GGUF or compiled diffusion generation hits the stub and returns NotImplementedError: Unsloth: 'triton.tools.experimental_descriptor.enable_in_pytorch' was called on Apple Silicon / MLX, where triton is stubbed out. so every image generation 500s with an Apple-Silicon message on a Linux CUDA host, while the load reports success. Found by loading Z-Image-Turbo GGUF through the API on a box where bitsandbytes could not initialise. Gate the retry on the host genuinely having no accelerator. The Xet stall watchdog is optional and already degrades with a warning; a process whose triton is stubbed out is not recoverable. The warning now says why it did not retry. * Fix the lost-generation proof set, the settle timeout and the hub inventory's diffusion gates Seven fixes from the latest review round on the Images page and the hub cache inventory. Images page: - The lost-POST settle path built its "already seen" gallery id set inside the catch, after the request failed. By then the earlier runs of the same batch had already prepended their records, so run 2 could accept run 1's image as proof that its own request reached the backend. The set is now captured once before the first POST and grows with every record the batch produces. - settleLostGeneration fell out of its SETTLE_MAX_MS loop and returned normally, so a wedged generation was counted as done and the next run started against a busy backend. It now throws on timeout. - Restoring a recipe cleared the ControlNet selection but left the workflow tab and the init / mask / reference images pointing at whatever was loaded, so the next Generate conditioned on an unrelated image. It now clears all of them and returns to Create. - The download plan omitted the adapter selection the load itself bakes in. A baked LoRA forces the dense build path, so the plan described a different file set than the load that followed and the rest was pulled inline, outside the download manager. Both now derive the list from one helper. Hub cache inventory: - A download for a repo an Images or Video load is staging was allowed to start: only the llama.cpp loader was consulted. Both diffusion backends already expose loading_repo_ids for the delete guard, and the download guard now reads them too. - A companion-only prefetch (pipeline manifest plus VAE and text encoder, no transformer) passed the snapshot-partial check, since every file its manifest expected did arrive, and was advertised as on-device although from_pretrained cannot load it. - The single-file flag never reached the picker through the hub inventory path, so a checkpoint-only diffusion repo read as a full pipeline and failed after the handoff. The two pipeline-shape helpers now live in hub/utils/inventory_scan.py so /api/models/cached and the hub inventory classify the same repos the same way. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop adopting an unknown scoped download, leaking raced blobs and resurrecting deleted clips Three items from the latest review round. A scoped download job carries a deliberate file subset, and every file set of one repo rides the same "@scope" slot. A client that adopts a live job from the backend had no file list to compare against: the active-downloads response never carried one, so an adopted job's set was unknown and any later scoped request for the same repo read as "already started". Selecting a different checkpoint then waited on the wrong transfer and tried to load a file nobody fetched. The response now publishes the scoped file list, adoption records it, and an unknown set no longer satisfies a scoped request. A gallery record can be deleted while its blob is still downloading. The delete revokes the URL present at that moment, so the fetch that lands afterwards inserted a fresh object URL for a record no card renders and nothing can revoke: a full MP4, tens to hundreds of MB, pinned for the rest of the session, and once per raced fetch. Both galleries now discard a blob whose record went away, with an epoch covering the video page's Clear all. The video backend keeps the last completed job until the next one starts, and the Video page merges that record on mount to cover a job that finished after the gallery fetch. Deleting the clip left the record in place, so every reload prepended a ghost card whose file request 404s until another generation replaced it. Deleting the clip, or clearing the gallery, now clears the matching terminal record, and the page skips a record it deleted itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Resolve revisions from the live cache, serialize dataset imports, and stop pinning every gallery blob Five more items from the review round. The conditioning-cache revision marker read huggingface_hub's import-time HF_HUB_CACHE constant. Studio can move its cache during a session and loading follows the live setting, so after a move the marker went unresolved (or pointed into the previous root) and pulling a new revision of the same checkpoint no longer invalidated the cache: a warm run could reuse the old encoder's embeddings and the old VAE's latents. It now looks in the active Studio cache first and keeps the environment and the library constant as fallbacks, which the trainer subprocess still needs. The dataset interlock counts mutations rather than excluding them, so two imports of different examples into the same empty name both got past the emptiness check. The winner promoted its staging directory atomically; the loser found the folder non-empty, fell back to a per-file move, and merged its images and captions into the winner's dataset. Imports now take a per-folder lock, a second one is refused with 409, and the emptiness check is repeated under the lock. On Windows the sd.cpp asset resolver filtered only by accelerator token, so a Windows arm64 host matched an x64 zip, downloaded and installed it, and failed later when the binary would not run. It now filters by architecture the way the Darwin and Linux branches do. Every gallery page fetched every PNG up front and kept the object URL for the session, so scrolling a large gallery grew memory without bound for tiles the user may never look at. The Images strip now fetches a tile as it nears view, like the Video strip, and keeps the eager path only where IntersectionObserver is unavailable. A 503 carrying a JSON body comes from the application, not a proxy, so it is surfaced as the error it is instead of entering lost-response settlement and being reported as a request that never reached the server. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the class of a failed generation instead of a bare "Image generation failed." Found on a macOS runner: the native renderer aborts inside its own text encoder there, and the page showed only "Image generation failed." with the sd-server backtrace left in the server log, so nothing about the failure reached the user. The failure is now classified into fixed text, out of memory and native-process death, so the message says what happened and what to try. None of the engine's own output is echoed, since a native tail carries local paths and argv; that stays in the log, and an unrecognised failure keeps the original literal. * Treat an undecodable caption sidecar as the tombstone the trainer sees Uploads store .txt and .caption sidecars as raw bytes, so one can hold invalid UTF-8. The trainer treats any existing sidecar, decodable or not, as an empty tombstone and never falls back to the metadata row for that image. The labeling grid and the dataset summary read an undecodable sidecar as absent instead, so both showed a metadata caption that the run would silently replace with the instance prompt, and counted the image as captioned. Both now track sidecar presence separately, so what the user reviews is what the run trains on. * Keep the reason a native server died, not just its backtrace A ggml abort prints its cause first and then a stack trace, so reporting the last twenty captured lines gave twenty addresses and nothing about the failure: on the macOS runner the native server died on an unimplemented Metal op and the message carried only frame pointers. The captured tail now leads with the lines that name a cause and keeps recent context after them, for both the startup failure and the mid-request death. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop staging the dense text encoder for an fp8 video load Two halves of the same gap, found while measuring the LTX-2.3 download plan: - The video download plan and the scoped pre-download never saw text_encoder_quant. An fp8 request loads a hosted pre-cast encoder, so asking for one still staged and downloaded the base repo's dense Gemma3 (48.79 GB of Lightricks/LTX-2 on the 2.3 distilled pick) that the pipeline then never opened. The plan now drops those shards and stages the pre-cast checkpoint instead; their configs stay, since the pre-cast loader still meta-inits the encoder from the base repo's component config. - The LTX-2.3 assembly builds every component itself, so pipe_kwargs (which carries the pre-cast encoder for from_pretrained) never reached it and an fp8 request silently loaded the dense encoder anyway. It is passed across explicitly now. The dense skip is earned, not assumed: only a pre-cast checkpoint that resolves on the Hub lets the plan drop the dense shards, and only one already fetched to disk lets the pull drop them, so an unpublished or gated artifact leaves both exactly as they were. If injection still fails after that, the load tops the dense weights back up rather than handing from_pretrained a snapshot with no encoder in it. Measured against the real Hub on the 2.3 distilled Q4_K_M pick: 67.24 GB before, 18.92 GB with a 0.43 GB stand-in for the pre-cast artifact (the base entry drops from 24 files / 48.79 GB to 13 files / 0.04 GB). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Match the file's typing idiom for flow_shift models/training.py annotates with typing constructs throughout (105 Optional[...], no Union), and flow_shift was the one place using PEP 604. Union[] reads like the rest of the file, and it also drops the runtime evaluation that would raise on Python 3.9. * Bound the gallery blob cache, and three interlock fixes Four review findings, all reproduced first: - The gallery object-URL caches were unbounded. A clip runs from a few MB to a few hundred, both pages stay mounted after their first visit, and entries were only dropped on delete, so scrolling pinned everything for the session. Both pages now share a byte-budgeted LRU (512 MB video / 192 MB images) keyed off the visibility signal the near-viewport fetching already provides. On-screen media, the selected clip or image, and the item just fetched are never evicted, so eviction is invisible and a single item larger than the whole budget cannot evict itself into a refetch loop. - The image, video and chat load guards ran two independent training probes but returned early when the FIRST one raised, so an unreadable LLM backend disabled the diffusion interlock and a load could proceed straight into an active diffusion trainer on the same GPU. The probes are independent now. - An engine switch swallowed a failed teardown and published the new engine anyway, which is exactly the leak the unload exists to prevent: the arbiter's evictor, /images/unload and the next load all resolve through get_active_diffusion_engine(), so the still-resident pipeline (or a live sd-server) became unreachable and the next load allocated on top of it. The switch now fails and leaves the old engine published, so it stays reclaimable. - The native generation timeout was 30 minutes while the Images page waits up to 6 hours (SETTLE_MAX_MS), so slow-but-progressing CPU jobs died deterministically at the deadline. Measured on GPU-less runners, a 512x512 4-step Q2_K generation took 900 s on Linux and 1465 s on Windows, so larger images or step counts clear half an hour easily. The ceiling now matches the page's window and applies to the whole request: chunks of a split batch share one deadline instead of each getting a full budget. Cancellation is unchanged. Declined: gating the huggingfacenotorch extra off Python 3.9 over the conditional diffusers marker. The marker is deliberate and its comment says why: diffusers dropped 3.9 in 0.38, so pinning >=0.39 outright leaves pip no candidate and the whole extra unresolvable there. The pipelines it names live in studio/backend, which cannot install on 3.9 anyway (studio.txt pins matplotlib==3.10.9 and fastmcp>=3.0.2, both requires_python >=3.10), and the extra is the general core one, so the alternative drops 3.9 for library users who never touch Studio. * Close the load-versus-training-start race, and two picker fixes - The image and video load guards read is_active() and only then selected an engine, acquired the arbiter and registered the load. A /train/diffusion/start reserving inside that window freed residents the load had not registered yet, so the trainer came up beside a brand-new pipeline. The service already had exactly the right pattern for this in dataset_mutation, so gpu_load_admission mirrors it: reserve() refuses while an admission is open, an admission refuses once a start is reserved, both decided under the one lock. The span is only the registration, since begin_load returns as soon as the load is registered and _free_gpu_for_diffusion_training preempts an in-flight load from that point. Chat is deliberately not covered: its load spans an eviction plus a multi-minute GGUF load, and it admits models that fit beside training by design, which is a different contract from the diffusion pipeline's all-or-nothing one. - Hugging Face gives the LTX-2 family the image-to-video pipeline_tag (both Lightricks/LTX-2 and unsloth/LTX-2.3-GGUF report it), so a text-to-video-only filter dropped the flagship audio family out of Video Hub search while the rest of the app routed it to Video. - Task-scoped quant fit sized picks against the LARGEST visible device while resolve_diffusion_device_target returns a bare "cuda" and torch places on the current one. On a heterogeneous host that recommended a checkpoint sized for the bigger card and then loaded it onto the smaller one. Fit now uses the device the load actually lands on; identical on a homogeneous host. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Expose the persistent conditioning cache in the start schema DiffusionLoraConfig has carried cond_cache_dir for a while and the DiT trainer acts on it, but DiffusionTrainingStartRequest omitted the field, so Pydantic dropped it silently and every API-driven run fell back to the in-memory cache that is rebuilt from scratch each time. The warm path skips loading the VAE and the multi-GB text encoders on a rerun whose images, captions and resolution are unchanged, so this was a real capability that could not be reached. Contained like output_dir rather than left to the trainer subprocess's cwd, since it is another directory the trainer writes to. Blank or omitted still means the in-memory cache, so it must not resolve to the outputs root. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix diffusion policy and classification issues from review fp8 auto precision defaulted to precise accumulate on any non-consumer GPU, which made fp8 2.05x slower than int8 on RTX 6000 Ada and slower than not quantising at all. NVIDIA's professional whitepapers do publish equal FP8 rates for both accumulate modes there, so the hardware premise held, but the cost is in the cuBLAS path rather than the published rate. Default to fast accumulate: measured on B200 the flag is a no-op (4096^3 _scaled_mm at 3023.8 vs 3041.8 TFLOP/s, bitwise-identical output, 1.213 s vs 1.230 s end to end), so it is a large win where it bites and free where it does not. Precise accumulate stays available via transformer_quant_fast_accum. Z-Image's DiT is a Lumina2 derivative, so unsloth/Z-Image-GGUF and unsloth/Z-Image-Turbo-GGUF both declare general.architecture = "lumina2" and the whole line was tagged image-diffusion-unsupported and hidden from the Images "On Device" list, though validate_load_request loads them. Resolve shared archs from the repo/file name like bare "wan" already does, with a test asserting the picker and the loader agree for every family. The sage attention on-demand install ran an unpinned `pip install sageattention`, but PyPI's newest wheel is 1.0.6 and diffusers refuses anything below 2.1.1: the install always "succeeded", wrote an unusable version into the running venv, and was rejected on the next line. Carry the dispatcher's floor so pip resolves nothing instead. The dense-quant disk gate sized the download from the bf16-RESIDENT table. The fp32 families download twice that (Z-Image: 23,479 MiB against a 21,970 MiB gate), leaving a window where the check passed and the download filled the disk; Ideogram 4 ships fp8 and was overcharged the other way. Size the gate by published bytes, verified against HF sibling metadata for all 12 families. Patch installs went through unsloth_zoo, which refuses to import unless UNSLOTH_IS_PRESENT is set, and that is set by unsloth itself. The server imports unsloth at boot so it never showed there, but any other process ran silently unpatched with every install returning False, which is 13 test failures on a clean environment. Import unsloth and retry once, memoised per process. Also: the GGUF+LoRA refusal pointed at the native engine without saying a GPU host only selects it under UNSLOTH_DIFFUSION_ENGINE=sd_cpp, so the suggestion was unreachable; the gallery recipe recorded loras from the generate request alone, losing a load-time bake; load-progress claimed "40.07 GB downloaded" for a fully cached load; and pickers.tsx imported three catalog-group helpers it never used. Reported by oobabooga. * Keep the sd.cpp text encoder on CPU under Metal macos-14 loads FLUX.2-klein-4B Q2_K natively on mps and then dies on the first generation with exit code -6: ggml_metal_op_encode_impl: error: unsupported op 'RMS_NORM' -> ggml_abort LLMEmbedder::encode_prompt -> LLMRunner::compute -> GGMLRunner::compute ggml's Metal backend gates RMS_NORM on contiguous rows and aborts the process when that does not hold, with no per-op CPU fallback, so any LLM text encoder (Qwen3 for FLUX.2 and Z-Image, T5 for FLUX.1) takes sd-server down. The encoder runs once per prompt while the DiT runs every step, so pinning only the encoder keeps Metal for the part that matters. UNSLOTH_DIFFUSION_SD_CPP_METAL_TE_GPU=1 opts back in once ggml grows the kernel. * Gate the unsloth retry in the diffusion patch backend The retry added for the clean-environment patch failures is not free: importing unsloth pulls torch in behind it, which costs ~940 MB of RSS measured in a process that had neither, and on a host with no accelerator it fails anyway. A cross-platform CI job that had generated fine at ~900 s later died 19 s in with SIGTERM and every 'if: always()' step skipped, which is the runner being torn down rather than a step failing. Retry only when torch is already imported (true of the server and of anything patching a real module, and the condition that stops the retry from being what loads torch), unsloth is installed but not yet imported, and the first failure was the ImportError the sentinel guard raises. The clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only retry the unsloth import where it can succeed The gate still let the retry run on hosts unsloth does not support, which is where it is most harmful: a 7 GB macOS runner lost the Studio server 26 s into a load, and the Linux runner was torn down mid-generation. Neither MPS nor plain CPU can complete the import, so the retry there pays the cost and fails anyway. Require an accelerator unsloth actually supports (CUDA/ROCm via torch.cuda, or XPU), with UNSLOTH_ALLOW_CPU as the documented override, and hoist the predicate to module level so it is tested directly rather than through the import system. On a CPU-only host the retry no longer fires at all; on CUDA the clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Guard old diffusers, stream video exports, record conditioned recipes Three fixes from review. The 0.39-only pipeline classes (Flux2Klein, Z-Image, Krea 2, LTX-2, HunyuanImage) were resolved by getattr deep in the load, so on the older diffusers that packaging still allows on Python 3.9 -- diffusers dropped 3.9 in 0.38 and this project still supports it, so the 0.39 floor has to be conditional or the extra becomes unresolvable -- an advertised model failed with a bare AttributeError after its checkpoint had already been downloaded. Krea 2 already guarded itself this way; assert_pipeline_class_available now runs the same check for every image and video family from validation, before any fetch, and names the version and the fix. WebM export accumulated the whole VP9 output in a BytesIO and returned it as one bytes object that the response held again. The request caps allow 2048x2048 for 1024 frames, so an export runs to hundreds of MB and concurrent clicks could exhaust the process, while the MP4 route beside it already streamed from disk. transcode_to_file encodes to a temp file and the route returns a FileResponse with a background unlink, so nothing large is resident. A conditioned generation's recipe carried only the txt2img fields, so the gallery presented an inpaint or upscale result as a complete Create recipe and restoring it replayed an unrelated text-to-image request. The images themselves are still not persisted (user uploads with their own lifetime), but the workflow and its scalars are, restore reapplies them, and the toast now names the inputs that have to be supplied again instead of silently landing on Create. Reported by Codex. * Per-load video cancel event, family-gated image picker, cond cache refusal A cancelled video load could resume: begin_load cleared the shared cancel event, and unload() drops _loading without waiting for the worker, so the next load cleared the very object the cancelled worker was watching and its multi-gigabyte pull ran on alongside the replacement until the token check at the end. Each load now gets its own threading.Event, passed down through _fetch_te_prequant and _predownload_base, so a cancelled worker stays cancelled. A cached repo with a model_index.json was advertised as text-to-image on the trust rule alone, but validate_load_request also requires a detected image family, so a trusted pipeline of an unsupported class produced a picker row that deterministically 400s. The picker now applies both gates, mirroring the video branch. cond_cache_dir was accepted for sdxl and then ignored: only the DiT trainer reads it, while the SDXL trainer builds a per-run in-memory latent cache, so the promised cross-run reuse never happened. The route now refuses it with a 400 that names the families which do support it, checked against the resolved family so an omitted model_family with an SDXL base is caught too. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the frontend build broken by the gallery blob cache tsc -b failed on the branch head, so npm run build produced no dist and every platform job fell back to --api-only: blob-url-cache.ts(29,15): TS1294: This syntax is not allowed when 'erasableSyntaxOnly' is enabled dataset-labeling-grid.tsx / dataset-showcase.tsx: Argument of type '{ url: string; bytes: number; }' is not assignable to parameter of type 'string' The cache took its budget as a constructor parameter property, which the project's tsconfig forbids, and fetchGalleryObjectUrl now returns the blob size alongside the URL for that budget, which the two dataset thumbnail components still consumed as a bare string. Declare the field explicitly and destructure the URL at both call sites. tsc -b is clean and vite build emits dist again. * Recover from a ggml unsupported-op abort by restarting on the CPU backend ggml checks every node against the device's supports_op and calls GGML_ABORT when one is not implemented, because a single-backend graph has nowhere else to put it: there is no per-op CPU fallback. The whole sd-server dies with SIGABRT mid-generation and the user gets "the native image renderer stopped unexpectedly" with no way forward. Seen on macos-14 arm64 with FLUX.2-klein-4B Q2_K through the cross-platform CI: the text encoder is already pinned to CPU, and the abort moved into the denoise loop instead. ggml_metal_op_encode_impl: error: unsupported op 'MUL_MAT' -> ggml_abort StableDiffusionGGML::sample -> sample_k_diffusion A retry on the same backend would abort identically, so the load is restarted once with --backend cpu (the only flag that changes which backend executes the graph; --offload-to-cpu moves parameters, not compute) and the generation is re-submitted. The same checkpoint then renders slower rather than not at all. Strictly bounded: the signature must carry both the unsupported-op line and ggml_abort, the device must not already be CPU, and it happens once per load, so an OOM kill or a genuine crash still surfaces as itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix two tests that only fail in a full-suite run The 3.10 CI leg resolves PyAV 17, where av.container.OutputContainer is an immutable C type, so the no-libopus export test died on "cannot set 'add_stream' attribute of immutable type" before it asserted anything. Inject the refusal by wrapping the container av.open() returns instead; modules stay patchable on every build. Removing the injection makes the test fail again, so it still covers the branch it is named for. The Xet shim's degraded-path tests drop utils.hf_xet_fallback from sys.modules and import a throwaway copy. Restoring only the sys.modules entry left the utils package attribute bound to the throwaway, and the two disagreed for the rest of the process: a later monkeypatch of the dotted target patched one copy while the code under test imported the other, so the patch did nothing and test_fetch_te_prequant_only_reports_what_it_downloaded reached the real Hub and got a 401. Restore both bindings. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop an ignored-cancel sd-server, guard deletes during diffusion training, repair unusable managed binaries sd-server does not interrupt an in-flight job, so when it ignores a cancel the grace branch abandoned the poll and reported cancellation while the native job kept a core (or the GPU) busy to completion and held the server's job slot. The comment said the caller stops the server, but only unload does that immediately: a superseding load stops it after its multi-gigabyte download, and a load that then fails never gets there. Stop it here, as the deadline branch already does. DELETE /api/models/delete-finetuned checked only the LLM trainer, so it could rmtree the output directory a live diffusion LoRA run was about to write its adapter into. Consult the diffusion training service too, like the dataset mutation and model-load routes. find_sd_*_binary only checks is_file(), so an interrupted extraction (or a prebuilt for the wrong CPU) left a present-but-unrunnable binary the installer never retried: every load probed it, fell back to diffusers, and native inference stayed off until the directory was deleted by hand. Probe it and reinstall, but only for a copy under the installer-owned root -- SD_CLI_PATH, UNSLOTH_SD_CPP_PATH, an in-tree build and anything on PATH are the user's. * Plan the pre-cast text encoder, and make the cross-trainer GPU admission atomic An fp8 text-encoder request loads a hosted PRE-CAST checkpoint, but the image download plan never received text_encoder_quant, so the manager staged the base repo's dense encoder (FLUX.2-dev's Mistral-24B is ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) and the load then pulled the pre-cast file inline, outside the manager's progress and disk preflight. The plan now takes the field, resolves the hosted artifact with the same resolver the injection uses, stages that file, and drops only those components' dense weight shards. The load's own prefetch takes the same treatment, since it paid the same cost. Only a checkpoint that really resolves on the Hub earns the drop, so a gated or renamed artifact still stages the dense encoder the load will fall back to. The two trainers admitted each other with independent check-then-act guards: the diffusion route checks the LLM backend several network-bound preflights before it reserves, and the LLM route checks the diffusion service well before it spawns, so two near-simultaneous starts could both pass and train on one GPU. reserve() now re-tests the LLM backend under its own lock, and the LLM route holds the diffusion service's gpu_load_admission across its spawn, so exactly one of the two wins. Both halves fail open, so a chat-only install still trains. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the Advanced options a staged download planned against Staging does not set busy, so while a multi-gigabyte download runs the user can still change precision, memory mode, speed or the baked LoRA selection. The pending record held only the repo and artifact, and the completed download fired a load that read the CURRENT state: the staged file set could then be missing files that load needs (fetched inline, with no progress and no disk preflight) or hold gigabytes it no longer uses. One snapshot of every Advanced control is now taken when the plan is built, and it travels with the pending record into the load, so the load that runs is the one the download was planned for. * Do not advertise a family the installed diffusers cannot build The newer families (Z-Image, Krea 2, FLUX.2, LTX-2, HunyuanImage) exist only from diffusers 0.39, and 0.39 cannot be installed on Python 3.9 at all -- diffusers dropped 3.9 in 0.38, so the requirement is conditional or the whole extra becomes unresolvable. On such an environment the picker still offered those rows, every pick failed deterministically, and the error's advice to run pip install -U diffusers could not fix it without also upgrading Python. The cached-repo picker now applies the same availability check validate_load_request does, which is keyed on the pipeline class actually present rather than on the Python version, so it is also right for an intentionally pinned older diffusers on 3.10+. Fails open when diffusers cannot be imported at all: that is a different problem and the load path reports it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore the diffusion engine selection after each router test The active engine is module state, and several tests set it by plain assignment because what _activate does to it is the thing under test, so monkeypatch could not undo it. A leaked ENGINE_SD_CPP left get_active_diffusion_engine() handing back the sd.cpp backend for the rest of the process, and every later route that reads the active engine then saw an unloaded model: eight tests in test_openai_images_generations_route.py returned 503 in a full-suite run while passing on their own. The autouse fixture now snapshots and restores it. * Stream gallery clips, and close three races around them Four fixes from the latest review pass. The video gallery downloaded each clip into a blob before it could play, so playback waited on the whole file (tens to hundreds of MB), seeking was limited to what had arrived, and every viewed clip stayed pinned in the webview. The file route already streams and serves ranges; it just could not be a <video src> because it is bearer-gated. Mint a short-lived signed link instead (its own HMAC secret, 12 hour TTL, separate from the image links) and hand it to the element, which then fetches only the ranges it plays. That removes the blob budget, its LRU and every revoke on this page. The sd.cpp readiness probe accepted any process answering on the port, so a foreign server that grabbed the port between the bind check and the spawn was adopted as ours. Confirm the listener is our child before reporting ready, and stay best-effort (psutil missing, an unknown owner, or any probe error still passes) so the check can only reject a definitely foreign process. Dataset import held its lock for the extract but not for the upload path, so two concurrent uploads into the same folder interleaved; take the same lock and return 409. And reject Windows device names (CON, NUL, COM1..9, LPT1..9, with or without an extension) plus trailing periods in dataset names, which are unopenable on Windows. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Build the image download plan for the engine that will load /images/download-plan always asked the diffusers backend, while /images/load picks the engine per host: a GGUF pick on a machine with no usable GPU routes to native sd.cpp, which reads a single-file VAE plus text encoders and never opens the base repo's sharded components. Measured on unsloth/FLUX.2-klein-4B-GGUF (Q2_K): the plan staged 7.66 GB of FLUX.2-klein-4B components the native load discards, and the 7.80 GB sd-cli actually needs was then fetched inline by the loader, outside the download manager's progress and its disk preflight. Z-Image-Turbo is the same shape. The plan now asks whichever engine the load will select. predict_engine() applies the selection policy without any side effect: it activates nothing (staging a download must not unload the resident model) and only locates the binary rather than installing it, but still counts an installable binary as available, since that is what the load does on a fresh host. The native backend gains a download_plan built from the same _asset_specs the loader fetches, returning the same envelope, so the manager stages exactly the files sd-cli opens. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not let a queued generation outlive the model, and three scan fixes Five items from the latest review; four were real. An unload or arbiter eviction only cancels the generation holding _generate_lock. A second request queued behind it holds no cancel event yet, and Python locks are not FIFO, so it could take the lock the instant the active denoise released it, still see a loaded pipeline, and run a whole new denoise after the model was told to go away: the eviction then waits minutes for it and an image lands after the eject. Unload and a superseding load now raise a fence under _lock before they queue, and a generation that wins the lock while one is pending refuses instead. The cached-model scan judged pipeline completeness across every revision, so a repo holding an older complete snapshot plus a newer companion-only one read as complete while the snapshot from_pretrained actually opens has no transformer. Both scans now look at the revision the loader will open. Deleting a dataset image deleted its caption sidecar unconditionally, which for cat.jpg alongside cat.png removed the caption the survivor still resolves to. The sidecar now goes only with the last image of that stem, matching what the thumbnail cleanup beside it already did. Importing an example into a folder that holds no images but does hold files fell back to promoting the staging dir one file at a time, so an interruption left a partial dataset that the image_count check accepts as complete on retry. Those files are folded into the staging dir instead and the promotion stays a single atomic rename. The MPS generator report does not apply: torch.Generator(device="mps") has worked since PyTorch 2.0 (pytorch/pytorch#91348) and the studio installer pins torch>=2.4. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten diffusion comments Collapse the multi-line comment blocks across the image, video, sd.cpp and diffusion-training code to one or two lines each, and drop comments that only restate the statement below them. Comments only, no code or behaviour changes. * Tighten diffusion comments (second pass) Collapse the remaining multi-line comment blocks in the video page, training routes and service, sd.cpp server and installer, memory and speed planners, and the shared request models. Comments only, no code or behaviour changes. * Tighten diffusion comments (third pass) Collapse the remaining multi-line comment blocks in the attention, cache, LoRA, prequant, precision and compile-cache modules, the sd.cpp arg builder and engine, the video routes, the Ideogram 4 assembly, the model picker, and the diffusion test suites. Comments only, no code or behaviour changes. * Restore the dataset when an example import cannot be promoted Promotion folds the folder's pre-existing entries into the staging dir so the swap is one atomic rename. Every failure after that fold left the user with nothing: the 409 path and an os.replace error both fell through to 'finally: shutil.rmtree(staging)', which deleted the entries that had just been moved in there, while the response said 'Nothing was written'. A same-named entry was also unlinked outright before the promotion was known to succeed. Park superseded same-name entries in a rescue dir instead of deleting them, record every move, and restore all of them if any step of the promotion fails. The fold loop itself is covered too: renaming a non-writable directory raises EACCES on POSIX, which previously escaped as a 500 after the earlier entries had already been moved out. A failed rename now maps to the same retryable 409 as the rmdir conflict. Verified with the reported trigger (a non-empty mode-500 directory whose name collides with an imported file): the folder listing is now identical before and after the failed import. * Drain the teardown fence on a failing unload, give each load its own cancel event Two independent leaks on the image path, both already solved elsewhere in the same file. unload() incremented _teardown_waiters, ran _unload_locked() and decremented, with no try/finally, while the superseding-load path used a finally for the same pair. _unload_locked ends in clear_gpu_cache(), whose CUDA branch calls synchronize/empty_cache/ipc_collect unguarded, and a sticky CUDA fault makes those raise. The count then never drained, so every later generation was refused as cancelled for the life of the process, a fresh load included, since begin_load's own increment and decrement are symmetric. unload() is reached from the chat/video GPU handoff, the engine router and two training routes, so one fault during an ordinary handoff wedged image generation until restart. Release it in a finally. The image and native backends each cleared one shared cancel Event on a new load. unload() sets that event to cancel an in-flight multi-GB download and drops _loading in the same breath, so a replacement load is admitted while the cancelled worker is still inside the fetch, and its clear() re-enabled the very object that worker was watching: the cancelled download resumed and ran alongside the replacement. Take a fresh Event per load and thread it to the worker, as the video backend already does, and set it under the lock since begin_load now rebinds the attribute. * Name utf-8 on the diffusion text I/O and the sd.cpp subprocess pipes tests/test_text_io_encoding.py failed on five files this branch adds. Text I/O without an explicit encoding falls back to the Windows ANSI codepage, so a non-ASCII path or manifest value round-trips corrupted, and the three sd.cpp pipes decode the child's UTF-8 output as ANSI on Windows despite already passing errors = 'replace'. Eleven read_text() / write_text() sites across diffusion_compile_cache, diffusion_ideogram4 and diffusion_krea2, plus text = True on the sd-cli version probe, the sd-cli run and the sd-server pipe. * Record the load-time build on a gallery image's recipe A gallery record documents itself as the image's full generation recipe and is embedded in the PNG, but the only load-related field it carried was the repo id. A GGUF repo holds many quants, so that does not say which one made the pixels, and it says nothing about an adapter baked in at load time. The fallback meant to cover the baked case could never fire: with no loras on the request _adjust_baked_loras zeroes every baked adapter and _active_lora_pairs drops zero-weight entries, so active_loras was always empty. A baked-and-disabled build is not the same pipeline as a never-baked one, so the recipe could not reconstruct the image once the model was rebuilt. Persist model_kind, gguf_filename, transformer_quant and the baked adapter names, read off the load state rather than the request, and show them in the recipe popover. The new fields are optional with defaults, which matters because list_gallery_images drops any record that fails validation, so a required field would have emptied every existing gallery; a regression test pins that. * Drop eleven duplicated comment tails, restore the mxfp8 denial note The comment passes collapsed several wrapped blocks onto one line without deleting the last physical line of the original wrap, leaving the tail of each sentence repeated as its own comment underneath. Two of the eleven were re-worded rather than byte-identical, so a strict suffix match missed them. 6e16ad16f also dropped the line justifying the qwen-image mxfp8 denial while that rule stayed live in _FAMILY_SCHEME_DENY, under a header that then documented only fp8 and nvfp4. Restored. Comments only; verified with the AST gate. * Only let the dense-quant fallback use shards the prefetch actually staged The prefetch skips the base repo's transformer/ shards whenever a prequant checkpoint is expected, since that checkpoint replaces them. But a prequant fetch can fail for reasons the planner cannot see: an unpublished, gated or renamed artifact, a hub 5xx, a proxy, a checkpoint the validator rejects. The loader then fell through to from_pretrained(subfolder = 'transformer') and pulled those shards inside the load lock during 'finalizing', after the previous pipeline was already evicted, where the cancel event has no reach, load_progress has already reported bytes_downloaded == bytes_total, and the cache-disk gate had only reserved the small prequant checkpoint. That is verbatim the situation _dense_quant_prefetch_needed's own docstring exists to prevent. Gate the in-loader dense fallback on the shards being staged, read off the returned file list rather than the request so a failed size estimate closes it too, and let the GGUF build take over otherwise, which is what the prequant-sized replan already does one branch over. It is also the invariant the text-encoder path already enforces: only a component that really resolves may have its dense weights dropped from a plan or a prefetch. Adds the missing coverage for the gate's prequant arm, which the existing disk-gate tests never reached. * Refuse a training output_dir that resolves to the outputs root resolve_output_dir strips a leading 'outputs' and drops '' / '.' segments, so '.', './', './.', 'outputs', 'outputs/outputs' and ' . ' all clean away to nothing and land on the outputs root itself rather than a run directory under it. The DiT trainer then writes pytorch_lora_weights.safetensors flat into the root, where scan_trained_models and scan_checkpoints cannot see it (both filter is_dir()), and a second such run overwrites the first. The UI only checks the field is non-empty, so 'outputs' is one plausible run name away. Refuse it with a 400 that says what to do instead. cond_cache_dir collapses the same way but has an honest 'off' to fall back to, so a root-resolving value now means the in-memory cache, which is what the comment above it already promised: otherwise a run drops one flat safetensors per cached latent and caption into the directory trained models live in. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Repair three defects the main merge left behind Git merged all three files without a conflict, but the result was wrong in each case. The tests only started failing once the merged tree was run. routes/inference.py duplicated the GGUF load block: this branch had moved the gguf_load_in_flight marker and the _hub_download_blocks_gguf_load guard under "if config.is_gguf and config.gguf_hf_repo", and the conflict resolution re-added main's copy at the old position, so both ran. Dropped main's copy; the earlier placement is the deliberate one, so a 409 from the hub guard cannot tear down a resident Images or Video pipeline. test_gpu_selection.py still called _hf_offline_if_dns_dead, which main renamed to _hf_offline_if_unreachable_for (#7591). Disjoint edits, so no conflict, but four route-error tests referenced a function that no longer exists. test_gguf_load_cache_reuse.py anchored its ordering assertion with rindex over "if config.is_gguf:", taking the last one before the load marker. That only held while _resolve_inherited_extra_args sat above every such line; main has since hoisted it above the gpu_ids preflight, so the anchor landed between the call and the marker and the assertion compared against an unrelated later call site. The ordering it checks is a property of _load_model_impl as a whole, so it now anchors on the function. The ordering itself is intact: _resolve_inherited_extra_args, then the gguf_load_in_flight marker, then the hub guard, then the chat handoff, then unload_model. * Stop a real sd.cpp install from breaking its own discovery tests The five "nothing is installed" assertions in test_sd_cpp_engine.py cleared SD_CLI_PATH and UNSLOTH_SD_CPP_PATH and patched Path.home, which covers hops 1, 2 and the fallback half of hop 3. It leaves two hops live. Hop 3 goes through managed_install_root(), which honors UNSLOTH_STUDIO_HOME and STUDIO_HOME and resolves to the stable-diffusion.cpp directory beside the studio home. That is the documented way to run side-by-side Studios, so anyone who has one set gets a real sd-cli back from a finder the test expects to return None. Hop 4 is the in-tree developer build, which does the same for anyone who built sd.cpp inside the checkout. Isolated with an autouse fixture rather than another helper call, because reaching the failure needs no fixture: SdCppEngine(binary = None) runs the finder from its constructor, which is why test_generate_raises_when_binary_missing failed too. The fixture points the studio home and the in-tree root at an empty tmp tree, so every hop is answered by the test rather than by the box. The in-tree root moves into a named in_tree_install_root() so it can be pointed somewhere empty; behaviour is unchanged, including the OSError and IndexError guard on an unexpected layout. * Make the sd.cpp uninstall test actually extract the code it tests The two sed ranges anchored the production fragments at column 0, but both blocks sit inside the main removal function and are indented, so each range matched nothing and LOOP_FILE / DEFAULT_FILE came out empty. Sourcing an empty file is a no-op, so the suite reported 4 passed / 7 failed: the seven removal assertions failed because nothing ran, and the four that passed were 'kept' assertions that pass trivially when nothing runs. The shell job auto-discovers tests/sh/test_*.sh and runs each under set -e, and this file is not in its skip list, so it was a deterministic red. Anchor on optional leading whitespace and fail loudly on an empty or _remove_path-less fragment, so a future reshuffle of uninstall.sh cannot make the suite vacuous again. Now 11 passed, 0 failed, against the real removal loop and the real default-mode block. * Agree on what an engine can build, and on what the installer owns Two gates, each half-applied. The unbuildable-family gate had one caller, the image branch of the cached repo picker. The GGUF classifier, the local-model classifier and the video branch had none, so on a diffusers too old for a family the picker still offered its GGUF and the load then failed. Meanwhile the loader asserted the diffusers pipeline class before engine selection, so a GGUF this host routes to native sd.cpp, which instantiates no pipeline class at all, was refused with an upgrade instruction that could not help. Both are wrong one-sidedly: hide a family only when NEITHER engine can build it, and demand the diffusers class only when diffusers is what will load it. One predicate, family_buildable_here, now answers both, so the picker and the loader cannot disagree. The population is real on Python 3.9, whose diffusers ceiling is 0.36: Flux2KleinPipeline, Krea2Pipeline and LTX2Pipeline are all absent there, and FLUX.2-klein GGUF is a repo in this PR's title. That assertion also raised RuntimeError, which /images/load maps to 409, the status that otherwise means a load is already in progress, and which escaped /images/download-plan (it catches ValueError and FileNotFoundError) as a bare 500 with the message lost. It is an unloadable pick like every other, so it raises ValueError and both routes answer 400 with the text intact. The managed-binary repair used a path test for ownership while the installer uses a marker. On a managed root without the marker, which is any install predating it, the repair deleted sd-server and the reinstall was then refused, permanently, because the surviving sd-cli keeps the directory non-empty so the marker can never be claimed: the user went from an unrunnable binary to no binary and no way back. Require the marker before discarding, which is the same definition of ours that uninstall.sh already uses to keep a user's own stable-diffusion.cpp checkout. A genuinely interrupted extraction still self-heals, since install() writes the marker before it extracts. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add gguf and av to the studio extra so the wheel install matches studio.txt This branch added both to studio/backend/requirements/studio.txt, which is what install.sh uses, but never to the studio extra in pyproject.toml, which is what 'pip install unsloth[studio]' uses. Nothing else keeps the two in sync, so tests/studio/install/test_studio_extra_matches_requirements.py failed deterministically, and the CI job runs the whole tests/ tree. The drift is not only a red test: gguf backs diffusers' GGUFQuantizationConfig and av does the MP4 encode and audio mux, so a wheel install got a Studio that cannot read a GGUF or export a video, which is most of what this PR adds. * Fix two test-suite defects: the CPU-only patch gate and a popped module The patch backend reaches unsloth_zoo's helpers only through an ~940 MB 'import unsloth', which a CPU-only host cannot complete, so every patch install returned False and nine arch/eager/compile tests failed there -- exactly the runners the retry was narrowed to protect. unsloth_zoo only wants UNSLOTH_IS_PRESENT in the environment, which costs nothing and needs no accelerator, and the conftest already sets its sibling UNSLOTH_ALLOW_CPU. Set it there too, as run.py and main.py already do at module scope, so no real server ever took the expensive route. CPU-only goes from 9 failed to 24 passed, 1 skipped; the GPU run is unchanged at 29 passed. test_setup_cache_env_hf_home popped utils.hf_cache_settings to model a fresh process and never restored it, so a later import built a second module object and rebound it on the utils package. test_hf_cache_settings then wrote its setting into one object while core.inference.diffusion read the other, the same split-module failure the xet shim already had. Restore both bindings in teardown: 21 pass together, and each file still passes alone. * Restore the setup_fail assertions the main merge reverted A fourth instance of the class 9541cc535 fixed: the merge took main's studio/setup.sh, which #7644 changed to abort through the setup_fail helper so desktop mode still emits [TAURI:ERROR], but kept this branch's older copy of the test, which still asserted the literal 'exit 1'. setup.sh is byte-identical to main here and the only diff in this file was the reverted assertions, so take main's version. 60 passed. * Fence the video teardown, reject non-finite clipping knobs, stop a test installing sd.cpp The video backend never had the teardown fence the image backend documents. Its unload and its superseding-load path both signalled the active generation, then did 'with self._generate_lock: pass' and tore the state down with the lock free. A generation queued behind that barrier holds no cancel event yet, so the signal cannot reach it, and Python locks are not FIFO: it won the lock the instant the barrier released it, read a still-loaded state and denoised a whole clip against the pipeline being freed. Reproduced on both paths, where the queued generation returned a finished MP4. Mirror the image backend: count waiters, refuse a generation while one is pending, and tear down inside the barrier with the counter released in a finally. _teardown_state becomes _teardown_state_locked since the caller now holds both locks. max_grad_norm and snr_gamma were bounded on one side only, so 1e309 floated to inf and passed. clip_grad_norm_ then computes an infinite clip coefficient, clamps it to 1.0 and scales nothing, and min(snr, inf) / snr makes every min-SNR weight 1.0. The run starts, reports normal progress and trains with the requested knob silently disabled. Measured both. NaN already failed the bounds; allow_inf_nan makes that explicit. learning_rate and flow_shift already guard this exact vector. test_load_routes_to_sd_cpp_on_cpu stubbed ensure_sd_cpp_binary but not ensure_sd_server_binary, which select_and_activate_engine probes first with installs enabled, so the unit test downloaded and unpacked 108 MB into the developer's real ~/.unsloth when UNSLOTH_STUDIO_HOME was unset. Now 0 bytes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Key the video schedule reset on the resolved defaults, not the repo id A GGUF repo holds several variants, so another client swapping a distilled build for the base one from the same repo changes the steps and guidance the backend reports while repo_id stays put. modelChanged stayed false, the page kept the previous schedule, and later generations ran an 8-step distilled setting on a model that expects roughly 40 steps and CFG 4. Include the reported defaults in the key. They only move when the resident artifact does, so a manual steps change still survives a status poll. * Retire the release-lag pin now that 2026.7.6 carries the relaxed gates The virgin Windows container row that installs from PyPI on purpose was tolerated with continue-on-error, because a Server Core container has no Microsoft Store and so no winget, and the released studio/setup.ps1 hard-stopped on a winget-only git gate and reached for winget again for the VC++ runtime. #7549 relaxed both, but it landed on 2026-07-28 and the newest wheel was 2026.7.5 from the 23rd, so the row could not get past it. The pin carried its own tripwire for exactly this moment, and it has fired: unsloth 2026.7.6 shipped 2026-07-29 and the released wheel now installs end to end, so the step reported success and the assertion errored with "delete this pin and drop continue-on-error from the Install step so this row gates". Doing that. Both rows now gate unconditionally, and the assertion that enumerated the accepted failure signatures goes with the pin it protected, since there is no longer an accepted failure for it to describe. The other three release-lag mentions in this file are descriptive rather than tolerated-failure pins: they explain why the fedora and ubuntu-nonroot legs run with overlay: true, and both already refuse a triton failure as a regression rather than accepting it as lag. Left alone. * Pin the hosted pre-quant onto the plain-torch fp8 kernel A Z-Image GGUF pick at the DEFAULT speed mode, once the hosted fp8 pre-quant is actually reachable, dies at generate with an HTTP 500: torch._dynamo.exc.Unsupported: Operator does not support running with fake tensors. Developer debug context: unsupported operator: mslk.f8f8bf16_rowwise.default _fp8_config already pins KernelPreference.TORCH when it BUILDS a config, precisely because the default AUTO switches to the MSLK kernel wherever an mslk package is importable (sm90+). A hosted checkpoint escapes that pin completely: the preference is serialized per Float8Tensor, and all 239 weights in the published Z-Image-Turbo-FP8 checkpoint carry AUTO. Loading it re-arms the very kernel the pin exists to avoid, mslk.f8f8bf16_rowwise has no fake impl, and the first compiled generate therefore cannot be traced. Isolated away from the product to be sure of the mechanism: quantise one Linear three ways on this box and compile each. KernelPreference.TORCH eager ok, compiled ok KernelPreference.AUTO eager ok, compiled FAILS on mslk.f8f8bf16_rowwise library default eager ok, compiled FAILS the same way So the pin is correct and necessary, and the only gap is that the hosted path never got it. _validate_checkpoint checks scheme, granularity, base model, min_features, exclude tokens and fast_accum, but not this. Rewriting the preference on load is safe: it selects a matmul kernel, it is not weight data, so the tensors stay bit-identical and the checkpoint's own state_dict_sha256 still describes them. It is also the faster path compiled, since the opaque extern call blocks inductor quantize fusion. Why this went unnoticed: the pre-quant repos are private, so nothing that could not read them ever took this path. It becomes the default the moment they are readable. Verified end to end at 1024x1024, 8 steps, speed_mode default, on the published checkpoint: before HTTP 500 at generate, diffusion.generate_failed after pinned 239 weights to the plain-torch fp8 kernel, load 10.0 s, transformer_quant fp8 with compiled engaged, cold 7.9 s, warm 0.87 0.87 0.87 0.86 0.87 s That warm number also beats the 1.4 s recorded for this shape on 2026-07-26. 159 prequant and transformer-quant tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cover the fp8 kernel pin so the 500 cannot come back quietly The hosted pre-quant re-arming MSLK was only visible as an HTTP 500 on a compiled generate, which needs a GPU, a readable private repo and an importable mslk to reproduce. None of that is available in CI, so the pin would rot unnoticed. Three hermetic cases on _pin_kernel_preference instead: AUTO weights are rewritten and already-TORCH ones are left alone (counting only what changed), a weight that refuses the assignment does not sink the whole load, and with no torchao enum available the checkpoint is left exactly as saved. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: tighten comments across the image generation, video and training code Rewrite the comments added by this branch to be shorter and clearer: collapse multi-line explanations into one or two lines, drop comments that only restate the code, and keep the rationale that explains why a choice was made. No code changes. * Studio: let the pipeline-class guard survive a host with no diffusers Backend CI installs the CPU-only dependency set, which has no diffusers, and runs without a GPU. 13 tests failed there for three separate reasons, one of which is a real product bug rather than a test-environment gap. assert_pipeline_class_available did a bare "import diffusers", so on any host without it the call raised ModuleNotFoundError instead of the ValueError its own contract promises. /images/download-plan catches only ValueError and FileNotFoundError, so that escaped as a bare 500 with the message lost, which is the exact failure the guard exists to prevent. It is reachable in production: the native sd.cpp engine serves GGUF picks on a CPU or Apple host where diffusers is never installed. Absent diffusers now returns, since the check answers "is the installed diffusers new enough for this family" and there is no version to judge; a pick that genuinely needs diffusers still fails in the loader with its own message. The rest were tests asserting through gates unrelated to what they cover: - Three cond_cache_dir route tests drive a DiT family, which the start route refuses without an accelerator. They now take the existing dit_train_host fixture, written for this case, so they keep testing the schema on every host. - test_download_plan_forwards_the_load_time_controls stubs the diffusers planner but sends a GGUF pick, which routes to the native planner on a GPU-less host. It now pins the engine, so it tests kwarg forwarding rather than the hardware. Engine selection keeps its own tests. - The pipeline-class guard test needed a real diffusers only for its sweep over every shipped family. That half is split out and skips on its own; the stub-driven refusal needs no diffusers and still runs. Added a test for the absent-diffusers contract above. 409 pass on a GPU host with diffusers; 301 pass and 1 skips with diffusers blocked and no GPU. * Studio: cut the image generation comments down further Second pass over the comments this branch adds. Collapse the multi-line blocks that still read as paragraphs, rewrite the longest one-liners so they say the same thing in fewer words, and drop a stale comment that had drifted away from the constant it described. No code changes: only comments and whitespace. * Studio: workflow rail, sidebar Video row, and Train settings polish Sidebar - Pin Video under Images by default; persist store bumped to v3 so an untouched install adopts it while a custom arrangement is left alone. - Add "Customize sidebar" to the end of the More flyout, opening Settings > Appearance scrolled to the sidebar nav section. Images - Replace the workflow dropdown with an icon rail down the left edge: all seven workflows visible, keyboard nav, tooltips carry the labels. - Workflows stay selectable with no model loaded, so one can be set up before picking a model. Generate is already gated on a loaded model. - Video moves out of the Create/Train strip into a link at the far right, with a matching Images link on the Video page. - Wider gutters around the settings column, headings matched to Train. Train - Field guidance moves into "i" tooltips; only state that limits a control stays on the page. Steps, LoRA rank and the rest gain hints. - /info reports params, qlora_vram_gb, gated and note as fields, so the family note renders as chips. vram_note is rebuilt from them unchanged for older clients. Shared - Number steppers appear on hover or focus. - Lighter control border (#e9e9e9) on Images and Video, light mode only. * Studio: move Advanced inline on Images and Video - Replace the top-bar toggle and right-docked Advanced panel with a disclosure under Seed, so load-time tuning sits with the settings it affects and opening it no longer shrinks the preview canvas. - Video uses the bordered variant, since offload and memory decide whether a model fits at all. - Open state persists per page in localStorage. - Tighten the Steps unit trigger: 14px chevron, less right padding. * Studio: full width media pages and layout polish - Drop the 1100px cap on Images Create, Images Train and Video, so the preview canvas grows with the window instead of sitting in a band. - Align the workflow rail to the model selector label, widen the gap before its divider, and restore the divider itself. - Advanced: more room above and below, icon and label sized to the slider rows, and the same quiet row on Video as on Images. - Give Video a pane heading and description, matching Images. - More space above Seed on both pages. - More no longer takes the active style when the current page is one of its own rows. * Studio: fix the Extend side toggles The resting outline used a ring, and index.css blanks the ring on the button holding mouse focus, so the side you just clicked lost its border until focus moved. Use a border for the outline and leave the ring to focus-visible. Also separates the two states properly (the off state had no surface of its own), drops the border in dark mode as the inputs do, and adds the aria-pressed and focus-visible styling the buttons were missing. * Studio: drop the border on the Extend side toggles Light mode should carry no border either, so the fill alone marks the state: a muted surface when off, a primary tint when on. Matches the borderless treatment dark mode already had. * Studio: move the Images workflows into the sidebar The workflow switcher was a vertical icon rail on the page. It now lives in the sidebar under Images, so the page keeps that width for the canvas. - Workflows list under the Images row, with a chevron to fold them away. Hovering the row peeks them in a flyout on the standard menu surface. - Clicking Images while already there toggles the list instead of navigating. - The listed workflow carries the highlight, so the Images row drops it. - Create takes its own icon: it was sharing the New chat pencil. - Model hub moves above Projects. Persisted layouts bump to v4 and only adopt the new order where the stored one is still a shipped default. - Images and Video content both start at 32px, clear of the sidebar. * Studio: simplify the Images workflow submenu Dropping the hover flyout: two ways to reach the same seven workflows, one of them overlapping the list right below it, read as clutter. - The flyout is gone. The workflows are rows under Images and nothing else. - On the Images page they are always open, since they are that page's switcher. Elsewhere they stay folded, and the row's chevron opens them. - Create takes the sparkles icon. - Create and Train both pad their settings column to 40px a side, so the two tabs line up with each other and with the model selector. * Studio: dock the generate action and tidy the Create controls The primary action sat at the foot of a long scroll, so it was off screen until you scrolled for it. - Generate, Video's Generate and Train's Start training float at the bottom of their settings column. No bar behind them: hover lightens the fill rather than thinning it, and the disabled state is opaque, so the controls underneath never show through. - Aspect ratios read as names: Square (1:1), Widescreen (16:9) and so on. - Width and height replace their sliders with two compact boxes. Type a size or pick one from the menu; the value still snaps into range and still drives the locked ratio. - Negative prompt gets a quiet disclosure under the prompt. It was there before but only above guidance 0, which is not the default, so it never showed. - The info "i" is smaller across the settings UI. * Studio: reveal Images and Video field hints on hover The "i" next to every field label sat there permanently, which made a column of settings read as busier than it is. Scoped to the diffusion pages by CSS rather than per component: the hints come from the page's own Field and SelectRow, the train panel's FieldLabel, and the chat ParamSlider, which is shared and should not change for chat. Reveals when the pointer is inside the field, on focus-within for keyboard, and stays inert while hidden. InfoHint grows a data-slot so the rule has a stable hook, rather than keying off its aria-label. * Studio: one-line sliders, edge fades, shared negative prompt - Steps, Guidance and the rest put label, track and value on one row. ParamSlider is shared with chat, so this is an opt-in prop and chat keeps the stacked layout. - The settings columns fade at whichever edge they run past, as the sidebar and model picker do, instead of cutting off. New useScrollFades hook drives all three. - Negative prompt moves to a shared component and Video picks it up, so both pages collapse it the same way. - Right gutter sits closer to the rule, Steps takes a bigger break above it, and the Advanced rule sits between its neighbours rather than up against the field above. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: add the API monitor to the sidebar nav Sits under More, below Export. Points at the monitor page, not the API keys dialog the profile menu opens. No store version bump: an id missing from a stored layout is appended with its default pinned state, so existing sidebars gain the row at the end and keep their own order. * Studio: take main's copy of the sidebar spinner test The merge kept this branch's older version, which pins the Recents row to h-[33px]. Main already relaxed that to any height, since row density moves independently of the trailing column the test is about. * Studio: fix FLUX.2 klein size resolution, HunyuanVideo tier routing, and the personalization save Five fixes found while auditing the model registries against the live Hub. FLUX.2 klein 9B loaded against the 4B config. One family covers both klein sizes and defaults to 4B, relying on the base_model card tag for the real base, but the trust allowlist only had klein-4B, so the correct tag was discarded and a 9B GGUF (inner_dim 4096) was loaded against a 4B config (3072). That surfaced as a bare shape mismatch from inside the GGUF quantizer naming neither the file nor the repo. Adds the missing allowlist entries and a header-level size check that fails early with a legible message; the check is fail-open, so an unreadable file, a non-FLUX.2 family, or an unmapped base leaves the load exactly as it was. The sd.cpp text encoder rule matched the literal "klein-9b", so klein-base-9B was handed the 4B encoder. HunyuanVideo-1.5 720p checkpoints routed to the 480p family. Only the literal 720p_t2v path was aliased, so 720p_i2v and every GGUF repack fell through to the generic token and inherited the 480p base repo, which also supplies the VAE and text encoder. The tier is baked into the weights (transformer target_size 640 vs 960, scheduler shift 5.0 vs 9.0) and the two bucket lists are disjoint, so this ran the whole pipeline off-tier. Resolution presets that the checkpoints were never trained for. Wan2.2 TI2V-5B is 720P-only upstream (SUPPORTED_SIZES is exactly 704x1280 and 1280x704, asserted in generate.py), and the HunyuanVideo 480p square preset 624x624 is not a bucket of its tier; 640x640 is. Personalization could never be saved. The frontend ships eight sidebar nav ids and refills every missing one on each save, but the backend Literal had seven and no "api", so every PUT to /api/settings/personalization returned 422. Aligns the server defaults and the Literal with the shipped layout. A catalog row pointing at a file that does not exist. unsloth/Qwen-Image-2512-FP8 holds torch prequant .pt checkpoints, not qwen-image-2512-fp8.safetensors, and fp8 is denied for this family anyway because it renders black, yet the row was the auto-route target on any GPU above 40 GB. Also marks FLUX.1-schnell, Krea-2-Turbo and the two Ideogram repos gated: all four are gated on the Hub today, and schnell being Apache-2.0 does not make it ungated. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: stop a case-only duplicate upload destroying an image, and size the load against the right GPU Two fixes from the review round. A dataset upload containing two names that differ only by case silently loses one on Windows and macOS. The in-batch duplicate check is an exact string compare, so "Cat.png" and "cat.png" both pass, but on a case-folding filesystem they are one destination: the commit step moves the first staged part aside, writes the second over it, then deletes the backup, while the response still reports both as uploaded. Captions collapse the same way and never even reach the image branch. No single folder can hold such a pair, but the Windows and macOS open dialogs flatten search and Recents results across folders into one multi-selectable list, which is a normal way a LoRA dataset gets assembled. Rejecting the pair everywhere would regress Linux, where the two really are different files with their own sidecars, so the check probes the filesystem once per process instead of keying off sys.platform: macOS also ships case-sensitive APFS volumes, and a Linux host can keep its Studio home on an exFAT or NTFS mount. A failed probe answers "case-sensitive", which leaves today's behaviour untouched. The quantization fit budget was sized against the wrong card under a reordering CUDA_VISIBLE_DEVICES. A bare "cuda" load lands on visible ordinal 0, but the reducer took the lowest PHYSICAL index, which the nvidia-smi path reports as index_kind "physical". Under "3,1" that sizes against GPU 1 while the pipeline loads onto GPU 3. The hook's own interface doc already promised the lowest visible ordinal. Ranks by visible_ordinal, falling back to index only for an older backend that omits it. * Studio: rework the diffusion dataset panel and align the pane gutters Upload: - one Upload button beside the dataset name, with an upload icon - picking files uploads them, no second confirm click - Add button beside the Training images dropdown, so images can go into a set that already exists - fall back to the upload form when a selected dataset no longer resolves Labeling grid: - two columns at any width; sm:grid-cols-3 crushed the tiles in a fixed width column - pin the caption size so the Textarea's md:text-sm does not outsize a tile - name the tile hover group; a bare one revealed every tile's Remove at once - Remove is an icon button on the image, not a word over the artwork - drop the amber tile fill, keep the No caption label - match the header, toggle and status text to the section's type scale Gutters: - the run area and the preview canvas now sit 40px off the rule, the gutter the settings column has off the page edge * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: give the dataset preview per-image hover and remove, and wrap it onto rows - each thumbnail is its own button, so it can carry an X that removes just that image; the strip was one button and could not nest a second control - hover lightens the image instead of drawing a border - wraps onto more rows rather than scrolling sideways, which also keeps the bottom corners rounded - samples 12 thumbnails, up from 8, now that a second row is available - a delete refreshes the panel's dataset counts * Studio: show the workflow icon beside the Images and Video headings Images reads it off WORKFLOW_TABS, so it is the same icon the sidebar submenu shows and it follows the active workflow. Video's heading is also Create, so it takes the Create icon rather than the nav row's film slate. * Studio: reject a duplicate LoRA id on the diffusion load path too DiffusionGenerateRequest already refuses a repeated adapter id, with a comment naming the hazard: _resolve_lora_set suffixes colliding adapter names, so the same id resolves the SAME adapter twice and set_adapters stacks both copies past the per-adapter weight bound. DiffusionLoadRequest bounds only the list length, and it matters more there. Generation-time stacking spoils one image; on the load path the adapters are baked into the quantized build before compilation, so the unintended combination rides every image until a reload. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: teach the GGUF reuse test double about holds_no_vram The resident-GGUF fast path consults llama_backend.holds_no_vram before asserting CHAT ownership, but two tests hand it a SimpleNamespace built before that attribute existed, so they raised AttributeError and returned a 500. A real LlamaCppBackend exposes it as a property; the doubles now carry it too. Product code is unchanged: these pass on main and were the branch breaking its own test, not a defect in the guard. --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: michaelhan <michaelhan2050@gmail.com> Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: Michael Han <107991372+shimmyshimmer@users.noreply.github.com> | 1 个月前 | |
Reduce and tighten code comments and docstrings repo-wide (#6095) Trim and tighten code comments and docstrings across the repository. Comment-only: every changed file verified code-identical to main via AST/token comparison. | 2 个月前 | |
Replace standalone Studio wording with Unsloth (#7221) * Replace standalone Studio wording with Unsloth Replace the single word Studio with Unsloth wherever it is used as shorthand for Unsloth Studio in docs, CLI output, UI strings, i18n locales, workflow display names, comments and docstrings. Kept unchanged: the full name Unsloth Studio, third party product names (LM Studio, Visual Studio, Mac Studio), feature names (Recipe Studio, Fine-tuning Studio and its translations), and all identifiers such as env vars, commands, paths and filenames. * Address review feedback on the Studio wording rename Use "an" before Unsloth where the rename left the article as "a". Restore the split brand where Unsloth and Studio render as two halves of the full product name: the onboarding sidebar subtitle and the IPv6 localhost warning. Scope two messages to the full name Unsloth Studio where plain Unsloth was misleading: the AMD README bullet and the CLI studio setup error. | 1 个月前 | |
scan_packages: key baseline on matched-code hash so payloads in baselined files are not auto-suppressed (#6552) * scan_packages: key baseline on matched-code hash The baseline matched on (package, package-relative file, check), which excluded the matched code, so a future finding of the same check in the same file was suppressed regardless of what the code did. A malicious future version of an already-baselined package could place a payload in the same file under the same check and pass the enforcing gate. Key the baseline on a hash of the matched code too. The hash is over the deduped, sorted set of matched spans with L<NN>: line markers stripped, so version bumps, line shifts and match reordering stay stable while new or changed flagged code reopens the finding. Version is left out of the key so routine dependency bumps do not reopen every entry. The hash is capped and recomputable from the stored evidence. Regenerate scan_packages_baseline.json against the current dependency set; the hf-stack, studio and extras scan shards pass enforcing (no active CRITICAL or HIGH). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * scan_packages: refresh baseline for newer unsloth-zoo release A newer unsloth-zoo published after the first regenerate added tests/test_mlx_save_export_regressions.py, a benign test fixture (temporary_location="/tmp/ignored") that trips the /tmp dropper check. Regenerate the hf-stack shard against the current set so the entry is allowlisted; studio and extras are unchanged. * scan_packages: harden baseline loading against malformed JSON Guard against a non-dict top-level baseline and non-dict entries so a corrupt or hand-edited allowlist warns and fails closed instead of crashing with AttributeError, and treat an explicit evidence: null as empty. * scan_packages: hash the full match set, keep indentation, strip only the marker Address the evidence-hash review feedback: - Capture every matching line, not the first three, so a payload appended after existing matches in a baselined file and check reopens the finding instead of riding the sample. - Preserve leading indentation so a flagged line moved out of a guarded block reads as changed. - Strip only each span's prefix up to the first L<NN>: marker, so an L<NN>: inside the matched code is kept and a change to it reopens the finding. Evidence and its hash are stored in full and stay recomputable from the stored field. Regenerate the baseline; hf-stack, studio and extras pass enforcing with no active CRITICAL or HIGH. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * scan_packages: bind baseline evidence to full matched code Address review feedback on the evidence-hash baseline key: - Split evidence only on real span delimiters (" | " before an L<NN>: marker, or a newline), so a bitwise-or or union type in matched code is no longer split apart into separate spans. - Record matched lines in full (drop the 160-char per-line cap) and record every distinct multiline match, so code appended past the cap or a second cross-line match reopens the finding instead of riding the first one. - Give the large-JS-bundle and .pth base64-blob findings a content digest instead of empty or prefix-only evidence, and record all .pth import lines, so a changed bundle, blob or import no longer inherits a baselined empty or truncated key. - Warn when a loaded baseline has entries without evidence_hash so a legacy baseline is regenerated rather than silently degraded. Regenerate scripts/scan_packages_baseline.json against the current dep set and add regression tests for each case. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * scan_packages: harden multiline and duplicate evidence handling Follow-up hardening so the evidence hash tracks the full matched code: - For DOTALL patterns that match across lines, record every line the match spans (not just the start line), so a change on a continuation line (the URL inside a baselined C2 loop, a swapped credential path) reopens the finding. A pathological greedy span is bounded to its head line plus a digest of the rest. - Keep duplicate spans in the canonical evidence so a second identical matched line in a new code path changes the key instead of deduping away. - Anchor the evidence prefix to strip only a genuine leading label or line-number marker, leaving a marker-like "L<NN>:" inside raw .pth code intact. - Make the legacy-baseline warning explicit that entries without an evidence_hash reopen rather than suppress under a coarse key. Regenerate scripts/scan_packages_baseline.json (same finding set; entries for same-file repeated checks are now tracked separately) and add tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * scan_packages: bind every combo and large finding to its full content Close the remaining asymmetric-evidence gaps so a changed payload cannot ride a reviewed baseline entry: - Digest a capped multiline span from the code without line markers, so a pure line shift stays stable while a continuation-line change reopens. - Give the "Unusually large executable .pth" finding a content digest instead of keying on byte size and import-line count alone. - Record both contributing signals for the JS credential+network stealer, the shell credential+network and persistence-hook combos, and the hidden network+exec docstring payload, so changing the network/exec side reopens. - Allow punctuation in an evidence label prefix so a "network+exec:" label is stripped and line shifts do not change the key. Regenerate scripts/scan_packages_baseline.json and add tests for each case. * scan_packages: bind remaining Python combos; key npm baseline on evidence Python scanner: the openssl+key, anti-analysis, DNS-exfil and base64+exec+blob combos recorded only one contributing signal, so a changed payload on the other side could ride a reviewed baseline entry. Each now binds every co-occurring signal (and the blob is digested, since it can sit on a separate line from the decode call). npm scanner: scan_npm_packages.py keyed its allowlist on (package, path, pattern) only, the same coarse-key bypass the Python scanner just closed. Add an evidence hash to the key (schema v3, fail-closed on older baselines) and store full evidence. The committed baseline stays empty by design. Regenerate scripts/scan_packages_baseline.json and add tests for each case. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * scan_npm_packages: bind full blob evidence and harden baseline loader Follow-up on the npm evidence-hash key: - _evidence now records every match and, when a snippet is truncated for display, appends a digest of the full match. The obfuscated-blob key was hashing only the truncated first-match snippet, so a changed payload tail or an appended blob in the same package/file/pattern could ride a reviewed entry. - _load_baseline guards that the root is an object, entries is a list, and each entry is a dict before reading it, so a malformed baseline warns and fails closed instead of raising AttributeError. Add tests for a changed blob tail reopening the key and for malformed entries. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * scan_packages: symmetric baseline-loader guards; bind npm outbound host context - Python _load_baseline now rejects a non-list "entries" with a warning instead of raising TypeError, matching the npm loader. - npm cred-surface-host (outbound) records the host with its URL path / fetch call / host config, so a changed outbound path, headers or body reopens the key rather than riding the bare host literal. Add tests for both. * scan_npm_packages: migrate v2 baselines and bind host-config outbound context - _load_baseline now migrates schema v2 entries by recomputing the evidence hash from stored evidence (with a legacy warning), matching the Python loader, instead of discarding them; only pre-v2 basename schemas are rejected. - The cred-surface-host (outbound) host-config branch now captures the whole line (path, headers, body), so a changed outbound payload on the same hostname line reopens the key instead of riding the bare host snippet. Add tests for v2 migration and the host-config context binding. * scan packages: bind PEM key bodies and npm windowed evidence to baseline keys scan_packages: embedded-key findings now pin the full PEM block (BEGIN..END) via a content digest, so a key body swapped under the same marker reopens the finding instead of riding the unchanged BEGIN line. Single-line and DER keys were already bound by their full matched line; marker-only references with no END block (validation header lists) are unaffected, so the committed baseline is unchanged. scan_npm_packages: _evidence now digests the full containing line whenever the shown snippet is only a window into it (short match on a long line, or a truncated payload), so a changed payload tail outside the display window reopens the key. The npm baseline is empty, so this changes no suppressions. Adds regression tests for both cases. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * scan packages: bind multi-line evidence and every blob to baseline keys _extract_evidence now extends each single-line match over its bracket continuations, so a multi-line call binds its argument lines and a changed URL or body on a continuation line reopens the key. After the per-line pass it also records cross-line matches the scan cannot otherwise see (a DOTALL regex, or a multi-line construct appended under a check that already had a one-line match), so an appended multiline payload reopens instead of riding the key. _blob_digest hashes every large base64 blob (not just the first) for the base64+exec finding and the .pth large-blob finding, so an appended or swapped second encoded payload reopens; single-blob files keep the same digest. scan_npm_packages _evidence digests the full logical line (the matched line plus its bracket-continuation lines), so a multi-line fetch's option and header lines bind and a changed payload on a following line reopens the outbound key. Regenerated the Python baseline: same package/file/check set, 24 entries pick up the wider multi-line evidence. Adds regression tests for each case. * scan packages: stop giant greedy spans from binding a whole-file digest When a greedy DOTALL pattern (reverse shell socket...subprocess, C2 loop) has its anchor tokens far apart, the match span covers the whole file. Digesting that span bound thousands of unrelated lines, so the evidence hash drifted on any edit between the anchors (a dependency bump reshuffling the file), which made a baselined finding reopen on an upstream release. The multiline pass now skips an oversized span when the per-line pass already bound the signal lines, so the evidence is the stable matched lines; a genuinely appended multi-line construct stays under the cap and is still recorded. Regenerated the Python baseline against Python 3.12 (the version the scan CI shards run) so the resolved dependency set matches CI. Same package/file/check set. Adds a regression test. * scan packages: tighten evidence binding (order, string brackets, span size) Address review follow-ups on the evidence extraction: - _canon_evidence keeps discovery (line) order instead of sorting. Line-shift stability already comes from stripping the L<NN>: markers, so order stays significant and reordering matched lines (a multi-line call's arguments) reopens the finding. - _logical_line_end (Python) and _logical_line_text (npm) blank string literals before counting brackets, so a ) inside a string argument does not close the logical line early and drop later argument lines. - The oversized-span skip now only drops a giant whole-file bridge (over 60 lines); a genuinely appended multi-line construct is recorded so its payload reopens, rather than riding an existing one-line match. - npm _logical_line_text binds the enclosing bracket group, so a host-config object whose { is on a prior line binds its path/headers/body lines. Regenerated the Python baseline (Python 3.12, matching the scan CI shards): same package/file/check set. Adds regression tests for each. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * scan npm packages: normalize and bound the logical-line digest - _evidence whitespace-normalizes the logical line before digesting (matching _evidence_hash), so a formatter-only reindent of the bound continuation lines does not change the sha256 suffix and reopen an unchanged finding. - _logical_line_text follows a bracket group to its close up to a hard 200-line cap (digest input only), so a config object longer than the backward window still binds its whole tail instead of silently truncating. Adds regression tests. npm baseline is empty, so no regeneration is needed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * scan: cap single-line evidence and widen npm opener window Cap each rendered evidence line at 200 chars in scan_packages.py: a long or minified one-line file is shown as a bounded prefix plus a sha256 of the full line, so a packed payload cannot dump unbounded content into the CI logs or baseline while a change past the cutoff still changes the digest and reopens the finding. Mirrors how the npm scanner bounds its snippets. Widen the npm backward opener window (_MAX_CONT_LINES 12 to 200, symmetric with the forward cap) so a host deep inside a large options object binds the whole object, not just its own line; a changed path, header, or body on any property reopens. Regenerate the Python baseline with Python 3.12: only the protobuf nspkg.pth and unsloth-zoo compiler.py evidence change, both from the new line cap; the package/file/check key set is unchanged. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * scan: bind all host contexts, deep call continuations, far-back npm openers Three fail-closed evidence gaps surfaced by review of the previous round. scan_npm_packages.py: measure the forward bracket-group cap from the matched line (idx + _MAX_GROUP_LINES) instead of the opener, so an opener found near the widened backward limit no longer consumes the forward budget and drops the path, headers, or body that follow the host. scan_npm_packages.py: _outbound_host_evidence now records every outbound context form for a host (URL, fetch-context, host-config), claiming each non-overlapping match in form order, so a separate host-config request added beside an already-baselined URL changes the evidence and reopens the key. The common single-context case keeps its existing snippet. scan_packages.py: follow a matched Python call over its continuations up to a separate _MAX_CALL_LINES (40), decoupled from the 12-line display threshold, so a multi-line requests.post( binds its whole argument list in the digest and a changed body deep in the call reopens; bounded so a miscounted bracket cannot swallow unrelated code. No baseline change: the current dependency set has no matched call that closes between 13 and 40 lines, confirmed by a Python 3.12 regenerate that produced a byte-identical baseline. * scan: clamp npm depth, pin large bundles, follow backslash and bound .pth dump Four fail-closed evidence gaps surfaced by review of the previous round. scan_npm_packages.py: clamp the backward opener scan at depth 0 so a leading unmatched closer (a preceding block whose opener is outside the backward window) no longer drives depth negative and masks the real enclosing opener that follows; a host-config object after such a block now binds and a changed path reopens. scan_packages.py: a large JS bundle now pins its whole content even when another JS heuristic already fired. The bundle digest was only added when no other finding existed; it is now appended to every finding's evidence on a large bundle, so an unchanged obfuscation signature no longer lets changed payload elsewhere ride the matched-line key. scan_packages.py: _logical_line_end follows explicit backslash line continuations, so a call split with a backslash before its parenthesis binds the continuation line (URL/body) instead of returning at the zero-depth API line. scan_packages.py: the catch-all .pth import evidence is bounded through _cap_line (prefix plus a digest of every line) so a large .pth of benign imports cannot dump the whole member into the logs or baseline while an appended or swapped import still reopens. Baseline regenerated with Python 3.12: key set unchanged; one entry (unsloth-zoo compiler.py) gains the backslash-continued banner lines now bound by the continuation fix. * scan: handle multi-line strings, lifecycle bodies, and de-quadratic evidence Addresses a review round plus a performance audit of the evidence extractor. Correctness (fail-closed): - Bind the UNION of the single-line-blanked and multi-line-blanked bracket spans in both scanners. The multi-line view blanks a triple-quoted Python string or a backtick template literal that spans lines, so a `)` inside such a string no longer closes the enclosing call early and drop later arguments. The single-line view still counts a payload embedded INSIDE a string, so a dropper that hides a call in a string keeps its argument lines bound. Taking the larger span never shrinks the binding below either view, avoiding a fail-open regression. - cred-env-in-lifecycle now pins the whole lifecycle script body via a digest, so a changed non-token line (e.g. adding a curl exfil beside the token reference) reopens, not just a change on the token line. Performance / DoS (the scanner runs on attacker-controlled package files up to the 64 MiB / 16 MiB member caps, with no per-file time budget): - _extract_evidence precomputes newline offsets once and maps match offsets with bisect, removing the O(matches) whole-file content.count per match that made the finditer fallback quadratic (a crafted minified file went from ~13 s/MiB and hours at the cap to linear). - npm _index_text splits and string-blanks the file once per evidence call instead of per match (was O(matches x file) time and allocation). - Bound evidence output: _MAX_EVIDENCE_SPANS (Python) and _MAX_EVIDENCE_MATCHES (npm) fold the remainder into a digest so a file with thousands of matches cannot build a multi-megabyte evidence/baseline blob while an added/removed match past the cap still changes the key. - _outbound_host_evidence caps matches per form and bounds the overlap claim so a host repeated many times cannot make it quadratic. No baseline change: a Python 3.12 regenerate is byte-identical (the union equals the legacy single-line span for every current dependency file; the cap thresholds sit above the largest real entry), so these are forward-looking hardening with no drift. * scan: count all overflow matches, bind their context, blank JS regex literals Follow-ups on the evidence output caps from the previous commit. - _outbound_host_evidence no longer truncates each pattern's match iterator with islice; it iterates every match and runs the overlap dedup only while the display list is below the cap (so claimed stays bounded and the check is O(cap) per match, not quadratic), folding every match past the cap into the overflow digest. A host context beyond the 64th is counted again, so it reopens. - The overflow digest (both scanners, via a shared _overflow_digest) binds each overflow match's logical-line context, not just the regex match text, so a changed payload on an over-cap line reopens even with the matched token unchanged. - The multi-line JS blanked view now blanks regex-literal bodies (tracking the previous significant char for regex-vs-division and char classes for a literal `/` inside `[...]`), so a `)` inside `/)/` no longer closes an outbound call early. The bound span is the union of the single-line and multi-line views, so an imperfect regex decision only ever grows the span, never shrinks it. - The Python overflow digest canonicalizes spans (strips L<NN>: markers via _canon_evidence) before hashing, restoring line-shift stability for the over-cap region. No baseline change: the overflow branches only trigger above the per-finding caps (above the largest real entry), and the npm baseline is empty, so a Python 3.12 regenerate is byte-identical. * scan: refresh baseline for ipython interactiveshell.py span drift A newer ipython release changed the filesystem-enumeration span in IPython/core/interactiveshell.py, so its content digest no longer matched the baselined evidence and the studio scan shard flagged it as a non-baselined CRITICAL. Regenerated with Python 3.12: only the ipython entry's evidence_hash changes; the package/file/check key set is unchanged, and a studio enforcing spot-check exits 0. * Bound scanner evidence memory: stream overflow spans and cap lifecycle baseline size scan_packages.py: _extract_evidence no longer materializes a rendered span per match before slicing at the display cap. Once out holds _MAX_EVIDENCE_SPANS spans, further spans fold straight into a running digest, so a minified or padded file with hundreds of thousands of matching lines keeps memory bounded to the display cap instead of the match count. The fold reproduces _canon_evidence(" | ".join(overflow)) byte for byte, so the overflow digest and every baseline key are unchanged. scan_npm_packages.py: lifecycle-fetch-exec and cred-path-in-lifecycle stored the entire install script body as evidence, so --write-baseline on a package with a multi-MiB lifecycle script bloated the baseline JSON. Both now store a bounded matched snippet plus a body-sha256 digest, matching cred-env-in-lifecycle. The digest still binds the whole body, so a change to any line reopens the finding. Adds tests for the streamed overflow bound and the bounded-but-reopens lifecycle evidence. Baseline unchanged (byte-identical Python evidence; npm baseline empty). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make npm bracket-group scan order-aware so a same-line close-then-open binds _scan_group counted brackets with a per-line net (opens minus closes), which collapses intra-line order: a line that closes a prior block and then opens the host-config object on the same line, e.g. `}); const opts = {`, nets to <= 0, so the trailing `{` was dropped and the group started at the hostname line. A changed path/headers on the following lines then hashed to the same evidence and could ride an existing baseline key. Replace the net count with an order-aware (L, R) reduction per line (L closers needing an opener to the left, R openers needing a closer to the right) and apply it in order in both the backward and forward scans, clamping stray closers at 0. The trailing opener now stays visible so the whole object binds and a changed payload reopens. Per-line cost is unchanged (one C-level bracket findall), so the existing outbound-host evidence is byte-identical on all prior shapes; only the previously-dropped same-line case changes. Adds a regression test for it. * Harden scanner evidence: bound memory and bind Python call tails fail-closed Five fixes across both scanners, none of which change the committed baseline (a full regen of all three pip shards produced a byte-identical 185-key set). scan_npm_packages.py: _evidence and _outbound_host_evidence collected every regex match into a list before applying the 64-match display cap, so a text file under the size cap that repeats a cheap signal (such as NPM_TOKEN) millions of times could allocate a huge list of re.Match objects and stall or OOM before the overflow digest ran. They now stream from finditer and fold overflow as matches arrive via a shared _fold_overflow_match helper, byte-identical to the prior digest. scan_packages.py: - _extract_evidence kept inserting every unique over-cap span into the seen set even after it stopped appending to the display list, so a generated file with millions of one-line matches still grew that set unbounded. It now tracks spans only while filling the display list (per-line spans are unique by line number, so dropping them past the cap cannot miss a dedup). - _scan_line_end counted brackets with a per-line net, so a continued statement that closes on the same line it opens a flagged call (a leading "]" before "requests.post(") had the call's open paren cancelled and bound only the opener line. It now applies brackets in order via _bracket_lr (leading closers clamp at 0), matching the npm bracket fix. - a single-quoted string continued by a trailing backslash was not tracked across lines, so a close paren inside the continued string on the next line closed the call early; _blank_code_strings now carries the continuation. - a call with more argument lines than the soft cap was hashed only through the cap, so a changed data=/headers tail past it stayed suppressed; a closing call is now followed to its real close under a 200-line hard limit (a never-closing opener still stops at the 40-line soft cap so it cannot swallow the file). Adds regression tests for each. npm baseline is empty; the Python baseline is unchanged (verified byte-identical by regenerating all three shards). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Bind giant DOTALL span anchors and add context to constant IOC evidence Two fail-closed gaps where a changed payload could keep the same evidence hash and stay suppressed by the baseline. scan_packages.py: a giant greedy DOTALL span (a cross-line IOC match bridging more than 60 lines, e.g. RE_TEMP_EXEC matching a /tmp line and a much-later subprocess line) was dropped entirely once the per-line pass had any match, so an appended cross-line payload -- a new /tmp line plus a later subprocess line that share no single line, so the per-line pass never binds them -- produced the same evidence and rode the key. The span is no longer dropped: it is bound by its head and tail anchor lines plus a digest over just those (no line numbers, so a pure line shift is stable). An added or moved anchor reopens the finding, while churn in the bridged interior stays stable, so this does not reintroduce whole-file drift. Two baseline entries (multiprocess test, unsloth-zoo scanner file) carry such a span and are refreshed; a full three-shard regen confirmed only those two keys change. scan_npm_packages.py: known-ioc-string and cred-surface-host (always-bad) recorded only the bare needle/host as evidence, so a reviewed tarball that kept the IOC string while altering the adjacent fetch/exfil body produced an identical key. They now bind matched-line context: known-ioc-string via the matched line and its bracket-group continuation, cred-surface-host (always-bad) via the outbound call context (path/headers/body, falling back to the bare host when not in an outbound call). A changed payload on the same call now reopens. Adds regression tests for each. npm baseline is empty; the Python baseline updates only the two giant-span entries. * Hash giant-span interiors, bind exec/eval trigger, JS content, intra-literal whitespace Four fail-closed gaps where a changed payload could keep the same evidence hash. scan_packages.py: - A giant bridged DOTALL span was bound only by its head and tail anchors, so a cross-line payload inserted into the bridged interior between unchanged outer anchors kept the same key. The whole span content is now digested (via _render), so any interior change reopens; a pure line shift stays stable because the digest is over the markerless code. Two baseline entries (multiprocess test, unsloth-zoo scanner file) carry such a span; with full-interior binding, multiprocess resolved at two versions across shards now yields two distinct entries where the anchor digest had collapsed them into one. - The exec/eval-with-hidden-payload findings omitted the visible exec/eval line that makes the hidden string executable, so flipping a harmless eval("1+1") to exec(__doc__) kept the same key while arming the payload. The trigger line from the real-code view is now bound into the evidence. - check_js_file extracted evidence with the Python-string-aware extractor, which does not blank JS backtick template literals, so a template containing a close paren closed a call's bracket span early and omitted later option/body lines. The full file content digest is now pinned to every JS finding (not just large bundles), binding the whole call. scan_npm_packages.py: the evidence canon collapsed all whitespace via split(), erasing whitespace inside JS string literals along with harmless indentation, so a changed request body 'a b' -> 'a b' kept the same key. A new _canon_preserve_strings collapses whitespace only OUTSIDE string literals (reindent-stable) while preserving it INSIDE single/double/backtick literals (intra-payload edits reopen). Used for the evidence hash and the logical-line digests. Adds regression tests for each. npm baseline is empty; the Python baseline updates the two giant-span entries and adds the second multiprocess version's entry. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> | 2 个月前 | |
Stop os.dup2 alone being a reverse shell, and allowlist the zoo URL guard (#9994) * Stop os.dup2 alone being a reverse shell, and allowlist the zoo URL guard Security audit has been red on main since 2026-08-27, last green d617be270. Both failing shards are false positives, and neither came from a pull request here: each arrived when a floating dependency published a release. pip scan-packages :: extras -- triton 3.8.0 (published 2026-08-28) CRITICAL Reverse shell / bind shell pattern triton/_internal_testing.py L259: os.dup2(tmp_stderr.fileno(), 2) | L270: os.dup2(saved_stderr_fd, 2) That is stderr being pointed at a temporary file and then put back. RE_REVERSE_SHELL carried `os.dup2(` as a bare alternative, the only single-token one in a regex whose every other branch names two co-occurring signals (socket + connect + subprocess, /bin/sh + socket). dup2 is simply how you point a descriptor at a file, so it fired on capture helpers and redirect plumbing: ten of the nineteen reverse-shell entries in the committed baseline are dup2 with no socket anywhere in the file, in click/testing.py, numba/tests/support.py, rich/console.py, torch's elastic redirects.py and sentencepiece, and not one of them is a true positive. triton shipped the eleventh. A reverse shell dup2s onto a SOCKET, so the socket is the half carrying the meaning. The alternative now requires it, which puts it in line with its three siblings. Detection is not reduced: a payload has to name socket to get the descriptor in the first place, and the socket + connect + subprocess branch catches it independently. Both halves go into the evidence, so the entry reopens if either changes. pip scan-packages :: hf-stack -- unsloth_zoo 2026.8.16 (published 2026-08-27) CRITICAL Harvests environment variables/secrets AND makes network calls CRITICAL Accesses cloud metadata/IMDS AND makes network calls unsloth_zoo/vision_utils.py This is the SSRF guard that unslothai/unsloth-zoo#1106 added, being flagged for defending against the thing it defends against: the IMDS hit is _BLOCKED_HOSTNAMES, a frozenset of metadata names the guard refuses to fetch, and the env read is UNSLOTH_ALLOW_PRIVATE_URL_FETCH, the opt-out. They sit next to the requests session they exist to police. Telling a blocklist from an access is not something a pattern scan can do, so these are reviewed and allowlisted, which is what the baseline is for. The compiler.py obfuscation HIGH is also re-baselined; its hash moved because the banner text changed. Confirmed by scanning each side rather than reading the timeline: triton 3.7.1 exit 0 triton 3.8.0 exit 1, 1 CRITICAL unsloth_zoo 8.15 exit 0 unsloth_zoo 8.16 exit 1, 2 CRITICAL + 1 HIGH After this change both scan clean, and tests/security/test_scan_packages.py is 123 -> 128. Dropping the socket requirement fails the two dup2 tests; dropping the new baseline entries fails the zoo one. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the evidence unchanged when a named alternative already matched First push widened the evidence for every reverse-shell finding in a file that had a socket and a dup2. evidence_hash is taken over the evidence, so that reopened reviewed baseline entries that had nothing to do with the change: multiprocess/tests/__init__.py holds a socket, a connect, a subprocess AND a dup2, its two entries stopped matching, and the hf-stack and studio shards went red on this pull request having been green on main. The dup2 pairing now only applies when no named alternative matched, which is the only case it exists for. A file that fires on socket + connect + subprocess keeps byte-identical evidence to the scanner before this branch, so its entries stay matched, and a file that fires solely on the dup2 pairing still records both halves. Covered by test_a_named_reverse_shell_keeps_its_original_evidence. tests/security/test_scan_packages.py: 129 passed. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Decide dup2 in the gate, and leave the evidence pattern alone Second attempt at the same regression, because the first correction was aimed at the wrong half. The studio and hf-stack shards stayed red on multiprocess/tests/__init__.py: CRITICAL Reverse shell / bind shell pattern Evidence: L19: import socket sha256:26a745abdc7e89da28ab943394234d8ccb415e80... main's studio shard suppresses 90 CRITICAL, this branch suppressed 79. RE_REVERSE_SHELL is re.DOTALL, so a match runs from the first signal to the last and a span past the display cap renders as a digest of the whole span, not as the matched lines. That digest is what evidence_hash is taken over. Deleting the dup2 branch moved which span matched, moved the digest, and unmatched every reviewed baseline entry taken against it. Narrowing the pattern at all is therefore a baseline-wide edit, whatever it does to the verdict. So the pattern goes back exactly as it was, dup2 included, and stays the only thing evidence is extracted with. The question it used to answer implicitly is asked explicitly in check_py_file instead: a file with a socket is judged as before, and a file without one has to match RE_REVERSE_SHELL_WITHOUT_DUP, the same alternation minus dup2. Nothing that still fires is re-rendered, so no baseline entry moves. Verified against origin/main's scanner on the same inputs. Every branch that fires renders byte-identical evidence and an identical evidence_hash: socket + connect + subprocess + dup2 (the multiprocess shape) identical /bin/sh + socket + dup2 identical socket import + dup2, no connect identical pty.spawn identical webbrowser data: URI identical dup2 alone, no socket no longer fires dup2 alone, no socket (triton 3.8.0 shape) no longer fires The last two are the point of the change. triton 3.8.0, unsloth_zoo 2026.8.16, torch and accelerate all scan exit 0. tests/security/test_scan_packages.py: 131 passed. Putting dup2 back into the gate fails two of them; narrowing RE_REVERSE_SHELL fails the guard that exists because that is what went wrong here. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: danielhanchen <unslothai@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> | 5 天前 | |
CI: fix llama.cpp seed caching, Diffusers wheel installs, and isolated test coverage (#9979) * CI: fix llama.cpp seed caching, Diffusers wheel installs, and isolated test coverage * Keep a llama.cpp install only when its binaries actually run * Fix the prompt-cache, Diffusers, and MCP menu CI legs * Check the kept install's runtime payload, so Windows is not exempt * Give the CLI launch tests a writable studio home * Give each isolated unsloth_zoo file its own pytest process * Require the paired cudart trio and keep an install that already runs the stored backend * Tolerate an empty collection in the zoo rerun and re-anchor the node dir test * Skip the inline code scan when no backtick escapes the block regions * Answer the AppImage E2E preflight with the headers it asked for * Ask an unreadable marker for the payload every kind of its platform needs * Keep the mlx family out of the parallel zoo pass and run it serially * Shorten verbose PR comments * Reject a kept install whose binaries the OS will not execute * Probe the launched entrypoint, read Windows loader statuses, and honour this run's selection * Give the probe the recorded runtime line and stop reading GPU detection as a choice * Shorten the comments added since the last pass * Backfill the paired runtime asset when an install is reused * Compare the seed in llama.cpp's uint32 domain and require the published payload * allow cpu security import probe * Treat every uint32 sentinel seed as random in the per-choice offset _apply_seeded_llama_request compares the seed in llama.cpp's uint32 domain, so -1, 4294967295 and 2**64-1 are all LLAMA_DEFAULT_SEED and all keep prompt caching. _choice_seed still exempted the literal -1, so for n > 1 the two helpers disagreed inside one request: choice 0 kept the caller's random seed and its cache, while choice 1 was offset to ((seed & 0xFFFFFFFF) + 1) % 0xFFFFFFFF, a fixed seed with the cache turned off. seed=4294967295 with n=3 served [4294967295, 1, 2]. Confirmed against a real llama-server: -1, 4294967295 and 2**64-1 each varied across repeated requests, while 4294967294 and 0 did not. llama.cpp applies the same test at its own multi-choice offset in tools/server/server-task.h, where the guard is seed != LLAMA_DEFAULT_SEED rather than a literal. * Stop reporting a kept install as one that was just installed The keep path answers a transient update failure by exiting 0 with the tree already on disk. Both setup scripts only distinguish exit 0 by looking for "already matches", so a kept install took the other branch and printed "prebuilt installed and validated", naming a release that was never fetched. setup.ps1 also went on to print the old release as the installed one. The in-app updater had the same problem on macOS. start_update passes no pin_release_tag there, so the "pinned release X but installer produced Y" check that catches this on Linux and Windows does not run, and the phase reported "Updated llama.cpp to <the tag the user already had>". It now compares the marker before and after and says the install is already up to date, which is what happened. * Fail the zoo step when its mlx group glob matches nothing mlx_group is passed to pytest unquoted so the list word-splits into file arguments, which means an empty list passes no files at all and pytest collects the whole rootdir serially instead: green, much slower, and no longer the group the step describes. The glob only has to stop matching once, upstream renaming the family for instance, for that to be what runs. Also pass each rerun's own filename to _keep. Exit 5 stays tolerated, but a module-level skip and a file that stopped collecting for a new reason both produce it, and only the log says which file it was. * Cover the kept-install path against every marker shape that has shipped _existing_install_runs and _kept_install_payload_is_healthy decide whether an install the user already has survives a transient update failure, and they had no direct tests: the existing keep-path cases drive install_prebuilt with a hand-built two-key marker. UNSLOTH_PREBUILT_INFO.json has grown twelve shapes since #4562 with no version field and no migration, so an absent key is the normal case for anything installed before the release that added it. Covers each shipped shape on Linux, Windows and macOS hosts, including the S1-S8 markers that carry no backend key and resolve through the asset name, and the S1-S11 Windows CUDA installs that predate runtime_asset and so must not be asked for a cudart trio they cannot record. Also pins two boundaries that were unasserted: a source build has no marker, so confirm_install_tree rejects it and it still falls through to the source-build fallback, while a corrupt marker keeps the tree eligible and holds it to every kind's shared payload. Adds the missing SIGSEGV and hang cases for the --version probe. Platforms are simulated through HostInfo, as the rest of this suite does. That covers the path decisions and the payload tables; it does not cover the Windows loader or macOS dyld, which no Linux runner can answer. * Cover the keep path across every OS and accelerator, and its exit contract Extends the marker-shape cases with the product the installer actually has to serve. Fifteen cells across Linux, WSL, Windows and macOS by NVIDIA, AMD, CPU and Vulkan, each asserted three ways: a complete tree is kept, a gutted one is refused, and an accelerator tree missing only its own backend library is refused. That last case is the one that matters after a partial extraction, where the binaries still start and only fail once a model is loaded. Vulkan, windows-hip, windows-rocm, windows-arm64 and linux-arm64-cuda had no payload coverage at all before this. WSL is asserted to behave exactly like Linux, which is the claim worth pinning: HostInfo carries no WSL field and the WSL2 ROCDXG handling lives in host detection and setup.sh, upstream of the keep path. Adds the exit contract the setup scripts branch on, driven through install_prebuilt rather than the deciders: a transient failure returns 0 for every shipped marker shape, a tree that is not runnable still reaches EXIT_FALLBACK so a source build can proceed, an explicit version request is never answered with the old install, and a marker recording a backend this build cannot install exits EXIT_ERROR before the keep branch is reached. That last ordering was untested and is what stops a newer marker being answered with the tree already on disk. Also covers a marker whose keys were grafted on by sync_marker_selection rather than written in one go, which is the shape a long-lived install really has, and a marker naming another platform's backend, which falls open rather than being judged by a table that does not apply. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim the comments added for PR #9979 Same behaviour, fewer lines. AST comparison confirms every changed .py file is identical once docstrings are stripped, so this is comments only. setup.sh passes bash -n, setup.ps1 parses, the workflow YAML parses and its zoo step body still passes bash -n. * Report a kept llama.cpp install as an unavailable update for PR #9979 --------- Co-authored-by: mahiatlinux <110882203+mahiatlinux@users.noreply.github.com> Co-authored-by: Daniel Han-Chen <danielhanchen@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> | 4 天前 | |
Studio: add image generation, editing workflows and LoRA training with Unsloth GGUFs (#6763) * Tighten comments in the image stack tests and scripts * Close video single-file, training reservation, and image mount-resume gaps Route on-device single-checkpoint video folders through the single_file loader: a bare local .safetensors directory (no model_index.json) is advertised as a pipeline with no filename, so validation rejected it before it could load. Reinterpret the pick as a single_file load of the sole checkpoint, mirroring the image load route. Treat a reserved-but-not-yet-spawned LLM training start as active in is_training_active() so /images/load, /video/load, and /diffusion/start cannot race the reserved run for VRAM during the pre-spawn free window. Mirrors the diffusion training service reservation. Resume an in-flight image generation on the Images page mount: probe generate-progress, re-enter the poll loop, and refresh the gallery on completion so a run started elsewhere is reflected and its saved image appears without a manual refresh. Seed resident image defaults from the resolved base_repo rather than a possibly path-shaped repo_id so the first resident generation uses the right recipe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Publish image generation active state before pre-denoise setup generate() assigned self._gen only at the pipe() call, after deferred compile, LoRA resolution/application, and ControlNet download/build had run. Across that setup window generate_progress() reported inactive even though _generate_lock was held, so a reloaded page's mount probe showed idle and let a second generate queue behind the first. Publish an active step-0 _GenState the moment the generation lock is acquired, before the setup work, and clear it in the outer finally so a setup-time error cannot leave the UI stuck active. Mirrors the video backend's queued phase and the training start guard. * Studio: fix diffusion install ownership, dataset upload atomicity, gallery pagination, and teardown races install_sd_cpp_prebuilt: only write the .unsloth-studio-owned marker when the install created the target directory or it was empty. Adopting a pre-existing, unowned, non-empty directory (a user's own stable-diffusion.cpp checkout) made it eligible for the uninstaller's recursive delete. routes/training upload: make the multi-file promotion transactional. Back up each displaced original and roll every destination back on any failure, so a mid-loop rename error can no longer partially overwrite the live dataset. routes/training _resolve_dataset_folder: reject a symlinked dataset directory and prove the resolved folder stays under the datasets root, so image read/caption/delete cannot escape the root through a link. routes/training delete: escape glob metacharacters in the thumbnail filename so deleting an image named like [ab].png removes only its own thumbnails. image_gallery / video_gallery listing: filter records against the response schema inside the pager via a valid callback, so offset/limit/has_more all count over accepted records. A leading schema-invalid record no longer returns an empty page with has_more=true and stalls infinite scroll at offset 0. image_gallery / video_gallery save: publish via a temp file plus atomic rename (the sidecar is the video pair's commit marker) and clean up on failure, so a partial write never surfaces a truncated PNG or strands an orphan MP4. diffusion_train_common discovery: treat an empty caption sidecar as a metadata tombstone that still falls through to the dreambooth instance prompt, so clearing every metadata caption no longer fails with no captioned images found. diffusion backend unload: wait for an in-flight denoise to exit before tearing down process-wide patches and state, mirroring the load path. diffusion_engine_router: serialize the whole check/unload/publish transition so a concurrent selection cannot return the engine being unloaded. uninstall.ps1: gate the default sd.cpp process stop on the owner marker so a user's own sd-server is not terminated for a directory we then keep. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reject native batch seeds outside the JSON-safe range * Refuse sd.cpp install into unowned non-empty target dir When the install target already exists, is non-empty and lacks the .unsloth-studio-owned marker (a user's own stable-diffusion.cpp checkout, or unrelated files beside a custom Studio root), install() previously still extracted the release into it. Skipping the ownership marker only stopped the uninstaller from deleting the directory; extraction still merged binaries into the user's working tree and could overwrite same-named files. Fail up front with a clear message pointing the user at a fresh/empty location before any download or extraction, leaving their directory untouched. Update the ownership test suite to assert the refusal. * Studio: gate dataset uploads on the symlink check and surface local video single-file checkpoints * Tighten comments and docstrings added by the image-generation fixes * Studio: close arbiter load-registration race and surface native progress + local pipeline folders Publish native sd.cpp generate progress (_gen) before LoRA resolution so a reload probe reads active during setup, matching the diffusers path. Register the diffusion/video GPU load under the arbiter lock (acquire_for now takes a register callback) so a competing acquire cannot evict an owner before its load is marked in-flight and let two loaders allocate VRAM at once. Admit local diffusers pipeline folders (root model_index.json, weights in component subdirs) in the local model scan so they reach task tagging and the On Device picker. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: include marker-owned custom sd.cpp roots in the uninstall stop scan; harden Pester install against the nuget.exe PSGallery bootstrap * Studio: surface local pipeline scan roots, tag single-file checkpoints by filename, mark companion-only pipelines partial - _scan_models_dir: admit a scan folder that is itself a diffusers pipeline (root model_index.json, weights in transformer/ vae/ subdirs). _is_model_directory rejects such a root, so the child scan would list the component subdirs as bogus models and hide the real pipeline; treat the root as one model via _local_pipeline_index. - _local_is_diffusers / _local_model_task: include the sole checkpoint filename in the family-detection needles (_local_family_needles, resolved via resolve_local_single_file). A generically named folder holding one loadable qwen-image-*.safetensors / ltx-*.safetensors identifies its family only from the filename; the load route already resolves that file, so tag it or the task-scoped picker (which rejects task=null) hides the on-device model. - list_cached_models: mark a companion-only base snapshot partial. A GGUF image load prefetches the base repo's VAE / text-encoder / model_index.json but skips the transformer (the GGUF supplies it); the snapshot has a pipeline manifest yet is not a loadable BF16 pipeline, and _cached_repo_partial misses it. _repo_pipeline_missing_denoiser flags a pipeline snapshot whose transformer/ or unet/ component carries no weight, so the picker drops it instead of advertising it as fully on-device. * Studio: preserve foreign gallery files, force safetensors on remote ControlNets, and close dataset/seed/GPU gaps Gallery clear/delete now scope to Studio-owned files: image_gallery and video_gallery skip PNGs / MP4s without a readable recipe (a hand-dropped or orphan file the listing already hides), so clear() and a guessed-id delete no longer destroy files the gallery never surfaced. Remote ControlNets now force use_safetensors: a bare owner/name reaches from_pretrained without the base trust gate, and the Hub scan fails open when unavailable, so requiring safetensors closes the pickle deserialization vector. POSIX uninstall now stops resident sd-server / sd-cli under an owned sd.cpp root before removing the tree (marker-gated), mirroring the Windows stop-before-delete scan; a live native server no longer survives unlinking its binary. Diffusion dataset containment: the training-start read path and the discovery picker route bare names through the protected resolver, so a symlinked dataset is rejected / not advertised like the caption/delete routes already do. Uploads gain the inference decode guard (oversized real images 400 before OOMing the trainer) and dataset upload/caption/delete/import are blocked with 409 while a diffusion run is active. JSONL readers (trainer + routes) tolerate non-object JSON and invalid UTF-8 instead of raising AttributeError / 500. LoRA family compatibility is enforced in the shared resolver, not only the picker, so a direct API client cannot apply a mismatched-family adapter. GPU arbiter gains release_if so the image/video unload idle-check and release are atomic against a concurrent same-owner load's registration. Native batch recipes persist the base batch_seed and restore replays from it, so a native batch_index>0 image no longer advances its seed twice. FLUX.2-klein selects its sd.cpp text encoder by variant (4B -> Qwen3-4B, 9B -> Qwen3-8B) instead of the single family default. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: track the loaded GGUF filename so native companion resolution reproduces the load identity * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: gate local-pipeline image tagging on a real family; validate video sidecars before delete/clear * Studio: tighten image-generation fix comments and docstrings * Studio: gate gallery serve/export on ownership; keep image progress active until persisted; reserve diffusion training before the dataset scan * Studio: restore Reapply target on async image/video load errors; recheck training state before dataset commit; reclaim partial sd.cpp installs on retry Images/Video: a background model load that fails AFTER starting (error/eviction during download) leaves the previous pipeline resident, but handleLoad had already overwritten lastLoad.current with the failed pick, so "Reapply to loaded model" reloaded the failed model. Carry the prior Reapply target into the poller and restore it on the async error/null paths, mirroring the quant rollback. Training: an in-flight diffusion dataset upload passed _require_diffusion_dataset_mutable() at entry but could still commit files after a concurrent /diffusion/start reserved the training slot, mutating the dataset underneath the trainer. Re-check the interlock immediately before the commit phase; a 409 there leaves the staged temps for the finally to clean. sd.cpp install: an interrupted extraction (disk full, killed process, a raising post-extract cudart fetch) left the target non-empty with no owner marker, so the next lazy install tripped the "not a Studio-managed directory" refusal and wedged native install. Write the ownership marker before the partial writes when the target is reclaimable, so a retry recognises the debris as ours and re-extracts. * fp8 DiT quant: floor the dynamic activation scale with activation_value_lb An all-zero activation token row makes the dynamic per-row fp8 scale 0, which turns the quantized data to NaN and the render to black frames on torchao's plain-torch kernel path. The fused fbgemm/mslk quantize kernels clamp zero rows internally, so the bug only reproduces on machines without them, which is most user environments. Zero rows are real inputs, not a corner case: Wan 2.2 zero-pads its text conditioning, and Hunyuan-1.5 and Qwen-Image regenerate zero rows inside their transformer blocks every step. Pass activation_value_lb=1e-12 to Float8DynamicActivationFloat8WeightConfig whenever the installed torchao supports the kwarg (Float8Tensor rework, 0.13+), checked via inspect.signature so older torchao keeps exactly the current behaviour; the existing Float8MMConfig fallback chain is unchanged. Verified on GPU: with the forced plain-torch kernel path a zero-row input NaNs without the floor and stays finite with it, and end to end on HunyuanVideo-1.5 fp8 goes from a solid black frame (LPIPS 1.00) to a normal render (LPIPS 0.225); on Wan the floor matches the condition_embedder exclusion (LPIPS 0.211 vs 0.206). Same-seed renders with fused kernels present are unaffected, and pre-quantized fp8 checkpoints stay valid since weight scales are untouched. * Wire hosted pre-quantized DiT checkpoints into the image families Point prequant_repos for flux.1, flux.2-klein, flux.2-dev, qwen-image (int8 only there; fp8 is family-denied), z-image and krea-2 at the unsloth/<Model>-FP8 Hub repos carrying gate-validated int8 and fp8 transformer checkpoints, so the fast quant path loads the small pre-quantized file instead of materialising the dense bf16 transformer and quantising on device. Measured on FLUX.2-dev int8: build peak drops from 60.7 GB (dense + quantize) to 30.7 GB (hosted prequant), identical 30.7 GB resident after either path since loading a checkpoint is bit-identical to on-the-fly quantisation. The hosted repos name files <Model>-<SCHEME>.pt, so resolve_prequant_source now derives that model-name filename from the repo id (scheme suffix stripped case-insensitively) and carries the legacy transformer_<scheme>.pt as a fallback the resolver tries when the primary 404s, keeping older repos loadable. Wiring a repo also exposed a fallback hazard: with a prequant source present, the dense-fit preflight used to be skipped entirely, so a failed prequant download would fall through to the dense bf16 load the memory plan never budgeted, OOMing after eviction. The preflight now always runs and gates an allow_dense_fallback flag through _load_dense_quant_pipeline: a dense misfit still skips the fast path when no prequant exists, but with one it proceeds and a prequant failure raises to the GGUF build instead of loading dense. The same flag is set when the auto-policy replans an offloaded GGUF against a prequant-sized transient. Tests updated to the new filename convention plus new coverage for the derivation and the legacy-name fallback; the prequant-skips-refit test now asserts the re-check runs and forbids the dense fallback. Verified end to end on GPU: z-image int8 resolves the hosted repo, downloads the model-name file and renders (6.8s load, 5.9 GB peak). * Route krea-2 through its per-component loader on the transformer-quant fast path _assemble_pipe used Pipeline.from_pretrained for every family, but the krea repo ships transformers-5.x configs and no top-level tokenizer files, so the tokenizer dies with vocab_file=None. The pre-quantized checkpoint loaded fine and then the assembly crashed, dropping the load to the GGUF build, which krea-2 cannot take (Krea2Transformer2DModel has no from_single_file). Assemble per-component via load_krea2_pipeline like the pipeline-kind and single-file paths already do. Verified live: Krea-2-Turbo int8 and fp8 hosted prequant loads now assemble and render through the Studio images tab. * Keep Qwen-Image's text-stream linears bf16 on int8 (short prompts break torch._int_mm) Qwen-Image's MMDiT runs every text-stream Linear at M = actual prompt tokens: the Qwen2.5-VL embeds are not padded to a fixed length like FLUX's 512-token T5. A short prompt (13 tokens) or the near-empty negative prompt drives torch._int_mm below its M > 16 floor and the first denoise step raises 'self.size(0) needs to be greater than 16, but got 13' (measured on B200 through the Studio images tab). Add per-family int8 exclusions (txt_in, add_q/k/v_proj, to_add_out, txt_mlp) for qwen-image and qwen-image-edit, threaded through exclude_tokens_for_scheme(scheme, family) and the prequant checkpoint validation, so a checkpoint baked under the old token list is rejected and re-quantised instead of loaded crashing. The text stream runs at M = tens vs the image stream's M ~ 4k, so the exclusion costs nothing; the rebuilt hosted checkpoint gates 28/28 PASS with LPIPS mean 0.057 (was 0.069). * Harden the diffusion memory plan against transient free-VRAM undercounts A cold FLUX.2-dev int8 load on an idle 183 GB B200 planned offload=model (companions exceed budget) and silently served the GGUF as-is; the identical retry went resident and engaged the hosted prequant. The plan arithmetic was byte-identical across both loads (required 90,228 MiB, resident needs free of about 124 GB); the only divergent input was torch.cuda.mem_get_info, which is device-wide and instantaneous: a transient foreign CUDA context briefly held about 100 GB at the first snapshot, and the planner trusted that single read. Three changes: - settled_snapshot_device_memory: on cuda, synchronize + empty_cache (best-effort) and take the MAX free over up to 3 spaced reads. A transient can only shrink free, so the max rejects transient undercounts while a persistent tenant still caps every read. _plan_memory now uses it. - plan_fits_total_capacity + one replan retry: when the dense/prequant candidate fits TOTAL device capacity under the standard reserve and the 0.85 resident margin, an offload verdict can only stem from the free reading, so the loader re-snapshots and replans once before declining the fast path. Explicit balanced/low_vram modes skip the retry (they offload by mode). - diffusion.transformer_quant_declined log line with required/budget/free and the plan reasons, so the next decline is diagnosable from the server log (previously silent). Verified: cold FLUX.2-dev int8 first load in a fresh server now engages the hosted prequant resident (offload=none). * Add FLUX.2 Klein and FLUX.2-dev DiT LoRA training Register flux.2-klein and flux.2-dev in the DiT trainer following the upstream DreamBooth references: latents train patchified and batch-norm normalized from the VAE posterior mode, the packed forward reuses step-invariant position ids, and the guidance vector (3.5) is gated on the variant's guidance_embeds config. Conditioning stacks load per variant (Mistral via Flux2Pipeline for dev, Qwen3 via Flux2KleinPipeline for Klein) and are encoded and freed before the transformer lands on the device. The fused single-stream to_qkv_mlp_proj joins the attention projections in the LoRA targets; the single-stream out projection stays dense because its to_out suffix would also match the double-stream ModuleList container. Wire both families through the training registry (family set, labels, VRAM notes, rank 16 / lr 1e-4 defaults, bf16-only preflight), mark them trainable with train base repos in the family registry, add FLUX.2-dev to the gated-repo token check, and trust both official bases for training downloads. Verified on B200: 30-step klein int8 (19.6s) and nf4 (20.9s) and dev int8 (52.0s) runs train with finite decreasing loss and the saved adapters apply on the bf16 base pipeline (weight 0 reproduces the base image exactly, weight 1 visibly restyles it). * Support LoRA adapters on torchao int8/fp8 quantized image pipelines Adapters are baked at load time: they attach to the dense transformer, then quantize_ converts only the frozen base linears (the lora_ side path is excluded by name), then the loader compiles. Post-quant PEFT injection is not possible on a manually quantized module, so the prequant shortcut is skipped for a baked load and the memory plan is sized for the dense build (force_dense on the quant candidate). At generation time the baked topology is frozen: weight tweaks and disabling (scale 0 reproduces the quantized base exactly) go through set_adapters, while adding or removing adapters returns a clean 400 telling the client to reload with the new selection. supports_lora now returns True for int8/fp8 diffusers loads (checked before the gguf-kind early return, since the quant fast path keeps the picker kind); nvfp4/mxfp8 and GGUF-via-diffusers stay blocked. The load request model takes an optional loras list, threaded through begin_load on both engines (native ignores it and keeps applying LoRA at generation). Verified end to end on GPU: Z-Image GGUF picker + int8 + trained adapter loads through the API, bake marker logged, weight 1.0 vs 0 renders differ visibly, weight 0.5 accepted live, unknown adapter rejected as 400. Affected suites: 296 passed. * Add FLUX.1 Krea dev to the image model catalog Krea's guidance-distilled FLUX.1-dev finetune keeps the exact dev layout, so it runs under the existing flux.1 family unchanged. Wire it up end to end: - Catalog group with the gated official bf16 pipeline and the open QuantStack GGUF quants; the gated artifact is skipped on auto-routing when undownloaded. - Trust the official repo for non-GGUF from_pretrained loads, next to the other black-forest-labs bases. - Generation defaults: 28 steps at guidance 4.5 per the model card. The generic "krea" defaults key (Krea-2-Turbo's 8-step no-CFG recipe) used to swallow the id, which would have produced garbage output; the new flux.1-krea key precedes it on both the backend table and the images page table. - The flux.1 prequant checkpoints are schnell-based; the loader's baked base_model_id validation refuses them for the Krea-dev base, so int8/fp8 requests dense-quantize instead (covered by existing prequant tests). * Resolve pre-quantized checkpoints per base variant One family entry covers several published variants whose weights differ (flux.1: schnell, dev, Krea-dev), but prequant resolution was keyed on (family, scheme) alone, so only the default base could ever be served: the loader's baked base_model_id validation correctly refused the schnell checkpoint for dev and Krea-dev bases and every such load paid the dense download plus on-the-fly quantise. Add an optional prequant_variant_repos table on DiffusionFamily as (base_repo, scheme, repo_id) triples and thread the resolved base repo through resolve_prequant_source / usable_prequant_source and their three call sites (load fast path, memory-plan probe, auto-policy candidate). A base without its own entry keeps returning the family default, preserving the existing refuse-then-dense behavior exactly. Wire the flux.1 variants: the gate-validated unsloth/FLUX.1-dev-FP8 checkpoints (built in the earlier campaign but never reachable) and the new unsloth/FLUX.1-Krea-dev-FP8. * Add the Lumina Image 2.0 family to the image catalog Alpha-VLLM/Lumina-Image-2.0 is a 2.6B single-stream DiT with a Gemma2-2B encoder and a standard 16-channel VAE, all transformers-4.x-compatible, so the generic from_pretrained pipeline path loads it as a new lumina-2 family: - Family entry (Lumina2Pipeline / Lumina2Transformer2DModel), aliased to lumina-image-2.0 / lumina-image-2 / lumina2. No bare lumina alias: Lumina-Next checkpoints are a different arch and must stay unknown rather than crash mid-load. bf16-only upstream, so the fp16 fallback stays off like z-image. - Trust the official repo for non-GGUF loads; bf16 component table entry (ships fp32, ~5.2 GB transformer + 5.2 GB encoder bf16-resident). - Generation defaults 50 steps / guidance 4.0 per the model card, and the generate call passes the card's cfg_trunc_ratio=0.25 itself (family-gated, signature-gated): the pipeline default (1.0) runs the CFG double-forward on every step and oversaturates output. - Catalog group with the single ungated bf16 pipeline artifact (11 GB resident) plus routing assertions; images page defaults row. - No GGUF artifact: none exists upstream (only finetune/LLM quants), so the dense transformer_quant fast path (GGUF-kind-only) stays unreachable for now. Offline probes of the future prequant campaign: int8 and fp8 both engage and render cleanly (fp8 LPIPS 0.11 vs bf16, int8 0.33 from 50-step trajectory drift with intact quality), so neither scheme is family-denied. * Wire the hosted Lumina Image 2.0 int8/fp8 checkpoints Gate-validated against same-seed bf16 renders (28/28 pairs per scheme, zero failures): int8 LPIPS mean 0.146 / SSIM 0.937, fp8 LPIPS mean 0.116 / SSIM 0.946. Uploaded to unsloth/Lumina-Image-2.0-FP8 following the existing checkpoint repo conventions. * Add the HunyuanImage 2.1 family to the image backend The hunyuanvideo-community diffusers mirror carries the full stack in standard layout: a 17B dual-stream DiT (32.5 GB bf16), a Qwen2.5-VL text encoder, a ByT5 glyph encoder, the 32x HunyuanImage VAE, and guider/ocr_guider components (AdaptiveProjectedMixGuidance) that diffusers 0.39 loads natively, so the generic from_pretrained pipeline path covers everything with no per-component assembly. Family notes: - The call's guidance knob is distilled_guidance_scale (there is no guidance_scale kwarg), so cfg_kwarg routes the UI value there; real CFG runs inside the repo's guider at its baked scale. Defaults follow the card recipe: 50 steps, 3.25. - 2K-native: verified live at both 1024 and 2048. - Coexists with the HunyuanImage-3.0 structured exclusion (3.0 has no diffusers pipeline and stays excluded with its stated reason). - int8/fp8 dense quantization verified live (LPIPS 0.186 both vs same-seed bf16); a short prompt does not trip the int8 torch._int_mm minimum on this arch, so no family exclude entry is needed. - bf16 component table for the memory planner: (32.5, 16.3, 0.8) GB. * Surface HunyuanImage 2.1 in the image model catalog Catalog group with the open bf16 mirror pipeline (~50 GB resident, so a bare click on a consumer card routes to the QuantStack GGUF quants, which load and render through the generic GGUF path, verified live) plus the images page defaults (50 steps, guidance 3.25 feeding distilled_guidance_scale). * Add the HiDream-I1 family to the image backend A 17B MoE DiT (16 double + 32 single layers, 4 routed experts) with four text encoders, on HiDreamImagePipeline (diffusers 0.39). One family covers the open Full / Dev / Fast repos (same arch); per-variant generation defaults follow the upstream inference recipes (Full 50 steps at guidance 5, the distilled Dev 28 and Fast 16 guidance-free). The repos name a Llama-3.1-8B text_encoder_4 in their model_index but do not ship its weights; the official example passes the gated meta-llama repo in by hand. The loader instead assembles the component from the open unsloth mirror (byte-identical weights, already inside the non-GGUF trust gate), injected at the three pipeline from_pretrained sites, with output_hidden_states matching the official example. Memory planning counts the assembled TE4: 34.2 GB DiT + 28.8 GB encoders, ~63 GB bf16-resident. * Surface HiDream I1 in the image model catalog One catalog group with the three official bf16 pipelines (Full, plus the Dev and Fast distillations as labeled artifacts) at their ~63 GB resident size, so auto-routing keeps this a datacenter-GPU pick. city96's GGUF is deliberately not wired: the GGUF path would need the same Llama TE4 assembly for very small demand. Images-page defaults mirror the backend table with the variant keys ahead of the generic hidream key. * Pin the measured HiDream quant verdict in tests int8 and fp8 both engage and render cleanly on this family, including short prompts on int8: the routed MoE expert Linears only ever see the concatenated image+text stream (M far above the torch._int_mm minimum), so no deny entry and no family exclude tokens are warranted. Assert that so a future table edit cannot silently regress the measured behavior. * Wire the hosted HunyuanImage 2.1 int8/fp8 checkpoints Verified bit-identical to on-the-fly quantize: all 1264 state dict tensors (456 quantized) dequantize equal between the loaded checkpoint and a fresh quantize_ pass, so quality matches the runtime Dtype path exactly. Same-seed LPIPS suite means (0.35 int8 / 0.28 fp8) blend trajectory divergence with this family's own run-to-run nondeterminism (identical weights and seed reproduce a 17/255 mean pixel delta through the 50-step guider pipeline); per-case hard checks pass and the drift is compositional, reviewed visually. Uploaded to unsloth/HunyuanImage-2.1-FP8. * Fix silent LoRA drop and wasted transformer prefetch on GGUF quant loads Two live-test findings on the images load path: - transformer_quant with baked LoRAs, when the dense quantized build is declined for memory or fails: the load completed as a plain GGUF with the adapters silently dropped (HTTP success, supports_lora=false after the fact) -- wrong output with no signal. The load now fails with the recovery options (drop the adapters, free VRAM, or pick a smaller model). Weight-0 adapters still count as no bake request, and the plain no-LoRA decline keeps its silent GGUF fallback. - A fresh GGUF load on a small GPU prefetched the base repo's full bf16 transformer shards (~47 GB on Qwen-Image) because the dense-quant prefetch widening only checked scheme viability, not whether the device could ever hold the candidate resident. Gate the widening on total device capacity (reserve + 0.85 margin, the plan_fits_total_capacity bar) so a card that is certain to decline the dense build never pays the download; capable devices keep the prefetch. * Fix video progress under-reporting during load and generate Two live-test findings on the video progress endpoints: - load-progress downloaded_bytes froze mid-download: the counter used scan_cache_dir, which skips in-flight *.incomplete blobs, so it sat at the last completed blob for the whole multi-GB shard pull while the disk kept filling. Count the repo's cache directory directly (completed plus incomplete blobs, snapshot symlinks skipped so nothing is double-counted). - generate-progress reported total_steps=null / fraction=0 while step advanced: the video API only carried the native total field while the image API exposes total_steps and fraction, so one poller could not work against both. Derive the image-compatible aliases in generate_progress and declare them on the response model; the native total stays for back-compat. * Wire the hosted HiDream I1 int8/fp8 checkpoints Gate-validated: all 28 per-case pairs pass per scheme (LPIPS suite means 0.291 int8 / 0.278 fp8, in the 50-step trajectory-divergence band; CLIP delta means 0.007-0.008), and the int8 checkpoint is verified bit-identical to on-the-fly quantize across all 1615 state dict tensors (1073 quantized, max abs diff 0.0). Uploaded to unsloth/HiDream-I1-Full-FP8. * Add a pre-cast text-encoder loader for the layerwise fp8 scheme The runtime text_encoder_quant=fp8 path downloads the full bf16 text encoder and layerwise-casts it in place on every fresh load. For the heavyweight encoders (LTX's Gemma3-27B ~50 GB, FLUX.2-dev's Mistral-24B ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) that download dominates load time on a fresh machine. diffusion_te_prequant.py loads a pre-cast fp8-storage state dict instead: meta-init the encoder skeleton from the checkpoint's te_class, load_state_dict(assign=True), rebuild on CPU if non-persistent buffers stay on meta, then re-apply the same layerwise cast to install the upcast hooks. The cast is a deterministic storage transform, so the loaded encoder is bit-identical to dense-load-then-cast by construction. v1 hosts the layerwise fp8 storage scheme only: its state dict is plain tensors (torch.load(weights_only=True), no pickle execution). The dynamic-compute schemes (fp8_dynamic, int8, nvfp4) build torchao subclass wrappers at runtime and are deliberately not hosted. Checkpoints validate format, scheme, component and base_model_id before use and any problem falls back to the dense download and cast. Local path overrides reuse the DiT prequant allowlist env var. Families opt in via a new te_prequant_repos (scheme, component, repo_id) field on both DiffusionFamily and VideoFamily; the field defaults empty so nothing changes until a gate-validated artifact is wired. * Inject hosted pre-cast text encoders during pipeline assembly Wire te_prequant_pipe_kwargs into the three pipeline assembly sites: the diffusion full-pipeline branch, the diffusion transformer-only and GGUF branch (where the companion TE is the big remaining download), and the shared video assembly path before the pipeline/component split. Injection is gated exactly like the runtime cast (mode normalized to fp8, device supported, family not denied), so it can never engage where quantize_text_encoders would not; the later quantize_text_encoders call re-applies the cast idempotently and keeps status reporting truthful. With no hosted checkpoint configured the call returns {} and assembly loads the dense encoder as before. * Add the pre-cast text-encoder checkpoint builder Applies the runtime layerwise fp8 storage cast to a model's dense text encoder once and saves the cast state dict with baked metadata (format tag, base_model_id, family, scheme, component, te_class, versions) in the layout diffusion_te_prequant.py validates. Resolves the encoder class from the checkpoint's config.architectures so the recorded te_class matches what the pipeline instantiates. CPU-runnable: the cast touches storage dtypes only. * Test the pre-cast text-encoder load path Hermetic CPU coverage for diffusion_te_prequant: the checkpoint filename convention, family-table resolution by scheme and component with malformed entries skipped, resolution priority (path override, hosted repo, none) and the fp8-only scheme gate, the checkpoint validation matrix (wrong format, missing state_dict, wrong scheme, wrong component, wrong or missing base_model_id) with base case folding, the local-path allowlist refusal and missing-file fallback, and the assembly injection gating (mode, hosted entry, device support, family deny, load failure, successful injection). Also pins the te_prequant_repos field on both family dataclasses and that no family ships a hosted TE checkpoint until the campaign wires one. * Fix pre-cast TE checkpoint loading and engagement reporting Two bugs found while building the hosted checkpoints: - The builder recorded torch.__version__ (a TorchVersion object) in the checkpoint metadata, so torch.load(weights_only=True) rejected every artifact and the loader silently fell back to the dense download. Record plain strings. - Re-applying the layerwise fp8 cast to an injected pre-cast encoder raised on the duplicate hook registration, making quantize_text_encoders report the engaged cast as failed (status showed no TE quant while the encoder ran fp8). _cast_fp8 now returns early when the hooks are already installed. Also corrects the LTX TE size note: Gemma3-12B stored fp32 (~49 GB), not 27B. * Wire the hosted pre-cast fp8 text encoders qwen-image and flux.2-dev (diffusion) and ltx-2 (video) now resolve a hosted pre-cast fp8 text encoder from their unsloth -FP8 repos: - unsloth/Qwen-Image-FP8: Qwen2.5-VL-7B, 16.6 GB dense -> 8.8 GB - unsloth/FLUX.2-dev-FP8: Mistral-Small-24B, 48.0 GB dense -> 24.7 GB - unsloth/LTX-2-FP8: Gemma3-12B, 48.7 GB fp32 store -> 13.2 GB Every checkpoint verified bit-identical to dense-load-then-cast (729 / 585 / 1066 tensors, zero mismatches) and smoke-tested through the real backends with the repo engagement marker. Tests cover the wired entries, the resolver filenames, builder metadata weights_only survival, and the idempotent re-cast. * Report the compute dtype on fp8-cast encoders and inject the pre-cast TE on the dense fast path Two more findings from the hosted-TE GPU smokes: - Module.dtype reports the first floating parameter, which after the layerwise fp8 cast is the fp8 STORAGE dtype. Flux2 derives its prompt embed and latent dtypes from encoder.dtype and feeds them to randn_tensor, which has no fp8 kernel, so ANY flux.2 load with text_encoder_quant=fp8 crashed at generation (pre-existing, runtime cast included). The cast now swaps in a subclass whose dtype property reports the compute dtype; forward behaviour is unchanged. - The dense transformer_quant fast path assembles companions through _assemble_pipe, which never received the pre-cast TE injection, so the hosted encoder engaged on full-pipeline and GGUF builds but not on the fast path. Threaded through like the other two branches. Verified live on B200: qwen-image (full pipeline), flux.2-dev (GGUF picker with int8 DiT prequant), ltx-2 (video backend) all engage the hosted TE, render non-black, and report text_encoder_quant=fp8 truthfully. * Key the fp8 cast idempotency on an explicit completion marker Hook presence alone cannot distinguish a legitimately pre-cast text encoder from leftover hooks after a cast that failed mid-pass, so the early return now requires the completion marker _cast_fp8 sets once the hooks are fully installed. Leftover partial state keeps failing closed. Also tolerates non-Module encoder doubles in the hook probe and the dtype override. * Extend the fp8 TE quant to HiDream's Llama text_encoder_4 The generic quantize_text_encoders pass only covers text_encoder.._3, so HiDream's HEAVIEST encoder (Llama-3.1-8B TE4, 16.1 GB bf16) always stayed dense. TE4 is assembled separately (hidream_te4_kwargs), so the fp8 path now lives there: when the requested TE quant is layerwise fp8 and the device/family qualify, TE4 prefers the hosted pre-cast checkpoint (unsloth/HiDream-I1-Full-FP8, 8.6 GB) and falls back to dense-load-then- cast; a mid-pass cast failure reloads a fresh dense encoder instead of shipping partial state. The pre-cast loader and builder gain config_subfolder/config_overrides for standalone encoder repos whose config sits at the root and whose pipeline needs forward flags (output_hidden_states/attentions). Verified on B200: bit-identity 291 tensors (225 fp8, 0 mismatches), hosted checkpoint engages through the real backend (marker + status fp8), load 24.3 s vs 48.0 s dense, LPIPS 0.133 mean over 3 same-seed pairs vs the dense-TE render (gate 0.25), non-black frames. * Correct the ltx-2 resident TE estimate to the bf16 cast size The memory plan's bf16_components_gb held 50.4 GB for the LTX text encoder, which is the fp32 hub store of Gemma3-12B (~49 GB download), not what sits on device: the pipeline loads it torch_dtype=bf16, ~24.4 GB resident. The 26 GB over-estimate pushed the auto plan toward offload on cards that fit the real footprint. Comments and the size-table test now pin the resident semantics. * Host pre-cast fp8 text encoders for four more families Round 2 of the hosted TE set, each bit-identical to dense-load-then-cast and gated through the real backend (marker + status fp8 + same-seed LPIPS vs dense TEs): - FLUX.1 T5-XXL (text_encoder_2): 9.52 -> 5.90 GB, one artifact for schnell/dev/Krea-dev (T5 shards byte-identical across all three, verified sha256). 220 tensors, 144 fp8, LPIPS 0.109. - Lumina Gemma2-2B: fp32 hub store 10.46 -> 3.20 GB (3.3x download cut). 288 tensors, 182 fp8, LPIPS 0.041. - Z-Image Qwen3-4B: 8.04 -> 4.41 GB. 399 tensors, 252 fp8, LPIPS 0.112. NOT shared with flux.2-klein-4B: klein retrained layer 35's MLP (verified tensor diff, maxdiff 0.86), so klein hosts no entry. - Krea-2 Qwen3-VL-4B: 8.88 -> 4.83 GB. 713 tensors, 460 fp8, LPIPS 0.082. The constructor-assembled krea pipeline takes the encoder directly (load_krea2_pipeline text_encoder kwarg); the loader remaps 5.x rope_parameters and re-ties weights after assign so the rebuilt encoder matches the builder's structure. HunyuanImage 2.1 reuses the Qwen-Image artifact outright: its Qwen2.5-VL text encoder is byte-identical (every shard sha256, 16,584,414,544 bytes), recorded in the new component-level base-equivalence table the checkpoint validator consults. The injection loop now covers text_encoder.._3 so a family can host several components. Live check: LPIPS 0.123 vs dense. * Report the fp8-cast compute dtype without swapping the encoder class The dtype override swapped encoder.__class__ to a dynamic subclass, which breaks transformers' kwargs-based output recording: a fp8-cast Qwen3VLModel stopped returning hidden_states and every krea-2 generation with text_encoder_quant=fp8 crashed at encode_prompt (regression from the HiDream TE4 change; caught by the krea hosted-TE live smoke). The override is now a property shadowed on the ORIGINAL class that prefers a per-instance compute-dtype attribute, so class identity is preserved and uncast instances keep the stock behaviour. The idempotency test now pins exact class identity and the uncast-sibling fallback. * Pass the calibrated distilled sigma curve to LTX-2.3 8-step runs The 22B distilled DiT was trained against ltx_core's fixed DISTILLED_SIGMA_VALUES, but the diffusers scheduler derives 8-step spacing from resolution-shifted flow matching and lands far off at every reachable mu (second sigma 0.945-0.981 vs 0.99375, tail 0.37-0.61 -> 0.1 vs 0.725 -> 0.42 -> 0). At the distilled default step count the backend now passes the list verbatim, neutralising the scheduler's dynamic shift and terminal stretch for the call (they distort even explicit sigmas) and restoring them afterwards. Other step counts and the dev/base DiT keep the scheduler's own spacing. Live-verified on B200 through the video branch backend: the scheduler holds the exact curve after an 8-step distilled GGUF generation, config restored, healthy clip. Also reword the transformer_quant resolved reason to the measured reality: quant halves resident weights and hosted checkpoints cut load time, while per-step speed is roughly bf16 parity. * Pin the fp8 weight-quantize kernel against silent MSLK switching torchao's Float8Tensor KernelPreference defaults to AUTO, which switches the weight-quantize kernel to MSLK whenever an mslk package is importable on sm90+. Measured on B200: that changes fp8 scale rounding bitwise (8/8 FLUX matrices differ, scales ~55 percent of bytes), so a box that merely gains mslk would break the hosted-prequant bit-identity invariant; the mslk path is also slower under torch.compile (opaque extern call blocks inductor's quantize fusion, FLUX.1 fp8 e2e 1.149 to 1.624 s). Pin KernelPreference.TORCH explicitly, matching current no-mslk behaviour bit for bit; signature-gated for older torchao. GPU-smoked (finite, rel err 0.037) and pinned by test. * Shift Qwen-Image training sigmas to the inference distribution Qwen-Image's scheduler skips its static shift under use_dynamic_shifting, so the DiT trainer was drawing UNSHIFTED uniform-schedule sigmas for it (mean sigma 0.50) while inference always runs the exponential mu = log 3 shift plus the shift_terminal 0.02 stretch. Add a flow_shift config lever: "auto" (the new qwen-image default) rebuilds the training sigma table through the scheduler's own time_shift and stretch_shift_to_terminal so the draw matches the inference distribution exactly (mean sigma 0.72); a numeric value applies the standard linear shift s*u/(1+(s-1)*u); 1.0 keeps the historical identity behavior and stays the default for FLUX, Z-Image and Krea 2. The model timestep conditioning follows the shifted sigma, gathered in fp32 so bf16 rounding never skews it. Also wire two opt-in levers with off defaults: cfg_dropout (per-sample empty-prompt conditioning dropout, encoded alongside the captions before the text encoders are freed) and weighting_scheme="bell" (bsmntw-style mid-schedule Gaussian loss weighting normalized to mean 1). Verified with two 80-step rank-8 bf16 LoRA runs on Qwen/Qwen-Image (identity vs auto, same seed): both converge with finite decreasing loss and produce coherent same-seed previews. Unit tests cover the exact transform, the shifted sampling distribution, per-family defaults and config plumbing. * Add LoRA EMA, a persistent conditioning cache, and aspect bucketing helpers diffusion_train_extras hosts the opt-in training extras: LoRAEMA shadows only the trainable adapter params (warmup-ramped decay, default 0.99, exported as a second adapter under output_dir/ema), PersistentConditioningCache stores latent posterior stats and caption embeddings as safetensors keyed by content hash + family + resolution, and the aspect-ratio bucketing helpers group mixed-aspect datasets into same-area divisor-snapped shapes. The DiT trainer wires the first two behind config flags that default to the current behavior: ema_decay (0 disables) and cond_cache_dir (None disables). A fully warm cache skips loading the VAE and text encoders entirely; a cache hit is bit-identical to a fresh encode, including the per-channel qwen latent normalization. Also fixes the stale _gather_sigmas call in the perf test that still passed the scheduler instead of the sigma table. * Tighten torchao configs and note the FSDP2 design for the DiT trainer nf4 loads now enable double quantization (~0.4 bits/param off the frozen base scales at no fidelity cost), fp8 training uses the rowwise recipe when the torchao build ships it (per-row scaling confines the DiT activation outliers that a tensor-wide scale collapses), and the inference quant filter gains a per-scheme GEMM-tiling divisibility floor (16 for scaled_mm, 32 for MX blocks) so one ragged Linear cannot crash the first denoise after a clean quantize pass. plans/fsdp2_diffusion_design.md records the multi-GPU design: bf16/fp8 over FSDP2 with per-block units, LoRA attached before sharding, int8 out of scope (DTensor over the quantized subclass is undefined), per-family notes. * Batch diffusion inference with per-image seeds, an inference conditioning cache, and GGUF loader fixes Batched generation: /images/generate takes a prompts list (one image per prompt, txt2img only) or a seeds list (one prompt, one image per seed); the legacy batch_size path derives per-image seeds base..base+n-1 like the native engine. Every image gets its own torch.Generator so any batch member replays alone from its gallery recipe; the whole list runs as one forward by default with OOM backoff that halves a failed chunk, and an explicit batch_size caps images per forward. Validated 10-22x over serial engines on 32-image suites with LPIPS deltas within 0.002. Conditioning cache on the inference path: UNSLOTH_DIFFUSION_COND_CACHE_DIR (the inference sibling of the trainers' cond_cache_dir, same persistent store) wraps encode_prompt so repeated prompts skip the text-encoder forward entirely; verified bit-identical outputs. Bypassed while LoRA adapters are attached; tensor-argument calls pass through uncached. Compile cache: GGUF loads fingerprint their own bundles (quant=gguf, a different compiled graph than the dense family) and batched calls register every distinct (w, h, batch) chunk shape they ran, so the heavy GGUF batched warmups (~159 s at batch 32 on 12B-class, ~655 s on 20B CFG-batched) are paid once ever. GGUF loader: strip the sd.cpp model.diffusion_model. container prefix in the single-file converter; diffusers' FLUX.2 converter KeyErrors on it and the Qwen-Image identity mapping strands the model on meta. * Correct batched seed-replay docs to match measured behavior Same-seed images at the same batch shape are bit-identical; a solo regeneration with the recorded seed matches its batched rendition up to batch-size-dependent kernel numerics (mean abs pixel delta about 2.5/255, LPIPS delta under 0.002), not bit-exactly. The previous wording overclaimed bit-identity across batch shapes. * Note that batched bit-identity assumes a settled compiled graph The first generation issued while the deferred compile is still in flight can deviate transiently (observed once on a cold fp8 build: mean abs pixel delta 0.063/255); once the graph is settled, same-seed same-batch-shape images are bit-identical across runs. * Studio sidebar: Image03/FlimSlate icons, More flyout, Train row, New pills - Images uses Image03Icon and Video uses FlimSlateIcon. - New "More" row (MoreHorizontalIcon) opens a right-side flyout on click or hover holding Video, Recipes and Export; the close is delayed 180ms so the pointer can cross the gap. Its SidebarMenuButton deliberately takes `title` rather than `tooltip`: with `tooltip` the button returns a Tooltip root and DropdownMenuTrigger asChild would hand its ref to a non-DOM node. - Dropped the "Train" section heading; Train is now a top-level row between Images and More. data-tour="navbar" moves to the surviving nav group so the product tour keeps its anchor. - "New" pill beside Images and (inside the flyout) Video, via NavBadge. * Studio sidebar: match flyout rows and New pills to the existing scales - More flyout rows dropped their sidebar-row typography and size-icon override, which fought DropdownMenuItem's own scale (text-sm, gap-2.5, px-3 py-2 and size-4 icons) and rendered oversized glyphs and text next to the nav. - New pill reuses the brand "beta" badge recipe (nav-badge font, --ui-font-scale sizing, nav token colours) rather than hardcoded 9px values. - The More row's native title tooltip (an OS box on hover) is replaced by the app's Tooltip, wrapped around DropdownMenuTrigger so both triggers compose onto the same button, and shown only on the collapsed rail like other nav rows. * Settings: pin and reorder the sidebar navigation Adds a "Sidebar navigation" section to Settings -> Appearance, above the existing profile-menu customizer, with the same drag-to-reorder + switch UI. - New sidebarNav preference: one { id, pinned } entry per navigable row (projects, hub, images, train, video, recipes, export), array order = render order. Defaults match the shipped layout, so an untouched install is unchanged. - Unpinning moves a row into the More flyout rather than hiding it, so no page becomes unreachable. New chat and Search stay fixed as actions. - app-sidebar now renders from one navRows descriptor map, so a pinned row and its flyout counterpart cannot drift; the More row appears only when something is unpinned and highlights off whatever it actually holds. - Mirrored in the backend PersonalizationCustomization: without it the model's extra="ignore" would drop the field, and because sync replaces local state with the server's copy once customization is saved, the user's pin order would reset on the next sync. The validator dedupes and back-fills like sidebarMenu but preserves the client's order, since here order is meaningful. Frontend typecheck, i18n parity and catalog checks pass; 32 personalization tests pass, including a round-trip asserting a reordered list survives a save. * Sidebar customizer: drop the Search row, skip More for a lone item - Search is reached from the top bar, so it is no longer previewed as a fixed sidebar nav row; New chat stays. - More now appears only when it would hold two or more rows. A single unpinned row renders inline in its saved order position instead: a flyout wrapping one item costs a click and earns nothing. The customizer's More preview follows the same threshold. * Sidebar settings: hide a lone unpinned tab, match New chat icon, rename Profile menu - With exactly one tab unpinned, both More and that tab are dropped, so nothing is drawn for it (previously it rendered inline). The page stays reachable by URL. - The customizer's New chat preview uses PencilEdit02Icon, the icon the real row renders; Edit03Icon was a different glyph. - "Sidebar menu" is now "Profile menu", described as the shortcuts behind your name at the bottom of the sidebar, so it no longer reads as a second name for the navigation section above it. * Tighten comments in the new sidebar and delete-guard code * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio sidebar: keep the More row highlighted while its panel is open Moving the pointer into the flyout left the row unhighlighted while the panel stayed open. The row now carries data-menu-open, added to the nav hover selector list. Not data-state: the tooltip and menu triggers both write that attribute, so whichever lands last wins. * Images: use the shared pill toggle for Create/Train and pad the panels - Create/Train was the only segmented control on its own Tabs styling. It now uses PillTabs, the same control as the model picker and Hub toggles, pinned to the header row's 34px. PillTabs takes an icon per tab, so the inline-span workaround for TabsTrigger goes away. - pt-3 on both the Create and Train panels, which sat flush against the model selector row. * Images: make the workflow picker a dropdown instead of a 7-up strip Seven workflows in a 340px rail left ~48px each, so the labels crowded and the hints were only reachable as title tooltips. The strip is now a dropdown: the trigger shows the current workflow and its hint, and each row carries its own description. A row the loaded model can't run is disabled and shows the reason in place of the hint, so the gating explains itself. Adding a workflow no longer shrinks the others. * Images: workflow icons, hint under the trigger, more top room, unclipped Train cards - Each workflow carries an icon, shown on the closed trigger and on every row. - The trigger is one line (icon plus name). The selected workflow's description moved below it, where it reads like the Field hints further down the rail. - pt-6 instead of pt-3 on both Create and Train, so the cards clear the model selector row. - The Train right column scrolls while its cards use ring-1, which draws outside the box and was clipped at the scroll edges. p-px gives the ring room. * Images: one-line workflow rows, stronger trigger fill, roomier mode tabs - Dropdown rows are icon plus name only. The selected row's description already shows under the trigger, and a disabled row keeps its reason as a title. - Trigger fill moves to the bg-foreground/[0.07] dark:bg-foreground/[0.12] pair the hub cards use, so it reads against the card in both modes. - Description under the trigger goes from text-ui-10 to text-ui-11p5. - More horizontal padding on the Create / Train tabs. * Images: drop card borders for the composer shadow, keep scrollbars inside, use app controls in Train - Cards lose ring-1 for .panel-soft-surface: the composer's shadow in light, flat in dark, matching .chat-composer-surface and the menus. - Both rails now clip (overflow-hidden) with the scroller inside, so the scrollbar can't ride over the rounded corner. Same shape video-page already uses. - Train's 9 native selects become the app Select, so they no longer open an OS-native menu, and the native file input is hidden behind a Choose images button that reports the count. - Image previews use explicit 8-10px radii: this theme sets --radius to 1.1rem, so rounded-md was 15.6px and the thumbnails read as circles. * Images: one card for controls and preview, chat sliders, wider softer shadow - Controls and preview were two floating cards; they now share one card split by a divider. The Advanced dock stays separate since it toggles. - SliderField wraps Chat's ParamSlider, so the sliders match Chat (label row with the value, full-width neutral track) instead of a green track with a spin box. All 14 call sites keep their props. - panel-soft-surface goes from 0 2px 8px -2px /0.16 to 0 4px 22px -6px /0.10: lighter, spread wider. * Images: flat Create and Train panes, hover-only scrollbars, tidier Train dataset step Both Images tabs now sit on the page background like the Hub: no card, no shadow, no bounding box. A single rule divides the controls rail from the preview canvas (Create) and from the run area (Train), and the settings and previous-runs sections read as panes rather than nested cards. Also: - Scrollbars in these panes use the existing hover-scrollbar recipe, so the thumb only shows while the pane is hovered. - Workflow rows explain themselves with a tooltip after a short hover, which also works on disabled rows, and the descriptions are much shorter. - Training images rows are name plus image count; the license stays on the example card. - The upload step loses its dashed box, the buttons match the sizes around them, and Upload only appears once files are picked. - The empty preview uses the same icon as the Images nav item. * Images: full-height panes, wider settings rail, Create/Train offset from the selector The rule between the panes now runs the whole page height (the row drops its bottom padding and each pane pads its own content), the settings rail is wider on both Create and Train, and the Create/Train switch sits further right of the model selector. * Images: put both tabs on the Hub's centered measure Top bar and content now share mx-auto max-w-1100 with px-5 / sm:px-8, so Create and Train sit at the same width and position as the Hub instead of running edge to edge. * Images: restore the top bar position, drop the panes lower under it * Images: center the mode switch, flip the arrow with the orientation, app tooltips everywhere The Create/Train switch is centered on the page instead of trailing the model selector, with wider buttons. The flip control's arrows now rotate with the orientation and its label says which way the flip goes. Every native title tooltip on the page is now the app's tooltip, so they all get the rounded surface instead of the OS box. * Images Train: plainer field text, no green buttons, columns that stop colliding - The dataset name, trigger prompt, adapter name and custom base fields now say what they are in plain words instead of leaning on example values. - Import, Upload, Back, Back to settings and Train another are outline buttons, not green ones. - Example thumbnails are landscape tiles, so photos are not cropped to chunky squares. - Settings cells get min-w-0 and the select value truncates, so a long option like the nf4 label no longer widens its column into the next one. - The number stepper sits a little further in from the field edge. - Create and Train are wider. * Images Train: roomier example cards with Import on the thumbnail row * Video: same treatment as the Images tabs - No cards: the rail and the canvas sit on the page background, divided by a rule that runs the full page height, on the Hub's centered measure. - Wider rail, chat's sliders, hover-only scrollbars. - Every native title tooltip is now the app's tooltip, including the clip cards. - Reapply and Cancel are outline buttons, the empty state uses the Video nav icon, and the clip tiles are less rounded. * Images and Video: narrower generation rail, matching Train headings Create and Video rails go from 392px to 368px. Train a LoRA and Training settings are now the same size and both in the heading font: the h2 already picks it up from the base rule, so the settings header opts in with font-heading and the weight that rule pins. * Images Train: shorter copy throughout Family notes, example descriptions, precision labels and every helper line are trimmed so they stop wrapping to three lines and colliding with the next column. The nf4 label now fits its select without truncating. * Images Train: a little more spacing between field groups * Images and Video: tighten code comments * Fix training start NameError, the load-order guard test and CPU-only diffusion tests - start_training forwards resume_source_run_id to _start_training_impl, which reads it. Without it every start raised NameError. - Restore main's anchor in the load-marker order test: the file now has an earlier `if config.is_gguf:`, so indexing the first one compared the wrong branch. - The two diffusion tests that reach diffusers now skip when it is absent, matching the CPU repo-test env. - The UI smoke finds nav rows that live in the sidebar's More flyout. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Treat a null metadata caption as no caption str(None) stored the literal "None" as the caption, so a null row counted as captioned and would have trained on that text. Also drops an unused import. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix invalid-UTF-8 500s, the flat Canny map and the dropped DiT knobs read_text raises UnicodeDecodeError, which is not an OSError, so one bad caption sidecar or video sidecar 500d the info, upload and gallery routes. A flat image now yields the all-black edge map instead of its own luminance, and the four DiT loss knobs the trainer implements are declared so model_dump keeps them. * Use the ui font-size tokens instead of raw px text utilities text-[11px] and friends ignore the UI font size preference, which the repo's font-scale contract test enforces. Same rendered size at the default scale. * Fix diffusion dataset 500s, the dropout-1.0 no-op run and the reset base pick Four correctness fixes on the training side: - The labeling grid read caption sidecars under except OSError, but a non-UTF-8 sidecar raises UnicodeDecodeError (a ValueError), so one bad file 500d /diffusion/dataset/{name}/images and the grid could not be opened to repair it. Read it as no caption, matching the info summary. - An image past Pillow's own hard limit raises DecompressionBombError, which derives straight from Exception and so escaped the upload guard's (OSError, UnidentifiedImageError, ValueError) and returned 500 instead of the intended 400. - lora_dropout accepted 1.0, which makes PEFT build nn.Dropout(p=1.0): lora_A and lora_B receive no gradient and the run saves an untrained adapter while reporting normal progress. Bound it below 1.0, matching the LLM request schema. - The train panel re-seeded the base repo on every dataset refresh because the family object identity changes on each info fetch, so an upload or caption save silently replaced the user's chosen base and the run started on a different model. Track the pick and only re-seed on a real family change. * Show the retained failure when a video page mounts after a failed job Mount-time recovery handled only phase=completed, so reloading the page after a multi-minute generation failed left an idle view with no diagnosis: the backend keeps the terminal failed record only until the next job, and nothing else survives the reload. Surface it the same way the poll does, filtering the cancelled sentinel. * Fix batched generation crashes, cache keying and unreplayable recipes Four bugs in the batched inference path, all found by review: - A mixed-prompt batch sent a scalar negative prompt against a prompt list. Z-Image asserts on the length, and Qwen-Image, Krea 2 and FLUX true-CFG encode a batch-1 negative against batch-N latents and fail in the transformer's text/image concat. Broadcast it to match the batch. - The FBCache step-cache reset sat above the chunk loop. diffusers only resets that state at the end of a successful call, so a forward that raised (the OOM the backoff is meant to recover) left its own residual behind and the halved retry died on a shape mismatch. Reset before every forward instead. - The conditioning cache keyed on the checkpoint alone, but a GGUF or single-file load takes its text encoders from the companion base, so the same checkpoint against a different base reused the previous base's embeddings. Key the base too. - Gallery records stored the base seed and the requested batch size even when a prompts/seeds list drove the run, so restoring the second image of seeds=[5, 99] replayed seed 5. List-driven outputs now record as single-image recipes on their own seed. Also bound strength above 0: every img2img pipeline derives its step count from it, so 0 leaves zero denoising steps and either raises or, on SDXL, crashes on empty latents. * Fix quantized-load LoRA bake, prequant family exclusions and outpaint canvas Six review findings across the Images page and model scanning: - The quantized (int8/fp8) load path can only attach LoRA adapters before quantization, but the frontend load request had no loras field, so every generation after such a load was rejected and each reload repeated it. Send the selection with the load. - build_prequant_checkpoint passed no family to the scheme exclusions while recording the family in metadata, so a Qwen int8 artifact baked the short-M text-stream linears and was then rejected wholesale by the loader's family-keyed check. - Registering a bare single-file checkpoint directory produced no On Device row even though the images loader can load it; only its parent worked. Admit that shape when nothing else matched. - Unload left the Reapply target set, so the repair path was skipped and Reapply reloaded the ejected model. Clear it, as the video page does. - Both FLUX.2 bases were trusted for training but not inference, so Deploy to Create rejected every FLUX.2 adapter. - Outpaint allocated the grown canvas before downscaling, exceeding the browser canvas area cap on a large photo; an over-cap canvas is unusable, so Extend silently posted a fully transparent image and mask. Scale the source first. * Send the picked GGUF filename with the quant so diffusion loads fire The variant expander emitted only the quant label, and nothing else in the frontend set ggufFilename, so the Images and Video pages could never take their GGUF branch: both gate it on meta.ggufVariant and meta.ggufFilename, then fall through to the single-file path, which returns because the id is a repo id and not a .gguf name. Every quant pick was a silent dead click, with no load request reaching the backend. The filename was already on the variant row (the picker keys its list on it, and the variant validator requires a non-empty string), so thread it through the click handler. The chat path is unaffected: it reads ggufVariant and never needed the filename. * Version the conditioning cache key and reject non-finite flow_shift Two correctness fixes: - The cache keyed the checkpoint and its companion base by name only, so a Hub repo advancing to a new commit, or a local directory updated in place, kept returning embeddings from the previous text encoder. Pair both with a revision marker: the locally resolved commit sha for a Hub repo, config plus text-encoder file stats for a directory. Neither loads the encoders, so a warm run still keeps them off the GPU. - flow_shift only checked positivity, but JSON accepts 1e309, which floats to inf, and inf <= 0 is False while NaN fails every comparison. The sigma table then evaluates s * u / (1 + (s - 1) * u) as NaN, which poisons every sampled sigma and saves a corrupted adapter while progress looks normal. Require a finite value. * Keep curated models listed, guard the video companion repo, pin diffusers Three review findings: - The picker filtered every catalog member out of Recommended and Hub search on the way to canonical group rows, but nothing renders those rows yet (catalogGroupFitsDevice and groupMatchesQuery are imported and unused). A task-scoped picker's models list is catalogToModelOptions(), i.e. group members exclusively, so both lists came back empty and no curated model could be discovered or downloaded. Keep the artifacts listed until the grouped UI exists. - The video delete guard compared only repo_id, so deleting the companion base of a loaded GGUF video model was allowed even though it supplies the VAE and text encoders. Compare base_repo too, matching what the images guard already does for its companions. - diffusers was declared unversioned while the diffusion stack requires 0.39 (Krea2Pipeline, the cache_context child registries, the Flux2 and Z-Image pipelines), so an upgrade could keep an older release and selecting an advertised model failed until the user upgraded by hand. * Namespace the trainer conditioning cache per checkpoint, bound the learning rate - The trainer keyed its persistent conditioning cache on family and resolution only, while the keys themselves carry just the caption or image content and crop variant. One cache directory reused for two checkpoints, or for the same repo at a new revision, let a warm run skip loading its encoders and train on the other model's embeddings and latent statistics. Namespace on the base checkpoint and its resolved revision as well. The revision helper now lives beside the cache in diffusion_train_extras and the inference wrapper delegates to it, so the two cannot disagree about what counts as the same source. - The diffusion learning rate only checked positivity, but 1e309 floats to inf and satisfies gt, so the route evicted the resident models and started AdamW with an infinite rate: the first step destroys the adapter while progress looks normal and the result is saved. Bound it below 1.0, matching the LLM schema, which rejects inf for the same reason. * Fix GGUF image model picks doing nothing, and pick the train base in the top bar The quant rows never forwarded the .gguf filename, so every hub GGUF pick on Images/Video fell through to a silent return. On Train the top bar now picks the training base instead of a generation model, which is GGUF-only and untrainable. * Pin diffusion and video loads to the live HF cache root Both read huggingface_hub's import-time HF_HUB_CACHE, which changing the cache folder does not update: progress counted the old root while the download wrote to the new one, and from_pretrained could split one model across both. * Add the diffusion download plan endpoint Reports the repos and exact files a pick needs so the download manager can stage them with the loader's own file scope. A plain snapshot would add the packaged root single, transformer shards and fp16 twins the loader never opens. * Add a file-scoped flavour to the Hub download job Lets a consumer that reads a deliberate subset of a repo stage it through the normal download manager. Keyed as "@scope" so it never collides with a quant or with the repo's full snapshot, and the file list rides the registry so an XET to HTTP retry respawns the same scoped job. * Stage image and video downloads through the Hub download manager They downloaded inline inside the load, so they had none of the manager's disk preflight, manifest verification, resume or panel progress. Picks now stage as scoped jobs carrying the loader's own file list, then load from a warm cache. * Fetch staged GGUF checkpoints as scoped jobs, and stop calling diffusion models unsupported A GGUF entry went out as a full snapshot, whose ignore list drops *.gguf: the job finished at once having fetched only docs, and the repo landed on device unloadable. Every entry is scoped now. The Hub also no longer tags image/video models as unsupported (they run on their own pages), and those pickers name what they select. * Apply the picker task filter to local model sections LM Studio, ./models and custom-folder rows ignored it, so the Images picker listed chat GGUFs that 400 on a diffusion load. The backend already tags every local model with a task for this purpose. * Route a chat pick of a diffusion model to the Images or Video page Chat cannot load one, so it was either hidden or failed on load. The unfiltered picker now lists on-device diffusion models and navigates to the page that runs them, passing the repo and quant so that page loads it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Route the real GGUF filename, keep non-GGUF curated models, key scoped downloads by file set Five review findings, four of them ways a click did nothing or fetched the wrong thing: - A chat pick of a diffusion model routed ggufVariant (a label like Q4_K_M) in the search param the target page uses verbatim as the GGUF filename, so the load asked for a file that does not exist. Route ggufFilename; no filename means a curated non-GGUF pick, loaded as a pipeline. - The task-scoped pickers kept only GGUF repos, so the catalog's bf16, bnb-4bit and single-file fp8 artifacts could not be discovered or downloaded on the Images and Video pages even though loadSpecFor knows how to load them. Keep curated artifacts whatever their format, in Recommended and in Hub search. - Both pages deduplicated routed selections on the model alone, and they now stay mounted, so picking the same repo again -- another quant, or the same one after chat evicted it -- returned early without loading or clearing the query string. Key on model and quant. - Every scoped image download shared one @diffusion job key regardless of the requested files, so switching quant mid-download adopted the running job: the UI waited on the first file set, then loaded a file that was never fetched. Include a digest of the file set in the key. - A scoped plan silently dropped requested files missing from Hub metadata, and snapshot_download succeeds when an allow pattern matches nothing, so the job reported completion and triggered a load with required files absent. Fail the job instead. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the scoped download key derivable, and stop the hidden page hijacking a route Four review findings, the first a regression from my own last commit: - Keying scoped download jobs by a digest of the file set broke the download manager: it builds that key client-side (it polls and cancels before any response tells it a key), so it watched and cancelled a key no worker owned and never fired its ready callback. Keep the derivable "@scope" key and refuse the second request instead when a live job on the slot is fetching a different file set -- decided inside the registry claim, under the lock, so a concurrent claim cannot slip past it. The manager records the file set on the job as well, so a sibling quant's transfer is not adopted locally either. - Both diffusion pages read the route query through a loose useSearch and both stay mounted once visited, so the hidden one consumed the other's ?model=: it navigated back to its own route and tried to load, say, an image checkpoint as a video model. Only the visible page consumes it. - The staged download plan was built without the configured HF token or the Advanced values the load itself sends. The token matters most: the backend's Hub metadata lookup is best-effort, so a gated base silently planned no companion entry and the load pulled those multi-GB files inline, outside the manager. The memory/quant controls decide whether the base transformer/ shards are needed at all, and the route dropped memory_mode, cpu_offload, the prequant path and the LoRA selection before asking for the plan. - The video preview kept playing after leaving the page: the keep-alive layout only hides it, and display:none does not pause a media element, so a clip the user unmuted kept its audio going over the next page. Pause on the active transition and do not auto-replay while hidden. Also completes the hand-built request bodies in the hub download tests: the scoped-files field this branch added to the route read as an AttributeError against them, failing five tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Serialize the GPU handoffs, gate DiT training on a GPU, and keep 3.9 installable Six review findings, three of them evict-then-fail orderings: - The chat load reclaimed the GPU without telling the arbiter it existed. A chat load holds no llama-server process until its GGUF has downloaded, which is minutes, so a competing Images/Video acquire in that window found nothing to cancel, took the GPU, and the chat load then spawned onto the same device. It now registers an in-flight marker through acquire_for's register hook (under the arbiter lock, as the image and video loads do), the evictor cancels a marked load, and the route undoes itself if ownership moved while it loaded. - The Hub-download conflict check ran after that handoff, so a GGUF the download manager already owns destroyed the resident Images/Video pipeline and then 409'd, having loaded nothing. It moves above the handoff, together with the marker it handshakes with. - The image load released the engine router's transition lock before registering the load, so a second load choosing the other engine could unload the still-idle engine this one captured; the load then landed on a deactivated engine, where generate, status, unload and the arbiter's evictor can no longer reach it. Registration now happens under that lock and refuses if the engine changed. - Training a DiT family on a host with no GPU was accepted: nf4 is not a CPU fallback, its 4-bit load goes through bitsandbytes, which requires CUDA, XPU or MPS. The start unloaded the working Images pipeline, pulled the text encoders, and only then died in the child. Rejected before the teardown now, and /info stops advertising a precision that always 400s. SDXL keeps its documented fp32-on-CPU path. - Both diffusion pages kept the routed-pick marker forever, so re-picking the same checkpoint (after chat evicted it) neither loaded nor cleared the query string. The marker is released once the query is gone. The Images key also carried a stray NUL byte, which made the file read as binary to grep and other tooling. - diffusers dropped Python 3.9 in 0.38, so the unconditional >=0.39.0 pin left pip no candidate at all on 3.9 and made every install that composes the huggingface extras unresolvable there. The floor is conditional now. Also fixes tests that were already red on the branch: two hand-built request fakes had gone stale against fields this branch added, and the handoff-ordering test only failed on a host with fewer than two GPUs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the GGUF variant contract test against the merged handler signature The assertion pinned the exact single-line call handleVariantClick(v.quant, v.downloaded, expectedBytes, v.filename), but the handler takes (quant, filename, downloaded, sizeBytes) and prettier wraps the call across lines, so the mandatory repository test job failed on every push. Match the call structurally and assert the filename really is forwarded in the handler's argument order. * Stop a background page and a stale record taking the GPU or a download with them Five fixes from a review pass over the diffusion work. delete-finetuned rmtree'd a model the Images or Video engine was holding: every guard on that route is chat-only, and Images loads any local path, so deleting a local diffusion model under the storage root pulled the weights (and the companion VAE / text encoders sd.cpp re-reads each generation) out from under a live pipeline. The cached-model route already refuses this; the trained/exported one now does too, matching by path rather than repo id, and failing open on a chat-only install so it cannot block ordinary deletes. A staged download finishing while its page was hidden loaded the model and evicted whatever the user was actually using: both diffusion pages stay mounted behind the router and a load takes the GPU unconditionally. The pick is now held until its page is on screen again, which is also what chat does. A scoped download could report success having fetched nothing. With Hugging Face metadata unavailable no manifest is written, so verification is a no-op, and snapshot_download returns an existing snapshot folder without downloading when its own repo_info call fails. A repo already on disk from a full snapshot job (which ignores *.gguf) therefore completed with no weights and auto-loaded against them. The requested file list needs no network, so it is checked against the disk directly. The XET to HTTP retry reclaimed the job slot without the scoped file list, and that claim overwrites the stored record, so a later identical scoped start compared an empty list against the real one and 409'd instead of adopting the running download. The DiT accelerator gate probed torch.mps.is_available(), which only exists from torch 2.5 while the supported floor is 2.4. All three probes shared one try/except, so on torch 2.4 the AttributeError read as 'no block' and a CPU-only host still evicted the resident pipeline, downloaded the encoders and died in the child. Each accelerator is probed on its own now, through torch.backends.mps. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stage what an LTX-2.3 load reads, keep a routed file's load kind, drop an unbakeable LoRA Three from the latest review. The video download plan always asked for the wide base file list, so an LTX-2.3 pick staged the 2.0 base's VAEs, vocoder and connectors that the checkpoint supplies itself, while the companion files the 2.3 assembly does read were left out of the plan and pulled inline at load, outside the panel's progress, cancel and disk preflight. The plan now recognises a 2.3 pick by name (the load keeps the authoritative header probe, and under-guessing only falls back to the load-time pull), narrows the base list, and stages the extras in the same entry as the checkpoint so one repo stays one scoped job. A pick routed from the chat picker arrives as ?model= and ?quant= with no picker metadata, so a bare local .gguf or .safetensors was loaded as a pipeline: an explicit model_kind wins over the backend's filename sniffing, so it evicted the resident model and then failed on the missing model_index.json. Both pages now derive the load kind from the path, the same way their own picker handlers do. A torchao int8/fp8 build takes adapters only at load time. Switching artifact inside one family keeps the LoRA selection, since the family did not change, but the load did not bake it, so the next generation was rejected with 'reload the model with the adapter selection' while the picker still showed the adapter as active. The selection is now dropped once per resident build, with a message saying to pick and load again. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cancel an evicted safetensors load, spare the arbiter for CPU-only chat, fetch clips lazily Four fixes from the latest review round: - The GPU arbiter's chat evictor only cancelled the llama.cpp side. The orchestrator publishes active_model_name once its worker reports success, so an in-flight safetensors load was visible only as an entry in loading_models and finished onto the GPU after ownership had transferred. Cancel every pending load, and give the safetensors branch the post-load ownership recheck the GGUF branch already had. - A manual gpu_layers=0 GGUF load runs on the CPU with the GPUs hidden from the child, yet it took the arbiter unconditionally: it cancelled a running image or video generation for a model needing no VRAM, then held CHAT ownership so the next GPU workload unloaded it for nothing. Gate the acquire on the same predicate the launch-time CPU-only mask uses, as the image and video loaders gate on their resolved device. - The staged-download hook subscribes per repo, not per job, so another job on the same repo advanced the staged queue (starting a load whose scoped files were still downloading) or wiped a queue that was still running. Compare the variant each callback carries, like the chat page's auto-load does. - The video gallery fetched every record of a page into an object URL that lives until the page closes: 50 clips at tens to hundreds of MB each, for cards the user may never scroll to. Fetch a clip as its card nears the strip's edge, plus the selected one the player needs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim the comments across the diffusion backend Comment-only pass over the Python this PR touches: drop what the code already says, collapse multi-line explanations that still read on one line, and keep the reasoning that is not recoverable from the code. No code, docstring semantics or behaviour changes; verified with an AST comparison against the previous revision, and the backend suite is unchanged (same 37 environment failures as before: the API integration tests that need a live keyed server, the flash-attn install hooks, and the GPU memory fields). * Invalidate latents on a VAE swap, keep a cut-off generation, surface the EMA adapter - source_revision() scanned the checkpoint root plus text_encoder/tokenizer but not vae, so swapping or fine-tuning the VAE in place left the conditioning cache namespace unchanged and a warm run trained against latents from the old checkpoint. Include the vae directory, like any other component the cached tensors come from. - /images/generate answers only when the images are saved, and secure mode's tunnel caps an origin response near 100 seconds, which a native CPU or a high-step run passes routinely. The page reported failure while the work kept running, and a retry would duplicate it. A lost response (fetch rejection or a gateway status the origin never answered) is now told apart from a refusal: the page waits out generate-progress and reloads the gallery, so the run it started still lands. - The trainer emits the EMA adapter's path with the terminal event, but the state update dropped it, so neither the run history nor either response schema carried it and an enabled EMA left nothing discoverable. Keep it, and show it next to the primary adapter. - weighting_scheme advertised a choice of timestep sampling; sampling is always logit-normal and the flag only selects the bell loss weights. Describe what it does. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Hide unloadable cached rows, hold the dataset interlock, bound a GIF export - The cached-model listing tagged any repo with a model_index.json as text-to-image, so a community pipeline the image loader's trust rule refuses still got a row in the Images picker, and a detected-but-untrusted video repo fell through to that same tag. Gate the image tag on the load path's rule and hide an untrusted video repo outright. - A routed diffusion pick only carries a GGUF filename, which is all the chat picker has, so a curated single-file artifact arrived with no quant and was loaded as a pipeline: from_pretrained on a repo with no model_index.json. Pass the page's own catalog spec into the route pick, so a routed pick resolves to exactly what a direct pick on that page resolves to. - The dataset mutation endpoints checked is_active() and only then handed their filesystem work to a thread, so a start reserving in that gap changed captions or removed images underneath the preflight or the running trainer. The interlock is now registered for the whole request under the lock reserve() uses, and a start refuses while a mutation is open rather than waiting on it. - GIF export held every kept frame as a paletted image before encoding; a clip may be 2048x2048 for 1024 frames, and at the 12 fps target the step is 1, so one export click could allocate over 4 GB and take the backend down. Downscale past 720 px and widen the step to keep at most 300 frames. - seed accepted any Python int, so an out-of-range one passed every preflight, evicted the resident models, spawned the trainer and only then died in torch.manual_seed. Bound it to torch's 64-bit range in the request and config. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the accelerator probes in the DiT family-metadata tests Six tests read family_train_infos() (or a start preflight) without pinning the host probes, so they only held on a machine with a bf16 accelerator: on a GPU-less runner the DiT gate empties precision_modes, turns supports_compile off, and replaces any other preflight message with the no-accelerator note, and all six failed there. A conftest fixture pins both probes for exactly those tests, so they assert the family metadata they are about on every host. The gate's own CPU-only behaviour keeps its dedicated tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Carry the pipeline task into the hub inventory the pickers read - The task-scoped pickers filter On Device rows on a task, and the chat picker routes a diffusion pick by the same field, but those rows come from the /api/hub inventory, which never carried one: the Images and Video pickers listed nothing on device and the routing never fired. Both cached scans and the local listing now tag rows with the classifiers the models API already uses, the schemas and the frontend adapter carry it through, and a row the backend classified as a generation task is exempt from the chat-only guard that was also dropping it. - The local routing map was keyed by model_id while the row click passes id (a filesystem load id for a models_dir or LM Studio entry), so the lookup missed and the pick fell through to the chat loader. Key both. - A staged download whose start answered "error" left its head in place, where the effect never re-runs and onReady never fires, so the pick was stranded until the user reselected. Clear the queue and say so. - Every scoped pick in a repo shares the @diffusion variant, so the variant alone cannot tell two file sets apart: restaging while the first job finished let its completion pass for the new pick and load a checkpoint that had not downloaded. Bind the callbacks to the repo + file set they started, and to the staging generation. - A rejected generate POST does not say whether it reached the backend, so an immediately idle progress read was ambiguous and a submission that never landed looked like a finished image. Require evidence: progress seen active, or a gallery record that was not there before the POST. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the OpenAI image URL fetchable, keep WebM audio, stream example imports Four review items on the diffusion Studio work: - response_format=url returned the bearer-gated gallery route, which a standard image client downloads with no Authorization header, so the default response format was unusable. Mint a short-lived HMAC link instead (the shape RAG already uses for pdf.js) served by a signed route, and leave the gallery route itself bearer-only. - A manual gpu_layers=0 load carrying speculative_type="off" -- a value the UI persists and sends -- read as GPU-bearing, so it took the GPU arbiter and evicted a resident image/video pipeline even though the launcher hides the GPUs for it. Canonicalize the mode and exempt "off". - The curated example import prepared the whole split before the loop stopped at the 10-100 image cap; m1guelpf/nouns is 49,859 rows / 328 MB. Stream instead, with the prepared load kept as a fallback for a repo that cannot stream. - WebM export dropped the audio track an LTX-2 clip carries, silently, on the format offered for web embeds. Mux it as Opus through a resampler + FIFO, and keep exporting the video alone on a build without libopus. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not stub out triton on a GPU host when the Xet backend fails to import The lazy loader retries `import unsloth_zoo.hf_xet_fallback` under UNSLOTH_ZOO_DISABLE_GPU_INIT=1 whenever the first attempt raises. That flag makes unsloth_zoo take its MLX/CPU path, which injects triton and bitsandbytes STUBS into sys.modules for the rest of the process. On a working GPU box whose first import failed for an unrelated reason (a bitsandbytes/CUDA mismatch, say) the retry succeeds, so Studio boots looking healthy and then dies at the first CUDA-only kernel: a GGUF or compiled diffusion generation hits the stub and returns NotImplementedError: Unsloth: 'triton.tools.experimental_descriptor.enable_in_pytorch' was called on Apple Silicon / MLX, where triton is stubbed out. so every image generation 500s with an Apple-Silicon message on a Linux CUDA host, while the load reports success. Found by loading Z-Image-Turbo GGUF through the API on a box where bitsandbytes could not initialise. Gate the retry on the host genuinely having no accelerator. The Xet stall watchdog is optional and already degrades with a warning; a process whose triton is stubbed out is not recoverable. The warning now says why it did not retry. * Fix the lost-generation proof set, the settle timeout and the hub inventory's diffusion gates Seven fixes from the latest review round on the Images page and the hub cache inventory. Images page: - The lost-POST settle path built its "already seen" gallery id set inside the catch, after the request failed. By then the earlier runs of the same batch had already prepended their records, so run 2 could accept run 1's image as proof that its own request reached the backend. The set is now captured once before the first POST and grows with every record the batch produces. - settleLostGeneration fell out of its SETTLE_MAX_MS loop and returned normally, so a wedged generation was counted as done and the next run started against a busy backend. It now throws on timeout. - Restoring a recipe cleared the ControlNet selection but left the workflow tab and the init / mask / reference images pointing at whatever was loaded, so the next Generate conditioned on an unrelated image. It now clears all of them and returns to Create. - The download plan omitted the adapter selection the load itself bakes in. A baked LoRA forces the dense build path, so the plan described a different file set than the load that followed and the rest was pulled inline, outside the download manager. Both now derive the list from one helper. Hub cache inventory: - A download for a repo an Images or Video load is staging was allowed to start: only the llama.cpp loader was consulted. Both diffusion backends already expose loading_repo_ids for the delete guard, and the download guard now reads them too. - A companion-only prefetch (pipeline manifest plus VAE and text encoder, no transformer) passed the snapshot-partial check, since every file its manifest expected did arrive, and was advertised as on-device although from_pretrained cannot load it. - The single-file flag never reached the picker through the hub inventory path, so a checkpoint-only diffusion repo read as a full pipeline and failed after the handoff. The two pipeline-shape helpers now live in hub/utils/inventory_scan.py so /api/models/cached and the hub inventory classify the same repos the same way. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop adopting an unknown scoped download, leaking raced blobs and resurrecting deleted clips Three items from the latest review round. A scoped download job carries a deliberate file subset, and every file set of one repo rides the same "@scope" slot. A client that adopts a live job from the backend had no file list to compare against: the active-downloads response never carried one, so an adopted job's set was unknown and any later scoped request for the same repo read as "already started". Selecting a different checkpoint then waited on the wrong transfer and tried to load a file nobody fetched. The response now publishes the scoped file list, adoption records it, and an unknown set no longer satisfies a scoped request. A gallery record can be deleted while its blob is still downloading. The delete revokes the URL present at that moment, so the fetch that lands afterwards inserted a fresh object URL for a record no card renders and nothing can revoke: a full MP4, tens to hundreds of MB, pinned for the rest of the session, and once per raced fetch. Both galleries now discard a blob whose record went away, with an epoch covering the video page's Clear all. The video backend keeps the last completed job until the next one starts, and the Video page merges that record on mount to cover a job that finished after the gallery fetch. Deleting the clip left the record in place, so every reload prepended a ghost card whose file request 404s until another generation replaced it. Deleting the clip, or clearing the gallery, now clears the matching terminal record, and the page skips a record it deleted itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Resolve revisions from the live cache, serialize dataset imports, and stop pinning every gallery blob Five more items from the review round. The conditioning-cache revision marker read huggingface_hub's import-time HF_HUB_CACHE constant. Studio can move its cache during a session and loading follows the live setting, so after a move the marker went unresolved (or pointed into the previous root) and pulling a new revision of the same checkpoint no longer invalidated the cache: a warm run could reuse the old encoder's embeddings and the old VAE's latents. It now looks in the active Studio cache first and keeps the environment and the library constant as fallbacks, which the trainer subprocess still needs. The dataset interlock counts mutations rather than excluding them, so two imports of different examples into the same empty name both got past the emptiness check. The winner promoted its staging directory atomically; the loser found the folder non-empty, fell back to a per-file move, and merged its images and captions into the winner's dataset. Imports now take a per-folder lock, a second one is refused with 409, and the emptiness check is repeated under the lock. On Windows the sd.cpp asset resolver filtered only by accelerator token, so a Windows arm64 host matched an x64 zip, downloaded and installed it, and failed later when the binary would not run. It now filters by architecture the way the Darwin and Linux branches do. Every gallery page fetched every PNG up front and kept the object URL for the session, so scrolling a large gallery grew memory without bound for tiles the user may never look at. The Images strip now fetches a tile as it nears view, like the Video strip, and keeps the eager path only where IntersectionObserver is unavailable. A 503 carrying a JSON body comes from the application, not a proxy, so it is surfaced as the error it is instead of entering lost-response settlement and being reported as a request that never reached the server. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the class of a failed generation instead of a bare "Image generation failed." Found on a macOS runner: the native renderer aborts inside its own text encoder there, and the page showed only "Image generation failed." with the sd-server backtrace left in the server log, so nothing about the failure reached the user. The failure is now classified into fixed text, out of memory and native-process death, so the message says what happened and what to try. None of the engine's own output is echoed, since a native tail carries local paths and argv; that stays in the log, and an unrecognised failure keeps the original literal. * Treat an undecodable caption sidecar as the tombstone the trainer sees Uploads store .txt and .caption sidecars as raw bytes, so one can hold invalid UTF-8. The trainer treats any existing sidecar, decodable or not, as an empty tombstone and never falls back to the metadata row for that image. The labeling grid and the dataset summary read an undecodable sidecar as absent instead, so both showed a metadata caption that the run would silently replace with the instance prompt, and counted the image as captioned. Both now track sidecar presence separately, so what the user reviews is what the run trains on. * Keep the reason a native server died, not just its backtrace A ggml abort prints its cause first and then a stack trace, so reporting the last twenty captured lines gave twenty addresses and nothing about the failure: on the macOS runner the native server died on an unimplemented Metal op and the message carried only frame pointers. The captured tail now leads with the lines that name a cause and keeps recent context after them, for both the startup failure and the mid-request death. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop staging the dense text encoder for an fp8 video load Two halves of the same gap, found while measuring the LTX-2.3 download plan: - The video download plan and the scoped pre-download never saw text_encoder_quant. An fp8 request loads a hosted pre-cast encoder, so asking for one still staged and downloaded the base repo's dense Gemma3 (48.79 GB of Lightricks/LTX-2 on the 2.3 distilled pick) that the pipeline then never opened. The plan now drops those shards and stages the pre-cast checkpoint instead; their configs stay, since the pre-cast loader still meta-inits the encoder from the base repo's component config. - The LTX-2.3 assembly builds every component itself, so pipe_kwargs (which carries the pre-cast encoder for from_pretrained) never reached it and an fp8 request silently loaded the dense encoder anyway. It is passed across explicitly now. The dense skip is earned, not assumed: only a pre-cast checkpoint that resolves on the Hub lets the plan drop the dense shards, and only one already fetched to disk lets the pull drop them, so an unpublished or gated artifact leaves both exactly as they were. If injection still fails after that, the load tops the dense weights back up rather than handing from_pretrained a snapshot with no encoder in it. Measured against the real Hub on the 2.3 distilled Q4_K_M pick: 67.24 GB before, 18.92 GB with a 0.43 GB stand-in for the pre-cast artifact (the base entry drops from 24 files / 48.79 GB to 13 files / 0.04 GB). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Match the file's typing idiom for flow_shift models/training.py annotates with typing constructs throughout (105 Optional[...], no Union), and flow_shift was the one place using PEP 604. Union[] reads like the rest of the file, and it also drops the runtime evaluation that would raise on Python 3.9. * Bound the gallery blob cache, and three interlock fixes Four review findings, all reproduced first: - The gallery object-URL caches were unbounded. A clip runs from a few MB to a few hundred, both pages stay mounted after their first visit, and entries were only dropped on delete, so scrolling pinned everything for the session. Both pages now share a byte-budgeted LRU (512 MB video / 192 MB images) keyed off the visibility signal the near-viewport fetching already provides. On-screen media, the selected clip or image, and the item just fetched are never evicted, so eviction is invisible and a single item larger than the whole budget cannot evict itself into a refetch loop. - The image, video and chat load guards ran two independent training probes but returned early when the FIRST one raised, so an unreadable LLM backend disabled the diffusion interlock and a load could proceed straight into an active diffusion trainer on the same GPU. The probes are independent now. - An engine switch swallowed a failed teardown and published the new engine anyway, which is exactly the leak the unload exists to prevent: the arbiter's evictor, /images/unload and the next load all resolve through get_active_diffusion_engine(), so the still-resident pipeline (or a live sd-server) became unreachable and the next load allocated on top of it. The switch now fails and leaves the old engine published, so it stays reclaimable. - The native generation timeout was 30 minutes while the Images page waits up to 6 hours (SETTLE_MAX_MS), so slow-but-progressing CPU jobs died deterministically at the deadline. Measured on GPU-less runners, a 512x512 4-step Q2_K generation took 900 s on Linux and 1465 s on Windows, so larger images or step counts clear half an hour easily. The ceiling now matches the page's window and applies to the whole request: chunks of a split batch share one deadline instead of each getting a full budget. Cancellation is unchanged. Declined: gating the huggingfacenotorch extra off Python 3.9 over the conditional diffusers marker. The marker is deliberate and its comment says why: diffusers dropped 3.9 in 0.38, so pinning >=0.39 outright leaves pip no candidate and the whole extra unresolvable there. The pipelines it names live in studio/backend, which cannot install on 3.9 anyway (studio.txt pins matplotlib==3.10.9 and fastmcp>=3.0.2, both requires_python >=3.10), and the extra is the general core one, so the alternative drops 3.9 for library users who never touch Studio. * Close the load-versus-training-start race, and two picker fixes - The image and video load guards read is_active() and only then selected an engine, acquired the arbiter and registered the load. A /train/diffusion/start reserving inside that window freed residents the load had not registered yet, so the trainer came up beside a brand-new pipeline. The service already had exactly the right pattern for this in dataset_mutation, so gpu_load_admission mirrors it: reserve() refuses while an admission is open, an admission refuses once a start is reserved, both decided under the one lock. The span is only the registration, since begin_load returns as soon as the load is registered and _free_gpu_for_diffusion_training preempts an in-flight load from that point. Chat is deliberately not covered: its load spans an eviction plus a multi-minute GGUF load, and it admits models that fit beside training by design, which is a different contract from the diffusion pipeline's all-or-nothing one. - Hugging Face gives the LTX-2 family the image-to-video pipeline_tag (both Lightricks/LTX-2 and unsloth/LTX-2.3-GGUF report it), so a text-to-video-only filter dropped the flagship audio family out of Video Hub search while the rest of the app routed it to Video. - Task-scoped quant fit sized picks against the LARGEST visible device while resolve_diffusion_device_target returns a bare "cuda" and torch places on the current one. On a heterogeneous host that recommended a checkpoint sized for the bigger card and then loaded it onto the smaller one. Fit now uses the device the load actually lands on; identical on a homogeneous host. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Expose the persistent conditioning cache in the start schema DiffusionLoraConfig has carried cond_cache_dir for a while and the DiT trainer acts on it, but DiffusionTrainingStartRequest omitted the field, so Pydantic dropped it silently and every API-driven run fell back to the in-memory cache that is rebuilt from scratch each time. The warm path skips loading the VAE and the multi-GB text encoders on a rerun whose images, captions and resolution are unchanged, so this was a real capability that could not be reached. Contained like output_dir rather than left to the trainer subprocess's cwd, since it is another directory the trainer writes to. Blank or omitted still means the in-memory cache, so it must not resolve to the outputs root. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix diffusion policy and classification issues from review fp8 auto precision defaulted to precise accumulate on any non-consumer GPU, which made fp8 2.05x slower than int8 on RTX 6000 Ada and slower than not quantising at all. NVIDIA's professional whitepapers do publish equal FP8 rates for both accumulate modes there, so the hardware premise held, but the cost is in the cuBLAS path rather than the published rate. Default to fast accumulate: measured on B200 the flag is a no-op (4096^3 _scaled_mm at 3023.8 vs 3041.8 TFLOP/s, bitwise-identical output, 1.213 s vs 1.230 s end to end), so it is a large win where it bites and free where it does not. Precise accumulate stays available via transformer_quant_fast_accum. Z-Image's DiT is a Lumina2 derivative, so unsloth/Z-Image-GGUF and unsloth/Z-Image-Turbo-GGUF both declare general.architecture = "lumina2" and the whole line was tagged image-diffusion-unsupported and hidden from the Images "On Device" list, though validate_load_request loads them. Resolve shared archs from the repo/file name like bare "wan" already does, with a test asserting the picker and the loader agree for every family. The sage attention on-demand install ran an unpinned `pip install sageattention`, but PyPI's newest wheel is 1.0.6 and diffusers refuses anything below 2.1.1: the install always "succeeded", wrote an unusable version into the running venv, and was rejected on the next line. Carry the dispatcher's floor so pip resolves nothing instead. The dense-quant disk gate sized the download from the bf16-RESIDENT table. The fp32 families download twice that (Z-Image: 23,479 MiB against a 21,970 MiB gate), leaving a window where the check passed and the download filled the disk; Ideogram 4 ships fp8 and was overcharged the other way. Size the gate by published bytes, verified against HF sibling metadata for all 12 families. Patch installs went through unsloth_zoo, which refuses to import unless UNSLOTH_IS_PRESENT is set, and that is set by unsloth itself. The server imports unsloth at boot so it never showed there, but any other process ran silently unpatched with every install returning False, which is 13 test failures on a clean environment. Import unsloth and retry once, memoised per process. Also: the GGUF+LoRA refusal pointed at the native engine without saying a GPU host only selects it under UNSLOTH_DIFFUSION_ENGINE=sd_cpp, so the suggestion was unreachable; the gallery recipe recorded loras from the generate request alone, losing a load-time bake; load-progress claimed "40.07 GB downloaded" for a fully cached load; and pickers.tsx imported three catalog-group helpers it never used. Reported by oobabooga. * Keep the sd.cpp text encoder on CPU under Metal macos-14 loads FLUX.2-klein-4B Q2_K natively on mps and then dies on the first generation with exit code -6: ggml_metal_op_encode_impl: error: unsupported op 'RMS_NORM' -> ggml_abort LLMEmbedder::encode_prompt -> LLMRunner::compute -> GGMLRunner::compute ggml's Metal backend gates RMS_NORM on contiguous rows and aborts the process when that does not hold, with no per-op CPU fallback, so any LLM text encoder (Qwen3 for FLUX.2 and Z-Image, T5 for FLUX.1) takes sd-server down. The encoder runs once per prompt while the DiT runs every step, so pinning only the encoder keeps Metal for the part that matters. UNSLOTH_DIFFUSION_SD_CPP_METAL_TE_GPU=1 opts back in once ggml grows the kernel. * Gate the unsloth retry in the diffusion patch backend The retry added for the clean-environment patch failures is not free: importing unsloth pulls torch in behind it, which costs ~940 MB of RSS measured in a process that had neither, and on a host with no accelerator it fails anyway. A cross-platform CI job that had generated fine at ~900 s later died 19 s in with SIGTERM and every 'if: always()' step skipped, which is the runner being torn down rather than a step failing. Retry only when torch is already imported (true of the server and of anything patching a real module, and the condition that stops the retry from being what loads torch), unsloth is installed but not yet imported, and the first failure was the ImportError the sentinel guard raises. The clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only retry the unsloth import where it can succeed The gate still let the retry run on hosts unsloth does not support, which is where it is most harmful: a 7 GB macOS runner lost the Studio server 26 s into a load, and the Linux runner was torn down mid-generation. Neither MPS nor plain CPU can complete the import, so the retry there pays the cost and fails anyway. Require an accelerator unsloth actually supports (CUDA/ROCm via torch.cuda, or XPU), with UNSLOTH_ALLOW_CPU as the documented override, and hoist the predicate to module level so it is tested directly rather than through the import system. On a CPU-only host the retry no longer fires at all; on CUDA the clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Guard old diffusers, stream video exports, record conditioned recipes Three fixes from review. The 0.39-only pipeline classes (Flux2Klein, Z-Image, Krea 2, LTX-2, HunyuanImage) were resolved by getattr deep in the load, so on the older diffusers that packaging still allows on Python 3.9 -- diffusers dropped 3.9 in 0.38 and this project still supports it, so the 0.39 floor has to be conditional or the extra becomes unresolvable -- an advertised model failed with a bare AttributeError after its checkpoint had already been downloaded. Krea 2 already guarded itself this way; assert_pipeline_class_available now runs the same check for every image and video family from validation, before any fetch, and names the version and the fix. WebM export accumulated the whole VP9 output in a BytesIO and returned it as one bytes object that the response held again. The request caps allow 2048x2048 for 1024 frames, so an export runs to hundreds of MB and concurrent clicks could exhaust the process, while the MP4 route beside it already streamed from disk. transcode_to_file encodes to a temp file and the route returns a FileResponse with a background unlink, so nothing large is resident. A conditioned generation's recipe carried only the txt2img fields, so the gallery presented an inpaint or upscale result as a complete Create recipe and restoring it replayed an unrelated text-to-image request. The images themselves are still not persisted (user uploads with their own lifetime), but the workflow and its scalars are, restore reapplies them, and the toast now names the inputs that have to be supplied again instead of silently landing on Create. Reported by Codex. * Per-load video cancel event, family-gated image picker, cond cache refusal A cancelled video load could resume: begin_load cleared the shared cancel event, and unload() drops _loading without waiting for the worker, so the next load cleared the very object the cancelled worker was watching and its multi-gigabyte pull ran on alongside the replacement until the token check at the end. Each load now gets its own threading.Event, passed down through _fetch_te_prequant and _predownload_base, so a cancelled worker stays cancelled. A cached repo with a model_index.json was advertised as text-to-image on the trust rule alone, but validate_load_request also requires a detected image family, so a trusted pipeline of an unsupported class produced a picker row that deterministically 400s. The picker now applies both gates, mirroring the video branch. cond_cache_dir was accepted for sdxl and then ignored: only the DiT trainer reads it, while the SDXL trainer builds a per-run in-memory latent cache, so the promised cross-run reuse never happened. The route now refuses it with a 400 that names the families which do support it, checked against the resolved family so an omitted model_family with an SDXL base is caught too. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the frontend build broken by the gallery blob cache tsc -b failed on the branch head, so npm run build produced no dist and every platform job fell back to --api-only: blob-url-cache.ts(29,15): TS1294: This syntax is not allowed when 'erasableSyntaxOnly' is enabled dataset-labeling-grid.tsx / dataset-showcase.tsx: Argument of type '{ url: string; bytes: number; }' is not assignable to parameter of type 'string' The cache took its budget as a constructor parameter property, which the project's tsconfig forbids, and fetchGalleryObjectUrl now returns the blob size alongside the URL for that budget, which the two dataset thumbnail components still consumed as a bare string. Declare the field explicitly and destructure the URL at both call sites. tsc -b is clean and vite build emits dist again. * Recover from a ggml unsupported-op abort by restarting on the CPU backend ggml checks every node against the device's supports_op and calls GGML_ABORT when one is not implemented, because a single-backend graph has nowhere else to put it: there is no per-op CPU fallback. The whole sd-server dies with SIGABRT mid-generation and the user gets "the native image renderer stopped unexpectedly" with no way forward. Seen on macos-14 arm64 with FLUX.2-klein-4B Q2_K through the cross-platform CI: the text encoder is already pinned to CPU, and the abort moved into the denoise loop instead. ggml_metal_op_encode_impl: error: unsupported op 'MUL_MAT' -> ggml_abort StableDiffusionGGML::sample -> sample_k_diffusion A retry on the same backend would abort identically, so the load is restarted once with --backend cpu (the only flag that changes which backend executes the graph; --offload-to-cpu moves parameters, not compute) and the generation is re-submitted. The same checkpoint then renders slower rather than not at all. Strictly bounded: the signature must carry both the unsupported-op line and ggml_abort, the device must not already be CPU, and it happens once per load, so an OOM kill or a genuine crash still surfaces as itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix two tests that only fail in a full-suite run The 3.10 CI leg resolves PyAV 17, where av.container.OutputContainer is an immutable C type, so the no-libopus export test died on "cannot set 'add_stream' attribute of immutable type" before it asserted anything. Inject the refusal by wrapping the container av.open() returns instead; modules stay patchable on every build. Removing the injection makes the test fail again, so it still covers the branch it is named for. The Xet shim's degraded-path tests drop utils.hf_xet_fallback from sys.modules and import a throwaway copy. Restoring only the sys.modules entry left the utils package attribute bound to the throwaway, and the two disagreed for the rest of the process: a later monkeypatch of the dotted target patched one copy while the code under test imported the other, so the patch did nothing and test_fetch_te_prequant_only_reports_what_it_downloaded reached the real Hub and got a 401. Restore both bindings. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop an ignored-cancel sd-server, guard deletes during diffusion training, repair unusable managed binaries sd-server does not interrupt an in-flight job, so when it ignores a cancel the grace branch abandoned the poll and reported cancellation while the native job kept a core (or the GPU) busy to completion and held the server's job slot. The comment said the caller stops the server, but only unload does that immediately: a superseding load stops it after its multi-gigabyte download, and a load that then fails never gets there. Stop it here, as the deadline branch already does. DELETE /api/models/delete-finetuned checked only the LLM trainer, so it could rmtree the output directory a live diffusion LoRA run was about to write its adapter into. Consult the diffusion training service too, like the dataset mutation and model-load routes. find_sd_*_binary only checks is_file(), so an interrupted extraction (or a prebuilt for the wrong CPU) left a present-but-unrunnable binary the installer never retried: every load probed it, fell back to diffusers, and native inference stayed off until the directory was deleted by hand. Probe it and reinstall, but only for a copy under the installer-owned root -- SD_CLI_PATH, UNSLOTH_SD_CPP_PATH, an in-tree build and anything on PATH are the user's. * Plan the pre-cast text encoder, and make the cross-trainer GPU admission atomic An fp8 text-encoder request loads a hosted PRE-CAST checkpoint, but the image download plan never received text_encoder_quant, so the manager staged the base repo's dense encoder (FLUX.2-dev's Mistral-24B is ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) and the load then pulled the pre-cast file inline, outside the manager's progress and disk preflight. The plan now takes the field, resolves the hosted artifact with the same resolver the injection uses, stages that file, and drops only those components' dense weight shards. The load's own prefetch takes the same treatment, since it paid the same cost. Only a checkpoint that really resolves on the Hub earns the drop, so a gated or renamed artifact still stages the dense encoder the load will fall back to. The two trainers admitted each other with independent check-then-act guards: the diffusion route checks the LLM backend several network-bound preflights before it reserves, and the LLM route checks the diffusion service well before it spawns, so two near-simultaneous starts could both pass and train on one GPU. reserve() now re-tests the LLM backend under its own lock, and the LLM route holds the diffusion service's gpu_load_admission across its spawn, so exactly one of the two wins. Both halves fail open, so a chat-only install still trains. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the Advanced options a staged download planned against Staging does not set busy, so while a multi-gigabyte download runs the user can still change precision, memory mode, speed or the baked LoRA selection. The pending record held only the repo and artifact, and the completed download fired a load that read the CURRENT state: the staged file set could then be missing files that load needs (fetched inline, with no progress and no disk preflight) or hold gigabytes it no longer uses. One snapshot of every Advanced control is now taken when the plan is built, and it travels with the pending record into the load, so the load that runs is the one the download was planned for. * Do not advertise a family the installed diffusers cannot build The newer families (Z-Image, Krea 2, FLUX.2, LTX-2, HunyuanImage) exist only from diffusers 0.39, and 0.39 cannot be installed on Python 3.9 at all -- diffusers dropped 3.9 in 0.38, so the requirement is conditional or the whole extra becomes unresolvable. On such an environment the picker still offered those rows, every pick failed deterministically, and the error's advice to run pip install -U diffusers could not fix it without also upgrading Python. The cached-repo picker now applies the same availability check validate_load_request does, which is keyed on the pipeline class actually present rather than on the Python version, so it is also right for an intentionally pinned older diffusers on 3.10+. Fails open when diffusers cannot be imported at all: that is a different problem and the load path reports it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore the diffusion engine selection after each router test The active engine is module state, and several tests set it by plain assignment because what _activate does to it is the thing under test, so monkeypatch could not undo it. A leaked ENGINE_SD_CPP left get_active_diffusion_engine() handing back the sd.cpp backend for the rest of the process, and every later route that reads the active engine then saw an unloaded model: eight tests in test_openai_images_generations_route.py returned 503 in a full-suite run while passing on their own. The autouse fixture now snapshots and restores it. * Stream gallery clips, and close three races around them Four fixes from the latest review pass. The video gallery downloaded each clip into a blob before it could play, so playback waited on the whole file (tens to hundreds of MB), seeking was limited to what had arrived, and every viewed clip stayed pinned in the webview. The file route already streams and serves ranges; it just could not be a <video src> because it is bearer-gated. Mint a short-lived signed link instead (its own HMAC secret, 12 hour TTL, separate from the image links) and hand it to the element, which then fetches only the ranges it plays. That removes the blob budget, its LRU and every revoke on this page. The sd.cpp readiness probe accepted any process answering on the port, so a foreign server that grabbed the port between the bind check and the spawn was adopted as ours. Confirm the listener is our child before reporting ready, and stay best-effort (psutil missing, an unknown owner, or any probe error still passes) so the check can only reject a definitely foreign process. Dataset import held its lock for the extract but not for the upload path, so two concurrent uploads into the same folder interleaved; take the same lock and return 409. And reject Windows device names (CON, NUL, COM1..9, LPT1..9, with or without an extension) plus trailing periods in dataset names, which are unopenable on Windows. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Build the image download plan for the engine that will load /images/download-plan always asked the diffusers backend, while /images/load picks the engine per host: a GGUF pick on a machine with no usable GPU routes to native sd.cpp, which reads a single-file VAE plus text encoders and never opens the base repo's sharded components. Measured on unsloth/FLUX.2-klein-4B-GGUF (Q2_K): the plan staged 7.66 GB of FLUX.2-klein-4B components the native load discards, and the 7.80 GB sd-cli actually needs was then fetched inline by the loader, outside the download manager's progress and its disk preflight. Z-Image-Turbo is the same shape. The plan now asks whichever engine the load will select. predict_engine() applies the selection policy without any side effect: it activates nothing (staging a download must not unload the resident model) and only locates the binary rather than installing it, but still counts an installable binary as available, since that is what the load does on a fresh host. The native backend gains a download_plan built from the same _asset_specs the loader fetches, returning the same envelope, so the manager stages exactly the files sd-cli opens. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not let a queued generation outlive the model, and three scan fixes Five items from the latest review; four were real. An unload or arbiter eviction only cancels the generation holding _generate_lock. A second request queued behind it holds no cancel event yet, and Python locks are not FIFO, so it could take the lock the instant the active denoise released it, still see a loaded pipeline, and run a whole new denoise after the model was told to go away: the eviction then waits minutes for it and an image lands after the eject. Unload and a superseding load now raise a fence under _lock before they queue, and a generation that wins the lock while one is pending refuses instead. The cached-model scan judged pipeline completeness across every revision, so a repo holding an older complete snapshot plus a newer companion-only one read as complete while the snapshot from_pretrained actually opens has no transformer. Both scans now look at the revision the loader will open. Deleting a dataset image deleted its caption sidecar unconditionally, which for cat.jpg alongside cat.png removed the caption the survivor still resolves to. The sidecar now goes only with the last image of that stem, matching what the thumbnail cleanup beside it already did. Importing an example into a folder that holds no images but does hold files fell back to promoting the staging dir one file at a time, so an interruption left a partial dataset that the image_count check accepts as complete on retry. Those files are folded into the staging dir instead and the promotion stays a single atomic rename. The MPS generator report does not apply: torch.Generator(device="mps") has worked since PyTorch 2.0 (pytorch/pytorch#91348) and the studio installer pins torch>=2.4. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten diffusion comments Collapse the multi-line comment blocks across the image, video, sd.cpp and diffusion-training code to one or two lines each, and drop comments that only restate the statement below them. Comments only, no code or behaviour changes. * Tighten diffusion comments (second pass) Collapse the remaining multi-line comment blocks in the video page, training routes and service, sd.cpp server and installer, memory and speed planners, and the shared request models. Comments only, no code or behaviour changes. * Tighten diffusion comments (third pass) Collapse the remaining multi-line comment blocks in the attention, cache, LoRA, prequant, precision and compile-cache modules, the sd.cpp arg builder and engine, the video routes, the Ideogram 4 assembly, the model picker, and the diffusion test suites. Comments only, no code or behaviour changes. * Restore the dataset when an example import cannot be promoted Promotion folds the folder's pre-existing entries into the staging dir so the swap is one atomic rename. Every failure after that fold left the user with nothing: the 409 path and an os.replace error both fell through to 'finally: shutil.rmtree(staging)', which deleted the entries that had just been moved in there, while the response said 'Nothing was written'. A same-named entry was also unlinked outright before the promotion was known to succeed. Park superseded same-name entries in a rescue dir instead of deleting them, record every move, and restore all of them if any step of the promotion fails. The fold loop itself is covered too: renaming a non-writable directory raises EACCES on POSIX, which previously escaped as a 500 after the earlier entries had already been moved out. A failed rename now maps to the same retryable 409 as the rmdir conflict. Verified with the reported trigger (a non-empty mode-500 directory whose name collides with an imported file): the folder listing is now identical before and after the failed import. * Drain the teardown fence on a failing unload, give each load its own cancel event Two independent leaks on the image path, both already solved elsewhere in the same file. unload() incremented _teardown_waiters, ran _unload_locked() and decremented, with no try/finally, while the superseding-load path used a finally for the same pair. _unload_locked ends in clear_gpu_cache(), whose CUDA branch calls synchronize/empty_cache/ipc_collect unguarded, and a sticky CUDA fault makes those raise. The count then never drained, so every later generation was refused as cancelled for the life of the process, a fresh load included, since begin_load's own increment and decrement are symmetric. unload() is reached from the chat/video GPU handoff, the engine router and two training routes, so one fault during an ordinary handoff wedged image generation until restart. Release it in a finally. The image and native backends each cleared one shared cancel Event on a new load. unload() sets that event to cancel an in-flight multi-GB download and drops _loading in the same breath, so a replacement load is admitted while the cancelled worker is still inside the fetch, and its clear() re-enabled the very object that worker was watching: the cancelled download resumed and ran alongside the replacement. Take a fresh Event per load and thread it to the worker, as the video backend already does, and set it under the lock since begin_load now rebinds the attribute. * Name utf-8 on the diffusion text I/O and the sd.cpp subprocess pipes tests/test_text_io_encoding.py failed on five files this branch adds. Text I/O without an explicit encoding falls back to the Windows ANSI codepage, so a non-ASCII path or manifest value round-trips corrupted, and the three sd.cpp pipes decode the child's UTF-8 output as ANSI on Windows despite already passing errors = 'replace'. Eleven read_text() / write_text() sites across diffusion_compile_cache, diffusion_ideogram4 and diffusion_krea2, plus text = True on the sd-cli version probe, the sd-cli run and the sd-server pipe. * Record the load-time build on a gallery image's recipe A gallery record documents itself as the image's full generation recipe and is embedded in the PNG, but the only load-related field it carried was the repo id. A GGUF repo holds many quants, so that does not say which one made the pixels, and it says nothing about an adapter baked in at load time. The fallback meant to cover the baked case could never fire: with no loras on the request _adjust_baked_loras zeroes every baked adapter and _active_lora_pairs drops zero-weight entries, so active_loras was always empty. A baked-and-disabled build is not the same pipeline as a never-baked one, so the recipe could not reconstruct the image once the model was rebuilt. Persist model_kind, gguf_filename, transformer_quant and the baked adapter names, read off the load state rather than the request, and show them in the recipe popover. The new fields are optional with defaults, which matters because list_gallery_images drops any record that fails validation, so a required field would have emptied every existing gallery; a regression test pins that. * Drop eleven duplicated comment tails, restore the mxfp8 denial note The comment passes collapsed several wrapped blocks onto one line without deleting the last physical line of the original wrap, leaving the tail of each sentence repeated as its own comment underneath. Two of the eleven were re-worded rather than byte-identical, so a strict suffix match missed them. 6e16ad16f also dropped the line justifying the qwen-image mxfp8 denial while that rule stayed live in _FAMILY_SCHEME_DENY, under a header that then documented only fp8 and nvfp4. Restored. Comments only; verified with the AST gate. * Only let the dense-quant fallback use shards the prefetch actually staged The prefetch skips the base repo's transformer/ shards whenever a prequant checkpoint is expected, since that checkpoint replaces them. But a prequant fetch can fail for reasons the planner cannot see: an unpublished, gated or renamed artifact, a hub 5xx, a proxy, a checkpoint the validator rejects. The loader then fell through to from_pretrained(subfolder = 'transformer') and pulled those shards inside the load lock during 'finalizing', after the previous pipeline was already evicted, where the cancel event has no reach, load_progress has already reported bytes_downloaded == bytes_total, and the cache-disk gate had only reserved the small prequant checkpoint. That is verbatim the situation _dense_quant_prefetch_needed's own docstring exists to prevent. Gate the in-loader dense fallback on the shards being staged, read off the returned file list rather than the request so a failed size estimate closes it too, and let the GGUF build take over otherwise, which is what the prequant-sized replan already does one branch over. It is also the invariant the text-encoder path already enforces: only a component that really resolves may have its dense weights dropped from a plan or a prefetch. Adds the missing coverage for the gate's prequant arm, which the existing disk-gate tests never reached. * Refuse a training output_dir that resolves to the outputs root resolve_output_dir strips a leading 'outputs' and drops '' / '.' segments, so '.', './', './.', 'outputs', 'outputs/outputs' and ' . ' all clean away to nothing and land on the outputs root itself rather than a run directory under it. The DiT trainer then writes pytorch_lora_weights.safetensors flat into the root, where scan_trained_models and scan_checkpoints cannot see it (both filter is_dir()), and a second such run overwrites the first. The UI only checks the field is non-empty, so 'outputs' is one plausible run name away. Refuse it with a 400 that says what to do instead. cond_cache_dir collapses the same way but has an honest 'off' to fall back to, so a root-resolving value now means the in-memory cache, which is what the comment above it already promised: otherwise a run drops one flat safetensors per cached latent and caption into the directory trained models live in. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Repair three defects the main merge left behind Git merged all three files without a conflict, but the result was wrong in each case. The tests only started failing once the merged tree was run. routes/inference.py duplicated the GGUF load block: this branch had moved the gguf_load_in_flight marker and the _hub_download_blocks_gguf_load guard under "if config.is_gguf and config.gguf_hf_repo", and the conflict resolution re-added main's copy at the old position, so both ran. Dropped main's copy; the earlier placement is the deliberate one, so a 409 from the hub guard cannot tear down a resident Images or Video pipeline. test_gpu_selection.py still called _hf_offline_if_dns_dead, which main renamed to _hf_offline_if_unreachable_for (#7591). Disjoint edits, so no conflict, but four route-error tests referenced a function that no longer exists. test_gguf_load_cache_reuse.py anchored its ordering assertion with rindex over "if config.is_gguf:", taking the last one before the load marker. That only held while _resolve_inherited_extra_args sat above every such line; main has since hoisted it above the gpu_ids preflight, so the anchor landed between the call and the marker and the assertion compared against an unrelated later call site. The ordering it checks is a property of _load_model_impl as a whole, so it now anchors on the function. The ordering itself is intact: _resolve_inherited_extra_args, then the gguf_load_in_flight marker, then the hub guard, then the chat handoff, then unload_model. * Stop a real sd.cpp install from breaking its own discovery tests The five "nothing is installed" assertions in test_sd_cpp_engine.py cleared SD_CLI_PATH and UNSLOTH_SD_CPP_PATH and patched Path.home, which covers hops 1, 2 and the fallback half of hop 3. It leaves two hops live. Hop 3 goes through managed_install_root(), which honors UNSLOTH_STUDIO_HOME and STUDIO_HOME and resolves to the stable-diffusion.cpp directory beside the studio home. That is the documented way to run side-by-side Studios, so anyone who has one set gets a real sd-cli back from a finder the test expects to return None. Hop 4 is the in-tree developer build, which does the same for anyone who built sd.cpp inside the checkout. Isolated with an autouse fixture rather than another helper call, because reaching the failure needs no fixture: SdCppEngine(binary = None) runs the finder from its constructor, which is why test_generate_raises_when_binary_missing failed too. The fixture points the studio home and the in-tree root at an empty tmp tree, so every hop is answered by the test rather than by the box. The in-tree root moves into a named in_tree_install_root() so it can be pointed somewhere empty; behaviour is unchanged, including the OSError and IndexError guard on an unexpected layout. * Make the sd.cpp uninstall test actually extract the code it tests The two sed ranges anchored the production fragments at column 0, but both blocks sit inside the main removal function and are indented, so each range matched nothing and LOOP_FILE / DEFAULT_FILE came out empty. Sourcing an empty file is a no-op, so the suite reported 4 passed / 7 failed: the seven removal assertions failed because nothing ran, and the four that passed were 'kept' assertions that pass trivially when nothing runs. The shell job auto-discovers tests/sh/test_*.sh and runs each under set -e, and this file is not in its skip list, so it was a deterministic red. Anchor on optional leading whitespace and fail loudly on an empty or _remove_path-less fragment, so a future reshuffle of uninstall.sh cannot make the suite vacuous again. Now 11 passed, 0 failed, against the real removal loop and the real default-mode block. * Agree on what an engine can build, and on what the installer owns Two gates, each half-applied. The unbuildable-family gate had one caller, the image branch of the cached repo picker. The GGUF classifier, the local-model classifier and the video branch had none, so on a diffusers too old for a family the picker still offered its GGUF and the load then failed. Meanwhile the loader asserted the diffusers pipeline class before engine selection, so a GGUF this host routes to native sd.cpp, which instantiates no pipeline class at all, was refused with an upgrade instruction that could not help. Both are wrong one-sidedly: hide a family only when NEITHER engine can build it, and demand the diffusers class only when diffusers is what will load it. One predicate, family_buildable_here, now answers both, so the picker and the loader cannot disagree. The population is real on Python 3.9, whose diffusers ceiling is 0.36: Flux2KleinPipeline, Krea2Pipeline and LTX2Pipeline are all absent there, and FLUX.2-klein GGUF is a repo in this PR's title. That assertion also raised RuntimeError, which /images/load maps to 409, the status that otherwise means a load is already in progress, and which escaped /images/download-plan (it catches ValueError and FileNotFoundError) as a bare 500 with the message lost. It is an unloadable pick like every other, so it raises ValueError and both routes answer 400 with the text intact. The managed-binary repair used a path test for ownership while the installer uses a marker. On a managed root without the marker, which is any install predating it, the repair deleted sd-server and the reinstall was then refused, permanently, because the surviving sd-cli keeps the directory non-empty so the marker can never be claimed: the user went from an unrunnable binary to no binary and no way back. Require the marker before discarding, which is the same definition of ours that uninstall.sh already uses to keep a user's own stable-diffusion.cpp checkout. A genuinely interrupted extraction still self-heals, since install() writes the marker before it extracts. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add gguf and av to the studio extra so the wheel install matches studio.txt This branch added both to studio/backend/requirements/studio.txt, which is what install.sh uses, but never to the studio extra in pyproject.toml, which is what 'pip install unsloth[studio]' uses. Nothing else keeps the two in sync, so tests/studio/install/test_studio_extra_matches_requirements.py failed deterministically, and the CI job runs the whole tests/ tree. The drift is not only a red test: gguf backs diffusers' GGUFQuantizationConfig and av does the MP4 encode and audio mux, so a wheel install got a Studio that cannot read a GGUF or export a video, which is most of what this PR adds. * Fix two test-suite defects: the CPU-only patch gate and a popped module The patch backend reaches unsloth_zoo's helpers only through an ~940 MB 'import unsloth', which a CPU-only host cannot complete, so every patch install returned False and nine arch/eager/compile tests failed there -- exactly the runners the retry was narrowed to protect. unsloth_zoo only wants UNSLOTH_IS_PRESENT in the environment, which costs nothing and needs no accelerator, and the conftest already sets its sibling UNSLOTH_ALLOW_CPU. Set it there too, as run.py and main.py already do at module scope, so no real server ever took the expensive route. CPU-only goes from 9 failed to 24 passed, 1 skipped; the GPU run is unchanged at 29 passed. test_setup_cache_env_hf_home popped utils.hf_cache_settings to model a fresh process and never restored it, so a later import built a second module object and rebound it on the utils package. test_hf_cache_settings then wrote its setting into one object while core.inference.diffusion read the other, the same split-module failure the xet shim already had. Restore both bindings in teardown: 21 pass together, and each file still passes alone. * Restore the setup_fail assertions the main merge reverted A fourth instance of the class 9541cc535 fixed: the merge took main's studio/setup.sh, which #7644 changed to abort through the setup_fail helper so desktop mode still emits [TAURI:ERROR], but kept this branch's older copy of the test, which still asserted the literal 'exit 1'. setup.sh is byte-identical to main here and the only diff in this file was the reverted assertions, so take main's version. 60 passed. * Fence the video teardown, reject non-finite clipping knobs, stop a test installing sd.cpp The video backend never had the teardown fence the image backend documents. Its unload and its superseding-load path both signalled the active generation, then did 'with self._generate_lock: pass' and tore the state down with the lock free. A generation queued behind that barrier holds no cancel event yet, so the signal cannot reach it, and Python locks are not FIFO: it won the lock the instant the barrier released it, read a still-loaded state and denoised a whole clip against the pipeline being freed. Reproduced on both paths, where the queued generation returned a finished MP4. Mirror the image backend: count waiters, refuse a generation while one is pending, and tear down inside the barrier with the counter released in a finally. _teardown_state becomes _teardown_state_locked since the caller now holds both locks. max_grad_norm and snr_gamma were bounded on one side only, so 1e309 floated to inf and passed. clip_grad_norm_ then computes an infinite clip coefficient, clamps it to 1.0 and scales nothing, and min(snr, inf) / snr makes every min-SNR weight 1.0. The run starts, reports normal progress and trains with the requested knob silently disabled. Measured both. NaN already failed the bounds; allow_inf_nan makes that explicit. learning_rate and flow_shift already guard this exact vector. test_load_routes_to_sd_cpp_on_cpu stubbed ensure_sd_cpp_binary but not ensure_sd_server_binary, which select_and_activate_engine probes first with installs enabled, so the unit test downloaded and unpacked 108 MB into the developer's real ~/.unsloth when UNSLOTH_STUDIO_HOME was unset. Now 0 bytes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Key the video schedule reset on the resolved defaults, not the repo id A GGUF repo holds several variants, so another client swapping a distilled build for the base one from the same repo changes the steps and guidance the backend reports while repo_id stays put. modelChanged stayed false, the page kept the previous schedule, and later generations ran an 8-step distilled setting on a model that expects roughly 40 steps and CFG 4. Include the reported defaults in the key. They only move when the resident artifact does, so a manual steps change still survives a status poll. * Retire the release-lag pin now that 2026.7.6 carries the relaxed gates The virgin Windows container row that installs from PyPI on purpose was tolerated with continue-on-error, because a Server Core container has no Microsoft Store and so no winget, and the released studio/setup.ps1 hard-stopped on a winget-only git gate and reached for winget again for the VC++ runtime. #7549 relaxed both, but it landed on 2026-07-28 and the newest wheel was 2026.7.5 from the 23rd, so the row could not get past it. The pin carried its own tripwire for exactly this moment, and it has fired: unsloth 2026.7.6 shipped 2026-07-29 and the released wheel now installs end to end, so the step reported success and the assertion errored with "delete this pin and drop continue-on-error from the Install step so this row gates". Doing that. Both rows now gate unconditionally, and the assertion that enumerated the accepted failure signatures goes with the pin it protected, since there is no longer an accepted failure for it to describe. The other three release-lag mentions in this file are descriptive rather than tolerated-failure pins: they explain why the fedora and ubuntu-nonroot legs run with overlay: true, and both already refuse a triton failure as a regression rather than accepting it as lag. Left alone. * Pin the hosted pre-quant onto the plain-torch fp8 kernel A Z-Image GGUF pick at the DEFAULT speed mode, once the hosted fp8 pre-quant is actually reachable, dies at generate with an HTTP 500: torch._dynamo.exc.Unsupported: Operator does not support running with fake tensors. Developer debug context: unsupported operator: mslk.f8f8bf16_rowwise.default _fp8_config already pins KernelPreference.TORCH when it BUILDS a config, precisely because the default AUTO switches to the MSLK kernel wherever an mslk package is importable (sm90+). A hosted checkpoint escapes that pin completely: the preference is serialized per Float8Tensor, and all 239 weights in the published Z-Image-Turbo-FP8 checkpoint carry AUTO. Loading it re-arms the very kernel the pin exists to avoid, mslk.f8f8bf16_rowwise has no fake impl, and the first compiled generate therefore cannot be traced. Isolated away from the product to be sure of the mechanism: quantise one Linear three ways on this box and compile each. KernelPreference.TORCH eager ok, compiled ok KernelPreference.AUTO eager ok, compiled FAILS on mslk.f8f8bf16_rowwise library default eager ok, compiled FAILS the same way So the pin is correct and necessary, and the only gap is that the hosted path never got it. _validate_checkpoint checks scheme, granularity, base model, min_features, exclude tokens and fast_accum, but not this. Rewriting the preference on load is safe: it selects a matmul kernel, it is not weight data, so the tensors stay bit-identical and the checkpoint's own state_dict_sha256 still describes them. It is also the faster path compiled, since the opaque extern call blocks inductor quantize fusion. Why this went unnoticed: the pre-quant repos are private, so nothing that could not read them ever took this path. It becomes the default the moment they are readable. Verified end to end at 1024x1024, 8 steps, speed_mode default, on the published checkpoint: before HTTP 500 at generate, diffusion.generate_failed after pinned 239 weights to the plain-torch fp8 kernel, load 10.0 s, transformer_quant fp8 with compiled engaged, cold 7.9 s, warm 0.87 0.87 0.87 0.86 0.87 s That warm number also beats the 1.4 s recorded for this shape on 2026-07-26. 159 prequant and transformer-quant tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cover the fp8 kernel pin so the 500 cannot come back quietly The hosted pre-quant re-arming MSLK was only visible as an HTTP 500 on a compiled generate, which needs a GPU, a readable private repo and an importable mslk to reproduce. None of that is available in CI, so the pin would rot unnoticed. Three hermetic cases on _pin_kernel_preference instead: AUTO weights are rewritten and already-TORCH ones are left alone (counting only what changed), a weight that refuses the assignment does not sink the whole load, and with no torchao enum available the checkpoint is left exactly as saved. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: tighten comments across the image generation, video and training code Rewrite the comments added by this branch to be shorter and clearer: collapse multi-line explanations into one or two lines, drop comments that only restate the code, and keep the rationale that explains why a choice was made. No code changes. * Studio: let the pipeline-class guard survive a host with no diffusers Backend CI installs the CPU-only dependency set, which has no diffusers, and runs without a GPU. 13 tests failed there for three separate reasons, one of which is a real product bug rather than a test-environment gap. assert_pipeline_class_available did a bare "import diffusers", so on any host without it the call raised ModuleNotFoundError instead of the ValueError its own contract promises. /images/download-plan catches only ValueError and FileNotFoundError, so that escaped as a bare 500 with the message lost, which is the exact failure the guard exists to prevent. It is reachable in production: the native sd.cpp engine serves GGUF picks on a CPU or Apple host where diffusers is never installed. Absent diffusers now returns, since the check answers "is the installed diffusers new enough for this family" and there is no version to judge; a pick that genuinely needs diffusers still fails in the loader with its own message. The rest were tests asserting through gates unrelated to what they cover: - Three cond_cache_dir route tests drive a DiT family, which the start route refuses without an accelerator. They now take the existing dit_train_host fixture, written for this case, so they keep testing the schema on every host. - test_download_plan_forwards_the_load_time_controls stubs the diffusers planner but sends a GGUF pick, which routes to the native planner on a GPU-less host. It now pins the engine, so it tests kwarg forwarding rather than the hardware. Engine selection keeps its own tests. - The pipeline-class guard test needed a real diffusers only for its sweep over every shipped family. That half is split out and skips on its own; the stub-driven refusal needs no diffusers and still runs. Added a test for the absent-diffusers contract above. 409 pass on a GPU host with diffusers; 301 pass and 1 skips with diffusers blocked and no GPU. * Studio: cut the image generation comments down further Second pass over the comments this branch adds. Collapse the multi-line blocks that still read as paragraphs, rewrite the longest one-liners so they say the same thing in fewer words, and drop a stale comment that had drifted away from the constant it described. No code changes: only comments and whitespace. * Studio: workflow rail, sidebar Video row, and Train settings polish Sidebar - Pin Video under Images by default; persist store bumped to v3 so an untouched install adopts it while a custom arrangement is left alone. - Add "Customize sidebar" to the end of the More flyout, opening Settings > Appearance scrolled to the sidebar nav section. Images - Replace the workflow dropdown with an icon rail down the left edge: all seven workflows visible, keyboard nav, tooltips carry the labels. - Workflows stay selectable with no model loaded, so one can be set up before picking a model. Generate is already gated on a loaded model. - Video moves out of the Create/Train strip into a link at the far right, with a matching Images link on the Video page. - Wider gutters around the settings column, headings matched to Train. Train - Field guidance moves into "i" tooltips; only state that limits a control stays on the page. Steps, LoRA rank and the rest gain hints. - /info reports params, qlora_vram_gb, gated and note as fields, so the family note renders as chips. vram_note is rebuilt from them unchanged for older clients. Shared - Number steppers appear on hover or focus. - Lighter control border (#e9e9e9) on Images and Video, light mode only. * Studio: move Advanced inline on Images and Video - Replace the top-bar toggle and right-docked Advanced panel with a disclosure under Seed, so load-time tuning sits with the settings it affects and opening it no longer shrinks the preview canvas. - Video uses the bordered variant, since offload and memory decide whether a model fits at all. - Open state persists per page in localStorage. - Tighten the Steps unit trigger: 14px chevron, less right padding. * Studio: full width media pages and layout polish - Drop the 1100px cap on Images Create, Images Train and Video, so the preview canvas grows with the window instead of sitting in a band. - Align the workflow rail to the model selector label, widen the gap before its divider, and restore the divider itself. - Advanced: more room above and below, icon and label sized to the slider rows, and the same quiet row on Video as on Images. - Give Video a pane heading and description, matching Images. - More space above Seed on both pages. - More no longer takes the active style when the current page is one of its own rows. * Studio: fix the Extend side toggles The resting outline used a ring, and index.css blanks the ring on the button holding mouse focus, so the side you just clicked lost its border until focus moved. Use a border for the outline and leave the ring to focus-visible. Also separates the two states properly (the off state had no surface of its own), drops the border in dark mode as the inputs do, and adds the aria-pressed and focus-visible styling the buttons were missing. * Studio: drop the border on the Extend side toggles Light mode should carry no border either, so the fill alone marks the state: a muted surface when off, a primary tint when on. Matches the borderless treatment dark mode already had. * Studio: move the Images workflows into the sidebar The workflow switcher was a vertical icon rail on the page. It now lives in the sidebar under Images, so the page keeps that width for the canvas. - Workflows list under the Images row, with a chevron to fold them away. Hovering the row peeks them in a flyout on the standard menu surface. - Clicking Images while already there toggles the list instead of navigating. - The listed workflow carries the highlight, so the Images row drops it. - Create takes its own icon: it was sharing the New chat pencil. - Model hub moves above Projects. Persisted layouts bump to v4 and only adopt the new order where the stored one is still a shipped default. - Images and Video content both start at 32px, clear of the sidebar. * Studio: simplify the Images workflow submenu Dropping the hover flyout: two ways to reach the same seven workflows, one of them overlapping the list right below it, read as clutter. - The flyout is gone. The workflows are rows under Images and nothing else. - On the Images page they are always open, since they are that page's switcher. Elsewhere they stay folded, and the row's chevron opens them. - Create takes the sparkles icon. - Create and Train both pad their settings column to 40px a side, so the two tabs line up with each other and with the model selector. * Studio: dock the generate action and tidy the Create controls The primary action sat at the foot of a long scroll, so it was off screen until you scrolled for it. - Generate, Video's Generate and Train's Start training float at the bottom of their settings column. No bar behind them: hover lightens the fill rather than thinning it, and the disabled state is opaque, so the controls underneath never show through. - Aspect ratios read as names: Square (1:1), Widescreen (16:9) and so on. - Width and height replace their sliders with two compact boxes. Type a size or pick one from the menu; the value still snaps into range and still drives the locked ratio. - Negative prompt gets a quiet disclosure under the prompt. It was there before but only above guidance 0, which is not the default, so it never showed. - The info "i" is smaller across the settings UI. * Studio: reveal Images and Video field hints on hover The "i" next to every field label sat there permanently, which made a column of settings read as busier than it is. Scoped to the diffusion pages by CSS rather than per component: the hints come from the page's own Field and SelectRow, the train panel's FieldLabel, and the chat ParamSlider, which is shared and should not change for chat. Reveals when the pointer is inside the field, on focus-within for keyboard, and stays inert while hidden. InfoHint grows a data-slot so the rule has a stable hook, rather than keying off its aria-label. * Studio: one-line sliders, edge fades, shared negative prompt - Steps, Guidance and the rest put label, track and value on one row. ParamSlider is shared with chat, so this is an opt-in prop and chat keeps the stacked layout. - The settings columns fade at whichever edge they run past, as the sidebar and model picker do, instead of cutting off. New useScrollFades hook drives all three. - Negative prompt moves to a shared component and Video picks it up, so both pages collapse it the same way. - Right gutter sits closer to the rule, Steps takes a bigger break above it, and the Advanced rule sits between its neighbours rather than up against the field above. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: add the API monitor to the sidebar nav Sits under More, below Export. Points at the monitor page, not the API keys dialog the profile menu opens. No store version bump: an id missing from a stored layout is appended with its default pinned state, so existing sidebars gain the row at the end and keep their own order. * Studio: take main's copy of the sidebar spinner test The merge kept this branch's older version, which pins the Recents row to h-[33px]. Main already relaxed that to any height, since row density moves independently of the trailing column the test is about. * Studio: fix FLUX.2 klein size resolution, HunyuanVideo tier routing, and the personalization save Five fixes found while auditing the model registries against the live Hub. FLUX.2 klein 9B loaded against the 4B config. One family covers both klein sizes and defaults to 4B, relying on the base_model card tag for the real base, but the trust allowlist only had klein-4B, so the correct tag was discarded and a 9B GGUF (inner_dim 4096) was loaded against a 4B config (3072). That surfaced as a bare shape mismatch from inside the GGUF quantizer naming neither the file nor the repo. Adds the missing allowlist entries and a header-level size check that fails early with a legible message; the check is fail-open, so an unreadable file, a non-FLUX.2 family, or an unmapped base leaves the load exactly as it was. The sd.cpp text encoder rule matched the literal "klein-9b", so klein-base-9B was handed the 4B encoder. HunyuanVideo-1.5 720p checkpoints routed to the 480p family. Only the literal 720p_t2v path was aliased, so 720p_i2v and every GGUF repack fell through to the generic token and inherited the 480p base repo, which also supplies the VAE and text encoder. The tier is baked into the weights (transformer target_size 640 vs 960, scheduler shift 5.0 vs 9.0) and the two bucket lists are disjoint, so this ran the whole pipeline off-tier. Resolution presets that the checkpoints were never trained for. Wan2.2 TI2V-5B is 720P-only upstream (SUPPORTED_SIZES is exactly 704x1280 and 1280x704, asserted in generate.py), and the HunyuanVideo 480p square preset 624x624 is not a bucket of its tier; 640x640 is. Personalization could never be saved. The frontend ships eight sidebar nav ids and refills every missing one on each save, but the backend Literal had seven and no "api", so every PUT to /api/settings/personalization returned 422. Aligns the server defaults and the Literal with the shipped layout. A catalog row pointing at a file that does not exist. unsloth/Qwen-Image-2512-FP8 holds torch prequant .pt checkpoints, not qwen-image-2512-fp8.safetensors, and fp8 is denied for this family anyway because it renders black, yet the row was the auto-route target on any GPU above 40 GB. Also marks FLUX.1-schnell, Krea-2-Turbo and the two Ideogram repos gated: all four are gated on the Hub today, and schnell being Apache-2.0 does not make it ungated. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: stop a case-only duplicate upload destroying an image, and size the load against the right GPU Two fixes from the review round. A dataset upload containing two names that differ only by case silently loses one on Windows and macOS. The in-batch duplicate check is an exact string compare, so "Cat.png" and "cat.png" both pass, but on a case-folding filesystem they are one destination: the commit step moves the first staged part aside, writes the second over it, then deletes the backup, while the response still reports both as uploaded. Captions collapse the same way and never even reach the image branch. No single folder can hold such a pair, but the Windows and macOS open dialogs flatten search and Recents results across folders into one multi-selectable list, which is a normal way a LoRA dataset gets assembled. Rejecting the pair everywhere would regress Linux, where the two really are different files with their own sidecars, so the check probes the filesystem once per process instead of keying off sys.platform: macOS also ships case-sensitive APFS volumes, and a Linux host can keep its Studio home on an exFAT or NTFS mount. A failed probe answers "case-sensitive", which leaves today's behaviour untouched. The quantization fit budget was sized against the wrong card under a reordering CUDA_VISIBLE_DEVICES. A bare "cuda" load lands on visible ordinal 0, but the reducer took the lowest PHYSICAL index, which the nvidia-smi path reports as index_kind "physical". Under "3,1" that sizes against GPU 1 while the pipeline loads onto GPU 3. The hook's own interface doc already promised the lowest visible ordinal. Ranks by visible_ordinal, falling back to index only for an older backend that omits it. * Studio: rework the diffusion dataset panel and align the pane gutters Upload: - one Upload button beside the dataset name, with an upload icon - picking files uploads them, no second confirm click - Add button beside the Training images dropdown, so images can go into a set that already exists - fall back to the upload form when a selected dataset no longer resolves Labeling grid: - two columns at any width; sm:grid-cols-3 crushed the tiles in a fixed width column - pin the caption size so the Textarea's md:text-sm does not outsize a tile - name the tile hover group; a bare one revealed every tile's Remove at once - Remove is an icon button on the image, not a word over the artwork - drop the amber tile fill, keep the No caption label - match the header, toggle and status text to the section's type scale Gutters: - the run area and the preview canvas now sit 40px off the rule, the gutter the settings column has off the page edge * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: give the dataset preview per-image hover and remove, and wrap it onto rows - each thumbnail is its own button, so it can carry an X that removes just that image; the strip was one button and could not nest a second control - hover lightens the image instead of drawing a border - wraps onto more rows rather than scrolling sideways, which also keeps the bottom corners rounded - samples 12 thumbnails, up from 8, now that a second row is available - a delete refreshes the panel's dataset counts * Studio: show the workflow icon beside the Images and Video headings Images reads it off WORKFLOW_TABS, so it is the same icon the sidebar submenu shows and it follows the active workflow. Video's heading is also Create, so it takes the Create icon rather than the nav row's film slate. * Studio: reject a duplicate LoRA id on the diffusion load path too DiffusionGenerateRequest already refuses a repeated adapter id, with a comment naming the hazard: _resolve_lora_set suffixes colliding adapter names, so the same id resolves the SAME adapter twice and set_adapters stacks both copies past the per-adapter weight bound. DiffusionLoadRequest bounds only the list length, and it matters more there. Generation-time stacking spoils one image; on the load path the adapters are baked into the quantized build before compilation, so the unintended combination rides every image until a reload. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: teach the GGUF reuse test double about holds_no_vram The resident-GGUF fast path consults llama_backend.holds_no_vram before asserting CHAT ownership, but two tests hand it a SimpleNamespace built before that attribute existed, so they raised AttributeError and returned a 500. A real LlamaCppBackend exposes it as a property; the doubles now carry it too. Product code is unchanged: these pass on main and were the branch breaking its own test, not a defect in the guard. --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: michaelhan <michaelhan2050@gmail.com> Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: Michael Han <107991372+shimmyshimmer@users.noreply.github.com> | 1 个月前 | |
Diffusion: replace the attention perf claims with re-measured numbers (#8021) * Diffusion: replace the attention perf claims with re-measured numbers The three performance claims in diffusion_attention.py came from the stale #7021 and were measured on an older torch/torchao stack. Re-measured all of them on torch 2.12.1+cu130 / B200. One does not reproduce, one was off by a third, and the third turns out to understate the win badly. 1. "~1.18x end-to-end on B200, LPIPS ~0.004" for the cuDNN swap does not reproduce, and cannot: at Qwen-Image's 1024px shape the default SDPA dispatch ALREADY selects cuDNN, so pinning it is bitwise-identical (LPIPS 0.000000, np.array_equal True). Measured 1.02x compiled (1.599s -> 1.572s) and 0.93x eager, the per-call sdpa_kernel wrapper cost that compile folds away. The original claim is self-consistent with an older torch where the default was NOT cuDNN, since a nonzero LPIPS means the output changed, which cannot happen when you select the kernel already in use. Pinning is still right, for a better reason than a speedup: torch's dispatch is a heuristic. FLASH and EFFICIENT at that same shape run 3.9x and 9.0x slower, so pinning is insurance against the heuristic picking one of them on another card, head_dim or torch build. 2. "421 ms with the dense mask vs 19 ms, a ~22x tax" is now 296 ms vs 15 ms, ~20x. The structural claims all reproduce: FLASH refuses a dense bool mask, cuDNN silently falls back, MATH OOMs on the 75.5 GiB score matrix. 3. The trim's END TO END effect was never stated. It is 10.4x: a 121-frame 832x480 10-step HunyuanVideo-1.5 render goes 353.8s to 33.9s, medians of 3, reproduced across two independent runs. Also qualify "exact", which was too strong. No information is discarded, but the render is not bit-reproducible: a masked-to-fused SDPA swap perturbs each step at bf16 rounding scale (one DiT forward on identical inputs differs by 6.6e-3 relative at cosine 0.99998) and 10 denoising steps amplify that, so the video is visibly a different sample. That is the kernel change, not the trim. Rendering the SAME dense-mask path under two different exact SDPA kernels diverges MORE than the trim does (LPIPS 0.303 vs 0.285, SSIM 0.744 vs 0.767 over 13 sampled frames), which is the control that establishes it. Whole-video LPIPS cannot judge a kernel change at this step count; the single-forward relative error is the metric that can, and the comment now says so. Comments and docstrings only; no code change. * Name the real dense-mask fallback: cuDNN, not math The paragraph said a dense bool mask 'forces the slow math path', but the probe this same commit cites shows MATH OOMing on the 75.5 GiB score matrix while the measured dense attention completes in 296 ms. Both cannot be true. Settled by output identity rather than timing, since dispatch overhead makes timings ambiguous on their own. At the production shape (B=1 H=16 N=50345 D=128 bf16) on torch 2.12 / B200: MATH OOM FLASH refuses a non-null mask EFFICIENT runs, 168.25 ms, differs from default by 2.44e-4 CUDNN runs, 296.00 ms, BITWISE-equal to the default's output Default dense measures 296.11 ms, so cuDNN is what the dispatcher picks, on a masked path 20x slower than its own unmasked one. Recorded the EFFICIENT number too: the dispatcher's masked choice is not the fastest available, which is worth knowing before anyone optimises this path further. * Make the cited probe actually reproduce the backend-identity claim The comment concludes cuDNN is what the dispatcher picks under a dense mask, says explicitly that timings cannot establish that, and then cites scripts/sdpa_mask_backend_probe.py as the reproducer. The probe only times each backend. It never compared outputs, so it could not reach the conclusion it was cited for. It now captures the default dispatch's dense output once and reports, per backend, whether the forced result is bitwise-identical to it. Exactly one backend answering yes names the kernel the dispatcher chose. A backend that cannot run the dense mask reports OOM or unsupported instead, so the column never reads as a mismatch when nothing ran. Output on torch 2.12 / B200: backend mask=dense(ms) mask=None(ms) ==default(dense) default(dispatch) 295.98 14.84 yes MATH OOM OOM OOM FLASH UNSUPPORTED (RuntimeError) 51.46 unsupported EFFICIENT 168.21 123.12 no (2.4e-04) CUDNN 295.96 14.77 yes This makes the PR no longer comment-only. The change is confined to a standalone developer probe that nothing imports, and it exists to make a claim in the shipped comment verifiable rather than asserted. | 29 天前 | |
Studio: add image generation, editing workflows and LoRA training with Unsloth GGUFs (#6763) * Tighten comments in the image stack tests and scripts * Close video single-file, training reservation, and image mount-resume gaps Route on-device single-checkpoint video folders through the single_file loader: a bare local .safetensors directory (no model_index.json) is advertised as a pipeline with no filename, so validation rejected it before it could load. Reinterpret the pick as a single_file load of the sole checkpoint, mirroring the image load route. Treat a reserved-but-not-yet-spawned LLM training start as active in is_training_active() so /images/load, /video/load, and /diffusion/start cannot race the reserved run for VRAM during the pre-spawn free window. Mirrors the diffusion training service reservation. Resume an in-flight image generation on the Images page mount: probe generate-progress, re-enter the poll loop, and refresh the gallery on completion so a run started elsewhere is reflected and its saved image appears without a manual refresh. Seed resident image defaults from the resolved base_repo rather than a possibly path-shaped repo_id so the first resident generation uses the right recipe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Publish image generation active state before pre-denoise setup generate() assigned self._gen only at the pipe() call, after deferred compile, LoRA resolution/application, and ControlNet download/build had run. Across that setup window generate_progress() reported inactive even though _generate_lock was held, so a reloaded page's mount probe showed idle and let a second generate queue behind the first. Publish an active step-0 _GenState the moment the generation lock is acquired, before the setup work, and clear it in the outer finally so a setup-time error cannot leave the UI stuck active. Mirrors the video backend's queued phase and the training start guard. * Studio: fix diffusion install ownership, dataset upload atomicity, gallery pagination, and teardown races install_sd_cpp_prebuilt: only write the .unsloth-studio-owned marker when the install created the target directory or it was empty. Adopting a pre-existing, unowned, non-empty directory (a user's own stable-diffusion.cpp checkout) made it eligible for the uninstaller's recursive delete. routes/training upload: make the multi-file promotion transactional. Back up each displaced original and roll every destination back on any failure, so a mid-loop rename error can no longer partially overwrite the live dataset. routes/training _resolve_dataset_folder: reject a symlinked dataset directory and prove the resolved folder stays under the datasets root, so image read/caption/delete cannot escape the root through a link. routes/training delete: escape glob metacharacters in the thumbnail filename so deleting an image named like [ab].png removes only its own thumbnails. image_gallery / video_gallery listing: filter records against the response schema inside the pager via a valid callback, so offset/limit/has_more all count over accepted records. A leading schema-invalid record no longer returns an empty page with has_more=true and stalls infinite scroll at offset 0. image_gallery / video_gallery save: publish via a temp file plus atomic rename (the sidecar is the video pair's commit marker) and clean up on failure, so a partial write never surfaces a truncated PNG or strands an orphan MP4. diffusion_train_common discovery: treat an empty caption sidecar as a metadata tombstone that still falls through to the dreambooth instance prompt, so clearing every metadata caption no longer fails with no captioned images found. diffusion backend unload: wait for an in-flight denoise to exit before tearing down process-wide patches and state, mirroring the load path. diffusion_engine_router: serialize the whole check/unload/publish transition so a concurrent selection cannot return the engine being unloaded. uninstall.ps1: gate the default sd.cpp process stop on the owner marker so a user's own sd-server is not terminated for a directory we then keep. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reject native batch seeds outside the JSON-safe range * Refuse sd.cpp install into unowned non-empty target dir When the install target already exists, is non-empty and lacks the .unsloth-studio-owned marker (a user's own stable-diffusion.cpp checkout, or unrelated files beside a custom Studio root), install() previously still extracted the release into it. Skipping the ownership marker only stopped the uninstaller from deleting the directory; extraction still merged binaries into the user's working tree and could overwrite same-named files. Fail up front with a clear message pointing the user at a fresh/empty location before any download or extraction, leaving their directory untouched. Update the ownership test suite to assert the refusal. * Studio: gate dataset uploads on the symlink check and surface local video single-file checkpoints * Tighten comments and docstrings added by the image-generation fixes * Studio: close arbiter load-registration race and surface native progress + local pipeline folders Publish native sd.cpp generate progress (_gen) before LoRA resolution so a reload probe reads active during setup, matching the diffusers path. Register the diffusion/video GPU load under the arbiter lock (acquire_for now takes a register callback) so a competing acquire cannot evict an owner before its load is marked in-flight and let two loaders allocate VRAM at once. Admit local diffusers pipeline folders (root model_index.json, weights in component subdirs) in the local model scan so they reach task tagging and the On Device picker. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: include marker-owned custom sd.cpp roots in the uninstall stop scan; harden Pester install against the nuget.exe PSGallery bootstrap * Studio: surface local pipeline scan roots, tag single-file checkpoints by filename, mark companion-only pipelines partial - _scan_models_dir: admit a scan folder that is itself a diffusers pipeline (root model_index.json, weights in transformer/ vae/ subdirs). _is_model_directory rejects such a root, so the child scan would list the component subdirs as bogus models and hide the real pipeline; treat the root as one model via _local_pipeline_index. - _local_is_diffusers / _local_model_task: include the sole checkpoint filename in the family-detection needles (_local_family_needles, resolved via resolve_local_single_file). A generically named folder holding one loadable qwen-image-*.safetensors / ltx-*.safetensors identifies its family only from the filename; the load route already resolves that file, so tag it or the task-scoped picker (which rejects task=null) hides the on-device model. - list_cached_models: mark a companion-only base snapshot partial. A GGUF image load prefetches the base repo's VAE / text-encoder / model_index.json but skips the transformer (the GGUF supplies it); the snapshot has a pipeline manifest yet is not a loadable BF16 pipeline, and _cached_repo_partial misses it. _repo_pipeline_missing_denoiser flags a pipeline snapshot whose transformer/ or unet/ component carries no weight, so the picker drops it instead of advertising it as fully on-device. * Studio: preserve foreign gallery files, force safetensors on remote ControlNets, and close dataset/seed/GPU gaps Gallery clear/delete now scope to Studio-owned files: image_gallery and video_gallery skip PNGs / MP4s without a readable recipe (a hand-dropped or orphan file the listing already hides), so clear() and a guessed-id delete no longer destroy files the gallery never surfaced. Remote ControlNets now force use_safetensors: a bare owner/name reaches from_pretrained without the base trust gate, and the Hub scan fails open when unavailable, so requiring safetensors closes the pickle deserialization vector. POSIX uninstall now stops resident sd-server / sd-cli under an owned sd.cpp root before removing the tree (marker-gated), mirroring the Windows stop-before-delete scan; a live native server no longer survives unlinking its binary. Diffusion dataset containment: the training-start read path and the discovery picker route bare names through the protected resolver, so a symlinked dataset is rejected / not advertised like the caption/delete routes already do. Uploads gain the inference decode guard (oversized real images 400 before OOMing the trainer) and dataset upload/caption/delete/import are blocked with 409 while a diffusion run is active. JSONL readers (trainer + routes) tolerate non-object JSON and invalid UTF-8 instead of raising AttributeError / 500. LoRA family compatibility is enforced in the shared resolver, not only the picker, so a direct API client cannot apply a mismatched-family adapter. GPU arbiter gains release_if so the image/video unload idle-check and release are atomic against a concurrent same-owner load's registration. Native batch recipes persist the base batch_seed and restore replays from it, so a native batch_index>0 image no longer advances its seed twice. FLUX.2-klein selects its sd.cpp text encoder by variant (4B -> Qwen3-4B, 9B -> Qwen3-8B) instead of the single family default. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: track the loaded GGUF filename so native companion resolution reproduces the load identity * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: gate local-pipeline image tagging on a real family; validate video sidecars before delete/clear * Studio: tighten image-generation fix comments and docstrings * Studio: gate gallery serve/export on ownership; keep image progress active until persisted; reserve diffusion training before the dataset scan * Studio: restore Reapply target on async image/video load errors; recheck training state before dataset commit; reclaim partial sd.cpp installs on retry Images/Video: a background model load that fails AFTER starting (error/eviction during download) leaves the previous pipeline resident, but handleLoad had already overwritten lastLoad.current with the failed pick, so "Reapply to loaded model" reloaded the failed model. Carry the prior Reapply target into the poller and restore it on the async error/null paths, mirroring the quant rollback. Training: an in-flight diffusion dataset upload passed _require_diffusion_dataset_mutable() at entry but could still commit files after a concurrent /diffusion/start reserved the training slot, mutating the dataset underneath the trainer. Re-check the interlock immediately before the commit phase; a 409 there leaves the staged temps for the finally to clean. sd.cpp install: an interrupted extraction (disk full, killed process, a raising post-extract cudart fetch) left the target non-empty with no owner marker, so the next lazy install tripped the "not a Studio-managed directory" refusal and wedged native install. Write the ownership marker before the partial writes when the target is reclaimable, so a retry recognises the debris as ours and re-extracts. * fp8 DiT quant: floor the dynamic activation scale with activation_value_lb An all-zero activation token row makes the dynamic per-row fp8 scale 0, which turns the quantized data to NaN and the render to black frames on torchao's plain-torch kernel path. The fused fbgemm/mslk quantize kernels clamp zero rows internally, so the bug only reproduces on machines without them, which is most user environments. Zero rows are real inputs, not a corner case: Wan 2.2 zero-pads its text conditioning, and Hunyuan-1.5 and Qwen-Image regenerate zero rows inside their transformer blocks every step. Pass activation_value_lb=1e-12 to Float8DynamicActivationFloat8WeightConfig whenever the installed torchao supports the kwarg (Float8Tensor rework, 0.13+), checked via inspect.signature so older torchao keeps exactly the current behaviour; the existing Float8MMConfig fallback chain is unchanged. Verified on GPU: with the forced plain-torch kernel path a zero-row input NaNs without the floor and stays finite with it, and end to end on HunyuanVideo-1.5 fp8 goes from a solid black frame (LPIPS 1.00) to a normal render (LPIPS 0.225); on Wan the floor matches the condition_embedder exclusion (LPIPS 0.211 vs 0.206). Same-seed renders with fused kernels present are unaffected, and pre-quantized fp8 checkpoints stay valid since weight scales are untouched. * Wire hosted pre-quantized DiT checkpoints into the image families Point prequant_repos for flux.1, flux.2-klein, flux.2-dev, qwen-image (int8 only there; fp8 is family-denied), z-image and krea-2 at the unsloth/<Model>-FP8 Hub repos carrying gate-validated int8 and fp8 transformer checkpoints, so the fast quant path loads the small pre-quantized file instead of materialising the dense bf16 transformer and quantising on device. Measured on FLUX.2-dev int8: build peak drops from 60.7 GB (dense + quantize) to 30.7 GB (hosted prequant), identical 30.7 GB resident after either path since loading a checkpoint is bit-identical to on-the-fly quantisation. The hosted repos name files <Model>-<SCHEME>.pt, so resolve_prequant_source now derives that model-name filename from the repo id (scheme suffix stripped case-insensitively) and carries the legacy transformer_<scheme>.pt as a fallback the resolver tries when the primary 404s, keeping older repos loadable. Wiring a repo also exposed a fallback hazard: with a prequant source present, the dense-fit preflight used to be skipped entirely, so a failed prequant download would fall through to the dense bf16 load the memory plan never budgeted, OOMing after eviction. The preflight now always runs and gates an allow_dense_fallback flag through _load_dense_quant_pipeline: a dense misfit still skips the fast path when no prequant exists, but with one it proceeds and a prequant failure raises to the GGUF build instead of loading dense. The same flag is set when the auto-policy replans an offloaded GGUF against a prequant-sized transient. Tests updated to the new filename convention plus new coverage for the derivation and the legacy-name fallback; the prequant-skips-refit test now asserts the re-check runs and forbids the dense fallback. Verified end to end on GPU: z-image int8 resolves the hosted repo, downloads the model-name file and renders (6.8s load, 5.9 GB peak). * Route krea-2 through its per-component loader on the transformer-quant fast path _assemble_pipe used Pipeline.from_pretrained for every family, but the krea repo ships transformers-5.x configs and no top-level tokenizer files, so the tokenizer dies with vocab_file=None. The pre-quantized checkpoint loaded fine and then the assembly crashed, dropping the load to the GGUF build, which krea-2 cannot take (Krea2Transformer2DModel has no from_single_file). Assemble per-component via load_krea2_pipeline like the pipeline-kind and single-file paths already do. Verified live: Krea-2-Turbo int8 and fp8 hosted prequant loads now assemble and render through the Studio images tab. * Keep Qwen-Image's text-stream linears bf16 on int8 (short prompts break torch._int_mm) Qwen-Image's MMDiT runs every text-stream Linear at M = actual prompt tokens: the Qwen2.5-VL embeds are not padded to a fixed length like FLUX's 512-token T5. A short prompt (13 tokens) or the near-empty negative prompt drives torch._int_mm below its M > 16 floor and the first denoise step raises 'self.size(0) needs to be greater than 16, but got 13' (measured on B200 through the Studio images tab). Add per-family int8 exclusions (txt_in, add_q/k/v_proj, to_add_out, txt_mlp) for qwen-image and qwen-image-edit, threaded through exclude_tokens_for_scheme(scheme, family) and the prequant checkpoint validation, so a checkpoint baked under the old token list is rejected and re-quantised instead of loaded crashing. The text stream runs at M = tens vs the image stream's M ~ 4k, so the exclusion costs nothing; the rebuilt hosted checkpoint gates 28/28 PASS with LPIPS mean 0.057 (was 0.069). * Harden the diffusion memory plan against transient free-VRAM undercounts A cold FLUX.2-dev int8 load on an idle 183 GB B200 planned offload=model (companions exceed budget) and silently served the GGUF as-is; the identical retry went resident and engaged the hosted prequant. The plan arithmetic was byte-identical across both loads (required 90,228 MiB, resident needs free of about 124 GB); the only divergent input was torch.cuda.mem_get_info, which is device-wide and instantaneous: a transient foreign CUDA context briefly held about 100 GB at the first snapshot, and the planner trusted that single read. Three changes: - settled_snapshot_device_memory: on cuda, synchronize + empty_cache (best-effort) and take the MAX free over up to 3 spaced reads. A transient can only shrink free, so the max rejects transient undercounts while a persistent tenant still caps every read. _plan_memory now uses it. - plan_fits_total_capacity + one replan retry: when the dense/prequant candidate fits TOTAL device capacity under the standard reserve and the 0.85 resident margin, an offload verdict can only stem from the free reading, so the loader re-snapshots and replans once before declining the fast path. Explicit balanced/low_vram modes skip the retry (they offload by mode). - diffusion.transformer_quant_declined log line with required/budget/free and the plan reasons, so the next decline is diagnosable from the server log (previously silent). Verified: cold FLUX.2-dev int8 first load in a fresh server now engages the hosted prequant resident (offload=none). * Add FLUX.2 Klein and FLUX.2-dev DiT LoRA training Register flux.2-klein and flux.2-dev in the DiT trainer following the upstream DreamBooth references: latents train patchified and batch-norm normalized from the VAE posterior mode, the packed forward reuses step-invariant position ids, and the guidance vector (3.5) is gated on the variant's guidance_embeds config. Conditioning stacks load per variant (Mistral via Flux2Pipeline for dev, Qwen3 via Flux2KleinPipeline for Klein) and are encoded and freed before the transformer lands on the device. The fused single-stream to_qkv_mlp_proj joins the attention projections in the LoRA targets; the single-stream out projection stays dense because its to_out suffix would also match the double-stream ModuleList container. Wire both families through the training registry (family set, labels, VRAM notes, rank 16 / lr 1e-4 defaults, bf16-only preflight), mark them trainable with train base repos in the family registry, add FLUX.2-dev to the gated-repo token check, and trust both official bases for training downloads. Verified on B200: 30-step klein int8 (19.6s) and nf4 (20.9s) and dev int8 (52.0s) runs train with finite decreasing loss and the saved adapters apply on the bf16 base pipeline (weight 0 reproduces the base image exactly, weight 1 visibly restyles it). * Support LoRA adapters on torchao int8/fp8 quantized image pipelines Adapters are baked at load time: they attach to the dense transformer, then quantize_ converts only the frozen base linears (the lora_ side path is excluded by name), then the loader compiles. Post-quant PEFT injection is not possible on a manually quantized module, so the prequant shortcut is skipped for a baked load and the memory plan is sized for the dense build (force_dense on the quant candidate). At generation time the baked topology is frozen: weight tweaks and disabling (scale 0 reproduces the quantized base exactly) go through set_adapters, while adding or removing adapters returns a clean 400 telling the client to reload with the new selection. supports_lora now returns True for int8/fp8 diffusers loads (checked before the gguf-kind early return, since the quant fast path keeps the picker kind); nvfp4/mxfp8 and GGUF-via-diffusers stay blocked. The load request model takes an optional loras list, threaded through begin_load on both engines (native ignores it and keeps applying LoRA at generation). Verified end to end on GPU: Z-Image GGUF picker + int8 + trained adapter loads through the API, bake marker logged, weight 1.0 vs 0 renders differ visibly, weight 0.5 accepted live, unknown adapter rejected as 400. Affected suites: 296 passed. * Add FLUX.1 Krea dev to the image model catalog Krea's guidance-distilled FLUX.1-dev finetune keeps the exact dev layout, so it runs under the existing flux.1 family unchanged. Wire it up end to end: - Catalog group with the gated official bf16 pipeline and the open QuantStack GGUF quants; the gated artifact is skipped on auto-routing when undownloaded. - Trust the official repo for non-GGUF from_pretrained loads, next to the other black-forest-labs bases. - Generation defaults: 28 steps at guidance 4.5 per the model card. The generic "krea" defaults key (Krea-2-Turbo's 8-step no-CFG recipe) used to swallow the id, which would have produced garbage output; the new flux.1-krea key precedes it on both the backend table and the images page table. - The flux.1 prequant checkpoints are schnell-based; the loader's baked base_model_id validation refuses them for the Krea-dev base, so int8/fp8 requests dense-quantize instead (covered by existing prequant tests). * Resolve pre-quantized checkpoints per base variant One family entry covers several published variants whose weights differ (flux.1: schnell, dev, Krea-dev), but prequant resolution was keyed on (family, scheme) alone, so only the default base could ever be served: the loader's baked base_model_id validation correctly refused the schnell checkpoint for dev and Krea-dev bases and every such load paid the dense download plus on-the-fly quantise. Add an optional prequant_variant_repos table on DiffusionFamily as (base_repo, scheme, repo_id) triples and thread the resolved base repo through resolve_prequant_source / usable_prequant_source and their three call sites (load fast path, memory-plan probe, auto-policy candidate). A base without its own entry keeps returning the family default, preserving the existing refuse-then-dense behavior exactly. Wire the flux.1 variants: the gate-validated unsloth/FLUX.1-dev-FP8 checkpoints (built in the earlier campaign but never reachable) and the new unsloth/FLUX.1-Krea-dev-FP8. * Add the Lumina Image 2.0 family to the image catalog Alpha-VLLM/Lumina-Image-2.0 is a 2.6B single-stream DiT with a Gemma2-2B encoder and a standard 16-channel VAE, all transformers-4.x-compatible, so the generic from_pretrained pipeline path loads it as a new lumina-2 family: - Family entry (Lumina2Pipeline / Lumina2Transformer2DModel), aliased to lumina-image-2.0 / lumina-image-2 / lumina2. No bare lumina alias: Lumina-Next checkpoints are a different arch and must stay unknown rather than crash mid-load. bf16-only upstream, so the fp16 fallback stays off like z-image. - Trust the official repo for non-GGUF loads; bf16 component table entry (ships fp32, ~5.2 GB transformer + 5.2 GB encoder bf16-resident). - Generation defaults 50 steps / guidance 4.0 per the model card, and the generate call passes the card's cfg_trunc_ratio=0.25 itself (family-gated, signature-gated): the pipeline default (1.0) runs the CFG double-forward on every step and oversaturates output. - Catalog group with the single ungated bf16 pipeline artifact (11 GB resident) plus routing assertions; images page defaults row. - No GGUF artifact: none exists upstream (only finetune/LLM quants), so the dense transformer_quant fast path (GGUF-kind-only) stays unreachable for now. Offline probes of the future prequant campaign: int8 and fp8 both engage and render cleanly (fp8 LPIPS 0.11 vs bf16, int8 0.33 from 50-step trajectory drift with intact quality), so neither scheme is family-denied. * Wire the hosted Lumina Image 2.0 int8/fp8 checkpoints Gate-validated against same-seed bf16 renders (28/28 pairs per scheme, zero failures): int8 LPIPS mean 0.146 / SSIM 0.937, fp8 LPIPS mean 0.116 / SSIM 0.946. Uploaded to unsloth/Lumina-Image-2.0-FP8 following the existing checkpoint repo conventions. * Add the HunyuanImage 2.1 family to the image backend The hunyuanvideo-community diffusers mirror carries the full stack in standard layout: a 17B dual-stream DiT (32.5 GB bf16), a Qwen2.5-VL text encoder, a ByT5 glyph encoder, the 32x HunyuanImage VAE, and guider/ocr_guider components (AdaptiveProjectedMixGuidance) that diffusers 0.39 loads natively, so the generic from_pretrained pipeline path covers everything with no per-component assembly. Family notes: - The call's guidance knob is distilled_guidance_scale (there is no guidance_scale kwarg), so cfg_kwarg routes the UI value there; real CFG runs inside the repo's guider at its baked scale. Defaults follow the card recipe: 50 steps, 3.25. - 2K-native: verified live at both 1024 and 2048. - Coexists with the HunyuanImage-3.0 structured exclusion (3.0 has no diffusers pipeline and stays excluded with its stated reason). - int8/fp8 dense quantization verified live (LPIPS 0.186 both vs same-seed bf16); a short prompt does not trip the int8 torch._int_mm minimum on this arch, so no family exclude entry is needed. - bf16 component table for the memory planner: (32.5, 16.3, 0.8) GB. * Surface HunyuanImage 2.1 in the image model catalog Catalog group with the open bf16 mirror pipeline (~50 GB resident, so a bare click on a consumer card routes to the QuantStack GGUF quants, which load and render through the generic GGUF path, verified live) plus the images page defaults (50 steps, guidance 3.25 feeding distilled_guidance_scale). * Add the HiDream-I1 family to the image backend A 17B MoE DiT (16 double + 32 single layers, 4 routed experts) with four text encoders, on HiDreamImagePipeline (diffusers 0.39). One family covers the open Full / Dev / Fast repos (same arch); per-variant generation defaults follow the upstream inference recipes (Full 50 steps at guidance 5, the distilled Dev 28 and Fast 16 guidance-free). The repos name a Llama-3.1-8B text_encoder_4 in their model_index but do not ship its weights; the official example passes the gated meta-llama repo in by hand. The loader instead assembles the component from the open unsloth mirror (byte-identical weights, already inside the non-GGUF trust gate), injected at the three pipeline from_pretrained sites, with output_hidden_states matching the official example. Memory planning counts the assembled TE4: 34.2 GB DiT + 28.8 GB encoders, ~63 GB bf16-resident. * Surface HiDream I1 in the image model catalog One catalog group with the three official bf16 pipelines (Full, plus the Dev and Fast distillations as labeled artifacts) at their ~63 GB resident size, so auto-routing keeps this a datacenter-GPU pick. city96's GGUF is deliberately not wired: the GGUF path would need the same Llama TE4 assembly for very small demand. Images-page defaults mirror the backend table with the variant keys ahead of the generic hidream key. * Pin the measured HiDream quant verdict in tests int8 and fp8 both engage and render cleanly on this family, including short prompts on int8: the routed MoE expert Linears only ever see the concatenated image+text stream (M far above the torch._int_mm minimum), so no deny entry and no family exclude tokens are warranted. Assert that so a future table edit cannot silently regress the measured behavior. * Wire the hosted HunyuanImage 2.1 int8/fp8 checkpoints Verified bit-identical to on-the-fly quantize: all 1264 state dict tensors (456 quantized) dequantize equal between the loaded checkpoint and a fresh quantize_ pass, so quality matches the runtime Dtype path exactly. Same-seed LPIPS suite means (0.35 int8 / 0.28 fp8) blend trajectory divergence with this family's own run-to-run nondeterminism (identical weights and seed reproduce a 17/255 mean pixel delta through the 50-step guider pipeline); per-case hard checks pass and the drift is compositional, reviewed visually. Uploaded to unsloth/HunyuanImage-2.1-FP8. * Fix silent LoRA drop and wasted transformer prefetch on GGUF quant loads Two live-test findings on the images load path: - transformer_quant with baked LoRAs, when the dense quantized build is declined for memory or fails: the load completed as a plain GGUF with the adapters silently dropped (HTTP success, supports_lora=false after the fact) -- wrong output with no signal. The load now fails with the recovery options (drop the adapters, free VRAM, or pick a smaller model). Weight-0 adapters still count as no bake request, and the plain no-LoRA decline keeps its silent GGUF fallback. - A fresh GGUF load on a small GPU prefetched the base repo's full bf16 transformer shards (~47 GB on Qwen-Image) because the dense-quant prefetch widening only checked scheme viability, not whether the device could ever hold the candidate resident. Gate the widening on total device capacity (reserve + 0.85 margin, the plan_fits_total_capacity bar) so a card that is certain to decline the dense build never pays the download; capable devices keep the prefetch. * Fix video progress under-reporting during load and generate Two live-test findings on the video progress endpoints: - load-progress downloaded_bytes froze mid-download: the counter used scan_cache_dir, which skips in-flight *.incomplete blobs, so it sat at the last completed blob for the whole multi-GB shard pull while the disk kept filling. Count the repo's cache directory directly (completed plus incomplete blobs, snapshot symlinks skipped so nothing is double-counted). - generate-progress reported total_steps=null / fraction=0 while step advanced: the video API only carried the native total field while the image API exposes total_steps and fraction, so one poller could not work against both. Derive the image-compatible aliases in generate_progress and declare them on the response model; the native total stays for back-compat. * Wire the hosted HiDream I1 int8/fp8 checkpoints Gate-validated: all 28 per-case pairs pass per scheme (LPIPS suite means 0.291 int8 / 0.278 fp8, in the 50-step trajectory-divergence band; CLIP delta means 0.007-0.008), and the int8 checkpoint is verified bit-identical to on-the-fly quantize across all 1615 state dict tensors (1073 quantized, max abs diff 0.0). Uploaded to unsloth/HiDream-I1-Full-FP8. * Add a pre-cast text-encoder loader for the layerwise fp8 scheme The runtime text_encoder_quant=fp8 path downloads the full bf16 text encoder and layerwise-casts it in place on every fresh load. For the heavyweight encoders (LTX's Gemma3-27B ~50 GB, FLUX.2-dev's Mistral-24B ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) that download dominates load time on a fresh machine. diffusion_te_prequant.py loads a pre-cast fp8-storage state dict instead: meta-init the encoder skeleton from the checkpoint's te_class, load_state_dict(assign=True), rebuild on CPU if non-persistent buffers stay on meta, then re-apply the same layerwise cast to install the upcast hooks. The cast is a deterministic storage transform, so the loaded encoder is bit-identical to dense-load-then-cast by construction. v1 hosts the layerwise fp8 storage scheme only: its state dict is plain tensors (torch.load(weights_only=True), no pickle execution). The dynamic-compute schemes (fp8_dynamic, int8, nvfp4) build torchao subclass wrappers at runtime and are deliberately not hosted. Checkpoints validate format, scheme, component and base_model_id before use and any problem falls back to the dense download and cast. Local path overrides reuse the DiT prequant allowlist env var. Families opt in via a new te_prequant_repos (scheme, component, repo_id) field on both DiffusionFamily and VideoFamily; the field defaults empty so nothing changes until a gate-validated artifact is wired. * Inject hosted pre-cast text encoders during pipeline assembly Wire te_prequant_pipe_kwargs into the three pipeline assembly sites: the diffusion full-pipeline branch, the diffusion transformer-only and GGUF branch (where the companion TE is the big remaining download), and the shared video assembly path before the pipeline/component split. Injection is gated exactly like the runtime cast (mode normalized to fp8, device supported, family not denied), so it can never engage where quantize_text_encoders would not; the later quantize_text_encoders call re-applies the cast idempotently and keeps status reporting truthful. With no hosted checkpoint configured the call returns {} and assembly loads the dense encoder as before. * Add the pre-cast text-encoder checkpoint builder Applies the runtime layerwise fp8 storage cast to a model's dense text encoder once and saves the cast state dict with baked metadata (format tag, base_model_id, family, scheme, component, te_class, versions) in the layout diffusion_te_prequant.py validates. Resolves the encoder class from the checkpoint's config.architectures so the recorded te_class matches what the pipeline instantiates. CPU-runnable: the cast touches storage dtypes only. * Test the pre-cast text-encoder load path Hermetic CPU coverage for diffusion_te_prequant: the checkpoint filename convention, family-table resolution by scheme and component with malformed entries skipped, resolution priority (path override, hosted repo, none) and the fp8-only scheme gate, the checkpoint validation matrix (wrong format, missing state_dict, wrong scheme, wrong component, wrong or missing base_model_id) with base case folding, the local-path allowlist refusal and missing-file fallback, and the assembly injection gating (mode, hosted entry, device support, family deny, load failure, successful injection). Also pins the te_prequant_repos field on both family dataclasses and that no family ships a hosted TE checkpoint until the campaign wires one. * Fix pre-cast TE checkpoint loading and engagement reporting Two bugs found while building the hosted checkpoints: - The builder recorded torch.__version__ (a TorchVersion object) in the checkpoint metadata, so torch.load(weights_only=True) rejected every artifact and the loader silently fell back to the dense download. Record plain strings. - Re-applying the layerwise fp8 cast to an injected pre-cast encoder raised on the duplicate hook registration, making quantize_text_encoders report the engaged cast as failed (status showed no TE quant while the encoder ran fp8). _cast_fp8 now returns early when the hooks are already installed. Also corrects the LTX TE size note: Gemma3-12B stored fp32 (~49 GB), not 27B. * Wire the hosted pre-cast fp8 text encoders qwen-image and flux.2-dev (diffusion) and ltx-2 (video) now resolve a hosted pre-cast fp8 text encoder from their unsloth -FP8 repos: - unsloth/Qwen-Image-FP8: Qwen2.5-VL-7B, 16.6 GB dense -> 8.8 GB - unsloth/FLUX.2-dev-FP8: Mistral-Small-24B, 48.0 GB dense -> 24.7 GB - unsloth/LTX-2-FP8: Gemma3-12B, 48.7 GB fp32 store -> 13.2 GB Every checkpoint verified bit-identical to dense-load-then-cast (729 / 585 / 1066 tensors, zero mismatches) and smoke-tested through the real backends with the repo engagement marker. Tests cover the wired entries, the resolver filenames, builder metadata weights_only survival, and the idempotent re-cast. * Report the compute dtype on fp8-cast encoders and inject the pre-cast TE on the dense fast path Two more findings from the hosted-TE GPU smokes: - Module.dtype reports the first floating parameter, which after the layerwise fp8 cast is the fp8 STORAGE dtype. Flux2 derives its prompt embed and latent dtypes from encoder.dtype and feeds them to randn_tensor, which has no fp8 kernel, so ANY flux.2 load with text_encoder_quant=fp8 crashed at generation (pre-existing, runtime cast included). The cast now swaps in a subclass whose dtype property reports the compute dtype; forward behaviour is unchanged. - The dense transformer_quant fast path assembles companions through _assemble_pipe, which never received the pre-cast TE injection, so the hosted encoder engaged on full-pipeline and GGUF builds but not on the fast path. Threaded through like the other two branches. Verified live on B200: qwen-image (full pipeline), flux.2-dev (GGUF picker with int8 DiT prequant), ltx-2 (video backend) all engage the hosted TE, render non-black, and report text_encoder_quant=fp8 truthfully. * Key the fp8 cast idempotency on an explicit completion marker Hook presence alone cannot distinguish a legitimately pre-cast text encoder from leftover hooks after a cast that failed mid-pass, so the early return now requires the completion marker _cast_fp8 sets once the hooks are fully installed. Leftover partial state keeps failing closed. Also tolerates non-Module encoder doubles in the hook probe and the dtype override. * Extend the fp8 TE quant to HiDream's Llama text_encoder_4 The generic quantize_text_encoders pass only covers text_encoder.._3, so HiDream's HEAVIEST encoder (Llama-3.1-8B TE4, 16.1 GB bf16) always stayed dense. TE4 is assembled separately (hidream_te4_kwargs), so the fp8 path now lives there: when the requested TE quant is layerwise fp8 and the device/family qualify, TE4 prefers the hosted pre-cast checkpoint (unsloth/HiDream-I1-Full-FP8, 8.6 GB) and falls back to dense-load-then- cast; a mid-pass cast failure reloads a fresh dense encoder instead of shipping partial state. The pre-cast loader and builder gain config_subfolder/config_overrides for standalone encoder repos whose config sits at the root and whose pipeline needs forward flags (output_hidden_states/attentions). Verified on B200: bit-identity 291 tensors (225 fp8, 0 mismatches), hosted checkpoint engages through the real backend (marker + status fp8), load 24.3 s vs 48.0 s dense, LPIPS 0.133 mean over 3 same-seed pairs vs the dense-TE render (gate 0.25), non-black frames. * Correct the ltx-2 resident TE estimate to the bf16 cast size The memory plan's bf16_components_gb held 50.4 GB for the LTX text encoder, which is the fp32 hub store of Gemma3-12B (~49 GB download), not what sits on device: the pipeline loads it torch_dtype=bf16, ~24.4 GB resident. The 26 GB over-estimate pushed the auto plan toward offload on cards that fit the real footprint. Comments and the size-table test now pin the resident semantics. * Host pre-cast fp8 text encoders for four more families Round 2 of the hosted TE set, each bit-identical to dense-load-then-cast and gated through the real backend (marker + status fp8 + same-seed LPIPS vs dense TEs): - FLUX.1 T5-XXL (text_encoder_2): 9.52 -> 5.90 GB, one artifact for schnell/dev/Krea-dev (T5 shards byte-identical across all three, verified sha256). 220 tensors, 144 fp8, LPIPS 0.109. - Lumina Gemma2-2B: fp32 hub store 10.46 -> 3.20 GB (3.3x download cut). 288 tensors, 182 fp8, LPIPS 0.041. - Z-Image Qwen3-4B: 8.04 -> 4.41 GB. 399 tensors, 252 fp8, LPIPS 0.112. NOT shared with flux.2-klein-4B: klein retrained layer 35's MLP (verified tensor diff, maxdiff 0.86), so klein hosts no entry. - Krea-2 Qwen3-VL-4B: 8.88 -> 4.83 GB. 713 tensors, 460 fp8, LPIPS 0.082. The constructor-assembled krea pipeline takes the encoder directly (load_krea2_pipeline text_encoder kwarg); the loader remaps 5.x rope_parameters and re-ties weights after assign so the rebuilt encoder matches the builder's structure. HunyuanImage 2.1 reuses the Qwen-Image artifact outright: its Qwen2.5-VL text encoder is byte-identical (every shard sha256, 16,584,414,544 bytes), recorded in the new component-level base-equivalence table the checkpoint validator consults. The injection loop now covers text_encoder.._3 so a family can host several components. Live check: LPIPS 0.123 vs dense. * Report the fp8-cast compute dtype without swapping the encoder class The dtype override swapped encoder.__class__ to a dynamic subclass, which breaks transformers' kwargs-based output recording: a fp8-cast Qwen3VLModel stopped returning hidden_states and every krea-2 generation with text_encoder_quant=fp8 crashed at encode_prompt (regression from the HiDream TE4 change; caught by the krea hosted-TE live smoke). The override is now a property shadowed on the ORIGINAL class that prefers a per-instance compute-dtype attribute, so class identity is preserved and uncast instances keep the stock behaviour. The idempotency test now pins exact class identity and the uncast-sibling fallback. * Pass the calibrated distilled sigma curve to LTX-2.3 8-step runs The 22B distilled DiT was trained against ltx_core's fixed DISTILLED_SIGMA_VALUES, but the diffusers scheduler derives 8-step spacing from resolution-shifted flow matching and lands far off at every reachable mu (second sigma 0.945-0.981 vs 0.99375, tail 0.37-0.61 -> 0.1 vs 0.725 -> 0.42 -> 0). At the distilled default step count the backend now passes the list verbatim, neutralising the scheduler's dynamic shift and terminal stretch for the call (they distort even explicit sigmas) and restoring them afterwards. Other step counts and the dev/base DiT keep the scheduler's own spacing. Live-verified on B200 through the video branch backend: the scheduler holds the exact curve after an 8-step distilled GGUF generation, config restored, healthy clip. Also reword the transformer_quant resolved reason to the measured reality: quant halves resident weights and hosted checkpoints cut load time, while per-step speed is roughly bf16 parity. * Pin the fp8 weight-quantize kernel against silent MSLK switching torchao's Float8Tensor KernelPreference defaults to AUTO, which switches the weight-quantize kernel to MSLK whenever an mslk package is importable on sm90+. Measured on B200: that changes fp8 scale rounding bitwise (8/8 FLUX matrices differ, scales ~55 percent of bytes), so a box that merely gains mslk would break the hosted-prequant bit-identity invariant; the mslk path is also slower under torch.compile (opaque extern call blocks inductor's quantize fusion, FLUX.1 fp8 e2e 1.149 to 1.624 s). Pin KernelPreference.TORCH explicitly, matching current no-mslk behaviour bit for bit; signature-gated for older torchao. GPU-smoked (finite, rel err 0.037) and pinned by test. * Shift Qwen-Image training sigmas to the inference distribution Qwen-Image's scheduler skips its static shift under use_dynamic_shifting, so the DiT trainer was drawing UNSHIFTED uniform-schedule sigmas for it (mean sigma 0.50) while inference always runs the exponential mu = log 3 shift plus the shift_terminal 0.02 stretch. Add a flow_shift config lever: "auto" (the new qwen-image default) rebuilds the training sigma table through the scheduler's own time_shift and stretch_shift_to_terminal so the draw matches the inference distribution exactly (mean sigma 0.72); a numeric value applies the standard linear shift s*u/(1+(s-1)*u); 1.0 keeps the historical identity behavior and stays the default for FLUX, Z-Image and Krea 2. The model timestep conditioning follows the shifted sigma, gathered in fp32 so bf16 rounding never skews it. Also wire two opt-in levers with off defaults: cfg_dropout (per-sample empty-prompt conditioning dropout, encoded alongside the captions before the text encoders are freed) and weighting_scheme="bell" (bsmntw-style mid-schedule Gaussian loss weighting normalized to mean 1). Verified with two 80-step rank-8 bf16 LoRA runs on Qwen/Qwen-Image (identity vs auto, same seed): both converge with finite decreasing loss and produce coherent same-seed previews. Unit tests cover the exact transform, the shifted sampling distribution, per-family defaults and config plumbing. * Add LoRA EMA, a persistent conditioning cache, and aspect bucketing helpers diffusion_train_extras hosts the opt-in training extras: LoRAEMA shadows only the trainable adapter params (warmup-ramped decay, default 0.99, exported as a second adapter under output_dir/ema), PersistentConditioningCache stores latent posterior stats and caption embeddings as safetensors keyed by content hash + family + resolution, and the aspect-ratio bucketing helpers group mixed-aspect datasets into same-area divisor-snapped shapes. The DiT trainer wires the first two behind config flags that default to the current behavior: ema_decay (0 disables) and cond_cache_dir (None disables). A fully warm cache skips loading the VAE and text encoders entirely; a cache hit is bit-identical to a fresh encode, including the per-channel qwen latent normalization. Also fixes the stale _gather_sigmas call in the perf test that still passed the scheduler instead of the sigma table. * Tighten torchao configs and note the FSDP2 design for the DiT trainer nf4 loads now enable double quantization (~0.4 bits/param off the frozen base scales at no fidelity cost), fp8 training uses the rowwise recipe when the torchao build ships it (per-row scaling confines the DiT activation outliers that a tensor-wide scale collapses), and the inference quant filter gains a per-scheme GEMM-tiling divisibility floor (16 for scaled_mm, 32 for MX blocks) so one ragged Linear cannot crash the first denoise after a clean quantize pass. plans/fsdp2_diffusion_design.md records the multi-GPU design: bf16/fp8 over FSDP2 with per-block units, LoRA attached before sharding, int8 out of scope (DTensor over the quantized subclass is undefined), per-family notes. * Batch diffusion inference with per-image seeds, an inference conditioning cache, and GGUF loader fixes Batched generation: /images/generate takes a prompts list (one image per prompt, txt2img only) or a seeds list (one prompt, one image per seed); the legacy batch_size path derives per-image seeds base..base+n-1 like the native engine. Every image gets its own torch.Generator so any batch member replays alone from its gallery recipe; the whole list runs as one forward by default with OOM backoff that halves a failed chunk, and an explicit batch_size caps images per forward. Validated 10-22x over serial engines on 32-image suites with LPIPS deltas within 0.002. Conditioning cache on the inference path: UNSLOTH_DIFFUSION_COND_CACHE_DIR (the inference sibling of the trainers' cond_cache_dir, same persistent store) wraps encode_prompt so repeated prompts skip the text-encoder forward entirely; verified bit-identical outputs. Bypassed while LoRA adapters are attached; tensor-argument calls pass through uncached. Compile cache: GGUF loads fingerprint their own bundles (quant=gguf, a different compiled graph than the dense family) and batched calls register every distinct (w, h, batch) chunk shape they ran, so the heavy GGUF batched warmups (~159 s at batch 32 on 12B-class, ~655 s on 20B CFG-batched) are paid once ever. GGUF loader: strip the sd.cpp model.diffusion_model. container prefix in the single-file converter; diffusers' FLUX.2 converter KeyErrors on it and the Qwen-Image identity mapping strands the model on meta. * Correct batched seed-replay docs to match measured behavior Same-seed images at the same batch shape are bit-identical; a solo regeneration with the recorded seed matches its batched rendition up to batch-size-dependent kernel numerics (mean abs pixel delta about 2.5/255, LPIPS delta under 0.002), not bit-exactly. The previous wording overclaimed bit-identity across batch shapes. * Note that batched bit-identity assumes a settled compiled graph The first generation issued while the deferred compile is still in flight can deviate transiently (observed once on a cold fp8 build: mean abs pixel delta 0.063/255); once the graph is settled, same-seed same-batch-shape images are bit-identical across runs. * Studio sidebar: Image03/FlimSlate icons, More flyout, Train row, New pills - Images uses Image03Icon and Video uses FlimSlateIcon. - New "More" row (MoreHorizontalIcon) opens a right-side flyout on click or hover holding Video, Recipes and Export; the close is delayed 180ms so the pointer can cross the gap. Its SidebarMenuButton deliberately takes `title` rather than `tooltip`: with `tooltip` the button returns a Tooltip root and DropdownMenuTrigger asChild would hand its ref to a non-DOM node. - Dropped the "Train" section heading; Train is now a top-level row between Images and More. data-tour="navbar" moves to the surviving nav group so the product tour keeps its anchor. - "New" pill beside Images and (inside the flyout) Video, via NavBadge. * Studio sidebar: match flyout rows and New pills to the existing scales - More flyout rows dropped their sidebar-row typography and size-icon override, which fought DropdownMenuItem's own scale (text-sm, gap-2.5, px-3 py-2 and size-4 icons) and rendered oversized glyphs and text next to the nav. - New pill reuses the brand "beta" badge recipe (nav-badge font, --ui-font-scale sizing, nav token colours) rather than hardcoded 9px values. - The More row's native title tooltip (an OS box on hover) is replaced by the app's Tooltip, wrapped around DropdownMenuTrigger so both triggers compose onto the same button, and shown only on the collapsed rail like other nav rows. * Settings: pin and reorder the sidebar navigation Adds a "Sidebar navigation" section to Settings -> Appearance, above the existing profile-menu customizer, with the same drag-to-reorder + switch UI. - New sidebarNav preference: one { id, pinned } entry per navigable row (projects, hub, images, train, video, recipes, export), array order = render order. Defaults match the shipped layout, so an untouched install is unchanged. - Unpinning moves a row into the More flyout rather than hiding it, so no page becomes unreachable. New chat and Search stay fixed as actions. - app-sidebar now renders from one navRows descriptor map, so a pinned row and its flyout counterpart cannot drift; the More row appears only when something is unpinned and highlights off whatever it actually holds. - Mirrored in the backend PersonalizationCustomization: without it the model's extra="ignore" would drop the field, and because sync replaces local state with the server's copy once customization is saved, the user's pin order would reset on the next sync. The validator dedupes and back-fills like sidebarMenu but preserves the client's order, since here order is meaningful. Frontend typecheck, i18n parity and catalog checks pass; 32 personalization tests pass, including a round-trip asserting a reordered list survives a save. * Sidebar customizer: drop the Search row, skip More for a lone item - Search is reached from the top bar, so it is no longer previewed as a fixed sidebar nav row; New chat stays. - More now appears only when it would hold two or more rows. A single unpinned row renders inline in its saved order position instead: a flyout wrapping one item costs a click and earns nothing. The customizer's More preview follows the same threshold. * Sidebar settings: hide a lone unpinned tab, match New chat icon, rename Profile menu - With exactly one tab unpinned, both More and that tab are dropped, so nothing is drawn for it (previously it rendered inline). The page stays reachable by URL. - The customizer's New chat preview uses PencilEdit02Icon, the icon the real row renders; Edit03Icon was a different glyph. - "Sidebar menu" is now "Profile menu", described as the shortcuts behind your name at the bottom of the sidebar, so it no longer reads as a second name for the navigation section above it. * Tighten comments in the new sidebar and delete-guard code * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio sidebar: keep the More row highlighted while its panel is open Moving the pointer into the flyout left the row unhighlighted while the panel stayed open. The row now carries data-menu-open, added to the nav hover selector list. Not data-state: the tooltip and menu triggers both write that attribute, so whichever lands last wins. * Images: use the shared pill toggle for Create/Train and pad the panels - Create/Train was the only segmented control on its own Tabs styling. It now uses PillTabs, the same control as the model picker and Hub toggles, pinned to the header row's 34px. PillTabs takes an icon per tab, so the inline-span workaround for TabsTrigger goes away. - pt-3 on both the Create and Train panels, which sat flush against the model selector row. * Images: make the workflow picker a dropdown instead of a 7-up strip Seven workflows in a 340px rail left ~48px each, so the labels crowded and the hints were only reachable as title tooltips. The strip is now a dropdown: the trigger shows the current workflow and its hint, and each row carries its own description. A row the loaded model can't run is disabled and shows the reason in place of the hint, so the gating explains itself. Adding a workflow no longer shrinks the others. * Images: workflow icons, hint under the trigger, more top room, unclipped Train cards - Each workflow carries an icon, shown on the closed trigger and on every row. - The trigger is one line (icon plus name). The selected workflow's description moved below it, where it reads like the Field hints further down the rail. - pt-6 instead of pt-3 on both Create and Train, so the cards clear the model selector row. - The Train right column scrolls while its cards use ring-1, which draws outside the box and was clipped at the scroll edges. p-px gives the ring room. * Images: one-line workflow rows, stronger trigger fill, roomier mode tabs - Dropdown rows are icon plus name only. The selected row's description already shows under the trigger, and a disabled row keeps its reason as a title. - Trigger fill moves to the bg-foreground/[0.07] dark:bg-foreground/[0.12] pair the hub cards use, so it reads against the card in both modes. - Description under the trigger goes from text-ui-10 to text-ui-11p5. - More horizontal padding on the Create / Train tabs. * Images: drop card borders for the composer shadow, keep scrollbars inside, use app controls in Train - Cards lose ring-1 for .panel-soft-surface: the composer's shadow in light, flat in dark, matching .chat-composer-surface and the menus. - Both rails now clip (overflow-hidden) with the scroller inside, so the scrollbar can't ride over the rounded corner. Same shape video-page already uses. - Train's 9 native selects become the app Select, so they no longer open an OS-native menu, and the native file input is hidden behind a Choose images button that reports the count. - Image previews use explicit 8-10px radii: this theme sets --radius to 1.1rem, so rounded-md was 15.6px and the thumbnails read as circles. * Images: one card for controls and preview, chat sliders, wider softer shadow - Controls and preview were two floating cards; they now share one card split by a divider. The Advanced dock stays separate since it toggles. - SliderField wraps Chat's ParamSlider, so the sliders match Chat (label row with the value, full-width neutral track) instead of a green track with a spin box. All 14 call sites keep their props. - panel-soft-surface goes from 0 2px 8px -2px /0.16 to 0 4px 22px -6px /0.10: lighter, spread wider. * Images: flat Create and Train panes, hover-only scrollbars, tidier Train dataset step Both Images tabs now sit on the page background like the Hub: no card, no shadow, no bounding box. A single rule divides the controls rail from the preview canvas (Create) and from the run area (Train), and the settings and previous-runs sections read as panes rather than nested cards. Also: - Scrollbars in these panes use the existing hover-scrollbar recipe, so the thumb only shows while the pane is hovered. - Workflow rows explain themselves with a tooltip after a short hover, which also works on disabled rows, and the descriptions are much shorter. - Training images rows are name plus image count; the license stays on the example card. - The upload step loses its dashed box, the buttons match the sizes around them, and Upload only appears once files are picked. - The empty preview uses the same icon as the Images nav item. * Images: full-height panes, wider settings rail, Create/Train offset from the selector The rule between the panes now runs the whole page height (the row drops its bottom padding and each pane pads its own content), the settings rail is wider on both Create and Train, and the Create/Train switch sits further right of the model selector. * Images: put both tabs on the Hub's centered measure Top bar and content now share mx-auto max-w-1100 with px-5 / sm:px-8, so Create and Train sit at the same width and position as the Hub instead of running edge to edge. * Images: restore the top bar position, drop the panes lower under it * Images: center the mode switch, flip the arrow with the orientation, app tooltips everywhere The Create/Train switch is centered on the page instead of trailing the model selector, with wider buttons. The flip control's arrows now rotate with the orientation and its label says which way the flip goes. Every native title tooltip on the page is now the app's tooltip, so they all get the rounded surface instead of the OS box. * Images Train: plainer field text, no green buttons, columns that stop colliding - The dataset name, trigger prompt, adapter name and custom base fields now say what they are in plain words instead of leaning on example values. - Import, Upload, Back, Back to settings and Train another are outline buttons, not green ones. - Example thumbnails are landscape tiles, so photos are not cropped to chunky squares. - Settings cells get min-w-0 and the select value truncates, so a long option like the nf4 label no longer widens its column into the next one. - The number stepper sits a little further in from the field edge. - Create and Train are wider. * Images Train: roomier example cards with Import on the thumbnail row * Video: same treatment as the Images tabs - No cards: the rail and the canvas sit on the page background, divided by a rule that runs the full page height, on the Hub's centered measure. - Wider rail, chat's sliders, hover-only scrollbars. - Every native title tooltip is now the app's tooltip, including the clip cards. - Reapply and Cancel are outline buttons, the empty state uses the Video nav icon, and the clip tiles are less rounded. * Images and Video: narrower generation rail, matching Train headings Create and Video rails go from 392px to 368px. Train a LoRA and Training settings are now the same size and both in the heading font: the h2 already picks it up from the base rule, so the settings header opts in with font-heading and the weight that rule pins. * Images Train: shorter copy throughout Family notes, example descriptions, precision labels and every helper line are trimmed so they stop wrapping to three lines and colliding with the next column. The nf4 label now fits its select without truncating. * Images Train: a little more spacing between field groups * Images and Video: tighten code comments * Fix training start NameError, the load-order guard test and CPU-only diffusion tests - start_training forwards resume_source_run_id to _start_training_impl, which reads it. Without it every start raised NameError. - Restore main's anchor in the load-marker order test: the file now has an earlier `if config.is_gguf:`, so indexing the first one compared the wrong branch. - The two diffusion tests that reach diffusers now skip when it is absent, matching the CPU repo-test env. - The UI smoke finds nav rows that live in the sidebar's More flyout. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Treat a null metadata caption as no caption str(None) stored the literal "None" as the caption, so a null row counted as captioned and would have trained on that text. Also drops an unused import. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix invalid-UTF-8 500s, the flat Canny map and the dropped DiT knobs read_text raises UnicodeDecodeError, which is not an OSError, so one bad caption sidecar or video sidecar 500d the info, upload and gallery routes. A flat image now yields the all-black edge map instead of its own luminance, and the four DiT loss knobs the trainer implements are declared so model_dump keeps them. * Use the ui font-size tokens instead of raw px text utilities text-[11px] and friends ignore the UI font size preference, which the repo's font-scale contract test enforces. Same rendered size at the default scale. * Fix diffusion dataset 500s, the dropout-1.0 no-op run and the reset base pick Four correctness fixes on the training side: - The labeling grid read caption sidecars under except OSError, but a non-UTF-8 sidecar raises UnicodeDecodeError (a ValueError), so one bad file 500d /diffusion/dataset/{name}/images and the grid could not be opened to repair it. Read it as no caption, matching the info summary. - An image past Pillow's own hard limit raises DecompressionBombError, which derives straight from Exception and so escaped the upload guard's (OSError, UnidentifiedImageError, ValueError) and returned 500 instead of the intended 400. - lora_dropout accepted 1.0, which makes PEFT build nn.Dropout(p=1.0): lora_A and lora_B receive no gradient and the run saves an untrained adapter while reporting normal progress. Bound it below 1.0, matching the LLM request schema. - The train panel re-seeded the base repo on every dataset refresh because the family object identity changes on each info fetch, so an upload or caption save silently replaced the user's chosen base and the run started on a different model. Track the pick and only re-seed on a real family change. * Show the retained failure when a video page mounts after a failed job Mount-time recovery handled only phase=completed, so reloading the page after a multi-minute generation failed left an idle view with no diagnosis: the backend keeps the terminal failed record only until the next job, and nothing else survives the reload. Surface it the same way the poll does, filtering the cancelled sentinel. * Fix batched generation crashes, cache keying and unreplayable recipes Four bugs in the batched inference path, all found by review: - A mixed-prompt batch sent a scalar negative prompt against a prompt list. Z-Image asserts on the length, and Qwen-Image, Krea 2 and FLUX true-CFG encode a batch-1 negative against batch-N latents and fail in the transformer's text/image concat. Broadcast it to match the batch. - The FBCache step-cache reset sat above the chunk loop. diffusers only resets that state at the end of a successful call, so a forward that raised (the OOM the backoff is meant to recover) left its own residual behind and the halved retry died on a shape mismatch. Reset before every forward instead. - The conditioning cache keyed on the checkpoint alone, but a GGUF or single-file load takes its text encoders from the companion base, so the same checkpoint against a different base reused the previous base's embeddings. Key the base too. - Gallery records stored the base seed and the requested batch size even when a prompts/seeds list drove the run, so restoring the second image of seeds=[5, 99] replayed seed 5. List-driven outputs now record as single-image recipes on their own seed. Also bound strength above 0: every img2img pipeline derives its step count from it, so 0 leaves zero denoising steps and either raises or, on SDXL, crashes on empty latents. * Fix quantized-load LoRA bake, prequant family exclusions and outpaint canvas Six review findings across the Images page and model scanning: - The quantized (int8/fp8) load path can only attach LoRA adapters before quantization, but the frontend load request had no loras field, so every generation after such a load was rejected and each reload repeated it. Send the selection with the load. - build_prequant_checkpoint passed no family to the scheme exclusions while recording the family in metadata, so a Qwen int8 artifact baked the short-M text-stream linears and was then rejected wholesale by the loader's family-keyed check. - Registering a bare single-file checkpoint directory produced no On Device row even though the images loader can load it; only its parent worked. Admit that shape when nothing else matched. - Unload left the Reapply target set, so the repair path was skipped and Reapply reloaded the ejected model. Clear it, as the video page does. - Both FLUX.2 bases were trusted for training but not inference, so Deploy to Create rejected every FLUX.2 adapter. - Outpaint allocated the grown canvas before downscaling, exceeding the browser canvas area cap on a large photo; an over-cap canvas is unusable, so Extend silently posted a fully transparent image and mask. Scale the source first. * Send the picked GGUF filename with the quant so diffusion loads fire The variant expander emitted only the quant label, and nothing else in the frontend set ggufFilename, so the Images and Video pages could never take their GGUF branch: both gate it on meta.ggufVariant and meta.ggufFilename, then fall through to the single-file path, which returns because the id is a repo id and not a .gguf name. Every quant pick was a silent dead click, with no load request reaching the backend. The filename was already on the variant row (the picker keys its list on it, and the variant validator requires a non-empty string), so thread it through the click handler. The chat path is unaffected: it reads ggufVariant and never needed the filename. * Version the conditioning cache key and reject non-finite flow_shift Two correctness fixes: - The cache keyed the checkpoint and its companion base by name only, so a Hub repo advancing to a new commit, or a local directory updated in place, kept returning embeddings from the previous text encoder. Pair both with a revision marker: the locally resolved commit sha for a Hub repo, config plus text-encoder file stats for a directory. Neither loads the encoders, so a warm run still keeps them off the GPU. - flow_shift only checked positivity, but JSON accepts 1e309, which floats to inf, and inf <= 0 is False while NaN fails every comparison. The sigma table then evaluates s * u / (1 + (s - 1) * u) as NaN, which poisons every sampled sigma and saves a corrupted adapter while progress looks normal. Require a finite value. * Keep curated models listed, guard the video companion repo, pin diffusers Three review findings: - The picker filtered every catalog member out of Recommended and Hub search on the way to canonical group rows, but nothing renders those rows yet (catalogGroupFitsDevice and groupMatchesQuery are imported and unused). A task-scoped picker's models list is catalogToModelOptions(), i.e. group members exclusively, so both lists came back empty and no curated model could be discovered or downloaded. Keep the artifacts listed until the grouped UI exists. - The video delete guard compared only repo_id, so deleting the companion base of a loaded GGUF video model was allowed even though it supplies the VAE and text encoders. Compare base_repo too, matching what the images guard already does for its companions. - diffusers was declared unversioned while the diffusion stack requires 0.39 (Krea2Pipeline, the cache_context child registries, the Flux2 and Z-Image pipelines), so an upgrade could keep an older release and selecting an advertised model failed until the user upgraded by hand. * Namespace the trainer conditioning cache per checkpoint, bound the learning rate - The trainer keyed its persistent conditioning cache on family and resolution only, while the keys themselves carry just the caption or image content and crop variant. One cache directory reused for two checkpoints, or for the same repo at a new revision, let a warm run skip loading its encoders and train on the other model's embeddings and latent statistics. Namespace on the base checkpoint and its resolved revision as well. The revision helper now lives beside the cache in diffusion_train_extras and the inference wrapper delegates to it, so the two cannot disagree about what counts as the same source. - The diffusion learning rate only checked positivity, but 1e309 floats to inf and satisfies gt, so the route evicted the resident models and started AdamW with an infinite rate: the first step destroys the adapter while progress looks normal and the result is saved. Bound it below 1.0, matching the LLM schema, which rejects inf for the same reason. * Fix GGUF image model picks doing nothing, and pick the train base in the top bar The quant rows never forwarded the .gguf filename, so every hub GGUF pick on Images/Video fell through to a silent return. On Train the top bar now picks the training base instead of a generation model, which is GGUF-only and untrainable. * Pin diffusion and video loads to the live HF cache root Both read huggingface_hub's import-time HF_HUB_CACHE, which changing the cache folder does not update: progress counted the old root while the download wrote to the new one, and from_pretrained could split one model across both. * Add the diffusion download plan endpoint Reports the repos and exact files a pick needs so the download manager can stage them with the loader's own file scope. A plain snapshot would add the packaged root single, transformer shards and fp16 twins the loader never opens. * Add a file-scoped flavour to the Hub download job Lets a consumer that reads a deliberate subset of a repo stage it through the normal download manager. Keyed as "@scope" so it never collides with a quant or with the repo's full snapshot, and the file list rides the registry so an XET to HTTP retry respawns the same scoped job. * Stage image and video downloads through the Hub download manager They downloaded inline inside the load, so they had none of the manager's disk preflight, manifest verification, resume or panel progress. Picks now stage as scoped jobs carrying the loader's own file list, then load from a warm cache. * Fetch staged GGUF checkpoints as scoped jobs, and stop calling diffusion models unsupported A GGUF entry went out as a full snapshot, whose ignore list drops *.gguf: the job finished at once having fetched only docs, and the repo landed on device unloadable. Every entry is scoped now. The Hub also no longer tags image/video models as unsupported (they run on their own pages), and those pickers name what they select. * Apply the picker task filter to local model sections LM Studio, ./models and custom-folder rows ignored it, so the Images picker listed chat GGUFs that 400 on a diffusion load. The backend already tags every local model with a task for this purpose. * Route a chat pick of a diffusion model to the Images or Video page Chat cannot load one, so it was either hidden or failed on load. The unfiltered picker now lists on-device diffusion models and navigates to the page that runs them, passing the repo and quant so that page loads it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Route the real GGUF filename, keep non-GGUF curated models, key scoped downloads by file set Five review findings, four of them ways a click did nothing or fetched the wrong thing: - A chat pick of a diffusion model routed ggufVariant (a label like Q4_K_M) in the search param the target page uses verbatim as the GGUF filename, so the load asked for a file that does not exist. Route ggufFilename; no filename means a curated non-GGUF pick, loaded as a pipeline. - The task-scoped pickers kept only GGUF repos, so the catalog's bf16, bnb-4bit and single-file fp8 artifacts could not be discovered or downloaded on the Images and Video pages even though loadSpecFor knows how to load them. Keep curated artifacts whatever their format, in Recommended and in Hub search. - Both pages deduplicated routed selections on the model alone, and they now stay mounted, so picking the same repo again -- another quant, or the same one after chat evicted it -- returned early without loading or clearing the query string. Key on model and quant. - Every scoped image download shared one @diffusion job key regardless of the requested files, so switching quant mid-download adopted the running job: the UI waited on the first file set, then loaded a file that was never fetched. Include a digest of the file set in the key. - A scoped plan silently dropped requested files missing from Hub metadata, and snapshot_download succeeds when an allow pattern matches nothing, so the job reported completion and triggered a load with required files absent. Fail the job instead. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the scoped download key derivable, and stop the hidden page hijacking a route Four review findings, the first a regression from my own last commit: - Keying scoped download jobs by a digest of the file set broke the download manager: it builds that key client-side (it polls and cancels before any response tells it a key), so it watched and cancelled a key no worker owned and never fired its ready callback. Keep the derivable "@scope" key and refuse the second request instead when a live job on the slot is fetching a different file set -- decided inside the registry claim, under the lock, so a concurrent claim cannot slip past it. The manager records the file set on the job as well, so a sibling quant's transfer is not adopted locally either. - Both diffusion pages read the route query through a loose useSearch and both stay mounted once visited, so the hidden one consumed the other's ?model=: it navigated back to its own route and tried to load, say, an image checkpoint as a video model. Only the visible page consumes it. - The staged download plan was built without the configured HF token or the Advanced values the load itself sends. The token matters most: the backend's Hub metadata lookup is best-effort, so a gated base silently planned no companion entry and the load pulled those multi-GB files inline, outside the manager. The memory/quant controls decide whether the base transformer/ shards are needed at all, and the route dropped memory_mode, cpu_offload, the prequant path and the LoRA selection before asking for the plan. - The video preview kept playing after leaving the page: the keep-alive layout only hides it, and display:none does not pause a media element, so a clip the user unmuted kept its audio going over the next page. Pause on the active transition and do not auto-replay while hidden. Also completes the hand-built request bodies in the hub download tests: the scoped-files field this branch added to the route read as an AttributeError against them, failing five tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Serialize the GPU handoffs, gate DiT training on a GPU, and keep 3.9 installable Six review findings, three of them evict-then-fail orderings: - The chat load reclaimed the GPU without telling the arbiter it existed. A chat load holds no llama-server process until its GGUF has downloaded, which is minutes, so a competing Images/Video acquire in that window found nothing to cancel, took the GPU, and the chat load then spawned onto the same device. It now registers an in-flight marker through acquire_for's register hook (under the arbiter lock, as the image and video loads do), the evictor cancels a marked load, and the route undoes itself if ownership moved while it loaded. - The Hub-download conflict check ran after that handoff, so a GGUF the download manager already owns destroyed the resident Images/Video pipeline and then 409'd, having loaded nothing. It moves above the handoff, together with the marker it handshakes with. - The image load released the engine router's transition lock before registering the load, so a second load choosing the other engine could unload the still-idle engine this one captured; the load then landed on a deactivated engine, where generate, status, unload and the arbiter's evictor can no longer reach it. Registration now happens under that lock and refuses if the engine changed. - Training a DiT family on a host with no GPU was accepted: nf4 is not a CPU fallback, its 4-bit load goes through bitsandbytes, which requires CUDA, XPU or MPS. The start unloaded the working Images pipeline, pulled the text encoders, and only then died in the child. Rejected before the teardown now, and /info stops advertising a precision that always 400s. SDXL keeps its documented fp32-on-CPU path. - Both diffusion pages kept the routed-pick marker forever, so re-picking the same checkpoint (after chat evicted it) neither loaded nor cleared the query string. The marker is released once the query is gone. The Images key also carried a stray NUL byte, which made the file read as binary to grep and other tooling. - diffusers dropped Python 3.9 in 0.38, so the unconditional >=0.39.0 pin left pip no candidate at all on 3.9 and made every install that composes the huggingface extras unresolvable there. The floor is conditional now. Also fixes tests that were already red on the branch: two hand-built request fakes had gone stale against fields this branch added, and the handoff-ordering test only failed on a host with fewer than two GPUs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the GGUF variant contract test against the merged handler signature The assertion pinned the exact single-line call handleVariantClick(v.quant, v.downloaded, expectedBytes, v.filename), but the handler takes (quant, filename, downloaded, sizeBytes) and prettier wraps the call across lines, so the mandatory repository test job failed on every push. Match the call structurally and assert the filename really is forwarded in the handler's argument order. * Stop a background page and a stale record taking the GPU or a download with them Five fixes from a review pass over the diffusion work. delete-finetuned rmtree'd a model the Images or Video engine was holding: every guard on that route is chat-only, and Images loads any local path, so deleting a local diffusion model under the storage root pulled the weights (and the companion VAE / text encoders sd.cpp re-reads each generation) out from under a live pipeline. The cached-model route already refuses this; the trained/exported one now does too, matching by path rather than repo id, and failing open on a chat-only install so it cannot block ordinary deletes. A staged download finishing while its page was hidden loaded the model and evicted whatever the user was actually using: both diffusion pages stay mounted behind the router and a load takes the GPU unconditionally. The pick is now held until its page is on screen again, which is also what chat does. A scoped download could report success having fetched nothing. With Hugging Face metadata unavailable no manifest is written, so verification is a no-op, and snapshot_download returns an existing snapshot folder without downloading when its own repo_info call fails. A repo already on disk from a full snapshot job (which ignores *.gguf) therefore completed with no weights and auto-loaded against them. The requested file list needs no network, so it is checked against the disk directly. The XET to HTTP retry reclaimed the job slot without the scoped file list, and that claim overwrites the stored record, so a later identical scoped start compared an empty list against the real one and 409'd instead of adopting the running download. The DiT accelerator gate probed torch.mps.is_available(), which only exists from torch 2.5 while the supported floor is 2.4. All three probes shared one try/except, so on torch 2.4 the AttributeError read as 'no block' and a CPU-only host still evicted the resident pipeline, downloaded the encoders and died in the child. Each accelerator is probed on its own now, through torch.backends.mps. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stage what an LTX-2.3 load reads, keep a routed file's load kind, drop an unbakeable LoRA Three from the latest review. The video download plan always asked for the wide base file list, so an LTX-2.3 pick staged the 2.0 base's VAEs, vocoder and connectors that the checkpoint supplies itself, while the companion files the 2.3 assembly does read were left out of the plan and pulled inline at load, outside the panel's progress, cancel and disk preflight. The plan now recognises a 2.3 pick by name (the load keeps the authoritative header probe, and under-guessing only falls back to the load-time pull), narrows the base list, and stages the extras in the same entry as the checkpoint so one repo stays one scoped job. A pick routed from the chat picker arrives as ?model= and ?quant= with no picker metadata, so a bare local .gguf or .safetensors was loaded as a pipeline: an explicit model_kind wins over the backend's filename sniffing, so it evicted the resident model and then failed on the missing model_index.json. Both pages now derive the load kind from the path, the same way their own picker handlers do. A torchao int8/fp8 build takes adapters only at load time. Switching artifact inside one family keeps the LoRA selection, since the family did not change, but the load did not bake it, so the next generation was rejected with 'reload the model with the adapter selection' while the picker still showed the adapter as active. The selection is now dropped once per resident build, with a message saying to pick and load again. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cancel an evicted safetensors load, spare the arbiter for CPU-only chat, fetch clips lazily Four fixes from the latest review round: - The GPU arbiter's chat evictor only cancelled the llama.cpp side. The orchestrator publishes active_model_name once its worker reports success, so an in-flight safetensors load was visible only as an entry in loading_models and finished onto the GPU after ownership had transferred. Cancel every pending load, and give the safetensors branch the post-load ownership recheck the GGUF branch already had. - A manual gpu_layers=0 GGUF load runs on the CPU with the GPUs hidden from the child, yet it took the arbiter unconditionally: it cancelled a running image or video generation for a model needing no VRAM, then held CHAT ownership so the next GPU workload unloaded it for nothing. Gate the acquire on the same predicate the launch-time CPU-only mask uses, as the image and video loaders gate on their resolved device. - The staged-download hook subscribes per repo, not per job, so another job on the same repo advanced the staged queue (starting a load whose scoped files were still downloading) or wiped a queue that was still running. Compare the variant each callback carries, like the chat page's auto-load does. - The video gallery fetched every record of a page into an object URL that lives until the page closes: 50 clips at tens to hundreds of MB each, for cards the user may never scroll to. Fetch a clip as its card nears the strip's edge, plus the selected one the player needs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim the comments across the diffusion backend Comment-only pass over the Python this PR touches: drop what the code already says, collapse multi-line explanations that still read on one line, and keep the reasoning that is not recoverable from the code. No code, docstring semantics or behaviour changes; verified with an AST comparison against the previous revision, and the backend suite is unchanged (same 37 environment failures as before: the API integration tests that need a live keyed server, the flash-attn install hooks, and the GPU memory fields). * Invalidate latents on a VAE swap, keep a cut-off generation, surface the EMA adapter - source_revision() scanned the checkpoint root plus text_encoder/tokenizer but not vae, so swapping or fine-tuning the VAE in place left the conditioning cache namespace unchanged and a warm run trained against latents from the old checkpoint. Include the vae directory, like any other component the cached tensors come from. - /images/generate answers only when the images are saved, and secure mode's tunnel caps an origin response near 100 seconds, which a native CPU or a high-step run passes routinely. The page reported failure while the work kept running, and a retry would duplicate it. A lost response (fetch rejection or a gateway status the origin never answered) is now told apart from a refusal: the page waits out generate-progress and reloads the gallery, so the run it started still lands. - The trainer emits the EMA adapter's path with the terminal event, but the state update dropped it, so neither the run history nor either response schema carried it and an enabled EMA left nothing discoverable. Keep it, and show it next to the primary adapter. - weighting_scheme advertised a choice of timestep sampling; sampling is always logit-normal and the flag only selects the bell loss weights. Describe what it does. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Hide unloadable cached rows, hold the dataset interlock, bound a GIF export - The cached-model listing tagged any repo with a model_index.json as text-to-image, so a community pipeline the image loader's trust rule refuses still got a row in the Images picker, and a detected-but-untrusted video repo fell through to that same tag. Gate the image tag on the load path's rule and hide an untrusted video repo outright. - A routed diffusion pick only carries a GGUF filename, which is all the chat picker has, so a curated single-file artifact arrived with no quant and was loaded as a pipeline: from_pretrained on a repo with no model_index.json. Pass the page's own catalog spec into the route pick, so a routed pick resolves to exactly what a direct pick on that page resolves to. - The dataset mutation endpoints checked is_active() and only then handed their filesystem work to a thread, so a start reserving in that gap changed captions or removed images underneath the preflight or the running trainer. The interlock is now registered for the whole request under the lock reserve() uses, and a start refuses while a mutation is open rather than waiting on it. - GIF export held every kept frame as a paletted image before encoding; a clip may be 2048x2048 for 1024 frames, and at the 12 fps target the step is 1, so one export click could allocate over 4 GB and take the backend down. Downscale past 720 px and widen the step to keep at most 300 frames. - seed accepted any Python int, so an out-of-range one passed every preflight, evicted the resident models, spawned the trainer and only then died in torch.manual_seed. Bound it to torch's 64-bit range in the request and config. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the accelerator probes in the DiT family-metadata tests Six tests read family_train_infos() (or a start preflight) without pinning the host probes, so they only held on a machine with a bf16 accelerator: on a GPU-less runner the DiT gate empties precision_modes, turns supports_compile off, and replaces any other preflight message with the no-accelerator note, and all six failed there. A conftest fixture pins both probes for exactly those tests, so they assert the family metadata they are about on every host. The gate's own CPU-only behaviour keeps its dedicated tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Carry the pipeline task into the hub inventory the pickers read - The task-scoped pickers filter On Device rows on a task, and the chat picker routes a diffusion pick by the same field, but those rows come from the /api/hub inventory, which never carried one: the Images and Video pickers listed nothing on device and the routing never fired. Both cached scans and the local listing now tag rows with the classifiers the models API already uses, the schemas and the frontend adapter carry it through, and a row the backend classified as a generation task is exempt from the chat-only guard that was also dropping it. - The local routing map was keyed by model_id while the row click passes id (a filesystem load id for a models_dir or LM Studio entry), so the lookup missed and the pick fell through to the chat loader. Key both. - A staged download whose start answered "error" left its head in place, where the effect never re-runs and onReady never fires, so the pick was stranded until the user reselected. Clear the queue and say so. - Every scoped pick in a repo shares the @diffusion variant, so the variant alone cannot tell two file sets apart: restaging while the first job finished let its completion pass for the new pick and load a checkpoint that had not downloaded. Bind the callbacks to the repo + file set they started, and to the staging generation. - A rejected generate POST does not say whether it reached the backend, so an immediately idle progress read was ambiguous and a submission that never landed looked like a finished image. Require evidence: progress seen active, or a gallery record that was not there before the POST. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the OpenAI image URL fetchable, keep WebM audio, stream example imports Four review items on the diffusion Studio work: - response_format=url returned the bearer-gated gallery route, which a standard image client downloads with no Authorization header, so the default response format was unusable. Mint a short-lived HMAC link instead (the shape RAG already uses for pdf.js) served by a signed route, and leave the gallery route itself bearer-only. - A manual gpu_layers=0 load carrying speculative_type="off" -- a value the UI persists and sends -- read as GPU-bearing, so it took the GPU arbiter and evicted a resident image/video pipeline even though the launcher hides the GPUs for it. Canonicalize the mode and exempt "off". - The curated example import prepared the whole split before the loop stopped at the 10-100 image cap; m1guelpf/nouns is 49,859 rows / 328 MB. Stream instead, with the prepared load kept as a fallback for a repo that cannot stream. - WebM export dropped the audio track an LTX-2 clip carries, silently, on the format offered for web embeds. Mux it as Opus through a resampler + FIFO, and keep exporting the video alone on a build without libopus. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not stub out triton on a GPU host when the Xet backend fails to import The lazy loader retries `import unsloth_zoo.hf_xet_fallback` under UNSLOTH_ZOO_DISABLE_GPU_INIT=1 whenever the first attempt raises. That flag makes unsloth_zoo take its MLX/CPU path, which injects triton and bitsandbytes STUBS into sys.modules for the rest of the process. On a working GPU box whose first import failed for an unrelated reason (a bitsandbytes/CUDA mismatch, say) the retry succeeds, so Studio boots looking healthy and then dies at the first CUDA-only kernel: a GGUF or compiled diffusion generation hits the stub and returns NotImplementedError: Unsloth: 'triton.tools.experimental_descriptor.enable_in_pytorch' was called on Apple Silicon / MLX, where triton is stubbed out. so every image generation 500s with an Apple-Silicon message on a Linux CUDA host, while the load reports success. Found by loading Z-Image-Turbo GGUF through the API on a box where bitsandbytes could not initialise. Gate the retry on the host genuinely having no accelerator. The Xet stall watchdog is optional and already degrades with a warning; a process whose triton is stubbed out is not recoverable. The warning now says why it did not retry. * Fix the lost-generation proof set, the settle timeout and the hub inventory's diffusion gates Seven fixes from the latest review round on the Images page and the hub cache inventory. Images page: - The lost-POST settle path built its "already seen" gallery id set inside the catch, after the request failed. By then the earlier runs of the same batch had already prepended their records, so run 2 could accept run 1's image as proof that its own request reached the backend. The set is now captured once before the first POST and grows with every record the batch produces. - settleLostGeneration fell out of its SETTLE_MAX_MS loop and returned normally, so a wedged generation was counted as done and the next run started against a busy backend. It now throws on timeout. - Restoring a recipe cleared the ControlNet selection but left the workflow tab and the init / mask / reference images pointing at whatever was loaded, so the next Generate conditioned on an unrelated image. It now clears all of them and returns to Create. - The download plan omitted the adapter selection the load itself bakes in. A baked LoRA forces the dense build path, so the plan described a different file set than the load that followed and the rest was pulled inline, outside the download manager. Both now derive the list from one helper. Hub cache inventory: - A download for a repo an Images or Video load is staging was allowed to start: only the llama.cpp loader was consulted. Both diffusion backends already expose loading_repo_ids for the delete guard, and the download guard now reads them too. - A companion-only prefetch (pipeline manifest plus VAE and text encoder, no transformer) passed the snapshot-partial check, since every file its manifest expected did arrive, and was advertised as on-device although from_pretrained cannot load it. - The single-file flag never reached the picker through the hub inventory path, so a checkpoint-only diffusion repo read as a full pipeline and failed after the handoff. The two pipeline-shape helpers now live in hub/utils/inventory_scan.py so /api/models/cached and the hub inventory classify the same repos the same way. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop adopting an unknown scoped download, leaking raced blobs and resurrecting deleted clips Three items from the latest review round. A scoped download job carries a deliberate file subset, and every file set of one repo rides the same "@scope" slot. A client that adopts a live job from the backend had no file list to compare against: the active-downloads response never carried one, so an adopted job's set was unknown and any later scoped request for the same repo read as "already started". Selecting a different checkpoint then waited on the wrong transfer and tried to load a file nobody fetched. The response now publishes the scoped file list, adoption records it, and an unknown set no longer satisfies a scoped request. A gallery record can be deleted while its blob is still downloading. The delete revokes the URL present at that moment, so the fetch that lands afterwards inserted a fresh object URL for a record no card renders and nothing can revoke: a full MP4, tens to hundreds of MB, pinned for the rest of the session, and once per raced fetch. Both galleries now discard a blob whose record went away, with an epoch covering the video page's Clear all. The video backend keeps the last completed job until the next one starts, and the Video page merges that record on mount to cover a job that finished after the gallery fetch. Deleting the clip left the record in place, so every reload prepended a ghost card whose file request 404s until another generation replaced it. Deleting the clip, or clearing the gallery, now clears the matching terminal record, and the page skips a record it deleted itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Resolve revisions from the live cache, serialize dataset imports, and stop pinning every gallery blob Five more items from the review round. The conditioning-cache revision marker read huggingface_hub's import-time HF_HUB_CACHE constant. Studio can move its cache during a session and loading follows the live setting, so after a move the marker went unresolved (or pointed into the previous root) and pulling a new revision of the same checkpoint no longer invalidated the cache: a warm run could reuse the old encoder's embeddings and the old VAE's latents. It now looks in the active Studio cache first and keeps the environment and the library constant as fallbacks, which the trainer subprocess still needs. The dataset interlock counts mutations rather than excluding them, so two imports of different examples into the same empty name both got past the emptiness check. The winner promoted its staging directory atomically; the loser found the folder non-empty, fell back to a per-file move, and merged its images and captions into the winner's dataset. Imports now take a per-folder lock, a second one is refused with 409, and the emptiness check is repeated under the lock. On Windows the sd.cpp asset resolver filtered only by accelerator token, so a Windows arm64 host matched an x64 zip, downloaded and installed it, and failed later when the binary would not run. It now filters by architecture the way the Darwin and Linux branches do. Every gallery page fetched every PNG up front and kept the object URL for the session, so scrolling a large gallery grew memory without bound for tiles the user may never look at. The Images strip now fetches a tile as it nears view, like the Video strip, and keeps the eager path only where IntersectionObserver is unavailable. A 503 carrying a JSON body comes from the application, not a proxy, so it is surfaced as the error it is instead of entering lost-response settlement and being reported as a request that never reached the server. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the class of a failed generation instead of a bare "Image generation failed." Found on a macOS runner: the native renderer aborts inside its own text encoder there, and the page showed only "Image generation failed." with the sd-server backtrace left in the server log, so nothing about the failure reached the user. The failure is now classified into fixed text, out of memory and native-process death, so the message says what happened and what to try. None of the engine's own output is echoed, since a native tail carries local paths and argv; that stays in the log, and an unrecognised failure keeps the original literal. * Treat an undecodable caption sidecar as the tombstone the trainer sees Uploads store .txt and .caption sidecars as raw bytes, so one can hold invalid UTF-8. The trainer treats any existing sidecar, decodable or not, as an empty tombstone and never falls back to the metadata row for that image. The labeling grid and the dataset summary read an undecodable sidecar as absent instead, so both showed a metadata caption that the run would silently replace with the instance prompt, and counted the image as captioned. Both now track sidecar presence separately, so what the user reviews is what the run trains on. * Keep the reason a native server died, not just its backtrace A ggml abort prints its cause first and then a stack trace, so reporting the last twenty captured lines gave twenty addresses and nothing about the failure: on the macOS runner the native server died on an unimplemented Metal op and the message carried only frame pointers. The captured tail now leads with the lines that name a cause and keeps recent context after them, for both the startup failure and the mid-request death. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop staging the dense text encoder for an fp8 video load Two halves of the same gap, found while measuring the LTX-2.3 download plan: - The video download plan and the scoped pre-download never saw text_encoder_quant. An fp8 request loads a hosted pre-cast encoder, so asking for one still staged and downloaded the base repo's dense Gemma3 (48.79 GB of Lightricks/LTX-2 on the 2.3 distilled pick) that the pipeline then never opened. The plan now drops those shards and stages the pre-cast checkpoint instead; their configs stay, since the pre-cast loader still meta-inits the encoder from the base repo's component config. - The LTX-2.3 assembly builds every component itself, so pipe_kwargs (which carries the pre-cast encoder for from_pretrained) never reached it and an fp8 request silently loaded the dense encoder anyway. It is passed across explicitly now. The dense skip is earned, not assumed: only a pre-cast checkpoint that resolves on the Hub lets the plan drop the dense shards, and only one already fetched to disk lets the pull drop them, so an unpublished or gated artifact leaves both exactly as they were. If injection still fails after that, the load tops the dense weights back up rather than handing from_pretrained a snapshot with no encoder in it. Measured against the real Hub on the 2.3 distilled Q4_K_M pick: 67.24 GB before, 18.92 GB with a 0.43 GB stand-in for the pre-cast artifact (the base entry drops from 24 files / 48.79 GB to 13 files / 0.04 GB). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Match the file's typing idiom for flow_shift models/training.py annotates with typing constructs throughout (105 Optional[...], no Union), and flow_shift was the one place using PEP 604. Union[] reads like the rest of the file, and it also drops the runtime evaluation that would raise on Python 3.9. * Bound the gallery blob cache, and three interlock fixes Four review findings, all reproduced first: - The gallery object-URL caches were unbounded. A clip runs from a few MB to a few hundred, both pages stay mounted after their first visit, and entries were only dropped on delete, so scrolling pinned everything for the session. Both pages now share a byte-budgeted LRU (512 MB video / 192 MB images) keyed off the visibility signal the near-viewport fetching already provides. On-screen media, the selected clip or image, and the item just fetched are never evicted, so eviction is invisible and a single item larger than the whole budget cannot evict itself into a refetch loop. - The image, video and chat load guards ran two independent training probes but returned early when the FIRST one raised, so an unreadable LLM backend disabled the diffusion interlock and a load could proceed straight into an active diffusion trainer on the same GPU. The probes are independent now. - An engine switch swallowed a failed teardown and published the new engine anyway, which is exactly the leak the unload exists to prevent: the arbiter's evictor, /images/unload and the next load all resolve through get_active_diffusion_engine(), so the still-resident pipeline (or a live sd-server) became unreachable and the next load allocated on top of it. The switch now fails and leaves the old engine published, so it stays reclaimable. - The native generation timeout was 30 minutes while the Images page waits up to 6 hours (SETTLE_MAX_MS), so slow-but-progressing CPU jobs died deterministically at the deadline. Measured on GPU-less runners, a 512x512 4-step Q2_K generation took 900 s on Linux and 1465 s on Windows, so larger images or step counts clear half an hour easily. The ceiling now matches the page's window and applies to the whole request: chunks of a split batch share one deadline instead of each getting a full budget. Cancellation is unchanged. Declined: gating the huggingfacenotorch extra off Python 3.9 over the conditional diffusers marker. The marker is deliberate and its comment says why: diffusers dropped 3.9 in 0.38, so pinning >=0.39 outright leaves pip no candidate and the whole extra unresolvable there. The pipelines it names live in studio/backend, which cannot install on 3.9 anyway (studio.txt pins matplotlib==3.10.9 and fastmcp>=3.0.2, both requires_python >=3.10), and the extra is the general core one, so the alternative drops 3.9 for library users who never touch Studio. * Close the load-versus-training-start race, and two picker fixes - The image and video load guards read is_active() and only then selected an engine, acquired the arbiter and registered the load. A /train/diffusion/start reserving inside that window freed residents the load had not registered yet, so the trainer came up beside a brand-new pipeline. The service already had exactly the right pattern for this in dataset_mutation, so gpu_load_admission mirrors it: reserve() refuses while an admission is open, an admission refuses once a start is reserved, both decided under the one lock. The span is only the registration, since begin_load returns as soon as the load is registered and _free_gpu_for_diffusion_training preempts an in-flight load from that point. Chat is deliberately not covered: its load spans an eviction plus a multi-minute GGUF load, and it admits models that fit beside training by design, which is a different contract from the diffusion pipeline's all-or-nothing one. - Hugging Face gives the LTX-2 family the image-to-video pipeline_tag (both Lightricks/LTX-2 and unsloth/LTX-2.3-GGUF report it), so a text-to-video-only filter dropped the flagship audio family out of Video Hub search while the rest of the app routed it to Video. - Task-scoped quant fit sized picks against the LARGEST visible device while resolve_diffusion_device_target returns a bare "cuda" and torch places on the current one. On a heterogeneous host that recommended a checkpoint sized for the bigger card and then loaded it onto the smaller one. Fit now uses the device the load actually lands on; identical on a homogeneous host. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Expose the persistent conditioning cache in the start schema DiffusionLoraConfig has carried cond_cache_dir for a while and the DiT trainer acts on it, but DiffusionTrainingStartRequest omitted the field, so Pydantic dropped it silently and every API-driven run fell back to the in-memory cache that is rebuilt from scratch each time. The warm path skips loading the VAE and the multi-GB text encoders on a rerun whose images, captions and resolution are unchanged, so this was a real capability that could not be reached. Contained like output_dir rather than left to the trainer subprocess's cwd, since it is another directory the trainer writes to. Blank or omitted still means the in-memory cache, so it must not resolve to the outputs root. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix diffusion policy and classification issues from review fp8 auto precision defaulted to precise accumulate on any non-consumer GPU, which made fp8 2.05x slower than int8 on RTX 6000 Ada and slower than not quantising at all. NVIDIA's professional whitepapers do publish equal FP8 rates for both accumulate modes there, so the hardware premise held, but the cost is in the cuBLAS path rather than the published rate. Default to fast accumulate: measured on B200 the flag is a no-op (4096^3 _scaled_mm at 3023.8 vs 3041.8 TFLOP/s, bitwise-identical output, 1.213 s vs 1.230 s end to end), so it is a large win where it bites and free where it does not. Precise accumulate stays available via transformer_quant_fast_accum. Z-Image's DiT is a Lumina2 derivative, so unsloth/Z-Image-GGUF and unsloth/Z-Image-Turbo-GGUF both declare general.architecture = "lumina2" and the whole line was tagged image-diffusion-unsupported and hidden from the Images "On Device" list, though validate_load_request loads them. Resolve shared archs from the repo/file name like bare "wan" already does, with a test asserting the picker and the loader agree for every family. The sage attention on-demand install ran an unpinned `pip install sageattention`, but PyPI's newest wheel is 1.0.6 and diffusers refuses anything below 2.1.1: the install always "succeeded", wrote an unusable version into the running venv, and was rejected on the next line. Carry the dispatcher's floor so pip resolves nothing instead. The dense-quant disk gate sized the download from the bf16-RESIDENT table. The fp32 families download twice that (Z-Image: 23,479 MiB against a 21,970 MiB gate), leaving a window where the check passed and the download filled the disk; Ideogram 4 ships fp8 and was overcharged the other way. Size the gate by published bytes, verified against HF sibling metadata for all 12 families. Patch installs went through unsloth_zoo, which refuses to import unless UNSLOTH_IS_PRESENT is set, and that is set by unsloth itself. The server imports unsloth at boot so it never showed there, but any other process ran silently unpatched with every install returning False, which is 13 test failures on a clean environment. Import unsloth and retry once, memoised per process. Also: the GGUF+LoRA refusal pointed at the native engine without saying a GPU host only selects it under UNSLOTH_DIFFUSION_ENGINE=sd_cpp, so the suggestion was unreachable; the gallery recipe recorded loras from the generate request alone, losing a load-time bake; load-progress claimed "40.07 GB downloaded" for a fully cached load; and pickers.tsx imported three catalog-group helpers it never used. Reported by oobabooga. * Keep the sd.cpp text encoder on CPU under Metal macos-14 loads FLUX.2-klein-4B Q2_K natively on mps and then dies on the first generation with exit code -6: ggml_metal_op_encode_impl: error: unsupported op 'RMS_NORM' -> ggml_abort LLMEmbedder::encode_prompt -> LLMRunner::compute -> GGMLRunner::compute ggml's Metal backend gates RMS_NORM on contiguous rows and aborts the process when that does not hold, with no per-op CPU fallback, so any LLM text encoder (Qwen3 for FLUX.2 and Z-Image, T5 for FLUX.1) takes sd-server down. The encoder runs once per prompt while the DiT runs every step, so pinning only the encoder keeps Metal for the part that matters. UNSLOTH_DIFFUSION_SD_CPP_METAL_TE_GPU=1 opts back in once ggml grows the kernel. * Gate the unsloth retry in the diffusion patch backend The retry added for the clean-environment patch failures is not free: importing unsloth pulls torch in behind it, which costs ~940 MB of RSS measured in a process that had neither, and on a host with no accelerator it fails anyway. A cross-platform CI job that had generated fine at ~900 s later died 19 s in with SIGTERM and every 'if: always()' step skipped, which is the runner being torn down rather than a step failing. Retry only when torch is already imported (true of the server and of anything patching a real module, and the condition that stops the retry from being what loads torch), unsloth is installed but not yet imported, and the first failure was the ImportError the sentinel guard raises. The clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only retry the unsloth import where it can succeed The gate still let the retry run on hosts unsloth does not support, which is where it is most harmful: a 7 GB macOS runner lost the Studio server 26 s into a load, and the Linux runner was torn down mid-generation. Neither MPS nor plain CPU can complete the import, so the retry there pays the cost and fails anyway. Require an accelerator unsloth actually supports (CUDA/ROCm via torch.cuda, or XPU), with UNSLOTH_ALLOW_CPU as the documented override, and hoist the predicate to module level so it is tested directly rather than through the import system. On a CPU-only host the retry no longer fires at all; on CUDA the clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Guard old diffusers, stream video exports, record conditioned recipes Three fixes from review. The 0.39-only pipeline classes (Flux2Klein, Z-Image, Krea 2, LTX-2, HunyuanImage) were resolved by getattr deep in the load, so on the older diffusers that packaging still allows on Python 3.9 -- diffusers dropped 3.9 in 0.38 and this project still supports it, so the 0.39 floor has to be conditional or the extra becomes unresolvable -- an advertised model failed with a bare AttributeError after its checkpoint had already been downloaded. Krea 2 already guarded itself this way; assert_pipeline_class_available now runs the same check for every image and video family from validation, before any fetch, and names the version and the fix. WebM export accumulated the whole VP9 output in a BytesIO and returned it as one bytes object that the response held again. The request caps allow 2048x2048 for 1024 frames, so an export runs to hundreds of MB and concurrent clicks could exhaust the process, while the MP4 route beside it already streamed from disk. transcode_to_file encodes to a temp file and the route returns a FileResponse with a background unlink, so nothing large is resident. A conditioned generation's recipe carried only the txt2img fields, so the gallery presented an inpaint or upscale result as a complete Create recipe and restoring it replayed an unrelated text-to-image request. The images themselves are still not persisted (user uploads with their own lifetime), but the workflow and its scalars are, restore reapplies them, and the toast now names the inputs that have to be supplied again instead of silently landing on Create. Reported by Codex. * Per-load video cancel event, family-gated image picker, cond cache refusal A cancelled video load could resume: begin_load cleared the shared cancel event, and unload() drops _loading without waiting for the worker, so the next load cleared the very object the cancelled worker was watching and its multi-gigabyte pull ran on alongside the replacement until the token check at the end. Each load now gets its own threading.Event, passed down through _fetch_te_prequant and _predownload_base, so a cancelled worker stays cancelled. A cached repo with a model_index.json was advertised as text-to-image on the trust rule alone, but validate_load_request also requires a detected image family, so a trusted pipeline of an unsupported class produced a picker row that deterministically 400s. The picker now applies both gates, mirroring the video branch. cond_cache_dir was accepted for sdxl and then ignored: only the DiT trainer reads it, while the SDXL trainer builds a per-run in-memory latent cache, so the promised cross-run reuse never happened. The route now refuses it with a 400 that names the families which do support it, checked against the resolved family so an omitted model_family with an SDXL base is caught too. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the frontend build broken by the gallery blob cache tsc -b failed on the branch head, so npm run build produced no dist and every platform job fell back to --api-only: blob-url-cache.ts(29,15): TS1294: This syntax is not allowed when 'erasableSyntaxOnly' is enabled dataset-labeling-grid.tsx / dataset-showcase.tsx: Argument of type '{ url: string; bytes: number; }' is not assignable to parameter of type 'string' The cache took its budget as a constructor parameter property, which the project's tsconfig forbids, and fetchGalleryObjectUrl now returns the blob size alongside the URL for that budget, which the two dataset thumbnail components still consumed as a bare string. Declare the field explicitly and destructure the URL at both call sites. tsc -b is clean and vite build emits dist again. * Recover from a ggml unsupported-op abort by restarting on the CPU backend ggml checks every node against the device's supports_op and calls GGML_ABORT when one is not implemented, because a single-backend graph has nowhere else to put it: there is no per-op CPU fallback. The whole sd-server dies with SIGABRT mid-generation and the user gets "the native image renderer stopped unexpectedly" with no way forward. Seen on macos-14 arm64 with FLUX.2-klein-4B Q2_K through the cross-platform CI: the text encoder is already pinned to CPU, and the abort moved into the denoise loop instead. ggml_metal_op_encode_impl: error: unsupported op 'MUL_MAT' -> ggml_abort StableDiffusionGGML::sample -> sample_k_diffusion A retry on the same backend would abort identically, so the load is restarted once with --backend cpu (the only flag that changes which backend executes the graph; --offload-to-cpu moves parameters, not compute) and the generation is re-submitted. The same checkpoint then renders slower rather than not at all. Strictly bounded: the signature must carry both the unsupported-op line and ggml_abort, the device must not already be CPU, and it happens once per load, so an OOM kill or a genuine crash still surfaces as itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix two tests that only fail in a full-suite run The 3.10 CI leg resolves PyAV 17, where av.container.OutputContainer is an immutable C type, so the no-libopus export test died on "cannot set 'add_stream' attribute of immutable type" before it asserted anything. Inject the refusal by wrapping the container av.open() returns instead; modules stay patchable on every build. Removing the injection makes the test fail again, so it still covers the branch it is named for. The Xet shim's degraded-path tests drop utils.hf_xet_fallback from sys.modules and import a throwaway copy. Restoring only the sys.modules entry left the utils package attribute bound to the throwaway, and the two disagreed for the rest of the process: a later monkeypatch of the dotted target patched one copy while the code under test imported the other, so the patch did nothing and test_fetch_te_prequant_only_reports_what_it_downloaded reached the real Hub and got a 401. Restore both bindings. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop an ignored-cancel sd-server, guard deletes during diffusion training, repair unusable managed binaries sd-server does not interrupt an in-flight job, so when it ignores a cancel the grace branch abandoned the poll and reported cancellation while the native job kept a core (or the GPU) busy to completion and held the server's job slot. The comment said the caller stops the server, but only unload does that immediately: a superseding load stops it after its multi-gigabyte download, and a load that then fails never gets there. Stop it here, as the deadline branch already does. DELETE /api/models/delete-finetuned checked only the LLM trainer, so it could rmtree the output directory a live diffusion LoRA run was about to write its adapter into. Consult the diffusion training service too, like the dataset mutation and model-load routes. find_sd_*_binary only checks is_file(), so an interrupted extraction (or a prebuilt for the wrong CPU) left a present-but-unrunnable binary the installer never retried: every load probed it, fell back to diffusers, and native inference stayed off until the directory was deleted by hand. Probe it and reinstall, but only for a copy under the installer-owned root -- SD_CLI_PATH, UNSLOTH_SD_CPP_PATH, an in-tree build and anything on PATH are the user's. * Plan the pre-cast text encoder, and make the cross-trainer GPU admission atomic An fp8 text-encoder request loads a hosted PRE-CAST checkpoint, but the image download plan never received text_encoder_quant, so the manager staged the base repo's dense encoder (FLUX.2-dev's Mistral-24B is ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) and the load then pulled the pre-cast file inline, outside the manager's progress and disk preflight. The plan now takes the field, resolves the hosted artifact with the same resolver the injection uses, stages that file, and drops only those components' dense weight shards. The load's own prefetch takes the same treatment, since it paid the same cost. Only a checkpoint that really resolves on the Hub earns the drop, so a gated or renamed artifact still stages the dense encoder the load will fall back to. The two trainers admitted each other with independent check-then-act guards: the diffusion route checks the LLM backend several network-bound preflights before it reserves, and the LLM route checks the diffusion service well before it spawns, so two near-simultaneous starts could both pass and train on one GPU. reserve() now re-tests the LLM backend under its own lock, and the LLM route holds the diffusion service's gpu_load_admission across its spawn, so exactly one of the two wins. Both halves fail open, so a chat-only install still trains. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the Advanced options a staged download planned against Staging does not set busy, so while a multi-gigabyte download runs the user can still change precision, memory mode, speed or the baked LoRA selection. The pending record held only the repo and artifact, and the completed download fired a load that read the CURRENT state: the staged file set could then be missing files that load needs (fetched inline, with no progress and no disk preflight) or hold gigabytes it no longer uses. One snapshot of every Advanced control is now taken when the plan is built, and it travels with the pending record into the load, so the load that runs is the one the download was planned for. * Do not advertise a family the installed diffusers cannot build The newer families (Z-Image, Krea 2, FLUX.2, LTX-2, HunyuanImage) exist only from diffusers 0.39, and 0.39 cannot be installed on Python 3.9 at all -- diffusers dropped 3.9 in 0.38, so the requirement is conditional or the whole extra becomes unresolvable. On such an environment the picker still offered those rows, every pick failed deterministically, and the error's advice to run pip install -U diffusers could not fix it without also upgrading Python. The cached-repo picker now applies the same availability check validate_load_request does, which is keyed on the pipeline class actually present rather than on the Python version, so it is also right for an intentionally pinned older diffusers on 3.10+. Fails open when diffusers cannot be imported at all: that is a different problem and the load path reports it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore the diffusion engine selection after each router test The active engine is module state, and several tests set it by plain assignment because what _activate does to it is the thing under test, so monkeypatch could not undo it. A leaked ENGINE_SD_CPP left get_active_diffusion_engine() handing back the sd.cpp backend for the rest of the process, and every later route that reads the active engine then saw an unloaded model: eight tests in test_openai_images_generations_route.py returned 503 in a full-suite run while passing on their own. The autouse fixture now snapshots and restores it. * Stream gallery clips, and close three races around them Four fixes from the latest review pass. The video gallery downloaded each clip into a blob before it could play, so playback waited on the whole file (tens to hundreds of MB), seeking was limited to what had arrived, and every viewed clip stayed pinned in the webview. The file route already streams and serves ranges; it just could not be a <video src> because it is bearer-gated. Mint a short-lived signed link instead (its own HMAC secret, 12 hour TTL, separate from the image links) and hand it to the element, which then fetches only the ranges it plays. That removes the blob budget, its LRU and every revoke on this page. The sd.cpp readiness probe accepted any process answering on the port, so a foreign server that grabbed the port between the bind check and the spawn was adopted as ours. Confirm the listener is our child before reporting ready, and stay best-effort (psutil missing, an unknown owner, or any probe error still passes) so the check can only reject a definitely foreign process. Dataset import held its lock for the extract but not for the upload path, so two concurrent uploads into the same folder interleaved; take the same lock and return 409. And reject Windows device names (CON, NUL, COM1..9, LPT1..9, with or without an extension) plus trailing periods in dataset names, which are unopenable on Windows. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Build the image download plan for the engine that will load /images/download-plan always asked the diffusers backend, while /images/load picks the engine per host: a GGUF pick on a machine with no usable GPU routes to native sd.cpp, which reads a single-file VAE plus text encoders and never opens the base repo's sharded components. Measured on unsloth/FLUX.2-klein-4B-GGUF (Q2_K): the plan staged 7.66 GB of FLUX.2-klein-4B components the native load discards, and the 7.80 GB sd-cli actually needs was then fetched inline by the loader, outside the download manager's progress and its disk preflight. Z-Image-Turbo is the same shape. The plan now asks whichever engine the load will select. predict_engine() applies the selection policy without any side effect: it activates nothing (staging a download must not unload the resident model) and only locates the binary rather than installing it, but still counts an installable binary as available, since that is what the load does on a fresh host. The native backend gains a download_plan built from the same _asset_specs the loader fetches, returning the same envelope, so the manager stages exactly the files sd-cli opens. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not let a queued generation outlive the model, and three scan fixes Five items from the latest review; four were real. An unload or arbiter eviction only cancels the generation holding _generate_lock. A second request queued behind it holds no cancel event yet, and Python locks are not FIFO, so it could take the lock the instant the active denoise released it, still see a loaded pipeline, and run a whole new denoise after the model was told to go away: the eviction then waits minutes for it and an image lands after the eject. Unload and a superseding load now raise a fence under _lock before they queue, and a generation that wins the lock while one is pending refuses instead. The cached-model scan judged pipeline completeness across every revision, so a repo holding an older complete snapshot plus a newer companion-only one read as complete while the snapshot from_pretrained actually opens has no transformer. Both scans now look at the revision the loader will open. Deleting a dataset image deleted its caption sidecar unconditionally, which for cat.jpg alongside cat.png removed the caption the survivor still resolves to. The sidecar now goes only with the last image of that stem, matching what the thumbnail cleanup beside it already did. Importing an example into a folder that holds no images but does hold files fell back to promoting the staging dir one file at a time, so an interruption left a partial dataset that the image_count check accepts as complete on retry. Those files are folded into the staging dir instead and the promotion stays a single atomic rename. The MPS generator report does not apply: torch.Generator(device="mps") has worked since PyTorch 2.0 (pytorch/pytorch#91348) and the studio installer pins torch>=2.4. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten diffusion comments Collapse the multi-line comment blocks across the image, video, sd.cpp and diffusion-training code to one or two lines each, and drop comments that only restate the statement below them. Comments only, no code or behaviour changes. * Tighten diffusion comments (second pass) Collapse the remaining multi-line comment blocks in the video page, training routes and service, sd.cpp server and installer, memory and speed planners, and the shared request models. Comments only, no code or behaviour changes. * Tighten diffusion comments (third pass) Collapse the remaining multi-line comment blocks in the attention, cache, LoRA, prequant, precision and compile-cache modules, the sd.cpp arg builder and engine, the video routes, the Ideogram 4 assembly, the model picker, and the diffusion test suites. Comments only, no code or behaviour changes. * Restore the dataset when an example import cannot be promoted Promotion folds the folder's pre-existing entries into the staging dir so the swap is one atomic rename. Every failure after that fold left the user with nothing: the 409 path and an os.replace error both fell through to 'finally: shutil.rmtree(staging)', which deleted the entries that had just been moved in there, while the response said 'Nothing was written'. A same-named entry was also unlinked outright before the promotion was known to succeed. Park superseded same-name entries in a rescue dir instead of deleting them, record every move, and restore all of them if any step of the promotion fails. The fold loop itself is covered too: renaming a non-writable directory raises EACCES on POSIX, which previously escaped as a 500 after the earlier entries had already been moved out. A failed rename now maps to the same retryable 409 as the rmdir conflict. Verified with the reported trigger (a non-empty mode-500 directory whose name collides with an imported file): the folder listing is now identical before and after the failed import. * Drain the teardown fence on a failing unload, give each load its own cancel event Two independent leaks on the image path, both already solved elsewhere in the same file. unload() incremented _teardown_waiters, ran _unload_locked() and decremented, with no try/finally, while the superseding-load path used a finally for the same pair. _unload_locked ends in clear_gpu_cache(), whose CUDA branch calls synchronize/empty_cache/ipc_collect unguarded, and a sticky CUDA fault makes those raise. The count then never drained, so every later generation was refused as cancelled for the life of the process, a fresh load included, since begin_load's own increment and decrement are symmetric. unload() is reached from the chat/video GPU handoff, the engine router and two training routes, so one fault during an ordinary handoff wedged image generation until restart. Release it in a finally. The image and native backends each cleared one shared cancel Event on a new load. unload() sets that event to cancel an in-flight multi-GB download and drops _loading in the same breath, so a replacement load is admitted while the cancelled worker is still inside the fetch, and its clear() re-enabled the very object that worker was watching: the cancelled download resumed and ran alongside the replacement. Take a fresh Event per load and thread it to the worker, as the video backend already does, and set it under the lock since begin_load now rebinds the attribute. * Name utf-8 on the diffusion text I/O and the sd.cpp subprocess pipes tests/test_text_io_encoding.py failed on five files this branch adds. Text I/O without an explicit encoding falls back to the Windows ANSI codepage, so a non-ASCII path or manifest value round-trips corrupted, and the three sd.cpp pipes decode the child's UTF-8 output as ANSI on Windows despite already passing errors = 'replace'. Eleven read_text() / write_text() sites across diffusion_compile_cache, diffusion_ideogram4 and diffusion_krea2, plus text = True on the sd-cli version probe, the sd-cli run and the sd-server pipe. * Record the load-time build on a gallery image's recipe A gallery record documents itself as the image's full generation recipe and is embedded in the PNG, but the only load-related field it carried was the repo id. A GGUF repo holds many quants, so that does not say which one made the pixels, and it says nothing about an adapter baked in at load time. The fallback meant to cover the baked case could never fire: with no loras on the request _adjust_baked_loras zeroes every baked adapter and _active_lora_pairs drops zero-weight entries, so active_loras was always empty. A baked-and-disabled build is not the same pipeline as a never-baked one, so the recipe could not reconstruct the image once the model was rebuilt. Persist model_kind, gguf_filename, transformer_quant and the baked adapter names, read off the load state rather than the request, and show them in the recipe popover. The new fields are optional with defaults, which matters because list_gallery_images drops any record that fails validation, so a required field would have emptied every existing gallery; a regression test pins that. * Drop eleven duplicated comment tails, restore the mxfp8 denial note The comment passes collapsed several wrapped blocks onto one line without deleting the last physical line of the original wrap, leaving the tail of each sentence repeated as its own comment underneath. Two of the eleven were re-worded rather than byte-identical, so a strict suffix match missed them. 6e16ad16f also dropped the line justifying the qwen-image mxfp8 denial while that rule stayed live in _FAMILY_SCHEME_DENY, under a header that then documented only fp8 and nvfp4. Restored. Comments only; verified with the AST gate. * Only let the dense-quant fallback use shards the prefetch actually staged The prefetch skips the base repo's transformer/ shards whenever a prequant checkpoint is expected, since that checkpoint replaces them. But a prequant fetch can fail for reasons the planner cannot see: an unpublished, gated or renamed artifact, a hub 5xx, a proxy, a checkpoint the validator rejects. The loader then fell through to from_pretrained(subfolder = 'transformer') and pulled those shards inside the load lock during 'finalizing', after the previous pipeline was already evicted, where the cancel event has no reach, load_progress has already reported bytes_downloaded == bytes_total, and the cache-disk gate had only reserved the small prequant checkpoint. That is verbatim the situation _dense_quant_prefetch_needed's own docstring exists to prevent. Gate the in-loader dense fallback on the shards being staged, read off the returned file list rather than the request so a failed size estimate closes it too, and let the GGUF build take over otherwise, which is what the prequant-sized replan already does one branch over. It is also the invariant the text-encoder path already enforces: only a component that really resolves may have its dense weights dropped from a plan or a prefetch. Adds the missing coverage for the gate's prequant arm, which the existing disk-gate tests never reached. * Refuse a training output_dir that resolves to the outputs root resolve_output_dir strips a leading 'outputs' and drops '' / '.' segments, so '.', './', './.', 'outputs', 'outputs/outputs' and ' . ' all clean away to nothing and land on the outputs root itself rather than a run directory under it. The DiT trainer then writes pytorch_lora_weights.safetensors flat into the root, where scan_trained_models and scan_checkpoints cannot see it (both filter is_dir()), and a second such run overwrites the first. The UI only checks the field is non-empty, so 'outputs' is one plausible run name away. Refuse it with a 400 that says what to do instead. cond_cache_dir collapses the same way but has an honest 'off' to fall back to, so a root-resolving value now means the in-memory cache, which is what the comment above it already promised: otherwise a run drops one flat safetensors per cached latent and caption into the directory trained models live in. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Repair three defects the main merge left behind Git merged all three files without a conflict, but the result was wrong in each case. The tests only started failing once the merged tree was run. routes/inference.py duplicated the GGUF load block: this branch had moved the gguf_load_in_flight marker and the _hub_download_blocks_gguf_load guard under "if config.is_gguf and config.gguf_hf_repo", and the conflict resolution re-added main's copy at the old position, so both ran. Dropped main's copy; the earlier placement is the deliberate one, so a 409 from the hub guard cannot tear down a resident Images or Video pipeline. test_gpu_selection.py still called _hf_offline_if_dns_dead, which main renamed to _hf_offline_if_unreachable_for (#7591). Disjoint edits, so no conflict, but four route-error tests referenced a function that no longer exists. test_gguf_load_cache_reuse.py anchored its ordering assertion with rindex over "if config.is_gguf:", taking the last one before the load marker. That only held while _resolve_inherited_extra_args sat above every such line; main has since hoisted it above the gpu_ids preflight, so the anchor landed between the call and the marker and the assertion compared against an unrelated later call site. The ordering it checks is a property of _load_model_impl as a whole, so it now anchors on the function. The ordering itself is intact: _resolve_inherited_extra_args, then the gguf_load_in_flight marker, then the hub guard, then the chat handoff, then unload_model. * Stop a real sd.cpp install from breaking its own discovery tests The five "nothing is installed" assertions in test_sd_cpp_engine.py cleared SD_CLI_PATH and UNSLOTH_SD_CPP_PATH and patched Path.home, which covers hops 1, 2 and the fallback half of hop 3. It leaves two hops live. Hop 3 goes through managed_install_root(), which honors UNSLOTH_STUDIO_HOME and STUDIO_HOME and resolves to the stable-diffusion.cpp directory beside the studio home. That is the documented way to run side-by-side Studios, so anyone who has one set gets a real sd-cli back from a finder the test expects to return None. Hop 4 is the in-tree developer build, which does the same for anyone who built sd.cpp inside the checkout. Isolated with an autouse fixture rather than another helper call, because reaching the failure needs no fixture: SdCppEngine(binary = None) runs the finder from its constructor, which is why test_generate_raises_when_binary_missing failed too. The fixture points the studio home and the in-tree root at an empty tmp tree, so every hop is answered by the test rather than by the box. The in-tree root moves into a named in_tree_install_root() so it can be pointed somewhere empty; behaviour is unchanged, including the OSError and IndexError guard on an unexpected layout. * Make the sd.cpp uninstall test actually extract the code it tests The two sed ranges anchored the production fragments at column 0, but both blocks sit inside the main removal function and are indented, so each range matched nothing and LOOP_FILE / DEFAULT_FILE came out empty. Sourcing an empty file is a no-op, so the suite reported 4 passed / 7 failed: the seven removal assertions failed because nothing ran, and the four that passed were 'kept' assertions that pass trivially when nothing runs. The shell job auto-discovers tests/sh/test_*.sh and runs each under set -e, and this file is not in its skip list, so it was a deterministic red. Anchor on optional leading whitespace and fail loudly on an empty or _remove_path-less fragment, so a future reshuffle of uninstall.sh cannot make the suite vacuous again. Now 11 passed, 0 failed, against the real removal loop and the real default-mode block. * Agree on what an engine can build, and on what the installer owns Two gates, each half-applied. The unbuildable-family gate had one caller, the image branch of the cached repo picker. The GGUF classifier, the local-model classifier and the video branch had none, so on a diffusers too old for a family the picker still offered its GGUF and the load then failed. Meanwhile the loader asserted the diffusers pipeline class before engine selection, so a GGUF this host routes to native sd.cpp, which instantiates no pipeline class at all, was refused with an upgrade instruction that could not help. Both are wrong one-sidedly: hide a family only when NEITHER engine can build it, and demand the diffusers class only when diffusers is what will load it. One predicate, family_buildable_here, now answers both, so the picker and the loader cannot disagree. The population is real on Python 3.9, whose diffusers ceiling is 0.36: Flux2KleinPipeline, Krea2Pipeline and LTX2Pipeline are all absent there, and FLUX.2-klein GGUF is a repo in this PR's title. That assertion also raised RuntimeError, which /images/load maps to 409, the status that otherwise means a load is already in progress, and which escaped /images/download-plan (it catches ValueError and FileNotFoundError) as a bare 500 with the message lost. It is an unloadable pick like every other, so it raises ValueError and both routes answer 400 with the text intact. The managed-binary repair used a path test for ownership while the installer uses a marker. On a managed root without the marker, which is any install predating it, the repair deleted sd-server and the reinstall was then refused, permanently, because the surviving sd-cli keeps the directory non-empty so the marker can never be claimed: the user went from an unrunnable binary to no binary and no way back. Require the marker before discarding, which is the same definition of ours that uninstall.sh already uses to keep a user's own stable-diffusion.cpp checkout. A genuinely interrupted extraction still self-heals, since install() writes the marker before it extracts. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add gguf and av to the studio extra so the wheel install matches studio.txt This branch added both to studio/backend/requirements/studio.txt, which is what install.sh uses, but never to the studio extra in pyproject.toml, which is what 'pip install unsloth[studio]' uses. Nothing else keeps the two in sync, so tests/studio/install/test_studio_extra_matches_requirements.py failed deterministically, and the CI job runs the whole tests/ tree. The drift is not only a red test: gguf backs diffusers' GGUFQuantizationConfig and av does the MP4 encode and audio mux, so a wheel install got a Studio that cannot read a GGUF or export a video, which is most of what this PR adds. * Fix two test-suite defects: the CPU-only patch gate and a popped module The patch backend reaches unsloth_zoo's helpers only through an ~940 MB 'import unsloth', which a CPU-only host cannot complete, so every patch install returned False and nine arch/eager/compile tests failed there -- exactly the runners the retry was narrowed to protect. unsloth_zoo only wants UNSLOTH_IS_PRESENT in the environment, which costs nothing and needs no accelerator, and the conftest already sets its sibling UNSLOTH_ALLOW_CPU. Set it there too, as run.py and main.py already do at module scope, so no real server ever took the expensive route. CPU-only goes from 9 failed to 24 passed, 1 skipped; the GPU run is unchanged at 29 passed. test_setup_cache_env_hf_home popped utils.hf_cache_settings to model a fresh process and never restored it, so a later import built a second module object and rebound it on the utils package. test_hf_cache_settings then wrote its setting into one object while core.inference.diffusion read the other, the same split-module failure the xet shim already had. Restore both bindings in teardown: 21 pass together, and each file still passes alone. * Restore the setup_fail assertions the main merge reverted A fourth instance of the class 9541cc535 fixed: the merge took main's studio/setup.sh, which #7644 changed to abort through the setup_fail helper so desktop mode still emits [TAURI:ERROR], but kept this branch's older copy of the test, which still asserted the literal 'exit 1'. setup.sh is byte-identical to main here and the only diff in this file was the reverted assertions, so take main's version. 60 passed. * Fence the video teardown, reject non-finite clipping knobs, stop a test installing sd.cpp The video backend never had the teardown fence the image backend documents. Its unload and its superseding-load path both signalled the active generation, then did 'with self._generate_lock: pass' and tore the state down with the lock free. A generation queued behind that barrier holds no cancel event yet, so the signal cannot reach it, and Python locks are not FIFO: it won the lock the instant the barrier released it, read a still-loaded state and denoised a whole clip against the pipeline being freed. Reproduced on both paths, where the queued generation returned a finished MP4. Mirror the image backend: count waiters, refuse a generation while one is pending, and tear down inside the barrier with the counter released in a finally. _teardown_state becomes _teardown_state_locked since the caller now holds both locks. max_grad_norm and snr_gamma were bounded on one side only, so 1e309 floated to inf and passed. clip_grad_norm_ then computes an infinite clip coefficient, clamps it to 1.0 and scales nothing, and min(snr, inf) / snr makes every min-SNR weight 1.0. The run starts, reports normal progress and trains with the requested knob silently disabled. Measured both. NaN already failed the bounds; allow_inf_nan makes that explicit. learning_rate and flow_shift already guard this exact vector. test_load_routes_to_sd_cpp_on_cpu stubbed ensure_sd_cpp_binary but not ensure_sd_server_binary, which select_and_activate_engine probes first with installs enabled, so the unit test downloaded and unpacked 108 MB into the developer's real ~/.unsloth when UNSLOTH_STUDIO_HOME was unset. Now 0 bytes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Key the video schedule reset on the resolved defaults, not the repo id A GGUF repo holds several variants, so another client swapping a distilled build for the base one from the same repo changes the steps and guidance the backend reports while repo_id stays put. modelChanged stayed false, the page kept the previous schedule, and later generations ran an 8-step distilled setting on a model that expects roughly 40 steps and CFG 4. Include the reported defaults in the key. They only move when the resident artifact does, so a manual steps change still survives a status poll. * Retire the release-lag pin now that 2026.7.6 carries the relaxed gates The virgin Windows container row that installs from PyPI on purpose was tolerated with continue-on-error, because a Server Core container has no Microsoft Store and so no winget, and the released studio/setup.ps1 hard-stopped on a winget-only git gate and reached for winget again for the VC++ runtime. #7549 relaxed both, but it landed on 2026-07-28 and the newest wheel was 2026.7.5 from the 23rd, so the row could not get past it. The pin carried its own tripwire for exactly this moment, and it has fired: unsloth 2026.7.6 shipped 2026-07-29 and the released wheel now installs end to end, so the step reported success and the assertion errored with "delete this pin and drop continue-on-error from the Install step so this row gates". Doing that. Both rows now gate unconditionally, and the assertion that enumerated the accepted failure signatures goes with the pin it protected, since there is no longer an accepted failure for it to describe. The other three release-lag mentions in this file are descriptive rather than tolerated-failure pins: they explain why the fedora and ubuntu-nonroot legs run with overlay: true, and both already refuse a triton failure as a regression rather than accepting it as lag. Left alone. * Pin the hosted pre-quant onto the plain-torch fp8 kernel A Z-Image GGUF pick at the DEFAULT speed mode, once the hosted fp8 pre-quant is actually reachable, dies at generate with an HTTP 500: torch._dynamo.exc.Unsupported: Operator does not support running with fake tensors. Developer debug context: unsupported operator: mslk.f8f8bf16_rowwise.default _fp8_config already pins KernelPreference.TORCH when it BUILDS a config, precisely because the default AUTO switches to the MSLK kernel wherever an mslk package is importable (sm90+). A hosted checkpoint escapes that pin completely: the preference is serialized per Float8Tensor, and all 239 weights in the published Z-Image-Turbo-FP8 checkpoint carry AUTO. Loading it re-arms the very kernel the pin exists to avoid, mslk.f8f8bf16_rowwise has no fake impl, and the first compiled generate therefore cannot be traced. Isolated away from the product to be sure of the mechanism: quantise one Linear three ways on this box and compile each. KernelPreference.TORCH eager ok, compiled ok KernelPreference.AUTO eager ok, compiled FAILS on mslk.f8f8bf16_rowwise library default eager ok, compiled FAILS the same way So the pin is correct and necessary, and the only gap is that the hosted path never got it. _validate_checkpoint checks scheme, granularity, base model, min_features, exclude tokens and fast_accum, but not this. Rewriting the preference on load is safe: it selects a matmul kernel, it is not weight data, so the tensors stay bit-identical and the checkpoint's own state_dict_sha256 still describes them. It is also the faster path compiled, since the opaque extern call blocks inductor quantize fusion. Why this went unnoticed: the pre-quant repos are private, so nothing that could not read them ever took this path. It becomes the default the moment they are readable. Verified end to end at 1024x1024, 8 steps, speed_mode default, on the published checkpoint: before HTTP 500 at generate, diffusion.generate_failed after pinned 239 weights to the plain-torch fp8 kernel, load 10.0 s, transformer_quant fp8 with compiled engaged, cold 7.9 s, warm 0.87 0.87 0.87 0.86 0.87 s That warm number also beats the 1.4 s recorded for this shape on 2026-07-26. 159 prequant and transformer-quant tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cover the fp8 kernel pin so the 500 cannot come back quietly The hosted pre-quant re-arming MSLK was only visible as an HTTP 500 on a compiled generate, which needs a GPU, a readable private repo and an importable mslk to reproduce. None of that is available in CI, so the pin would rot unnoticed. Three hermetic cases on _pin_kernel_preference instead: AUTO weights are rewritten and already-TORCH ones are left alone (counting only what changed), a weight that refuses the assignment does not sink the whole load, and with no torchao enum available the checkpoint is left exactly as saved. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: tighten comments across the image generation, video and training code Rewrite the comments added by this branch to be shorter and clearer: collapse multi-line explanations into one or two lines, drop comments that only restate the code, and keep the rationale that explains why a choice was made. No code changes. * Studio: let the pipeline-class guard survive a host with no diffusers Backend CI installs the CPU-only dependency set, which has no diffusers, and runs without a GPU. 13 tests failed there for three separate reasons, one of which is a real product bug rather than a test-environment gap. assert_pipeline_class_available did a bare "import diffusers", so on any host without it the call raised ModuleNotFoundError instead of the ValueError its own contract promises. /images/download-plan catches only ValueError and FileNotFoundError, so that escaped as a bare 500 with the message lost, which is the exact failure the guard exists to prevent. It is reachable in production: the native sd.cpp engine serves GGUF picks on a CPU or Apple host where diffusers is never installed. Absent diffusers now returns, since the check answers "is the installed diffusers new enough for this family" and there is no version to judge; a pick that genuinely needs diffusers still fails in the loader with its own message. The rest were tests asserting through gates unrelated to what they cover: - Three cond_cache_dir route tests drive a DiT family, which the start route refuses without an accelerator. They now take the existing dit_train_host fixture, written for this case, so they keep testing the schema on every host. - test_download_plan_forwards_the_load_time_controls stubs the diffusers planner but sends a GGUF pick, which routes to the native planner on a GPU-less host. It now pins the engine, so it tests kwarg forwarding rather than the hardware. Engine selection keeps its own tests. - The pipeline-class guard test needed a real diffusers only for its sweep over every shipped family. That half is split out and skips on its own; the stub-driven refusal needs no diffusers and still runs. Added a test for the absent-diffusers contract above. 409 pass on a GPU host with diffusers; 301 pass and 1 skips with diffusers blocked and no GPU. * Studio: cut the image generation comments down further Second pass over the comments this branch adds. Collapse the multi-line blocks that still read as paragraphs, rewrite the longest one-liners so they say the same thing in fewer words, and drop a stale comment that had drifted away from the constant it described. No code changes: only comments and whitespace. * Studio: workflow rail, sidebar Video row, and Train settings polish Sidebar - Pin Video under Images by default; persist store bumped to v3 so an untouched install adopts it while a custom arrangement is left alone. - Add "Customize sidebar" to the end of the More flyout, opening Settings > Appearance scrolled to the sidebar nav section. Images - Replace the workflow dropdown with an icon rail down the left edge: all seven workflows visible, keyboard nav, tooltips carry the labels. - Workflows stay selectable with no model loaded, so one can be set up before picking a model. Generate is already gated on a loaded model. - Video moves out of the Create/Train strip into a link at the far right, with a matching Images link on the Video page. - Wider gutters around the settings column, headings matched to Train. Train - Field guidance moves into "i" tooltips; only state that limits a control stays on the page. Steps, LoRA rank and the rest gain hints. - /info reports params, qlora_vram_gb, gated and note as fields, so the family note renders as chips. vram_note is rebuilt from them unchanged for older clients. Shared - Number steppers appear on hover or focus. - Lighter control border (#e9e9e9) on Images and Video, light mode only. * Studio: move Advanced inline on Images and Video - Replace the top-bar toggle and right-docked Advanced panel with a disclosure under Seed, so load-time tuning sits with the settings it affects and opening it no longer shrinks the preview canvas. - Video uses the bordered variant, since offload and memory decide whether a model fits at all. - Open state persists per page in localStorage. - Tighten the Steps unit trigger: 14px chevron, less right padding. * Studio: full width media pages and layout polish - Drop the 1100px cap on Images Create, Images Train and Video, so the preview canvas grows with the window instead of sitting in a band. - Align the workflow rail to the model selector label, widen the gap before its divider, and restore the divider itself. - Advanced: more room above and below, icon and label sized to the slider rows, and the same quiet row on Video as on Images. - Give Video a pane heading and description, matching Images. - More space above Seed on both pages. - More no longer takes the active style when the current page is one of its own rows. * Studio: fix the Extend side toggles The resting outline used a ring, and index.css blanks the ring on the button holding mouse focus, so the side you just clicked lost its border until focus moved. Use a border for the outline and leave the ring to focus-visible. Also separates the two states properly (the off state had no surface of its own), drops the border in dark mode as the inputs do, and adds the aria-pressed and focus-visible styling the buttons were missing. * Studio: drop the border on the Extend side toggles Light mode should carry no border either, so the fill alone marks the state: a muted surface when off, a primary tint when on. Matches the borderless treatment dark mode already had. * Studio: move the Images workflows into the sidebar The workflow switcher was a vertical icon rail on the page. It now lives in the sidebar under Images, so the page keeps that width for the canvas. - Workflows list under the Images row, with a chevron to fold them away. Hovering the row peeks them in a flyout on the standard menu surface. - Clicking Images while already there toggles the list instead of navigating. - The listed workflow carries the highlight, so the Images row drops it. - Create takes its own icon: it was sharing the New chat pencil. - Model hub moves above Projects. Persisted layouts bump to v4 and only adopt the new order where the stored one is still a shipped default. - Images and Video content both start at 32px, clear of the sidebar. * Studio: simplify the Images workflow submenu Dropping the hover flyout: two ways to reach the same seven workflows, one of them overlapping the list right below it, read as clutter. - The flyout is gone. The workflows are rows under Images and nothing else. - On the Images page they are always open, since they are that page's switcher. Elsewhere they stay folded, and the row's chevron opens them. - Create takes the sparkles icon. - Create and Train both pad their settings column to 40px a side, so the two tabs line up with each other and with the model selector. * Studio: dock the generate action and tidy the Create controls The primary action sat at the foot of a long scroll, so it was off screen until you scrolled for it. - Generate, Video's Generate and Train's Start training float at the bottom of their settings column. No bar behind them: hover lightens the fill rather than thinning it, and the disabled state is opaque, so the controls underneath never show through. - Aspect ratios read as names: Square (1:1), Widescreen (16:9) and so on. - Width and height replace their sliders with two compact boxes. Type a size or pick one from the menu; the value still snaps into range and still drives the locked ratio. - Negative prompt gets a quiet disclosure under the prompt. It was there before but only above guidance 0, which is not the default, so it never showed. - The info "i" is smaller across the settings UI. * Studio: reveal Images and Video field hints on hover The "i" next to every field label sat there permanently, which made a column of settings read as busier than it is. Scoped to the diffusion pages by CSS rather than per component: the hints come from the page's own Field and SelectRow, the train panel's FieldLabel, and the chat ParamSlider, which is shared and should not change for chat. Reveals when the pointer is inside the field, on focus-within for keyboard, and stays inert while hidden. InfoHint grows a data-slot so the rule has a stable hook, rather than keying off its aria-label. * Studio: one-line sliders, edge fades, shared negative prompt - Steps, Guidance and the rest put label, track and value on one row. ParamSlider is shared with chat, so this is an opt-in prop and chat keeps the stacked layout. - The settings columns fade at whichever edge they run past, as the sidebar and model picker do, instead of cutting off. New useScrollFades hook drives all three. - Negative prompt moves to a shared component and Video picks it up, so both pages collapse it the same way. - Right gutter sits closer to the rule, Steps takes a bigger break above it, and the Advanced rule sits between its neighbours rather than up against the field above. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: add the API monitor to the sidebar nav Sits under More, below Export. Points at the monitor page, not the API keys dialog the profile menu opens. No store version bump: an id missing from a stored layout is appended with its default pinned state, so existing sidebars gain the row at the end and keep their own order. * Studio: take main's copy of the sidebar spinner test The merge kept this branch's older version, which pins the Recents row to h-[33px]. Main already relaxed that to any height, since row density moves independently of the trailing column the test is about. * Studio: fix FLUX.2 klein size resolution, HunyuanVideo tier routing, and the personalization save Five fixes found while auditing the model registries against the live Hub. FLUX.2 klein 9B loaded against the 4B config. One family covers both klein sizes and defaults to 4B, relying on the base_model card tag for the real base, but the trust allowlist only had klein-4B, so the correct tag was discarded and a 9B GGUF (inner_dim 4096) was loaded against a 4B config (3072). That surfaced as a bare shape mismatch from inside the GGUF quantizer naming neither the file nor the repo. Adds the missing allowlist entries and a header-level size check that fails early with a legible message; the check is fail-open, so an unreadable file, a non-FLUX.2 family, or an unmapped base leaves the load exactly as it was. The sd.cpp text encoder rule matched the literal "klein-9b", so klein-base-9B was handed the 4B encoder. HunyuanVideo-1.5 720p checkpoints routed to the 480p family. Only the literal 720p_t2v path was aliased, so 720p_i2v and every GGUF repack fell through to the generic token and inherited the 480p base repo, which also supplies the VAE and text encoder. The tier is baked into the weights (transformer target_size 640 vs 960, scheduler shift 5.0 vs 9.0) and the two bucket lists are disjoint, so this ran the whole pipeline off-tier. Resolution presets that the checkpoints were never trained for. Wan2.2 TI2V-5B is 720P-only upstream (SUPPORTED_SIZES is exactly 704x1280 and 1280x704, asserted in generate.py), and the HunyuanVideo 480p square preset 624x624 is not a bucket of its tier; 640x640 is. Personalization could never be saved. The frontend ships eight sidebar nav ids and refills every missing one on each save, but the backend Literal had seven and no "api", so every PUT to /api/settings/personalization returned 422. Aligns the server defaults and the Literal with the shipped layout. A catalog row pointing at a file that does not exist. unsloth/Qwen-Image-2512-FP8 holds torch prequant .pt checkpoints, not qwen-image-2512-fp8.safetensors, and fp8 is denied for this family anyway because it renders black, yet the row was the auto-route target on any GPU above 40 GB. Also marks FLUX.1-schnell, Krea-2-Turbo and the two Ideogram repos gated: all four are gated on the Hub today, and schnell being Apache-2.0 does not make it ungated. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: stop a case-only duplicate upload destroying an image, and size the load against the right GPU Two fixes from the review round. A dataset upload containing two names that differ only by case silently loses one on Windows and macOS. The in-batch duplicate check is an exact string compare, so "Cat.png" and "cat.png" both pass, but on a case-folding filesystem they are one destination: the commit step moves the first staged part aside, writes the second over it, then deletes the backup, while the response still reports both as uploaded. Captions collapse the same way and never even reach the image branch. No single folder can hold such a pair, but the Windows and macOS open dialogs flatten search and Recents results across folders into one multi-selectable list, which is a normal way a LoRA dataset gets assembled. Rejecting the pair everywhere would regress Linux, where the two really are different files with their own sidecars, so the check probes the filesystem once per process instead of keying off sys.platform: macOS also ships case-sensitive APFS volumes, and a Linux host can keep its Studio home on an exFAT or NTFS mount. A failed probe answers "case-sensitive", which leaves today's behaviour untouched. The quantization fit budget was sized against the wrong card under a reordering CUDA_VISIBLE_DEVICES. A bare "cuda" load lands on visible ordinal 0, but the reducer took the lowest PHYSICAL index, which the nvidia-smi path reports as index_kind "physical". Under "3,1" that sizes against GPU 1 while the pipeline loads onto GPU 3. The hook's own interface doc already promised the lowest visible ordinal. Ranks by visible_ordinal, falling back to index only for an older backend that omits it. * Studio: rework the diffusion dataset panel and align the pane gutters Upload: - one Upload button beside the dataset name, with an upload icon - picking files uploads them, no second confirm click - Add button beside the Training images dropdown, so images can go into a set that already exists - fall back to the upload form when a selected dataset no longer resolves Labeling grid: - two columns at any width; sm:grid-cols-3 crushed the tiles in a fixed width column - pin the caption size so the Textarea's md:text-sm does not outsize a tile - name the tile hover group; a bare one revealed every tile's Remove at once - Remove is an icon button on the image, not a word over the artwork - drop the amber tile fill, keep the No caption label - match the header, toggle and status text to the section's type scale Gutters: - the run area and the preview canvas now sit 40px off the rule, the gutter the settings column has off the page edge * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: give the dataset preview per-image hover and remove, and wrap it onto rows - each thumbnail is its own button, so it can carry an X that removes just that image; the strip was one button and could not nest a second control - hover lightens the image instead of drawing a border - wraps onto more rows rather than scrolling sideways, which also keeps the bottom corners rounded - samples 12 thumbnails, up from 8, now that a second row is available - a delete refreshes the panel's dataset counts * Studio: show the workflow icon beside the Images and Video headings Images reads it off WORKFLOW_TABS, so it is the same icon the sidebar submenu shows and it follows the active workflow. Video's heading is also Create, so it takes the Create icon rather than the nav row's film slate. * Studio: reject a duplicate LoRA id on the diffusion load path too DiffusionGenerateRequest already refuses a repeated adapter id, with a comment naming the hazard: _resolve_lora_set suffixes colliding adapter names, so the same id resolves the SAME adapter twice and set_adapters stacks both copies past the per-adapter weight bound. DiffusionLoadRequest bounds only the list length, and it matters more there. Generation-time stacking spoils one image; on the load path the adapters are baked into the quantized build before compilation, so the unintended combination rides every image until a reload. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: teach the GGUF reuse test double about holds_no_vram The resident-GGUF fast path consults llama_backend.holds_no_vram before asserting CHAT ownership, but two tests hand it a SimpleNamespace built before that attribute existed, so they raised AttributeError and returned a 500. A real LlamaCppBackend exposes it as a property; the doubles now carry it too. Product code is unchanged: these pass on main and were the branch breaking its own test, not a defect in the guard. --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: michaelhan <michaelhan2050@gmail.com> Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: Michael Han <107991372+shimmyshimmer@users.noreply.github.com> | 1 个月前 | |
Replace standalone Studio wording with Unsloth (#7221) * Replace standalone Studio wording with Unsloth Replace the single word Studio with Unsloth wherever it is used as shorthand for Unsloth Studio in docs, CLI output, UI strings, i18n locales, workflow display names, comments and docstrings. Kept unchanged: the full name Unsloth Studio, third party product names (LM Studio, Visual Studio, Mac Studio), feature names (Recipe Studio, Fine-tuning Studio and its translations), and all identifiers such as env vars, commands, paths and filenames. * Address review feedback on the Studio wording rename Use "an" before Unsloth where the rename left the article as "a". Restore the split brand where Unsloth and Studio render as two halves of the full product name: the onboarding sidebar subtitle and the IPv6 localhost warning. Scope two messages to the full name Unsloth Studio where plain Unsloth was misleading: the AMD README bullet and the CLI studio setup error. | 1 个月前 | |
Studio: auto-sync allowScripts pins after dependency bumps (#6136) * Studio: npm v12 readiness for install-script gating npm 12 (July 2026) stops running dependency install scripts unless they are approved via allowScripts, and npm 11.16 already warns. Studio has no git or remote URL deps anywhere, so script gating is the only exposure: - commit the allowScripts policy that npm approve-scripts writes for @biomejs/biome and msw, plus a manual fsevents entry: the tooling cannot match a darwin-only optional dep from Linux, but the strict check walks the platform independent ideal tree and flags it anyway - drop the minimum-release-age npmrc alias; npm >=11.16 flags it as an unknown project config that stops working in npm 12 - approve bun's postinstall in the setup.sh / setup.ps1 bun bootstrap; under npm 12 defaults npm install -g bun otherwise leaves a broken stub and setup falls back to the slower npm install path - fix the stale esbuild comment in studio-frontend-ci.yml: the vite 8 chain ships napi binaries with no install scripts * Studio: auto-sync allowScripts pins after dependency bumps The allowScripts entries from #6128 are version pinned, so a biome or msw bump strands the pin and the approval silently stops matching. Dependabot cannot maintain the field, so: - scripts/sync_allow_scripts_pins.py re-pins existing entries from the versions package-lock.json actually resolves. It never adds or removes entries, so approving a new script-bearing package stays a human decision. Bare names and non-exact specs are left alone. - a pre-commit hook runs it with --fix; pre-commit.ci pushes the fix commit to PR branches, Dependabot's included, so stale pins heal without a human in the loop - a Frontend CI step runs --check plus the offline unit tests as the backstop when pre-commit.ci is skipped No dependabot.yml change needed: the /studio/frontend entry already suppresses version PRs (security only) behind a 7 day cooldown. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the sync hook robust to lost executable bits The pre-commit.ci autofix commit dropped the script's exec bit, which breaks a shebang-style entry. Invoke via python instead and restore the bit. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> | 2 个月前 | |
Use Unsloth for bare Studio product references (#9732) * Studio: use Unsloth for product shorthand * Tests: restore case-folding fixture path * Complete Unsloth product reference rename * Tests: restore internal Studio references | 9 天前 | |
Use Unsloth for bare Studio product references (#9732) * Studio: use Unsloth for product shorthand * Tests: restore case-folding fixture path * Complete Unsloth product reference rename * Tests: restore internal Studio references | 9 天前 | |
Reduce and tighten code comments and docstrings repo-wide (#6095) Trim and tighten code comments and docstrings across the repository. Comment-only: every changed file verified code-identical to main via AST/token comparison. | 2 个月前 | |
Read quoted annotations in the import lint, and build the video load doubles off the real backend (#9601) * Read quoted annotations, and build the video load doubles off the real backend Two CI safety nets that fail on correct code, both found while looking at why the backend leg was red across the open PRs. verify_import_hoist.py walked annotations for ast.Name only, so a name inside a QUOTED forward reference was invisible to it. Optional["TePrequantSource"] keeps the name in an ast.Constant, so a TYPE_CHECKING import reached only that way read as unused and was reported as a BLOCKER, which is a lint failure nobody can fix without deleting a needed import. Annotation strings are now parsed and walked as the type they denote, to a depth of three so Optional["Dict[str, 'T']"] resolves and a pathological string cannot recurse without end. Literal[...] arguments are deliberately skipped. Those are values, not type names, and crediting a soft use is the one direction that can lose a real finding, so a literal that happens to spell an identifier must not mark an import used. Four self-tests cover both directions plus an annotation string that is prose rather than a type. The other one is the pair of hand-rolled backend doubles in the video load route tests. One returned a bare object() where the route gets a forty-field frozen VideoFamily, and neither declared more than two methods, so the first route step to reach for anything else failed on an AttributeError that said nothing about what the test was checking. Those two tests are about provenance and about the local_files_only the loader is handed, and they should stay about that. They now take a real VideoBackend with only the asserted methods replaced. Its __init__ allocates locks and empty state, and the load route's preflight and reservation helpers are pure resolution over the family registry, so nothing reaches the hub, a GPU or any weights. Overrides are checked against the real class, so a stub for a method that does not exist fails loudly instead of quietly never being called, and the family comes from the registry rather than being hand-copied. Added a guard for the direction a double cannot cover: every backend.<name> in routes/video.py must name something VideoBackend actually has. That call site is untyped, so a rename on the class or a typo in the route is not a syntax error, not a lint finding, and not visible to a test whose double stubs the old name. It is an AttributeError on a real load. Read off the parse tree, not the text. Backend suite is 29469 passed, 121 skipped on this tree. * Tighten the comments added in this PR Comment and docstring only. Verified the executable AST of both files is identical to the previous commit with docstrings stripped. | 11 天前 | |
Studio: add image generation, editing workflows and LoRA training with Unsloth GGUFs (#6763) * Tighten comments in the image stack tests and scripts * Close video single-file, training reservation, and image mount-resume gaps Route on-device single-checkpoint video folders through the single_file loader: a bare local .safetensors directory (no model_index.json) is advertised as a pipeline with no filename, so validation rejected it before it could load. Reinterpret the pick as a single_file load of the sole checkpoint, mirroring the image load route. Treat a reserved-but-not-yet-spawned LLM training start as active in is_training_active() so /images/load, /video/load, and /diffusion/start cannot race the reserved run for VRAM during the pre-spawn free window. Mirrors the diffusion training service reservation. Resume an in-flight image generation on the Images page mount: probe generate-progress, re-enter the poll loop, and refresh the gallery on completion so a run started elsewhere is reflected and its saved image appears without a manual refresh. Seed resident image defaults from the resolved base_repo rather than a possibly path-shaped repo_id so the first resident generation uses the right recipe. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Publish image generation active state before pre-denoise setup generate() assigned self._gen only at the pipe() call, after deferred compile, LoRA resolution/application, and ControlNet download/build had run. Across that setup window generate_progress() reported inactive even though _generate_lock was held, so a reloaded page's mount probe showed idle and let a second generate queue behind the first. Publish an active step-0 _GenState the moment the generation lock is acquired, before the setup work, and clear it in the outer finally so a setup-time error cannot leave the UI stuck active. Mirrors the video backend's queued phase and the training start guard. * Studio: fix diffusion install ownership, dataset upload atomicity, gallery pagination, and teardown races install_sd_cpp_prebuilt: only write the .unsloth-studio-owned marker when the install created the target directory or it was empty. Adopting a pre-existing, unowned, non-empty directory (a user's own stable-diffusion.cpp checkout) made it eligible for the uninstaller's recursive delete. routes/training upload: make the multi-file promotion transactional. Back up each displaced original and roll every destination back on any failure, so a mid-loop rename error can no longer partially overwrite the live dataset. routes/training _resolve_dataset_folder: reject a symlinked dataset directory and prove the resolved folder stays under the datasets root, so image read/caption/delete cannot escape the root through a link. routes/training delete: escape glob metacharacters in the thumbnail filename so deleting an image named like [ab].png removes only its own thumbnails. image_gallery / video_gallery listing: filter records against the response schema inside the pager via a valid callback, so offset/limit/has_more all count over accepted records. A leading schema-invalid record no longer returns an empty page with has_more=true and stalls infinite scroll at offset 0. image_gallery / video_gallery save: publish via a temp file plus atomic rename (the sidecar is the video pair's commit marker) and clean up on failure, so a partial write never surfaces a truncated PNG or strands an orphan MP4. diffusion_train_common discovery: treat an empty caption sidecar as a metadata tombstone that still falls through to the dreambooth instance prompt, so clearing every metadata caption no longer fails with no captioned images found. diffusion backend unload: wait for an in-flight denoise to exit before tearing down process-wide patches and state, mirroring the load path. diffusion_engine_router: serialize the whole check/unload/publish transition so a concurrent selection cannot return the engine being unloaded. uninstall.ps1: gate the default sd.cpp process stop on the owner marker so a user's own sd-server is not terminated for a directory we then keep. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reject native batch seeds outside the JSON-safe range * Refuse sd.cpp install into unowned non-empty target dir When the install target already exists, is non-empty and lacks the .unsloth-studio-owned marker (a user's own stable-diffusion.cpp checkout, or unrelated files beside a custom Studio root), install() previously still extracted the release into it. Skipping the ownership marker only stopped the uninstaller from deleting the directory; extraction still merged binaries into the user's working tree and could overwrite same-named files. Fail up front with a clear message pointing the user at a fresh/empty location before any download or extraction, leaving their directory untouched. Update the ownership test suite to assert the refusal. * Studio: gate dataset uploads on the symlink check and surface local video single-file checkpoints * Tighten comments and docstrings added by the image-generation fixes * Studio: close arbiter load-registration race and surface native progress + local pipeline folders Publish native sd.cpp generate progress (_gen) before LoRA resolution so a reload probe reads active during setup, matching the diffusers path. Register the diffusion/video GPU load under the arbiter lock (acquire_for now takes a register callback) so a competing acquire cannot evict an owner before its load is marked in-flight and let two loaders allocate VRAM at once. Admit local diffusers pipeline folders (root model_index.json, weights in component subdirs) in the local model scan so they reach task tagging and the On Device picker. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: include marker-owned custom sd.cpp roots in the uninstall stop scan; harden Pester install against the nuget.exe PSGallery bootstrap * Studio: surface local pipeline scan roots, tag single-file checkpoints by filename, mark companion-only pipelines partial - _scan_models_dir: admit a scan folder that is itself a diffusers pipeline (root model_index.json, weights in transformer/ vae/ subdirs). _is_model_directory rejects such a root, so the child scan would list the component subdirs as bogus models and hide the real pipeline; treat the root as one model via _local_pipeline_index. - _local_is_diffusers / _local_model_task: include the sole checkpoint filename in the family-detection needles (_local_family_needles, resolved via resolve_local_single_file). A generically named folder holding one loadable qwen-image-*.safetensors / ltx-*.safetensors identifies its family only from the filename; the load route already resolves that file, so tag it or the task-scoped picker (which rejects task=null) hides the on-device model. - list_cached_models: mark a companion-only base snapshot partial. A GGUF image load prefetches the base repo's VAE / text-encoder / model_index.json but skips the transformer (the GGUF supplies it); the snapshot has a pipeline manifest yet is not a loadable BF16 pipeline, and _cached_repo_partial misses it. _repo_pipeline_missing_denoiser flags a pipeline snapshot whose transformer/ or unet/ component carries no weight, so the picker drops it instead of advertising it as fully on-device. * Studio: preserve foreign gallery files, force safetensors on remote ControlNets, and close dataset/seed/GPU gaps Gallery clear/delete now scope to Studio-owned files: image_gallery and video_gallery skip PNGs / MP4s without a readable recipe (a hand-dropped or orphan file the listing already hides), so clear() and a guessed-id delete no longer destroy files the gallery never surfaced. Remote ControlNets now force use_safetensors: a bare owner/name reaches from_pretrained without the base trust gate, and the Hub scan fails open when unavailable, so requiring safetensors closes the pickle deserialization vector. POSIX uninstall now stops resident sd-server / sd-cli under an owned sd.cpp root before removing the tree (marker-gated), mirroring the Windows stop-before-delete scan; a live native server no longer survives unlinking its binary. Diffusion dataset containment: the training-start read path and the discovery picker route bare names through the protected resolver, so a symlinked dataset is rejected / not advertised like the caption/delete routes already do. Uploads gain the inference decode guard (oversized real images 400 before OOMing the trainer) and dataset upload/caption/delete/import are blocked with 409 while a diffusion run is active. JSONL readers (trainer + routes) tolerate non-object JSON and invalid UTF-8 instead of raising AttributeError / 500. LoRA family compatibility is enforced in the shared resolver, not only the picker, so a direct API client cannot apply a mismatched-family adapter. GPU arbiter gains release_if so the image/video unload idle-check and release are atomic against a concurrent same-owner load's registration. Native batch recipes persist the base batch_seed and restore replays from it, so a native batch_index>0 image no longer advances its seed twice. FLUX.2-klein selects its sd.cpp text encoder by variant (4B -> Qwen3-4B, 9B -> Qwen3-8B) instead of the single family default. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: track the loaded GGUF filename so native companion resolution reproduces the load identity * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: gate local-pipeline image tagging on a real family; validate video sidecars before delete/clear * Studio: tighten image-generation fix comments and docstrings * Studio: gate gallery serve/export on ownership; keep image progress active until persisted; reserve diffusion training before the dataset scan * Studio: restore Reapply target on async image/video load errors; recheck training state before dataset commit; reclaim partial sd.cpp installs on retry Images/Video: a background model load that fails AFTER starting (error/eviction during download) leaves the previous pipeline resident, but handleLoad had already overwritten lastLoad.current with the failed pick, so "Reapply to loaded model" reloaded the failed model. Carry the prior Reapply target into the poller and restore it on the async error/null paths, mirroring the quant rollback. Training: an in-flight diffusion dataset upload passed _require_diffusion_dataset_mutable() at entry but could still commit files after a concurrent /diffusion/start reserved the training slot, mutating the dataset underneath the trainer. Re-check the interlock immediately before the commit phase; a 409 there leaves the staged temps for the finally to clean. sd.cpp install: an interrupted extraction (disk full, killed process, a raising post-extract cudart fetch) left the target non-empty with no owner marker, so the next lazy install tripped the "not a Studio-managed directory" refusal and wedged native install. Write the ownership marker before the partial writes when the target is reclaimable, so a retry recognises the debris as ours and re-extracts. * fp8 DiT quant: floor the dynamic activation scale with activation_value_lb An all-zero activation token row makes the dynamic per-row fp8 scale 0, which turns the quantized data to NaN and the render to black frames on torchao's plain-torch kernel path. The fused fbgemm/mslk quantize kernels clamp zero rows internally, so the bug only reproduces on machines without them, which is most user environments. Zero rows are real inputs, not a corner case: Wan 2.2 zero-pads its text conditioning, and Hunyuan-1.5 and Qwen-Image regenerate zero rows inside their transformer blocks every step. Pass activation_value_lb=1e-12 to Float8DynamicActivationFloat8WeightConfig whenever the installed torchao supports the kwarg (Float8Tensor rework, 0.13+), checked via inspect.signature so older torchao keeps exactly the current behaviour; the existing Float8MMConfig fallback chain is unchanged. Verified on GPU: with the forced plain-torch kernel path a zero-row input NaNs without the floor and stays finite with it, and end to end on HunyuanVideo-1.5 fp8 goes from a solid black frame (LPIPS 1.00) to a normal render (LPIPS 0.225); on Wan the floor matches the condition_embedder exclusion (LPIPS 0.211 vs 0.206). Same-seed renders with fused kernels present are unaffected, and pre-quantized fp8 checkpoints stay valid since weight scales are untouched. * Wire hosted pre-quantized DiT checkpoints into the image families Point prequant_repos for flux.1, flux.2-klein, flux.2-dev, qwen-image (int8 only there; fp8 is family-denied), z-image and krea-2 at the unsloth/<Model>-FP8 Hub repos carrying gate-validated int8 and fp8 transformer checkpoints, so the fast quant path loads the small pre-quantized file instead of materialising the dense bf16 transformer and quantising on device. Measured on FLUX.2-dev int8: build peak drops from 60.7 GB (dense + quantize) to 30.7 GB (hosted prequant), identical 30.7 GB resident after either path since loading a checkpoint is bit-identical to on-the-fly quantisation. The hosted repos name files <Model>-<SCHEME>.pt, so resolve_prequant_source now derives that model-name filename from the repo id (scheme suffix stripped case-insensitively) and carries the legacy transformer_<scheme>.pt as a fallback the resolver tries when the primary 404s, keeping older repos loadable. Wiring a repo also exposed a fallback hazard: with a prequant source present, the dense-fit preflight used to be skipped entirely, so a failed prequant download would fall through to the dense bf16 load the memory plan never budgeted, OOMing after eviction. The preflight now always runs and gates an allow_dense_fallback flag through _load_dense_quant_pipeline: a dense misfit still skips the fast path when no prequant exists, but with one it proceeds and a prequant failure raises to the GGUF build instead of loading dense. The same flag is set when the auto-policy replans an offloaded GGUF against a prequant-sized transient. Tests updated to the new filename convention plus new coverage for the derivation and the legacy-name fallback; the prequant-skips-refit test now asserts the re-check runs and forbids the dense fallback. Verified end to end on GPU: z-image int8 resolves the hosted repo, downloads the model-name file and renders (6.8s load, 5.9 GB peak). * Route krea-2 through its per-component loader on the transformer-quant fast path _assemble_pipe used Pipeline.from_pretrained for every family, but the krea repo ships transformers-5.x configs and no top-level tokenizer files, so the tokenizer dies with vocab_file=None. The pre-quantized checkpoint loaded fine and then the assembly crashed, dropping the load to the GGUF build, which krea-2 cannot take (Krea2Transformer2DModel has no from_single_file). Assemble per-component via load_krea2_pipeline like the pipeline-kind and single-file paths already do. Verified live: Krea-2-Turbo int8 and fp8 hosted prequant loads now assemble and render through the Studio images tab. * Keep Qwen-Image's text-stream linears bf16 on int8 (short prompts break torch._int_mm) Qwen-Image's MMDiT runs every text-stream Linear at M = actual prompt tokens: the Qwen2.5-VL embeds are not padded to a fixed length like FLUX's 512-token T5. A short prompt (13 tokens) or the near-empty negative prompt drives torch._int_mm below its M > 16 floor and the first denoise step raises 'self.size(0) needs to be greater than 16, but got 13' (measured on B200 through the Studio images tab). Add per-family int8 exclusions (txt_in, add_q/k/v_proj, to_add_out, txt_mlp) for qwen-image and qwen-image-edit, threaded through exclude_tokens_for_scheme(scheme, family) and the prequant checkpoint validation, so a checkpoint baked under the old token list is rejected and re-quantised instead of loaded crashing. The text stream runs at M = tens vs the image stream's M ~ 4k, so the exclusion costs nothing; the rebuilt hosted checkpoint gates 28/28 PASS with LPIPS mean 0.057 (was 0.069). * Harden the diffusion memory plan against transient free-VRAM undercounts A cold FLUX.2-dev int8 load on an idle 183 GB B200 planned offload=model (companions exceed budget) and silently served the GGUF as-is; the identical retry went resident and engaged the hosted prequant. The plan arithmetic was byte-identical across both loads (required 90,228 MiB, resident needs free of about 124 GB); the only divergent input was torch.cuda.mem_get_info, which is device-wide and instantaneous: a transient foreign CUDA context briefly held about 100 GB at the first snapshot, and the planner trusted that single read. Three changes: - settled_snapshot_device_memory: on cuda, synchronize + empty_cache (best-effort) and take the MAX free over up to 3 spaced reads. A transient can only shrink free, so the max rejects transient undercounts while a persistent tenant still caps every read. _plan_memory now uses it. - plan_fits_total_capacity + one replan retry: when the dense/prequant candidate fits TOTAL device capacity under the standard reserve and the 0.85 resident margin, an offload verdict can only stem from the free reading, so the loader re-snapshots and replans once before declining the fast path. Explicit balanced/low_vram modes skip the retry (they offload by mode). - diffusion.transformer_quant_declined log line with required/budget/free and the plan reasons, so the next decline is diagnosable from the server log (previously silent). Verified: cold FLUX.2-dev int8 first load in a fresh server now engages the hosted prequant resident (offload=none). * Add FLUX.2 Klein and FLUX.2-dev DiT LoRA training Register flux.2-klein and flux.2-dev in the DiT trainer following the upstream DreamBooth references: latents train patchified and batch-norm normalized from the VAE posterior mode, the packed forward reuses step-invariant position ids, and the guidance vector (3.5) is gated on the variant's guidance_embeds config. Conditioning stacks load per variant (Mistral via Flux2Pipeline for dev, Qwen3 via Flux2KleinPipeline for Klein) and are encoded and freed before the transformer lands on the device. The fused single-stream to_qkv_mlp_proj joins the attention projections in the LoRA targets; the single-stream out projection stays dense because its to_out suffix would also match the double-stream ModuleList container. Wire both families through the training registry (family set, labels, VRAM notes, rank 16 / lr 1e-4 defaults, bf16-only preflight), mark them trainable with train base repos in the family registry, add FLUX.2-dev to the gated-repo token check, and trust both official bases for training downloads. Verified on B200: 30-step klein int8 (19.6s) and nf4 (20.9s) and dev int8 (52.0s) runs train with finite decreasing loss and the saved adapters apply on the bf16 base pipeline (weight 0 reproduces the base image exactly, weight 1 visibly restyles it). * Support LoRA adapters on torchao int8/fp8 quantized image pipelines Adapters are baked at load time: they attach to the dense transformer, then quantize_ converts only the frozen base linears (the lora_ side path is excluded by name), then the loader compiles. Post-quant PEFT injection is not possible on a manually quantized module, so the prequant shortcut is skipped for a baked load and the memory plan is sized for the dense build (force_dense on the quant candidate). At generation time the baked topology is frozen: weight tweaks and disabling (scale 0 reproduces the quantized base exactly) go through set_adapters, while adding or removing adapters returns a clean 400 telling the client to reload with the new selection. supports_lora now returns True for int8/fp8 diffusers loads (checked before the gguf-kind early return, since the quant fast path keeps the picker kind); nvfp4/mxfp8 and GGUF-via-diffusers stay blocked. The load request model takes an optional loras list, threaded through begin_load on both engines (native ignores it and keeps applying LoRA at generation). Verified end to end on GPU: Z-Image GGUF picker + int8 + trained adapter loads through the API, bake marker logged, weight 1.0 vs 0 renders differ visibly, weight 0.5 accepted live, unknown adapter rejected as 400. Affected suites: 296 passed. * Add FLUX.1 Krea dev to the image model catalog Krea's guidance-distilled FLUX.1-dev finetune keeps the exact dev layout, so it runs under the existing flux.1 family unchanged. Wire it up end to end: - Catalog group with the gated official bf16 pipeline and the open QuantStack GGUF quants; the gated artifact is skipped on auto-routing when undownloaded. - Trust the official repo for non-GGUF from_pretrained loads, next to the other black-forest-labs bases. - Generation defaults: 28 steps at guidance 4.5 per the model card. The generic "krea" defaults key (Krea-2-Turbo's 8-step no-CFG recipe) used to swallow the id, which would have produced garbage output; the new flux.1-krea key precedes it on both the backend table and the images page table. - The flux.1 prequant checkpoints are schnell-based; the loader's baked base_model_id validation refuses them for the Krea-dev base, so int8/fp8 requests dense-quantize instead (covered by existing prequant tests). * Resolve pre-quantized checkpoints per base variant One family entry covers several published variants whose weights differ (flux.1: schnell, dev, Krea-dev), but prequant resolution was keyed on (family, scheme) alone, so only the default base could ever be served: the loader's baked base_model_id validation correctly refused the schnell checkpoint for dev and Krea-dev bases and every such load paid the dense download plus on-the-fly quantise. Add an optional prequant_variant_repos table on DiffusionFamily as (base_repo, scheme, repo_id) triples and thread the resolved base repo through resolve_prequant_source / usable_prequant_source and their three call sites (load fast path, memory-plan probe, auto-policy candidate). A base without its own entry keeps returning the family default, preserving the existing refuse-then-dense behavior exactly. Wire the flux.1 variants: the gate-validated unsloth/FLUX.1-dev-FP8 checkpoints (built in the earlier campaign but never reachable) and the new unsloth/FLUX.1-Krea-dev-FP8. * Add the Lumina Image 2.0 family to the image catalog Alpha-VLLM/Lumina-Image-2.0 is a 2.6B single-stream DiT with a Gemma2-2B encoder and a standard 16-channel VAE, all transformers-4.x-compatible, so the generic from_pretrained pipeline path loads it as a new lumina-2 family: - Family entry (Lumina2Pipeline / Lumina2Transformer2DModel), aliased to lumina-image-2.0 / lumina-image-2 / lumina2. No bare lumina alias: Lumina-Next checkpoints are a different arch and must stay unknown rather than crash mid-load. bf16-only upstream, so the fp16 fallback stays off like z-image. - Trust the official repo for non-GGUF loads; bf16 component table entry (ships fp32, ~5.2 GB transformer + 5.2 GB encoder bf16-resident). - Generation defaults 50 steps / guidance 4.0 per the model card, and the generate call passes the card's cfg_trunc_ratio=0.25 itself (family-gated, signature-gated): the pipeline default (1.0) runs the CFG double-forward on every step and oversaturates output. - Catalog group with the single ungated bf16 pipeline artifact (11 GB resident) plus routing assertions; images page defaults row. - No GGUF artifact: none exists upstream (only finetune/LLM quants), so the dense transformer_quant fast path (GGUF-kind-only) stays unreachable for now. Offline probes of the future prequant campaign: int8 and fp8 both engage and render cleanly (fp8 LPIPS 0.11 vs bf16, int8 0.33 from 50-step trajectory drift with intact quality), so neither scheme is family-denied. * Wire the hosted Lumina Image 2.0 int8/fp8 checkpoints Gate-validated against same-seed bf16 renders (28/28 pairs per scheme, zero failures): int8 LPIPS mean 0.146 / SSIM 0.937, fp8 LPIPS mean 0.116 / SSIM 0.946. Uploaded to unsloth/Lumina-Image-2.0-FP8 following the existing checkpoint repo conventions. * Add the HunyuanImage 2.1 family to the image backend The hunyuanvideo-community diffusers mirror carries the full stack in standard layout: a 17B dual-stream DiT (32.5 GB bf16), a Qwen2.5-VL text encoder, a ByT5 glyph encoder, the 32x HunyuanImage VAE, and guider/ocr_guider components (AdaptiveProjectedMixGuidance) that diffusers 0.39 loads natively, so the generic from_pretrained pipeline path covers everything with no per-component assembly. Family notes: - The call's guidance knob is distilled_guidance_scale (there is no guidance_scale kwarg), so cfg_kwarg routes the UI value there; real CFG runs inside the repo's guider at its baked scale. Defaults follow the card recipe: 50 steps, 3.25. - 2K-native: verified live at both 1024 and 2048. - Coexists with the HunyuanImage-3.0 structured exclusion (3.0 has no diffusers pipeline and stays excluded with its stated reason). - int8/fp8 dense quantization verified live (LPIPS 0.186 both vs same-seed bf16); a short prompt does not trip the int8 torch._int_mm minimum on this arch, so no family exclude entry is needed. - bf16 component table for the memory planner: (32.5, 16.3, 0.8) GB. * Surface HunyuanImage 2.1 in the image model catalog Catalog group with the open bf16 mirror pipeline (~50 GB resident, so a bare click on a consumer card routes to the QuantStack GGUF quants, which load and render through the generic GGUF path, verified live) plus the images page defaults (50 steps, guidance 3.25 feeding distilled_guidance_scale). * Add the HiDream-I1 family to the image backend A 17B MoE DiT (16 double + 32 single layers, 4 routed experts) with four text encoders, on HiDreamImagePipeline (diffusers 0.39). One family covers the open Full / Dev / Fast repos (same arch); per-variant generation defaults follow the upstream inference recipes (Full 50 steps at guidance 5, the distilled Dev 28 and Fast 16 guidance-free). The repos name a Llama-3.1-8B text_encoder_4 in their model_index but do not ship its weights; the official example passes the gated meta-llama repo in by hand. The loader instead assembles the component from the open unsloth mirror (byte-identical weights, already inside the non-GGUF trust gate), injected at the three pipeline from_pretrained sites, with output_hidden_states matching the official example. Memory planning counts the assembled TE4: 34.2 GB DiT + 28.8 GB encoders, ~63 GB bf16-resident. * Surface HiDream I1 in the image model catalog One catalog group with the three official bf16 pipelines (Full, plus the Dev and Fast distillations as labeled artifacts) at their ~63 GB resident size, so auto-routing keeps this a datacenter-GPU pick. city96's GGUF is deliberately not wired: the GGUF path would need the same Llama TE4 assembly for very small demand. Images-page defaults mirror the backend table with the variant keys ahead of the generic hidream key. * Pin the measured HiDream quant verdict in tests int8 and fp8 both engage and render cleanly on this family, including short prompts on int8: the routed MoE expert Linears only ever see the concatenated image+text stream (M far above the torch._int_mm minimum), so no deny entry and no family exclude tokens are warranted. Assert that so a future table edit cannot silently regress the measured behavior. * Wire the hosted HunyuanImage 2.1 int8/fp8 checkpoints Verified bit-identical to on-the-fly quantize: all 1264 state dict tensors (456 quantized) dequantize equal between the loaded checkpoint and a fresh quantize_ pass, so quality matches the runtime Dtype path exactly. Same-seed LPIPS suite means (0.35 int8 / 0.28 fp8) blend trajectory divergence with this family's own run-to-run nondeterminism (identical weights and seed reproduce a 17/255 mean pixel delta through the 50-step guider pipeline); per-case hard checks pass and the drift is compositional, reviewed visually. Uploaded to unsloth/HunyuanImage-2.1-FP8. * Fix silent LoRA drop and wasted transformer prefetch on GGUF quant loads Two live-test findings on the images load path: - transformer_quant with baked LoRAs, when the dense quantized build is declined for memory or fails: the load completed as a plain GGUF with the adapters silently dropped (HTTP success, supports_lora=false after the fact) -- wrong output with no signal. The load now fails with the recovery options (drop the adapters, free VRAM, or pick a smaller model). Weight-0 adapters still count as no bake request, and the plain no-LoRA decline keeps its silent GGUF fallback. - A fresh GGUF load on a small GPU prefetched the base repo's full bf16 transformer shards (~47 GB on Qwen-Image) because the dense-quant prefetch widening only checked scheme viability, not whether the device could ever hold the candidate resident. Gate the widening on total device capacity (reserve + 0.85 margin, the plan_fits_total_capacity bar) so a card that is certain to decline the dense build never pays the download; capable devices keep the prefetch. * Fix video progress under-reporting during load and generate Two live-test findings on the video progress endpoints: - load-progress downloaded_bytes froze mid-download: the counter used scan_cache_dir, which skips in-flight *.incomplete blobs, so it sat at the last completed blob for the whole multi-GB shard pull while the disk kept filling. Count the repo's cache directory directly (completed plus incomplete blobs, snapshot symlinks skipped so nothing is double-counted). - generate-progress reported total_steps=null / fraction=0 while step advanced: the video API only carried the native total field while the image API exposes total_steps and fraction, so one poller could not work against both. Derive the image-compatible aliases in generate_progress and declare them on the response model; the native total stays for back-compat. * Wire the hosted HiDream I1 int8/fp8 checkpoints Gate-validated: all 28 per-case pairs pass per scheme (LPIPS suite means 0.291 int8 / 0.278 fp8, in the 50-step trajectory-divergence band; CLIP delta means 0.007-0.008), and the int8 checkpoint is verified bit-identical to on-the-fly quantize across all 1615 state dict tensors (1073 quantized, max abs diff 0.0). Uploaded to unsloth/HiDream-I1-Full-FP8. * Add a pre-cast text-encoder loader for the layerwise fp8 scheme The runtime text_encoder_quant=fp8 path downloads the full bf16 text encoder and layerwise-casts it in place on every fresh load. For the heavyweight encoders (LTX's Gemma3-27B ~50 GB, FLUX.2-dev's Mistral-24B ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) that download dominates load time on a fresh machine. diffusion_te_prequant.py loads a pre-cast fp8-storage state dict instead: meta-init the encoder skeleton from the checkpoint's te_class, load_state_dict(assign=True), rebuild on CPU if non-persistent buffers stay on meta, then re-apply the same layerwise cast to install the upcast hooks. The cast is a deterministic storage transform, so the loaded encoder is bit-identical to dense-load-then-cast by construction. v1 hosts the layerwise fp8 storage scheme only: its state dict is plain tensors (torch.load(weights_only=True), no pickle execution). The dynamic-compute schemes (fp8_dynamic, int8, nvfp4) build torchao subclass wrappers at runtime and are deliberately not hosted. Checkpoints validate format, scheme, component and base_model_id before use and any problem falls back to the dense download and cast. Local path overrides reuse the DiT prequant allowlist env var. Families opt in via a new te_prequant_repos (scheme, component, repo_id) field on both DiffusionFamily and VideoFamily; the field defaults empty so nothing changes until a gate-validated artifact is wired. * Inject hosted pre-cast text encoders during pipeline assembly Wire te_prequant_pipe_kwargs into the three pipeline assembly sites: the diffusion full-pipeline branch, the diffusion transformer-only and GGUF branch (where the companion TE is the big remaining download), and the shared video assembly path before the pipeline/component split. Injection is gated exactly like the runtime cast (mode normalized to fp8, device supported, family not denied), so it can never engage where quantize_text_encoders would not; the later quantize_text_encoders call re-applies the cast idempotently and keeps status reporting truthful. With no hosted checkpoint configured the call returns {} and assembly loads the dense encoder as before. * Add the pre-cast text-encoder checkpoint builder Applies the runtime layerwise fp8 storage cast to a model's dense text encoder once and saves the cast state dict with baked metadata (format tag, base_model_id, family, scheme, component, te_class, versions) in the layout diffusion_te_prequant.py validates. Resolves the encoder class from the checkpoint's config.architectures so the recorded te_class matches what the pipeline instantiates. CPU-runnable: the cast touches storage dtypes only. * Test the pre-cast text-encoder load path Hermetic CPU coverage for diffusion_te_prequant: the checkpoint filename convention, family-table resolution by scheme and component with malformed entries skipped, resolution priority (path override, hosted repo, none) and the fp8-only scheme gate, the checkpoint validation matrix (wrong format, missing state_dict, wrong scheme, wrong component, wrong or missing base_model_id) with base case folding, the local-path allowlist refusal and missing-file fallback, and the assembly injection gating (mode, hosted entry, device support, family deny, load failure, successful injection). Also pins the te_prequant_repos field on both family dataclasses and that no family ships a hosted TE checkpoint until the campaign wires one. * Fix pre-cast TE checkpoint loading and engagement reporting Two bugs found while building the hosted checkpoints: - The builder recorded torch.__version__ (a TorchVersion object) in the checkpoint metadata, so torch.load(weights_only=True) rejected every artifact and the loader silently fell back to the dense download. Record plain strings. - Re-applying the layerwise fp8 cast to an injected pre-cast encoder raised on the duplicate hook registration, making quantize_text_encoders report the engaged cast as failed (status showed no TE quant while the encoder ran fp8). _cast_fp8 now returns early when the hooks are already installed. Also corrects the LTX TE size note: Gemma3-12B stored fp32 (~49 GB), not 27B. * Wire the hosted pre-cast fp8 text encoders qwen-image and flux.2-dev (diffusion) and ltx-2 (video) now resolve a hosted pre-cast fp8 text encoder from their unsloth -FP8 repos: - unsloth/Qwen-Image-FP8: Qwen2.5-VL-7B, 16.6 GB dense -> 8.8 GB - unsloth/FLUX.2-dev-FP8: Mistral-Small-24B, 48.0 GB dense -> 24.7 GB - unsloth/LTX-2-FP8: Gemma3-12B, 48.7 GB fp32 store -> 13.2 GB Every checkpoint verified bit-identical to dense-load-then-cast (729 / 585 / 1066 tensors, zero mismatches) and smoke-tested through the real backends with the repo engagement marker. Tests cover the wired entries, the resolver filenames, builder metadata weights_only survival, and the idempotent re-cast. * Report the compute dtype on fp8-cast encoders and inject the pre-cast TE on the dense fast path Two more findings from the hosted-TE GPU smokes: - Module.dtype reports the first floating parameter, which after the layerwise fp8 cast is the fp8 STORAGE dtype. Flux2 derives its prompt embed and latent dtypes from encoder.dtype and feeds them to randn_tensor, which has no fp8 kernel, so ANY flux.2 load with text_encoder_quant=fp8 crashed at generation (pre-existing, runtime cast included). The cast now swaps in a subclass whose dtype property reports the compute dtype; forward behaviour is unchanged. - The dense transformer_quant fast path assembles companions through _assemble_pipe, which never received the pre-cast TE injection, so the hosted encoder engaged on full-pipeline and GGUF builds but not on the fast path. Threaded through like the other two branches. Verified live on B200: qwen-image (full pipeline), flux.2-dev (GGUF picker with int8 DiT prequant), ltx-2 (video backend) all engage the hosted TE, render non-black, and report text_encoder_quant=fp8 truthfully. * Key the fp8 cast idempotency on an explicit completion marker Hook presence alone cannot distinguish a legitimately pre-cast text encoder from leftover hooks after a cast that failed mid-pass, so the early return now requires the completion marker _cast_fp8 sets once the hooks are fully installed. Leftover partial state keeps failing closed. Also tolerates non-Module encoder doubles in the hook probe and the dtype override. * Extend the fp8 TE quant to HiDream's Llama text_encoder_4 The generic quantize_text_encoders pass only covers text_encoder.._3, so HiDream's HEAVIEST encoder (Llama-3.1-8B TE4, 16.1 GB bf16) always stayed dense. TE4 is assembled separately (hidream_te4_kwargs), so the fp8 path now lives there: when the requested TE quant is layerwise fp8 and the device/family qualify, TE4 prefers the hosted pre-cast checkpoint (unsloth/HiDream-I1-Full-FP8, 8.6 GB) and falls back to dense-load-then- cast; a mid-pass cast failure reloads a fresh dense encoder instead of shipping partial state. The pre-cast loader and builder gain config_subfolder/config_overrides for standalone encoder repos whose config sits at the root and whose pipeline needs forward flags (output_hidden_states/attentions). Verified on B200: bit-identity 291 tensors (225 fp8, 0 mismatches), hosted checkpoint engages through the real backend (marker + status fp8), load 24.3 s vs 48.0 s dense, LPIPS 0.133 mean over 3 same-seed pairs vs the dense-TE render (gate 0.25), non-black frames. * Correct the ltx-2 resident TE estimate to the bf16 cast size The memory plan's bf16_components_gb held 50.4 GB for the LTX text encoder, which is the fp32 hub store of Gemma3-12B (~49 GB download), not what sits on device: the pipeline loads it torch_dtype=bf16, ~24.4 GB resident. The 26 GB over-estimate pushed the auto plan toward offload on cards that fit the real footprint. Comments and the size-table test now pin the resident semantics. * Host pre-cast fp8 text encoders for four more families Round 2 of the hosted TE set, each bit-identical to dense-load-then-cast and gated through the real backend (marker + status fp8 + same-seed LPIPS vs dense TEs): - FLUX.1 T5-XXL (text_encoder_2): 9.52 -> 5.90 GB, one artifact for schnell/dev/Krea-dev (T5 shards byte-identical across all three, verified sha256). 220 tensors, 144 fp8, LPIPS 0.109. - Lumina Gemma2-2B: fp32 hub store 10.46 -> 3.20 GB (3.3x download cut). 288 tensors, 182 fp8, LPIPS 0.041. - Z-Image Qwen3-4B: 8.04 -> 4.41 GB. 399 tensors, 252 fp8, LPIPS 0.112. NOT shared with flux.2-klein-4B: klein retrained layer 35's MLP (verified tensor diff, maxdiff 0.86), so klein hosts no entry. - Krea-2 Qwen3-VL-4B: 8.88 -> 4.83 GB. 713 tensors, 460 fp8, LPIPS 0.082. The constructor-assembled krea pipeline takes the encoder directly (load_krea2_pipeline text_encoder kwarg); the loader remaps 5.x rope_parameters and re-ties weights after assign so the rebuilt encoder matches the builder's structure. HunyuanImage 2.1 reuses the Qwen-Image artifact outright: its Qwen2.5-VL text encoder is byte-identical (every shard sha256, 16,584,414,544 bytes), recorded in the new component-level base-equivalence table the checkpoint validator consults. The injection loop now covers text_encoder.._3 so a family can host several components. Live check: LPIPS 0.123 vs dense. * Report the fp8-cast compute dtype without swapping the encoder class The dtype override swapped encoder.__class__ to a dynamic subclass, which breaks transformers' kwargs-based output recording: a fp8-cast Qwen3VLModel stopped returning hidden_states and every krea-2 generation with text_encoder_quant=fp8 crashed at encode_prompt (regression from the HiDream TE4 change; caught by the krea hosted-TE live smoke). The override is now a property shadowed on the ORIGINAL class that prefers a per-instance compute-dtype attribute, so class identity is preserved and uncast instances keep the stock behaviour. The idempotency test now pins exact class identity and the uncast-sibling fallback. * Pass the calibrated distilled sigma curve to LTX-2.3 8-step runs The 22B distilled DiT was trained against ltx_core's fixed DISTILLED_SIGMA_VALUES, but the diffusers scheduler derives 8-step spacing from resolution-shifted flow matching and lands far off at every reachable mu (second sigma 0.945-0.981 vs 0.99375, tail 0.37-0.61 -> 0.1 vs 0.725 -> 0.42 -> 0). At the distilled default step count the backend now passes the list verbatim, neutralising the scheduler's dynamic shift and terminal stretch for the call (they distort even explicit sigmas) and restoring them afterwards. Other step counts and the dev/base DiT keep the scheduler's own spacing. Live-verified on B200 through the video branch backend: the scheduler holds the exact curve after an 8-step distilled GGUF generation, config restored, healthy clip. Also reword the transformer_quant resolved reason to the measured reality: quant halves resident weights and hosted checkpoints cut load time, while per-step speed is roughly bf16 parity. * Pin the fp8 weight-quantize kernel against silent MSLK switching torchao's Float8Tensor KernelPreference defaults to AUTO, which switches the weight-quantize kernel to MSLK whenever an mslk package is importable on sm90+. Measured on B200: that changes fp8 scale rounding bitwise (8/8 FLUX matrices differ, scales ~55 percent of bytes), so a box that merely gains mslk would break the hosted-prequant bit-identity invariant; the mslk path is also slower under torch.compile (opaque extern call blocks inductor's quantize fusion, FLUX.1 fp8 e2e 1.149 to 1.624 s). Pin KernelPreference.TORCH explicitly, matching current no-mslk behaviour bit for bit; signature-gated for older torchao. GPU-smoked (finite, rel err 0.037) and pinned by test. * Shift Qwen-Image training sigmas to the inference distribution Qwen-Image's scheduler skips its static shift under use_dynamic_shifting, so the DiT trainer was drawing UNSHIFTED uniform-schedule sigmas for it (mean sigma 0.50) while inference always runs the exponential mu = log 3 shift plus the shift_terminal 0.02 stretch. Add a flow_shift config lever: "auto" (the new qwen-image default) rebuilds the training sigma table through the scheduler's own time_shift and stretch_shift_to_terminal so the draw matches the inference distribution exactly (mean sigma 0.72); a numeric value applies the standard linear shift s*u/(1+(s-1)*u); 1.0 keeps the historical identity behavior and stays the default for FLUX, Z-Image and Krea 2. The model timestep conditioning follows the shifted sigma, gathered in fp32 so bf16 rounding never skews it. Also wire two opt-in levers with off defaults: cfg_dropout (per-sample empty-prompt conditioning dropout, encoded alongside the captions before the text encoders are freed) and weighting_scheme="bell" (bsmntw-style mid-schedule Gaussian loss weighting normalized to mean 1). Verified with two 80-step rank-8 bf16 LoRA runs on Qwen/Qwen-Image (identity vs auto, same seed): both converge with finite decreasing loss and produce coherent same-seed previews. Unit tests cover the exact transform, the shifted sampling distribution, per-family defaults and config plumbing. * Add LoRA EMA, a persistent conditioning cache, and aspect bucketing helpers diffusion_train_extras hosts the opt-in training extras: LoRAEMA shadows only the trainable adapter params (warmup-ramped decay, default 0.99, exported as a second adapter under output_dir/ema), PersistentConditioningCache stores latent posterior stats and caption embeddings as safetensors keyed by content hash + family + resolution, and the aspect-ratio bucketing helpers group mixed-aspect datasets into same-area divisor-snapped shapes. The DiT trainer wires the first two behind config flags that default to the current behavior: ema_decay (0 disables) and cond_cache_dir (None disables). A fully warm cache skips loading the VAE and text encoders entirely; a cache hit is bit-identical to a fresh encode, including the per-channel qwen latent normalization. Also fixes the stale _gather_sigmas call in the perf test that still passed the scheduler instead of the sigma table. * Tighten torchao configs and note the FSDP2 design for the DiT trainer nf4 loads now enable double quantization (~0.4 bits/param off the frozen base scales at no fidelity cost), fp8 training uses the rowwise recipe when the torchao build ships it (per-row scaling confines the DiT activation outliers that a tensor-wide scale collapses), and the inference quant filter gains a per-scheme GEMM-tiling divisibility floor (16 for scaled_mm, 32 for MX blocks) so one ragged Linear cannot crash the first denoise after a clean quantize pass. plans/fsdp2_diffusion_design.md records the multi-GPU design: bf16/fp8 over FSDP2 with per-block units, LoRA attached before sharding, int8 out of scope (DTensor over the quantized subclass is undefined), per-family notes. * Batch diffusion inference with per-image seeds, an inference conditioning cache, and GGUF loader fixes Batched generation: /images/generate takes a prompts list (one image per prompt, txt2img only) or a seeds list (one prompt, one image per seed); the legacy batch_size path derives per-image seeds base..base+n-1 like the native engine. Every image gets its own torch.Generator so any batch member replays alone from its gallery recipe; the whole list runs as one forward by default with OOM backoff that halves a failed chunk, and an explicit batch_size caps images per forward. Validated 10-22x over serial engines on 32-image suites with LPIPS deltas within 0.002. Conditioning cache on the inference path: UNSLOTH_DIFFUSION_COND_CACHE_DIR (the inference sibling of the trainers' cond_cache_dir, same persistent store) wraps encode_prompt so repeated prompts skip the text-encoder forward entirely; verified bit-identical outputs. Bypassed while LoRA adapters are attached; tensor-argument calls pass through uncached. Compile cache: GGUF loads fingerprint their own bundles (quant=gguf, a different compiled graph than the dense family) and batched calls register every distinct (w, h, batch) chunk shape they ran, so the heavy GGUF batched warmups (~159 s at batch 32 on 12B-class, ~655 s on 20B CFG-batched) are paid once ever. GGUF loader: strip the sd.cpp model.diffusion_model. container prefix in the single-file converter; diffusers' FLUX.2 converter KeyErrors on it and the Qwen-Image identity mapping strands the model on meta. * Correct batched seed-replay docs to match measured behavior Same-seed images at the same batch shape are bit-identical; a solo regeneration with the recorded seed matches its batched rendition up to batch-size-dependent kernel numerics (mean abs pixel delta about 2.5/255, LPIPS delta under 0.002), not bit-exactly. The previous wording overclaimed bit-identity across batch shapes. * Note that batched bit-identity assumes a settled compiled graph The first generation issued while the deferred compile is still in flight can deviate transiently (observed once on a cold fp8 build: mean abs pixel delta 0.063/255); once the graph is settled, same-seed same-batch-shape images are bit-identical across runs. * Studio sidebar: Image03/FlimSlate icons, More flyout, Train row, New pills - Images uses Image03Icon and Video uses FlimSlateIcon. - New "More" row (MoreHorizontalIcon) opens a right-side flyout on click or hover holding Video, Recipes and Export; the close is delayed 180ms so the pointer can cross the gap. Its SidebarMenuButton deliberately takes `title` rather than `tooltip`: with `tooltip` the button returns a Tooltip root and DropdownMenuTrigger asChild would hand its ref to a non-DOM node. - Dropped the "Train" section heading; Train is now a top-level row between Images and More. data-tour="navbar" moves to the surviving nav group so the product tour keeps its anchor. - "New" pill beside Images and (inside the flyout) Video, via NavBadge. * Studio sidebar: match flyout rows and New pills to the existing scales - More flyout rows dropped their sidebar-row typography and size-icon override, which fought DropdownMenuItem's own scale (text-sm, gap-2.5, px-3 py-2 and size-4 icons) and rendered oversized glyphs and text next to the nav. - New pill reuses the brand "beta" badge recipe (nav-badge font, --ui-font-scale sizing, nav token colours) rather than hardcoded 9px values. - The More row's native title tooltip (an OS box on hover) is replaced by the app's Tooltip, wrapped around DropdownMenuTrigger so both triggers compose onto the same button, and shown only on the collapsed rail like other nav rows. * Settings: pin and reorder the sidebar navigation Adds a "Sidebar navigation" section to Settings -> Appearance, above the existing profile-menu customizer, with the same drag-to-reorder + switch UI. - New sidebarNav preference: one { id, pinned } entry per navigable row (projects, hub, images, train, video, recipes, export), array order = render order. Defaults match the shipped layout, so an untouched install is unchanged. - Unpinning moves a row into the More flyout rather than hiding it, so no page becomes unreachable. New chat and Search stay fixed as actions. - app-sidebar now renders from one navRows descriptor map, so a pinned row and its flyout counterpart cannot drift; the More row appears only when something is unpinned and highlights off whatever it actually holds. - Mirrored in the backend PersonalizationCustomization: without it the model's extra="ignore" would drop the field, and because sync replaces local state with the server's copy once customization is saved, the user's pin order would reset on the next sync. The validator dedupes and back-fills like sidebarMenu but preserves the client's order, since here order is meaningful. Frontend typecheck, i18n parity and catalog checks pass; 32 personalization tests pass, including a round-trip asserting a reordered list survives a save. * Sidebar customizer: drop the Search row, skip More for a lone item - Search is reached from the top bar, so it is no longer previewed as a fixed sidebar nav row; New chat stays. - More now appears only when it would hold two or more rows. A single unpinned row renders inline in its saved order position instead: a flyout wrapping one item costs a click and earns nothing. The customizer's More preview follows the same threshold. * Sidebar settings: hide a lone unpinned tab, match New chat icon, rename Profile menu - With exactly one tab unpinned, both More and that tab are dropped, so nothing is drawn for it (previously it rendered inline). The page stays reachable by URL. - The customizer's New chat preview uses PencilEdit02Icon, the icon the real row renders; Edit03Icon was a different glyph. - "Sidebar menu" is now "Profile menu", described as the shortcuts behind your name at the bottom of the sidebar, so it no longer reads as a second name for the navigation section above it. * Tighten comments in the new sidebar and delete-guard code * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio sidebar: keep the More row highlighted while its panel is open Moving the pointer into the flyout left the row unhighlighted while the panel stayed open. The row now carries data-menu-open, added to the nav hover selector list. Not data-state: the tooltip and menu triggers both write that attribute, so whichever lands last wins. * Images: use the shared pill toggle for Create/Train and pad the panels - Create/Train was the only segmented control on its own Tabs styling. It now uses PillTabs, the same control as the model picker and Hub toggles, pinned to the header row's 34px. PillTabs takes an icon per tab, so the inline-span workaround for TabsTrigger goes away. - pt-3 on both the Create and Train panels, which sat flush against the model selector row. * Images: make the workflow picker a dropdown instead of a 7-up strip Seven workflows in a 340px rail left ~48px each, so the labels crowded and the hints were only reachable as title tooltips. The strip is now a dropdown: the trigger shows the current workflow and its hint, and each row carries its own description. A row the loaded model can't run is disabled and shows the reason in place of the hint, so the gating explains itself. Adding a workflow no longer shrinks the others. * Images: workflow icons, hint under the trigger, more top room, unclipped Train cards - Each workflow carries an icon, shown on the closed trigger and on every row. - The trigger is one line (icon plus name). The selected workflow's description moved below it, where it reads like the Field hints further down the rail. - pt-6 instead of pt-3 on both Create and Train, so the cards clear the model selector row. - The Train right column scrolls while its cards use ring-1, which draws outside the box and was clipped at the scroll edges. p-px gives the ring room. * Images: one-line workflow rows, stronger trigger fill, roomier mode tabs - Dropdown rows are icon plus name only. The selected row's description already shows under the trigger, and a disabled row keeps its reason as a title. - Trigger fill moves to the bg-foreground/[0.07] dark:bg-foreground/[0.12] pair the hub cards use, so it reads against the card in both modes. - Description under the trigger goes from text-ui-10 to text-ui-11p5. - More horizontal padding on the Create / Train tabs. * Images: drop card borders for the composer shadow, keep scrollbars inside, use app controls in Train - Cards lose ring-1 for .panel-soft-surface: the composer's shadow in light, flat in dark, matching .chat-composer-surface and the menus. - Both rails now clip (overflow-hidden) with the scroller inside, so the scrollbar can't ride over the rounded corner. Same shape video-page already uses. - Train's 9 native selects become the app Select, so they no longer open an OS-native menu, and the native file input is hidden behind a Choose images button that reports the count. - Image previews use explicit 8-10px radii: this theme sets --radius to 1.1rem, so rounded-md was 15.6px and the thumbnails read as circles. * Images: one card for controls and preview, chat sliders, wider softer shadow - Controls and preview were two floating cards; they now share one card split by a divider. The Advanced dock stays separate since it toggles. - SliderField wraps Chat's ParamSlider, so the sliders match Chat (label row with the value, full-width neutral track) instead of a green track with a spin box. All 14 call sites keep their props. - panel-soft-surface goes from 0 2px 8px -2px /0.16 to 0 4px 22px -6px /0.10: lighter, spread wider. * Images: flat Create and Train panes, hover-only scrollbars, tidier Train dataset step Both Images tabs now sit on the page background like the Hub: no card, no shadow, no bounding box. A single rule divides the controls rail from the preview canvas (Create) and from the run area (Train), and the settings and previous-runs sections read as panes rather than nested cards. Also: - Scrollbars in these panes use the existing hover-scrollbar recipe, so the thumb only shows while the pane is hovered. - Workflow rows explain themselves with a tooltip after a short hover, which also works on disabled rows, and the descriptions are much shorter. - Training images rows are name plus image count; the license stays on the example card. - The upload step loses its dashed box, the buttons match the sizes around them, and Upload only appears once files are picked. - The empty preview uses the same icon as the Images nav item. * Images: full-height panes, wider settings rail, Create/Train offset from the selector The rule between the panes now runs the whole page height (the row drops its bottom padding and each pane pads its own content), the settings rail is wider on both Create and Train, and the Create/Train switch sits further right of the model selector. * Images: put both tabs on the Hub's centered measure Top bar and content now share mx-auto max-w-1100 with px-5 / sm:px-8, so Create and Train sit at the same width and position as the Hub instead of running edge to edge. * Images: restore the top bar position, drop the panes lower under it * Images: center the mode switch, flip the arrow with the orientation, app tooltips everywhere The Create/Train switch is centered on the page instead of trailing the model selector, with wider buttons. The flip control's arrows now rotate with the orientation and its label says which way the flip goes. Every native title tooltip on the page is now the app's tooltip, so they all get the rounded surface instead of the OS box. * Images Train: plainer field text, no green buttons, columns that stop colliding - The dataset name, trigger prompt, adapter name and custom base fields now say what they are in plain words instead of leaning on example values. - Import, Upload, Back, Back to settings and Train another are outline buttons, not green ones. - Example thumbnails are landscape tiles, so photos are not cropped to chunky squares. - Settings cells get min-w-0 and the select value truncates, so a long option like the nf4 label no longer widens its column into the next one. - The number stepper sits a little further in from the field edge. - Create and Train are wider. * Images Train: roomier example cards with Import on the thumbnail row * Video: same treatment as the Images tabs - No cards: the rail and the canvas sit on the page background, divided by a rule that runs the full page height, on the Hub's centered measure. - Wider rail, chat's sliders, hover-only scrollbars. - Every native title tooltip is now the app's tooltip, including the clip cards. - Reapply and Cancel are outline buttons, the empty state uses the Video nav icon, and the clip tiles are less rounded. * Images and Video: narrower generation rail, matching Train headings Create and Video rails go from 392px to 368px. Train a LoRA and Training settings are now the same size and both in the heading font: the h2 already picks it up from the base rule, so the settings header opts in with font-heading and the weight that rule pins. * Images Train: shorter copy throughout Family notes, example descriptions, precision labels and every helper line are trimmed so they stop wrapping to three lines and colliding with the next column. The nf4 label now fits its select without truncating. * Images Train: a little more spacing between field groups * Images and Video: tighten code comments * Fix training start NameError, the load-order guard test and CPU-only diffusion tests - start_training forwards resume_source_run_id to _start_training_impl, which reads it. Without it every start raised NameError. - Restore main's anchor in the load-marker order test: the file now has an earlier `if config.is_gguf:`, so indexing the first one compared the wrong branch. - The two diffusion tests that reach diffusers now skip when it is absent, matching the CPU repo-test env. - The UI smoke finds nav rows that live in the sidebar's More flyout. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Treat a null metadata caption as no caption str(None) stored the literal "None" as the caption, so a null row counted as captioned and would have trained on that text. Also drops an unused import. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix invalid-UTF-8 500s, the flat Canny map and the dropped DiT knobs read_text raises UnicodeDecodeError, which is not an OSError, so one bad caption sidecar or video sidecar 500d the info, upload and gallery routes. A flat image now yields the all-black edge map instead of its own luminance, and the four DiT loss knobs the trainer implements are declared so model_dump keeps them. * Use the ui font-size tokens instead of raw px text utilities text-[11px] and friends ignore the UI font size preference, which the repo's font-scale contract test enforces. Same rendered size at the default scale. * Fix diffusion dataset 500s, the dropout-1.0 no-op run and the reset base pick Four correctness fixes on the training side: - The labeling grid read caption sidecars under except OSError, but a non-UTF-8 sidecar raises UnicodeDecodeError (a ValueError), so one bad file 500d /diffusion/dataset/{name}/images and the grid could not be opened to repair it. Read it as no caption, matching the info summary. - An image past Pillow's own hard limit raises DecompressionBombError, which derives straight from Exception and so escaped the upload guard's (OSError, UnidentifiedImageError, ValueError) and returned 500 instead of the intended 400. - lora_dropout accepted 1.0, which makes PEFT build nn.Dropout(p=1.0): lora_A and lora_B receive no gradient and the run saves an untrained adapter while reporting normal progress. Bound it below 1.0, matching the LLM request schema. - The train panel re-seeded the base repo on every dataset refresh because the family object identity changes on each info fetch, so an upload or caption save silently replaced the user's chosen base and the run started on a different model. Track the pick and only re-seed on a real family change. * Show the retained failure when a video page mounts after a failed job Mount-time recovery handled only phase=completed, so reloading the page after a multi-minute generation failed left an idle view with no diagnosis: the backend keeps the terminal failed record only until the next job, and nothing else survives the reload. Surface it the same way the poll does, filtering the cancelled sentinel. * Fix batched generation crashes, cache keying and unreplayable recipes Four bugs in the batched inference path, all found by review: - A mixed-prompt batch sent a scalar negative prompt against a prompt list. Z-Image asserts on the length, and Qwen-Image, Krea 2 and FLUX true-CFG encode a batch-1 negative against batch-N latents and fail in the transformer's text/image concat. Broadcast it to match the batch. - The FBCache step-cache reset sat above the chunk loop. diffusers only resets that state at the end of a successful call, so a forward that raised (the OOM the backoff is meant to recover) left its own residual behind and the halved retry died on a shape mismatch. Reset before every forward instead. - The conditioning cache keyed on the checkpoint alone, but a GGUF or single-file load takes its text encoders from the companion base, so the same checkpoint against a different base reused the previous base's embeddings. Key the base too. - Gallery records stored the base seed and the requested batch size even when a prompts/seeds list drove the run, so restoring the second image of seeds=[5, 99] replayed seed 5. List-driven outputs now record as single-image recipes on their own seed. Also bound strength above 0: every img2img pipeline derives its step count from it, so 0 leaves zero denoising steps and either raises or, on SDXL, crashes on empty latents. * Fix quantized-load LoRA bake, prequant family exclusions and outpaint canvas Six review findings across the Images page and model scanning: - The quantized (int8/fp8) load path can only attach LoRA adapters before quantization, but the frontend load request had no loras field, so every generation after such a load was rejected and each reload repeated it. Send the selection with the load. - build_prequant_checkpoint passed no family to the scheme exclusions while recording the family in metadata, so a Qwen int8 artifact baked the short-M text-stream linears and was then rejected wholesale by the loader's family-keyed check. - Registering a bare single-file checkpoint directory produced no On Device row even though the images loader can load it; only its parent worked. Admit that shape when nothing else matched. - Unload left the Reapply target set, so the repair path was skipped and Reapply reloaded the ejected model. Clear it, as the video page does. - Both FLUX.2 bases were trusted for training but not inference, so Deploy to Create rejected every FLUX.2 adapter. - Outpaint allocated the grown canvas before downscaling, exceeding the browser canvas area cap on a large photo; an over-cap canvas is unusable, so Extend silently posted a fully transparent image and mask. Scale the source first. * Send the picked GGUF filename with the quant so diffusion loads fire The variant expander emitted only the quant label, and nothing else in the frontend set ggufFilename, so the Images and Video pages could never take their GGUF branch: both gate it on meta.ggufVariant and meta.ggufFilename, then fall through to the single-file path, which returns because the id is a repo id and not a .gguf name. Every quant pick was a silent dead click, with no load request reaching the backend. The filename was already on the variant row (the picker keys its list on it, and the variant validator requires a non-empty string), so thread it through the click handler. The chat path is unaffected: it reads ggufVariant and never needed the filename. * Version the conditioning cache key and reject non-finite flow_shift Two correctness fixes: - The cache keyed the checkpoint and its companion base by name only, so a Hub repo advancing to a new commit, or a local directory updated in place, kept returning embeddings from the previous text encoder. Pair both with a revision marker: the locally resolved commit sha for a Hub repo, config plus text-encoder file stats for a directory. Neither loads the encoders, so a warm run still keeps them off the GPU. - flow_shift only checked positivity, but JSON accepts 1e309, which floats to inf, and inf <= 0 is False while NaN fails every comparison. The sigma table then evaluates s * u / (1 + (s - 1) * u) as NaN, which poisons every sampled sigma and saves a corrupted adapter while progress looks normal. Require a finite value. * Keep curated models listed, guard the video companion repo, pin diffusers Three review findings: - The picker filtered every catalog member out of Recommended and Hub search on the way to canonical group rows, but nothing renders those rows yet (catalogGroupFitsDevice and groupMatchesQuery are imported and unused). A task-scoped picker's models list is catalogToModelOptions(), i.e. group members exclusively, so both lists came back empty and no curated model could be discovered or downloaded. Keep the artifacts listed until the grouped UI exists. - The video delete guard compared only repo_id, so deleting the companion base of a loaded GGUF video model was allowed even though it supplies the VAE and text encoders. Compare base_repo too, matching what the images guard already does for its companions. - diffusers was declared unversioned while the diffusion stack requires 0.39 (Krea2Pipeline, the cache_context child registries, the Flux2 and Z-Image pipelines), so an upgrade could keep an older release and selecting an advertised model failed until the user upgraded by hand. * Namespace the trainer conditioning cache per checkpoint, bound the learning rate - The trainer keyed its persistent conditioning cache on family and resolution only, while the keys themselves carry just the caption or image content and crop variant. One cache directory reused for two checkpoints, or for the same repo at a new revision, let a warm run skip loading its encoders and train on the other model's embeddings and latent statistics. Namespace on the base checkpoint and its resolved revision as well. The revision helper now lives beside the cache in diffusion_train_extras and the inference wrapper delegates to it, so the two cannot disagree about what counts as the same source. - The diffusion learning rate only checked positivity, but 1e309 floats to inf and satisfies gt, so the route evicted the resident models and started AdamW with an infinite rate: the first step destroys the adapter while progress looks normal and the result is saved. Bound it below 1.0, matching the LLM schema, which rejects inf for the same reason. * Fix GGUF image model picks doing nothing, and pick the train base in the top bar The quant rows never forwarded the .gguf filename, so every hub GGUF pick on Images/Video fell through to a silent return. On Train the top bar now picks the training base instead of a generation model, which is GGUF-only and untrainable. * Pin diffusion and video loads to the live HF cache root Both read huggingface_hub's import-time HF_HUB_CACHE, which changing the cache folder does not update: progress counted the old root while the download wrote to the new one, and from_pretrained could split one model across both. * Add the diffusion download plan endpoint Reports the repos and exact files a pick needs so the download manager can stage them with the loader's own file scope. A plain snapshot would add the packaged root single, transformer shards and fp16 twins the loader never opens. * Add a file-scoped flavour to the Hub download job Lets a consumer that reads a deliberate subset of a repo stage it through the normal download manager. Keyed as "@scope" so it never collides with a quant or with the repo's full snapshot, and the file list rides the registry so an XET to HTTP retry respawns the same scoped job. * Stage image and video downloads through the Hub download manager They downloaded inline inside the load, so they had none of the manager's disk preflight, manifest verification, resume or panel progress. Picks now stage as scoped jobs carrying the loader's own file list, then load from a warm cache. * Fetch staged GGUF checkpoints as scoped jobs, and stop calling diffusion models unsupported A GGUF entry went out as a full snapshot, whose ignore list drops *.gguf: the job finished at once having fetched only docs, and the repo landed on device unloadable. Every entry is scoped now. The Hub also no longer tags image/video models as unsupported (they run on their own pages), and those pickers name what they select. * Apply the picker task filter to local model sections LM Studio, ./models and custom-folder rows ignored it, so the Images picker listed chat GGUFs that 400 on a diffusion load. The backend already tags every local model with a task for this purpose. * Route a chat pick of a diffusion model to the Images or Video page Chat cannot load one, so it was either hidden or failed on load. The unfiltered picker now lists on-device diffusion models and navigates to the page that runs them, passing the repo and quant so that page loads it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Route the real GGUF filename, keep non-GGUF curated models, key scoped downloads by file set Five review findings, four of them ways a click did nothing or fetched the wrong thing: - A chat pick of a diffusion model routed ggufVariant (a label like Q4_K_M) in the search param the target page uses verbatim as the GGUF filename, so the load asked for a file that does not exist. Route ggufFilename; no filename means a curated non-GGUF pick, loaded as a pipeline. - The task-scoped pickers kept only GGUF repos, so the catalog's bf16, bnb-4bit and single-file fp8 artifacts could not be discovered or downloaded on the Images and Video pages even though loadSpecFor knows how to load them. Keep curated artifacts whatever their format, in Recommended and in Hub search. - Both pages deduplicated routed selections on the model alone, and they now stay mounted, so picking the same repo again -- another quant, or the same one after chat evicted it -- returned early without loading or clearing the query string. Key on model and quant. - Every scoped image download shared one @diffusion job key regardless of the requested files, so switching quant mid-download adopted the running job: the UI waited on the first file set, then loaded a file that was never fetched. Include a digest of the file set in the key. - A scoped plan silently dropped requested files missing from Hub metadata, and snapshot_download succeeds when an allow pattern matches nothing, so the job reported completion and triggered a load with required files absent. Fail the job instead. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the scoped download key derivable, and stop the hidden page hijacking a route Four review findings, the first a regression from my own last commit: - Keying scoped download jobs by a digest of the file set broke the download manager: it builds that key client-side (it polls and cancels before any response tells it a key), so it watched and cancelled a key no worker owned and never fired its ready callback. Keep the derivable "@scope" key and refuse the second request instead when a live job on the slot is fetching a different file set -- decided inside the registry claim, under the lock, so a concurrent claim cannot slip past it. The manager records the file set on the job as well, so a sibling quant's transfer is not adopted locally either. - Both diffusion pages read the route query through a loose useSearch and both stay mounted once visited, so the hidden one consumed the other's ?model=: it navigated back to its own route and tried to load, say, an image checkpoint as a video model. Only the visible page consumes it. - The staged download plan was built without the configured HF token or the Advanced values the load itself sends. The token matters most: the backend's Hub metadata lookup is best-effort, so a gated base silently planned no companion entry and the load pulled those multi-GB files inline, outside the manager. The memory/quant controls decide whether the base transformer/ shards are needed at all, and the route dropped memory_mode, cpu_offload, the prequant path and the LoRA selection before asking for the plan. - The video preview kept playing after leaving the page: the keep-alive layout only hides it, and display:none does not pause a media element, so a clip the user unmuted kept its audio going over the next page. Pause on the active transition and do not auto-replay while hidden. Also completes the hand-built request bodies in the hub download tests: the scoped-files field this branch added to the route read as an AttributeError against them, failing five tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Serialize the GPU handoffs, gate DiT training on a GPU, and keep 3.9 installable Six review findings, three of them evict-then-fail orderings: - The chat load reclaimed the GPU without telling the arbiter it existed. A chat load holds no llama-server process until its GGUF has downloaded, which is minutes, so a competing Images/Video acquire in that window found nothing to cancel, took the GPU, and the chat load then spawned onto the same device. It now registers an in-flight marker through acquire_for's register hook (under the arbiter lock, as the image and video loads do), the evictor cancels a marked load, and the route undoes itself if ownership moved while it loaded. - The Hub-download conflict check ran after that handoff, so a GGUF the download manager already owns destroyed the resident Images/Video pipeline and then 409'd, having loaded nothing. It moves above the handoff, together with the marker it handshakes with. - The image load released the engine router's transition lock before registering the load, so a second load choosing the other engine could unload the still-idle engine this one captured; the load then landed on a deactivated engine, where generate, status, unload and the arbiter's evictor can no longer reach it. Registration now happens under that lock and refuses if the engine changed. - Training a DiT family on a host with no GPU was accepted: nf4 is not a CPU fallback, its 4-bit load goes through bitsandbytes, which requires CUDA, XPU or MPS. The start unloaded the working Images pipeline, pulled the text encoders, and only then died in the child. Rejected before the teardown now, and /info stops advertising a precision that always 400s. SDXL keeps its documented fp32-on-CPU path. - Both diffusion pages kept the routed-pick marker forever, so re-picking the same checkpoint (after chat evicted it) neither loaded nor cleared the query string. The marker is released once the query is gone. The Images key also carried a stray NUL byte, which made the file read as binary to grep and other tooling. - diffusers dropped Python 3.9 in 0.38, so the unconditional >=0.39.0 pin left pip no candidate at all on 3.9 and made every install that composes the huggingface extras unresolvable there. The floor is conditional now. Also fixes tests that were already red on the branch: two hand-built request fakes had gone stale against fields this branch added, and the handoff-ordering test only failed on a host with fewer than two GPUs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the GGUF variant contract test against the merged handler signature The assertion pinned the exact single-line call handleVariantClick(v.quant, v.downloaded, expectedBytes, v.filename), but the handler takes (quant, filename, downloaded, sizeBytes) and prettier wraps the call across lines, so the mandatory repository test job failed on every push. Match the call structurally and assert the filename really is forwarded in the handler's argument order. * Stop a background page and a stale record taking the GPU or a download with them Five fixes from a review pass over the diffusion work. delete-finetuned rmtree'd a model the Images or Video engine was holding: every guard on that route is chat-only, and Images loads any local path, so deleting a local diffusion model under the storage root pulled the weights (and the companion VAE / text encoders sd.cpp re-reads each generation) out from under a live pipeline. The cached-model route already refuses this; the trained/exported one now does too, matching by path rather than repo id, and failing open on a chat-only install so it cannot block ordinary deletes. A staged download finishing while its page was hidden loaded the model and evicted whatever the user was actually using: both diffusion pages stay mounted behind the router and a load takes the GPU unconditionally. The pick is now held until its page is on screen again, which is also what chat does. A scoped download could report success having fetched nothing. With Hugging Face metadata unavailable no manifest is written, so verification is a no-op, and snapshot_download returns an existing snapshot folder without downloading when its own repo_info call fails. A repo already on disk from a full snapshot job (which ignores *.gguf) therefore completed with no weights and auto-loaded against them. The requested file list needs no network, so it is checked against the disk directly. The XET to HTTP retry reclaimed the job slot without the scoped file list, and that claim overwrites the stored record, so a later identical scoped start compared an empty list against the real one and 409'd instead of adopting the running download. The DiT accelerator gate probed torch.mps.is_available(), which only exists from torch 2.5 while the supported floor is 2.4. All three probes shared one try/except, so on torch 2.4 the AttributeError read as 'no block' and a CPU-only host still evicted the resident pipeline, downloaded the encoders and died in the child. Each accelerator is probed on its own now, through torch.backends.mps. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stage what an LTX-2.3 load reads, keep a routed file's load kind, drop an unbakeable LoRA Three from the latest review. The video download plan always asked for the wide base file list, so an LTX-2.3 pick staged the 2.0 base's VAEs, vocoder and connectors that the checkpoint supplies itself, while the companion files the 2.3 assembly does read were left out of the plan and pulled inline at load, outside the panel's progress, cancel and disk preflight. The plan now recognises a 2.3 pick by name (the load keeps the authoritative header probe, and under-guessing only falls back to the load-time pull), narrows the base list, and stages the extras in the same entry as the checkpoint so one repo stays one scoped job. A pick routed from the chat picker arrives as ?model= and ?quant= with no picker metadata, so a bare local .gguf or .safetensors was loaded as a pipeline: an explicit model_kind wins over the backend's filename sniffing, so it evicted the resident model and then failed on the missing model_index.json. Both pages now derive the load kind from the path, the same way their own picker handlers do. A torchao int8/fp8 build takes adapters only at load time. Switching artifact inside one family keeps the LoRA selection, since the family did not change, but the load did not bake it, so the next generation was rejected with 'reload the model with the adapter selection' while the picker still showed the adapter as active. The selection is now dropped once per resident build, with a message saying to pick and load again. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cancel an evicted safetensors load, spare the arbiter for CPU-only chat, fetch clips lazily Four fixes from the latest review round: - The GPU arbiter's chat evictor only cancelled the llama.cpp side. The orchestrator publishes active_model_name once its worker reports success, so an in-flight safetensors load was visible only as an entry in loading_models and finished onto the GPU after ownership had transferred. Cancel every pending load, and give the safetensors branch the post-load ownership recheck the GGUF branch already had. - A manual gpu_layers=0 GGUF load runs on the CPU with the GPUs hidden from the child, yet it took the arbiter unconditionally: it cancelled a running image or video generation for a model needing no VRAM, then held CHAT ownership so the next GPU workload unloaded it for nothing. Gate the acquire on the same predicate the launch-time CPU-only mask uses, as the image and video loaders gate on their resolved device. - The staged-download hook subscribes per repo, not per job, so another job on the same repo advanced the staged queue (starting a load whose scoped files were still downloading) or wiped a queue that was still running. Compare the variant each callback carries, like the chat page's auto-load does. - The video gallery fetched every record of a page into an object URL that lives until the page closes: 50 clips at tens to hundreds of MB each, for cards the user may never scroll to. Fetch a clip as its card nears the strip's edge, plus the selected one the player needs. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Trim the comments across the diffusion backend Comment-only pass over the Python this PR touches: drop what the code already says, collapse multi-line explanations that still read on one line, and keep the reasoning that is not recoverable from the code. No code, docstring semantics or behaviour changes; verified with an AST comparison against the previous revision, and the backend suite is unchanged (same 37 environment failures as before: the API integration tests that need a live keyed server, the flash-attn install hooks, and the GPU memory fields). * Invalidate latents on a VAE swap, keep a cut-off generation, surface the EMA adapter - source_revision() scanned the checkpoint root plus text_encoder/tokenizer but not vae, so swapping or fine-tuning the VAE in place left the conditioning cache namespace unchanged and a warm run trained against latents from the old checkpoint. Include the vae directory, like any other component the cached tensors come from. - /images/generate answers only when the images are saved, and secure mode's tunnel caps an origin response near 100 seconds, which a native CPU or a high-step run passes routinely. The page reported failure while the work kept running, and a retry would duplicate it. A lost response (fetch rejection or a gateway status the origin never answered) is now told apart from a refusal: the page waits out generate-progress and reloads the gallery, so the run it started still lands. - The trainer emits the EMA adapter's path with the terminal event, but the state update dropped it, so neither the run history nor either response schema carried it and an enabled EMA left nothing discoverable. Keep it, and show it next to the primary adapter. - weighting_scheme advertised a choice of timestep sampling; sampling is always logit-normal and the flag only selects the bell loss weights. Describe what it does. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Hide unloadable cached rows, hold the dataset interlock, bound a GIF export - The cached-model listing tagged any repo with a model_index.json as text-to-image, so a community pipeline the image loader's trust rule refuses still got a row in the Images picker, and a detected-but-untrusted video repo fell through to that same tag. Gate the image tag on the load path's rule and hide an untrusted video repo outright. - A routed diffusion pick only carries a GGUF filename, which is all the chat picker has, so a curated single-file artifact arrived with no quant and was loaded as a pipeline: from_pretrained on a repo with no model_index.json. Pass the page's own catalog spec into the route pick, so a routed pick resolves to exactly what a direct pick on that page resolves to. - The dataset mutation endpoints checked is_active() and only then handed their filesystem work to a thread, so a start reserving in that gap changed captions or removed images underneath the preflight or the running trainer. The interlock is now registered for the whole request under the lock reserve() uses, and a start refuses while a mutation is open rather than waiting on it. - GIF export held every kept frame as a paletted image before encoding; a clip may be 2048x2048 for 1024 frames, and at the 12 fps target the step is 1, so one export click could allocate over 4 GB and take the backend down. Downscale past 720 px and widen the step to keep at most 300 frames. - seed accepted any Python int, so an out-of-range one passed every preflight, evicted the resident models, spawned the trainer and only then died in torch.manual_seed. Bound it to torch's 64-bit range in the request and config. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the accelerator probes in the DiT family-metadata tests Six tests read family_train_infos() (or a start preflight) without pinning the host probes, so they only held on a machine with a bf16 accelerator: on a GPU-less runner the DiT gate empties precision_modes, turns supports_compile off, and replaces any other preflight message with the no-accelerator note, and all six failed there. A conftest fixture pins both probes for exactly those tests, so they assert the family metadata they are about on every host. The gate's own CPU-only behaviour keeps its dedicated tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Carry the pipeline task into the hub inventory the pickers read - The task-scoped pickers filter On Device rows on a task, and the chat picker routes a diffusion pick by the same field, but those rows come from the /api/hub inventory, which never carried one: the Images and Video pickers listed nothing on device and the routing never fired. Both cached scans and the local listing now tag rows with the classifiers the models API already uses, the schemas and the frontend adapter carry it through, and a row the backend classified as a generation task is exempt from the chat-only guard that was also dropping it. - The local routing map was keyed by model_id while the row click passes id (a filesystem load id for a models_dir or LM Studio entry), so the lookup missed and the pick fell through to the chat loader. Key both. - A staged download whose start answered "error" left its head in place, where the effect never re-runs and onReady never fires, so the pick was stranded until the user reselected. Clear the queue and say so. - Every scoped pick in a repo shares the @diffusion variant, so the variant alone cannot tell two file sets apart: restaging while the first job finished let its completion pass for the new pick and load a checkpoint that had not downloaded. Bind the callbacks to the repo + file set they started, and to the staging generation. - A rejected generate POST does not say whether it reached the backend, so an immediately idle progress read was ambiguous and a submission that never landed looked like a finished image. Require evidence: progress seen active, or a gallery record that was not there before the POST. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make the OpenAI image URL fetchable, keep WebM audio, stream example imports Four review items on the diffusion Studio work: - response_format=url returned the bearer-gated gallery route, which a standard image client downloads with no Authorization header, so the default response format was unusable. Mint a short-lived HMAC link instead (the shape RAG already uses for pdf.js) served by a signed route, and leave the gallery route itself bearer-only. - A manual gpu_layers=0 load carrying speculative_type="off" -- a value the UI persists and sends -- read as GPU-bearing, so it took the GPU arbiter and evicted a resident image/video pipeline even though the launcher hides the GPUs for it. Canonicalize the mode and exempt "off". - The curated example import prepared the whole split before the loop stopped at the 10-100 image cap; m1guelpf/nouns is 49,859 rows / 328 MB. Stream instead, with the prepared load kept as a fallback for a repo that cannot stream. - WebM export dropped the audio track an LTX-2 clip carries, silently, on the format offered for web embeds. Mux it as Opus through a resampler + FIFO, and keep exporting the video alone on a build without libopus. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not stub out triton on a GPU host when the Xet backend fails to import The lazy loader retries `import unsloth_zoo.hf_xet_fallback` under UNSLOTH_ZOO_DISABLE_GPU_INIT=1 whenever the first attempt raises. That flag makes unsloth_zoo take its MLX/CPU path, which injects triton and bitsandbytes STUBS into sys.modules for the rest of the process. On a working GPU box whose first import failed for an unrelated reason (a bitsandbytes/CUDA mismatch, say) the retry succeeds, so Studio boots looking healthy and then dies at the first CUDA-only kernel: a GGUF or compiled diffusion generation hits the stub and returns NotImplementedError: Unsloth: 'triton.tools.experimental_descriptor.enable_in_pytorch' was called on Apple Silicon / MLX, where triton is stubbed out. so every image generation 500s with an Apple-Silicon message on a Linux CUDA host, while the load reports success. Found by loading Z-Image-Turbo GGUF through the API on a box where bitsandbytes could not initialise. Gate the retry on the host genuinely having no accelerator. The Xet stall watchdog is optional and already degrades with a warning; a process whose triton is stubbed out is not recoverable. The warning now says why it did not retry. * Fix the lost-generation proof set, the settle timeout and the hub inventory's diffusion gates Seven fixes from the latest review round on the Images page and the hub cache inventory. Images page: - The lost-POST settle path built its "already seen" gallery id set inside the catch, after the request failed. By then the earlier runs of the same batch had already prepended their records, so run 2 could accept run 1's image as proof that its own request reached the backend. The set is now captured once before the first POST and grows with every record the batch produces. - settleLostGeneration fell out of its SETTLE_MAX_MS loop and returned normally, so a wedged generation was counted as done and the next run started against a busy backend. It now throws on timeout. - Restoring a recipe cleared the ControlNet selection but left the workflow tab and the init / mask / reference images pointing at whatever was loaded, so the next Generate conditioned on an unrelated image. It now clears all of them and returns to Create. - The download plan omitted the adapter selection the load itself bakes in. A baked LoRA forces the dense build path, so the plan described a different file set than the load that followed and the rest was pulled inline, outside the download manager. Both now derive the list from one helper. Hub cache inventory: - A download for a repo an Images or Video load is staging was allowed to start: only the llama.cpp loader was consulted. Both diffusion backends already expose loading_repo_ids for the delete guard, and the download guard now reads them too. - A companion-only prefetch (pipeline manifest plus VAE and text encoder, no transformer) passed the snapshot-partial check, since every file its manifest expected did arrive, and was advertised as on-device although from_pretrained cannot load it. - The single-file flag never reached the picker through the hub inventory path, so a checkpoint-only diffusion repo read as a full pipeline and failed after the handoff. The two pipeline-shape helpers now live in hub/utils/inventory_scan.py so /api/models/cached and the hub inventory classify the same repos the same way. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop adopting an unknown scoped download, leaking raced blobs and resurrecting deleted clips Three items from the latest review round. A scoped download job carries a deliberate file subset, and every file set of one repo rides the same "@scope" slot. A client that adopts a live job from the backend had no file list to compare against: the active-downloads response never carried one, so an adopted job's set was unknown and any later scoped request for the same repo read as "already started". Selecting a different checkpoint then waited on the wrong transfer and tried to load a file nobody fetched. The response now publishes the scoped file list, adoption records it, and an unknown set no longer satisfies a scoped request. A gallery record can be deleted while its blob is still downloading. The delete revokes the URL present at that moment, so the fetch that lands afterwards inserted a fresh object URL for a record no card renders and nothing can revoke: a full MP4, tens to hundreds of MB, pinned for the rest of the session, and once per raced fetch. Both galleries now discard a blob whose record went away, with an epoch covering the video page's Clear all. The video backend keeps the last completed job until the next one starts, and the Video page merges that record on mount to cover a job that finished after the gallery fetch. Deleting the clip left the record in place, so every reload prepended a ghost card whose file request 404s until another generation replaced it. Deleting the clip, or clearing the gallery, now clears the matching terminal record, and the page skips a record it deleted itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Resolve revisions from the live cache, serialize dataset imports, and stop pinning every gallery blob Five more items from the review round. The conditioning-cache revision marker read huggingface_hub's import-time HF_HUB_CACHE constant. Studio can move its cache during a session and loading follows the live setting, so after a move the marker went unresolved (or pointed into the previous root) and pulling a new revision of the same checkpoint no longer invalidated the cache: a warm run could reuse the old encoder's embeddings and the old VAE's latents. It now looks in the active Studio cache first and keeps the environment and the library constant as fallbacks, which the trainer subprocess still needs. The dataset interlock counts mutations rather than excluding them, so two imports of different examples into the same empty name both got past the emptiness check. The winner promoted its staging directory atomically; the loser found the folder non-empty, fell back to a per-file move, and merged its images and captions into the winner's dataset. Imports now take a per-folder lock, a second one is refused with 409, and the emptiness check is repeated under the lock. On Windows the sd.cpp asset resolver filtered only by accelerator token, so a Windows arm64 host matched an x64 zip, downloaded and installed it, and failed later when the binary would not run. It now filters by architecture the way the Darwin and Linux branches do. Every gallery page fetched every PNG up front and kept the object URL for the session, so scrolling a large gallery grew memory without bound for tiles the user may never look at. The Images strip now fetches a tile as it nears view, like the Video strip, and keeps the eager path only where IntersectionObserver is unavailable. A 503 carrying a JSON body comes from the application, not a proxy, so it is surfaced as the error it is instead of entering lost-response settlement and being reported as a request that never reached the server. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Name the class of a failed generation instead of a bare "Image generation failed." Found on a macOS runner: the native renderer aborts inside its own text encoder there, and the page showed only "Image generation failed." with the sd-server backtrace left in the server log, so nothing about the failure reached the user. The failure is now classified into fixed text, out of memory and native-process death, so the message says what happened and what to try. None of the engine's own output is echoed, since a native tail carries local paths and argv; that stays in the log, and an unrecognised failure keeps the original literal. * Treat an undecodable caption sidecar as the tombstone the trainer sees Uploads store .txt and .caption sidecars as raw bytes, so one can hold invalid UTF-8. The trainer treats any existing sidecar, decodable or not, as an empty tombstone and never falls back to the metadata row for that image. The labeling grid and the dataset summary read an undecodable sidecar as absent instead, so both showed a metadata caption that the run would silently replace with the instance prompt, and counted the image as captioned. Both now track sidecar presence separately, so what the user reviews is what the run trains on. * Keep the reason a native server died, not just its backtrace A ggml abort prints its cause first and then a stack trace, so reporting the last twenty captured lines gave twenty addresses and nothing about the failure: on the macOS runner the native server died on an unimplemented Metal op and the message carried only frame pointers. The captured tail now leads with the lines that name a cause and keeps recent context after them, for both the startup failure and the mid-request death. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop staging the dense text encoder for an fp8 video load Two halves of the same gap, found while measuring the LTX-2.3 download plan: - The video download plan and the scoped pre-download never saw text_encoder_quant. An fp8 request loads a hosted pre-cast encoder, so asking for one still staged and downloaded the base repo's dense Gemma3 (48.79 GB of Lightricks/LTX-2 on the 2.3 distilled pick) that the pipeline then never opened. The plan now drops those shards and stages the pre-cast checkpoint instead; their configs stay, since the pre-cast loader still meta-inits the encoder from the base repo's component config. - The LTX-2.3 assembly builds every component itself, so pipe_kwargs (which carries the pre-cast encoder for from_pretrained) never reached it and an fp8 request silently loaded the dense encoder anyway. It is passed across explicitly now. The dense skip is earned, not assumed: only a pre-cast checkpoint that resolves on the Hub lets the plan drop the dense shards, and only one already fetched to disk lets the pull drop them, so an unpublished or gated artifact leaves both exactly as they were. If injection still fails after that, the load tops the dense weights back up rather than handing from_pretrained a snapshot with no encoder in it. Measured against the real Hub on the 2.3 distilled Q4_K_M pick: 67.24 GB before, 18.92 GB with a 0.43 GB stand-in for the pre-cast artifact (the base entry drops from 24 files / 48.79 GB to 13 files / 0.04 GB). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Match the file's typing idiom for flow_shift models/training.py annotates with typing constructs throughout (105 Optional[...], no Union), and flow_shift was the one place using PEP 604. Union[] reads like the rest of the file, and it also drops the runtime evaluation that would raise on Python 3.9. * Bound the gallery blob cache, and three interlock fixes Four review findings, all reproduced first: - The gallery object-URL caches were unbounded. A clip runs from a few MB to a few hundred, both pages stay mounted after their first visit, and entries were only dropped on delete, so scrolling pinned everything for the session. Both pages now share a byte-budgeted LRU (512 MB video / 192 MB images) keyed off the visibility signal the near-viewport fetching already provides. On-screen media, the selected clip or image, and the item just fetched are never evicted, so eviction is invisible and a single item larger than the whole budget cannot evict itself into a refetch loop. - The image, video and chat load guards ran two independent training probes but returned early when the FIRST one raised, so an unreadable LLM backend disabled the diffusion interlock and a load could proceed straight into an active diffusion trainer on the same GPU. The probes are independent now. - An engine switch swallowed a failed teardown and published the new engine anyway, which is exactly the leak the unload exists to prevent: the arbiter's evictor, /images/unload and the next load all resolve through get_active_diffusion_engine(), so the still-resident pipeline (or a live sd-server) became unreachable and the next load allocated on top of it. The switch now fails and leaves the old engine published, so it stays reclaimable. - The native generation timeout was 30 minutes while the Images page waits up to 6 hours (SETTLE_MAX_MS), so slow-but-progressing CPU jobs died deterministically at the deadline. Measured on GPU-less runners, a 512x512 4-step Q2_K generation took 900 s on Linux and 1465 s on Windows, so larger images or step counts clear half an hour easily. The ceiling now matches the page's window and applies to the whole request: chunks of a split batch share one deadline instead of each getting a full budget. Cancellation is unchanged. Declined: gating the huggingfacenotorch extra off Python 3.9 over the conditional diffusers marker. The marker is deliberate and its comment says why: diffusers dropped 3.9 in 0.38, so pinning >=0.39 outright leaves pip no candidate and the whole extra unresolvable there. The pipelines it names live in studio/backend, which cannot install on 3.9 anyway (studio.txt pins matplotlib==3.10.9 and fastmcp>=3.0.2, both requires_python >=3.10), and the extra is the general core one, so the alternative drops 3.9 for library users who never touch Studio. * Close the load-versus-training-start race, and two picker fixes - The image and video load guards read is_active() and only then selected an engine, acquired the arbiter and registered the load. A /train/diffusion/start reserving inside that window freed residents the load had not registered yet, so the trainer came up beside a brand-new pipeline. The service already had exactly the right pattern for this in dataset_mutation, so gpu_load_admission mirrors it: reserve() refuses while an admission is open, an admission refuses once a start is reserved, both decided under the one lock. The span is only the registration, since begin_load returns as soon as the load is registered and _free_gpu_for_diffusion_training preempts an in-flight load from that point. Chat is deliberately not covered: its load spans an eviction plus a multi-minute GGUF load, and it admits models that fit beside training by design, which is a different contract from the diffusion pipeline's all-or-nothing one. - Hugging Face gives the LTX-2 family the image-to-video pipeline_tag (both Lightricks/LTX-2 and unsloth/LTX-2.3-GGUF report it), so a text-to-video-only filter dropped the flagship audio family out of Video Hub search while the rest of the app routed it to Video. - Task-scoped quant fit sized picks against the LARGEST visible device while resolve_diffusion_device_target returns a bare "cuda" and torch places on the current one. On a heterogeneous host that recommended a checkpoint sized for the bigger card and then loaded it onto the smaller one. Fit now uses the device the load actually lands on; identical on a homogeneous host. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Expose the persistent conditioning cache in the start schema DiffusionLoraConfig has carried cond_cache_dir for a while and the DiT trainer acts on it, but DiffusionTrainingStartRequest omitted the field, so Pydantic dropped it silently and every API-driven run fell back to the in-memory cache that is rebuilt from scratch each time. The warm path skips loading the VAE and the multi-GB text encoders on a rerun whose images, captions and resolution are unchanged, so this was a real capability that could not be reached. Contained like output_dir rather than left to the trainer subprocess's cwd, since it is another directory the trainer writes to. Blank or omitted still means the in-memory cache, so it must not resolve to the outputs root. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix diffusion policy and classification issues from review fp8 auto precision defaulted to precise accumulate on any non-consumer GPU, which made fp8 2.05x slower than int8 on RTX 6000 Ada and slower than not quantising at all. NVIDIA's professional whitepapers do publish equal FP8 rates for both accumulate modes there, so the hardware premise held, but the cost is in the cuBLAS path rather than the published rate. Default to fast accumulate: measured on B200 the flag is a no-op (4096^3 _scaled_mm at 3023.8 vs 3041.8 TFLOP/s, bitwise-identical output, 1.213 s vs 1.230 s end to end), so it is a large win where it bites and free where it does not. Precise accumulate stays available via transformer_quant_fast_accum. Z-Image's DiT is a Lumina2 derivative, so unsloth/Z-Image-GGUF and unsloth/Z-Image-Turbo-GGUF both declare general.architecture = "lumina2" and the whole line was tagged image-diffusion-unsupported and hidden from the Images "On Device" list, though validate_load_request loads them. Resolve shared archs from the repo/file name like bare "wan" already does, with a test asserting the picker and the loader agree for every family. The sage attention on-demand install ran an unpinned `pip install sageattention`, but PyPI's newest wheel is 1.0.6 and diffusers refuses anything below 2.1.1: the install always "succeeded", wrote an unusable version into the running venv, and was rejected on the next line. Carry the dispatcher's floor so pip resolves nothing instead. The dense-quant disk gate sized the download from the bf16-RESIDENT table. The fp32 families download twice that (Z-Image: 23,479 MiB against a 21,970 MiB gate), leaving a window where the check passed and the download filled the disk; Ideogram 4 ships fp8 and was overcharged the other way. Size the gate by published bytes, verified against HF sibling metadata for all 12 families. Patch installs went through unsloth_zoo, which refuses to import unless UNSLOTH_IS_PRESENT is set, and that is set by unsloth itself. The server imports unsloth at boot so it never showed there, but any other process ran silently unpatched with every install returning False, which is 13 test failures on a clean environment. Import unsloth and retry once, memoised per process. Also: the GGUF+LoRA refusal pointed at the native engine without saying a GPU host only selects it under UNSLOTH_DIFFUSION_ENGINE=sd_cpp, so the suggestion was unreachable; the gallery recipe recorded loras from the generate request alone, losing a load-time bake; load-progress claimed "40.07 GB downloaded" for a fully cached load; and pickers.tsx imported three catalog-group helpers it never used. Reported by oobabooga. * Keep the sd.cpp text encoder on CPU under Metal macos-14 loads FLUX.2-klein-4B Q2_K natively on mps and then dies on the first generation with exit code -6: ggml_metal_op_encode_impl: error: unsupported op 'RMS_NORM' -> ggml_abort LLMEmbedder::encode_prompt -> LLMRunner::compute -> GGMLRunner::compute ggml's Metal backend gates RMS_NORM on contiguous rows and aborts the process when that does not hold, with no per-op CPU fallback, so any LLM text encoder (Qwen3 for FLUX.2 and Z-Image, T5 for FLUX.1) takes sd-server down. The encoder runs once per prompt while the DiT runs every step, so pinning only the encoder keeps Metal for the part that matters. UNSLOTH_DIFFUSION_SD_CPP_METAL_TE_GPU=1 opts back in once ggml grows the kernel. * Gate the unsloth retry in the diffusion patch backend The retry added for the clean-environment patch failures is not free: importing unsloth pulls torch in behind it, which costs ~940 MB of RSS measured in a process that had neither, and on a host with no accelerator it fails anyway. A cross-platform CI job that had generated fine at ~900 s later died 19 s in with SIGTERM and every 'if: always()' step skipped, which is the runner being torn down rather than a step failing. Retry only when torch is already imported (true of the server and of anything patching a real module, and the condition that stops the retry from being what loads torch), unsloth is installed but not yet imported, and the first failure was the ImportError the sentinel guard raises. The clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Only retry the unsloth import where it can succeed The gate still let the retry run on hosts unsloth does not support, which is where it is most harmful: a 7 GB macOS runner lost the Studio server 26 s into a load, and the Linux runner was torn down mid-generation. Neither MPS nor plain CPU can complete the import, so the retry there pays the cost and fails anyway. Require an accelerator unsloth actually supports (CUDA/ROCm via torch.cuda, or XPU), with UNSLOTH_ALLOW_CPU as the documented override, and hoist the predicate to module level so it is tested directly rather than through the import system. On a CPU-only host the retry no longer fires at all; on CUDA the clean-environment case it was added for still passes 29/29. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Guard old diffusers, stream video exports, record conditioned recipes Three fixes from review. The 0.39-only pipeline classes (Flux2Klein, Z-Image, Krea 2, LTX-2, HunyuanImage) were resolved by getattr deep in the load, so on the older diffusers that packaging still allows on Python 3.9 -- diffusers dropped 3.9 in 0.38 and this project still supports it, so the 0.39 floor has to be conditional or the extra becomes unresolvable -- an advertised model failed with a bare AttributeError after its checkpoint had already been downloaded. Krea 2 already guarded itself this way; assert_pipeline_class_available now runs the same check for every image and video family from validation, before any fetch, and names the version and the fix. WebM export accumulated the whole VP9 output in a BytesIO and returned it as one bytes object that the response held again. The request caps allow 2048x2048 for 1024 frames, so an export runs to hundreds of MB and concurrent clicks could exhaust the process, while the MP4 route beside it already streamed from disk. transcode_to_file encodes to a temp file and the route returns a FileResponse with a background unlink, so nothing large is resident. A conditioned generation's recipe carried only the txt2img fields, so the gallery presented an inpaint or upscale result as a complete Create recipe and restoring it replayed an unrelated text-to-image request. The images themselves are still not persisted (user uploads with their own lifetime), but the workflow and its scalars are, restore reapplies them, and the toast now names the inputs that have to be supplied again instead of silently landing on Create. Reported by Codex. * Per-load video cancel event, family-gated image picker, cond cache refusal A cancelled video load could resume: begin_load cleared the shared cancel event, and unload() drops _loading without waiting for the worker, so the next load cleared the very object the cancelled worker was watching and its multi-gigabyte pull ran on alongside the replacement until the token check at the end. Each load now gets its own threading.Event, passed down through _fetch_te_prequant and _predownload_base, so a cancelled worker stays cancelled. A cached repo with a model_index.json was advertised as text-to-image on the trust rule alone, but validate_load_request also requires a detected image family, so a trusted pipeline of an unsupported class produced a picker row that deterministically 400s. The picker now applies both gates, mirroring the video branch. cond_cache_dir was accepted for sdxl and then ignored: only the DiT trainer reads it, while the SDXL trainer builds a per-run in-memory latent cache, so the promised cross-run reuse never happened. The route now refuses it with a 400 that names the families which do support it, checked against the resolved family so an omitted model_family with an SDXL base is caught too. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix the frontend build broken by the gallery blob cache tsc -b failed on the branch head, so npm run build produced no dist and every platform job fell back to --api-only: blob-url-cache.ts(29,15): TS1294: This syntax is not allowed when 'erasableSyntaxOnly' is enabled dataset-labeling-grid.tsx / dataset-showcase.tsx: Argument of type '{ url: string; bytes: number; }' is not assignable to parameter of type 'string' The cache took its budget as a constructor parameter property, which the project's tsconfig forbids, and fetchGalleryObjectUrl now returns the blob size alongside the URL for that budget, which the two dataset thumbnail components still consumed as a bare string. Declare the field explicitly and destructure the URL at both call sites. tsc -b is clean and vite build emits dist again. * Recover from a ggml unsupported-op abort by restarting on the CPU backend ggml checks every node against the device's supports_op and calls GGML_ABORT when one is not implemented, because a single-backend graph has nowhere else to put it: there is no per-op CPU fallback. The whole sd-server dies with SIGABRT mid-generation and the user gets "the native image renderer stopped unexpectedly" with no way forward. Seen on macos-14 arm64 with FLUX.2-klein-4B Q2_K through the cross-platform CI: the text encoder is already pinned to CPU, and the abort moved into the denoise loop instead. ggml_metal_op_encode_impl: error: unsupported op 'MUL_MAT' -> ggml_abort StableDiffusionGGML::sample -> sample_k_diffusion A retry on the same backend would abort identically, so the load is restarted once with --backend cpu (the only flag that changes which backend executes the graph; --offload-to-cpu moves parameters, not compute) and the generation is re-submitted. The same checkpoint then renders slower rather than not at all. Strictly bounded: the signature must carry both the unsupported-op line and ggml_abort, the device must not already be CPU, and it happens once per load, so an OOM kill or a genuine crash still surfaces as itself. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix two tests that only fail in a full-suite run The 3.10 CI leg resolves PyAV 17, where av.container.OutputContainer is an immutable C type, so the no-libopus export test died on "cannot set 'add_stream' attribute of immutable type" before it asserted anything. Inject the refusal by wrapping the container av.open() returns instead; modules stay patchable on every build. Removing the injection makes the test fail again, so it still covers the branch it is named for. The Xet shim's degraded-path tests drop utils.hf_xet_fallback from sys.modules and import a throwaway copy. Restoring only the sys.modules entry left the utils package attribute bound to the throwaway, and the two disagreed for the rest of the process: a later monkeypatch of the dotted target patched one copy while the code under test imported the other, so the patch did nothing and test_fetch_te_prequant_only_reports_what_it_downloaded reached the real Hub and got a 401. Restore both bindings. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Stop an ignored-cancel sd-server, guard deletes during diffusion training, repair unusable managed binaries sd-server does not interrupt an in-flight job, so when it ignores a cancel the grace branch abandoned the poll and reported cancellation while the native job kept a core (or the GPU) busy to completion and held the server's job slot. The comment said the caller stops the server, but only unload does that immediately: a superseding load stops it after its multi-gigabyte download, and a load that then fails never gets there. Stop it here, as the deadline branch already does. DELETE /api/models/delete-finetuned checked only the LLM trainer, so it could rmtree the output directory a live diffusion LoRA run was about to write its adapter into. Consult the diffusion training service too, like the dataset mutation and model-load routes. find_sd_*_binary only checks is_file(), so an interrupted extraction (or a prebuilt for the wrong CPU) left a present-but-unrunnable binary the installer never retried: every load probed it, fell back to diffusers, and native inference stayed off until the directory was deleted by hand. Probe it and reinstall, but only for a copy under the installer-owned root -- SD_CLI_PATH, UNSLOTH_SD_CPP_PATH, an in-tree build and anything on PATH are the user's. * Plan the pre-cast text encoder, and make the cross-trainer GPU admission atomic An fp8 text-encoder request loads a hosted PRE-CAST checkpoint, but the image download plan never received text_encoder_quant, so the manager staged the base repo's dense encoder (FLUX.2-dev's Mistral-24B is ~48 GB, Qwen-Image's Qwen2.5-VL ~16.6 GB) and the load then pulled the pre-cast file inline, outside the manager's progress and disk preflight. The plan now takes the field, resolves the hosted artifact with the same resolver the injection uses, stages that file, and drops only those components' dense weight shards. The load's own prefetch takes the same treatment, since it paid the same cost. Only a checkpoint that really resolves on the Hub earns the drop, so a gated or renamed artifact still stages the dense encoder the load will fall back to. The two trainers admitted each other with independent check-then-act guards: the diffusion route checks the LLM backend several network-bound preflights before it reserves, and the LLM route checks the diffusion service well before it spawns, so two near-simultaneous starts could both pass and train on one GPU. reserve() now re-tests the LLM backend under its own lock, and the LLM route holds the diffusion service's gpu_load_admission across its spawn, so exactly one of the two wins. Both halves fail open, so a chat-only install still trains. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pin the Advanced options a staged download planned against Staging does not set busy, so while a multi-gigabyte download runs the user can still change precision, memory mode, speed or the baked LoRA selection. The pending record held only the repo and artifact, and the completed download fired a load that read the CURRENT state: the staged file set could then be missing files that load needs (fetched inline, with no progress and no disk preflight) or hold gigabytes it no longer uses. One snapshot of every Advanced control is now taken when the plan is built, and it travels with the pending record into the load, so the load that runs is the one the download was planned for. * Do not advertise a family the installed diffusers cannot build The newer families (Z-Image, Krea 2, FLUX.2, LTX-2, HunyuanImage) exist only from diffusers 0.39, and 0.39 cannot be installed on Python 3.9 at all -- diffusers dropped 3.9 in 0.38, so the requirement is conditional or the whole extra becomes unresolvable. On such an environment the picker still offered those rows, every pick failed deterministically, and the error's advice to run pip install -U diffusers could not fix it without also upgrading Python. The cached-repo picker now applies the same availability check validate_load_request does, which is keyed on the pipeline class actually present rather than on the Python version, so it is also right for an intentionally pinned older diffusers on 3.10+. Fails open when diffusers cannot be imported at all: that is a different problem and the load path reports it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Restore the diffusion engine selection after each router test The active engine is module state, and several tests set it by plain assignment because what _activate does to it is the thing under test, so monkeypatch could not undo it. A leaked ENGINE_SD_CPP left get_active_diffusion_engine() handing back the sd.cpp backend for the rest of the process, and every later route that reads the active engine then saw an unloaded model: eight tests in test_openai_images_generations_route.py returned 503 in a full-suite run while passing on their own. The autouse fixture now snapshots and restores it. * Stream gallery clips, and close three races around them Four fixes from the latest review pass. The video gallery downloaded each clip into a blob before it could play, so playback waited on the whole file (tens to hundreds of MB), seeking was limited to what had arrived, and every viewed clip stayed pinned in the webview. The file route already streams and serves ranges; it just could not be a <video src> because it is bearer-gated. Mint a short-lived signed link instead (its own HMAC secret, 12 hour TTL, separate from the image links) and hand it to the element, which then fetches only the ranges it plays. That removes the blob budget, its LRU and every revoke on this page. The sd.cpp readiness probe accepted any process answering on the port, so a foreign server that grabbed the port between the bind check and the spawn was adopted as ours. Confirm the listener is our child before reporting ready, and stay best-effort (psutil missing, an unknown owner, or any probe error still passes) so the check can only reject a definitely foreign process. Dataset import held its lock for the extract but not for the upload path, so two concurrent uploads into the same folder interleaved; take the same lock and return 409. And reject Windows device names (CON, NUL, COM1..9, LPT1..9, with or without an extension) plus trailing periods in dataset names, which are unopenable on Windows. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Build the image download plan for the engine that will load /images/download-plan always asked the diffusers backend, while /images/load picks the engine per host: a GGUF pick on a machine with no usable GPU routes to native sd.cpp, which reads a single-file VAE plus text encoders and never opens the base repo's sharded components. Measured on unsloth/FLUX.2-klein-4B-GGUF (Q2_K): the plan staged 7.66 GB of FLUX.2-klein-4B components the native load discards, and the 7.80 GB sd-cli actually needs was then fetched inline by the loader, outside the download manager's progress and its disk preflight. Z-Image-Turbo is the same shape. The plan now asks whichever engine the load will select. predict_engine() applies the selection policy without any side effect: it activates nothing (staging a download must not unload the resident model) and only locates the binary rather than installing it, but still counts an installable binary as available, since that is what the load does on a fresh host. The native backend gains a download_plan built from the same _asset_specs the loader fetches, returning the same envelope, so the manager stages exactly the files sd-cli opens. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not let a queued generation outlive the model, and three scan fixes Five items from the latest review; four were real. An unload or arbiter eviction only cancels the generation holding _generate_lock. A second request queued behind it holds no cancel event yet, and Python locks are not FIFO, so it could take the lock the instant the active denoise released it, still see a loaded pipeline, and run a whole new denoise after the model was told to go away: the eviction then waits minutes for it and an image lands after the eject. Unload and a superseding load now raise a fence under _lock before they queue, and a generation that wins the lock while one is pending refuses instead. The cached-model scan judged pipeline completeness across every revision, so a repo holding an older complete snapshot plus a newer companion-only one read as complete while the snapshot from_pretrained actually opens has no transformer. Both scans now look at the revision the loader will open. Deleting a dataset image deleted its caption sidecar unconditionally, which for cat.jpg alongside cat.png removed the caption the survivor still resolves to. The sidecar now goes only with the last image of that stem, matching what the thumbnail cleanup beside it already did. Importing an example into a folder that holds no images but does hold files fell back to promoting the staging dir one file at a time, so an interruption left a partial dataset that the image_count check accepts as complete on retry. Those files are folded into the staging dir instead and the promotion stays a single atomic rename. The MPS generator report does not apply: torch.Generator(device="mps") has worked since PyTorch 2.0 (pytorch/pytorch#91348) and the studio installer pins torch>=2.4. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten diffusion comments Collapse the multi-line comment blocks across the image, video, sd.cpp and diffusion-training code to one or two lines each, and drop comments that only restate the statement below them. Comments only, no code or behaviour changes. * Tighten diffusion comments (second pass) Collapse the remaining multi-line comment blocks in the video page, training routes and service, sd.cpp server and installer, memory and speed planners, and the shared request models. Comments only, no code or behaviour changes. * Tighten diffusion comments (third pass) Collapse the remaining multi-line comment blocks in the attention, cache, LoRA, prequant, precision and compile-cache modules, the sd.cpp arg builder and engine, the video routes, the Ideogram 4 assembly, the model picker, and the diffusion test suites. Comments only, no code or behaviour changes. * Restore the dataset when an example import cannot be promoted Promotion folds the folder's pre-existing entries into the staging dir so the swap is one atomic rename. Every failure after that fold left the user with nothing: the 409 path and an os.replace error both fell through to 'finally: shutil.rmtree(staging)', which deleted the entries that had just been moved in there, while the response said 'Nothing was written'. A same-named entry was also unlinked outright before the promotion was known to succeed. Park superseded same-name entries in a rescue dir instead of deleting them, record every move, and restore all of them if any step of the promotion fails. The fold loop itself is covered too: renaming a non-writable directory raises EACCES on POSIX, which previously escaped as a 500 after the earlier entries had already been moved out. A failed rename now maps to the same retryable 409 as the rmdir conflict. Verified with the reported trigger (a non-empty mode-500 directory whose name collides with an imported file): the folder listing is now identical before and after the failed import. * Drain the teardown fence on a failing unload, give each load its own cancel event Two independent leaks on the image path, both already solved elsewhere in the same file. unload() incremented _teardown_waiters, ran _unload_locked() and decremented, with no try/finally, while the superseding-load path used a finally for the same pair. _unload_locked ends in clear_gpu_cache(), whose CUDA branch calls synchronize/empty_cache/ipc_collect unguarded, and a sticky CUDA fault makes those raise. The count then never drained, so every later generation was refused as cancelled for the life of the process, a fresh load included, since begin_load's own increment and decrement are symmetric. unload() is reached from the chat/video GPU handoff, the engine router and two training routes, so one fault during an ordinary handoff wedged image generation until restart. Release it in a finally. The image and native backends each cleared one shared cancel Event on a new load. unload() sets that event to cancel an in-flight multi-GB download and drops _loading in the same breath, so a replacement load is admitted while the cancelled worker is still inside the fetch, and its clear() re-enabled the very object that worker was watching: the cancelled download resumed and ran alongside the replacement. Take a fresh Event per load and thread it to the worker, as the video backend already does, and set it under the lock since begin_load now rebinds the attribute. * Name utf-8 on the diffusion text I/O and the sd.cpp subprocess pipes tests/test_text_io_encoding.py failed on five files this branch adds. Text I/O without an explicit encoding falls back to the Windows ANSI codepage, so a non-ASCII path or manifest value round-trips corrupted, and the three sd.cpp pipes decode the child's UTF-8 output as ANSI on Windows despite already passing errors = 'replace'. Eleven read_text() / write_text() sites across diffusion_compile_cache, diffusion_ideogram4 and diffusion_krea2, plus text = True on the sd-cli version probe, the sd-cli run and the sd-server pipe. * Record the load-time build on a gallery image's recipe A gallery record documents itself as the image's full generation recipe and is embedded in the PNG, but the only load-related field it carried was the repo id. A GGUF repo holds many quants, so that does not say which one made the pixels, and it says nothing about an adapter baked in at load time. The fallback meant to cover the baked case could never fire: with no loras on the request _adjust_baked_loras zeroes every baked adapter and _active_lora_pairs drops zero-weight entries, so active_loras was always empty. A baked-and-disabled build is not the same pipeline as a never-baked one, so the recipe could not reconstruct the image once the model was rebuilt. Persist model_kind, gguf_filename, transformer_quant and the baked adapter names, read off the load state rather than the request, and show them in the recipe popover. The new fields are optional with defaults, which matters because list_gallery_images drops any record that fails validation, so a required field would have emptied every existing gallery; a regression test pins that. * Drop eleven duplicated comment tails, restore the mxfp8 denial note The comment passes collapsed several wrapped blocks onto one line without deleting the last physical line of the original wrap, leaving the tail of each sentence repeated as its own comment underneath. Two of the eleven were re-worded rather than byte-identical, so a strict suffix match missed them. 6e16ad16f also dropped the line justifying the qwen-image mxfp8 denial while that rule stayed live in _FAMILY_SCHEME_DENY, under a header that then documented only fp8 and nvfp4. Restored. Comments only; verified with the AST gate. * Only let the dense-quant fallback use shards the prefetch actually staged The prefetch skips the base repo's transformer/ shards whenever a prequant checkpoint is expected, since that checkpoint replaces them. But a prequant fetch can fail for reasons the planner cannot see: an unpublished, gated or renamed artifact, a hub 5xx, a proxy, a checkpoint the validator rejects. The loader then fell through to from_pretrained(subfolder = 'transformer') and pulled those shards inside the load lock during 'finalizing', after the previous pipeline was already evicted, where the cancel event has no reach, load_progress has already reported bytes_downloaded == bytes_total, and the cache-disk gate had only reserved the small prequant checkpoint. That is verbatim the situation _dense_quant_prefetch_needed's own docstring exists to prevent. Gate the in-loader dense fallback on the shards being staged, read off the returned file list rather than the request so a failed size estimate closes it too, and let the GGUF build take over otherwise, which is what the prequant-sized replan already does one branch over. It is also the invariant the text-encoder path already enforces: only a component that really resolves may have its dense weights dropped from a plan or a prefetch. Adds the missing coverage for the gate's prequant arm, which the existing disk-gate tests never reached. * Refuse a training output_dir that resolves to the outputs root resolve_output_dir strips a leading 'outputs' and drops '' / '.' segments, so '.', './', './.', 'outputs', 'outputs/outputs' and ' . ' all clean away to nothing and land on the outputs root itself rather than a run directory under it. The DiT trainer then writes pytorch_lora_weights.safetensors flat into the root, where scan_trained_models and scan_checkpoints cannot see it (both filter is_dir()), and a second such run overwrites the first. The UI only checks the field is non-empty, so 'outputs' is one plausible run name away. Refuse it with a 400 that says what to do instead. cond_cache_dir collapses the same way but has an honest 'off' to fall back to, so a root-resolving value now means the in-memory cache, which is what the comment above it already promised: otherwise a run drops one flat safetensors per cached latent and caption into the directory trained models live in. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Repair three defects the main merge left behind Git merged all three files without a conflict, but the result was wrong in each case. The tests only started failing once the merged tree was run. routes/inference.py duplicated the GGUF load block: this branch had moved the gguf_load_in_flight marker and the _hub_download_blocks_gguf_load guard under "if config.is_gguf and config.gguf_hf_repo", and the conflict resolution re-added main's copy at the old position, so both ran. Dropped main's copy; the earlier placement is the deliberate one, so a 409 from the hub guard cannot tear down a resident Images or Video pipeline. test_gpu_selection.py still called _hf_offline_if_dns_dead, which main renamed to _hf_offline_if_unreachable_for (#7591). Disjoint edits, so no conflict, but four route-error tests referenced a function that no longer exists. test_gguf_load_cache_reuse.py anchored its ordering assertion with rindex over "if config.is_gguf:", taking the last one before the load marker. That only held while _resolve_inherited_extra_args sat above every such line; main has since hoisted it above the gpu_ids preflight, so the anchor landed between the call and the marker and the assertion compared against an unrelated later call site. The ordering it checks is a property of _load_model_impl as a whole, so it now anchors on the function. The ordering itself is intact: _resolve_inherited_extra_args, then the gguf_load_in_flight marker, then the hub guard, then the chat handoff, then unload_model. * Stop a real sd.cpp install from breaking its own discovery tests The five "nothing is installed" assertions in test_sd_cpp_engine.py cleared SD_CLI_PATH and UNSLOTH_SD_CPP_PATH and patched Path.home, which covers hops 1, 2 and the fallback half of hop 3. It leaves two hops live. Hop 3 goes through managed_install_root(), which honors UNSLOTH_STUDIO_HOME and STUDIO_HOME and resolves to the stable-diffusion.cpp directory beside the studio home. That is the documented way to run side-by-side Studios, so anyone who has one set gets a real sd-cli back from a finder the test expects to return None. Hop 4 is the in-tree developer build, which does the same for anyone who built sd.cpp inside the checkout. Isolated with an autouse fixture rather than another helper call, because reaching the failure needs no fixture: SdCppEngine(binary = None) runs the finder from its constructor, which is why test_generate_raises_when_binary_missing failed too. The fixture points the studio home and the in-tree root at an empty tmp tree, so every hop is answered by the test rather than by the box. The in-tree root moves into a named in_tree_install_root() so it can be pointed somewhere empty; behaviour is unchanged, including the OSError and IndexError guard on an unexpected layout. * Make the sd.cpp uninstall test actually extract the code it tests The two sed ranges anchored the production fragments at column 0, but both blocks sit inside the main removal function and are indented, so each range matched nothing and LOOP_FILE / DEFAULT_FILE came out empty. Sourcing an empty file is a no-op, so the suite reported 4 passed / 7 failed: the seven removal assertions failed because nothing ran, and the four that passed were 'kept' assertions that pass trivially when nothing runs. The shell job auto-discovers tests/sh/test_*.sh and runs each under set -e, and this file is not in its skip list, so it was a deterministic red. Anchor on optional leading whitespace and fail loudly on an empty or _remove_path-less fragment, so a future reshuffle of uninstall.sh cannot make the suite vacuous again. Now 11 passed, 0 failed, against the real removal loop and the real default-mode block. * Agree on what an engine can build, and on what the installer owns Two gates, each half-applied. The unbuildable-family gate had one caller, the image branch of the cached repo picker. The GGUF classifier, the local-model classifier and the video branch had none, so on a diffusers too old for a family the picker still offered its GGUF and the load then failed. Meanwhile the loader asserted the diffusers pipeline class before engine selection, so a GGUF this host routes to native sd.cpp, which instantiates no pipeline class at all, was refused with an upgrade instruction that could not help. Both are wrong one-sidedly: hide a family only when NEITHER engine can build it, and demand the diffusers class only when diffusers is what will load it. One predicate, family_buildable_here, now answers both, so the picker and the loader cannot disagree. The population is real on Python 3.9, whose diffusers ceiling is 0.36: Flux2KleinPipeline, Krea2Pipeline and LTX2Pipeline are all absent there, and FLUX.2-klein GGUF is a repo in this PR's title. That assertion also raised RuntimeError, which /images/load maps to 409, the status that otherwise means a load is already in progress, and which escaped /images/download-plan (it catches ValueError and FileNotFoundError) as a bare 500 with the message lost. It is an unloadable pick like every other, so it raises ValueError and both routes answer 400 with the text intact. The managed-binary repair used a path test for ownership while the installer uses a marker. On a managed root without the marker, which is any install predating it, the repair deleted sd-server and the reinstall was then refused, permanently, because the surviving sd-cli keeps the directory non-empty so the marker can never be claimed: the user went from an unrunnable binary to no binary and no way back. Require the marker before discarding, which is the same definition of ours that uninstall.sh already uses to keep a user's own stable-diffusion.cpp checkout. A genuinely interrupted extraction still self-heals, since install() writes the marker before it extracts. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add gguf and av to the studio extra so the wheel install matches studio.txt This branch added both to studio/backend/requirements/studio.txt, which is what install.sh uses, but never to the studio extra in pyproject.toml, which is what 'pip install unsloth[studio]' uses. Nothing else keeps the two in sync, so tests/studio/install/test_studio_extra_matches_requirements.py failed deterministically, and the CI job runs the whole tests/ tree. The drift is not only a red test: gguf backs diffusers' GGUFQuantizationConfig and av does the MP4 encode and audio mux, so a wheel install got a Studio that cannot read a GGUF or export a video, which is most of what this PR adds. * Fix two test-suite defects: the CPU-only patch gate and a popped module The patch backend reaches unsloth_zoo's helpers only through an ~940 MB 'import unsloth', which a CPU-only host cannot complete, so every patch install returned False and nine arch/eager/compile tests failed there -- exactly the runners the retry was narrowed to protect. unsloth_zoo only wants UNSLOTH_IS_PRESENT in the environment, which costs nothing and needs no accelerator, and the conftest already sets its sibling UNSLOTH_ALLOW_CPU. Set it there too, as run.py and main.py already do at module scope, so no real server ever took the expensive route. CPU-only goes from 9 failed to 24 passed, 1 skipped; the GPU run is unchanged at 29 passed. test_setup_cache_env_hf_home popped utils.hf_cache_settings to model a fresh process and never restored it, so a later import built a second module object and rebound it on the utils package. test_hf_cache_settings then wrote its setting into one object while core.inference.diffusion read the other, the same split-module failure the xet shim already had. Restore both bindings in teardown: 21 pass together, and each file still passes alone. * Restore the setup_fail assertions the main merge reverted A fourth instance of the class 9541cc535 fixed: the merge took main's studio/setup.sh, which #7644 changed to abort through the setup_fail helper so desktop mode still emits [TAURI:ERROR], but kept this branch's older copy of the test, which still asserted the literal 'exit 1'. setup.sh is byte-identical to main here and the only diff in this file was the reverted assertions, so take main's version. 60 passed. * Fence the video teardown, reject non-finite clipping knobs, stop a test installing sd.cpp The video backend never had the teardown fence the image backend documents. Its unload and its superseding-load path both signalled the active generation, then did 'with self._generate_lock: pass' and tore the state down with the lock free. A generation queued behind that barrier holds no cancel event yet, so the signal cannot reach it, and Python locks are not FIFO: it won the lock the instant the barrier released it, read a still-loaded state and denoised a whole clip against the pipeline being freed. Reproduced on both paths, where the queued generation returned a finished MP4. Mirror the image backend: count waiters, refuse a generation while one is pending, and tear down inside the barrier with the counter released in a finally. _teardown_state becomes _teardown_state_locked since the caller now holds both locks. max_grad_norm and snr_gamma were bounded on one side only, so 1e309 floated to inf and passed. clip_grad_norm_ then computes an infinite clip coefficient, clamps it to 1.0 and scales nothing, and min(snr, inf) / snr makes every min-SNR weight 1.0. The run starts, reports normal progress and trains with the requested knob silently disabled. Measured both. NaN already failed the bounds; allow_inf_nan makes that explicit. learning_rate and flow_shift already guard this exact vector. test_load_routes_to_sd_cpp_on_cpu stubbed ensure_sd_cpp_binary but not ensure_sd_server_binary, which select_and_activate_engine probes first with installs enabled, so the unit test downloaded and unpacked 108 MB into the developer's real ~/.unsloth when UNSLOTH_STUDIO_HOME was unset. Now 0 bytes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Key the video schedule reset on the resolved defaults, not the repo id A GGUF repo holds several variants, so another client swapping a distilled build for the base one from the same repo changes the steps and guidance the backend reports while repo_id stays put. modelChanged stayed false, the page kept the previous schedule, and later generations ran an 8-step distilled setting on a model that expects roughly 40 steps and CFG 4. Include the reported defaults in the key. They only move when the resident artifact does, so a manual steps change still survives a status poll. * Retire the release-lag pin now that 2026.7.6 carries the relaxed gates The virgin Windows container row that installs from PyPI on purpose was tolerated with continue-on-error, because a Server Core container has no Microsoft Store and so no winget, and the released studio/setup.ps1 hard-stopped on a winget-only git gate and reached for winget again for the VC++ runtime. #7549 relaxed both, but it landed on 2026-07-28 and the newest wheel was 2026.7.5 from the 23rd, so the row could not get past it. The pin carried its own tripwire for exactly this moment, and it has fired: unsloth 2026.7.6 shipped 2026-07-29 and the released wheel now installs end to end, so the step reported success and the assertion errored with "delete this pin and drop continue-on-error from the Install step so this row gates". Doing that. Both rows now gate unconditionally, and the assertion that enumerated the accepted failure signatures goes with the pin it protected, since there is no longer an accepted failure for it to describe. The other three release-lag mentions in this file are descriptive rather than tolerated-failure pins: they explain why the fedora and ubuntu-nonroot legs run with overlay: true, and both already refuse a triton failure as a regression rather than accepting it as lag. Left alone. * Pin the hosted pre-quant onto the plain-torch fp8 kernel A Z-Image GGUF pick at the DEFAULT speed mode, once the hosted fp8 pre-quant is actually reachable, dies at generate with an HTTP 500: torch._dynamo.exc.Unsupported: Operator does not support running with fake tensors. Developer debug context: unsupported operator: mslk.f8f8bf16_rowwise.default _fp8_config already pins KernelPreference.TORCH when it BUILDS a config, precisely because the default AUTO switches to the MSLK kernel wherever an mslk package is importable (sm90+). A hosted checkpoint escapes that pin completely: the preference is serialized per Float8Tensor, and all 239 weights in the published Z-Image-Turbo-FP8 checkpoint carry AUTO. Loading it re-arms the very kernel the pin exists to avoid, mslk.f8f8bf16_rowwise has no fake impl, and the first compiled generate therefore cannot be traced. Isolated away from the product to be sure of the mechanism: quantise one Linear three ways on this box and compile each. KernelPreference.TORCH eager ok, compiled ok KernelPreference.AUTO eager ok, compiled FAILS on mslk.f8f8bf16_rowwise library default eager ok, compiled FAILS the same way So the pin is correct and necessary, and the only gap is that the hosted path never got it. _validate_checkpoint checks scheme, granularity, base model, min_features, exclude tokens and fast_accum, but not this. Rewriting the preference on load is safe: it selects a matmul kernel, it is not weight data, so the tensors stay bit-identical and the checkpoint's own state_dict_sha256 still describes them. It is also the faster path compiled, since the opaque extern call blocks inductor quantize fusion. Why this went unnoticed: the pre-quant repos are private, so nothing that could not read them ever took this path. It becomes the default the moment they are readable. Verified end to end at 1024x1024, 8 steps, speed_mode default, on the published checkpoint: before HTTP 500 at generate, diffusion.generate_failed after pinned 239 weights to the plain-torch fp8 kernel, load 10.0 s, transformer_quant fp8 with compiled engaged, cold 7.9 s, warm 0.87 0.87 0.87 0.86 0.87 s That warm number also beats the 1.4 s recorded for this shape on 2026-07-26. 159 prequant and transformer-quant tests pass. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cover the fp8 kernel pin so the 500 cannot come back quietly The hosted pre-quant re-arming MSLK was only visible as an HTTP 500 on a compiled generate, which needs a GPU, a readable private repo and an importable mslk to reproduce. None of that is available in CI, so the pin would rot unnoticed. Three hermetic cases on _pin_kernel_preference instead: AUTO weights are rewritten and already-TORCH ones are left alone (counting only what changed), a weight that refuses the assignment does not sink the whole load, and with no torchao enum available the checkpoint is left exactly as saved. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: tighten comments across the image generation, video and training code Rewrite the comments added by this branch to be shorter and clearer: collapse multi-line explanations into one or two lines, drop comments that only restate the code, and keep the rationale that explains why a choice was made. No code changes. * Studio: let the pipeline-class guard survive a host with no diffusers Backend CI installs the CPU-only dependency set, which has no diffusers, and runs without a GPU. 13 tests failed there for three separate reasons, one of which is a real product bug rather than a test-environment gap. assert_pipeline_class_available did a bare "import diffusers", so on any host without it the call raised ModuleNotFoundError instead of the ValueError its own contract promises. /images/download-plan catches only ValueError and FileNotFoundError, so that escaped as a bare 500 with the message lost, which is the exact failure the guard exists to prevent. It is reachable in production: the native sd.cpp engine serves GGUF picks on a CPU or Apple host where diffusers is never installed. Absent diffusers now returns, since the check answers "is the installed diffusers new enough for this family" and there is no version to judge; a pick that genuinely needs diffusers still fails in the loader with its own message. The rest were tests asserting through gates unrelated to what they cover: - Three cond_cache_dir route tests drive a DiT family, which the start route refuses without an accelerator. They now take the existing dit_train_host fixture, written for this case, so they keep testing the schema on every host. - test_download_plan_forwards_the_load_time_controls stubs the diffusers planner but sends a GGUF pick, which routes to the native planner on a GPU-less host. It now pins the engine, so it tests kwarg forwarding rather than the hardware. Engine selection keeps its own tests. - The pipeline-class guard test needed a real diffusers only for its sweep over every shipped family. That half is split out and skips on its own; the stub-driven refusal needs no diffusers and still runs. Added a test for the absent-diffusers contract above. 409 pass on a GPU host with diffusers; 301 pass and 1 skips with diffusers blocked and no GPU. * Studio: cut the image generation comments down further Second pass over the comments this branch adds. Collapse the multi-line blocks that still read as paragraphs, rewrite the longest one-liners so they say the same thing in fewer words, and drop a stale comment that had drifted away from the constant it described. No code changes: only comments and whitespace. * Studio: workflow rail, sidebar Video row, and Train settings polish Sidebar - Pin Video under Images by default; persist store bumped to v3 so an untouched install adopts it while a custom arrangement is left alone. - Add "Customize sidebar" to the end of the More flyout, opening Settings > Appearance scrolled to the sidebar nav section. Images - Replace the workflow dropdown with an icon rail down the left edge: all seven workflows visible, keyboard nav, tooltips carry the labels. - Workflows stay selectable with no model loaded, so one can be set up before picking a model. Generate is already gated on a loaded model. - Video moves out of the Create/Train strip into a link at the far right, with a matching Images link on the Video page. - Wider gutters around the settings column, headings matched to Train. Train - Field guidance moves into "i" tooltips; only state that limits a control stays on the page. Steps, LoRA rank and the rest gain hints. - /info reports params, qlora_vram_gb, gated and note as fields, so the family note renders as chips. vram_note is rebuilt from them unchanged for older clients. Shared - Number steppers appear on hover or focus. - Lighter control border (#e9e9e9) on Images and Video, light mode only. * Studio: move Advanced inline on Images and Video - Replace the top-bar toggle and right-docked Advanced panel with a disclosure under Seed, so load-time tuning sits with the settings it affects and opening it no longer shrinks the preview canvas. - Video uses the bordered variant, since offload and memory decide whether a model fits at all. - Open state persists per page in localStorage. - Tighten the Steps unit trigger: 14px chevron, less right padding. * Studio: full width media pages and layout polish - Drop the 1100px cap on Images Create, Images Train and Video, so the preview canvas grows with the window instead of sitting in a band. - Align the workflow rail to the model selector label, widen the gap before its divider, and restore the divider itself. - Advanced: more room above and below, icon and label sized to the slider rows, and the same quiet row on Video as on Images. - Give Video a pane heading and description, matching Images. - More space above Seed on both pages. - More no longer takes the active style when the current page is one of its own rows. * Studio: fix the Extend side toggles The resting outline used a ring, and index.css blanks the ring on the button holding mouse focus, so the side you just clicked lost its border until focus moved. Use a border for the outline and leave the ring to focus-visible. Also separates the two states properly (the off state had no surface of its own), drops the border in dark mode as the inputs do, and adds the aria-pressed and focus-visible styling the buttons were missing. * Studio: drop the border on the Extend side toggles Light mode should carry no border either, so the fill alone marks the state: a muted surface when off, a primary tint when on. Matches the borderless treatment dark mode already had. * Studio: move the Images workflows into the sidebar The workflow switcher was a vertical icon rail on the page. It now lives in the sidebar under Images, so the page keeps that width for the canvas. - Workflows list under the Images row, with a chevron to fold them away. Hovering the row peeks them in a flyout on the standard menu surface. - Clicking Images while already there toggles the list instead of navigating. - The listed workflow carries the highlight, so the Images row drops it. - Create takes its own icon: it was sharing the New chat pencil. - Model hub moves above Projects. Persisted layouts bump to v4 and only adopt the new order where the stored one is still a shipped default. - Images and Video content both start at 32px, clear of the sidebar. * Studio: simplify the Images workflow submenu Dropping the hover flyout: two ways to reach the same seven workflows, one of them overlapping the list right below it, read as clutter. - The flyout is gone. The workflows are rows under Images and nothing else. - On the Images page they are always open, since they are that page's switcher. Elsewhere they stay folded, and the row's chevron opens them. - Create takes the sparkles icon. - Create and Train both pad their settings column to 40px a side, so the two tabs line up with each other and with the model selector. * Studio: dock the generate action and tidy the Create controls The primary action sat at the foot of a long scroll, so it was off screen until you scrolled for it. - Generate, Video's Generate and Train's Start training float at the bottom of their settings column. No bar behind them: hover lightens the fill rather than thinning it, and the disabled state is opaque, so the controls underneath never show through. - Aspect ratios read as names: Square (1:1), Widescreen (16:9) and so on. - Width and height replace their sliders with two compact boxes. Type a size or pick one from the menu; the value still snaps into range and still drives the locked ratio. - Negative prompt gets a quiet disclosure under the prompt. It was there before but only above guidance 0, which is not the default, so it never showed. - The info "i" is smaller across the settings UI. * Studio: reveal Images and Video field hints on hover The "i" next to every field label sat there permanently, which made a column of settings read as busier than it is. Scoped to the diffusion pages by CSS rather than per component: the hints come from the page's own Field and SelectRow, the train panel's FieldLabel, and the chat ParamSlider, which is shared and should not change for chat. Reveals when the pointer is inside the field, on focus-within for keyboard, and stays inert while hidden. InfoHint grows a data-slot so the rule has a stable hook, rather than keying off its aria-label. * Studio: one-line sliders, edge fades, shared negative prompt - Steps, Guidance and the rest put label, track and value on one row. ParamSlider is shared with chat, so this is an opt-in prop and chat keeps the stacked layout. - The settings columns fade at whichever edge they run past, as the sidebar and model picker do, instead of cutting off. New useScrollFades hook drives all three. - Negative prompt moves to a shared component and Video picks it up, so both pages collapse it the same way. - Right gutter sits closer to the rule, Steps takes a bigger break above it, and the Advanced rule sits between its neighbours rather than up against the field above. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: add the API monitor to the sidebar nav Sits under More, below Export. Points at the monitor page, not the API keys dialog the profile menu opens. No store version bump: an id missing from a stored layout is appended with its default pinned state, so existing sidebars gain the row at the end and keep their own order. * Studio: take main's copy of the sidebar spinner test The merge kept this branch's older version, which pins the Recents row to h-[33px]. Main already relaxed that to any height, since row density moves independently of the trailing column the test is about. * Studio: fix FLUX.2 klein size resolution, HunyuanVideo tier routing, and the personalization save Five fixes found while auditing the model registries against the live Hub. FLUX.2 klein 9B loaded against the 4B config. One family covers both klein sizes and defaults to 4B, relying on the base_model card tag for the real base, but the trust allowlist only had klein-4B, so the correct tag was discarded and a 9B GGUF (inner_dim 4096) was loaded against a 4B config (3072). That surfaced as a bare shape mismatch from inside the GGUF quantizer naming neither the file nor the repo. Adds the missing allowlist entries and a header-level size check that fails early with a legible message; the check is fail-open, so an unreadable file, a non-FLUX.2 family, or an unmapped base leaves the load exactly as it was. The sd.cpp text encoder rule matched the literal "klein-9b", so klein-base-9B was handed the 4B encoder. HunyuanVideo-1.5 720p checkpoints routed to the 480p family. Only the literal 720p_t2v path was aliased, so 720p_i2v and every GGUF repack fell through to the generic token and inherited the 480p base repo, which also supplies the VAE and text encoder. The tier is baked into the weights (transformer target_size 640 vs 960, scheduler shift 5.0 vs 9.0) and the two bucket lists are disjoint, so this ran the whole pipeline off-tier. Resolution presets that the checkpoints were never trained for. Wan2.2 TI2V-5B is 720P-only upstream (SUPPORTED_SIZES is exactly 704x1280 and 1280x704, asserted in generate.py), and the HunyuanVideo 480p square preset 624x624 is not a bucket of its tier; 640x640 is. Personalization could never be saved. The frontend ships eight sidebar nav ids and refills every missing one on each save, but the backend Literal had seven and no "api", so every PUT to /api/settings/personalization returned 422. Aligns the server defaults and the Literal with the shipped layout. A catalog row pointing at a file that does not exist. unsloth/Qwen-Image-2512-FP8 holds torch prequant .pt checkpoints, not qwen-image-2512-fp8.safetensors, and fp8 is denied for this family anyway because it renders black, yet the row was the auto-route target on any GPU above 40 GB. Also marks FLUX.1-schnell, Krea-2-Turbo and the two Ideogram repos gated: all four are gated on the Hub today, and schnell being Apache-2.0 does not make it ungated. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: stop a case-only duplicate upload destroying an image, and size the load against the right GPU Two fixes from the review round. A dataset upload containing two names that differ only by case silently loses one on Windows and macOS. The in-batch duplicate check is an exact string compare, so "Cat.png" and "cat.png" both pass, but on a case-folding filesystem they are one destination: the commit step moves the first staged part aside, writes the second over it, then deletes the backup, while the response still reports both as uploaded. Captions collapse the same way and never even reach the image branch. No single folder can hold such a pair, but the Windows and macOS open dialogs flatten search and Recents results across folders into one multi-selectable list, which is a normal way a LoRA dataset gets assembled. Rejecting the pair everywhere would regress Linux, where the two really are different files with their own sidecars, so the check probes the filesystem once per process instead of keying off sys.platform: macOS also ships case-sensitive APFS volumes, and a Linux host can keep its Studio home on an exFAT or NTFS mount. A failed probe answers "case-sensitive", which leaves today's behaviour untouched. The quantization fit budget was sized against the wrong card under a reordering CUDA_VISIBLE_DEVICES. A bare "cuda" load lands on visible ordinal 0, but the reducer took the lowest PHYSICAL index, which the nvidia-smi path reports as index_kind "physical". Under "3,1" that sizes against GPU 1 while the pipeline loads onto GPU 3. The hook's own interface doc already promised the lowest visible ordinal. Ranks by visible_ordinal, falling back to index only for an older backend that omits it. * Studio: rework the diffusion dataset panel and align the pane gutters Upload: - one Upload button beside the dataset name, with an upload icon - picking files uploads them, no second confirm click - Add button beside the Training images dropdown, so images can go into a set that already exists - fall back to the upload form when a selected dataset no longer resolves Labeling grid: - two columns at any width; sm:grid-cols-3 crushed the tiles in a fixed width column - pin the caption size so the Textarea's md:text-sm does not outsize a tile - name the tile hover group; a bare one revealed every tile's Remove at once - Remove is an icon button on the image, not a word over the artwork - drop the amber tile fill, keep the No caption label - match the header, toggle and status text to the section's type scale Gutters: - the run area and the preview canvas now sit 40px off the rule, the gutter the settings column has off the page edge * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: give the dataset preview per-image hover and remove, and wrap it onto rows - each thumbnail is its own button, so it can carry an X that removes just that image; the strip was one button and could not nest a second control - hover lightens the image instead of drawing a border - wraps onto more rows rather than scrolling sideways, which also keeps the bottom corners rounded - samples 12 thumbnails, up from 8, now that a second row is available - a delete refreshes the panel's dataset counts * Studio: show the workflow icon beside the Images and Video headings Images reads it off WORKFLOW_TABS, so it is the same icon the sidebar submenu shows and it follows the active workflow. Video's heading is also Create, so it takes the Create icon rather than the nav row's film slate. * Studio: reject a duplicate LoRA id on the diffusion load path too DiffusionGenerateRequest already refuses a repeated adapter id, with a comment naming the hazard: _resolve_lora_set suffixes colliding adapter names, so the same id resolves the SAME adapter twice and set_adapters stacks both copies past the per-adapter weight bound. DiffusionLoadRequest bounds only the list length, and it matters more there. Generation-time stacking spoils one image; on the load path the adapters are baked into the quantized build before compilation, so the unintended combination rides every image until a reload. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: teach the GGUF reuse test double about holds_no_vram The resident-GGUF fast path consults llama_backend.holds_no_vram before asserting CHAT ownership, but two tests hand it a SimpleNamespace built before that attribute existed, so they raised AttributeError and returned a 500. A real LlamaCppBackend exposes it as a property; the doubles now carry it too. Product code is unchanged: these pass on main and were the branch breaking its own test, not a defect in the guard. --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: michaelhan <michaelhan2050@gmail.com> Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local> Co-authored-by: Michael Han <107991372+shimmyshimmer@users.noreply.github.com> | 1 个月前 | |
Use Unsloth for bare Studio product references (#9732) * Studio: use Unsloth for product shorthand * Tests: restore case-folding fixture path * Complete Unsloth product reference rename * Tests: restore internal Studio references | 9 天前 | |
studio: add higgs, moss, and minimax audio models * Studio: add Higgs, MOSS, and MiniMax audio models * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten comments and source headers * Restore audio PR scope * Fix native audio review findings * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix native audio platform edge cases * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix native audio review follow-ups * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Complete native audio cancellation guards * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix native audio inventory and cancellation follow-ups * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Secure MOSS codec selection and instructions * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Reduce verbose comments * Fix native audio review findings * Fix Audio model loading and managed downloads * Keep cached Audio loads available offline * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix native audio placement and inventory * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix MOSS generation and audio history * Forward MOSS language through audio worker * Fix cached native audio handoff * Use MOSS model context for audio duration * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix cached native audio handoffs * Validate native audio prompt budgets * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix native audio placement and routing * Handle resident VRAM during audio model handoff * Apply repository Python formatting * Trim native audio test coverage * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix backend CI after main merge * Fix teardown padding CI stub * Fix native audio review findings * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fail closed on unknown llama gpu order * stage the higgs tts2 audio tokenizer * move the higgs tts2 codec to the model device * require remote code consent before audio staging * hide unsupported native audio adapters * load minimax components from the selected checkpoint * use safe attention backends for moss audio * fail closed on unavailable audio tokenizers * serialize audio generation and model swaps * use the released diffusers minimax support * neutralize moss tts prompt delimiters * honor speech request contracts * allow the full minimax music frame budget * expose native audio prompt controls * stabilize native audio generation * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * preserve frame-aligned minimax durations * resolve higgs codecs from checkpoint metadata * neutralize minimax music delimiters * load approved higgs3 codecs with auth * fail closed on oversized audio metadata * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * align native audio security preflight * invalidate stale audio mode work * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * reject unsupported audio plans before download --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: danielhanchen <danielhanchen@users.noreply.github.com> Co-authored-by: mahiatlinux <110882203+mahiatlinux@users.noreply.github.com> | 6 天前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 3 天前 | ||
| 9 天前 | ||
| 9 天前 | ||
| 6 天前 | ||
| 2 个月前 | ||
| 2 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 9 天前 | ||
| 9 天前 | ||
| 2 个月前 | ||
| 5 天前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 26 天前 | ||
| 3 个月前 | ||
| 28 天前 | ||
| 23 天前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 17 天前 | ||
| 10 天前 | ||
| 5 天前 | ||
| 26 天前 | ||
| 6 天前 | ||
| 10 天前 | ||
| 25 天前 | ||
| 3 天前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 9 天前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 9 天前 | ||
| 1 个月前 | ||
| 2 个月前 | ||
| 1 个月前 | ||
| 2 个月前 | ||
| 5 天前 | ||
| 4 天前 | ||
| 1 个月前 | ||
| 29 天前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 2 个月前 | ||
| 9 天前 | ||
| 9 天前 | ||
| 2 个月前 | ||
| 11 天前 | ||
| 1 个月前 | ||
| 9 天前 | ||
| 6 天前 |