| feat: ernie image/turbo (#9115) * Update to Transformers 5.1.0 * remove extra stuff * feat: ERNIE-Image integration (incl. diffusers 0.38 + transformers 5) Adds Baidu ERNIE-Image and ERNIE-Image-Turbo as a new BaseModelType, mirroring the FLUX.2 / Z-Image integration pattern. Both models share the ErnieImageTransformer2DModel architecture (3072 hidden, 24 layers, 24 heads) and an AutoencoderKLFlux2 VAE; they differ only in default inference settings (50 steps + CFG 4.0 vs 8 steps + CFG 1.0 for Turbo). Built on top of PR #8859 (transformers 5.1+) and additionally bumps diffusers 0.36.0 -> 0.38.0, which is the first release containing the ErnieImagePipeline and ErnieImageTransformer2DModel. Backend - BaseModelType.ErnieImage, ModelType.PromptEnhancer, two new SubModelTypes (pe, pe_tokenizer) for the bundled prompt enhancer - Main_Diffusers_ErnieImage_Config + Main_Checkpoint_ErnieImage_Config with state-dict-based detection (x_embedder + text_proj + adaLN_modulation) - Diffusers loader registered for ERNIE-Image; uses upstream subdir conventions, loads transformer / vae / text_encoder / tokenizer plus optional pe / pe_tokenizer - New invokeai/backend/ernie_image/ with sampling utilities (2x2 patchify, BN normalize/denormalize, sigma schedule, padded text packing) and a rectified-flow denoise loop supporting Euler/Heun/LCM - Five invocations: model_loader, text_encoder (with prompt-enhancer toggle), denoise, vae_encode, vae_decode - ErnieImageConditioningInfo + ConditioningField/Output + pickle allowlist - ERNIE_IMAGE_SCHEDULER_MAP reusing the FlowMatch* scheduler classes - New generation modes ernie_image_{txt2img,img2img,inpaint,outpaint} - Starter models for baidu/ERNIE-Image and baidu/ERNIE-Image-Turbo + STARTER_BUNDLES entry Frontend - Regenerated services/api/schema.ts to expose the new node types - Type unions extended (ImageOutput / LatentToImage / ImageToLatents / DenoiseLatents / MainModelLoaderNodes) - ParamsState gains ernieImageScheduler + ernieImageUsePromptEnhancer, with reducers, selectors, and selectIsErnieImage - buildErnieImageGraph (txt2img/img2img/inpaint/outpaint) wired into useEnqueueCanvas - ERNIE entries added to MODEL_BASE_TO_{COLOR,LONG_NAME,SHORT_NAME} and prompt_enhancer to MODEL_TYPE_TO_LONG_NAME - ParamErnieImageScheduler and ParamErnieImagePromptEnhancer rendered conditionally in GenerationSettingsAccordion - All add{TextTo,ImageTo,Inpaint,Outpaint}Image type guards extended to accept ernie_image_denoise; isMainModelWithoutUnet ditto Diffusers 0.38 fix-out - hotfixes.py: import LoRACompatibleConv directly; the lazy-module __getattr__ no longer exposes diffusers.models.lora as an attribute Verification - pytest tests/ -m "not slow": 619 passed, 0 failed - pnpm lint:tsc / lint:eslint / lint:prettier: clean - pnpm test:no-watch: 563 passed, 0 failed - Manual smoketest pending: requires baidu/ERNIE-Image weights and a GPU (8B parameters; CPU not practical) Out of scope (follow-up phases) - ControlNet, IP-Adapter, LoRA support for ERNIE-Image - Single-file checkpoint loading (defensive scaffolding only) - Metadata recall handlers in the gallery side panel * Chore Ruff + Typegen * Chore ruff * fix(ernie-image): timestep scale, live preview, missing graph case, and UI cleanup - Pass timesteps in [0, num_train_timesteps] to the transformer instead of [0, 1]; the diffusers Timesteps embedding expects the unnormalised range, which produced mosaic-pattern garbage instead of an image. - Unpatchify predicted-x0 before the denoise step callback and route through sd_step_callback so the canvas shows a live preview during sampling (uses FLUX.2's RGB factors -- same AutoencoderKLFlux2 / 32 latent channels). - Add the ernie-image case to useEnqueueGenerate (Generate tab); was only wired up in useEnqueueCanvas, so plain text-to-image failed with "No graph builders for base ernie-image". - Move the Prompt Enhancer toggle from the Generation accordion into a dedicated ERNIE-Image block in the Advanced accordion; hide the rest of the SD-style advanced controls (CLIP skip, CFG rescale, seamless, color comp., separate VAE) since none apply to ERNIE-Image. - Detect ERNIE-Image-Turbo by name in MainModelDefaultSettings.from_base so installs (starter or manual) get steps=8, cfg_scale=1.0 instead of the standard 50/4.0. - Pin compel to Cstannahill/compel5@chore/transformers5-diffusers-smoke for transformers>=5 compatibility (PR damian0815/compel#129). * Merged missed * Chore Ruff * Chore Fix UV lock * fix(ernie): restrict ERNIE-Image to text-to-image only ERNIE's denoise node has no denoise_mask input, so masked modes are unsupported. Drop img2img/inpaint/outpaint from the ERNIE graph builder, exclude ernie_image_denoise from a new MaskableDenoiseNodes type used by addInpaint/addOutpaint, and align addImageToImage unions with the node type aliases. Fixes tsc failures on the ernie-image branch. * fix(ernie): restrict ERNIE-Image to text-to-image only ERNIE-Image's denoise node has no denoise_mask input and no mask logic, so masked modes (inpaint/outpaint) are impossible and image-to-image is dropped as well. Frontend: - buildErnieImageGraph now builds txt2img only; asserts on other modes - add MaskableDenoiseNodes (DenoiseLatentsNodes minus ernie_image_denoise) and use it in addInpaint/addOutpaint - drop ernie_image_vae_encode from ImageToLatentsNodes; align addImageToImage unions with the LatentToImage/ImageToLatents aliases - regenerate schema.ts Backend: - delete the ernie_image_vae_encode invocation (i2l, only used by the removed image-input modes) - drop ernie_image_{img2img,inpaint,outpaint} from GENERATION_MODES Fixes the tsc failures on the ernie-image branch. * Chore OpenApi * fix(model_manager): remove duplicate _has_anima_keys shadowing ComfyUI-bundled prefix support A duplicate _has_anima_keys definition (older, net.-only) shadowed the complete version that also recognizes the `model.diffusion_model.` ComfyUI-bundled prefix, causing Anima identification to reject bundled checkpoints. * Chore Openapi * fix(ernie-image): exclude prompt enhancer from fp8, tighten turbo detection Address review findings on #9115: - Exclude SubModelType.PromptEnhancer and PromptEnhancerTokenizer from fp8 layerwise casting. The prompt enhancer is a causal LM driven by generate() — one full forward per generated token — so casting made the whole LM round-trip bf16<->fp8 per token, on top of fp8 rounding a model whose entire job is text quality. - Match turbo detection on the install directory's leaf name instead of the whole path string. An in-place install records an absolute path, so an ancestor directory like /mnt/turbo-nvme/ silently gave the base model Turbo's 8 steps and CFG 1.0. - Pass local_files_only=True on all ERNIE-Image from_pretrained calls and load tokenizers bare, matching the sibling loaders. - Cap the prompt enhancer's max_new_tokens at 1024. Driving it off model_max_length hangs the graph if the tokenizer config omits it and transformers substitutes its int(1e30) sentinel. Adds regression tests for the fp8 exclusion and the turbo path matching. * fix(ernie-image): honor denoising window, noise init latents, harden PE gate Self-review follow-ups on #9115: - Honor denoising_end. Every FlowMatch scheduler appends its own terminal 0 sigma, and passing the window minus its last entry let that zero stand in for the requested end sigma - so denoising_end < 1.0 ran a full denoise in fewer, coarser steps. Hand the scheduler the whole window and truncate its appended zero instead, which also keeps the scheduler's own `shift` applied to the terminal sigma. - Reject a denoising window that rounds down to a single sigma. It yielded zero steps, so the loop returned its input untouched and the graph decoded raw noise with no error at all. - Blend image-to-image init latents with noise at the first sigma, and reject denoising_start > 0 when no latents are provided. Both cases previously lied to the model about where the sample sits on the rectified-flow path. The shape check now covers batch and spatial dims, and its message no longer points at a VAE-encode node that does not exist. - Require both `pe` and `pe_tokenizer` to be present AND declared in model_index.json before offering the prompt enhancer. get_hf_load_class resolves submodels from that file, so a directory-only check let a partial install pass the gate and then hard-fail the generation - by default, since the toggle is on. - Pass the real generation dimensions to the prompt enhancer instead of leaving it on its 1024x1024 defaults. Adds regression tests for each, including a graph-builder test suite for ERNIE (which had none). Added ERNIE-Image + Krea 2 Raw into README.md under Supported Model. * Readme.me update with ERNIE-Image-Turbo * fix(ernie-image): seed the stochastic scheduler, blend at the shifted sigma Round-4 review follow-ups on #9115: - Pass a seeded generator into scheduler.step. FlowMatchLCMScheduler is stochastic - it re-noises the sample every step - so with generator=None it drew from the global RNG. The seed field only controlled the initial latent, so the same seed produced a different image on every run and seed recall from gallery metadata could not reproduce an LCM generation. Seeded from `seed ^ 0xFFFFFFFF` (as denoise_latents.py does) so the step noise stays decorrelated from the initial noise, and from a CPU generator so it is device-independent like the initial noise. - Blend image-to-image init latents at the scheduler's post-shift first sigma instead of the raw schedule value. get_schedule emits raw linspace values and the scheduler applies `shift` in set_timesteps, so the blend built the sample at one sigma and then told the first model call it was at another. The blend moves into denoise(), which is the only layer that knows the shifted sigma. - Add an `add_noise` field mirroring z_image_denoise. The only node that can currently feed `latents` is another ernie_image_denoise, whose output already sits at the handoff sigma - re-noising it broke the very multi-stage handoff the denoising_end fix enables. - Give the prompt enhancer the original size rather than the intermediate scaled render size. Regenerates openapi.json / schema.ts for the new field. Adds regression tests for the seed contract (same seed identical, different seeds different, euler unaffected) and for the post-shift blend. --------- Co-authored-by: Your Name <you@example.com> Co-authored-by: 4pointoh <97913726+4pointoh@users.noreply.github.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> | 2 个月前 |