| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
Feat(model support): ideogram4 support (#9303) * Update to Transformers 5.1.0 * remove extra stuff * chore(deps): compel fork + transformers>=5.9.0 + remove override Switches compel from PyPI 2.1.1 to invoke-ai/compel@main fork which supports transformers 5.x. Bumps transformers floor to 5.9.0. Removes the transformers>=5.1.0 uv override that was only needed to bypass compel 2.1.1's <5.0 constraint. NOTE: compel fork pulls notebook dep (full Jupyter stack); flag to maintainer for cleanup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(z_image): resolve rope_theta from rope_parameters for transformers 5.x transformers 5.x no longer exposes rope_theta as a top-level attribute on Qwen3Config; the value is stored in the rope_parameters (and rope_scaling) dict instead. Read it from there with a getattr fallback so the inv_freq buffer is computed from the configured base (1e6 / 256) instead of raising AttributeError. Applies to both the safetensors and GGUF Qwen3 encoder paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(model_manager): replace removed hf_hub get_token_permission with whoami huggingface_hub 1.x removed get_token_permission(). HFTokenHelper.get_status() now validates the token via whoami(), which returns user info for a valid token and raises HfHubHTTPError for an invalid one. Preserves the original three-way status: VALID on success, INVALID on HfHubHTTPError (e.g. 401), UNKNOWN on any other error (e.g. network failure). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): regenerate uv.lock after upstream merge Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sd3): resolve merge conflict marker, drop T5TokenizerFast The upstream merge left an unresolved conflict marker in _t5_encode and reintroduced T5TokenizerFast. Keep our v5 assertion (T5Tokenizer only) plus upstream's new t5_device logic, and drop the now-dead T5TokenizerFast monkeypatch in the test (the name no longer exists in the module). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: ruff fixes on merge-resolved files - flux_text_encoder.py: drop unused typing.Union (F401) left by v5 import merge - huggingface.py: ruff format (wrap append(SimpleNamespace(...))) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): pin transformers <5.6 (diffusers single-file CLIP incompat) transformers 5.6 flattened CLIPTextModel (removed the self.text_model wrapper, hoisted embeddings/encoder/final_layer_norm to the top level). diffusers' single-file checkpoint loader (create_diffusers_clip_model_from_ldm) still assumes the nested layout, so loading SD1.5 .safetensors checkpoints fails on 5.6+ with 'CLIPTextModel object has no attribute text_model' and, once that read is shimmed, 'Cannot copy out of meta tensor' (weights never populate the flattened model). Pin to >=5.5,<5.6 (last pre-flattening release) which keeps both the single-file and from_pretrained paths working. The invoke-ai/compel fork accepts any 5.x. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * @ chore(deps): replace compel fork with official compel 2.4.0 compel 2.4.0 (released 2026-05-30) merges the transformers-5 support that the invoke-ai fork carried (both descend from upstream PR #129), plus the maintainer-reviewed padding rework and added diffusers/T5 smoke coverage. Switch from the git fork to the PyPI release. - pyproject: compel git+main -> compel>=2.4.0,<3 - uv.lock: compel 2.3.1 (git 8f404b45) -> 2.4.0 (pypi) - transformers stays 5.5.4 (satisfies compel >=5,<6 and our <5.6 pin) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> @ * feat(ideogram4): backend + model-manager registration for Ideogram 4 Vendor the Apache-2.0 Ideogram 4 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler, nf4/fp8 quant loading) into invokeai/backend/ideogram4/, plus InvokeAI glue (Qwen3-VL text encoding, packed-input build, dual-branch Euler denoise loop). Register the model: BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config (detected via the Ideogram4Pipeline class name in model_index.json), and the Ideogram4DiffusersModel loader that loads both transformers as one Ideogram4TransformerPair submodel plus the Qwen3-VL encoder and VAE. Text-to-image only. * feat(ideogram4): Ideogram 4 backend — model manager, invocations, nf4 loading End-to-end text-to-image backend for Ideogram 4, validated through the real session runner. Vendors the Apache-2.0 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler) into invokeai/backend/ideogram4/ with InvokeAI glue. Registers BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config, and the Ideogram4DiffusersModel loader (two transformers as one Ideogram4TransformerPair; Qwen3-VL encoder + VAE). Both transformers and the encoder load via InvokeLinearNF4 so they work with the partial-load cache. Adds Ideogram4ConditioningInfo/Field/Output and the model_loader/text_encoder/denoise/l2i invocations. Text-to-image only. * feat(ideogram4): frontend — Regions→JSON prompt, graph builder, UI Wires Ideogram 4 into the canvas/generate UI. buildIdeogram4Prompt assembles the structured JSON caption from the global prompt + Canvas Regional Guidance layers (each region → an obj element with a 0–1000 bbox + desc), with raw-JSON passthrough and a plain-text fallback when there are no regions. Adds buildIdeogram4Graph (text-to-image only, no negative prompt) and the enqueue switch. Structured captions use a static string node + a decoy positive-prompt node so the linear batch can't clobber the assembled JSON; plain text uses the real node so dynamic prompts/batching still work. Registers the 'ideogram-4' base (enums, color, names, model picker, grid size 16), a sampler-preset param (V4_QUALITY_48/V4_DEFAULT_20/V4_TURBO_12) replacing the steps/CFG controls, ParamIdeogram4SamplerPreset, and metadata recall. Regenerates schema.ts. * feat(ideogram4): advanced sampler overrides + color palette Advanced accordion now shows only Ideogram 4-relevant controls. Adds optional overrides of the sampler preset — steps, guidance scale (overrides the main gw, preserves the preset's polish tail), and schedule shift (mu) — plus a color palette editor that injects style_description.color_palette into the auto-built JSON caption (uppercase #RRGGBB, max 16, ignored for raw-JSON prompts). All are nullable (null = use preset), recallable from metadata, and the irrelevant controls (VAE, CLIP skip, CFG rescale, seamless, color compensation) are hidden for Ideogram 4. Backend denoise gains steps/guidance_scale/mu fields; schema.ts regenerated. * Use existing keys + fix select size * Update Readme * feat(ideogram4): add Ideogram 4 to starter models with non-commercial license hint - Implement the weight-only fp8 text-encoder load path (was NotImplementedError); validated against the real fp8 build + add CPU unit tests for the fp8 mechanism - Show Ideogram 4 handlers in the Recall Parameters tab - Recall the assembled JSON caption back into the positive prompt - Translate the metadata "Auto" values - Document Ideogram 4 (install/license, regional-guidance JSON prompting, presets) - Add Ideogram 4 nf4 (CUDA) + fp8 (any device) starter models and bundle - Surface the FLUX-style Non-Commercial License popover for Ideogram 4 models - Note the gated HuggingFace license requirement in the model descriptions * Chore Ruff * Chore Ruff * Chore OpenApi * Chore Knit * fix(deps): regenerate uv.lock to remove duplicate packages from bad merge * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * Fix: address ideogram4 review (strict load, 1-step guidance, i18n, runtime caption) - Non-fp8 Ideogram 4 text-encoder load now validates the state dict: unexpected keys raise, missing keys warn (mirrors the fp8 helper) instead of silently accepting a partial load. - Guidance schedule: cap the polish tail at num_steps-1 so at least one main step always remains (the guidance_scale override was silently dropped at num_steps=1), and require steps >= 2 (backend field + frontend slider/marks). - Localize the Ideogram sampler-preset option labels via t() with the step count interpolated; add the three preset i18n keys. - Assemble the structured JSON caption at generation time in a new ideogram4_caption_builder node (Python port of buildIdeogram4Caption) instead of at graph-build time. The graph now wires the real prompt node -> caption builder -> text encoder and returns it as positivePrompt, so dynamic prompts / prompt batching vary the encoded caption (the decoy that dropped them is removed). The builder's output is wired to a new declared ideogram4_caption metadata field via an edge, so each batched image records its actual caption. Regenerates schema.ts for the new node + metadata field. Adds tests for caption assembly, the guidance schedule, and the graph wiring. * feat(ideogram4): step previews + document the model's built-in safety filter - Emit a low-res progress preview each denoise step so the forming image is visible during generation, like the other denoise nodes. Ideogram uses a FLUX.2-style 32-channel VAE, so the packed latent is unpatchified/denormalized (get_latent_norm) and run through the FLUX.2 latent->RGB factors — no full VAE decode per step. The denoise loop now hands the callback the packed grid latent. - Document Ideogram 4's built-in content safety filter in models.mdx: it lives in the model weights (not Invoke's NSFW checker, can't be disabled from Invoke) and false-positives on benign prompts; structured JSON prompts trip it less. * feat(ideogram4): avoid safety-filter false-positives + step previews + caption visibility The main fix: Ideogram 4's built-in safety filter (baked into the model weights) false-positives and returns an "Image blocked by safety filter" placeholder for "degenerate" captions — empirically, an empty `compositional_deconstruction.elements` list, or a single full-frame [0,0,1000,1000] element whose desc just repeats the high_level_description. Our assembly produced empty elements whenever the user drew no regions, so plain prompts were blocked. - Caption assembly (build_ideogram4_caption): - Always emit a structured JSON caption; never bare plain text (the filter false-positives far more on plain text). Raw-JSON pastes still pass through. - When there are no regions, synthesize one default element describing the whole scene from the prompt with a *partial* (non-full-frame) bbox [100,100,900,900]. This never yields an empty/degenerate elements list. Verified end-to-end against the model: the previously-blocked "golden retriever on a skateboard" now renders. - Metadata: always wire the caption builder's output to the ideogram4_caption metadata field, so the viewer's "Structured Caption" row shows the exact JSON that was encoded (not just the raw prompt) for every generation. - Denoise: emit a low-res progress preview each step (unpatchify + FLUX.2 latent->RGB factors, since Ideogram uses a FLUX.2-style 32-channel VAE) so the forming image is visible during generation, like the other denoise nodes. - Docs: document the model's built-in safety filter in models.mdx (it's not Invoke's NSFW checker, can't be disabled from Invoke, and false-positives). Updates the caption/graph tests accordingly (also fixes latent tsc errors in the graph-builder test's core_metadata / ideogram4_caption comparisons). * Chore typegen + openapi + Ruff * Fix Knit * fix(ideogram4): enforce steps>=2 client-side and validate region bbox Address review on the Ideogram 4 PR: - The backend denoise node requires steps >= 2, but the client still accepted ideogram4_steps = 1 in three places, letting a recalled or rehydrated value build a graph that violates the backend schema. Tighten the zod schema to min(2) with `.catch(null)` (a stale/out-of-range value normalizes to null = use the preset instead of failing the whole persisted slice), normalize dispatched values through the schema in setIdeogram4Steps, and refuse an out-of-range value in the ideogram4_steps metadata recall parser. The slider was already min=2. - Ideogram4Region.bbox was an unconstrained Optional[list[int]], so a workflow/API caller could pass a wrong-length or out-of-range box that the caption builder serialized verbatim into the structured prompt. Add a field validator requiring exactly four coordinates, each in 0..1000. Add tests for both: the region bbox contract (valid/None accepted; short, long, negative, and >1000 rejected) and the ideogram4Steps normalization (valid kept, null kept, stale 1 normalized to null on both dispatch and rehydrate). * fix(ideogram4): block unsupported canvas modes/bbox, warn dropped region inputs, constrain bbox Address the latest review on the Ideogram 4 PR: - Canvas readiness allowed unsupported generation modes: Ideogram 4 is txt2img-only (buildIdeogram4Graph asserts it), but a raster layer or inpaint mask makes the compositor pick img2img/outpaint/inpaint, failing only at graph build. Warn in getRasterLayerWarnings/getInpaintMaskWarnings for Ideogram 4 (these already flow into canvas readiness reasons), blocking enqueue up front. - Canvas readiness had no Ideogram 4 bbox check; the backend requires multiples of 16. Add the 16-grid check mirroring the other 16-grid models so an off-grid bbox (e.g. 1025x1024) is blocked instead of failing backend validation. - getRegionalGuidanceWarnings had no Ideogram 4 branch, so a region whose only input is a negative prompt, auto-negative, or reference image looked effective while the graph silently drops it. Warn those inputs are unsupported. - Ideogram4Region.bbox now also rejects inverted boxes (y_min <= y_max, x_min <= x_max). - The advanced-settings badge selector lumped Ideogram 4 into the generic branch, showing stale VAE/clip-skip/CFG-rescale/seamless badges for controls that are hidden for Ideogram. Exclude Ideogram 4 from that branch. - Model bbox as a constrained type (exactly 4 ints, each 0..1000) so the OpenAPI schema advertises minItems/maxItems and item minimum/maximum. Add readiness tests (bbox grid, raster/inpaint blocking, empty-layer allowance, regional-guidance negative/reference-image warnings) and bbox ordering-rejection tests. * fix(ideogram4): reject text encoders with weights left on the meta device _load_text_encoder() builds the encoder under accelerate.init_empty_weights() and previously downgraded missing keys to a warning (both the fp8 path via load_fp8_state_dict(strict=False) and the non-fp8 path). A missing non-tied weight therefore stayed on the meta device, so a bad or mismatched encoder appeared to load and only failed later during device movement or encoding. Add _verify_encoder_fully_materialized(): call tie_weights() to materialize tied weights from their source, then hard-fail if any parameter or buffer remains on the meta device. Wire it into both the fp8 and non-fp8 (incl. bnb-nf4) paths and drop the missing-key warning — genuinely missing non-tied weights are now caught as leftover meta tensors, while tied weights are tolerated. This is a state-based, path-agnostic check. Add tests: passes when fully materialized, raises on a leftover meta tensor from a missing non-tied weight, and tolerates a tied weight resolved by tie_weights(). * chore(ui): prettier formatting for AdvancedSettingsAccordion The Ideogram 4 badge-suppression branch left the wrapped block at its old indentation, failing `pnpm lint:prettier` in frontend-checks. Formatting only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Your Name <you@example.com> Co-authored-by: 4pointoh <97913726+4pointoh@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> | 1 个月前 | |
Feat(model support): ideogram4 support (#9303) * Update to Transformers 5.1.0 * remove extra stuff * chore(deps): compel fork + transformers>=5.9.0 + remove override Switches compel from PyPI 2.1.1 to invoke-ai/compel@main fork which supports transformers 5.x. Bumps transformers floor to 5.9.0. Removes the transformers>=5.1.0 uv override that was only needed to bypass compel 2.1.1's <5.0 constraint. NOTE: compel fork pulls notebook dep (full Jupyter stack); flag to maintainer for cleanup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(z_image): resolve rope_theta from rope_parameters for transformers 5.x transformers 5.x no longer exposes rope_theta as a top-level attribute on Qwen3Config; the value is stored in the rope_parameters (and rope_scaling) dict instead. Read it from there with a getattr fallback so the inv_freq buffer is computed from the configured base (1e6 / 256) instead of raising AttributeError. Applies to both the safetensors and GGUF Qwen3 encoder paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(model_manager): replace removed hf_hub get_token_permission with whoami huggingface_hub 1.x removed get_token_permission(). HFTokenHelper.get_status() now validates the token via whoami(), which returns user info for a valid token and raises HfHubHTTPError for an invalid one. Preserves the original three-way status: VALID on success, INVALID on HfHubHTTPError (e.g. 401), UNKNOWN on any other error (e.g. network failure). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): regenerate uv.lock after upstream merge Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sd3): resolve merge conflict marker, drop T5TokenizerFast The upstream merge left an unresolved conflict marker in _t5_encode and reintroduced T5TokenizerFast. Keep our v5 assertion (T5Tokenizer only) plus upstream's new t5_device logic, and drop the now-dead T5TokenizerFast monkeypatch in the test (the name no longer exists in the module). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: ruff fixes on merge-resolved files - flux_text_encoder.py: drop unused typing.Union (F401) left by v5 import merge - huggingface.py: ruff format (wrap append(SimpleNamespace(...))) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): pin transformers <5.6 (diffusers single-file CLIP incompat) transformers 5.6 flattened CLIPTextModel (removed the self.text_model wrapper, hoisted embeddings/encoder/final_layer_norm to the top level). diffusers' single-file checkpoint loader (create_diffusers_clip_model_from_ldm) still assumes the nested layout, so loading SD1.5 .safetensors checkpoints fails on 5.6+ with 'CLIPTextModel object has no attribute text_model' and, once that read is shimmed, 'Cannot copy out of meta tensor' (weights never populate the flattened model). Pin to >=5.5,<5.6 (last pre-flattening release) which keeps both the single-file and from_pretrained paths working. The invoke-ai/compel fork accepts any 5.x. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * @ chore(deps): replace compel fork with official compel 2.4.0 compel 2.4.0 (released 2026-05-30) merges the transformers-5 support that the invoke-ai fork carried (both descend from upstream PR #129), plus the maintainer-reviewed padding rework and added diffusers/T5 smoke coverage. Switch from the git fork to the PyPI release. - pyproject: compel git+main -> compel>=2.4.0,<3 - uv.lock: compel 2.3.1 (git 8f404b45) -> 2.4.0 (pypi) - transformers stays 5.5.4 (satisfies compel >=5,<6 and our <5.6 pin) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> @ * feat(ideogram4): backend + model-manager registration for Ideogram 4 Vendor the Apache-2.0 Ideogram 4 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler, nf4/fp8 quant loading) into invokeai/backend/ideogram4/, plus InvokeAI glue (Qwen3-VL text encoding, packed-input build, dual-branch Euler denoise loop). Register the model: BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config (detected via the Ideogram4Pipeline class name in model_index.json), and the Ideogram4DiffusersModel loader that loads both transformers as one Ideogram4TransformerPair submodel plus the Qwen3-VL encoder and VAE. Text-to-image only. * feat(ideogram4): Ideogram 4 backend — model manager, invocations, nf4 loading End-to-end text-to-image backend for Ideogram 4, validated through the real session runner. Vendors the Apache-2.0 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler) into invokeai/backend/ideogram4/ with InvokeAI glue. Registers BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config, and the Ideogram4DiffusersModel loader (two transformers as one Ideogram4TransformerPair; Qwen3-VL encoder + VAE). Both transformers and the encoder load via InvokeLinearNF4 so they work with the partial-load cache. Adds Ideogram4ConditioningInfo/Field/Output and the model_loader/text_encoder/denoise/l2i invocations. Text-to-image only. * feat(ideogram4): frontend — Regions→JSON prompt, graph builder, UI Wires Ideogram 4 into the canvas/generate UI. buildIdeogram4Prompt assembles the structured JSON caption from the global prompt + Canvas Regional Guidance layers (each region → an obj element with a 0–1000 bbox + desc), with raw-JSON passthrough and a plain-text fallback when there are no regions. Adds buildIdeogram4Graph (text-to-image only, no negative prompt) and the enqueue switch. Structured captions use a static string node + a decoy positive-prompt node so the linear batch can't clobber the assembled JSON; plain text uses the real node so dynamic prompts/batching still work. Registers the 'ideogram-4' base (enums, color, names, model picker, grid size 16), a sampler-preset param (V4_QUALITY_48/V4_DEFAULT_20/V4_TURBO_12) replacing the steps/CFG controls, ParamIdeogram4SamplerPreset, and metadata recall. Regenerates schema.ts. * feat(ideogram4): advanced sampler overrides + color palette Advanced accordion now shows only Ideogram 4-relevant controls. Adds optional overrides of the sampler preset — steps, guidance scale (overrides the main gw, preserves the preset's polish tail), and schedule shift (mu) — plus a color palette editor that injects style_description.color_palette into the auto-built JSON caption (uppercase #RRGGBB, max 16, ignored for raw-JSON prompts). All are nullable (null = use preset), recallable from metadata, and the irrelevant controls (VAE, CLIP skip, CFG rescale, seamless, color compensation) are hidden for Ideogram 4. Backend denoise gains steps/guidance_scale/mu fields; schema.ts regenerated. * Use existing keys + fix select size * Update Readme * feat(ideogram4): add Ideogram 4 to starter models with non-commercial license hint - Implement the weight-only fp8 text-encoder load path (was NotImplementedError); validated against the real fp8 build + add CPU unit tests for the fp8 mechanism - Show Ideogram 4 handlers in the Recall Parameters tab - Recall the assembled JSON caption back into the positive prompt - Translate the metadata "Auto" values - Document Ideogram 4 (install/license, regional-guidance JSON prompting, presets) - Add Ideogram 4 nf4 (CUDA) + fp8 (any device) starter models and bundle - Surface the FLUX-style Non-Commercial License popover for Ideogram 4 models - Note the gated HuggingFace license requirement in the model descriptions * Chore Ruff * Chore Ruff * Chore OpenApi * Chore Knit * fix(deps): regenerate uv.lock to remove duplicate packages from bad merge * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * Fix: address ideogram4 review (strict load, 1-step guidance, i18n, runtime caption) - Non-fp8 Ideogram 4 text-encoder load now validates the state dict: unexpected keys raise, missing keys warn (mirrors the fp8 helper) instead of silently accepting a partial load. - Guidance schedule: cap the polish tail at num_steps-1 so at least one main step always remains (the guidance_scale override was silently dropped at num_steps=1), and require steps >= 2 (backend field + frontend slider/marks). - Localize the Ideogram sampler-preset option labels via t() with the step count interpolated; add the three preset i18n keys. - Assemble the structured JSON caption at generation time in a new ideogram4_caption_builder node (Python port of buildIdeogram4Caption) instead of at graph-build time. The graph now wires the real prompt node -> caption builder -> text encoder and returns it as positivePrompt, so dynamic prompts / prompt batching vary the encoded caption (the decoy that dropped them is removed). The builder's output is wired to a new declared ideogram4_caption metadata field via an edge, so each batched image records its actual caption. Regenerates schema.ts for the new node + metadata field. Adds tests for caption assembly, the guidance schedule, and the graph wiring. * feat(ideogram4): step previews + document the model's built-in safety filter - Emit a low-res progress preview each denoise step so the forming image is visible during generation, like the other denoise nodes. Ideogram uses a FLUX.2-style 32-channel VAE, so the packed latent is unpatchified/denormalized (get_latent_norm) and run through the FLUX.2 latent->RGB factors — no full VAE decode per step. The denoise loop now hands the callback the packed grid latent. - Document Ideogram 4's built-in content safety filter in models.mdx: it lives in the model weights (not Invoke's NSFW checker, can't be disabled from Invoke) and false-positives on benign prompts; structured JSON prompts trip it less. * feat(ideogram4): avoid safety-filter false-positives + step previews + caption visibility The main fix: Ideogram 4's built-in safety filter (baked into the model weights) false-positives and returns an "Image blocked by safety filter" placeholder for "degenerate" captions — empirically, an empty `compositional_deconstruction.elements` list, or a single full-frame [0,0,1000,1000] element whose desc just repeats the high_level_description. Our assembly produced empty elements whenever the user drew no regions, so plain prompts were blocked. - Caption assembly (build_ideogram4_caption): - Always emit a structured JSON caption; never bare plain text (the filter false-positives far more on plain text). Raw-JSON pastes still pass through. - When there are no regions, synthesize one default element describing the whole scene from the prompt with a *partial* (non-full-frame) bbox [100,100,900,900]. This never yields an empty/degenerate elements list. Verified end-to-end against the model: the previously-blocked "golden retriever on a skateboard" now renders. - Metadata: always wire the caption builder's output to the ideogram4_caption metadata field, so the viewer's "Structured Caption" row shows the exact JSON that was encoded (not just the raw prompt) for every generation. - Denoise: emit a low-res progress preview each step (unpatchify + FLUX.2 latent->RGB factors, since Ideogram uses a FLUX.2-style 32-channel VAE) so the forming image is visible during generation, like the other denoise nodes. - Docs: document the model's built-in safety filter in models.mdx (it's not Invoke's NSFW checker, can't be disabled from Invoke, and false-positives). Updates the caption/graph tests accordingly (also fixes latent tsc errors in the graph-builder test's core_metadata / ideogram4_caption comparisons). * Chore typegen + openapi + Ruff * Fix Knit * fix(ideogram4): enforce steps>=2 client-side and validate region bbox Address review on the Ideogram 4 PR: - The backend denoise node requires steps >= 2, but the client still accepted ideogram4_steps = 1 in three places, letting a recalled or rehydrated value build a graph that violates the backend schema. Tighten the zod schema to min(2) with `.catch(null)` (a stale/out-of-range value normalizes to null = use the preset instead of failing the whole persisted slice), normalize dispatched values through the schema in setIdeogram4Steps, and refuse an out-of-range value in the ideogram4_steps metadata recall parser. The slider was already min=2. - Ideogram4Region.bbox was an unconstrained Optional[list[int]], so a workflow/API caller could pass a wrong-length or out-of-range box that the caption builder serialized verbatim into the structured prompt. Add a field validator requiring exactly four coordinates, each in 0..1000. Add tests for both: the region bbox contract (valid/None accepted; short, long, negative, and >1000 rejected) and the ideogram4Steps normalization (valid kept, null kept, stale 1 normalized to null on both dispatch and rehydrate). * fix(ideogram4): block unsupported canvas modes/bbox, warn dropped region inputs, constrain bbox Address the latest review on the Ideogram 4 PR: - Canvas readiness allowed unsupported generation modes: Ideogram 4 is txt2img-only (buildIdeogram4Graph asserts it), but a raster layer or inpaint mask makes the compositor pick img2img/outpaint/inpaint, failing only at graph build. Warn in getRasterLayerWarnings/getInpaintMaskWarnings for Ideogram 4 (these already flow into canvas readiness reasons), blocking enqueue up front. - Canvas readiness had no Ideogram 4 bbox check; the backend requires multiples of 16. Add the 16-grid check mirroring the other 16-grid models so an off-grid bbox (e.g. 1025x1024) is blocked instead of failing backend validation. - getRegionalGuidanceWarnings had no Ideogram 4 branch, so a region whose only input is a negative prompt, auto-negative, or reference image looked effective while the graph silently drops it. Warn those inputs are unsupported. - Ideogram4Region.bbox now also rejects inverted boxes (y_min <= y_max, x_min <= x_max). - The advanced-settings badge selector lumped Ideogram 4 into the generic branch, showing stale VAE/clip-skip/CFG-rescale/seamless badges for controls that are hidden for Ideogram. Exclude Ideogram 4 from that branch. - Model bbox as a constrained type (exactly 4 ints, each 0..1000) so the OpenAPI schema advertises minItems/maxItems and item minimum/maximum. Add readiness tests (bbox grid, raster/inpaint blocking, empty-layer allowance, regional-guidance negative/reference-image warnings) and bbox ordering-rejection tests. * fix(ideogram4): reject text encoders with weights left on the meta device _load_text_encoder() builds the encoder under accelerate.init_empty_weights() and previously downgraded missing keys to a warning (both the fp8 path via load_fp8_state_dict(strict=False) and the non-fp8 path). A missing non-tied weight therefore stayed on the meta device, so a bad or mismatched encoder appeared to load and only failed later during device movement or encoding. Add _verify_encoder_fully_materialized(): call tie_weights() to materialize tied weights from their source, then hard-fail if any parameter or buffer remains on the meta device. Wire it into both the fp8 and non-fp8 (incl. bnb-nf4) paths and drop the missing-key warning — genuinely missing non-tied weights are now caught as leftover meta tensors, while tied weights are tolerated. This is a state-based, path-agnostic check. Add tests: passes when fully materialized, raises on a leftover meta tensor from a missing non-tied weight, and tolerates a tied weight resolved by tie_weights(). * chore(ui): prettier formatting for AdvancedSettingsAccordion The Ideogram 4 badge-suppression branch left the wrapped block at its old indentation, failing `pnpm lint:prettier` in frontend-checks. Formatting only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Your Name <you@example.com> Co-authored-by: 4pointoh <97913726+4pointoh@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> | 1 个月前 | |
Feat(model support): ideogram4 support (#9303) * Update to Transformers 5.1.0 * remove extra stuff * chore(deps): compel fork + transformers>=5.9.0 + remove override Switches compel from PyPI 2.1.1 to invoke-ai/compel@main fork which supports transformers 5.x. Bumps transformers floor to 5.9.0. Removes the transformers>=5.1.0 uv override that was only needed to bypass compel 2.1.1's <5.0 constraint. NOTE: compel fork pulls notebook dep (full Jupyter stack); flag to maintainer for cleanup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(z_image): resolve rope_theta from rope_parameters for transformers 5.x transformers 5.x no longer exposes rope_theta as a top-level attribute on Qwen3Config; the value is stored in the rope_parameters (and rope_scaling) dict instead. Read it from there with a getattr fallback so the inv_freq buffer is computed from the configured base (1e6 / 256) instead of raising AttributeError. Applies to both the safetensors and GGUF Qwen3 encoder paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(model_manager): replace removed hf_hub get_token_permission with whoami huggingface_hub 1.x removed get_token_permission(). HFTokenHelper.get_status() now validates the token via whoami(), which returns user info for a valid token and raises HfHubHTTPError for an invalid one. Preserves the original three-way status: VALID on success, INVALID on HfHubHTTPError (e.g. 401), UNKNOWN on any other error (e.g. network failure). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): regenerate uv.lock after upstream merge Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sd3): resolve merge conflict marker, drop T5TokenizerFast The upstream merge left an unresolved conflict marker in _t5_encode and reintroduced T5TokenizerFast. Keep our v5 assertion (T5Tokenizer only) plus upstream's new t5_device logic, and drop the now-dead T5TokenizerFast monkeypatch in the test (the name no longer exists in the module). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: ruff fixes on merge-resolved files - flux_text_encoder.py: drop unused typing.Union (F401) left by v5 import merge - huggingface.py: ruff format (wrap append(SimpleNamespace(...))) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): pin transformers <5.6 (diffusers single-file CLIP incompat) transformers 5.6 flattened CLIPTextModel (removed the self.text_model wrapper, hoisted embeddings/encoder/final_layer_norm to the top level). diffusers' single-file checkpoint loader (create_diffusers_clip_model_from_ldm) still assumes the nested layout, so loading SD1.5 .safetensors checkpoints fails on 5.6+ with 'CLIPTextModel object has no attribute text_model' and, once that read is shimmed, 'Cannot copy out of meta tensor' (weights never populate the flattened model). Pin to >=5.5,<5.6 (last pre-flattening release) which keeps both the single-file and from_pretrained paths working. The invoke-ai/compel fork accepts any 5.x. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * @ chore(deps): replace compel fork with official compel 2.4.0 compel 2.4.0 (released 2026-05-30) merges the transformers-5 support that the invoke-ai fork carried (both descend from upstream PR #129), plus the maintainer-reviewed padding rework and added diffusers/T5 smoke coverage. Switch from the git fork to the PyPI release. - pyproject: compel git+main -> compel>=2.4.0,<3 - uv.lock: compel 2.3.1 (git 8f404b45) -> 2.4.0 (pypi) - transformers stays 5.5.4 (satisfies compel >=5,<6 and our <5.6 pin) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> @ * feat(ideogram4): backend + model-manager registration for Ideogram 4 Vendor the Apache-2.0 Ideogram 4 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler, nf4/fp8 quant loading) into invokeai/backend/ideogram4/, plus InvokeAI glue (Qwen3-VL text encoding, packed-input build, dual-branch Euler denoise loop). Register the model: BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config (detected via the Ideogram4Pipeline class name in model_index.json), and the Ideogram4DiffusersModel loader that loads both transformers as one Ideogram4TransformerPair submodel plus the Qwen3-VL encoder and VAE. Text-to-image only. * feat(ideogram4): Ideogram 4 backend — model manager, invocations, nf4 loading End-to-end text-to-image backend for Ideogram 4, validated through the real session runner. Vendors the Apache-2.0 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler) into invokeai/backend/ideogram4/ with InvokeAI glue. Registers BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config, and the Ideogram4DiffusersModel loader (two transformers as one Ideogram4TransformerPair; Qwen3-VL encoder + VAE). Both transformers and the encoder load via InvokeLinearNF4 so they work with the partial-load cache. Adds Ideogram4ConditioningInfo/Field/Output and the model_loader/text_encoder/denoise/l2i invocations. Text-to-image only. * feat(ideogram4): frontend — Regions→JSON prompt, graph builder, UI Wires Ideogram 4 into the canvas/generate UI. buildIdeogram4Prompt assembles the structured JSON caption from the global prompt + Canvas Regional Guidance layers (each region → an obj element with a 0–1000 bbox + desc), with raw-JSON passthrough and a plain-text fallback when there are no regions. Adds buildIdeogram4Graph (text-to-image only, no negative prompt) and the enqueue switch. Structured captions use a static string node + a decoy positive-prompt node so the linear batch can't clobber the assembled JSON; plain text uses the real node so dynamic prompts/batching still work. Registers the 'ideogram-4' base (enums, color, names, model picker, grid size 16), a sampler-preset param (V4_QUALITY_48/V4_DEFAULT_20/V4_TURBO_12) replacing the steps/CFG controls, ParamIdeogram4SamplerPreset, and metadata recall. Regenerates schema.ts. * feat(ideogram4): advanced sampler overrides + color palette Advanced accordion now shows only Ideogram 4-relevant controls. Adds optional overrides of the sampler preset — steps, guidance scale (overrides the main gw, preserves the preset's polish tail), and schedule shift (mu) — plus a color palette editor that injects style_description.color_palette into the auto-built JSON caption (uppercase #RRGGBB, max 16, ignored for raw-JSON prompts). All are nullable (null = use preset), recallable from metadata, and the irrelevant controls (VAE, CLIP skip, CFG rescale, seamless, color compensation) are hidden for Ideogram 4. Backend denoise gains steps/guidance_scale/mu fields; schema.ts regenerated. * Use existing keys + fix select size * Update Readme * feat(ideogram4): add Ideogram 4 to starter models with non-commercial license hint - Implement the weight-only fp8 text-encoder load path (was NotImplementedError); validated against the real fp8 build + add CPU unit tests for the fp8 mechanism - Show Ideogram 4 handlers in the Recall Parameters tab - Recall the assembled JSON caption back into the positive prompt - Translate the metadata "Auto" values - Document Ideogram 4 (install/license, regional-guidance JSON prompting, presets) - Add Ideogram 4 nf4 (CUDA) + fp8 (any device) starter models and bundle - Surface the FLUX-style Non-Commercial License popover for Ideogram 4 models - Note the gated HuggingFace license requirement in the model descriptions * Chore Ruff * Chore Ruff * Chore OpenApi * Chore Knit * fix(deps): regenerate uv.lock to remove duplicate packages from bad merge * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * Fix: address ideogram4 review (strict load, 1-step guidance, i18n, runtime caption) - Non-fp8 Ideogram 4 text-encoder load now validates the state dict: unexpected keys raise, missing keys warn (mirrors the fp8 helper) instead of silently accepting a partial load. - Guidance schedule: cap the polish tail at num_steps-1 so at least one main step always remains (the guidance_scale override was silently dropped at num_steps=1), and require steps >= 2 (backend field + frontend slider/marks). - Localize the Ideogram sampler-preset option labels via t() with the step count interpolated; add the three preset i18n keys. - Assemble the structured JSON caption at generation time in a new ideogram4_caption_builder node (Python port of buildIdeogram4Caption) instead of at graph-build time. The graph now wires the real prompt node -> caption builder -> text encoder and returns it as positivePrompt, so dynamic prompts / prompt batching vary the encoded caption (the decoy that dropped them is removed). The builder's output is wired to a new declared ideogram4_caption metadata field via an edge, so each batched image records its actual caption. Regenerates schema.ts for the new node + metadata field. Adds tests for caption assembly, the guidance schedule, and the graph wiring. * feat(ideogram4): step previews + document the model's built-in safety filter - Emit a low-res progress preview each denoise step so the forming image is visible during generation, like the other denoise nodes. Ideogram uses a FLUX.2-style 32-channel VAE, so the packed latent is unpatchified/denormalized (get_latent_norm) and run through the FLUX.2 latent->RGB factors — no full VAE decode per step. The denoise loop now hands the callback the packed grid latent. - Document Ideogram 4's built-in content safety filter in models.mdx: it lives in the model weights (not Invoke's NSFW checker, can't be disabled from Invoke) and false-positives on benign prompts; structured JSON prompts trip it less. * feat(ideogram4): avoid safety-filter false-positives + step previews + caption visibility The main fix: Ideogram 4's built-in safety filter (baked into the model weights) false-positives and returns an "Image blocked by safety filter" placeholder for "degenerate" captions — empirically, an empty `compositional_deconstruction.elements` list, or a single full-frame [0,0,1000,1000] element whose desc just repeats the high_level_description. Our assembly produced empty elements whenever the user drew no regions, so plain prompts were blocked. - Caption assembly (build_ideogram4_caption): - Always emit a structured JSON caption; never bare plain text (the filter false-positives far more on plain text). Raw-JSON pastes still pass through. - When there are no regions, synthesize one default element describing the whole scene from the prompt with a *partial* (non-full-frame) bbox [100,100,900,900]. This never yields an empty/degenerate elements list. Verified end-to-end against the model: the previously-blocked "golden retriever on a skateboard" now renders. - Metadata: always wire the caption builder's output to the ideogram4_caption metadata field, so the viewer's "Structured Caption" row shows the exact JSON that was encoded (not just the raw prompt) for every generation. - Denoise: emit a low-res progress preview each step (unpatchify + FLUX.2 latent->RGB factors, since Ideogram uses a FLUX.2-style 32-channel VAE) so the forming image is visible during generation, like the other denoise nodes. - Docs: document the model's built-in safety filter in models.mdx (it's not Invoke's NSFW checker, can't be disabled from Invoke, and false-positives). Updates the caption/graph tests accordingly (also fixes latent tsc errors in the graph-builder test's core_metadata / ideogram4_caption comparisons). * Chore typegen + openapi + Ruff * Fix Knit * fix(ideogram4): enforce steps>=2 client-side and validate region bbox Address review on the Ideogram 4 PR: - The backend denoise node requires steps >= 2, but the client still accepted ideogram4_steps = 1 in three places, letting a recalled or rehydrated value build a graph that violates the backend schema. Tighten the zod schema to min(2) with `.catch(null)` (a stale/out-of-range value normalizes to null = use the preset instead of failing the whole persisted slice), normalize dispatched values through the schema in setIdeogram4Steps, and refuse an out-of-range value in the ideogram4_steps metadata recall parser. The slider was already min=2. - Ideogram4Region.bbox was an unconstrained Optional[list[int]], so a workflow/API caller could pass a wrong-length or out-of-range box that the caption builder serialized verbatim into the structured prompt. Add a field validator requiring exactly four coordinates, each in 0..1000. Add tests for both: the region bbox contract (valid/None accepted; short, long, negative, and >1000 rejected) and the ideogram4Steps normalization (valid kept, null kept, stale 1 normalized to null on both dispatch and rehydrate). * fix(ideogram4): block unsupported canvas modes/bbox, warn dropped region inputs, constrain bbox Address the latest review on the Ideogram 4 PR: - Canvas readiness allowed unsupported generation modes: Ideogram 4 is txt2img-only (buildIdeogram4Graph asserts it), but a raster layer or inpaint mask makes the compositor pick img2img/outpaint/inpaint, failing only at graph build. Warn in getRasterLayerWarnings/getInpaintMaskWarnings for Ideogram 4 (these already flow into canvas readiness reasons), blocking enqueue up front. - Canvas readiness had no Ideogram 4 bbox check; the backend requires multiples of 16. Add the 16-grid check mirroring the other 16-grid models so an off-grid bbox (e.g. 1025x1024) is blocked instead of failing backend validation. - getRegionalGuidanceWarnings had no Ideogram 4 branch, so a region whose only input is a negative prompt, auto-negative, or reference image looked effective while the graph silently drops it. Warn those inputs are unsupported. - Ideogram4Region.bbox now also rejects inverted boxes (y_min <= y_max, x_min <= x_max). - The advanced-settings badge selector lumped Ideogram 4 into the generic branch, showing stale VAE/clip-skip/CFG-rescale/seamless badges for controls that are hidden for Ideogram. Exclude Ideogram 4 from that branch. - Model bbox as a constrained type (exactly 4 ints, each 0..1000) so the OpenAPI schema advertises minItems/maxItems and item minimum/maximum. Add readiness tests (bbox grid, raster/inpaint blocking, empty-layer allowance, regional-guidance negative/reference-image warnings) and bbox ordering-rejection tests. * fix(ideogram4): reject text encoders with weights left on the meta device _load_text_encoder() builds the encoder under accelerate.init_empty_weights() and previously downgraded missing keys to a warning (both the fp8 path via load_fp8_state_dict(strict=False) and the non-fp8 path). A missing non-tied weight therefore stayed on the meta device, so a bad or mismatched encoder appeared to load and only failed later during device movement or encoding. Add _verify_encoder_fully_materialized(): call tie_weights() to materialize tied weights from their source, then hard-fail if any parameter or buffer remains on the meta device. Wire it into both the fp8 and non-fp8 (incl. bnb-nf4) paths and drop the missing-key warning — genuinely missing non-tied weights are now caught as leftover meta tensors, while tied weights are tolerated. This is a state-based, path-agnostic check. Add tests: passes when fully materialized, raises on a leftover meta tensor from a missing non-tied weight, and tolerates a tied weight resolved by tie_weights(). * chore(ui): prettier formatting for AdvancedSettingsAccordion The Ideogram 4 badge-suppression branch left the wrapped block at its old indentation, failing `pnpm lint:prettier` in frontend-checks. Formatting only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Your Name <you@example.com> Co-authored-by: 4pointoh <97913726+4pointoh@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> | 1 个月前 | |
Feat(model support): ideogram4 support (#9303) * Update to Transformers 5.1.0 * remove extra stuff * chore(deps): compel fork + transformers>=5.9.0 + remove override Switches compel from PyPI 2.1.1 to invoke-ai/compel@main fork which supports transformers 5.x. Bumps transformers floor to 5.9.0. Removes the transformers>=5.1.0 uv override that was only needed to bypass compel 2.1.1's <5.0 constraint. NOTE: compel fork pulls notebook dep (full Jupyter stack); flag to maintainer for cleanup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(z_image): resolve rope_theta from rope_parameters for transformers 5.x transformers 5.x no longer exposes rope_theta as a top-level attribute on Qwen3Config; the value is stored in the rope_parameters (and rope_scaling) dict instead. Read it from there with a getattr fallback so the inv_freq buffer is computed from the configured base (1e6 / 256) instead of raising AttributeError. Applies to both the safetensors and GGUF Qwen3 encoder paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(model_manager): replace removed hf_hub get_token_permission with whoami huggingface_hub 1.x removed get_token_permission(). HFTokenHelper.get_status() now validates the token via whoami(), which returns user info for a valid token and raises HfHubHTTPError for an invalid one. Preserves the original three-way status: VALID on success, INVALID on HfHubHTTPError (e.g. 401), UNKNOWN on any other error (e.g. network failure). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): regenerate uv.lock after upstream merge Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sd3): resolve merge conflict marker, drop T5TokenizerFast The upstream merge left an unresolved conflict marker in _t5_encode and reintroduced T5TokenizerFast. Keep our v5 assertion (T5Tokenizer only) plus upstream's new t5_device logic, and drop the now-dead T5TokenizerFast monkeypatch in the test (the name no longer exists in the module). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: ruff fixes on merge-resolved files - flux_text_encoder.py: drop unused typing.Union (F401) left by v5 import merge - huggingface.py: ruff format (wrap append(SimpleNamespace(...))) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): pin transformers <5.6 (diffusers single-file CLIP incompat) transformers 5.6 flattened CLIPTextModel (removed the self.text_model wrapper, hoisted embeddings/encoder/final_layer_norm to the top level). diffusers' single-file checkpoint loader (create_diffusers_clip_model_from_ldm) still assumes the nested layout, so loading SD1.5 .safetensors checkpoints fails on 5.6+ with 'CLIPTextModel object has no attribute text_model' and, once that read is shimmed, 'Cannot copy out of meta tensor' (weights never populate the flattened model). Pin to >=5.5,<5.6 (last pre-flattening release) which keeps both the single-file and from_pretrained paths working. The invoke-ai/compel fork accepts any 5.x. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * @ chore(deps): replace compel fork with official compel 2.4.0 compel 2.4.0 (released 2026-05-30) merges the transformers-5 support that the invoke-ai fork carried (both descend from upstream PR #129), plus the maintainer-reviewed padding rework and added diffusers/T5 smoke coverage. Switch from the git fork to the PyPI release. - pyproject: compel git+main -> compel>=2.4.0,<3 - uv.lock: compel 2.3.1 (git 8f404b45) -> 2.4.0 (pypi) - transformers stays 5.5.4 (satisfies compel >=5,<6 and our <5.6 pin) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> @ * feat(ideogram4): backend + model-manager registration for Ideogram 4 Vendor the Apache-2.0 Ideogram 4 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler, nf4/fp8 quant loading) into invokeai/backend/ideogram4/, plus InvokeAI glue (Qwen3-VL text encoding, packed-input build, dual-branch Euler denoise loop). Register the model: BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config (detected via the Ideogram4Pipeline class name in model_index.json), and the Ideogram4DiffusersModel loader that loads both transformers as one Ideogram4TransformerPair submodel plus the Qwen3-VL encoder and VAE. Text-to-image only. * feat(ideogram4): Ideogram 4 backend — model manager, invocations, nf4 loading End-to-end text-to-image backend for Ideogram 4, validated through the real session runner. Vendors the Apache-2.0 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler) into invokeai/backend/ideogram4/ with InvokeAI glue. Registers BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config, and the Ideogram4DiffusersModel loader (two transformers as one Ideogram4TransformerPair; Qwen3-VL encoder + VAE). Both transformers and the encoder load via InvokeLinearNF4 so they work with the partial-load cache. Adds Ideogram4ConditioningInfo/Field/Output and the model_loader/text_encoder/denoise/l2i invocations. Text-to-image only. * feat(ideogram4): frontend — Regions→JSON prompt, graph builder, UI Wires Ideogram 4 into the canvas/generate UI. buildIdeogram4Prompt assembles the structured JSON caption from the global prompt + Canvas Regional Guidance layers (each region → an obj element with a 0–1000 bbox + desc), with raw-JSON passthrough and a plain-text fallback when there are no regions. Adds buildIdeogram4Graph (text-to-image only, no negative prompt) and the enqueue switch. Structured captions use a static string node + a decoy positive-prompt node so the linear batch can't clobber the assembled JSON; plain text uses the real node so dynamic prompts/batching still work. Registers the 'ideogram-4' base (enums, color, names, model picker, grid size 16), a sampler-preset param (V4_QUALITY_48/V4_DEFAULT_20/V4_TURBO_12) replacing the steps/CFG controls, ParamIdeogram4SamplerPreset, and metadata recall. Regenerates schema.ts. * feat(ideogram4): advanced sampler overrides + color palette Advanced accordion now shows only Ideogram 4-relevant controls. Adds optional overrides of the sampler preset — steps, guidance scale (overrides the main gw, preserves the preset's polish tail), and schedule shift (mu) — plus a color palette editor that injects style_description.color_palette into the auto-built JSON caption (uppercase #RRGGBB, max 16, ignored for raw-JSON prompts). All are nullable (null = use preset), recallable from metadata, and the irrelevant controls (VAE, CLIP skip, CFG rescale, seamless, color compensation) are hidden for Ideogram 4. Backend denoise gains steps/guidance_scale/mu fields; schema.ts regenerated. * Use existing keys + fix select size * Update Readme * feat(ideogram4): add Ideogram 4 to starter models with non-commercial license hint - Implement the weight-only fp8 text-encoder load path (was NotImplementedError); validated against the real fp8 build + add CPU unit tests for the fp8 mechanism - Show Ideogram 4 handlers in the Recall Parameters tab - Recall the assembled JSON caption back into the positive prompt - Translate the metadata "Auto" values - Document Ideogram 4 (install/license, regional-guidance JSON prompting, presets) - Add Ideogram 4 nf4 (CUDA) + fp8 (any device) starter models and bundle - Surface the FLUX-style Non-Commercial License popover for Ideogram 4 models - Note the gated HuggingFace license requirement in the model descriptions * Chore Ruff * Chore Ruff * Chore OpenApi * Chore Knit * fix(deps): regenerate uv.lock to remove duplicate packages from bad merge * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * Fix: address ideogram4 review (strict load, 1-step guidance, i18n, runtime caption) - Non-fp8 Ideogram 4 text-encoder load now validates the state dict: unexpected keys raise, missing keys warn (mirrors the fp8 helper) instead of silently accepting a partial load. - Guidance schedule: cap the polish tail at num_steps-1 so at least one main step always remains (the guidance_scale override was silently dropped at num_steps=1), and require steps >= 2 (backend field + frontend slider/marks). - Localize the Ideogram sampler-preset option labels via t() with the step count interpolated; add the three preset i18n keys. - Assemble the structured JSON caption at generation time in a new ideogram4_caption_builder node (Python port of buildIdeogram4Caption) instead of at graph-build time. The graph now wires the real prompt node -> caption builder -> text encoder and returns it as positivePrompt, so dynamic prompts / prompt batching vary the encoded caption (the decoy that dropped them is removed). The builder's output is wired to a new declared ideogram4_caption metadata field via an edge, so each batched image records its actual caption. Regenerates schema.ts for the new node + metadata field. Adds tests for caption assembly, the guidance schedule, and the graph wiring. * feat(ideogram4): step previews + document the model's built-in safety filter - Emit a low-res progress preview each denoise step so the forming image is visible during generation, like the other denoise nodes. Ideogram uses a FLUX.2-style 32-channel VAE, so the packed latent is unpatchified/denormalized (get_latent_norm) and run through the FLUX.2 latent->RGB factors — no full VAE decode per step. The denoise loop now hands the callback the packed grid latent. - Document Ideogram 4's built-in content safety filter in models.mdx: it lives in the model weights (not Invoke's NSFW checker, can't be disabled from Invoke) and false-positives on benign prompts; structured JSON prompts trip it less. * feat(ideogram4): avoid safety-filter false-positives + step previews + caption visibility The main fix: Ideogram 4's built-in safety filter (baked into the model weights) false-positives and returns an "Image blocked by safety filter" placeholder for "degenerate" captions — empirically, an empty `compositional_deconstruction.elements` list, or a single full-frame [0,0,1000,1000] element whose desc just repeats the high_level_description. Our assembly produced empty elements whenever the user drew no regions, so plain prompts were blocked. - Caption assembly (build_ideogram4_caption): - Always emit a structured JSON caption; never bare plain text (the filter false-positives far more on plain text). Raw-JSON pastes still pass through. - When there are no regions, synthesize one default element describing the whole scene from the prompt with a *partial* (non-full-frame) bbox [100,100,900,900]. This never yields an empty/degenerate elements list. Verified end-to-end against the model: the previously-blocked "golden retriever on a skateboard" now renders. - Metadata: always wire the caption builder's output to the ideogram4_caption metadata field, so the viewer's "Structured Caption" row shows the exact JSON that was encoded (not just the raw prompt) for every generation. - Denoise: emit a low-res progress preview each step (unpatchify + FLUX.2 latent->RGB factors, since Ideogram uses a FLUX.2-style 32-channel VAE) so the forming image is visible during generation, like the other denoise nodes. - Docs: document the model's built-in safety filter in models.mdx (it's not Invoke's NSFW checker, can't be disabled from Invoke, and false-positives). Updates the caption/graph tests accordingly (also fixes latent tsc errors in the graph-builder test's core_metadata / ideogram4_caption comparisons). * Chore typegen + openapi + Ruff * Fix Knit * fix(ideogram4): enforce steps>=2 client-side and validate region bbox Address review on the Ideogram 4 PR: - The backend denoise node requires steps >= 2, but the client still accepted ideogram4_steps = 1 in three places, letting a recalled or rehydrated value build a graph that violates the backend schema. Tighten the zod schema to min(2) with `.catch(null)` (a stale/out-of-range value normalizes to null = use the preset instead of failing the whole persisted slice), normalize dispatched values through the schema in setIdeogram4Steps, and refuse an out-of-range value in the ideogram4_steps metadata recall parser. The slider was already min=2. - Ideogram4Region.bbox was an unconstrained Optional[list[int]], so a workflow/API caller could pass a wrong-length or out-of-range box that the caption builder serialized verbatim into the structured prompt. Add a field validator requiring exactly four coordinates, each in 0..1000. Add tests for both: the region bbox contract (valid/None accepted; short, long, negative, and >1000 rejected) and the ideogram4Steps normalization (valid kept, null kept, stale 1 normalized to null on both dispatch and rehydrate). * fix(ideogram4): block unsupported canvas modes/bbox, warn dropped region inputs, constrain bbox Address the latest review on the Ideogram 4 PR: - Canvas readiness allowed unsupported generation modes: Ideogram 4 is txt2img-only (buildIdeogram4Graph asserts it), but a raster layer or inpaint mask makes the compositor pick img2img/outpaint/inpaint, failing only at graph build. Warn in getRasterLayerWarnings/getInpaintMaskWarnings for Ideogram 4 (these already flow into canvas readiness reasons), blocking enqueue up front. - Canvas readiness had no Ideogram 4 bbox check; the backend requires multiples of 16. Add the 16-grid check mirroring the other 16-grid models so an off-grid bbox (e.g. 1025x1024) is blocked instead of failing backend validation. - getRegionalGuidanceWarnings had no Ideogram 4 branch, so a region whose only input is a negative prompt, auto-negative, or reference image looked effective while the graph silently drops it. Warn those inputs are unsupported. - Ideogram4Region.bbox now also rejects inverted boxes (y_min <= y_max, x_min <= x_max). - The advanced-settings badge selector lumped Ideogram 4 into the generic branch, showing stale VAE/clip-skip/CFG-rescale/seamless badges for controls that are hidden for Ideogram. Exclude Ideogram 4 from that branch. - Model bbox as a constrained type (exactly 4 ints, each 0..1000) so the OpenAPI schema advertises minItems/maxItems and item minimum/maximum. Add readiness tests (bbox grid, raster/inpaint blocking, empty-layer allowance, regional-guidance negative/reference-image warnings) and bbox ordering-rejection tests. * fix(ideogram4): reject text encoders with weights left on the meta device _load_text_encoder() builds the encoder under accelerate.init_empty_weights() and previously downgraded missing keys to a warning (both the fp8 path via load_fp8_state_dict(strict=False) and the non-fp8 path). A missing non-tied weight therefore stayed on the meta device, so a bad or mismatched encoder appeared to load and only failed later during device movement or encoding. Add _verify_encoder_fully_materialized(): call tie_weights() to materialize tied weights from their source, then hard-fail if any parameter or buffer remains on the meta device. Wire it into both the fp8 and non-fp8 (incl. bnb-nf4) paths and drop the missing-key warning — genuinely missing non-tied weights are now caught as leftover meta tensors, while tied weights are tolerated. This is a state-based, path-agnostic check. Add tests: passes when fully materialized, raises on a leftover meta tensor from a missing non-tied weight, and tolerates a tied weight resolved by tie_weights(). * chore(ui): prettier formatting for AdvancedSettingsAccordion The Ideogram 4 badge-suppression branch left the wrapped block at its old indentation, failing `pnpm lint:prettier` in frontend-checks. Formatting only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Your Name <you@example.com> Co-authored-by: 4pointoh <97913726+4pointoh@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> | 1 个月前 | |
Feat(model support): ideogram4 support (#9303) * Update to Transformers 5.1.0 * remove extra stuff * chore(deps): compel fork + transformers>=5.9.0 + remove override Switches compel from PyPI 2.1.1 to invoke-ai/compel@main fork which supports transformers 5.x. Bumps transformers floor to 5.9.0. Removes the transformers>=5.1.0 uv override that was only needed to bypass compel 2.1.1's <5.0 constraint. NOTE: compel fork pulls notebook dep (full Jupyter stack); flag to maintainer for cleanup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(z_image): resolve rope_theta from rope_parameters for transformers 5.x transformers 5.x no longer exposes rope_theta as a top-level attribute on Qwen3Config; the value is stored in the rope_parameters (and rope_scaling) dict instead. Read it from there with a getattr fallback so the inv_freq buffer is computed from the configured base (1e6 / 256) instead of raising AttributeError. Applies to both the safetensors and GGUF Qwen3 encoder paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(model_manager): replace removed hf_hub get_token_permission with whoami huggingface_hub 1.x removed get_token_permission(). HFTokenHelper.get_status() now validates the token via whoami(), which returns user info for a valid token and raises HfHubHTTPError for an invalid one. Preserves the original three-way status: VALID on success, INVALID on HfHubHTTPError (e.g. 401), UNKNOWN on any other error (e.g. network failure). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): regenerate uv.lock after upstream merge Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sd3): resolve merge conflict marker, drop T5TokenizerFast The upstream merge left an unresolved conflict marker in _t5_encode and reintroduced T5TokenizerFast. Keep our v5 assertion (T5Tokenizer only) plus upstream's new t5_device logic, and drop the now-dead T5TokenizerFast monkeypatch in the test (the name no longer exists in the module). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: ruff fixes on merge-resolved files - flux_text_encoder.py: drop unused typing.Union (F401) left by v5 import merge - huggingface.py: ruff format (wrap append(SimpleNamespace(...))) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): pin transformers <5.6 (diffusers single-file CLIP incompat) transformers 5.6 flattened CLIPTextModel (removed the self.text_model wrapper, hoisted embeddings/encoder/final_layer_norm to the top level). diffusers' single-file checkpoint loader (create_diffusers_clip_model_from_ldm) still assumes the nested layout, so loading SD1.5 .safetensors checkpoints fails on 5.6+ with 'CLIPTextModel object has no attribute text_model' and, once that read is shimmed, 'Cannot copy out of meta tensor' (weights never populate the flattened model). Pin to >=5.5,<5.6 (last pre-flattening release) which keeps both the single-file and from_pretrained paths working. The invoke-ai/compel fork accepts any 5.x. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * @ chore(deps): replace compel fork with official compel 2.4.0 compel 2.4.0 (released 2026-05-30) merges the transformers-5 support that the invoke-ai fork carried (both descend from upstream PR #129), plus the maintainer-reviewed padding rework and added diffusers/T5 smoke coverage. Switch from the git fork to the PyPI release. - pyproject: compel git+main -> compel>=2.4.0,<3 - uv.lock: compel 2.3.1 (git 8f404b45) -> 2.4.0 (pypi) - transformers stays 5.5.4 (satisfies compel >=5,<6 and our <5.6 pin) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> @ * feat(ideogram4): backend + model-manager registration for Ideogram 4 Vendor the Apache-2.0 Ideogram 4 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler, nf4/fp8 quant loading) into invokeai/backend/ideogram4/, plus InvokeAI glue (Qwen3-VL text encoding, packed-input build, dual-branch Euler denoise loop). Register the model: BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config (detected via the Ideogram4Pipeline class name in model_index.json), and the Ideogram4DiffusersModel loader that loads both transformers as one Ideogram4TransformerPair submodel plus the Qwen3-VL encoder and VAE. Text-to-image only. * feat(ideogram4): Ideogram 4 backend — model manager, invocations, nf4 loading End-to-end text-to-image backend for Ideogram 4, validated through the real session runner. Vendors the Apache-2.0 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler) into invokeai/backend/ideogram4/ with InvokeAI glue. Registers BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config, and the Ideogram4DiffusersModel loader (two transformers as one Ideogram4TransformerPair; Qwen3-VL encoder + VAE). Both transformers and the encoder load via InvokeLinearNF4 so they work with the partial-load cache. Adds Ideogram4ConditioningInfo/Field/Output and the model_loader/text_encoder/denoise/l2i invocations. Text-to-image only. * feat(ideogram4): frontend — Regions→JSON prompt, graph builder, UI Wires Ideogram 4 into the canvas/generate UI. buildIdeogram4Prompt assembles the structured JSON caption from the global prompt + Canvas Regional Guidance layers (each region → an obj element with a 0–1000 bbox + desc), with raw-JSON passthrough and a plain-text fallback when there are no regions. Adds buildIdeogram4Graph (text-to-image only, no negative prompt) and the enqueue switch. Structured captions use a static string node + a decoy positive-prompt node so the linear batch can't clobber the assembled JSON; plain text uses the real node so dynamic prompts/batching still work. Registers the 'ideogram-4' base (enums, color, names, model picker, grid size 16), a sampler-preset param (V4_QUALITY_48/V4_DEFAULT_20/V4_TURBO_12) replacing the steps/CFG controls, ParamIdeogram4SamplerPreset, and metadata recall. Regenerates schema.ts. * feat(ideogram4): advanced sampler overrides + color palette Advanced accordion now shows only Ideogram 4-relevant controls. Adds optional overrides of the sampler preset — steps, guidance scale (overrides the main gw, preserves the preset's polish tail), and schedule shift (mu) — plus a color palette editor that injects style_description.color_palette into the auto-built JSON caption (uppercase #RRGGBB, max 16, ignored for raw-JSON prompts). All are nullable (null = use preset), recallable from metadata, and the irrelevant controls (VAE, CLIP skip, CFG rescale, seamless, color compensation) are hidden for Ideogram 4. Backend denoise gains steps/guidance_scale/mu fields; schema.ts regenerated. * Use existing keys + fix select size * Update Readme * feat(ideogram4): add Ideogram 4 to starter models with non-commercial license hint - Implement the weight-only fp8 text-encoder load path (was NotImplementedError); validated against the real fp8 build + add CPU unit tests for the fp8 mechanism - Show Ideogram 4 handlers in the Recall Parameters tab - Recall the assembled JSON caption back into the positive prompt - Translate the metadata "Auto" values - Document Ideogram 4 (install/license, regional-guidance JSON prompting, presets) - Add Ideogram 4 nf4 (CUDA) + fp8 (any device) starter models and bundle - Surface the FLUX-style Non-Commercial License popover for Ideogram 4 models - Note the gated HuggingFace license requirement in the model descriptions * Chore Ruff * Chore Ruff * Chore OpenApi * Chore Knit * fix(deps): regenerate uv.lock to remove duplicate packages from bad merge * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * Fix: address ideogram4 review (strict load, 1-step guidance, i18n, runtime caption) - Non-fp8 Ideogram 4 text-encoder load now validates the state dict: unexpected keys raise, missing keys warn (mirrors the fp8 helper) instead of silently accepting a partial load. - Guidance schedule: cap the polish tail at num_steps-1 so at least one main step always remains (the guidance_scale override was silently dropped at num_steps=1), and require steps >= 2 (backend field + frontend slider/marks). - Localize the Ideogram sampler-preset option labels via t() with the step count interpolated; add the three preset i18n keys. - Assemble the structured JSON caption at generation time in a new ideogram4_caption_builder node (Python port of buildIdeogram4Caption) instead of at graph-build time. The graph now wires the real prompt node -> caption builder -> text encoder and returns it as positivePrompt, so dynamic prompts / prompt batching vary the encoded caption (the decoy that dropped them is removed). The builder's output is wired to a new declared ideogram4_caption metadata field via an edge, so each batched image records its actual caption. Regenerates schema.ts for the new node + metadata field. Adds tests for caption assembly, the guidance schedule, and the graph wiring. * feat(ideogram4): step previews + document the model's built-in safety filter - Emit a low-res progress preview each denoise step so the forming image is visible during generation, like the other denoise nodes. Ideogram uses a FLUX.2-style 32-channel VAE, so the packed latent is unpatchified/denormalized (get_latent_norm) and run through the FLUX.2 latent->RGB factors — no full VAE decode per step. The denoise loop now hands the callback the packed grid latent. - Document Ideogram 4's built-in content safety filter in models.mdx: it lives in the model weights (not Invoke's NSFW checker, can't be disabled from Invoke) and false-positives on benign prompts; structured JSON prompts trip it less. * feat(ideogram4): avoid safety-filter false-positives + step previews + caption visibility The main fix: Ideogram 4's built-in safety filter (baked into the model weights) false-positives and returns an "Image blocked by safety filter" placeholder for "degenerate" captions — empirically, an empty `compositional_deconstruction.elements` list, or a single full-frame [0,0,1000,1000] element whose desc just repeats the high_level_description. Our assembly produced empty elements whenever the user drew no regions, so plain prompts were blocked. - Caption assembly (build_ideogram4_caption): - Always emit a structured JSON caption; never bare plain text (the filter false-positives far more on plain text). Raw-JSON pastes still pass through. - When there are no regions, synthesize one default element describing the whole scene from the prompt with a *partial* (non-full-frame) bbox [100,100,900,900]. This never yields an empty/degenerate elements list. Verified end-to-end against the model: the previously-blocked "golden retriever on a skateboard" now renders. - Metadata: always wire the caption builder's output to the ideogram4_caption metadata field, so the viewer's "Structured Caption" row shows the exact JSON that was encoded (not just the raw prompt) for every generation. - Denoise: emit a low-res progress preview each step (unpatchify + FLUX.2 latent->RGB factors, since Ideogram uses a FLUX.2-style 32-channel VAE) so the forming image is visible during generation, like the other denoise nodes. - Docs: document the model's built-in safety filter in models.mdx (it's not Invoke's NSFW checker, can't be disabled from Invoke, and false-positives). Updates the caption/graph tests accordingly (also fixes latent tsc errors in the graph-builder test's core_metadata / ideogram4_caption comparisons). * Chore typegen + openapi + Ruff * Fix Knit * fix(ideogram4): enforce steps>=2 client-side and validate region bbox Address review on the Ideogram 4 PR: - The backend denoise node requires steps >= 2, but the client still accepted ideogram4_steps = 1 in three places, letting a recalled or rehydrated value build a graph that violates the backend schema. Tighten the zod schema to min(2) with `.catch(null)` (a stale/out-of-range value normalizes to null = use the preset instead of failing the whole persisted slice), normalize dispatched values through the schema in setIdeogram4Steps, and refuse an out-of-range value in the ideogram4_steps metadata recall parser. The slider was already min=2. - Ideogram4Region.bbox was an unconstrained Optional[list[int]], so a workflow/API caller could pass a wrong-length or out-of-range box that the caption builder serialized verbatim into the structured prompt. Add a field validator requiring exactly four coordinates, each in 0..1000. Add tests for both: the region bbox contract (valid/None accepted; short, long, negative, and >1000 rejected) and the ideogram4Steps normalization (valid kept, null kept, stale 1 normalized to null on both dispatch and rehydrate). * fix(ideogram4): block unsupported canvas modes/bbox, warn dropped region inputs, constrain bbox Address the latest review on the Ideogram 4 PR: - Canvas readiness allowed unsupported generation modes: Ideogram 4 is txt2img-only (buildIdeogram4Graph asserts it), but a raster layer or inpaint mask makes the compositor pick img2img/outpaint/inpaint, failing only at graph build. Warn in getRasterLayerWarnings/getInpaintMaskWarnings for Ideogram 4 (these already flow into canvas readiness reasons), blocking enqueue up front. - Canvas readiness had no Ideogram 4 bbox check; the backend requires multiples of 16. Add the 16-grid check mirroring the other 16-grid models so an off-grid bbox (e.g. 1025x1024) is blocked instead of failing backend validation. - getRegionalGuidanceWarnings had no Ideogram 4 branch, so a region whose only input is a negative prompt, auto-negative, or reference image looked effective while the graph silently drops it. Warn those inputs are unsupported. - Ideogram4Region.bbox now also rejects inverted boxes (y_min <= y_max, x_min <= x_max). - The advanced-settings badge selector lumped Ideogram 4 into the generic branch, showing stale VAE/clip-skip/CFG-rescale/seamless badges for controls that are hidden for Ideogram. Exclude Ideogram 4 from that branch. - Model bbox as a constrained type (exactly 4 ints, each 0..1000) so the OpenAPI schema advertises minItems/maxItems and item minimum/maximum. Add readiness tests (bbox grid, raster/inpaint blocking, empty-layer allowance, regional-guidance negative/reference-image warnings) and bbox ordering-rejection tests. * fix(ideogram4): reject text encoders with weights left on the meta device _load_text_encoder() builds the encoder under accelerate.init_empty_weights() and previously downgraded missing keys to a warning (both the fp8 path via load_fp8_state_dict(strict=False) and the non-fp8 path). A missing non-tied weight therefore stayed on the meta device, so a bad or mismatched encoder appeared to load and only failed later during device movement or encoding. Add _verify_encoder_fully_materialized(): call tie_weights() to materialize tied weights from their source, then hard-fail if any parameter or buffer remains on the meta device. Wire it into both the fp8 and non-fp8 (incl. bnb-nf4) paths and drop the missing-key warning — genuinely missing non-tied weights are now caught as leftover meta tensors, while tied weights are tolerated. This is a state-based, path-agnostic check. Add tests: passes when fully materialized, raises on a leftover meta tensor from a missing non-tied weight, and tolerates a tied weight resolved by tie_weights(). * chore(ui): prettier formatting for AdvancedSettingsAccordion The Ideogram 4 badge-suppression branch left the wrapped block at its old indentation, failing `pnpm lint:prettier` in frontend-checks. Formatting only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Your Name <you@example.com> Co-authored-by: 4pointoh <97913726+4pointoh@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> | 1 个月前 | |
Feat(model support): ideogram4 support (#9303) * Update to Transformers 5.1.0 * remove extra stuff * chore(deps): compel fork + transformers>=5.9.0 + remove override Switches compel from PyPI 2.1.1 to invoke-ai/compel@main fork which supports transformers 5.x. Bumps transformers floor to 5.9.0. Removes the transformers>=5.1.0 uv override that was only needed to bypass compel 2.1.1's <5.0 constraint. NOTE: compel fork pulls notebook dep (full Jupyter stack); flag to maintainer for cleanup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(z_image): resolve rope_theta from rope_parameters for transformers 5.x transformers 5.x no longer exposes rope_theta as a top-level attribute on Qwen3Config; the value is stored in the rope_parameters (and rope_scaling) dict instead. Read it from there with a getattr fallback so the inv_freq buffer is computed from the configured base (1e6 / 256) instead of raising AttributeError. Applies to both the safetensors and GGUF Qwen3 encoder paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(model_manager): replace removed hf_hub get_token_permission with whoami huggingface_hub 1.x removed get_token_permission(). HFTokenHelper.get_status() now validates the token via whoami(), which returns user info for a valid token and raises HfHubHTTPError for an invalid one. Preserves the original three-way status: VALID on success, INVALID on HfHubHTTPError (e.g. 401), UNKNOWN on any other error (e.g. network failure). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): regenerate uv.lock after upstream merge Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sd3): resolve merge conflict marker, drop T5TokenizerFast The upstream merge left an unresolved conflict marker in _t5_encode and reintroduced T5TokenizerFast. Keep our v5 assertion (T5Tokenizer only) plus upstream's new t5_device logic, and drop the now-dead T5TokenizerFast monkeypatch in the test (the name no longer exists in the module). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: ruff fixes on merge-resolved files - flux_text_encoder.py: drop unused typing.Union (F401) left by v5 import merge - huggingface.py: ruff format (wrap append(SimpleNamespace(...))) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): pin transformers <5.6 (diffusers single-file CLIP incompat) transformers 5.6 flattened CLIPTextModel (removed the self.text_model wrapper, hoisted embeddings/encoder/final_layer_norm to the top level). diffusers' single-file checkpoint loader (create_diffusers_clip_model_from_ldm) still assumes the nested layout, so loading SD1.5 .safetensors checkpoints fails on 5.6+ with 'CLIPTextModel object has no attribute text_model' and, once that read is shimmed, 'Cannot copy out of meta tensor' (weights never populate the flattened model). Pin to >=5.5,<5.6 (last pre-flattening release) which keeps both the single-file and from_pretrained paths working. The invoke-ai/compel fork accepts any 5.x. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * @ chore(deps): replace compel fork with official compel 2.4.0 compel 2.4.0 (released 2026-05-30) merges the transformers-5 support that the invoke-ai fork carried (both descend from upstream PR #129), plus the maintainer-reviewed padding rework and added diffusers/T5 smoke coverage. Switch from the git fork to the PyPI release. - pyproject: compel git+main -> compel>=2.4.0,<3 - uv.lock: compel 2.3.1 (git 8f404b45) -> 2.4.0 (pypi) - transformers stays 5.5.4 (satisfies compel >=5,<6 and our <5.6 pin) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> @ * feat(ideogram4): backend + model-manager registration for Ideogram 4 Vendor the Apache-2.0 Ideogram 4 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler, nf4/fp8 quant loading) into invokeai/backend/ideogram4/, plus InvokeAI glue (Qwen3-VL text encoding, packed-input build, dual-branch Euler denoise loop). Register the model: BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config (detected via the Ideogram4Pipeline class name in model_index.json), and the Ideogram4DiffusersModel loader that loads both transformers as one Ideogram4TransformerPair submodel plus the Qwen3-VL encoder and VAE. Text-to-image only. * feat(ideogram4): Ideogram 4 backend — model manager, invocations, nf4 loading End-to-end text-to-image backend for Ideogram 4, validated through the real session runner. Vendors the Apache-2.0 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler) into invokeai/backend/ideogram4/ with InvokeAI glue. Registers BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config, and the Ideogram4DiffusersModel loader (two transformers as one Ideogram4TransformerPair; Qwen3-VL encoder + VAE). Both transformers and the encoder load via InvokeLinearNF4 so they work with the partial-load cache. Adds Ideogram4ConditioningInfo/Field/Output and the model_loader/text_encoder/denoise/l2i invocations. Text-to-image only. * feat(ideogram4): frontend — Regions→JSON prompt, graph builder, UI Wires Ideogram 4 into the canvas/generate UI. buildIdeogram4Prompt assembles the structured JSON caption from the global prompt + Canvas Regional Guidance layers (each region → an obj element with a 0–1000 bbox + desc), with raw-JSON passthrough and a plain-text fallback when there are no regions. Adds buildIdeogram4Graph (text-to-image only, no negative prompt) and the enqueue switch. Structured captions use a static string node + a decoy positive-prompt node so the linear batch can't clobber the assembled JSON; plain text uses the real node so dynamic prompts/batching still work. Registers the 'ideogram-4' base (enums, color, names, model picker, grid size 16), a sampler-preset param (V4_QUALITY_48/V4_DEFAULT_20/V4_TURBO_12) replacing the steps/CFG controls, ParamIdeogram4SamplerPreset, and metadata recall. Regenerates schema.ts. * feat(ideogram4): advanced sampler overrides + color palette Advanced accordion now shows only Ideogram 4-relevant controls. Adds optional overrides of the sampler preset — steps, guidance scale (overrides the main gw, preserves the preset's polish tail), and schedule shift (mu) — plus a color palette editor that injects style_description.color_palette into the auto-built JSON caption (uppercase #RRGGBB, max 16, ignored for raw-JSON prompts). All are nullable (null = use preset), recallable from metadata, and the irrelevant controls (VAE, CLIP skip, CFG rescale, seamless, color compensation) are hidden for Ideogram 4. Backend denoise gains steps/guidance_scale/mu fields; schema.ts regenerated. * Use existing keys + fix select size * Update Readme * feat(ideogram4): add Ideogram 4 to starter models with non-commercial license hint - Implement the weight-only fp8 text-encoder load path (was NotImplementedError); validated against the real fp8 build + add CPU unit tests for the fp8 mechanism - Show Ideogram 4 handlers in the Recall Parameters tab - Recall the assembled JSON caption back into the positive prompt - Translate the metadata "Auto" values - Document Ideogram 4 (install/license, regional-guidance JSON prompting, presets) - Add Ideogram 4 nf4 (CUDA) + fp8 (any device) starter models and bundle - Surface the FLUX-style Non-Commercial License popover for Ideogram 4 models - Note the gated HuggingFace license requirement in the model descriptions * Chore Ruff * Chore Ruff * Chore OpenApi * Chore Knit * fix(deps): regenerate uv.lock to remove duplicate packages from bad merge * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * Fix: address ideogram4 review (strict load, 1-step guidance, i18n, runtime caption) - Non-fp8 Ideogram 4 text-encoder load now validates the state dict: unexpected keys raise, missing keys warn (mirrors the fp8 helper) instead of silently accepting a partial load. - Guidance schedule: cap the polish tail at num_steps-1 so at least one main step always remains (the guidance_scale override was silently dropped at num_steps=1), and require steps >= 2 (backend field + frontend slider/marks). - Localize the Ideogram sampler-preset option labels via t() with the step count interpolated; add the three preset i18n keys. - Assemble the structured JSON caption at generation time in a new ideogram4_caption_builder node (Python port of buildIdeogram4Caption) instead of at graph-build time. The graph now wires the real prompt node -> caption builder -> text encoder and returns it as positivePrompt, so dynamic prompts / prompt batching vary the encoded caption (the decoy that dropped them is removed). The builder's output is wired to a new declared ideogram4_caption metadata field via an edge, so each batched image records its actual caption. Regenerates schema.ts for the new node + metadata field. Adds tests for caption assembly, the guidance schedule, and the graph wiring. * feat(ideogram4): step previews + document the model's built-in safety filter - Emit a low-res progress preview each denoise step so the forming image is visible during generation, like the other denoise nodes. Ideogram uses a FLUX.2-style 32-channel VAE, so the packed latent is unpatchified/denormalized (get_latent_norm) and run through the FLUX.2 latent->RGB factors — no full VAE decode per step. The denoise loop now hands the callback the packed grid latent. - Document Ideogram 4's built-in content safety filter in models.mdx: it lives in the model weights (not Invoke's NSFW checker, can't be disabled from Invoke) and false-positives on benign prompts; structured JSON prompts trip it less. * feat(ideogram4): avoid safety-filter false-positives + step previews + caption visibility The main fix: Ideogram 4's built-in safety filter (baked into the model weights) false-positives and returns an "Image blocked by safety filter" placeholder for "degenerate" captions — empirically, an empty `compositional_deconstruction.elements` list, or a single full-frame [0,0,1000,1000] element whose desc just repeats the high_level_description. Our assembly produced empty elements whenever the user drew no regions, so plain prompts were blocked. - Caption assembly (build_ideogram4_caption): - Always emit a structured JSON caption; never bare plain text (the filter false-positives far more on plain text). Raw-JSON pastes still pass through. - When there are no regions, synthesize one default element describing the whole scene from the prompt with a *partial* (non-full-frame) bbox [100,100,900,900]. This never yields an empty/degenerate elements list. Verified end-to-end against the model: the previously-blocked "golden retriever on a skateboard" now renders. - Metadata: always wire the caption builder's output to the ideogram4_caption metadata field, so the viewer's "Structured Caption" row shows the exact JSON that was encoded (not just the raw prompt) for every generation. - Denoise: emit a low-res progress preview each step (unpatchify + FLUX.2 latent->RGB factors, since Ideogram uses a FLUX.2-style 32-channel VAE) so the forming image is visible during generation, like the other denoise nodes. - Docs: document the model's built-in safety filter in models.mdx (it's not Invoke's NSFW checker, can't be disabled from Invoke, and false-positives). Updates the caption/graph tests accordingly (also fixes latent tsc errors in the graph-builder test's core_metadata / ideogram4_caption comparisons). * Chore typegen + openapi + Ruff * Fix Knit * fix(ideogram4): enforce steps>=2 client-side and validate region bbox Address review on the Ideogram 4 PR: - The backend denoise node requires steps >= 2, but the client still accepted ideogram4_steps = 1 in three places, letting a recalled or rehydrated value build a graph that violates the backend schema. Tighten the zod schema to min(2) with `.catch(null)` (a stale/out-of-range value normalizes to null = use the preset instead of failing the whole persisted slice), normalize dispatched values through the schema in setIdeogram4Steps, and refuse an out-of-range value in the ideogram4_steps metadata recall parser. The slider was already min=2. - Ideogram4Region.bbox was an unconstrained Optional[list[int]], so a workflow/API caller could pass a wrong-length or out-of-range box that the caption builder serialized verbatim into the structured prompt. Add a field validator requiring exactly four coordinates, each in 0..1000. Add tests for both: the region bbox contract (valid/None accepted; short, long, negative, and >1000 rejected) and the ideogram4Steps normalization (valid kept, null kept, stale 1 normalized to null on both dispatch and rehydrate). * fix(ideogram4): block unsupported canvas modes/bbox, warn dropped region inputs, constrain bbox Address the latest review on the Ideogram 4 PR: - Canvas readiness allowed unsupported generation modes: Ideogram 4 is txt2img-only (buildIdeogram4Graph asserts it), but a raster layer or inpaint mask makes the compositor pick img2img/outpaint/inpaint, failing only at graph build. Warn in getRasterLayerWarnings/getInpaintMaskWarnings for Ideogram 4 (these already flow into canvas readiness reasons), blocking enqueue up front. - Canvas readiness had no Ideogram 4 bbox check; the backend requires multiples of 16. Add the 16-grid check mirroring the other 16-grid models so an off-grid bbox (e.g. 1025x1024) is blocked instead of failing backend validation. - getRegionalGuidanceWarnings had no Ideogram 4 branch, so a region whose only input is a negative prompt, auto-negative, or reference image looked effective while the graph silently drops it. Warn those inputs are unsupported. - Ideogram4Region.bbox now also rejects inverted boxes (y_min <= y_max, x_min <= x_max). - The advanced-settings badge selector lumped Ideogram 4 into the generic branch, showing stale VAE/clip-skip/CFG-rescale/seamless badges for controls that are hidden for Ideogram. Exclude Ideogram 4 from that branch. - Model bbox as a constrained type (exactly 4 ints, each 0..1000) so the OpenAPI schema advertises minItems/maxItems and item minimum/maximum. Add readiness tests (bbox grid, raster/inpaint blocking, empty-layer allowance, regional-guidance negative/reference-image warnings) and bbox ordering-rejection tests. * fix(ideogram4): reject text encoders with weights left on the meta device _load_text_encoder() builds the encoder under accelerate.init_empty_weights() and previously downgraded missing keys to a warning (both the fp8 path via load_fp8_state_dict(strict=False) and the non-fp8 path). A missing non-tied weight therefore stayed on the meta device, so a bad or mismatched encoder appeared to load and only failed later during device movement or encoding. Add _verify_encoder_fully_materialized(): call tie_weights() to materialize tied weights from their source, then hard-fail if any parameter or buffer remains on the meta device. Wire it into both the fp8 and non-fp8 (incl. bnb-nf4) paths and drop the missing-key warning — genuinely missing non-tied weights are now caught as leftover meta tensors, while tied weights are tolerated. This is a state-based, path-agnostic check. Add tests: passes when fully materialized, raises on a leftover meta tensor from a missing non-tied weight, and tolerates a tied weight resolved by tie_weights(). * chore(ui): prettier formatting for AdvancedSettingsAccordion The Ideogram 4 badge-suppression branch left the wrapped block at its old indentation, failing `pnpm lint:prettier` in frontend-checks. Formatting only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Your Name <you@example.com> Co-authored-by: 4pointoh <97913726+4pointoh@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> | 1 个月前 | |
Feat(model support): ideogram4 support (#9303) * Update to Transformers 5.1.0 * remove extra stuff * chore(deps): compel fork + transformers>=5.9.0 + remove override Switches compel from PyPI 2.1.1 to invoke-ai/compel@main fork which supports transformers 5.x. Bumps transformers floor to 5.9.0. Removes the transformers>=5.1.0 uv override that was only needed to bypass compel 2.1.1's <5.0 constraint. NOTE: compel fork pulls notebook dep (full Jupyter stack); flag to maintainer for cleanup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(z_image): resolve rope_theta from rope_parameters for transformers 5.x transformers 5.x no longer exposes rope_theta as a top-level attribute on Qwen3Config; the value is stored in the rope_parameters (and rope_scaling) dict instead. Read it from there with a getattr fallback so the inv_freq buffer is computed from the configured base (1e6 / 256) instead of raising AttributeError. Applies to both the safetensors and GGUF Qwen3 encoder paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(model_manager): replace removed hf_hub get_token_permission with whoami huggingface_hub 1.x removed get_token_permission(). HFTokenHelper.get_status() now validates the token via whoami(), which returns user info for a valid token and raises HfHubHTTPError for an invalid one. Preserves the original three-way status: VALID on success, INVALID on HfHubHTTPError (e.g. 401), UNKNOWN on any other error (e.g. network failure). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): regenerate uv.lock after upstream merge Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sd3): resolve merge conflict marker, drop T5TokenizerFast The upstream merge left an unresolved conflict marker in _t5_encode and reintroduced T5TokenizerFast. Keep our v5 assertion (T5Tokenizer only) plus upstream's new t5_device logic, and drop the now-dead T5TokenizerFast monkeypatch in the test (the name no longer exists in the module). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: ruff fixes on merge-resolved files - flux_text_encoder.py: drop unused typing.Union (F401) left by v5 import merge - huggingface.py: ruff format (wrap append(SimpleNamespace(...))) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): pin transformers <5.6 (diffusers single-file CLIP incompat) transformers 5.6 flattened CLIPTextModel (removed the self.text_model wrapper, hoisted embeddings/encoder/final_layer_norm to the top level). diffusers' single-file checkpoint loader (create_diffusers_clip_model_from_ldm) still assumes the nested layout, so loading SD1.5 .safetensors checkpoints fails on 5.6+ with 'CLIPTextModel object has no attribute text_model' and, once that read is shimmed, 'Cannot copy out of meta tensor' (weights never populate the flattened model). Pin to >=5.5,<5.6 (last pre-flattening release) which keeps both the single-file and from_pretrained paths working. The invoke-ai/compel fork accepts any 5.x. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * @ chore(deps): replace compel fork with official compel 2.4.0 compel 2.4.0 (released 2026-05-30) merges the transformers-5 support that the invoke-ai fork carried (both descend from upstream PR #129), plus the maintainer-reviewed padding rework and added diffusers/T5 smoke coverage. Switch from the git fork to the PyPI release. - pyproject: compel git+main -> compel>=2.4.0,<3 - uv.lock: compel 2.3.1 (git 8f404b45) -> 2.4.0 (pypi) - transformers stays 5.5.4 (satisfies compel >=5,<6 and our <5.6 pin) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> @ * feat(ideogram4): backend + model-manager registration for Ideogram 4 Vendor the Apache-2.0 Ideogram 4 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler, nf4/fp8 quant loading) into invokeai/backend/ideogram4/, plus InvokeAI glue (Qwen3-VL text encoding, packed-input build, dual-branch Euler denoise loop). Register the model: BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config (detected via the Ideogram4Pipeline class name in model_index.json), and the Ideogram4DiffusersModel loader that loads both transformers as one Ideogram4TransformerPair submodel plus the Qwen3-VL encoder and VAE. Text-to-image only. * feat(ideogram4): Ideogram 4 backend — model manager, invocations, nf4 loading End-to-end text-to-image backend for Ideogram 4, validated through the real session runner. Vendors the Apache-2.0 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler) into invokeai/backend/ideogram4/ with InvokeAI glue. Registers BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config, and the Ideogram4DiffusersModel loader (two transformers as one Ideogram4TransformerPair; Qwen3-VL encoder + VAE). Both transformers and the encoder load via InvokeLinearNF4 so they work with the partial-load cache. Adds Ideogram4ConditioningInfo/Field/Output and the model_loader/text_encoder/denoise/l2i invocations. Text-to-image only. * feat(ideogram4): frontend — Regions→JSON prompt, graph builder, UI Wires Ideogram 4 into the canvas/generate UI. buildIdeogram4Prompt assembles the structured JSON caption from the global prompt + Canvas Regional Guidance layers (each region → an obj element with a 0–1000 bbox + desc), with raw-JSON passthrough and a plain-text fallback when there are no regions. Adds buildIdeogram4Graph (text-to-image only, no negative prompt) and the enqueue switch. Structured captions use a static string node + a decoy positive-prompt node so the linear batch can't clobber the assembled JSON; plain text uses the real node so dynamic prompts/batching still work. Registers the 'ideogram-4' base (enums, color, names, model picker, grid size 16), a sampler-preset param (V4_QUALITY_48/V4_DEFAULT_20/V4_TURBO_12) replacing the steps/CFG controls, ParamIdeogram4SamplerPreset, and metadata recall. Regenerates schema.ts. * feat(ideogram4): advanced sampler overrides + color palette Advanced accordion now shows only Ideogram 4-relevant controls. Adds optional overrides of the sampler preset — steps, guidance scale (overrides the main gw, preserves the preset's polish tail), and schedule shift (mu) — plus a color palette editor that injects style_description.color_palette into the auto-built JSON caption (uppercase #RRGGBB, max 16, ignored for raw-JSON prompts). All are nullable (null = use preset), recallable from metadata, and the irrelevant controls (VAE, CLIP skip, CFG rescale, seamless, color compensation) are hidden for Ideogram 4. Backend denoise gains steps/guidance_scale/mu fields; schema.ts regenerated. * Use existing keys + fix select size * Update Readme * feat(ideogram4): add Ideogram 4 to starter models with non-commercial license hint - Implement the weight-only fp8 text-encoder load path (was NotImplementedError); validated against the real fp8 build + add CPU unit tests for the fp8 mechanism - Show Ideogram 4 handlers in the Recall Parameters tab - Recall the assembled JSON caption back into the positive prompt - Translate the metadata "Auto" values - Document Ideogram 4 (install/license, regional-guidance JSON prompting, presets) - Add Ideogram 4 nf4 (CUDA) + fp8 (any device) starter models and bundle - Surface the FLUX-style Non-Commercial License popover for Ideogram 4 models - Note the gated HuggingFace license requirement in the model descriptions * Chore Ruff * Chore Ruff * Chore OpenApi * Chore Knit * fix(deps): regenerate uv.lock to remove duplicate packages from bad merge * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * Fix: address ideogram4 review (strict load, 1-step guidance, i18n, runtime caption) - Non-fp8 Ideogram 4 text-encoder load now validates the state dict: unexpected keys raise, missing keys warn (mirrors the fp8 helper) instead of silently accepting a partial load. - Guidance schedule: cap the polish tail at num_steps-1 so at least one main step always remains (the guidance_scale override was silently dropped at num_steps=1), and require steps >= 2 (backend field + frontend slider/marks). - Localize the Ideogram sampler-preset option labels via t() with the step count interpolated; add the three preset i18n keys. - Assemble the structured JSON caption at generation time in a new ideogram4_caption_builder node (Python port of buildIdeogram4Caption) instead of at graph-build time. The graph now wires the real prompt node -> caption builder -> text encoder and returns it as positivePrompt, so dynamic prompts / prompt batching vary the encoded caption (the decoy that dropped them is removed). The builder's output is wired to a new declared ideogram4_caption metadata field via an edge, so each batched image records its actual caption. Regenerates schema.ts for the new node + metadata field. Adds tests for caption assembly, the guidance schedule, and the graph wiring. * feat(ideogram4): step previews + document the model's built-in safety filter - Emit a low-res progress preview each denoise step so the forming image is visible during generation, like the other denoise nodes. Ideogram uses a FLUX.2-style 32-channel VAE, so the packed latent is unpatchified/denormalized (get_latent_norm) and run through the FLUX.2 latent->RGB factors — no full VAE decode per step. The denoise loop now hands the callback the packed grid latent. - Document Ideogram 4's built-in content safety filter in models.mdx: it lives in the model weights (not Invoke's NSFW checker, can't be disabled from Invoke) and false-positives on benign prompts; structured JSON prompts trip it less. * feat(ideogram4): avoid safety-filter false-positives + step previews + caption visibility The main fix: Ideogram 4's built-in safety filter (baked into the model weights) false-positives and returns an "Image blocked by safety filter" placeholder for "degenerate" captions — empirically, an empty `compositional_deconstruction.elements` list, or a single full-frame [0,0,1000,1000] element whose desc just repeats the high_level_description. Our assembly produced empty elements whenever the user drew no regions, so plain prompts were blocked. - Caption assembly (build_ideogram4_caption): - Always emit a structured JSON caption; never bare plain text (the filter false-positives far more on plain text). Raw-JSON pastes still pass through. - When there are no regions, synthesize one default element describing the whole scene from the prompt with a *partial* (non-full-frame) bbox [100,100,900,900]. This never yields an empty/degenerate elements list. Verified end-to-end against the model: the previously-blocked "golden retriever on a skateboard" now renders. - Metadata: always wire the caption builder's output to the ideogram4_caption metadata field, so the viewer's "Structured Caption" row shows the exact JSON that was encoded (not just the raw prompt) for every generation. - Denoise: emit a low-res progress preview each step (unpatchify + FLUX.2 latent->RGB factors, since Ideogram uses a FLUX.2-style 32-channel VAE) so the forming image is visible during generation, like the other denoise nodes. - Docs: document the model's built-in safety filter in models.mdx (it's not Invoke's NSFW checker, can't be disabled from Invoke, and false-positives). Updates the caption/graph tests accordingly (also fixes latent tsc errors in the graph-builder test's core_metadata / ideogram4_caption comparisons). * Chore typegen + openapi + Ruff * Fix Knit * fix(ideogram4): enforce steps>=2 client-side and validate region bbox Address review on the Ideogram 4 PR: - The backend denoise node requires steps >= 2, but the client still accepted ideogram4_steps = 1 in three places, letting a recalled or rehydrated value build a graph that violates the backend schema. Tighten the zod schema to min(2) with `.catch(null)` (a stale/out-of-range value normalizes to null = use the preset instead of failing the whole persisted slice), normalize dispatched values through the schema in setIdeogram4Steps, and refuse an out-of-range value in the ideogram4_steps metadata recall parser. The slider was already min=2. - Ideogram4Region.bbox was an unconstrained Optional[list[int]], so a workflow/API caller could pass a wrong-length or out-of-range box that the caption builder serialized verbatim into the structured prompt. Add a field validator requiring exactly four coordinates, each in 0..1000. Add tests for both: the region bbox contract (valid/None accepted; short, long, negative, and >1000 rejected) and the ideogram4Steps normalization (valid kept, null kept, stale 1 normalized to null on both dispatch and rehydrate). * fix(ideogram4): block unsupported canvas modes/bbox, warn dropped region inputs, constrain bbox Address the latest review on the Ideogram 4 PR: - Canvas readiness allowed unsupported generation modes: Ideogram 4 is txt2img-only (buildIdeogram4Graph asserts it), but a raster layer or inpaint mask makes the compositor pick img2img/outpaint/inpaint, failing only at graph build. Warn in getRasterLayerWarnings/getInpaintMaskWarnings for Ideogram 4 (these already flow into canvas readiness reasons), blocking enqueue up front. - Canvas readiness had no Ideogram 4 bbox check; the backend requires multiples of 16. Add the 16-grid check mirroring the other 16-grid models so an off-grid bbox (e.g. 1025x1024) is blocked instead of failing backend validation. - getRegionalGuidanceWarnings had no Ideogram 4 branch, so a region whose only input is a negative prompt, auto-negative, or reference image looked effective while the graph silently drops it. Warn those inputs are unsupported. - Ideogram4Region.bbox now also rejects inverted boxes (y_min <= y_max, x_min <= x_max). - The advanced-settings badge selector lumped Ideogram 4 into the generic branch, showing stale VAE/clip-skip/CFG-rescale/seamless badges for controls that are hidden for Ideogram. Exclude Ideogram 4 from that branch. - Model bbox as a constrained type (exactly 4 ints, each 0..1000) so the OpenAPI schema advertises minItems/maxItems and item minimum/maximum. Add readiness tests (bbox grid, raster/inpaint blocking, empty-layer allowance, regional-guidance negative/reference-image warnings) and bbox ordering-rejection tests. * fix(ideogram4): reject text encoders with weights left on the meta device _load_text_encoder() builds the encoder under accelerate.init_empty_weights() and previously downgraded missing keys to a warning (both the fp8 path via load_fp8_state_dict(strict=False) and the non-fp8 path). A missing non-tied weight therefore stayed on the meta device, so a bad or mismatched encoder appeared to load and only failed later during device movement or encoding. Add _verify_encoder_fully_materialized(): call tie_weights() to materialize tied weights from their source, then hard-fail if any parameter or buffer remains on the meta device. Wire it into both the fp8 and non-fp8 (incl. bnb-nf4) paths and drop the missing-key warning — genuinely missing non-tied weights are now caught as leftover meta tensors, while tied weights are tolerated. This is a state-based, path-agnostic check. Add tests: passes when fully materialized, raises on a leftover meta tensor from a missing non-tied weight, and tolerates a tied weight resolved by tie_weights(). * chore(ui): prettier formatting for AdvancedSettingsAccordion The Ideogram 4 badge-suppression branch left the wrapped block at its old indentation, failing `pnpm lint:prettier` in frontend-checks. Formatting only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Your Name <you@example.com> Co-authored-by: 4pointoh <97913726+4pointoh@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> | 1 个月前 | |
Feat(model support): ideogram4 support (#9303) * Update to Transformers 5.1.0 * remove extra stuff * chore(deps): compel fork + transformers>=5.9.0 + remove override Switches compel from PyPI 2.1.1 to invoke-ai/compel@main fork which supports transformers 5.x. Bumps transformers floor to 5.9.0. Removes the transformers>=5.1.0 uv override that was only needed to bypass compel 2.1.1's <5.0 constraint. NOTE: compel fork pulls notebook dep (full Jupyter stack); flag to maintainer for cleanup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(z_image): resolve rope_theta from rope_parameters for transformers 5.x transformers 5.x no longer exposes rope_theta as a top-level attribute on Qwen3Config; the value is stored in the rope_parameters (and rope_scaling) dict instead. Read it from there with a getattr fallback so the inv_freq buffer is computed from the configured base (1e6 / 256) instead of raising AttributeError. Applies to both the safetensors and GGUF Qwen3 encoder paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(model_manager): replace removed hf_hub get_token_permission with whoami huggingface_hub 1.x removed get_token_permission(). HFTokenHelper.get_status() now validates the token via whoami(), which returns user info for a valid token and raises HfHubHTTPError for an invalid one. Preserves the original three-way status: VALID on success, INVALID on HfHubHTTPError (e.g. 401), UNKNOWN on any other error (e.g. network failure). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): regenerate uv.lock after upstream merge Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sd3): resolve merge conflict marker, drop T5TokenizerFast The upstream merge left an unresolved conflict marker in _t5_encode and reintroduced T5TokenizerFast. Keep our v5 assertion (T5Tokenizer only) plus upstream's new t5_device logic, and drop the now-dead T5TokenizerFast monkeypatch in the test (the name no longer exists in the module). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: ruff fixes on merge-resolved files - flux_text_encoder.py: drop unused typing.Union (F401) left by v5 import merge - huggingface.py: ruff format (wrap append(SimpleNamespace(...))) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): pin transformers <5.6 (diffusers single-file CLIP incompat) transformers 5.6 flattened CLIPTextModel (removed the self.text_model wrapper, hoisted embeddings/encoder/final_layer_norm to the top level). diffusers' single-file checkpoint loader (create_diffusers_clip_model_from_ldm) still assumes the nested layout, so loading SD1.5 .safetensors checkpoints fails on 5.6+ with 'CLIPTextModel object has no attribute text_model' and, once that read is shimmed, 'Cannot copy out of meta tensor' (weights never populate the flattened model). Pin to >=5.5,<5.6 (last pre-flattening release) which keeps both the single-file and from_pretrained paths working. The invoke-ai/compel fork accepts any 5.x. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * @ chore(deps): replace compel fork with official compel 2.4.0 compel 2.4.0 (released 2026-05-30) merges the transformers-5 support that the invoke-ai fork carried (both descend from upstream PR #129), plus the maintainer-reviewed padding rework and added diffusers/T5 smoke coverage. Switch from the git fork to the PyPI release. - pyproject: compel git+main -> compel>=2.4.0,<3 - uv.lock: compel 2.3.1 (git 8f404b45) -> 2.4.0 (pypi) - transformers stays 5.5.4 (satisfies compel >=5,<6 and our <5.6 pin) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> @ * feat(ideogram4): backend + model-manager registration for Ideogram 4 Vendor the Apache-2.0 Ideogram 4 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler, nf4/fp8 quant loading) into invokeai/backend/ideogram4/, plus InvokeAI glue (Qwen3-VL text encoding, packed-input build, dual-branch Euler denoise loop). Register the model: BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config (detected via the Ideogram4Pipeline class name in model_index.json), and the Ideogram4DiffusersModel loader that loads both transformers as one Ideogram4TransformerPair submodel plus the Qwen3-VL encoder and VAE. Text-to-image only. * feat(ideogram4): Ideogram 4 backend — model manager, invocations, nf4 loading End-to-end text-to-image backend for Ideogram 4, validated through the real session runner. Vendors the Apache-2.0 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler) into invokeai/backend/ideogram4/ with InvokeAI glue. Registers BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config, and the Ideogram4DiffusersModel loader (two transformers as one Ideogram4TransformerPair; Qwen3-VL encoder + VAE). Both transformers and the encoder load via InvokeLinearNF4 so they work with the partial-load cache. Adds Ideogram4ConditioningInfo/Field/Output and the model_loader/text_encoder/denoise/l2i invocations. Text-to-image only. * feat(ideogram4): frontend — Regions→JSON prompt, graph builder, UI Wires Ideogram 4 into the canvas/generate UI. buildIdeogram4Prompt assembles the structured JSON caption from the global prompt + Canvas Regional Guidance layers (each region → an obj element with a 0–1000 bbox + desc), with raw-JSON passthrough and a plain-text fallback when there are no regions. Adds buildIdeogram4Graph (text-to-image only, no negative prompt) and the enqueue switch. Structured captions use a static string node + a decoy positive-prompt node so the linear batch can't clobber the assembled JSON; plain text uses the real node so dynamic prompts/batching still work. Registers the 'ideogram-4' base (enums, color, names, model picker, grid size 16), a sampler-preset param (V4_QUALITY_48/V4_DEFAULT_20/V4_TURBO_12) replacing the steps/CFG controls, ParamIdeogram4SamplerPreset, and metadata recall. Regenerates schema.ts. * feat(ideogram4): advanced sampler overrides + color palette Advanced accordion now shows only Ideogram 4-relevant controls. Adds optional overrides of the sampler preset — steps, guidance scale (overrides the main gw, preserves the preset's polish tail), and schedule shift (mu) — plus a color palette editor that injects style_description.color_palette into the auto-built JSON caption (uppercase #RRGGBB, max 16, ignored for raw-JSON prompts). All are nullable (null = use preset), recallable from metadata, and the irrelevant controls (VAE, CLIP skip, CFG rescale, seamless, color compensation) are hidden for Ideogram 4. Backend denoise gains steps/guidance_scale/mu fields; schema.ts regenerated. * Use existing keys + fix select size * Update Readme * feat(ideogram4): add Ideogram 4 to starter models with non-commercial license hint - Implement the weight-only fp8 text-encoder load path (was NotImplementedError); validated against the real fp8 build + add CPU unit tests for the fp8 mechanism - Show Ideogram 4 handlers in the Recall Parameters tab - Recall the assembled JSON caption back into the positive prompt - Translate the metadata "Auto" values - Document Ideogram 4 (install/license, regional-guidance JSON prompting, presets) - Add Ideogram 4 nf4 (CUDA) + fp8 (any device) starter models and bundle - Surface the FLUX-style Non-Commercial License popover for Ideogram 4 models - Note the gated HuggingFace license requirement in the model descriptions * Chore Ruff * Chore Ruff * Chore OpenApi * Chore Knit * fix(deps): regenerate uv.lock to remove duplicate packages from bad merge * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * Fix: address ideogram4 review (strict load, 1-step guidance, i18n, runtime caption) - Non-fp8 Ideogram 4 text-encoder load now validates the state dict: unexpected keys raise, missing keys warn (mirrors the fp8 helper) instead of silently accepting a partial load. - Guidance schedule: cap the polish tail at num_steps-1 so at least one main step always remains (the guidance_scale override was silently dropped at num_steps=1), and require steps >= 2 (backend field + frontend slider/marks). - Localize the Ideogram sampler-preset option labels via t() with the step count interpolated; add the three preset i18n keys. - Assemble the structured JSON caption at generation time in a new ideogram4_caption_builder node (Python port of buildIdeogram4Caption) instead of at graph-build time. The graph now wires the real prompt node -> caption builder -> text encoder and returns it as positivePrompt, so dynamic prompts / prompt batching vary the encoded caption (the decoy that dropped them is removed). The builder's output is wired to a new declared ideogram4_caption metadata field via an edge, so each batched image records its actual caption. Regenerates schema.ts for the new node + metadata field. Adds tests for caption assembly, the guidance schedule, and the graph wiring. * feat(ideogram4): step previews + document the model's built-in safety filter - Emit a low-res progress preview each denoise step so the forming image is visible during generation, like the other denoise nodes. Ideogram uses a FLUX.2-style 32-channel VAE, so the packed latent is unpatchified/denormalized (get_latent_norm) and run through the FLUX.2 latent->RGB factors — no full VAE decode per step. The denoise loop now hands the callback the packed grid latent. - Document Ideogram 4's built-in content safety filter in models.mdx: it lives in the model weights (not Invoke's NSFW checker, can't be disabled from Invoke) and false-positives on benign prompts; structured JSON prompts trip it less. * feat(ideogram4): avoid safety-filter false-positives + step previews + caption visibility The main fix: Ideogram 4's built-in safety filter (baked into the model weights) false-positives and returns an "Image blocked by safety filter" placeholder for "degenerate" captions — empirically, an empty `compositional_deconstruction.elements` list, or a single full-frame [0,0,1000,1000] element whose desc just repeats the high_level_description. Our assembly produced empty elements whenever the user drew no regions, so plain prompts were blocked. - Caption assembly (build_ideogram4_caption): - Always emit a structured JSON caption; never bare plain text (the filter false-positives far more on plain text). Raw-JSON pastes still pass through. - When there are no regions, synthesize one default element describing the whole scene from the prompt with a *partial* (non-full-frame) bbox [100,100,900,900]. This never yields an empty/degenerate elements list. Verified end-to-end against the model: the previously-blocked "golden retriever on a skateboard" now renders. - Metadata: always wire the caption builder's output to the ideogram4_caption metadata field, so the viewer's "Structured Caption" row shows the exact JSON that was encoded (not just the raw prompt) for every generation. - Denoise: emit a low-res progress preview each step (unpatchify + FLUX.2 latent->RGB factors, since Ideogram uses a FLUX.2-style 32-channel VAE) so the forming image is visible during generation, like the other denoise nodes. - Docs: document the model's built-in safety filter in models.mdx (it's not Invoke's NSFW checker, can't be disabled from Invoke, and false-positives). Updates the caption/graph tests accordingly (also fixes latent tsc errors in the graph-builder test's core_metadata / ideogram4_caption comparisons). * Chore typegen + openapi + Ruff * Fix Knit * fix(ideogram4): enforce steps>=2 client-side and validate region bbox Address review on the Ideogram 4 PR: - The backend denoise node requires steps >= 2, but the client still accepted ideogram4_steps = 1 in three places, letting a recalled or rehydrated value build a graph that violates the backend schema. Tighten the zod schema to min(2) with `.catch(null)` (a stale/out-of-range value normalizes to null = use the preset instead of failing the whole persisted slice), normalize dispatched values through the schema in setIdeogram4Steps, and refuse an out-of-range value in the ideogram4_steps metadata recall parser. The slider was already min=2. - Ideogram4Region.bbox was an unconstrained Optional[list[int]], so a workflow/API caller could pass a wrong-length or out-of-range box that the caption builder serialized verbatim into the structured prompt. Add a field validator requiring exactly four coordinates, each in 0..1000. Add tests for both: the region bbox contract (valid/None accepted; short, long, negative, and >1000 rejected) and the ideogram4Steps normalization (valid kept, null kept, stale 1 normalized to null on both dispatch and rehydrate). * fix(ideogram4): block unsupported canvas modes/bbox, warn dropped region inputs, constrain bbox Address the latest review on the Ideogram 4 PR: - Canvas readiness allowed unsupported generation modes: Ideogram 4 is txt2img-only (buildIdeogram4Graph asserts it), but a raster layer or inpaint mask makes the compositor pick img2img/outpaint/inpaint, failing only at graph build. Warn in getRasterLayerWarnings/getInpaintMaskWarnings for Ideogram 4 (these already flow into canvas readiness reasons), blocking enqueue up front. - Canvas readiness had no Ideogram 4 bbox check; the backend requires multiples of 16. Add the 16-grid check mirroring the other 16-grid models so an off-grid bbox (e.g. 1025x1024) is blocked instead of failing backend validation. - getRegionalGuidanceWarnings had no Ideogram 4 branch, so a region whose only input is a negative prompt, auto-negative, or reference image looked effective while the graph silently drops it. Warn those inputs are unsupported. - Ideogram4Region.bbox now also rejects inverted boxes (y_min <= y_max, x_min <= x_max). - The advanced-settings badge selector lumped Ideogram 4 into the generic branch, showing stale VAE/clip-skip/CFG-rescale/seamless badges for controls that are hidden for Ideogram. Exclude Ideogram 4 from that branch. - Model bbox as a constrained type (exactly 4 ints, each 0..1000) so the OpenAPI schema advertises minItems/maxItems and item minimum/maximum. Add readiness tests (bbox grid, raster/inpaint blocking, empty-layer allowance, regional-guidance negative/reference-image warnings) and bbox ordering-rejection tests. * fix(ideogram4): reject text encoders with weights left on the meta device _load_text_encoder() builds the encoder under accelerate.init_empty_weights() and previously downgraded missing keys to a warning (both the fp8 path via load_fp8_state_dict(strict=False) and the non-fp8 path). A missing non-tied weight therefore stayed on the meta device, so a bad or mismatched encoder appeared to load and only failed later during device movement or encoding. Add _verify_encoder_fully_materialized(): call tie_weights() to materialize tied weights from their source, then hard-fail if any parameter or buffer remains on the meta device. Wire it into both the fp8 and non-fp8 (incl. bnb-nf4) paths and drop the missing-key warning — genuinely missing non-tied weights are now caught as leftover meta tensors, while tied weights are tolerated. This is a state-based, path-agnostic check. Add tests: passes when fully materialized, raises on a leftover meta tensor from a missing non-tied weight, and tolerates a tied weight resolved by tie_weights(). * chore(ui): prettier formatting for AdvancedSettingsAccordion The Ideogram 4 badge-suppression branch left the wrapped block at its old indentation, failing `pnpm lint:prettier` in frontend-checks. Formatting only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Your Name <you@example.com> Co-authored-by: 4pointoh <97913726+4pointoh@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> | 1 个月前 | |
Feat(model support): ideogram4 support (#9303) * Update to Transformers 5.1.0 * remove extra stuff * chore(deps): compel fork + transformers>=5.9.0 + remove override Switches compel from PyPI 2.1.1 to invoke-ai/compel@main fork which supports transformers 5.x. Bumps transformers floor to 5.9.0. Removes the transformers>=5.1.0 uv override that was only needed to bypass compel 2.1.1's <5.0 constraint. NOTE: compel fork pulls notebook dep (full Jupyter stack); flag to maintainer for cleanup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(z_image): resolve rope_theta from rope_parameters for transformers 5.x transformers 5.x no longer exposes rope_theta as a top-level attribute on Qwen3Config; the value is stored in the rope_parameters (and rope_scaling) dict instead. Read it from there with a getattr fallback so the inv_freq buffer is computed from the configured base (1e6 / 256) instead of raising AttributeError. Applies to both the safetensors and GGUF Qwen3 encoder paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(model_manager): replace removed hf_hub get_token_permission with whoami huggingface_hub 1.x removed get_token_permission(). HFTokenHelper.get_status() now validates the token via whoami(), which returns user info for a valid token and raises HfHubHTTPError for an invalid one. Preserves the original three-way status: VALID on success, INVALID on HfHubHTTPError (e.g. 401), UNKNOWN on any other error (e.g. network failure). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): regenerate uv.lock after upstream merge Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sd3): resolve merge conflict marker, drop T5TokenizerFast The upstream merge left an unresolved conflict marker in _t5_encode and reintroduced T5TokenizerFast. Keep our v5 assertion (T5Tokenizer only) plus upstream's new t5_device logic, and drop the now-dead T5TokenizerFast monkeypatch in the test (the name no longer exists in the module). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: ruff fixes on merge-resolved files - flux_text_encoder.py: drop unused typing.Union (F401) left by v5 import merge - huggingface.py: ruff format (wrap append(SimpleNamespace(...))) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): pin transformers <5.6 (diffusers single-file CLIP incompat) transformers 5.6 flattened CLIPTextModel (removed the self.text_model wrapper, hoisted embeddings/encoder/final_layer_norm to the top level). diffusers' single-file checkpoint loader (create_diffusers_clip_model_from_ldm) still assumes the nested layout, so loading SD1.5 .safetensors checkpoints fails on 5.6+ with 'CLIPTextModel object has no attribute text_model' and, once that read is shimmed, 'Cannot copy out of meta tensor' (weights never populate the flattened model). Pin to >=5.5,<5.6 (last pre-flattening release) which keeps both the single-file and from_pretrained paths working. The invoke-ai/compel fork accepts any 5.x. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * @ chore(deps): replace compel fork with official compel 2.4.0 compel 2.4.0 (released 2026-05-30) merges the transformers-5 support that the invoke-ai fork carried (both descend from upstream PR #129), plus the maintainer-reviewed padding rework and added diffusers/T5 smoke coverage. Switch from the git fork to the PyPI release. - pyproject: compel git+main -> compel>=2.4.0,<3 - uv.lock: compel 2.3.1 (git 8f404b45) -> 2.4.0 (pypi) - transformers stays 5.5.4 (satisfies compel >=5,<6 and our <5.6 pin) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> @ * feat(ideogram4): backend + model-manager registration for Ideogram 4 Vendor the Apache-2.0 Ideogram 4 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler, nf4/fp8 quant loading) into invokeai/backend/ideogram4/, plus InvokeAI glue (Qwen3-VL text encoding, packed-input build, dual-branch Euler denoise loop). Register the model: BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config (detected via the Ideogram4Pipeline class name in model_index.json), and the Ideogram4DiffusersModel loader that loads both transformers as one Ideogram4TransformerPair submodel plus the Qwen3-VL encoder and VAE. Text-to-image only. * feat(ideogram4): Ideogram 4 backend — model manager, invocations, nf4 loading End-to-end text-to-image backend for Ideogram 4, validated through the real session runner. Vendors the Apache-2.0 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler) into invokeai/backend/ideogram4/ with InvokeAI glue. Registers BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config, and the Ideogram4DiffusersModel loader (two transformers as one Ideogram4TransformerPair; Qwen3-VL encoder + VAE). Both transformers and the encoder load via InvokeLinearNF4 so they work with the partial-load cache. Adds Ideogram4ConditioningInfo/Field/Output and the model_loader/text_encoder/denoise/l2i invocations. Text-to-image only. * feat(ideogram4): frontend — Regions→JSON prompt, graph builder, UI Wires Ideogram 4 into the canvas/generate UI. buildIdeogram4Prompt assembles the structured JSON caption from the global prompt + Canvas Regional Guidance layers (each region → an obj element with a 0–1000 bbox + desc), with raw-JSON passthrough and a plain-text fallback when there are no regions. Adds buildIdeogram4Graph (text-to-image only, no negative prompt) and the enqueue switch. Structured captions use a static string node + a decoy positive-prompt node so the linear batch can't clobber the assembled JSON; plain text uses the real node so dynamic prompts/batching still work. Registers the 'ideogram-4' base (enums, color, names, model picker, grid size 16), a sampler-preset param (V4_QUALITY_48/V4_DEFAULT_20/V4_TURBO_12) replacing the steps/CFG controls, ParamIdeogram4SamplerPreset, and metadata recall. Regenerates schema.ts. * feat(ideogram4): advanced sampler overrides + color palette Advanced accordion now shows only Ideogram 4-relevant controls. Adds optional overrides of the sampler preset — steps, guidance scale (overrides the main gw, preserves the preset's polish tail), and schedule shift (mu) — plus a color palette editor that injects style_description.color_palette into the auto-built JSON caption (uppercase #RRGGBB, max 16, ignored for raw-JSON prompts). All are nullable (null = use preset), recallable from metadata, and the irrelevant controls (VAE, CLIP skip, CFG rescale, seamless, color compensation) are hidden for Ideogram 4. Backend denoise gains steps/guidance_scale/mu fields; schema.ts regenerated. * Use existing keys + fix select size * Update Readme * feat(ideogram4): add Ideogram 4 to starter models with non-commercial license hint - Implement the weight-only fp8 text-encoder load path (was NotImplementedError); validated against the real fp8 build + add CPU unit tests for the fp8 mechanism - Show Ideogram 4 handlers in the Recall Parameters tab - Recall the assembled JSON caption back into the positive prompt - Translate the metadata "Auto" values - Document Ideogram 4 (install/license, regional-guidance JSON prompting, presets) - Add Ideogram 4 nf4 (CUDA) + fp8 (any device) starter models and bundle - Surface the FLUX-style Non-Commercial License popover for Ideogram 4 models - Note the gated HuggingFace license requirement in the model descriptions * Chore Ruff * Chore Ruff * Chore OpenApi * Chore Knit * fix(deps): regenerate uv.lock to remove duplicate packages from bad merge * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * Fix: address ideogram4 review (strict load, 1-step guidance, i18n, runtime caption) - Non-fp8 Ideogram 4 text-encoder load now validates the state dict: unexpected keys raise, missing keys warn (mirrors the fp8 helper) instead of silently accepting a partial load. - Guidance schedule: cap the polish tail at num_steps-1 so at least one main step always remains (the guidance_scale override was silently dropped at num_steps=1), and require steps >= 2 (backend field + frontend slider/marks). - Localize the Ideogram sampler-preset option labels via t() with the step count interpolated; add the three preset i18n keys. - Assemble the structured JSON caption at generation time in a new ideogram4_caption_builder node (Python port of buildIdeogram4Caption) instead of at graph-build time. The graph now wires the real prompt node -> caption builder -> text encoder and returns it as positivePrompt, so dynamic prompts / prompt batching vary the encoded caption (the decoy that dropped them is removed). The builder's output is wired to a new declared ideogram4_caption metadata field via an edge, so each batched image records its actual caption. Regenerates schema.ts for the new node + metadata field. Adds tests for caption assembly, the guidance schedule, and the graph wiring. * feat(ideogram4): step previews + document the model's built-in safety filter - Emit a low-res progress preview each denoise step so the forming image is visible during generation, like the other denoise nodes. Ideogram uses a FLUX.2-style 32-channel VAE, so the packed latent is unpatchified/denormalized (get_latent_norm) and run through the FLUX.2 latent->RGB factors — no full VAE decode per step. The denoise loop now hands the callback the packed grid latent. - Document Ideogram 4's built-in content safety filter in models.mdx: it lives in the model weights (not Invoke's NSFW checker, can't be disabled from Invoke) and false-positives on benign prompts; structured JSON prompts trip it less. * feat(ideogram4): avoid safety-filter false-positives + step previews + caption visibility The main fix: Ideogram 4's built-in safety filter (baked into the model weights) false-positives and returns an "Image blocked by safety filter" placeholder for "degenerate" captions — empirically, an empty `compositional_deconstruction.elements` list, or a single full-frame [0,0,1000,1000] element whose desc just repeats the high_level_description. Our assembly produced empty elements whenever the user drew no regions, so plain prompts were blocked. - Caption assembly (build_ideogram4_caption): - Always emit a structured JSON caption; never bare plain text (the filter false-positives far more on plain text). Raw-JSON pastes still pass through. - When there are no regions, synthesize one default element describing the whole scene from the prompt with a *partial* (non-full-frame) bbox [100,100,900,900]. This never yields an empty/degenerate elements list. Verified end-to-end against the model: the previously-blocked "golden retriever on a skateboard" now renders. - Metadata: always wire the caption builder's output to the ideogram4_caption metadata field, so the viewer's "Structured Caption" row shows the exact JSON that was encoded (not just the raw prompt) for every generation. - Denoise: emit a low-res progress preview each step (unpatchify + FLUX.2 latent->RGB factors, since Ideogram uses a FLUX.2-style 32-channel VAE) so the forming image is visible during generation, like the other denoise nodes. - Docs: document the model's built-in safety filter in models.mdx (it's not Invoke's NSFW checker, can't be disabled from Invoke, and false-positives). Updates the caption/graph tests accordingly (also fixes latent tsc errors in the graph-builder test's core_metadata / ideogram4_caption comparisons). * Chore typegen + openapi + Ruff * Fix Knit * fix(ideogram4): enforce steps>=2 client-side and validate region bbox Address review on the Ideogram 4 PR: - The backend denoise node requires steps >= 2, but the client still accepted ideogram4_steps = 1 in three places, letting a recalled or rehydrated value build a graph that violates the backend schema. Tighten the zod schema to min(2) with `.catch(null)` (a stale/out-of-range value normalizes to null = use the preset instead of failing the whole persisted slice), normalize dispatched values through the schema in setIdeogram4Steps, and refuse an out-of-range value in the ideogram4_steps metadata recall parser. The slider was already min=2. - Ideogram4Region.bbox was an unconstrained Optional[list[int]], so a workflow/API caller could pass a wrong-length or out-of-range box that the caption builder serialized verbatim into the structured prompt. Add a field validator requiring exactly four coordinates, each in 0..1000. Add tests for both: the region bbox contract (valid/None accepted; short, long, negative, and >1000 rejected) and the ideogram4Steps normalization (valid kept, null kept, stale 1 normalized to null on both dispatch and rehydrate). * fix(ideogram4): block unsupported canvas modes/bbox, warn dropped region inputs, constrain bbox Address the latest review on the Ideogram 4 PR: - Canvas readiness allowed unsupported generation modes: Ideogram 4 is txt2img-only (buildIdeogram4Graph asserts it), but a raster layer or inpaint mask makes the compositor pick img2img/outpaint/inpaint, failing only at graph build. Warn in getRasterLayerWarnings/getInpaintMaskWarnings for Ideogram 4 (these already flow into canvas readiness reasons), blocking enqueue up front. - Canvas readiness had no Ideogram 4 bbox check; the backend requires multiples of 16. Add the 16-grid check mirroring the other 16-grid models so an off-grid bbox (e.g. 1025x1024) is blocked instead of failing backend validation. - getRegionalGuidanceWarnings had no Ideogram 4 branch, so a region whose only input is a negative prompt, auto-negative, or reference image looked effective while the graph silently drops it. Warn those inputs are unsupported. - Ideogram4Region.bbox now also rejects inverted boxes (y_min <= y_max, x_min <= x_max). - The advanced-settings badge selector lumped Ideogram 4 into the generic branch, showing stale VAE/clip-skip/CFG-rescale/seamless badges for controls that are hidden for Ideogram. Exclude Ideogram 4 from that branch. - Model bbox as a constrained type (exactly 4 ints, each 0..1000) so the OpenAPI schema advertises minItems/maxItems and item minimum/maximum. Add readiness tests (bbox grid, raster/inpaint blocking, empty-layer allowance, regional-guidance negative/reference-image warnings) and bbox ordering-rejection tests. * fix(ideogram4): reject text encoders with weights left on the meta device _load_text_encoder() builds the encoder under accelerate.init_empty_weights() and previously downgraded missing keys to a warning (both the fp8 path via load_fp8_state_dict(strict=False) and the non-fp8 path). A missing non-tied weight therefore stayed on the meta device, so a bad or mismatched encoder appeared to load and only failed later during device movement or encoding. Add _verify_encoder_fully_materialized(): call tie_weights() to materialize tied weights from their source, then hard-fail if any parameter or buffer remains on the meta device. Wire it into both the fp8 and non-fp8 (incl. bnb-nf4) paths and drop the missing-key warning — genuinely missing non-tied weights are now caught as leftover meta tensors, while tied weights are tolerated. This is a state-based, path-agnostic check. Add tests: passes when fully materialized, raises on a leftover meta tensor from a missing non-tied weight, and tolerates a tied weight resolved by tie_weights(). * chore(ui): prettier formatting for AdvancedSettingsAccordion The Ideogram 4 badge-suppression branch left the wrapped block at its old indentation, failing `pnpm lint:prettier` in frontend-checks. Formatting only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Your Name <you@example.com> Co-authored-by: 4pointoh <97913726+4pointoh@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> | 1 个月前 | |
Feat(model support): ideogram4 support (#9303) * Update to Transformers 5.1.0 * remove extra stuff * chore(deps): compel fork + transformers>=5.9.0 + remove override Switches compel from PyPI 2.1.1 to invoke-ai/compel@main fork which supports transformers 5.x. Bumps transformers floor to 5.9.0. Removes the transformers>=5.1.0 uv override that was only needed to bypass compel 2.1.1's <5.0 constraint. NOTE: compel fork pulls notebook dep (full Jupyter stack); flag to maintainer for cleanup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(z_image): resolve rope_theta from rope_parameters for transformers 5.x transformers 5.x no longer exposes rope_theta as a top-level attribute on Qwen3Config; the value is stored in the rope_parameters (and rope_scaling) dict instead. Read it from there with a getattr fallback so the inv_freq buffer is computed from the configured base (1e6 / 256) instead of raising AttributeError. Applies to both the safetensors and GGUF Qwen3 encoder paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(model_manager): replace removed hf_hub get_token_permission with whoami huggingface_hub 1.x removed get_token_permission(). HFTokenHelper.get_status() now validates the token via whoami(), which returns user info for a valid token and raises HfHubHTTPError for an invalid one. Preserves the original three-way status: VALID on success, INVALID on HfHubHTTPError (e.g. 401), UNKNOWN on any other error (e.g. network failure). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): regenerate uv.lock after upstream merge Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sd3): resolve merge conflict marker, drop T5TokenizerFast The upstream merge left an unresolved conflict marker in _t5_encode and reintroduced T5TokenizerFast. Keep our v5 assertion (T5Tokenizer only) plus upstream's new t5_device logic, and drop the now-dead T5TokenizerFast monkeypatch in the test (the name no longer exists in the module). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: ruff fixes on merge-resolved files - flux_text_encoder.py: drop unused typing.Union (F401) left by v5 import merge - huggingface.py: ruff format (wrap append(SimpleNamespace(...))) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): pin transformers <5.6 (diffusers single-file CLIP incompat) transformers 5.6 flattened CLIPTextModel (removed the self.text_model wrapper, hoisted embeddings/encoder/final_layer_norm to the top level). diffusers' single-file checkpoint loader (create_diffusers_clip_model_from_ldm) still assumes the nested layout, so loading SD1.5 .safetensors checkpoints fails on 5.6+ with 'CLIPTextModel object has no attribute text_model' and, once that read is shimmed, 'Cannot copy out of meta tensor' (weights never populate the flattened model). Pin to >=5.5,<5.6 (last pre-flattening release) which keeps both the single-file and from_pretrained paths working. The invoke-ai/compel fork accepts any 5.x. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * @ chore(deps): replace compel fork with official compel 2.4.0 compel 2.4.0 (released 2026-05-30) merges the transformers-5 support that the invoke-ai fork carried (both descend from upstream PR #129), plus the maintainer-reviewed padding rework and added diffusers/T5 smoke coverage. Switch from the git fork to the PyPI release. - pyproject: compel git+main -> compel>=2.4.0,<3 - uv.lock: compel 2.3.1 (git 8f404b45) -> 2.4.0 (pypi) - transformers stays 5.5.4 (satisfies compel >=5,<6 and our <5.6 pin) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> @ * feat(ideogram4): backend + model-manager registration for Ideogram 4 Vendor the Apache-2.0 Ideogram 4 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler, nf4/fp8 quant loading) into invokeai/backend/ideogram4/, plus InvokeAI glue (Qwen3-VL text encoding, packed-input build, dual-branch Euler denoise loop). Register the model: BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config (detected via the Ideogram4Pipeline class name in model_index.json), and the Ideogram4DiffusersModel loader that loads both transformers as one Ideogram4TransformerPair submodel plus the Qwen3-VL encoder and VAE. Text-to-image only. * feat(ideogram4): Ideogram 4 backend — model manager, invocations, nf4 loading End-to-end text-to-image backend for Ideogram 4, validated through the real session runner. Vendors the Apache-2.0 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler) into invokeai/backend/ideogram4/ with InvokeAI glue. Registers BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config, and the Ideogram4DiffusersModel loader (two transformers as one Ideogram4TransformerPair; Qwen3-VL encoder + VAE). Both transformers and the encoder load via InvokeLinearNF4 so they work with the partial-load cache. Adds Ideogram4ConditioningInfo/Field/Output and the model_loader/text_encoder/denoise/l2i invocations. Text-to-image only. * feat(ideogram4): frontend — Regions→JSON prompt, graph builder, UI Wires Ideogram 4 into the canvas/generate UI. buildIdeogram4Prompt assembles the structured JSON caption from the global prompt + Canvas Regional Guidance layers (each region → an obj element with a 0–1000 bbox + desc), with raw-JSON passthrough and a plain-text fallback when there are no regions. Adds buildIdeogram4Graph (text-to-image only, no negative prompt) and the enqueue switch. Structured captions use a static string node + a decoy positive-prompt node so the linear batch can't clobber the assembled JSON; plain text uses the real node so dynamic prompts/batching still work. Registers the 'ideogram-4' base (enums, color, names, model picker, grid size 16), a sampler-preset param (V4_QUALITY_48/V4_DEFAULT_20/V4_TURBO_12) replacing the steps/CFG controls, ParamIdeogram4SamplerPreset, and metadata recall. Regenerates schema.ts. * feat(ideogram4): advanced sampler overrides + color palette Advanced accordion now shows only Ideogram 4-relevant controls. Adds optional overrides of the sampler preset — steps, guidance scale (overrides the main gw, preserves the preset's polish tail), and schedule shift (mu) — plus a color palette editor that injects style_description.color_palette into the auto-built JSON caption (uppercase #RRGGBB, max 16, ignored for raw-JSON prompts). All are nullable (null = use preset), recallable from metadata, and the irrelevant controls (VAE, CLIP skip, CFG rescale, seamless, color compensation) are hidden for Ideogram 4. Backend denoise gains steps/guidance_scale/mu fields; schema.ts regenerated. * Use existing keys + fix select size * Update Readme * feat(ideogram4): add Ideogram 4 to starter models with non-commercial license hint - Implement the weight-only fp8 text-encoder load path (was NotImplementedError); validated against the real fp8 build + add CPU unit tests for the fp8 mechanism - Show Ideogram 4 handlers in the Recall Parameters tab - Recall the assembled JSON caption back into the positive prompt - Translate the metadata "Auto" values - Document Ideogram 4 (install/license, regional-guidance JSON prompting, presets) - Add Ideogram 4 nf4 (CUDA) + fp8 (any device) starter models and bundle - Surface the FLUX-style Non-Commercial License popover for Ideogram 4 models - Note the gated HuggingFace license requirement in the model descriptions * Chore Ruff * Chore Ruff * Chore OpenApi * Chore Knit * fix(deps): regenerate uv.lock to remove duplicate packages from bad merge * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * Fix: address ideogram4 review (strict load, 1-step guidance, i18n, runtime caption) - Non-fp8 Ideogram 4 text-encoder load now validates the state dict: unexpected keys raise, missing keys warn (mirrors the fp8 helper) instead of silently accepting a partial load. - Guidance schedule: cap the polish tail at num_steps-1 so at least one main step always remains (the guidance_scale override was silently dropped at num_steps=1), and require steps >= 2 (backend field + frontend slider/marks). - Localize the Ideogram sampler-preset option labels via t() with the step count interpolated; add the three preset i18n keys. - Assemble the structured JSON caption at generation time in a new ideogram4_caption_builder node (Python port of buildIdeogram4Caption) instead of at graph-build time. The graph now wires the real prompt node -> caption builder -> text encoder and returns it as positivePrompt, so dynamic prompts / prompt batching vary the encoded caption (the decoy that dropped them is removed). The builder's output is wired to a new declared ideogram4_caption metadata field via an edge, so each batched image records its actual caption. Regenerates schema.ts for the new node + metadata field. Adds tests for caption assembly, the guidance schedule, and the graph wiring. * feat(ideogram4): step previews + document the model's built-in safety filter - Emit a low-res progress preview each denoise step so the forming image is visible during generation, like the other denoise nodes. Ideogram uses a FLUX.2-style 32-channel VAE, so the packed latent is unpatchified/denormalized (get_latent_norm) and run through the FLUX.2 latent->RGB factors — no full VAE decode per step. The denoise loop now hands the callback the packed grid latent. - Document Ideogram 4's built-in content safety filter in models.mdx: it lives in the model weights (not Invoke's NSFW checker, can't be disabled from Invoke) and false-positives on benign prompts; structured JSON prompts trip it less. * feat(ideogram4): avoid safety-filter false-positives + step previews + caption visibility The main fix: Ideogram 4's built-in safety filter (baked into the model weights) false-positives and returns an "Image blocked by safety filter" placeholder for "degenerate" captions — empirically, an empty `compositional_deconstruction.elements` list, or a single full-frame [0,0,1000,1000] element whose desc just repeats the high_level_description. Our assembly produced empty elements whenever the user drew no regions, so plain prompts were blocked. - Caption assembly (build_ideogram4_caption): - Always emit a structured JSON caption; never bare plain text (the filter false-positives far more on plain text). Raw-JSON pastes still pass through. - When there are no regions, synthesize one default element describing the whole scene from the prompt with a *partial* (non-full-frame) bbox [100,100,900,900]. This never yields an empty/degenerate elements list. Verified end-to-end against the model: the previously-blocked "golden retriever on a skateboard" now renders. - Metadata: always wire the caption builder's output to the ideogram4_caption metadata field, so the viewer's "Structured Caption" row shows the exact JSON that was encoded (not just the raw prompt) for every generation. - Denoise: emit a low-res progress preview each step (unpatchify + FLUX.2 latent->RGB factors, since Ideogram uses a FLUX.2-style 32-channel VAE) so the forming image is visible during generation, like the other denoise nodes. - Docs: document the model's built-in safety filter in models.mdx (it's not Invoke's NSFW checker, can't be disabled from Invoke, and false-positives). Updates the caption/graph tests accordingly (also fixes latent tsc errors in the graph-builder test's core_metadata / ideogram4_caption comparisons). * Chore typegen + openapi + Ruff * Fix Knit * fix(ideogram4): enforce steps>=2 client-side and validate region bbox Address review on the Ideogram 4 PR: - The backend denoise node requires steps >= 2, but the client still accepted ideogram4_steps = 1 in three places, letting a recalled or rehydrated value build a graph that violates the backend schema. Tighten the zod schema to min(2) with `.catch(null)` (a stale/out-of-range value normalizes to null = use the preset instead of failing the whole persisted slice), normalize dispatched values through the schema in setIdeogram4Steps, and refuse an out-of-range value in the ideogram4_steps metadata recall parser. The slider was already min=2. - Ideogram4Region.bbox was an unconstrained Optional[list[int]], so a workflow/API caller could pass a wrong-length or out-of-range box that the caption builder serialized verbatim into the structured prompt. Add a field validator requiring exactly four coordinates, each in 0..1000. Add tests for both: the region bbox contract (valid/None accepted; short, long, negative, and >1000 rejected) and the ideogram4Steps normalization (valid kept, null kept, stale 1 normalized to null on both dispatch and rehydrate). * fix(ideogram4): block unsupported canvas modes/bbox, warn dropped region inputs, constrain bbox Address the latest review on the Ideogram 4 PR: - Canvas readiness allowed unsupported generation modes: Ideogram 4 is txt2img-only (buildIdeogram4Graph asserts it), but a raster layer or inpaint mask makes the compositor pick img2img/outpaint/inpaint, failing only at graph build. Warn in getRasterLayerWarnings/getInpaintMaskWarnings for Ideogram 4 (these already flow into canvas readiness reasons), blocking enqueue up front. - Canvas readiness had no Ideogram 4 bbox check; the backend requires multiples of 16. Add the 16-grid check mirroring the other 16-grid models so an off-grid bbox (e.g. 1025x1024) is blocked instead of failing backend validation. - getRegionalGuidanceWarnings had no Ideogram 4 branch, so a region whose only input is a negative prompt, auto-negative, or reference image looked effective while the graph silently drops it. Warn those inputs are unsupported. - Ideogram4Region.bbox now also rejects inverted boxes (y_min <= y_max, x_min <= x_max). - The advanced-settings badge selector lumped Ideogram 4 into the generic branch, showing stale VAE/clip-skip/CFG-rescale/seamless badges for controls that are hidden for Ideogram. Exclude Ideogram 4 from that branch. - Model bbox as a constrained type (exactly 4 ints, each 0..1000) so the OpenAPI schema advertises minItems/maxItems and item minimum/maximum. Add readiness tests (bbox grid, raster/inpaint blocking, empty-layer allowance, regional-guidance negative/reference-image warnings) and bbox ordering-rejection tests. * fix(ideogram4): reject text encoders with weights left on the meta device _load_text_encoder() builds the encoder under accelerate.init_empty_weights() and previously downgraded missing keys to a warning (both the fp8 path via load_fp8_state_dict(strict=False) and the non-fp8 path). A missing non-tied weight therefore stayed on the meta device, so a bad or mismatched encoder appeared to load and only failed later during device movement or encoding. Add _verify_encoder_fully_materialized(): call tie_weights() to materialize tied weights from their source, then hard-fail if any parameter or buffer remains on the meta device. Wire it into both the fp8 and non-fp8 (incl. bnb-nf4) paths and drop the missing-key warning — genuinely missing non-tied weights are now caught as leftover meta tensors, while tied weights are tolerated. This is a state-based, path-agnostic check. Add tests: passes when fully materialized, raises on a leftover meta tensor from a missing non-tied weight, and tolerates a tied weight resolved by tie_weights(). * chore(ui): prettier formatting for AdvancedSettingsAccordion The Ideogram 4 badge-suppression branch left the wrapped block at its old indentation, failing `pnpm lint:prettier` in frontend-checks. Formatting only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Your Name <you@example.com> Co-authored-by: 4pointoh <97913726+4pointoh@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> | 1 个月前 | |
Feat(model support): ideogram4 support (#9303) * Update to Transformers 5.1.0 * remove extra stuff * chore(deps): compel fork + transformers>=5.9.0 + remove override Switches compel from PyPI 2.1.1 to invoke-ai/compel@main fork which supports transformers 5.x. Bumps transformers floor to 5.9.0. Removes the transformers>=5.1.0 uv override that was only needed to bypass compel 2.1.1's <5.0 constraint. NOTE: compel fork pulls notebook dep (full Jupyter stack); flag to maintainer for cleanup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(z_image): resolve rope_theta from rope_parameters for transformers 5.x transformers 5.x no longer exposes rope_theta as a top-level attribute on Qwen3Config; the value is stored in the rope_parameters (and rope_scaling) dict instead. Read it from there with a getattr fallback so the inv_freq buffer is computed from the configured base (1e6 / 256) instead of raising AttributeError. Applies to both the safetensors and GGUF Qwen3 encoder paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(model_manager): replace removed hf_hub get_token_permission with whoami huggingface_hub 1.x removed get_token_permission(). HFTokenHelper.get_status() now validates the token via whoami(), which returns user info for a valid token and raises HfHubHTTPError for an invalid one. Preserves the original three-way status: VALID on success, INVALID on HfHubHTTPError (e.g. 401), UNKNOWN on any other error (e.g. network failure). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): regenerate uv.lock after upstream merge Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sd3): resolve merge conflict marker, drop T5TokenizerFast The upstream merge left an unresolved conflict marker in _t5_encode and reintroduced T5TokenizerFast. Keep our v5 assertion (T5Tokenizer only) plus upstream's new t5_device logic, and drop the now-dead T5TokenizerFast monkeypatch in the test (the name no longer exists in the module). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: ruff fixes on merge-resolved files - flux_text_encoder.py: drop unused typing.Union (F401) left by v5 import merge - huggingface.py: ruff format (wrap append(SimpleNamespace(...))) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): pin transformers <5.6 (diffusers single-file CLIP incompat) transformers 5.6 flattened CLIPTextModel (removed the self.text_model wrapper, hoisted embeddings/encoder/final_layer_norm to the top level). diffusers' single-file checkpoint loader (create_diffusers_clip_model_from_ldm) still assumes the nested layout, so loading SD1.5 .safetensors checkpoints fails on 5.6+ with 'CLIPTextModel object has no attribute text_model' and, once that read is shimmed, 'Cannot copy out of meta tensor' (weights never populate the flattened model). Pin to >=5.5,<5.6 (last pre-flattening release) which keeps both the single-file and from_pretrained paths working. The invoke-ai/compel fork accepts any 5.x. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * @ chore(deps): replace compel fork with official compel 2.4.0 compel 2.4.0 (released 2026-05-30) merges the transformers-5 support that the invoke-ai fork carried (both descend from upstream PR #129), plus the maintainer-reviewed padding rework and added diffusers/T5 smoke coverage. Switch from the git fork to the PyPI release. - pyproject: compel git+main -> compel>=2.4.0,<3 - uv.lock: compel 2.3.1 (git 8f404b45) -> 2.4.0 (pypi) - transformers stays 5.5.4 (satisfies compel >=5,<6 and our <5.6 pin) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> @ * feat(ideogram4): backend + model-manager registration for Ideogram 4 Vendor the Apache-2.0 Ideogram 4 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler, nf4/fp8 quant loading) into invokeai/backend/ideogram4/, plus InvokeAI glue (Qwen3-VL text encoding, packed-input build, dual-branch Euler denoise loop). Register the model: BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config (detected via the Ideogram4Pipeline class name in model_index.json), and the Ideogram4DiffusersModel loader that loads both transformers as one Ideogram4TransformerPair submodel plus the Qwen3-VL encoder and VAE. Text-to-image only. * feat(ideogram4): Ideogram 4 backend — model manager, invocations, nf4 loading End-to-end text-to-image backend for Ideogram 4, validated through the real session runner. Vendors the Apache-2.0 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler) into invokeai/backend/ideogram4/ with InvokeAI glue. Registers BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config, and the Ideogram4DiffusersModel loader (two transformers as one Ideogram4TransformerPair; Qwen3-VL encoder + VAE). Both transformers and the encoder load via InvokeLinearNF4 so they work with the partial-load cache. Adds Ideogram4ConditioningInfo/Field/Output and the model_loader/text_encoder/denoise/l2i invocations. Text-to-image only. * feat(ideogram4): frontend — Regions→JSON prompt, graph builder, UI Wires Ideogram 4 into the canvas/generate UI. buildIdeogram4Prompt assembles the structured JSON caption from the global prompt + Canvas Regional Guidance layers (each region → an obj element with a 0–1000 bbox + desc), with raw-JSON passthrough and a plain-text fallback when there are no regions. Adds buildIdeogram4Graph (text-to-image only, no negative prompt) and the enqueue switch. Structured captions use a static string node + a decoy positive-prompt node so the linear batch can't clobber the assembled JSON; plain text uses the real node so dynamic prompts/batching still work. Registers the 'ideogram-4' base (enums, color, names, model picker, grid size 16), a sampler-preset param (V4_QUALITY_48/V4_DEFAULT_20/V4_TURBO_12) replacing the steps/CFG controls, ParamIdeogram4SamplerPreset, and metadata recall. Regenerates schema.ts. * feat(ideogram4): advanced sampler overrides + color palette Advanced accordion now shows only Ideogram 4-relevant controls. Adds optional overrides of the sampler preset — steps, guidance scale (overrides the main gw, preserves the preset's polish tail), and schedule shift (mu) — plus a color palette editor that injects style_description.color_palette into the auto-built JSON caption (uppercase #RRGGBB, max 16, ignored for raw-JSON prompts). All are nullable (null = use preset), recallable from metadata, and the irrelevant controls (VAE, CLIP skip, CFG rescale, seamless, color compensation) are hidden for Ideogram 4. Backend denoise gains steps/guidance_scale/mu fields; schema.ts regenerated. * Use existing keys + fix select size * Update Readme * feat(ideogram4): add Ideogram 4 to starter models with non-commercial license hint - Implement the weight-only fp8 text-encoder load path (was NotImplementedError); validated against the real fp8 build + add CPU unit tests for the fp8 mechanism - Show Ideogram 4 handlers in the Recall Parameters tab - Recall the assembled JSON caption back into the positive prompt - Translate the metadata "Auto" values - Document Ideogram 4 (install/license, regional-guidance JSON prompting, presets) - Add Ideogram 4 nf4 (CUDA) + fp8 (any device) starter models and bundle - Surface the FLUX-style Non-Commercial License popover for Ideogram 4 models - Note the gated HuggingFace license requirement in the model descriptions * Chore Ruff * Chore Ruff * Chore OpenApi * Chore Knit * fix(deps): regenerate uv.lock to remove duplicate packages from bad merge * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * Fix: address ideogram4 review (strict load, 1-step guidance, i18n, runtime caption) - Non-fp8 Ideogram 4 text-encoder load now validates the state dict: unexpected keys raise, missing keys warn (mirrors the fp8 helper) instead of silently accepting a partial load. - Guidance schedule: cap the polish tail at num_steps-1 so at least one main step always remains (the guidance_scale override was silently dropped at num_steps=1), and require steps >= 2 (backend field + frontend slider/marks). - Localize the Ideogram sampler-preset option labels via t() with the step count interpolated; add the three preset i18n keys. - Assemble the structured JSON caption at generation time in a new ideogram4_caption_builder node (Python port of buildIdeogram4Caption) instead of at graph-build time. The graph now wires the real prompt node -> caption builder -> text encoder and returns it as positivePrompt, so dynamic prompts / prompt batching vary the encoded caption (the decoy that dropped them is removed). The builder's output is wired to a new declared ideogram4_caption metadata field via an edge, so each batched image records its actual caption. Regenerates schema.ts for the new node + metadata field. Adds tests for caption assembly, the guidance schedule, and the graph wiring. * feat(ideogram4): step previews + document the model's built-in safety filter - Emit a low-res progress preview each denoise step so the forming image is visible during generation, like the other denoise nodes. Ideogram uses a FLUX.2-style 32-channel VAE, so the packed latent is unpatchified/denormalized (get_latent_norm) and run through the FLUX.2 latent->RGB factors — no full VAE decode per step. The denoise loop now hands the callback the packed grid latent. - Document Ideogram 4's built-in content safety filter in models.mdx: it lives in the model weights (not Invoke's NSFW checker, can't be disabled from Invoke) and false-positives on benign prompts; structured JSON prompts trip it less. * feat(ideogram4): avoid safety-filter false-positives + step previews + caption visibility The main fix: Ideogram 4's built-in safety filter (baked into the model weights) false-positives and returns an "Image blocked by safety filter" placeholder for "degenerate" captions — empirically, an empty `compositional_deconstruction.elements` list, or a single full-frame [0,0,1000,1000] element whose desc just repeats the high_level_description. Our assembly produced empty elements whenever the user drew no regions, so plain prompts were blocked. - Caption assembly (build_ideogram4_caption): - Always emit a structured JSON caption; never bare plain text (the filter false-positives far more on plain text). Raw-JSON pastes still pass through. - When there are no regions, synthesize one default element describing the whole scene from the prompt with a *partial* (non-full-frame) bbox [100,100,900,900]. This never yields an empty/degenerate elements list. Verified end-to-end against the model: the previously-blocked "golden retriever on a skateboard" now renders. - Metadata: always wire the caption builder's output to the ideogram4_caption metadata field, so the viewer's "Structured Caption" row shows the exact JSON that was encoded (not just the raw prompt) for every generation. - Denoise: emit a low-res progress preview each step (unpatchify + FLUX.2 latent->RGB factors, since Ideogram uses a FLUX.2-style 32-channel VAE) so the forming image is visible during generation, like the other denoise nodes. - Docs: document the model's built-in safety filter in models.mdx (it's not Invoke's NSFW checker, can't be disabled from Invoke, and false-positives). Updates the caption/graph tests accordingly (also fixes latent tsc errors in the graph-builder test's core_metadata / ideogram4_caption comparisons). * Chore typegen + openapi + Ruff * Fix Knit * fix(ideogram4): enforce steps>=2 client-side and validate region bbox Address review on the Ideogram 4 PR: - The backend denoise node requires steps >= 2, but the client still accepted ideogram4_steps = 1 in three places, letting a recalled or rehydrated value build a graph that violates the backend schema. Tighten the zod schema to min(2) with `.catch(null)` (a stale/out-of-range value normalizes to null = use the preset instead of failing the whole persisted slice), normalize dispatched values through the schema in setIdeogram4Steps, and refuse an out-of-range value in the ideogram4_steps metadata recall parser. The slider was already min=2. - Ideogram4Region.bbox was an unconstrained Optional[list[int]], so a workflow/API caller could pass a wrong-length or out-of-range box that the caption builder serialized verbatim into the structured prompt. Add a field validator requiring exactly four coordinates, each in 0..1000. Add tests for both: the region bbox contract (valid/None accepted; short, long, negative, and >1000 rejected) and the ideogram4Steps normalization (valid kept, null kept, stale 1 normalized to null on both dispatch and rehydrate). * fix(ideogram4): block unsupported canvas modes/bbox, warn dropped region inputs, constrain bbox Address the latest review on the Ideogram 4 PR: - Canvas readiness allowed unsupported generation modes: Ideogram 4 is txt2img-only (buildIdeogram4Graph asserts it), but a raster layer or inpaint mask makes the compositor pick img2img/outpaint/inpaint, failing only at graph build. Warn in getRasterLayerWarnings/getInpaintMaskWarnings for Ideogram 4 (these already flow into canvas readiness reasons), blocking enqueue up front. - Canvas readiness had no Ideogram 4 bbox check; the backend requires multiples of 16. Add the 16-grid check mirroring the other 16-grid models so an off-grid bbox (e.g. 1025x1024) is blocked instead of failing backend validation. - getRegionalGuidanceWarnings had no Ideogram 4 branch, so a region whose only input is a negative prompt, auto-negative, or reference image looked effective while the graph silently drops it. Warn those inputs are unsupported. - Ideogram4Region.bbox now also rejects inverted boxes (y_min <= y_max, x_min <= x_max). - The advanced-settings badge selector lumped Ideogram 4 into the generic branch, showing stale VAE/clip-skip/CFG-rescale/seamless badges for controls that are hidden for Ideogram. Exclude Ideogram 4 from that branch. - Model bbox as a constrained type (exactly 4 ints, each 0..1000) so the OpenAPI schema advertises minItems/maxItems and item minimum/maximum. Add readiness tests (bbox grid, raster/inpaint blocking, empty-layer allowance, regional-guidance negative/reference-image warnings) and bbox ordering-rejection tests. * fix(ideogram4): reject text encoders with weights left on the meta device _load_text_encoder() builds the encoder under accelerate.init_empty_weights() and previously downgraded missing keys to a warning (both the fp8 path via load_fp8_state_dict(strict=False) and the non-fp8 path). A missing non-tied weight therefore stayed on the meta device, so a bad or mismatched encoder appeared to load and only failed later during device movement or encoding. Add _verify_encoder_fully_materialized(): call tie_weights() to materialize tied weights from their source, then hard-fail if any parameter or buffer remains on the meta device. Wire it into both the fp8 and non-fp8 (incl. bnb-nf4) paths and drop the missing-key warning — genuinely missing non-tied weights are now caught as leftover meta tensors, while tied weights are tolerated. This is a state-based, path-agnostic check. Add tests: passes when fully materialized, raises on a leftover meta tensor from a missing non-tied weight, and tolerates a tied weight resolved by tie_weights(). * chore(ui): prettier formatting for AdvancedSettingsAccordion The Ideogram 4 badge-suppression branch left the wrapped block at its old indentation, failing `pnpm lint:prettier` in frontend-checks. Formatting only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Your Name <you@example.com> Co-authored-by: 4pointoh <97913726+4pointoh@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> | 1 个月前 | |
Feat(model support): ideogram4 support (#9303) * Update to Transformers 5.1.0 * remove extra stuff * chore(deps): compel fork + transformers>=5.9.0 + remove override Switches compel from PyPI 2.1.1 to invoke-ai/compel@main fork which supports transformers 5.x. Bumps transformers floor to 5.9.0. Removes the transformers>=5.1.0 uv override that was only needed to bypass compel 2.1.1's <5.0 constraint. NOTE: compel fork pulls notebook dep (full Jupyter stack); flag to maintainer for cleanup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(z_image): resolve rope_theta from rope_parameters for transformers 5.x transformers 5.x no longer exposes rope_theta as a top-level attribute on Qwen3Config; the value is stored in the rope_parameters (and rope_scaling) dict instead. Read it from there with a getattr fallback so the inv_freq buffer is computed from the configured base (1e6 / 256) instead of raising AttributeError. Applies to both the safetensors and GGUF Qwen3 encoder paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(model_manager): replace removed hf_hub get_token_permission with whoami huggingface_hub 1.x removed get_token_permission(). HFTokenHelper.get_status() now validates the token via whoami(), which returns user info for a valid token and raises HfHubHTTPError for an invalid one. Preserves the original three-way status: VALID on success, INVALID on HfHubHTTPError (e.g. 401), UNKNOWN on any other error (e.g. network failure). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): regenerate uv.lock after upstream merge Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sd3): resolve merge conflict marker, drop T5TokenizerFast The upstream merge left an unresolved conflict marker in _t5_encode and reintroduced T5TokenizerFast. Keep our v5 assertion (T5Tokenizer only) plus upstream's new t5_device logic, and drop the now-dead T5TokenizerFast monkeypatch in the test (the name no longer exists in the module). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: ruff fixes on merge-resolved files - flux_text_encoder.py: drop unused typing.Union (F401) left by v5 import merge - huggingface.py: ruff format (wrap append(SimpleNamespace(...))) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): pin transformers <5.6 (diffusers single-file CLIP incompat) transformers 5.6 flattened CLIPTextModel (removed the self.text_model wrapper, hoisted embeddings/encoder/final_layer_norm to the top level). diffusers' single-file checkpoint loader (create_diffusers_clip_model_from_ldm) still assumes the nested layout, so loading SD1.5 .safetensors checkpoints fails on 5.6+ with 'CLIPTextModel object has no attribute text_model' and, once that read is shimmed, 'Cannot copy out of meta tensor' (weights never populate the flattened model). Pin to >=5.5,<5.6 (last pre-flattening release) which keeps both the single-file and from_pretrained paths working. The invoke-ai/compel fork accepts any 5.x. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * @ chore(deps): replace compel fork with official compel 2.4.0 compel 2.4.0 (released 2026-05-30) merges the transformers-5 support that the invoke-ai fork carried (both descend from upstream PR #129), plus the maintainer-reviewed padding rework and added diffusers/T5 smoke coverage. Switch from the git fork to the PyPI release. - pyproject: compel git+main -> compel>=2.4.0,<3 - uv.lock: compel 2.3.1 (git 8f404b45) -> 2.4.0 (pypi) - transformers stays 5.5.4 (satisfies compel >=5,<6 and our <5.6 pin) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> @ * feat(ideogram4): backend + model-manager registration for Ideogram 4 Vendor the Apache-2.0 Ideogram 4 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler, nf4/fp8 quant loading) into invokeai/backend/ideogram4/, plus InvokeAI glue (Qwen3-VL text encoding, packed-input build, dual-branch Euler denoise loop). Register the model: BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config (detected via the Ideogram4Pipeline class name in model_index.json), and the Ideogram4DiffusersModel loader that loads both transformers as one Ideogram4TransformerPair submodel plus the Qwen3-VL encoder and VAE. Text-to-image only. * feat(ideogram4): Ideogram 4 backend — model manager, invocations, nf4 loading End-to-end text-to-image backend for Ideogram 4, validated through the real session runner. Vendors the Apache-2.0 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler) into invokeai/backend/ideogram4/ with InvokeAI glue. Registers BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config, and the Ideogram4DiffusersModel loader (two transformers as one Ideogram4TransformerPair; Qwen3-VL encoder + VAE). Both transformers and the encoder load via InvokeLinearNF4 so they work with the partial-load cache. Adds Ideogram4ConditioningInfo/Field/Output and the model_loader/text_encoder/denoise/l2i invocations. Text-to-image only. * feat(ideogram4): frontend — Regions→JSON prompt, graph builder, UI Wires Ideogram 4 into the canvas/generate UI. buildIdeogram4Prompt assembles the structured JSON caption from the global prompt + Canvas Regional Guidance layers (each region → an obj element with a 0–1000 bbox + desc), with raw-JSON passthrough and a plain-text fallback when there are no regions. Adds buildIdeogram4Graph (text-to-image only, no negative prompt) and the enqueue switch. Structured captions use a static string node + a decoy positive-prompt node so the linear batch can't clobber the assembled JSON; plain text uses the real node so dynamic prompts/batching still work. Registers the 'ideogram-4' base (enums, color, names, model picker, grid size 16), a sampler-preset param (V4_QUALITY_48/V4_DEFAULT_20/V4_TURBO_12) replacing the steps/CFG controls, ParamIdeogram4SamplerPreset, and metadata recall. Regenerates schema.ts. * feat(ideogram4): advanced sampler overrides + color palette Advanced accordion now shows only Ideogram 4-relevant controls. Adds optional overrides of the sampler preset — steps, guidance scale (overrides the main gw, preserves the preset's polish tail), and schedule shift (mu) — plus a color palette editor that injects style_description.color_palette into the auto-built JSON caption (uppercase #RRGGBB, max 16, ignored for raw-JSON prompts). All are nullable (null = use preset), recallable from metadata, and the irrelevant controls (VAE, CLIP skip, CFG rescale, seamless, color compensation) are hidden for Ideogram 4. Backend denoise gains steps/guidance_scale/mu fields; schema.ts regenerated. * Use existing keys + fix select size * Update Readme * feat(ideogram4): add Ideogram 4 to starter models with non-commercial license hint - Implement the weight-only fp8 text-encoder load path (was NotImplementedError); validated against the real fp8 build + add CPU unit tests for the fp8 mechanism - Show Ideogram 4 handlers in the Recall Parameters tab - Recall the assembled JSON caption back into the positive prompt - Translate the metadata "Auto" values - Document Ideogram 4 (install/license, regional-guidance JSON prompting, presets) - Add Ideogram 4 nf4 (CUDA) + fp8 (any device) starter models and bundle - Surface the FLUX-style Non-Commercial License popover for Ideogram 4 models - Note the gated HuggingFace license requirement in the model descriptions * Chore Ruff * Chore Ruff * Chore OpenApi * Chore Knit * fix(deps): regenerate uv.lock to remove duplicate packages from bad merge * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * Fix: address ideogram4 review (strict load, 1-step guidance, i18n, runtime caption) - Non-fp8 Ideogram 4 text-encoder load now validates the state dict: unexpected keys raise, missing keys warn (mirrors the fp8 helper) instead of silently accepting a partial load. - Guidance schedule: cap the polish tail at num_steps-1 so at least one main step always remains (the guidance_scale override was silently dropped at num_steps=1), and require steps >= 2 (backend field + frontend slider/marks). - Localize the Ideogram sampler-preset option labels via t() with the step count interpolated; add the three preset i18n keys. - Assemble the structured JSON caption at generation time in a new ideogram4_caption_builder node (Python port of buildIdeogram4Caption) instead of at graph-build time. The graph now wires the real prompt node -> caption builder -> text encoder and returns it as positivePrompt, so dynamic prompts / prompt batching vary the encoded caption (the decoy that dropped them is removed). The builder's output is wired to a new declared ideogram4_caption metadata field via an edge, so each batched image records its actual caption. Regenerates schema.ts for the new node + metadata field. Adds tests for caption assembly, the guidance schedule, and the graph wiring. * feat(ideogram4): step previews + document the model's built-in safety filter - Emit a low-res progress preview each denoise step so the forming image is visible during generation, like the other denoise nodes. Ideogram uses a FLUX.2-style 32-channel VAE, so the packed latent is unpatchified/denormalized (get_latent_norm) and run through the FLUX.2 latent->RGB factors — no full VAE decode per step. The denoise loop now hands the callback the packed grid latent. - Document Ideogram 4's built-in content safety filter in models.mdx: it lives in the model weights (not Invoke's NSFW checker, can't be disabled from Invoke) and false-positives on benign prompts; structured JSON prompts trip it less. * feat(ideogram4): avoid safety-filter false-positives + step previews + caption visibility The main fix: Ideogram 4's built-in safety filter (baked into the model weights) false-positives and returns an "Image blocked by safety filter" placeholder for "degenerate" captions — empirically, an empty `compositional_deconstruction.elements` list, or a single full-frame [0,0,1000,1000] element whose desc just repeats the high_level_description. Our assembly produced empty elements whenever the user drew no regions, so plain prompts were blocked. - Caption assembly (build_ideogram4_caption): - Always emit a structured JSON caption; never bare plain text (the filter false-positives far more on plain text). Raw-JSON pastes still pass through. - When there are no regions, synthesize one default element describing the whole scene from the prompt with a *partial* (non-full-frame) bbox [100,100,900,900]. This never yields an empty/degenerate elements list. Verified end-to-end against the model: the previously-blocked "golden retriever on a skateboard" now renders. - Metadata: always wire the caption builder's output to the ideogram4_caption metadata field, so the viewer's "Structured Caption" row shows the exact JSON that was encoded (not just the raw prompt) for every generation. - Denoise: emit a low-res progress preview each step (unpatchify + FLUX.2 latent->RGB factors, since Ideogram uses a FLUX.2-style 32-channel VAE) so the forming image is visible during generation, like the other denoise nodes. - Docs: document the model's built-in safety filter in models.mdx (it's not Invoke's NSFW checker, can't be disabled from Invoke, and false-positives). Updates the caption/graph tests accordingly (also fixes latent tsc errors in the graph-builder test's core_metadata / ideogram4_caption comparisons). * Chore typegen + openapi + Ruff * Fix Knit * fix(ideogram4): enforce steps>=2 client-side and validate region bbox Address review on the Ideogram 4 PR: - The backend denoise node requires steps >= 2, but the client still accepted ideogram4_steps = 1 in three places, letting a recalled or rehydrated value build a graph that violates the backend schema. Tighten the zod schema to min(2) with `.catch(null)` (a stale/out-of-range value normalizes to null = use the preset instead of failing the whole persisted slice), normalize dispatched values through the schema in setIdeogram4Steps, and refuse an out-of-range value in the ideogram4_steps metadata recall parser. The slider was already min=2. - Ideogram4Region.bbox was an unconstrained Optional[list[int]], so a workflow/API caller could pass a wrong-length or out-of-range box that the caption builder serialized verbatim into the structured prompt. Add a field validator requiring exactly four coordinates, each in 0..1000. Add tests for both: the region bbox contract (valid/None accepted; short, long, negative, and >1000 rejected) and the ideogram4Steps normalization (valid kept, null kept, stale 1 normalized to null on both dispatch and rehydrate). * fix(ideogram4): block unsupported canvas modes/bbox, warn dropped region inputs, constrain bbox Address the latest review on the Ideogram 4 PR: - Canvas readiness allowed unsupported generation modes: Ideogram 4 is txt2img-only (buildIdeogram4Graph asserts it), but a raster layer or inpaint mask makes the compositor pick img2img/outpaint/inpaint, failing only at graph build. Warn in getRasterLayerWarnings/getInpaintMaskWarnings for Ideogram 4 (these already flow into canvas readiness reasons), blocking enqueue up front. - Canvas readiness had no Ideogram 4 bbox check; the backend requires multiples of 16. Add the 16-grid check mirroring the other 16-grid models so an off-grid bbox (e.g. 1025x1024) is blocked instead of failing backend validation. - getRegionalGuidanceWarnings had no Ideogram 4 branch, so a region whose only input is a negative prompt, auto-negative, or reference image looked effective while the graph silently drops it. Warn those inputs are unsupported. - Ideogram4Region.bbox now also rejects inverted boxes (y_min <= y_max, x_min <= x_max). - The advanced-settings badge selector lumped Ideogram 4 into the generic branch, showing stale VAE/clip-skip/CFG-rescale/seamless badges for controls that are hidden for Ideogram. Exclude Ideogram 4 from that branch. - Model bbox as a constrained type (exactly 4 ints, each 0..1000) so the OpenAPI schema advertises minItems/maxItems and item minimum/maximum. Add readiness tests (bbox grid, raster/inpaint blocking, empty-layer allowance, regional-guidance negative/reference-image warnings) and bbox ordering-rejection tests. * fix(ideogram4): reject text encoders with weights left on the meta device _load_text_encoder() builds the encoder under accelerate.init_empty_weights() and previously downgraded missing keys to a warning (both the fp8 path via load_fp8_state_dict(strict=False) and the non-fp8 path). A missing non-tied weight therefore stayed on the meta device, so a bad or mismatched encoder appeared to load and only failed later during device movement or encoding. Add _verify_encoder_fully_materialized(): call tie_weights() to materialize tied weights from their source, then hard-fail if any parameter or buffer remains on the meta device. Wire it into both the fp8 and non-fp8 (incl. bnb-nf4) paths and drop the missing-key warning — genuinely missing non-tied weights are now caught as leftover meta tensors, while tied weights are tolerated. This is a state-based, path-agnostic check. Add tests: passes when fully materialized, raises on a leftover meta tensor from a missing non-tied weight, and tolerates a tied weight resolved by tie_weights(). * chore(ui): prettier formatting for AdvancedSettingsAccordion The Ideogram 4 badge-suppression branch left the wrapped block at its old indentation, failing `pnpm lint:prettier` in frontend-checks. Formatting only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Your Name <you@example.com> Co-authored-by: 4pointoh <97913726+4pointoh@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> | 1 个月前 | |
Feat(model support): ideogram4 support (#9303) * Update to Transformers 5.1.0 * remove extra stuff * chore(deps): compel fork + transformers>=5.9.0 + remove override Switches compel from PyPI 2.1.1 to invoke-ai/compel@main fork which supports transformers 5.x. Bumps transformers floor to 5.9.0. Removes the transformers>=5.1.0 uv override that was only needed to bypass compel 2.1.1's <5.0 constraint. NOTE: compel fork pulls notebook dep (full Jupyter stack); flag to maintainer for cleanup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(z_image): resolve rope_theta from rope_parameters for transformers 5.x transformers 5.x no longer exposes rope_theta as a top-level attribute on Qwen3Config; the value is stored in the rope_parameters (and rope_scaling) dict instead. Read it from there with a getattr fallback so the inv_freq buffer is computed from the configured base (1e6 / 256) instead of raising AttributeError. Applies to both the safetensors and GGUF Qwen3 encoder paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(model_manager): replace removed hf_hub get_token_permission with whoami huggingface_hub 1.x removed get_token_permission(). HFTokenHelper.get_status() now validates the token via whoami(), which returns user info for a valid token and raises HfHubHTTPError for an invalid one. Preserves the original three-way status: VALID on success, INVALID on HfHubHTTPError (e.g. 401), UNKNOWN on any other error (e.g. network failure). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): regenerate uv.lock after upstream merge Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sd3): resolve merge conflict marker, drop T5TokenizerFast The upstream merge left an unresolved conflict marker in _t5_encode and reintroduced T5TokenizerFast. Keep our v5 assertion (T5Tokenizer only) plus upstream's new t5_device logic, and drop the now-dead T5TokenizerFast monkeypatch in the test (the name no longer exists in the module). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: ruff fixes on merge-resolved files - flux_text_encoder.py: drop unused typing.Union (F401) left by v5 import merge - huggingface.py: ruff format (wrap append(SimpleNamespace(...))) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): pin transformers <5.6 (diffusers single-file CLIP incompat) transformers 5.6 flattened CLIPTextModel (removed the self.text_model wrapper, hoisted embeddings/encoder/final_layer_norm to the top level). diffusers' single-file checkpoint loader (create_diffusers_clip_model_from_ldm) still assumes the nested layout, so loading SD1.5 .safetensors checkpoints fails on 5.6+ with 'CLIPTextModel object has no attribute text_model' and, once that read is shimmed, 'Cannot copy out of meta tensor' (weights never populate the flattened model). Pin to >=5.5,<5.6 (last pre-flattening release) which keeps both the single-file and from_pretrained paths working. The invoke-ai/compel fork accepts any 5.x. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * @ chore(deps): replace compel fork with official compel 2.4.0 compel 2.4.0 (released 2026-05-30) merges the transformers-5 support that the invoke-ai fork carried (both descend from upstream PR #129), plus the maintainer-reviewed padding rework and added diffusers/T5 smoke coverage. Switch from the git fork to the PyPI release. - pyproject: compel git+main -> compel>=2.4.0,<3 - uv.lock: compel 2.3.1 (git 8f404b45) -> 2.4.0 (pypi) - transformers stays 5.5.4 (satisfies compel >=5,<6 and our <5.6 pin) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> @ * feat(ideogram4): backend + model-manager registration for Ideogram 4 Vendor the Apache-2.0 Ideogram 4 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler, nf4/fp8 quant loading) into invokeai/backend/ideogram4/, plus InvokeAI glue (Qwen3-VL text encoding, packed-input build, dual-branch Euler denoise loop). Register the model: BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config (detected via the Ideogram4Pipeline class name in model_index.json), and the Ideogram4DiffusersModel loader that loads both transformers as one Ideogram4TransformerPair submodel plus the Qwen3-VL encoder and VAE. Text-to-image only. * feat(ideogram4): Ideogram 4 backend — model manager, invocations, nf4 loading End-to-end text-to-image backend for Ideogram 4, validated through the real session runner. Vendors the Apache-2.0 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler) into invokeai/backend/ideogram4/ with InvokeAI glue. Registers BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config, and the Ideogram4DiffusersModel loader (two transformers as one Ideogram4TransformerPair; Qwen3-VL encoder + VAE). Both transformers and the encoder load via InvokeLinearNF4 so they work with the partial-load cache. Adds Ideogram4ConditioningInfo/Field/Output and the model_loader/text_encoder/denoise/l2i invocations. Text-to-image only. * feat(ideogram4): frontend — Regions→JSON prompt, graph builder, UI Wires Ideogram 4 into the canvas/generate UI. buildIdeogram4Prompt assembles the structured JSON caption from the global prompt + Canvas Regional Guidance layers (each region → an obj element with a 0–1000 bbox + desc), with raw-JSON passthrough and a plain-text fallback when there are no regions. Adds buildIdeogram4Graph (text-to-image only, no negative prompt) and the enqueue switch. Structured captions use a static string node + a decoy positive-prompt node so the linear batch can't clobber the assembled JSON; plain text uses the real node so dynamic prompts/batching still work. Registers the 'ideogram-4' base (enums, color, names, model picker, grid size 16), a sampler-preset param (V4_QUALITY_48/V4_DEFAULT_20/V4_TURBO_12) replacing the steps/CFG controls, ParamIdeogram4SamplerPreset, and metadata recall. Regenerates schema.ts. * feat(ideogram4): advanced sampler overrides + color palette Advanced accordion now shows only Ideogram 4-relevant controls. Adds optional overrides of the sampler preset — steps, guidance scale (overrides the main gw, preserves the preset's polish tail), and schedule shift (mu) — plus a color palette editor that injects style_description.color_palette into the auto-built JSON caption (uppercase #RRGGBB, max 16, ignored for raw-JSON prompts). All are nullable (null = use preset), recallable from metadata, and the irrelevant controls (VAE, CLIP skip, CFG rescale, seamless, color compensation) are hidden for Ideogram 4. Backend denoise gains steps/guidance_scale/mu fields; schema.ts regenerated. * Use existing keys + fix select size * Update Readme * feat(ideogram4): add Ideogram 4 to starter models with non-commercial license hint - Implement the weight-only fp8 text-encoder load path (was NotImplementedError); validated against the real fp8 build + add CPU unit tests for the fp8 mechanism - Show Ideogram 4 handlers in the Recall Parameters tab - Recall the assembled JSON caption back into the positive prompt - Translate the metadata "Auto" values - Document Ideogram 4 (install/license, regional-guidance JSON prompting, presets) - Add Ideogram 4 nf4 (CUDA) + fp8 (any device) starter models and bundle - Surface the FLUX-style Non-Commercial License popover for Ideogram 4 models - Note the gated HuggingFace license requirement in the model descriptions * Chore Ruff * Chore Ruff * Chore OpenApi * Chore Knit * fix(deps): regenerate uv.lock to remove duplicate packages from bad merge * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * Fix: address ideogram4 review (strict load, 1-step guidance, i18n, runtime caption) - Non-fp8 Ideogram 4 text-encoder load now validates the state dict: unexpected keys raise, missing keys warn (mirrors the fp8 helper) instead of silently accepting a partial load. - Guidance schedule: cap the polish tail at num_steps-1 so at least one main step always remains (the guidance_scale override was silently dropped at num_steps=1), and require steps >= 2 (backend field + frontend slider/marks). - Localize the Ideogram sampler-preset option labels via t() with the step count interpolated; add the three preset i18n keys. - Assemble the structured JSON caption at generation time in a new ideogram4_caption_builder node (Python port of buildIdeogram4Caption) instead of at graph-build time. The graph now wires the real prompt node -> caption builder -> text encoder and returns it as positivePrompt, so dynamic prompts / prompt batching vary the encoded caption (the decoy that dropped them is removed). The builder's output is wired to a new declared ideogram4_caption metadata field via an edge, so each batched image records its actual caption. Regenerates schema.ts for the new node + metadata field. Adds tests for caption assembly, the guidance schedule, and the graph wiring. * feat(ideogram4): step previews + document the model's built-in safety filter - Emit a low-res progress preview each denoise step so the forming image is visible during generation, like the other denoise nodes. Ideogram uses a FLUX.2-style 32-channel VAE, so the packed latent is unpatchified/denormalized (get_latent_norm) and run through the FLUX.2 latent->RGB factors — no full VAE decode per step. The denoise loop now hands the callback the packed grid latent. - Document Ideogram 4's built-in content safety filter in models.mdx: it lives in the model weights (not Invoke's NSFW checker, can't be disabled from Invoke) and false-positives on benign prompts; structured JSON prompts trip it less. * feat(ideogram4): avoid safety-filter false-positives + step previews + caption visibility The main fix: Ideogram 4's built-in safety filter (baked into the model weights) false-positives and returns an "Image blocked by safety filter" placeholder for "degenerate" captions — empirically, an empty `compositional_deconstruction.elements` list, or a single full-frame [0,0,1000,1000] element whose desc just repeats the high_level_description. Our assembly produced empty elements whenever the user drew no regions, so plain prompts were blocked. - Caption assembly (build_ideogram4_caption): - Always emit a structured JSON caption; never bare plain text (the filter false-positives far more on plain text). Raw-JSON pastes still pass through. - When there are no regions, synthesize one default element describing the whole scene from the prompt with a *partial* (non-full-frame) bbox [100,100,900,900]. This never yields an empty/degenerate elements list. Verified end-to-end against the model: the previously-blocked "golden retriever on a skateboard" now renders. - Metadata: always wire the caption builder's output to the ideogram4_caption metadata field, so the viewer's "Structured Caption" row shows the exact JSON that was encoded (not just the raw prompt) for every generation. - Denoise: emit a low-res progress preview each step (unpatchify + FLUX.2 latent->RGB factors, since Ideogram uses a FLUX.2-style 32-channel VAE) so the forming image is visible during generation, like the other denoise nodes. - Docs: document the model's built-in safety filter in models.mdx (it's not Invoke's NSFW checker, can't be disabled from Invoke, and false-positives). Updates the caption/graph tests accordingly (also fixes latent tsc errors in the graph-builder test's core_metadata / ideogram4_caption comparisons). * Chore typegen + openapi + Ruff * Fix Knit * fix(ideogram4): enforce steps>=2 client-side and validate region bbox Address review on the Ideogram 4 PR: - The backend denoise node requires steps >= 2, but the client still accepted ideogram4_steps = 1 in three places, letting a recalled or rehydrated value build a graph that violates the backend schema. Tighten the zod schema to min(2) with `.catch(null)` (a stale/out-of-range value normalizes to null = use the preset instead of failing the whole persisted slice), normalize dispatched values through the schema in setIdeogram4Steps, and refuse an out-of-range value in the ideogram4_steps metadata recall parser. The slider was already min=2. - Ideogram4Region.bbox was an unconstrained Optional[list[int]], so a workflow/API caller could pass a wrong-length or out-of-range box that the caption builder serialized verbatim into the structured prompt. Add a field validator requiring exactly four coordinates, each in 0..1000. Add tests for both: the region bbox contract (valid/None accepted; short, long, negative, and >1000 rejected) and the ideogram4Steps normalization (valid kept, null kept, stale 1 normalized to null on both dispatch and rehydrate). * fix(ideogram4): block unsupported canvas modes/bbox, warn dropped region inputs, constrain bbox Address the latest review on the Ideogram 4 PR: - Canvas readiness allowed unsupported generation modes: Ideogram 4 is txt2img-only (buildIdeogram4Graph asserts it), but a raster layer or inpaint mask makes the compositor pick img2img/outpaint/inpaint, failing only at graph build. Warn in getRasterLayerWarnings/getInpaintMaskWarnings for Ideogram 4 (these already flow into canvas readiness reasons), blocking enqueue up front. - Canvas readiness had no Ideogram 4 bbox check; the backend requires multiples of 16. Add the 16-grid check mirroring the other 16-grid models so an off-grid bbox (e.g. 1025x1024) is blocked instead of failing backend validation. - getRegionalGuidanceWarnings had no Ideogram 4 branch, so a region whose only input is a negative prompt, auto-negative, or reference image looked effective while the graph silently drops it. Warn those inputs are unsupported. - Ideogram4Region.bbox now also rejects inverted boxes (y_min <= y_max, x_min <= x_max). - The advanced-settings badge selector lumped Ideogram 4 into the generic branch, showing stale VAE/clip-skip/CFG-rescale/seamless badges for controls that are hidden for Ideogram. Exclude Ideogram 4 from that branch. - Model bbox as a constrained type (exactly 4 ints, each 0..1000) so the OpenAPI schema advertises minItems/maxItems and item minimum/maximum. Add readiness tests (bbox grid, raster/inpaint blocking, empty-layer allowance, regional-guidance negative/reference-image warnings) and bbox ordering-rejection tests. * fix(ideogram4): reject text encoders with weights left on the meta device _load_text_encoder() builds the encoder under accelerate.init_empty_weights() and previously downgraded missing keys to a warning (both the fp8 path via load_fp8_state_dict(strict=False) and the non-fp8 path). A missing non-tied weight therefore stayed on the meta device, so a bad or mismatched encoder appeared to load and only failed later during device movement or encoding. Add _verify_encoder_fully_materialized(): call tie_weights() to materialize tied weights from their source, then hard-fail if any parameter or buffer remains on the meta device. Wire it into both the fp8 and non-fp8 (incl. bnb-nf4) paths and drop the missing-key warning — genuinely missing non-tied weights are now caught as leftover meta tensors, while tied weights are tolerated. This is a state-based, path-agnostic check. Add tests: passes when fully materialized, raises on a leftover meta tensor from a missing non-tied weight, and tolerates a tied weight resolved by tie_weights(). * chore(ui): prettier formatting for AdvancedSettingsAccordion The Ideogram 4 badge-suppression branch left the wrapped block at its old indentation, failing `pnpm lint:prettier` in frontend-checks. Formatting only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Your Name <you@example.com> Co-authored-by: 4pointoh <97913726+4pointoh@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> | 1 个月前 | |
Feat(model support): ideogram4 support (#9303) * Update to Transformers 5.1.0 * remove extra stuff * chore(deps): compel fork + transformers>=5.9.0 + remove override Switches compel from PyPI 2.1.1 to invoke-ai/compel@main fork which supports transformers 5.x. Bumps transformers floor to 5.9.0. Removes the transformers>=5.1.0 uv override that was only needed to bypass compel 2.1.1's <5.0 constraint. NOTE: compel fork pulls notebook dep (full Jupyter stack); flag to maintainer for cleanup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(z_image): resolve rope_theta from rope_parameters for transformers 5.x transformers 5.x no longer exposes rope_theta as a top-level attribute on Qwen3Config; the value is stored in the rope_parameters (and rope_scaling) dict instead. Read it from there with a getattr fallback so the inv_freq buffer is computed from the configured base (1e6 / 256) instead of raising AttributeError. Applies to both the safetensors and GGUF Qwen3 encoder paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(model_manager): replace removed hf_hub get_token_permission with whoami huggingface_hub 1.x removed get_token_permission(). HFTokenHelper.get_status() now validates the token via whoami(), which returns user info for a valid token and raises HfHubHTTPError for an invalid one. Preserves the original three-way status: VALID on success, INVALID on HfHubHTTPError (e.g. 401), UNKNOWN on any other error (e.g. network failure). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): regenerate uv.lock after upstream merge Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(sd3): resolve merge conflict marker, drop T5TokenizerFast The upstream merge left an unresolved conflict marker in _t5_encode and reintroduced T5TokenizerFast. Keep our v5 assertion (T5Tokenizer only) plus upstream's new t5_device logic, and drop the now-dead T5TokenizerFast monkeypatch in the test (the name no longer exists in the module). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: ruff fixes on merge-resolved files - flux_text_encoder.py: drop unused typing.Union (F401) left by v5 import merge - huggingface.py: ruff format (wrap append(SimpleNamespace(...))) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(deps): pin transformers <5.6 (diffusers single-file CLIP incompat) transformers 5.6 flattened CLIPTextModel (removed the self.text_model wrapper, hoisted embeddings/encoder/final_layer_norm to the top level). diffusers' single-file checkpoint loader (create_diffusers_clip_model_from_ldm) still assumes the nested layout, so loading SD1.5 .safetensors checkpoints fails on 5.6+ with 'CLIPTextModel object has no attribute text_model' and, once that read is shimmed, 'Cannot copy out of meta tensor' (weights never populate the flattened model). Pin to >=5.5,<5.6 (last pre-flattening release) which keeps both the single-file and from_pretrained paths working. The invoke-ai/compel fork accepts any 5.x. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * @ chore(deps): replace compel fork with official compel 2.4.0 compel 2.4.0 (released 2026-05-30) merges the transformers-5 support that the invoke-ai fork carried (both descend from upstream PR #129), plus the maintainer-reviewed padding rework and added diffusers/T5 smoke coverage. Switch from the git fork to the PyPI release. - pyproject: compel git+main -> compel>=2.4.0,<3 - uv.lock: compel 2.3.1 (git 8f404b45) -> 2.4.0 (pypi) - transformers stays 5.5.4 (satisfies compel >=5,<6 and our <5.6 pin) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> @ * feat(ideogram4): backend + model-manager registration for Ideogram 4 Vendor the Apache-2.0 Ideogram 4 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler, nf4/fp8 quant loading) into invokeai/backend/ideogram4/, plus InvokeAI glue (Qwen3-VL text encoding, packed-input build, dual-branch Euler denoise loop). Register the model: BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config (detected via the Ideogram4Pipeline class name in model_index.json), and the Ideogram4DiffusersModel loader that loads both transformers as one Ideogram4TransformerPair submodel plus the Qwen3-VL encoder and VAE. Text-to-image only. * feat(ideogram4): Ideogram 4 backend — model manager, invocations, nf4 loading End-to-end text-to-image backend for Ideogram 4, validated through the real session runner. Vendors the Apache-2.0 reference model (DiT, FLUX2-style VAE, logit-normal flow-match scheduler) into invokeai/backend/ideogram4/ with InvokeAI glue. Registers BaseModelType.Ideogram4, Main_Diffusers_Ideogram4_Config, and the Ideogram4DiffusersModel loader (two transformers as one Ideogram4TransformerPair; Qwen3-VL encoder + VAE). Both transformers and the encoder load via InvokeLinearNF4 so they work with the partial-load cache. Adds Ideogram4ConditioningInfo/Field/Output and the model_loader/text_encoder/denoise/l2i invocations. Text-to-image only. * feat(ideogram4): frontend — Regions→JSON prompt, graph builder, UI Wires Ideogram 4 into the canvas/generate UI. buildIdeogram4Prompt assembles the structured JSON caption from the global prompt + Canvas Regional Guidance layers (each region → an obj element with a 0–1000 bbox + desc), with raw-JSON passthrough and a plain-text fallback when there are no regions. Adds buildIdeogram4Graph (text-to-image only, no negative prompt) and the enqueue switch. Structured captions use a static string node + a decoy positive-prompt node so the linear batch can't clobber the assembled JSON; plain text uses the real node so dynamic prompts/batching still work. Registers the 'ideogram-4' base (enums, color, names, model picker, grid size 16), a sampler-preset param (V4_QUALITY_48/V4_DEFAULT_20/V4_TURBO_12) replacing the steps/CFG controls, ParamIdeogram4SamplerPreset, and metadata recall. Regenerates schema.ts. * feat(ideogram4): advanced sampler overrides + color palette Advanced accordion now shows only Ideogram 4-relevant controls. Adds optional overrides of the sampler preset — steps, guidance scale (overrides the main gw, preserves the preset's polish tail), and schedule shift (mu) — plus a color palette editor that injects style_description.color_palette into the auto-built JSON caption (uppercase #RRGGBB, max 16, ignored for raw-JSON prompts). All are nullable (null = use preset), recallable from metadata, and the irrelevant controls (VAE, CLIP skip, CFG rescale, seamless, color compensation) are hidden for Ideogram 4. Backend denoise gains steps/guidance_scale/mu fields; schema.ts regenerated. * Use existing keys + fix select size * Update Readme * feat(ideogram4): add Ideogram 4 to starter models with non-commercial license hint - Implement the weight-only fp8 text-encoder load path (was NotImplementedError); validated against the real fp8 build + add CPU unit tests for the fp8 mechanism - Show Ideogram 4 handlers in the Recall Parameters tab - Recall the assembled JSON caption back into the positive prompt - Translate the metadata "Auto" values - Document Ideogram 4 (install/license, regional-guidance JSON prompting, presets) - Add Ideogram 4 nf4 (CUDA) + fp8 (any device) starter models and bundle - Surface the FLUX-style Non-Commercial License popover for Ideogram 4 models - Note the gated HuggingFace license requirement in the model descriptions * Chore Ruff * Chore Ruff * Chore OpenApi * Chore Knit * fix(deps): regenerate uv.lock to remove duplicate packages from bad merge * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * fix(ideogram4): make bitsandbytes import lazy in quantized_loading bitsandbytes has no macOS wheels and is excluded on darwin, but the module-level import broke test collection on macOS CI. Move the import into the two bnb-only functions and a TYPE_CHECKING block so the fp8 path imports without bitsandbytes installed. * Fix: address ideogram4 review (strict load, 1-step guidance, i18n, runtime caption) - Non-fp8 Ideogram 4 text-encoder load now validates the state dict: unexpected keys raise, missing keys warn (mirrors the fp8 helper) instead of silently accepting a partial load. - Guidance schedule: cap the polish tail at num_steps-1 so at least one main step always remains (the guidance_scale override was silently dropped at num_steps=1), and require steps >= 2 (backend field + frontend slider/marks). - Localize the Ideogram sampler-preset option labels via t() with the step count interpolated; add the three preset i18n keys. - Assemble the structured JSON caption at generation time in a new ideogram4_caption_builder node (Python port of buildIdeogram4Caption) instead of at graph-build time. The graph now wires the real prompt node -> caption builder -> text encoder and returns it as positivePrompt, so dynamic prompts / prompt batching vary the encoded caption (the decoy that dropped them is removed). The builder's output is wired to a new declared ideogram4_caption metadata field via an edge, so each batched image records its actual caption. Regenerates schema.ts for the new node + metadata field. Adds tests for caption assembly, the guidance schedule, and the graph wiring. * feat(ideogram4): step previews + document the model's built-in safety filter - Emit a low-res progress preview each denoise step so the forming image is visible during generation, like the other denoise nodes. Ideogram uses a FLUX.2-style 32-channel VAE, so the packed latent is unpatchified/denormalized (get_latent_norm) and run through the FLUX.2 latent->RGB factors — no full VAE decode per step. The denoise loop now hands the callback the packed grid latent. - Document Ideogram 4's built-in content safety filter in models.mdx: it lives in the model weights (not Invoke's NSFW checker, can't be disabled from Invoke) and false-positives on benign prompts; structured JSON prompts trip it less. * feat(ideogram4): avoid safety-filter false-positives + step previews + caption visibility The main fix: Ideogram 4's built-in safety filter (baked into the model weights) false-positives and returns an "Image blocked by safety filter" placeholder for "degenerate" captions — empirically, an empty `compositional_deconstruction.elements` list, or a single full-frame [0,0,1000,1000] element whose desc just repeats the high_level_description. Our assembly produced empty elements whenever the user drew no regions, so plain prompts were blocked. - Caption assembly (build_ideogram4_caption): - Always emit a structured JSON caption; never bare plain text (the filter false-positives far more on plain text). Raw-JSON pastes still pass through. - When there are no regions, synthesize one default element describing the whole scene from the prompt with a *partial* (non-full-frame) bbox [100,100,900,900]. This never yields an empty/degenerate elements list. Verified end-to-end against the model: the previously-blocked "golden retriever on a skateboard" now renders. - Metadata: always wire the caption builder's output to the ideogram4_caption metadata field, so the viewer's "Structured Caption" row shows the exact JSON that was encoded (not just the raw prompt) for every generation. - Denoise: emit a low-res progress preview each step (unpatchify + FLUX.2 latent->RGB factors, since Ideogram uses a FLUX.2-style 32-channel VAE) so the forming image is visible during generation, like the other denoise nodes. - Docs: document the model's built-in safety filter in models.mdx (it's not Invoke's NSFW checker, can't be disabled from Invoke, and false-positives). Updates the caption/graph tests accordingly (also fixes latent tsc errors in the graph-builder test's core_metadata / ideogram4_caption comparisons). * Chore typegen + openapi + Ruff * Fix Knit * fix(ideogram4): enforce steps>=2 client-side and validate region bbox Address review on the Ideogram 4 PR: - The backend denoise node requires steps >= 2, but the client still accepted ideogram4_steps = 1 in three places, letting a recalled or rehydrated value build a graph that violates the backend schema. Tighten the zod schema to min(2) with `.catch(null)` (a stale/out-of-range value normalizes to null = use the preset instead of failing the whole persisted slice), normalize dispatched values through the schema in setIdeogram4Steps, and refuse an out-of-range value in the ideogram4_steps metadata recall parser. The slider was already min=2. - Ideogram4Region.bbox was an unconstrained Optional[list[int]], so a workflow/API caller could pass a wrong-length or out-of-range box that the caption builder serialized verbatim into the structured prompt. Add a field validator requiring exactly four coordinates, each in 0..1000. Add tests for both: the region bbox contract (valid/None accepted; short, long, negative, and >1000 rejected) and the ideogram4Steps normalization (valid kept, null kept, stale 1 normalized to null on both dispatch and rehydrate). * fix(ideogram4): block unsupported canvas modes/bbox, warn dropped region inputs, constrain bbox Address the latest review on the Ideogram 4 PR: - Canvas readiness allowed unsupported generation modes: Ideogram 4 is txt2img-only (buildIdeogram4Graph asserts it), but a raster layer or inpaint mask makes the compositor pick img2img/outpaint/inpaint, failing only at graph build. Warn in getRasterLayerWarnings/getInpaintMaskWarnings for Ideogram 4 (these already flow into canvas readiness reasons), blocking enqueue up front. - Canvas readiness had no Ideogram 4 bbox check; the backend requires multiples of 16. Add the 16-grid check mirroring the other 16-grid models so an off-grid bbox (e.g. 1025x1024) is blocked instead of failing backend validation. - getRegionalGuidanceWarnings had no Ideogram 4 branch, so a region whose only input is a negative prompt, auto-negative, or reference image looked effective while the graph silently drops it. Warn those inputs are unsupported. - Ideogram4Region.bbox now also rejects inverted boxes (y_min <= y_max, x_min <= x_max). - The advanced-settings badge selector lumped Ideogram 4 into the generic branch, showing stale VAE/clip-skip/CFG-rescale/seamless badges for controls that are hidden for Ideogram. Exclude Ideogram 4 from that branch. - Model bbox as a constrained type (exactly 4 ints, each 0..1000) so the OpenAPI schema advertises minItems/maxItems and item minimum/maximum. Add readiness tests (bbox grid, raster/inpaint blocking, empty-layer allowance, regional-guidance negative/reference-image warnings) and bbox ordering-rejection tests. * fix(ideogram4): reject text encoders with weights left on the meta device _load_text_encoder() builds the encoder under accelerate.init_empty_weights() and previously downgraded missing keys to a warning (both the fp8 path via load_fp8_state_dict(strict=False) and the non-fp8 path). A missing non-tied weight therefore stayed on the meta device, so a bad or mismatched encoder appeared to load and only failed later during device movement or encoding. Add _verify_encoder_fully_materialized(): call tie_weights() to materialize tied weights from their source, then hard-fail if any parameter or buffer remains on the meta device. Wire it into both the fp8 and non-fp8 (incl. bnb-nf4) paths and drop the missing-key warning — genuinely missing non-tied weights are now caught as leftover meta tensors, while tied weights are tolerated. This is a state-based, path-agnostic check. Add tests: passes when fully materialized, raises on a leftover meta tensor from a missing non-tied weight, and tolerates a tied weight resolved by tie_weights(). * chore(ui): prettier formatting for AdvancedSettingsAccordion The Ideogram 4 badge-suppression branch left the wrapped block at its old indentation, failing `pnpm lint:prettier` in frontend-checks. Formatting only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Your Name <you@example.com> Co-authored-by: 4pointoh <97913726+4pointoh@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> | 1 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 1 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 1 个月前 |