Invoke is a leading creative engine for Stable Diffusion models, empowering professionals, artists, and enthusiasts to generate and create visual media using the latest AI-driven technologies. The solution offers an industry leading WebUI, and serves as the foundation for multiple commercial products.
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
Apply black | 3 年前 | |
ci: attach provenance and SBOM attestations to the published container (#9398) * ci: attach provenance and SBOM attestations to the published container * ci: restore trailing newline at end of file --------- Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> | 29 天前 | |
combine pytest.ini with pyproject.toml | 3 年前 | |
feat(deps): resolve torch from PyPI on linux/aarch64 to enable ARM64 installs and Docker builds (#9095) * feat(deps): exclude linux_aarch64 from PyTorch WHL-index sources PyTorch's WHL indexes (download.pytorch.org/whl/{cpu,cu128,rocm6.3}) only publish wheels for linux_x86_64 / win_amd64 / macOS arm64. On linux_aarch64, pinning torch==2.7.1+cpu (etc.) is unsatisfiable and `uv sync` fails. Gate the [tool.uv.sources] index overrides and the +cpu/+cu128/+rocm6.3 version pins behind a marker that excludes only linux_aarch64: sys_platform != 'linux' or platform_machine != 'aarch64' On linux_aarch64 the marker resolves to False, so uv falls back to the base `torch~=2.7.0` / `torchvision` declarations and pulls plain PyPI wheels. On every other platform -- Linux x86_64, Windows, and macOS -- it resolves to True, leaving existing behaviour unchanged. A negative-exclusion marker (only the broken platform) is used rather than a positive whitelist like `platform_machine == 'x86_64'`, which is fragile on Windows (Python reports AMD64 there) and would need to be extended for every newly-supported architecture. The standard docker/Dockerfile builds cleanly on aarch64 hosts with `--build-arg GPU_DRIVER=cpu`; no separate Dockerfile or build-time patching of pyproject.toml is needed. uv.lock is regenerated to match. * docs: update docker README/Dockerfile for arm64 CPU-only builds, clarify aarch64 wheel comments * docs: cover native (non-docker) ARM64 installs in system requirements, manual install, and dev environment guides --------- Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> | 1 个月前 | |
perf(api): make gzip compression level configurable (#9441) * fix(api): run gallery and search routes off the event loop The gallery list/name routes and the auth dependencies were declared `async def` while calling synchronous SQLite services, so their database work ran on the event loop. For its whole duration the process served no other request and delivered no socket.io event, which users experienced as the backend freezing mid-generation rather than as a slow gallery. Declaring them `def` hands them to Starlette's threadpool instead. Measured against a 200k-image, 1.7 GB database: the latency of an unrelated request issued while a gallery name query is in flight drops from 1662 ms to 4 ms (search) and from 2298 ms to 881 ms (no search). The queries themselves are unchanged; only the loop is freed. The residual 881 ms in the no-search case is response serialization of 202k items, which is tracked separately. Adds a regression test that stubs a blocking service call and asserts an unrelated route still answers during it, plus a contributor doc describing the rule. * perf(gallery): add a flat item-names endpoint and deprecate the legacy ones The name list that drives the virtualized gallery wrapped every entry in an object carrying a `kind` discriminator. Building those models cost 820ms of the 2225ms service call on a 200k-item library, and every consumer threw the field away — `itemRefsToNames` mapped it off immediately and each caller re-derived the kind from the file extension via `isVideoName`. Adds `GET /v1/gallery/item_names`, returning a flat name list in the same shape as the image-only `ImageNamesResult`. An optional `created_date` filter subsumes the separate by-date virtual-board route, so regular boards and virtual dates now share one endpoint, one cache and one query-args selector instead of a skipToken branch duplicated across the grid hook, range selection and both auto-select listeners. Measured on a 200k-image, 1.7 GB database: 2.51s -> 1.57s per request, 8.48 MB -> 3.85 MB of response, and the residual event-loop stall from serializing the response drops from 466ms to 102ms at p95. Existing integrations still call the old routes, so all five legacy name endpoints keep working and are marked `deprecated=True` with a pointer to the replacement. * perf(api): stop gzipping responses that are already compressed Starlette's GZipMiddleware compresses every response type except text/event-stream, so every image and video the gallery serves was being deflate-compressed a second time. Measured: a 1024x1024 PNG (3.00 MB) costs 52ms of event-loop time to gzip and comes back at 3.01 MB — larger than it went in; a 2048x2048 PNG costs 210ms for the same non-result. Compression runs on the event loop, so that time is a full stall of the process. With auto-switch enabled the UI fetches the full image after every generated image, so the cost lands repeatedly during a batch. Replaces it with a content-type-aware subclass that compresses an allowlist of text, JSON, XML and SVG responses and passes everything else through. The UI bundle and the API's JSON keep their compression unchanged. Lowering compresslevel is not an alternative for this case: on already-compressed input level 1 costs 51ms against level 9's 52ms, because deflate still scans the whole body. Making the level configurable is worthwhile for the *compressible* path and is tracked separately. Note for deployments: media responses no longer carry Content-Encoding: gzip. * feat(queue): add lightweight item summaries endpoint * build: unpin FastAPI and move to 0.141.1 The pin sat at 0.118.3 with a comment guessing the OpenAPI crash on 0.119 was "probably Invoke's [bug], because we are doing something unusual with AnyInvocation". It was not: fastapi/_compat/v2.py assumed every field mapping carries a `$ref` and raised KeyError otherwise. Upstream fixed it in 0.124.0 with no change needed here. Two later changes needed adapting to, both of which fail silently: - 0.130 emits `contentMediaType: application/octet-stream` instead of `format: binary` for file uploads. typegen.js mapped only the latter to `Blob`, so upload call sites would have started typing their `File` argument as `string`. It now maps both. - 0.141 keeps an included router as a single node in `app.routes` instead of copying its routes into it. The default-deny auth guard walked `app.routes` looking for APIRoute instances and found 2 of 197 — passing while inspecting almost nothing. It now walks `iter_route_contexts`, the traversal FastAPI's own OpenAPI generation uses, and asserts a floor on the route count so going blind fails loudly instead. Schema changes are limited to ValidationError gaining the optional `input`/`ctx` fields; upload fields still resolve to Blob. Starlette stays at 0.48.0. * perf(api): run every synchronous route handler off the event loop Package A converted the eight gallery and search routes that caused the reported multi-minute stalls. The same defect was present across the rest of the API: 167 route handlers were declared `async def` while awaiting nothing, so their synchronous service calls ran on the event loop. Each one stalls the entire process for its duration - no other request served, no socket.io event delivered - which is why the symptom looked like the application freezing rather than one slow endpoint. Candidates were identified by AST rather than by hand: `async def` route handlers with no `await`, `async with` or `async for` anywhere in the body, cross-checked for references to asyncio, anyio or the loop. Two flagged candidates were false positives (both the word "loop" in a comment). The diff is 167 signature lines plus one signature that ruff collapsed onto a single line once `async ` was removed. Adds tests/app/routers/test_no_blocking_async_routes.py, which enforces the rule for every handler including ones written later - a per-route test cannot cover a route that does not exist yet, and this failure mode is invisible until a user has a large enough library to notice. Two tests that invoked route handlers directly were updated to call them as the plain functions they now are. * Docs Changes * Chore openapi * feat(api): make gzip compression level configurable Starlette's GZipMiddleware compresses at level 9, the slowest setting, and there was no way to change that. Compression runs on the event loop, so it stalls every other request and every socket.io event for its duration. Measured on the flat image-name list of a 200k-image library (8.48 MB of JSON): level 1 takes 16.4ms and returns 6.1% of the input, level 9 takes 90.2ms and returns 5.7%. Level 9 spends 5.5x the event-loop time to save 0.4 percentage points of bandwidth — a poor deal for a locally-served app, where the saved bandwidth is worthless and the stall is not. Add `gzip_compresslevel` (range 0-9). The default stays at 9, so nothing changes for existing installs; users who feel the stall on a large library can now lower it, and the docs explain when that is worthwhile. At 0 the middleware is left out entirely rather than installed at level 0, so responses skip the responder instead of being buffered and re-emitted as a stored-only gzip stream. Deployments behind a compressing reverse proxy want that. This does not help the media case — on incompressible input level 1 costs about the same as level 9 — which is why the content-type exclusion remains the fix for that path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(api): rename gzip_compresslevel to http_compression_level The old name read like an image setting, easy to confuse with pil_compress_level. The new one says which layer it acts on. Renames the config field, its env var (INVOKEAI_HTTP_COMPRESSION_LEVEL), the docs, the tests and the generated schemas. No behaviour change: the default stays 9 and 0 still leaves the middleware uninstalled. The setting has never been in a release, so no deprecation alias is needed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(api): chunk queue-summary id lookups; document the compression env var Addresses review feedback on #9441. - get_queue_item_summaries_by_ids built one bound parameter per requested id, so a request above SQLite's SQLITE_MAX_VARIABLE_NUMBER (32766 on modern builds, 999 on older ones) failed with an OperationalError and answered 500. Deduplicate the ids and query in chunks of 900, reassembling in the order the caller asked for. Follows the existing UserService.get_many pattern. - The config docs did not name the environment variable. `http_compression_level` is settable as INVOKEAI_HTTP_COMPRESSION_LEVEL; the default of 9 is unchanged and was already correct in the source and in the generated settings reference. - Pin the Starlette responder contract that _ContentTypeAwareGZipResponder depends on (the exclusion flag exists; http.response.start is buffered, not forwarded) so an upgrade that breaks it fails with a named cause instead of silently gzipping PNGs. * Chore typegen * chore: add http_compression_level to the generated API schema The main merge resolved openapi.json and schema.ts in favour of main, which drops this branch's own contribution: http_compression_level is a field on InvokeAIAppConfig, so it lives inside a schema's properties rather than adding a path or a schema of its own. Comparing the two files by path and schema *names* - which is how the merge was checked - cannot see a difference at that depth, so the loss went unnoticed until openapi-checks and typegen-checks failed. Restores the property and the InvokeAIAppConfig description that lists it. schema.ts is regenerated from the corrected openapi.json rather than hand-edited. * chore: regenerate schema.ts with this branch's typegen The previous regeneration ran the typegen script from a different checkout, which only maps FastAPI's pre-0.130 'format: binary' to Blob. Since the schema is now generated by FastAPI 0.141, upload fields arrive as 'contentMediaType: application/octet-stream' and came out as string, breaking StylePresetImportButton with TS2345 and leaving typegen-checks red. Regenerated with the script and the locked openapi-typescript from this branch. tsc --noEmit passes and a second run reproduces the file byte for byte. --------- Co-authored-by: JPPhoto <jpollack@jpollackphoto.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com> | 18 天前 | |
feat(fp8): enable FP8 storage for Anima (#9415) * feat(fp8): enable FP8 storage for Z-Image Z-Image was excluded from FP8 storage in #8945 because diffusers' enable_layerwise_casting() was called with the global torch dtype (fp16) while Z-Image loads in bf16: skipped modules stayed bf16, hooked ones produced fp16, and attention crashed. That root cause was fixed later in the same PR — the compute dtype now comes from the model's own parameters — so the exclusion is obsolete. Removing it alone is not enough. Our hook-based cast (#9231) dropped one thing diffusers' enable_layerwise_casting() did: honoring the model's declared _skip_layerwise_casting_patterns. Z-Image needs it, and not for quality — TimestepEmbedder.forward reads self.mlp[0].weight.dtype and casts its *input* to it. With an fp8 weight the input becomes float8 before our pre-hook restores the weight, and F.linear dies with: RuntimeError: "addmm_cuda" not implemented for 'Float8_e4m3fn' which is why ZImageTransformer2DModel declares ['t_embedder', 'cap_embedder']. _apply_fp8_to_nn_module now takes extra_skip_patterns and the caller passes the model's list. For other models this is a strict superset of our defaults (FLUX/SD3 pos_embed+norm, UNet norm, CogView4 also proj_out), so it only ever skips more. Also wire the cast into ZImageCheckpointModel: only the diffusers loader called it, so the toggle was a silent no-op for single-file Z-Image models even though both paths build the same ZImageTransformer2DModel. Tested end to end on CUDA: transformer resident VRAM drops from ~11.5GB to 5880MB for both Z-Image-Turbo (diffusers) and Z-Image-Turbo (checkpoint, 14.37GB file), with clean output images in both cases. * Chore openapi * feat(fp8): enable FP8 storage for Anima The fp8_storage toggle was shown for Anima main models but did nothing: AnimaCheckpointModel never called _apply_fp8_layerwise_casting. Wire it in — the state dict is cast to a single model_dtype before load_state_dict, so the layerwise cast has one unambiguous compute dtype to restore to. Wiring alone renders a heavily dithered image with no fine detail. The cause is t_embedder: it produces the adaln_lora conditioning consumed by every block, so casting it to FP8 corrupts every token everywhere. None of the generic skip patterns match it — they target diffusers' module names (norm, pos_embed, patch_embed, proj_in/out) and this architecture names things differently. AnimaTransformer now declares _skip_layerwise_casting_patterns, the same attribute diffusers models use, so the loader needs no special-casing. Measured on CUDA, same seed/steps/CFG each run: casting nothing = broken at 1994MB; t_embedder alone = clean at 2010MB; adding x_embedder and final_layer changes nothing further (2012MB) and is kept as margin on the I/O layers; adaln_modulation was tested too and is deliberately not listed — it costs 168MB and made no difference. Against a bf16 reference (3988MB) the FP8 result keeps the same composition and loses only a little micro-detail. * test(fp8): drop the Z-Image entry from the exclusion parametrize main added a device-probe parametrize listing Z-Image as an excluded model. This branch removes that exclusion, so the entry contradicts `test_should_use_fp8_allows_z_image` and the case now returns the probe's value instead of False. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(fp8): pin Anima's skip patterns to real modules and guard the wiring Review follow-ups for #9415. Add `tests/.../test_anima_fp8_wiring.py`. Deleting the `_apply_fp8_layerwise_casting` call from the Anima single-file loader previously left the whole model_manager and anima suites green, so the dead `fp8_storage` toggle this PR fixes could come straight back with CI passing. The new boundary test fails on that mutation. The pattern test now instantiates the real `AnimaTransformer` under `accelerate.init_empty_weights()` and pins all three declared patterns to actual dotted module paths, instead of asserting a string is in a list against a hand-built stand-in. A second test records that `_FP8_DEFAULT_SKIP_PATTERNS` covers zero modules in this architecture, so the declared list is demonstrably not redundant. Lift the transformer kwargs to `ANIMA_TRANSFORMER_CONFIG` so tests build the real graph without duplicating them, mirroring `KREA2_TRANSFORMER_CONFIG`. Correct the skip-list comment. `adaln_modulation` "made no difference" was not supported by measurement: relative L2 against bf16 on a single forward goes 0.134 -> 0.091 when it is skipped, making it the largest remaining error source. The 168MB call still stands, but it rests on a 35-step A/B showing no visible difference, and the comment now says so. Also note that most of what the `final_layer` entry shields is `final_layer.adaln_modulation.*` (1.57 of 1.70M params). Stop offering FP8 storage for Anima LLLite ControlNets in the model manager. `AnimaControlNetLLLiteModel` never calls the layerwise cast, so the toggle was rendered and inert; at 16-63MB per adapter, hiding it beats wiring it. * fix(fp8): stop the hidden Anima ControlNet fp8 toggle from re-persisting Two fixes from an adversarial review of the merge: - `ControlAdapterModelDefaultSettings` hid the FP8 storage control for Anima LLLite adapters but kept sending its value. react-hook-form keeps unrendered fields in `defaultValues` (`shouldUnregister` defaults to false), so a value persisted before the control was hidden was re-sent verbatim on every save, with no UI left to clear it. Null it out wherever the control is hidden. - `test_single_file_loader_applies_fp8_layerwise_casting` passed `fp8_storage` as a top-level kwarg to `model_construct`. It is not a field of `Main_Checkpoint_Anima_Config` and the model has no `extra="allow"`, so pydantic silently discarded it and `default_settings` stayed `None` -- the toggle was off in the test that exists to prove the toggle is wired up. Build a real `MainModelDefaultSettings(fp8_storage=True)` instead. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> | 11 天前 | |
Add opt-in low-VRAM mode for Wan generation (#9462) * feat(video): optimize Wan memory usage * chore: openapi schema * Added additional optimizations * fix(wan): address video memory optimization review * test(cache): keep VRAM budget test CPU-safe * fix(wan): address memory optimization review * fix(wan): calibrate VAE from checkpoint files * feat(wan): add tiled VAE calibration mode * fix(wan): align VAE calibration with estimator | 19 天前 | |
feat(fp8): enable FP8 storage for Anima (#9415) * feat(fp8): enable FP8 storage for Z-Image Z-Image was excluded from FP8 storage in #8945 because diffusers' enable_layerwise_casting() was called with the global torch dtype (fp16) while Z-Image loads in bf16: skipped modules stayed bf16, hooked ones produced fp16, and attention crashed. That root cause was fixed later in the same PR — the compute dtype now comes from the model's own parameters — so the exclusion is obsolete. Removing it alone is not enough. Our hook-based cast (#9231) dropped one thing diffusers' enable_layerwise_casting() did: honoring the model's declared _skip_layerwise_casting_patterns. Z-Image needs it, and not for quality — TimestepEmbedder.forward reads self.mlp[0].weight.dtype and casts its *input* to it. With an fp8 weight the input becomes float8 before our pre-hook restores the weight, and F.linear dies with: RuntimeError: "addmm_cuda" not implemented for 'Float8_e4m3fn' which is why ZImageTransformer2DModel declares ['t_embedder', 'cap_embedder']. _apply_fp8_to_nn_module now takes extra_skip_patterns and the caller passes the model's list. For other models this is a strict superset of our defaults (FLUX/SD3 pos_embed+norm, UNet norm, CogView4 also proj_out), so it only ever skips more. Also wire the cast into ZImageCheckpointModel: only the diffusers loader called it, so the toggle was a silent no-op for single-file Z-Image models even though both paths build the same ZImageTransformer2DModel. Tested end to end on CUDA: transformer resident VRAM drops from ~11.5GB to 5880MB for both Z-Image-Turbo (diffusers) and Z-Image-Turbo (checkpoint, 14.37GB file), with clean output images in both cases. * Chore openapi * feat(fp8): enable FP8 storage for Anima The fp8_storage toggle was shown for Anima main models but did nothing: AnimaCheckpointModel never called _apply_fp8_layerwise_casting. Wire it in — the state dict is cast to a single model_dtype before load_state_dict, so the layerwise cast has one unambiguous compute dtype to restore to. Wiring alone renders a heavily dithered image with no fine detail. The cause is t_embedder: it produces the adaln_lora conditioning consumed by every block, so casting it to FP8 corrupts every token everywhere. None of the generic skip patterns match it — they target diffusers' module names (norm, pos_embed, patch_embed, proj_in/out) and this architecture names things differently. AnimaTransformer now declares _skip_layerwise_casting_patterns, the same attribute diffusers models use, so the loader needs no special-casing. Measured on CUDA, same seed/steps/CFG each run: casting nothing = broken at 1994MB; t_embedder alone = clean at 2010MB; adding x_embedder and final_layer changes nothing further (2012MB) and is kept as margin on the I/O layers; adaln_modulation was tested too and is deliberately not listed — it costs 168MB and made no difference. Against a bf16 reference (3988MB) the FP8 result keeps the same composition and loses only a little micro-detail. * test(fp8): drop the Z-Image entry from the exclusion parametrize main added a device-probe parametrize listing Z-Image as an excluded model. This branch removes that exclusion, so the entry contradicts `test_should_use_fp8_allows_z_image` and the case now returns the probe's value instead of False. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(fp8): pin Anima's skip patterns to real modules and guard the wiring Review follow-ups for #9415. Add `tests/.../test_anima_fp8_wiring.py`. Deleting the `_apply_fp8_layerwise_casting` call from the Anima single-file loader previously left the whole model_manager and anima suites green, so the dead `fp8_storage` toggle this PR fixes could come straight back with CI passing. The new boundary test fails on that mutation. The pattern test now instantiates the real `AnimaTransformer` under `accelerate.init_empty_weights()` and pins all three declared patterns to actual dotted module paths, instead of asserting a string is in a list against a hand-built stand-in. A second test records that `_FP8_DEFAULT_SKIP_PATTERNS` covers zero modules in this architecture, so the declared list is demonstrably not redundant. Lift the transformer kwargs to `ANIMA_TRANSFORMER_CONFIG` so tests build the real graph without duplicating them, mirroring `KREA2_TRANSFORMER_CONFIG`. Correct the skip-list comment. `adaln_modulation` "made no difference" was not supported by measurement: relative L2 against bf16 on a single forward goes 0.134 -> 0.091 when it is skipped, making it the largest remaining error source. The 168MB call still stands, but it rests on a 35-step A/B showing no visible difference, and the comment now says so. Also note that most of what the `final_layer` entry shields is `final_layer.adaln_modulation.*` (1.57 of 1.70M params). Stop offering FP8 storage for Anima LLLite ControlNets in the model manager. `AnimaControlNetLLLiteModel` never calls the layerwise cast, so the toggle was rendered and inert; at 16-63MB per adapter, hiding it beats wiring it. * fix(fp8): stop the hidden Anima ControlNet fp8 toggle from re-persisting Two fixes from an adversarial review of the merge: - `ControlAdapterModelDefaultSettings` hid the FP8 storage control for Anima LLLite adapters but kept sending its value. react-hook-form keeps unrendered fields in `defaultValues` (`shouldUnregister` defaults to false), so a value persisted before the control was hidden was re-sent verbatim on every save, with no UI left to clear it. Null it out wherever the control is hidden. - `test_single_file_loader_applies_fp8_layerwise_casting` passed `fp8_storage` as a top-level kwarg to `model_construct`. It is not a field of `Main_Checkpoint_Anima_Config` and the model has no `extra="allow"`, so pydantic silently discarded it and `default_settings` stayed `None` -- the toggle was off in the test that exists to prove the toggle is wired up. Build a real `MainModelDefaultSettings(fp8_storage=True)` instead. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> | 11 天前 | |
refactor Dockerfile; get rid of multi-stage build; upgrade to python 3.12 | 1 年前 | |
Merge dev into main for 2.2.0 (#1642) * Fixes inpainting + code cleanup * Disable stage info in Inpainting Tab * Mask Brush Preview now always at 0.5 opacity The new mask is only visible properly at max opacity but at max opacity the brush preview becomes fully opaque blocking the view. So the mask brush preview no remains at 0.5 no matter what the Brush opacity is. * Remove save button from Canvas Controls (cleanup) * Implements invert mask * Changes "Invert Mask" to "Preserve Masked Areas" * Fixes (?) spacebar issues * Patches redux-persist and redux-deep-persist with debounced persists Our app changes redux state very, very often. As our undo/redo history grows, the calls to persist state start to take in the 100ms range, due to a the deep cloning of the history. This causes very noticeable performance lag. The deep cloning is required because we need to blacklist certain items in redux from being persisted (e.g. the app's connection status). Debouncing the whole process of persistence is a simple and effective solution. Unfortunately, `redux-persist` dropped `debounce` between v4 and v5, replacing it with `throttle`. `throttle`, instead of delaying the expensive action until a period of X ms of inactivity, simply ensures the action is executed at least every X ms. Of course, this does not fix our performance issue. The patch is very simple. It adds a `debounce` argument - a number of milliseconds - and debounces `redux-persist`'s `update()` method (provided by `createPersistoid`) by that many ms. Before this, I also tried writing a custom storage adapter for `redux-persist` to debounce the calls to `localStorage.setItem()`. While this worked and was far less invasive, it doesn't actually address the issue. It turns out `setItem()` is a very fast part of the process. We use `redux-deep-persist` to simplify the `redux-persist` configuration, which can get complicated when you need to blacklist or whitelist deeply nested state. There is also a patch here for that library because it uses the same types as `redux-persist`. Unfortunately, the last release of `redux-persist` used a package `flat-stream` which was malicious and has been removed from npm. The latest commits to `redux-persist` (about 1 year ago) do not build; we cannot use the master branch. And between the last release and last commit, the changes have all been breaking. Patching this last release (about 3 years old at this point) directly is far simpler than attempting to fix the upstream library's master branch or figuring out an alternative to the malicious and now non-existent dependency. * Adds debouncing * Fixes AttributeError: 'dict' object has no attribute 'invert_mask' * Updates package.json to use redux-persist patches * Attempts to fix redux-persist debounce patch * Fixes undo/redo * Fixes invert mask * Debounce > 300ms * Limits history to 256 for each of undo and redo * Canvas styling * Hotkeys improvement * Add Metadata To Viewer * Increases CFG Scale max to 200 * Fix gallery width size for Outpainting Also fixes the canvas resizing failing n fast pushes * Fixes disappearing canvas grid lines * Adds staging area * Fixes "use all" not setting variationAmount Now sets to 0 when the image had variations. * Builds fresh bundle * Outpainting tab loads to empty canvas instead of upload * Fixes wonky canvas layer ordering & compositing * Fixes error on inpainting paste back `TypeError: 'float' object cannot be interpreted as an integer` * Hides staging area outline on mouseover prev/next * Fixes inpainting not doing img2img when no mask * Fixes bbox not resizing in outpainting if partially off screen * Fixes crashes during iterative outpaint. Still doesn't work correctly though. * Fix iterative outpainting by restoring original images * Moves image uploading to HTTP - It all seems to work fine - A lot of cleanup is still needed - Logging needs to be added - May need types to be reviewed * Fixes: outpainting temp images show in gallery * WIP refactor to unified canvas * Removes console.log from redux-persist patch * Initial unification of canvas * Removes all references to split inpainting/outpainting canvas * Add patchmatch and infill_method parameter to prompt2image (options are 'patchmatch' or 'tile'). * Fixes app after removing in/out-painting refs * Rebases on dev, updates new env files w/ patchmatch * Organises features/canvas * Fixes bounding box ending up offscreen * Organises features/canvas * Stops unnecessary canvas rescales on gallery state change * Fixes 2px layout shift on toggle canvas lock * Clips lines drawn while canvas locked When drawing with the locked canvas, if a brush stroke gets too close to the edge of the canvas and its stroke would extend past the edge of the canvas, the edge of that stroke will be seen after unlocking the canvas. This could cause a problem if you unlock the canvas and now have a bunch of strokes just outside the init image area, which are far back in undo history and you cannot easily erase. With this change, lines drawn while the canvas is locked get clipped to the initial image bbox, fixing this issue. Additionally, the merge and save to gallery functions have been updated to respect the initial image bbox so they function how you'd expect. * Fixes reset canvas view when locked * Fixes send to buttons * Fixes bounding box not being rounded to 64 * Abandons "inpainting" canvas lock * Fixes save to gallery including empty area, adds download and copy image * Fix Current Image display background going over image bounds * Sets status immediately when clicking Invoke * Adds hotkeys and refactors sharing of konva instances Adds hotkeys to canvas. As part of this change, the access to konva instance objects was refactored: Previously closure'd refs were used to indirectly get access to the konva instances outside of react components. Now, a getter and setter function are used to provide access directly to the konva objects. * Updates hotkeys * Fixes canvas showing spinner on first load Also adds good default canvas scale and positioning when no image is on it * Fixes possible hang on MaskCompositer * Improves behaviour when setting init canvas image/reset view * Resets bounding box coords/dims when no image present * Disables canvas actions which cannot be done during processing * Adds useToastWatcher hook - Dispatch an `addToast` action with standard Chakra toast options object to add a toast to the toastQueue - The hook is called in App.tsx and just useEffect's w/ toastQueue as dependency to create the toasts - So now you can add toasts anywhere you have access to `dispatch`, which includes middleware and thunks - Adds first usage of this for the save image buttons in canvas * Update Hotkey Info Add missing tooltip hotkeys and update the hotkeys modal to reflect the new hotkeys for the Unified Canvas. * Fix theme changer not displaying current theme on page refresh * Fix tab count in hotkeys panel * Unify Brush and Eraser Sizes * Fix staging area display toggle not working * Staging Area delete button is now red So it doesnt feel blended into to the rest of them. * Revert "Fix theme changer not displaying current theme on page refresh" This reverts commit 903edfb803e743500242589ff093a8a8a0912726. * Add arguments to use SSL to webserver * Integrates #1487 - touch events Need to add: - Pinch zoom - Touch-specific handling (some things aren't quite right) * Refactors upload-related async thunks - Now standard thunks instead of RTK createAsyncThunk() - Adds toasts for all canvas upload-related actions * Reorganises app file structure * Fixes Canvas Auto Save to Gallery * Fixes staging area outline * Adds staging area hotkeys, disables gallery left/right when staging * Fixes Use All Parameters * Fix metadata viewer image url length when viewing intermediate * Fixes intermediate images being tiny in txt2img/img2img * Removes stale code * Improves canvas status text and adds option to toggle debug info * Fixes paste image to upload * Adds model drop-down to site header * Adds theme changer popover * Fix missing key on ThemeChanger map * Fixes stage position changing on zoom * Hotkey Cleanup - Viewer is now Z - Canvas Move tool is V - sync with PS - Removed some unused hotkeys * Fix canvas resizing when both options and gallery are unpinned * Implements thumbnails for gallery - Thumbnails are saved whenever an image is saved, and when gallery requests images from server - Thumbnails saved at original image aspect ratio with width of 128px as WEBP - If the thumbnail property of an image is unavailable for whatever reason, the image's full size URL is used instead * Saves thumbnails to separate thumbnails directory * Thumbnail size = 256px * Fix Lightbox Issues * Disables canvas image saving functions when processing * Fix index error on going past last image in Gallery * WIP - Lightbox Fixes Still need to fix the images not being centered on load when the image res changes * Fixes another similar index error, simplifies logic * Reworks canvas toolbar * Fixes canvas toolbar upload button * Cleans up IAICanvasStatusText * Improves metadata handling, fixes #1450 - Removes model list from metadata - Adds generation's specific model to metadata - Displays full metadata in JSON viewer * Gracefully handles corrupted images; fixes #1486 - App does not crash if corrupted image loaded - Error is displayed in the UI console and CLI output if an image cannot be loaded * Adds hotkey to reset canvas interaction state If the canvas' interaction state (e.g. isMovingBoundingBox, isDrawing, etc) get stuck somehow, user can press Escape to reset the state. * Removes stray console.log() * Fixes bug causing gallery to close on context menu open * Minor bugfixes - When doing long-running canvas image exporting actions, display indeterminate progress bar - Fix staging area image outline not displaying after committing/discarding results * Removes unused imports * Fixes repo root .gitignore ignoring frontend things * Builds fresh bundle * Styling updates * Removes reasonsWhyNotReady The popover doesn't play well with the button being disabled, and I don't think adds any value. * Image gallery resize/style tweaks * Styles buttons for clearing canvas history and mask * First pass on Canvas options panel * Fixes bug where discarding staged images results in loss of history * Adds Save to Gallery button to staging toolbar * Rearrange some canvas toolbar icons Put brush stuff together and canvas movement stuff together * Fix gallery maxwidth on unified canvas * Update Layer hotkey display to UI * Adds option to crop to bounding box on save * Masking option tweaks * Crop to Bounding Box > Save Box Region Only * Adds clear temp folder * Updates mask options popover behavior * Builds fresh bundle * Fix styling on alert modals * Fix input checkbox styling being incorrect on light theme * Styling fixes * Improves gallery resize behaviour * Cap gallery size on canvas tab so it doesnt overflow * Fixes bug when postprocessing image with no metadata * Adds IAIAlertDialog component * Moves Loopback to app settings * Fixes metadata viewer not showing metadata after refresh Also adds Dream-style prompt to metadata * Adds outpainting specific options * Linting * Fixes gallery width on lightbox, fixes gallery button expansion * Builds fresh bundle * Fix Lightbox images of different res not centering * Update feature tooltip text * Highlight mask icon when on mask layer * Fix gallery not resizing correctly on open and close * Add loopback to just img2img. Remove from settings. * Fix to gallery resizing * Removes Advanced checkbox, cleans up options panel for unified canvas * Minor styling fixes to new options panel layout * Styling Updates * Adds infill method * Tab Styling Fixes * memoize outpainting options * Fix unnecessary gallery re-renders * Isolate Cursor Pos debug text on canvas to prevent rerenders * Fixes missing postprocessed image metadata before refresh * Builds fresh bundle * Fix rerenders on model select * Floating panel re-render fix * Simplify fullscreen hotkey selector * Add Training WIP Tab * Adds Training icon * Move full screen hotkey to floating to prevent tab rerenders * Adds single-column gallery layout * Fixes crash on cancel with intermediates enabled, fixes #1416 * Updates npm dependencies * Fixes img2img attempting inpaint when init image has transparency * Fixes missing threshold and perlin parameters in metadata viewer * Renames "Threshold" > "Noise Threshold" * Fixes postprocessing not being disabled when clicking use all * Builds fresh bundle * Adds color picker * Lints & builds fresh bundle * Fixes iterations being disabled when seed random & variations are off * Un-floors cursor position * Changes color picker preview to circles * Fixes variation params not set correctly when recalled * Fixes invoke hotkey not working in input fields * Simplifies Accordion Prep for adding reset buttons for each section * Fixes mask brush preview color * Committing color picker color changes tool to brush * Color picker does not overwrite user-selected alpha * Adds brush color alpha hotkey * Lints * Removes force_outpaint param * Add inpaint size options to inpaint at a larger size than the actual inpaint image, then scale back down for recombination * Bug fix for inpaint size * Adds inpaint size (as scale bounding box) to UI * Adds auto-scaling for inpaint size * Improves scaled bbox display logic * Fixes bug with clear mask and history * Fixes shouldShowStagingImage not resetting to true on commit * Builds fresh bundle * Fixes canvas failing to scale on first run * Builds fresh bundle * Fixes unnecessary canvas scaling * Adds gallery drag and drop to img2img/canvas * Builds fresh bundle * Fix desktop mode being broken with new versions of flaskwebgui * Fixes canvas dimensions not setting on first load * Builds fresh bundle * stop crash on !import_models call on model inside rootdir - addresses bug report #1546 * prevent "!switch state gets confused if model switching fails" - If !switch were to fail on a particular model, then generate got confused and wouldn't try again until you switch to a different working model and back again. - This commit fixes and closes #1547 * Revert "make the docstring more readable and improve the list_models logic" This reverts commit 248068fe5d57b5639ea7a87ee6cbf023104d957d. * fix model cache path * also set fail-fast to it's default (true) in this way the whole action fails if one job fails this should unblock the runners!!! * fix output path for Archive results * disable checks for python 3.9 * Update-requirements and test-invoke-pip workflow (#1574) * update requirements files * update test-invoke-pip workflow * move requirements-mkdocs.txt to docs folder (#1575) * move requirements-mkdocs.txt to docs folder * update copyright * Fixes outpainting with resized inpaint size * Interactive configuration (#1517) * Update scripts/configure_invokeai.py prevent crash if output exists Co-authored-by: psychedelicious <4822129+psychedelicious@users.noreply.github.com> * implement changes requested by reviews * default to correct root and output directory on Windows systems - Previously the script was relying on the readline buffer editing feature to set up the correct default. But this feature doesn't exist on windows. - This commit detects when user typed return with an empty directory value and replaces with the default directory. * improved readability of directory choices * Update scripts/configure_invokeai.py Co-authored-by: psychedelicious <4822129+psychedelicious@users.noreply.github.com> * better error reporting at startup - If user tries to run the script outside of the repo or runtime directory, a more informative message will appear explaining the problem. Co-authored-by: psychedelicious <4822129+psychedelicious@users.noreply.github.com> * Embedding merging (#1526) * add whole <style token> to vocab for concept library embeddings * add ability to load multiple concept .bin files * make --log_tokenization respect custom tokens * start working on concept downloading system * preliminary support for dynamic loading and merging of multiple embedded models - The embedding_manager is now enhanced with ldm.invoke.concepts_lib, which handles dynamic downloading and caching of embedded models from the Hugging Face concepts library (https://huggingface.co/sd-concepts-library) - Downloading of a embedded model is triggered by the presence of one or more <concept> tags in the prompt. - Once the embedded model is downloaded, its trigger phrase will be loaded into the embedding manager and the prompt's <concept> tag will be replaced with the <trigger_phrase> - The downloaded model stays on disk for fast loading later. - The CLI autocomplete will complete partial <concept> tags for you. Type a '<' and hit tab to get all ~700 concepts. BUGS AND LIMITATIONS: - MODEL NAME VS TRIGGER PHRASE You must use the name of the concept embed model from the SD library, and not the trigger phrase itself. Usually these are the same, but not always. For example, the model named "hoi4-leaders" corresponds to the trigger "<HOI4-Leader>" One reason for this design choice is that there is no apparent constraint on the uniqueness of the trigger phrases and one trigger phrase may map onto multiple models. So we use the model name instead. The second reason is that there is no way I know of to search Hugging Face for models with certain trigger phrases. So we'd have to download all 700 models to index the phrases. The problem this presents is that this may confuse users, who will want to reuse prompts from distributions that use the trigger phrase directly. Usually this will work, but not always. - WON'T WORK ON A FIREWALLED SYSTEM If the host running IAI has no internet connection, it can't download the concept libraries. I will add a script that allows users to preload a list of concept models. - BUG IN PROMPT REPLACEMENT WHEN MODEL NOT FOUND There's a small bug that occurs when the user provides an invalid model name. The <concept> gets replaced with <None> in the prompt. * fix loading .pt embeddings; allow multi-vector embeddings; warn on dupes * simplify replacement logic and remove cuda assumption * download list of concepts from hugging face * remove misleading customization of '*' placeholder the existing code as-is did not do anything; unclear what it was supposed to do. the obvious alternative -- setting using 'placeholder_strings' instead of 'placeholder_tokens' to match model.params.personalization_config.params.placeholder_strings -- caused a crash. i think this is because the passed string also needed to be handed over on init of the PersonalizedBase as the 'placeholder_token' argument. this is weird config dict magic and i don't want to touch it. put a breakpoint in personalzied.py line 116 (top of PersonalizedBase.__init__) if you want to have a crack at it yourself. * address all the issues raised by damian0815 in review of PR #1526 * actually resize the token_embeddings * multiple improvements to the concept loader based on code reviews 1. Activated the --embedding_directory option (alias --embedding_path) to load a single embedding or an entire directory of embeddings at startup time. 2. Can turn off automatic loading of embeddings using --no-embeddings. 3. Embedding checkpoints are scanned with the pickle scanner. 4. More informative error messages when a concept can't be loaded due either to a 404 not found error or a network error. * autocomplete terms end with ">" now * fix startup error and network unreachable 1. If the .invokeai file does not contain the --root and --outdir options, invoke.py will now fix it. 2. Catch and handle network problems when downloading hugging face textual inversion concepts. * fix misformatted error string Co-authored-by: Damian Stewart <d@damianstewart.com> * model_cache.py: fix list_models Signed-off-by: devops117 <55235206+devops117@users.noreply.github.com> * add statement of values (#1584) * this adds the Statement of Values Google doc source = https://docs.google.com/document/d/1-PrUKDJcxy8OyNGc8CyiHhv2VgLvjt7LRGlEpbg1nmQ/edit?usp=sharing * Fix heading * Update InvokeAI_Statement_of_Values.md * Update InvokeAI_Statement_of_Values.md * Update InvokeAI_Statement_of_Values.md * Update InvokeAI_Statement_of_Values.md * Update InvokeAI_Statement_of_Values.md * add keturn and mauwii to the team member list * Fix punctuation * this adds the Statement of Values Google doc source = https://docs.google.com/document/d/1-PrUKDJcxy8OyNGc8CyiHhv2VgLvjt7LRGlEpbg1nmQ/edit?usp=sharing * add keturn and mauwii to the team member list * fix formating - make sub bullets use * (decide to all use - or *) - indent sub bullets Sorry, first only looked at the code version and found this only after looking at the markdown rendered version * use multiparagraph numbered sections * Break up Statement Of Values as per comments on #1584 * remove duplicated word, reduce vagueness it's important not to overstate how many artists we are consulting. * fix typo (sorry blessedcoolant) Co-authored-by: mauwii <Mauwii@outlook.de> Co-authored-by: damian <git@damianstewart.com> * update dockerfile (#1551) * update dockerfile * remove not existing file from .dockerignore * remove bloat and unecesary step also use --no-cache-dir for pip install image is now close to 2GB * make Dockerfile a variable * set base image to `ubuntu:22.10` * add build-essential * link outputs folder for persistence * update tag variable * update docs * fix not customizeable build args, add reqs output * !model_import autocompletes in ROOTDIR * Adds psychedelicious to statement of values signature (#1602) * add a --no-patchmatch option to disable patchmatch loading (#1598) This feature was added to prevent the CI Macintosh tests from erroring out when patchmatch is unable to retrieve its shared library from github assets. * Fix #1599 by relaxing the `match_trigger` regex (#1601) * Fix #1599 by relaxing the `match_trigger` regex Also simplify logic and reduce duplication. * restrict trigger regex again (but not so far) * make concepts library work with Web UI This PR makes it possible to include a Hugging Face concepts library <style-or-subject-trigger> in the WebUI prompt. The metadata seems to be correctly handled. * documentation enhancements (#1603) - Add documentation for the Hugging Face concepts library and TI embedding. - Fixup index.md to point to each of the feature documentation files, including ones that are pending. * tweak setup and environment files for linux & pypatchmatch (#1580) * tweak setup and environment files for linux & pypatchmatch - Downgrade python requirements to 3.9 because 3.10 is not supported on Ubuntu 20.04 LTS (widely-used distro) - Use our github pypatchmatch 0.1.3 in order to install Makefile where it needs to be. - Restored "-e ." as the last install step on pip installs. Hopefully this will not trigger the high-CPU hang we've previously experienced. * keep windows on basicsr 1.4.1 * keep windows on basicsr 1.4.1 * bump pypatchmatch requirement to 0.1.4 - This brings in a version of pypatchmatch that will gracefully handle internet connection not available at startup time. - Also refactors and simplifies the handling of gfpgan's basicsr requirement across various platforms. * revert to older version of list_models() (#1611) This restores the correct behavior of list_models() and quenches the bug of list_models() returning a single model entry named "name". I have not investigated what was wrong with the new version, but I think it may have to do with changes to the behavior in dict.update() * Fixes for #1604 (#1605) * Converts ESRGAN image input to RGB - Also adds typing for image input. - Partially resolves #1604 * ensure there are unmasked pixels before color matching Co-authored-by: Kyle Schouviller <kyle0654@hotmail.com> * update index.md (#1609) - comment out non existing link - fix indention - add seperator between feature categories * Debloat-docker (#1612) * debloat Dockerfile - less options more but more userfriendly - better Entrypoint to simulate CLI usage - without command the container still starts the web-host * debloat build.sh * better syntax in run.sh * update Docker docs - fix description of VOLUMENAME - update run script example to reflect new entrypoint * Test installer (#1618) * test linux install * try removing http from parsed requirements * pip install confirmed working on linux * ready for linux testing - rebuilt py3.10-linux-x86_64-cuda-reqs.txt to include pypatchmatch dependency. - point install.sh and install.bat to test-installer branch. * Updates MPS reqs * detect broken readline history files * fix download.pytorch.org URL * Test installer (Win 11) (#1620) Co-authored-by: Cyrus Chan <cyruswkc@hku.hk> * Test installer (MacOS 13.0.1 w/ torch==1.12.0) (#1621) * Test installer (Win 11) * Test installer (MacOS 13.0.1 w/ torch==1.12.0) Co-authored-by: Cyrus Chan <cyruswkc@hku.hk> * change sourceball to development for testing * Test installer (MacOS 13.0.1 w/ torch==1.12.1 & torchvision==1.13.1) (#1622) * Test installer (Win 11) * Test installer (MacOS 13.0.1 w/ torch==1.12.0) * Test installer (MacOS 13.0.1 w/ torch==1.12.1 & torchvision==1.13.1) Co-authored-by: Cyrus Chan <cyruswkc@hku.hk> Co-authored-by: psychedelicious <4822129+psychedelicious@users.noreply.github.com> Co-authored-by: Cyrus Chan <82143712+cyruschan360@users.noreply.github.com> Co-authored-by: Cyrus Chan <cyruswkc@hku.hk> * 2.2 Doc Updates (#1589) * Unified Canvas Docs & Assets Unified Canvas draft Advanced Tools Updates Doc Updates (lstein feedback) * copy edits to Unified Canvas docs - consistent capitalisation and feature naming - more intimate address (replace "the user" with "you") for improved User Engagement(tm) - grammatical massaging and *poesie* Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> Co-authored-by: damian <git@damianstewart.com> * include a step after config to `cat ~/.invokeai` (#1629) * disable patchmatch in CI actions (#1626) * disable patchmatch in CI actions * fix indention * replace tab with spaces Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com> Co-authored-by: mauwii <Mauwii@outlook.de> * Fix installer script for macOS. (#1630) * refer to the platform as 'osx' instead of 'mac', otherwise the composed URL to micromamba is wrong. * move the `-O` option to `tar` to be grouped with the other tar flags to avoid the `-O` being interpreted as something to unarchive. * Removes symlinked environment.yaml (#1631) Was unintentionally added in #1621 * Fix inpainting with iterations (#1635) * fix error when inpainting using runwayml inpainting model (#1634) - error was "Omnibus object has no attribute pil_image" - closes #1596 * add k_dpmpp_2_a and k_dpmpp_2 solvers options (#1389) * add k_dpmpp_2_a and k_dpmpp_2 solvers options * update frontend Co-authored-by: Victor <victorca25@users.noreply.github.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> * add .editorconfig (#1636) * Web UI 2.2 bugfixes (#1572) * Fixes bug preventing multiple images from being generated * Fixes valid seam strength value range * Update Delete Alert Text Indicates to the user that images are not permanently deleted. * Fixes left/right arrows not working on gallery * Fixes initial image on load erroneously set to a user uploaded image Should be a result gallery image. * Lightbox Fixes - Lightbox is now a button in the current image buttons - Lightbox is also now available in the gallery context menu - Lightbox zoom issues fixed - Lightbox has a fade in animation. * Fix image display wrapper in current preview not overflow bounds * Revert "Fix image display wrapper in current preview not overflow bounds" This reverts commit 5511c82714dbf1d1999d64e8bc357bafa34ddf37. * Change Staging Area discard icon from Bin to X * Expose Snap Threshold and Move Snap Settings to BBox Panel * Changes img2img strength default to 0.75 * Fixes drawing triggering when mouse enters canvas w/ button down When we only supported inpainting and no zoom, this was useful. It allowed the cursor to leave the canvas (which was easy to do given the limited canvas dimensions) and without losing the "I am drawing" state. With a zoomable canvas this is no longer as useful. Additionally, we have more popovers and tools (like the color pickers) which result in unexpected brush strokes. This fixes that issue. * Revert "Expose Snap Threshold and Move Snap Settings to BBox Panel" We will handle this a bit differently - by allowing the grid origin to be moved. I will dig in at some point. This reverts commit 33c92ecf4da724c2f17d9d91c7ea31a43a2f6deb. * Adds Limit Strokes to Box * Adds fill bounding box button * Adds erase bounding box button * Changes Staging area discard icon to match others * Fixes right click breaking move tool * Fixes brush preview visibility issue with "darken outside box" * Fixes history bugs with addFillRect, addEraseRect, and other actions * Adds missing `key` * Fixes postprocessing being applied to canvas generations * Fixes bbox not getting scaled in various situations * Fixes staging area show image toggle not resetting on accept/discard * Locks down canvas while generating/staging * Fixes move tool breaking when canvas loses focus during move/transform * Hides cursor when restrict strokes is on and mouse outside bbox * Lints * Builds fresh bundle * Fix overlapping hotkey for Fill Bounding Box * Build Fresh Bundle * Fixes bug with mask and bbox overlay * Builds fresh bundle Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> * disable NSFW checker loading during the CI tests (#1641) * disable NSFW checker loading during the CI tests The NSFW filter apparently causes invoke.py to crash during CI testing, possibly due to out of memory errors. This workaround disables NSFW model loading. * doc change * fix formatting errors in yml files * Configure the NSFW checker at install time with default on (#1624) * configure the NSFW checker at install time with default on 1. Changes the --safety_checker argument to --nsfw_checker and --no-nsfw_checker. The original argument is recognized for backward compatibility. 2. The configure script asks users whether to enable the checker (default yes). Also offers users ability to select default sampler and number of generation steps. 3.Enables the pasting of the caution icon on blurred images when InvokeAI is installed into the package directory. 4. Adds documentation for the NSFW checker, including caveats about accuracy, memory requirements, and intermediate image dispaly. * use better fitting icon * NSFW defaults false for testing * set default back to nsfw active Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com> Co-authored-by: mauwii <Mauwii@outlook.de> Signed-off-by: devops117 <55235206+devops117@users.noreply.github.com> Co-authored-by: psychedelicious <4822129+psychedelicious@users.noreply.github.com> Co-authored-by: blessedcoolant <54517381+blessedcoolant@users.noreply.github.com> Co-authored-by: Kyle Schouviller <kyle0654@hotmail.com> Co-authored-by: javl <mail@jaspervanloenen.com> Co-authored-by: Kent Keirsey <31807370+hipsterusername@users.noreply.github.com> Co-authored-by: mauwii <Mauwii@outlook.de> Co-authored-by: Matthias Wild <40327258+mauwii@users.noreply.github.com> Co-authored-by: Damian Stewart <d@damianstewart.com> Co-authored-by: DevOps117 <55235206+devops117@users.noreply.github.com> Co-authored-by: damian <git@damianstewart.com> Co-authored-by: Damian Stewart <null@damianstewart.com> Co-authored-by: Cyrus Chan <82143712+cyruschan360@users.noreply.github.com> Co-authored-by: Cyrus Chan <cyruswkc@hku.hk> Co-authored-by: Andre LaBranche <dre@mac.com> Co-authored-by: victorca25 <41912303+victorca25@users.noreply.github.com> Co-authored-by: Victor <victorca25@users.noreply.github.com> | 3 年前 | |
Git blame ignore revs | 1 年前 | |
refactor: model manager v3 (#8607) * feat(mm): add UnknownModelConfig * refactor(ui): move model categorisation-ish logic to central location, simplify model manager models list * refactor(ui)refactor(ui): more cleanup of model categories * refactor(ui): remove unused excludeSubmodels I can't remember what this was for and don't see any reference to it. Maybe it's just remnants from a previous implementation? * feat(nodes): add unknown as model base * chore(ui): typegen * feat(ui): add unknown model base support in ui * feat(ui): allow changing model type in MM, fix up base and variant selects * feat(mm): omit model description instead of making it "base type filename model" * feat(app): add setting to allow unknown models * feat(ui): allow changing model format in MM * feat(app): add the installed model config to install complete events * chore(ui): typegen * feat(ui): toast warning when installed model is unidentified * docs: update config docstrings * chore(ui): typegen * tests(mm): fix test for MM, leave the UnknownModelConfig class in the list of configs * tidy(ui): prefer types from zod schemas for model attrs * chore(ui): lint * fix(ui): wrong translation string * feat(mm): normalized model storage Store models in a flat directory structure. Each model is in a dir named its unique key (a UUID). Inside that dir is either the model file or the model dir. * feat(mm): add migration to flat model storage * fix(mm): normalized multi-file/diffusers model installation no worky now worky * refactor: port MM probes to new api - Add concept of match certainty to new probe - Port CLIP Embed models to new API - Fiddle with stuff * feat(mm): port TIs to new API * tidy(mm): remove unused probes * feat(mm): port spandrel to new API * fix(mm): parsing for spandrel * fix(mm): loader for clip embed * fix(mm): tis use existing weight_files method * feat(mm): port vae to new API * fix(mm): vae class inheritance and config_path * tidy(mm): patcher types and import paths * feat(mm): better errors when invalid model config found in db * feat(mm): port t5 to new API * feat(mm): make config_path optional * refactor(mm): simplify model classification process Previously, we had a multi-phase strategy to identify models from their files on disk: 1. Run each model config classes' `matches()` method on the files. It checks if the model could possibly be an identified as the candidate model type. This was intended to be a quick check. Break on the first match. 2. If we have a match, run the config class's `parse()` method. It derive some additional model config attrs from the model files. This was intended to encapsulate heavier operations that may require loading the model into memory. 3. Derive the common model config attrs, like name, description, calculate the hash, etc. Some of these are also heavier operations. This strategy has some issues: - It is not clear how the pieces fit together. There is some back-and-forth between different methods and the config base class. It is hard to trace the flow of logic until you fully wrap your head around the system and therefore difficult to add a model architecture to the probe. - The assumption that we could do quick, lightweight checks before heavier checks is incorrect. We often _must_ load the model state dict in the `matches()` method. So there is no practical perf benefit to splitting up the responsibility of `matches()` and `parse()`. - Sometimes we need to do the same checks in `matches()` and `parse()`. In these cases, splitting the logic is has a negative perf impact because we are doing the same work twice. - As we introduce the concept of an "unknown" model config (i.e. a model that we cannot identify, but still record in the db; see #8582), we will _always_ run _all_ the checks for every model. Therefore we need not try to defer heavier checks or resource-intensive ops like hashing. We are going to do them anyways. - There are situations where a model may match multiple configs. One known case are SD pipeline models with merged LoRAs. In the old probe API, we relied on the implicit order of checks to know that if a model matched for pipeline _and_ LoRA, we prefer the pipeline match. But, in the new API, we do not have this implicit ordering of checks. To resolve this in a resilient way, we need to get all matches up front, then use tie-breaker logic to figure out which should win (or add "differential diagnosis" logic to the matchers). - Field overrides weren't handled well by this strategy. They were only applied at the very end, if a model matched successfully. This means we cannot tell the system "Hey, this model is type X with base Y. Trust me bro.". We cannot override the match logic. As we move towards letting users correct mis-identified models (see #8582), this is a requirement. We can simplify the process significantly and better support "unknown" models. Firstly, model config classes now have a single `from_model_on_disk()` method that attempts to construct an instance of the class from the model files. This replaces the `matches()` and `parse()` methods. If we fail to create the config instance, a special exception is raised that indicates why we think the files cannot be identified as the given model config class. Next, the flow for model identification is a bit simpler: - Derive all the common fields up-front (name, desc, hash, etc). - Merge in overrides. - Call `from_model_on_disk()` for every config class, passing in the fields. Overrides are handled in this method. - Record the results for each config class and choose the best one. The identification logic is a bit more verbose, with the special exceptions and handling of overrides, but it is very clear what is happening. The one downside I can think of for this strategy is we do need to check every model type, instead of stopping at the first match. It's a bit less efficient. In practice, however, this isn't a hot code path, and the improved clarity is worth far more than perf optimizations that the end user will likely never notice. * refactor(mm): remove unused methods in config.py * refactor(mm): add model config parsing utils * fix(mm): abstractmethod bork * tidy(mm): clarify that model id utils are private * fix(mm): fall back to UnknownModelConfig correctly * feat(mm): port CLIPVisionDiffusersConfig to new api * feat(mm): port SigLIPDiffusersConfig to new api * feat(mm): make match helpers more succint * feat(mm): port flux redux to new api * feat(mm): port ip adapter to new api * tidy(mm): skip optimistic override handling for now * refactor(mm): continue iterating on config * feat(mm): port flux "control lora" and t2i adapter to new api * tidy(ui): use Extract to get model config types * fix(mm): t2i base determination * feat(mm): port cnet to new api * refactor(mm): add config validation utils, make it all consistent and clean * feat(mm): wip port of main models to new api * feat(mm): wip port of main models to new api * feat(mm): wip port of main models to new api * docs(mm): add todos * tidy(mm): removed unused model merge class * feat(mm): wip port main models to new api * tidy(mm): clean up model heuristic utils * tidy(mm): clean up ModelOnDisk caching * tidy(mm): flux lora format util * refactor(mm): make config classes narrow Simpler logic to identify, less complexity to add new model, fewer useless attrs that do not relate to the model arch, etc * refactor(mm): diffusers loras w * feat(mm): consistent naming for all model config classes * fix(mm): tag generation & scattered probe fixes * tidy(mm): consistent class names * refactor(mm): split configs into separate files * docs(mm): add comments for identification utils * chore(ui): typegen * refactor(mm): remove legacy probe, new configs dir structure, update imports * fix(mm): inverted condition * docs(mm): update docsstrings in factory.py * docs(mm): document flux variant attr * feat(mm): add helper method for legacy configs * feat(mm): satisfy type checker in flux denoise * docs(mm): remove extraneous comment * fix(mm): ensure unknown model configs get unknown attrs * fix(mm): t5 identification * fix(mm): sdxl ip adapter identification * feat(mm): more flexible config matching utils * fix(mm): clip vision identification * feat(mm): add sanity checks before probing paths * docs(mm): add reminder for self for field migrations * feat(mm): clearer naming for main config class hierarchy * feat(mm): fix clip vision starter model bases, add ref to actual models * feat(mm): add model config schema migration logic * fix(mm): duplicate import * refactor(mm): split big migration into 3 Split the big migration that did all of these things into 3: - Migration 22: Remove unique contraint on base/name/type in models table - Migration 23: Migrate configs to v6.8.0 schemas - Migration 24: Normalize file storage * fix(mm): pop base/type/format when creating unknown model config * fix(db): migration 22 insert only real cols * fix(db): migration 23 fall back to unknown model when config change fails * feat(db): run migrations 23 and 24 * fix(mm): false negative on flux lora * fix(mm): vae checkpoint probe checking for dir instead of file * fix(mm): ModelOnDisk skips dirs when looking for weights Previously a path w/ any of the known weights suffixes would be seen as a weights file, even if it was a directory. We now check to ensure the candidate path is actually a file before adding it to the list of weights. * feat(mm): add method to get main model defaults from a base * feat(mm): do not log when multiple non-unknown model matches * refactor(mm): continued iteration on model identifcation * tests(mm): refactor model identification tests Overhaul of model identification (probing) tests. Previously we didn't test the correctness of probing except in a few narrow cases - now we do. See tests/model_identification/README.md for a detailed overview of the new test setup. It includes instructions for adding a new test case. In brief: - Download the model you want to add as a test case - Run a script against it to generate the test model files - Fill in the expected model type/format/base/etc in the generated test metadata JSON file Included test cases: - All starter models - A handful of other models that I had installed - Models present in the previous test cases as smoke tests, now also tested for correctness * fix(mm): omit type/format/base when creating unknown config instance * feat(mm): use ValueError for model id sanity checks * feat(mm): add flag for updating models to allow class changes * tests(mm): fix remaining MM tests * feat: allow users to edit models freely * feat(ui): add warning for model settings edit * tests(mm): flux state dict tests * tidy: remove unused file * fix(mm): lora state dict loading in model id * feat(ui): use translation string for model edit warning * docs(db): update version numbers in migration comments * chore: bump version to v6.9.0a1 * docs: update model id readme * tests(mm): attempt to fix windows model id tests * fix(mm): issue with deleting single file models * feat(mm): just delete the dir w/ rmtree when deleting model * tests(mm): windows CI issue * fix(ui): typegen schema sync * fix(mm): fixes for migration 23 - Handle CLIP Embed and Main SD models missing variant field - Handle errors when calling the discriminator function, previously only handled ValidationError but it could be a ValueError or something else - Better logging for config migration * chore: bump version to v6.9.0a2 * chore: bump version to v6.9.0a3 | 10 个月前 | |
Docs: Housekeeping + Remove docs-old (#9166) * chore(docs): remove old docs directory * chore(docs): transition tests, generators to new docs Focuses on continuing the move away from the old docs configuration by removing legacy packages, tests and generate scripts, and replacing them with new scripts for the new docs.ma - Adds docs commands to Makefile - Removes old docs tests - Removes old docs generate scripts - Removes docstring and mkfile usage - Regenerated UV lockfile * feat(docs): improve config organization Pulls out the long parts of the astro config into dedicated files located in a new `config/` directory. * feat(docs): add pages for contributing to docs * feat(docs): add image section describe how to add images to your docs pages in an organized fashion. * feat(docs): add guide for docs translations * feat(docs): add cover image * fix(docs): add social image metadata * fix(docs): pages in wrong locations also updated some orderings and frontmatter * chore(docs): upgrade deps * feat(docs): add missing docs pages * fix(docs): docs fixes - fix redirects - improve canvas projects docs - improve models docs --------- Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> | 3 个月前 | |
remove src directory, which is gumming up conda installs; addresses issue #77 | 3 年前 | |
update nodes schema / typegen | 1 年前 | |
chore: update pre-commit syntax; add check for uv.lock needing an update | 1 年前 | |
feat: automated releases via github action - Restructure & update code check workflows - Add release workflow to handle checks/tests, build and publish to PyPI - Add docs/RELEASE.md explaining the workflow & process - `create_installer.sh`: Update to work with the release workflow - `create_installer.sh` & `tag_release.sh`: Fix the ANSI escape codes for macOS - `tag_release.sh`: Add check for python binary name - `tag_release.sh`: Print `git remote -v` output - `tag_release.sh`: Fix error when deleting nonexistant tags | 2 年前 | |
Add @ebr to Contributors (#2095) * (docs) @ebr signs Statement of Values * (docs) add @ebr to Contributors page | 3 年前 | |
Update LICENSE | 3 年前 | |
Feature: HiDiffusion integration (#8787) * Initial implementation of HiDiffusion pipeline * Added URLs to original HiDiffusion repository in documentation and frontend popover. * Added comment next to HiDiffusion in pyproject.toml to clarify its purpose. * Formatting * Import order fix * Add HiDiffusion T1/T2 ratio controls and docs updates * Checks appeasing * Refactor HiDiffusion import and update T2 ratio default value to 0 * Changed to vendoring an updated version of HiDiffusion and removed deps for external dependency * Ruff * Ruff again * chore(api): refresh hidiffusion openapi schema * Fix HiDiffusion cleanup and metadata recall * chore: fix lint import ordering * Fix HiDiffusion cleanup and metadata recall * Move HiDiffusion license notice to repository root * Fix HiDiffusion modular denoise and docs formatting * Bump HiDiffusion node versions and refresh schema * Seed HiDiffusion window attention deterministically * fix: stale HiDiffusion state in cached UNet * Make HiDiffusion teardown transactional * Format HiDiffusion teardown with Ruff --------- Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com> | 1 个月前 | |
Feat: Add PiD (Pixel Diffusion Decoder) 4× super-resolution decode for FLUX / FLUX.2 / SD3 / SDXL / Z-Image / Qwen-Image (#9281) * feat(pid): vendor PiD decoder backend (phase A of integration) Adds a vendored subset of NVIDIA's PiD (Pixel Diffusion Decoder) at invokeai/backend/pid/ as the foundation for upcoming FLUX / FLUX.2 / SD3 / Z-Image PiD decode nodes plus a future PiD-based 4x upscale node. Upstream: https://github.com/nv-tlabs/PiD (Apache 2.0). Vendor scope: * _src/{networks,models,modules}: PidNet, PixDiT_T2I, LQProjection2D, PidModel, PidDistillModel, PixelDiTModel, GeneralConditioner. * _ext/imaginaire: minimal Imaginaire framework subset (lazy_config, model, utils/{log,misc,distributed,device,count_params}). * configs/, tokenizers/, checkpointer/, trainer.py, visualize/, _demo_*, from_*, easy_io/, S3/wandb training helpers were intentionally excluded. Dependency stripping (no new hard deps introduced): * loguru, termcolor -> stdlib logging shim * iopath PathManager -> stdlib pathlib stub * fvcore Registry -> minimal stdlib Registry * lazy_config/lazy.py: yaml/dill/cloudpickle/detectron2 save/load paths replaced with a minimal LazyCall stub * lazy_config/instantiate.py: omegaconf DictConfig/ListConfig branches removed; configs are plain dict / LazyCall mappings * megatron, pynvml, boto3/wandb imports are try/except-guarded or local to functions and stay inert in our inference path All pid.* imports rewritten to invokeai.backend.pid.*; SPDX-Apache-2.0 headers retained on vendored files; attribution and detailed list of local modifications added in LICENSE-PiD.txt. The pre-trained PiD checkpoints distributed by NVIDIA remain under NSCLv1 (non-commercial); this commit only vendors code. Smoke test: PidNet, PidModel, PidDistillModel, GeneralConditioner import cleanly; LazyCall -> instantiate round-trip resolves to the expected nn.Module. ruff check passes. * feat(pid): wire PiD + Gemma-2 into model-manager and add decode nodes Adds the model-manager plumbing and workflow nodes needed to use the vendored PiD decoder (phase A) end-to-end with FLUX, SD3 and Z-Image. Model manager (Phase B + B.5): * taxonomy: ModelType.PiDDecoder, PiDDecoderVariantType (Res2k_Sr4x / Res2kTo4k_Sr4x), ModelType.Gemma2Encoder + ModelFormat.Gemma2Encoder, both added to AnyVariant + variant_type_adapter. * configs/pid_decoder.py: per-backbone PiD configs (FLUX / FLUX.2 / SD3) with state-dict probing on 'lq_proj' substring and backbone/variant detection from the official NVIDIA filenames. * configs/gemma2_encoder.py: Gemma-2 directory probing on Gemma2ForCausalLM architecture + tokenizer files. * AnyModelConfig union updated. * model_loaders/pid_decoder.py: loads .pth / .safetensors, strips the upstream 'net.' prefix, supports torch.load(weights_only=True). * model_loaders/gemma2_encoder.py: SubModelType.{Tokenizer, TextEncoder} dispatch; returns the causal LM's inner Gemma2Model (transformers 4.56's get_decoder() returns None for Gemma2). Decode pipeline (Phase C): * backend/pid/decode.py: build_pid_net + load_pid_decoder (per-backbone PixDiT_T2I hyperparams derived from PiD's pid_sr4x base + per-experiment overrides), encode_caption_for_pid (chi-prompt + Gemma encoding, mirrors PixelDiTModel._encode_text_raw), and a PiDDecoder wrapper with a reimplemented few-step distill sampler (no autocast / no distributed / no PixelDiTModel init paths from upstream). Invocations (Phase 6.x): * Gemma2EncoderField + PiDDecoderField in invocations/model.py. * gemma2_encoder_loader / pid_decoder_loader: thin ModelIdentifierField pickers that emit the corresponding fields. * z_image_pid_decode (pilot), flux_pid_decode, sd3_pid_decode: caption encode -> Gemma offload -> PiD state dict load -> PidNet construct -> decode. Per-backbone latent denormalisation (FLUX1 ae_params, SD3 hardcoded 1.5305/0.0609, Z-Image piggybacks on FLUX VAE). End-to-end validated with the released PiD_res2k_sr4x_official_flux_distill_4step.pth checkpoint and gemma-2-2b-it: PidNet rebuilds at exactly 456 keys / 1.36B params, sampler runs at ~5 GB VRAM peak (Gemma dominates), output shape and range match. FLUX.2 PiD decode is deliberately deferred: it needs BN-based latent denormalisation and 32->128 channel packing, and we have no FLUX.2 checkpoint to validate against yet. * feat(pid): end-to-end PiD pixel-diffusion decoder integration Adds the NVIDIA PiD decoder as a 4x super-resolution alternative to the regular VAE/RAE decode path. Includes model-manager configs and loaders for both the PiD checkpoints and the Gemma-2 caption encoder they require, plus four invocations: latent-in decode for FLUX / SD3 / Z-Image and an image-in pid_upscale node. - Decode pipeline keeps PidNet params in fp32 and uses bf16 autocast only for matmuls; caption embeddings have outliers that overflow bf16 RMSNorm. - encode_caption_for_pid forces tokenizer padding_side="right" (Gemma defaults to left, PiD trained with right) and returns the attention mask as bool so it stays compatible with SDPA. - Z-Image reuses the FLUX-trained checkpoint and reads scale/shift from the VAE config at runtime (PiD upstream notes they are checkpoint-specific). - TextLLM config now excludes Gemma2ForCausalLM so it falls through to the dedicated Gemma2 encoder config instead of being misclassified. - Frontend: new model_type / model_format / variant enums, type guards and category metadata; schema.ts regenerated via pnpm typegen. * Chore Ruff * Chore Typegen * Chore Knip * fix(pid): remove unused vendored models/utils.py (broken easy_io import) * feat(pid): identify decoder backbone from weight shapes, not filename Read latent channel count from lq_proj.latent_proj.0.weight (FLUX.2=128, FLUX.1/SD3=16) as the primary discriminator; fall back to filename/dir name only to disambiguate the architecturally identical FLUX.1/SD3 pair. Fixes FLUX.2 checkpoints (model_ema_bf16.pth) not being recognised, and correctly rejects unsupported backbones (RAE/dinov2, 768ch). Fix Flux2 docstring 32->128. * feat(ui): PiD decode (Fit mode) for FLUX text-to-image Add a "PiD Decode" mode select (Off / Fit / Native) to the FLUX advanced settings with PiD decoder + Gemma-2 encoder pickers. In Fit mode the FLUX graph swaps the VAE decode for a PiD 4x super-resolution decode and downscales back to the requested size. Adds params state (pidMode, decoder, encoder, steps) with a v3->v4 migration, model hooks, readiness checks, and graph guards for the not-yet-wired Native and non-txt2img paths. * feat(ui): PiD Native 4x mode for FLUX text-to-image Make the generation dimension helpers PiD-aware via an optional pidScale: in Native mode the user-facing dimensions are the 4x target (grid 64, optimal 2048), generation runs at target/4, and PiD's 4x output is used directly with no downscale. Thread pidScale through the params dimension reducers and the optimal-dimension/grid-size selectors, resync dimensions when toggling Native, and wire the Native path in the FLUX graph builder. Add working_mem_bytes for PiD Decode * feat(ui): PiD Fit decode for FLUX image-to-image Extract the PiD decode chain into buildPidDecodeChain (loaders + decode + fit-downscale, no denoise setup) so it can substitute for the VAE decode across generation modes. Widen addImageToImage's l2i param to ImageOutputNodes (it only consumes .image) and wire the PiD chain into the img2img branch in Fit mode. Native stays txt2img-only (a 4x result can't composite onto the bbox); inpaint/outpaint remain gated off for now. * feat(ui): PiD Native 4x decode for FLUX image-to-image Add addPidImageToImageNative: the canvas bbox is the 4x target, so the init image is downscaled to bbox/4, denoised at that resolution, and PiD decodes straight back up to the full bbox with no post-decode downscale - preserving all PiD detail while still compositing cleanly onto the region. Wire it into the img2img branch of buildFLUXGraph (native vs fit vs off) and drop the native-txt2img-only guard. Make the canvas FLUX grid check PiD-aware so a native bbox must be a multiple of 64 (16 * 4) for bbox/4 to land on the grid. * feat(ui): add informational popover to PiD Decode setting Explain PiD usage on hover, mirroring the DyPE popover: what the decoder is (NVIDIA Pixel Diffusion Decoder, 4x SR, needs a PiD decoder + Gemma-2 encoder), Fit vs Native modes, the 2K / 2K-to-4K target resolutions, that Steps can be lowered, and that Scale Before Processing must be off. Links to nv-tlabs/PiD. * feat(models): add PiD decoder + Gemma-2 encoder to starter models Register NVIDIA's PiD FLUX decoders (2K and 2K-to-4K presets, from nvidia/PiD) and the Efficient-Large-Model/gemma-2-2b-it caption encoder as starter models so they can be installed from the Model Manager. The Gemma-2 encoder is wired as a dependency of each decoder (and offered standalone). * feat(pid): add FLUX.2 Klein PiD 4x-SR decode support Add a flux2_pid_decode node that packs the stored FLUX.2 latent (32ch @ H/8) into PiD's 128ch @ H/16 layout before decoding; FLUX.2's BatchNorm denormalization is already applied in flux2_denoise, so no scalar denorm is needed (optional vae input reads identity constants). Generalize the frontend PiD decode chain (decodeNodeType, optional vaeSource) and wire the isFlux2 graph path for txt2img/img2img (Fit & Native). Base-aware PiD gating/decoder-filter, FLUX.2 readiness checks, and two nvidia/PiD FLUX.2 starter decoders (2K, 2Kto4K). Standard FLUX PiD path unchanged. * feat(pid): add SD3 PiD 4x-SR decode support Wire the existing sd3_pid_decode node into the SD3 graph builder (txt2img and img2img, Fit & Native) with a PiD guard, base-aware gating/decoder-filter (sd-3), and SD3 readiness checks. Add two nvidia/PiD SD3 starter decoders (2K, 2Kto4K). Harden the PiD config probe against the 16-channel FLUX.1/SD3 ambiguity: when the checkpoint's directory name is silent (the HF single-file download renames it), trust an explicit base override so SD3 checkpoints are not misidentified as FLUX.1. Also benefits Qwen. FLUX / FLUX.2 identification is unchanged. * feat(pid): add SDXL PiD 4x-SR decode support Build the full SDXL PiD backend stack: _PER_BACKBONE[SDXL] (4ch/down8), PiDDecoder_Checkpoint_SDXL_Config with a 4-channel latent-map entry, factory union + loader registration, and a new sdxl_pid_decode node (reads the VAE's scaling_factor/shift at runtime; SDXL fallbacks 0.13025/0.0). 4-channel latents are unambiguous, so no directory-name disambiguation is needed. Generalize the shared PiD decode chain to support SD-family denoise: denoise_latents has no width/height, so thread an optional noise node for sizing and round to the model's native grid (8 for SDXL, 16 for FLUX). Wire buildSDXLGraph (txt2img + img2img, Fit & Native) with the VAE as the decode's scaling source, base-aware gating/readiness, and a starter decoder (SDXL 2Kto4K only). PiD + SDXL refiner is blocked for now via a graph guard and a readiness reason. FLUX/FLUX.2/SD3 paths are unchanged. * feat(pid): add Z-Image PiD 4x-SR decode support Wire the existing z_image_pid_decode node into the Z-Image graph builder (txt2img and img2img, Fit & Native) with a PiD guard and readiness checks. Z-Image shares FLUX.1's 16-channel VAE and has no PiD checkpoints of its own, so it reuses the FLUX decoder: the decoder filter maps z-image -> flux, showing FLUX PiD decoders when a Z-Image model is active. The Z-Image VAE is passed to the decode node so it reads the real scaling_factor / shift instead of the fallback constants. No backend, schema, or starter-model changes. FLUX/FLUX.2/SD3/SDXL paths are unchanged. * feat(pid): add Qwen-Image PiD 4x-SR decode support Build the full Qwen-Image PiD backend stack: _PER_BACKBONE[QwenImage] (16ch/down8), PiDDecoder_Checkpoint_QwenImage_Config (added to the 16-channel latent map + filename heuristic), factory union + loader registration, and a new qwen_image_pid_decode node. Unlike the scalar-scaling bases, the Qwen-Image VAE normalizes per channel (latents_mean / latents_std) and stores a 5D video-style latent, so the node denormalizes per-channel (z * std + mean, read from the VAE config) and drops the singleton temporal frame before decoding - matching qwen_image_l2i. Wire buildQwenImageGraph (txt2img + img2img, Fit & Native) with the Qwen-Image VAE as the decode's normalization source, base-aware gating/readiness, and a starter decoder (Qwen-Image 2Kto4K only). The 16-channel FLUX/SD3/Qwen ambiguity is handled by the existing trusted- base-override probe hardening. FLUX/FLUX.2/SD3/SDXL/Z-Image paths are unchanged. * Chore Ruff * Add Docs * fix(pid): green up frontend tests and knip for the PiD branch - graph-builder tests: set pidMode 'off' in the FLUX / Qwen-Image / SDXL+SD3 param fixtures so the PiD guard doesn't fire on an undefined pidMode and call the (unmocked) size helpers - paramsSlice migration test: expect _version 4 (v3→v4 adds the PiD fields) - remove the unused setPidSteps action and selectPidSteps selector flagged by knip; the pidSteps state field stays at its default of 4 * Chore openapi + typegen * docs: regenerate invocation-context data for offload_from_vram * fix(pid): resolve PiD decoder review findings (state, readiness, steps, race) Address all reviewer findings on the PiD decoder feature: - Guard offload_model_from_vram with @synchronized, matching its sibling drop_model, to prevent a race when the VRAM working set is mutated concurrently during model swaps. - On base change (modelChanged), clear a PiD decoder that is incompatible with the new main model's decoder base at the root, respecting the Z-Image -> FLUX decoder reuse so a still-valid decoder is kept. - On switching to a base without PiD support, reset pidMode to 'off' and refit the dimensions so no hidden 4x native grid survives. - Extend the scaled-grid bbox readiness validation to SD3, SDXL and Z-Image, mirroring the existing FLUX.2 native-grid check. - Add a PiD Steps control (slider + number input, 1-8, default 4) with a pidStepsChanged action and selectPidSteps selector, so the documented step count is actually configurable and flows into the graph. Add readiness and paramsSlice tests covering the scaled-grid blocking and the base-change decoder/pidMode behavior * fix(pid): address review — cap steps at 4, honor encoder device, decoder/base guards Merge blockers: - PiD steps 5-8 produced duplicate timesteps: the student schedule has only 4 transitions (a 5-point list), so sub-sampling to >4 steps rounded distinct indices onto the same point and wasted network forwards on repeated timesteps. Cap the public range at 4 across the backend fields (le=4), the UI slider/input (max=4), and the pidSteps zod schema (int, 1-4), and harden _get_t_list with a strictly-decreasing assertion as a safety net. - CPU-only Gemma encoders crashed on CUDA hosts: each PiD invocation passed the global compute device to caption encoding, pushing the tokenizer output to CUDA while a cpu_only encoder stayed on the CPU. Encode on the encoder's actual device instead (next(encoder.parameters()).device), honoring model_on_device(). - Gemma 2 could no longer be configured as a generic TextLLM: the specialised- architecture exclusion rejected Gemma2ForCausalLM unconditionally. Only defer to the encoder config during automatic classification; keep an explicit type=text_llm request valid (the generic causal-LM loader supports it). Follow-ups folded in: - Reject incompatible Gemma 2 sizes before execution: PiD's caption projection is fixed at Gemma-2-2b's 2304-dim hidden state, so the encoder config now rejects 9B (3584) / 27B (4608) up front instead of failing deep in inference. - Validate the PiD decoder's base against each base-specific decode node: the base-agnostic loader let the Nodes editor wire any decoder into any node. Add assert_pid_decoder_matches_base and call it in all seven decode nodes, preserving the Z-Image-reuses-FLUX-decoder case (its node backbone is FLUX). Add regression tests: the distill schedule (strictly decreasing 1-4, safety net trips at 5) and decoder/base validation; the Gemma2 hidden-size gate; and TextLLM classification (auto-defers, explicit type still matches, plain causal LMs match). The expand-prompt pipeline always sent a dedicated "system" role message, which some chat templates (notably Gemma) reject with "System role not supported", 500-ing prompt expansion for those models. When applying the chat template fails with a system-role error, fold the system prompt into the first user turn and retry instead of failing. Adds regression tests for both the fallback and the normal (system-supported) path. * Chore fix * fix(pid): keep large Gemma2 as TextLLM, raise (not assert) schedule guard, compute_device, narrow pid_upscale VAE Address the latest review on the PiD PR: - Merge blocker: automatic classification sent Gemma 2 9B/27B to Unknown. The PiD Gemma2 encoder config rejects their non-2304 hidden size, and TextLLM deferred *every* Gemma2ForCausalLM, so neither matched. TextLLM now defers only the size the encoder config accepts (2304 = Gemma-2-2b); larger variants stay TextLLM. - Schedule safety net used assert, which is stripped under `python -O`, leaving _get_t_list(num_steps=5) returning a duplicate schedule. Raise ValueError instead so the guard holds in optimized runtimes; the regression test now asserts ValueError and passes under `python -O`. - All seven PiD caption paths derived the Gemma device from the first parameter, which is wrong under partial loading (first param on CPU, later modules on CUDA). Use the cache contract's LoadedModel.compute_device instead. - pid_upscale advertised Z-Image / 16-channel VAEs but delegates to the FLUX-only vae_encode. Narrow the field description and validate the VAE is a FLUX AutoEncoder up front (a diffusers AutoencoderKL now fails with a clear error instead of a stripped-assert failure inside vae_encode). Update the TextLLM/Gemma2 tests (per-size config-level + a factory-level check that 2304 -> Gemma2Encoder and 3584/4608 -> TextLLM) and the schedule test (ValueError, green under python -O). * chore: regenerate OpenAPI schema and frontend types * feat(pid): accept single-file GGUF Gemma-2-2b as the PiD caption encoder The PiD Gemma encoder was directory + HuggingFace only, so a llama.cpp GGUF (e.g. gemma-2-2b-it-Q4_K_M.gguf) could not be used. Add GGUF support: - Gemma2Encoder_GGUF_Config: identifies a single .gguf file, reads the GGUF metadata and requires general.architecture == "gemma2" and <arch>.embedding_length == 2304 (Gemma-2-2b), rejecting 9B/27B as the directory config does. - Gemma2EncoderGGUFLoader (format gguf_quantized): loads via transformers from_pretrained(<dir>, gguf_file=<name>) — transformers dequantizes gemma2 GGUFs and reads the tokenizer from the GGUF metadata — then exposes the Gemma2Model decoder, matching the directory loader. PiD encodes the caption once and offloads the encoder, so dequantizing at load is acceptable. - Register the config in the AnyModelConfig union. No frontend change: the PiD encoder picker filters by type=gemma2_encoder, so the GGUF variant appears automatically. Verified end-to-end against a real q4_k_m file: it classifies as Gemma2Encoder_GGUF_Config and loads to a Gemma2Model producing 2304-dim hidden states. Adds config identification tests (match, 9B/27B rejected, non-gemma2 rejected, non-.gguf rejected). * Chore Ruff * fix(models): stop Qwen3 GGUF config from matching Gemma-2 GGUF encoders A Gemma-2 GGUF satisfies the generic Qwen3 GGUF key heuristic (token_embd.weight + blk.* keys), so it matched both Qwen3Encoder_GGUF_Config and the intended Gemma2Encoder_GGUF_Config. On a fresh install the Gemma config happened to win, but re-identification could pick Qwen3, mis-classifying the model. Add _has_gemma2_keys (Gemma uses blk.*.post_attention_norm / post_ffw_norm, which a Qwen3 encoder never has — Qwen3 has attn_q_norm/attn_k_norm instead) and reject such state dicts in both Qwen3 encoder configs' _validate_looks_like_qwen3_model (GGUF and checkpoint), mirroring the existing T5 / Qwen-VL exclusions. The Gemma config already rejects Qwen3 GGUFs via the general.architecture metadata, so the two are now mutually exclusive and identification is deterministic. Add regression tests: _has_gemma2_keys detection and that the Qwen3 GGUF config rejects a Gemma-keyed state dict. * test(models): assert re-identified Gemma GGUF drops the stale Qwen3 variant The Gemma2 GGUF encoder config has no `variant` field, so re-identifying a model previously mis-detected as a Qwen3 GGUF (which carries a variant) drops it — the serialized record has no variant key and replace_model overwrites it away. Assert this explicitly in the Gemma GGUF identification test. * fix(pid): point 2K-to-4K starter decoders at NVIDIA's v1.5 replacements NVIDIA deprecated the FLUX / FLUX.2 / Qwen-Image `res2kto4k_sr4x` PiD decoders and moved them to `checkpoints_deprecated/`, replacing them with the recommended `v1pt5_res2kto4k_sr4x` checkpoints. Our starter models still pointed at the old `checkpoints/` paths, which now 404 on install. Repoint the three affected 2K-to-4K starters (FLUX, FLUX.2, Qwen-Image) to the v1.5 successors and note the upgrade in their descriptions. The 2K (`res2k_sr4x`) decoders and the SD3 / SDXL 2K-to-4K decoders are not deprecated and are unchanged. Base and variant are still sent as explicit overrides, so config identification is unaffected by the new directory name (res2kto4k -> Res2kTo4k_Sr4x). * feat(pid): load Gemma-2 GGUF encoders natively (keep weights quantized) The GGUF Gemma encoder used transformers' from_pretrained(gguf_file=...), which dequantizes every weight at load — so a quantized Gemma cost the same VRAM as the unquantized model. Load it via InvokeAI's GGMLTensor path instead: read the config from GGUF metadata, map llama.cpp tensor names to Gemma2Model, and keep the 2D projection weights as GGMLTensor (dequantized on demand by the model cache). Materialize only the embedding and the RMSNorm weights, subtracting 1 from the norms (llama.cpp folds +1 in; Gemma2RMSNorm re-adds it), and assert nothing is left on meta. Verified: hidden states match the fully-dequantized loader within quantization tolerance. Adds key-mapping tests and a local load/compare test. NVIDIA's v1.5 decoders use a different network (lq_hidden_dim=1024, PiT injection) that build_pid_net (512-dim legacy) cannot load, causing a size-mismatch crash. - Point the FLUX/FLUX.2/Qwen 2K-to-4K starters back at the legacy checkpoints (moved to checkpoints_deprecated/) that the current network loads. - Reject a checkpoint whose lq_proj hidden dim is not the supported 512 at identification time, instead of accepting it and failing inside the decode. - Enumerate all supported backbones (add SDXL, Qwen-Image) in the loader title and correct the variant enum docs (not every backbone ships both presets). Full v1.5 architecture support is planned as a follow-up. Adds PiD decoder identification tests (legacy accepted, 1024-dim v1.5 rejected). * Chore openapi + typegen * Docs update --------- Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com> Co-authored-by: JPPhoto <jpollack@jpollackphoto.com> | 1 个月前 | |
updated LICENSE files and added information about watermarking | 3 年前 | |
updated LICENSE files and added information about watermarking | 3 年前 | |
Docs: Housekeeping + Remove docs-old (#9166) * chore(docs): remove old docs directory * chore(docs): transition tests, generators to new docs Focuses on continuing the move away from the old docs configuration by removing legacy packages, tests and generate scripts, and replacing them with new scripts for the new docs.ma - Adds docs commands to Makefile - Removes old docs tests - Removes old docs generate scripts - Removes docstring and mkfile usage - Regenerated UV lockfile * feat(docs): improve config organization Pulls out the long parts of the astro config into dedicated files located in a new `config/` directory. * feat(docs): add pages for contributing to docs * feat(docs): add image section describe how to add images to your docs pages in an organized fashion. * feat(docs): add guide for docs translations * feat(docs): add cover image * fix(docs): add social image metadata * fix(docs): pages in wrong locations also updated some orderings and frontmatter * chore(docs): upgrade deps * feat(docs): add missing docs pages * fix(docs): docs fixes - fix redirects - improve canvas projects docs - improve models docs --------- Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> | 3 个月前 | |
Feat: flux2 dev support (#9234) * feat(flux2): add FLUX.2 [dev] support Adds end-to-end support for FLUX.2 [dev] alongside the existing Klein implementation. Dev uses Mistral Small 3.1 (24B) as its sole text encoder instead of Klein's Qwen3, with joint_attention_dim=15360 and the guidance-distilled 32B transformer. Backend - taxonomy: Flux2VariantType.Dev, ModelType.MistralEncoder, ModelFormat.MistralEncoder, MistralVariantType - configs: probe dev via context_in_dim=15360 (main + LoRA); new mistral_encoder.py with Diffusers / Checkpoint / GGUF configs; Main_Diffusers_Flux2_Config accepts Flux2Pipeline class name - loaders: new mistral_encoder.py (AutoModel for Diffusers folder, MistralModel for single-file + GGUF with llama.cpp key conversion). Existing Klein transformer loaders are generic enough for dev - ModelRecordChanges.variant union extended with MistralVariantType Invocations - flux2_dev_model_loader, flux2_dev_text_encoder (Mistral chat-template with FLUX2_DEV_SYSTEM_MESSAGE and layer-stacking 10/20/30), flux2_dev_lora_loader (+ collection variant) - MistralEncoderField on model.py; flux2_denoise / flux2_vae_decode / flux2_vae_encode reused unchanged (already model-agnostic) Frontend - types/hooks/selectors for MistralEncoder, isFlux2DevMainModelConfig, selectFlux2DevDiffusersModels, useMistralEncoderModels - params slice fields flux2DevVaeModel / flux2DevMistralEncoderModel / flux2DevSourceModel + reducers, selectIsFlux2Dev / selectIsFlux2Klein - ParamFlux2DevModelSelect component, wired into AdvancedSettingsAccordion - buildFLUXGraph dev branch with full txt2img / img2img / inpaint / outpaint + multi-reference image editing (same flux_kontext + collect chain as Klein, since Flux2RefImageExtension is model-agnostic) - addFlux2DevLoRAs helper for dev LoRA wiring - zModelType / zModelFormat / zFlux2VariantType extended for mistral_encoder / mistral_small_3_1 / dev - OpenAPI schema regenerated, TS types updated Starter models - FLUX.2 [dev] Diffusers (bf16 + NF4), three GGUFs (Q4/Q6/Q8), Mistral encoder (bf16 + NF4) * fix(flux2): wire dev path end-to-end, harden Mistral encoder loader Follow-up fixes after first end-to-end run with FLUX.2 [dev] GGUF + Mistral 3.x GGUF + standalone FLUX.2 VAE. Frontend - buildFLUXGraph: wire dev model loader's vae into both flux2_denoise (required for BN statistics / inpaint) and flux2_vae_decode; missing edge was raising RequiredConnectionException at runtime - readiness.ts: variant-aware FLUX.2 readiness check — dev requires flux2DevVaeModel + flux2DevMistralEncoderModel (or a Dev diffusers source); Klein keeps Qwen3/VAE check. Threads hasFlux2DevDiffusersSource through generate + canvas tabs and updates buildGenerateTabArg / buildCanvasTabArg test helpers - en.json: noFlux2DevVaeModelSelected, noFlux2DevMistralEncoderModelSelected Mistral encoder loader (GGUF / single-file) - Fix "Cannot copy out of meta tensor": llama.cpp conversion produced `model.*` keys but loader instantiated bare MistralModel (no `model.` prefix). Add _convert_for_bare_mistral_model to strip the prefix and drop lm_head before load_state_dict - _materialize_remaining_meta_tensors: after load_state_dict, replace any still-meta parameters (norms→ones, others→zeros) and buffers so the cache→VRAM move can't fail on partial state dicts, with a warning listing what was missing - llama.cpp converter: map attn_q_norm/attn_k_norm (Mistral 3.x qk-norm variants), with ordering before attn_q/attn_k to avoid bad rewrites Tokenizer / processor fallback - _load_processor_with_offline_fallback walks a list of sources (black-forest-labs/FLUX.2-dev tokenizer subfolder, then mistralai/Mistral-Small-3.1-… and 3.2-…), trying AutoProcessor then AutoTokenizer for each, cache-first then online. Final error spells out the three workarounds (install Diffusers folder, set HF_ENDPOINT, pre-cache the tokenizer) - flux2_dev_text_encoder: try multimodal `[{type, text}]` chat template first (PixtralProcessor / Mistral3Processor), fall back to plain string content (AutoTokenizer), then to manual [INST]…[/INST] Qwen3 encoder probe strictness - _get_qwen3_variant_from_state_dict and _get_variant_from_config now return None / raise NotAMatchError for unknown hidden_size instead of silently defaulting to qwen3_4b. The old fallback meant any llama.cpp GGUF causal LM (Mistral, Llama, …) was wrongly classified as Qwen3 — visible when the Mistral 3.x GGUF was identified as a Qwen3-4B encoder - Checkpoint / GGUF / Diffusers loaders propagate the strictness * Chore Path fix * FLUX.2 [dev]: restrict Mistral encoder to 30-layer cow + add recall handlers Upstream Mistral Small 3.1/3.2 (40 layers) produces off-distribution embeddings under FLUX.2's static (10, 20, 30) hidden-state extraction. The joint attention was actually trained against BFL's 30-layer cow-mistral3-small distillation — both Comfy-Org's safetensors and gguf-org's cow GGUFs ship the same 30-layer weights, just packaged differently. - Probing (configs/mistral_encoder.py) now rejects non-cow Mistrals across all three formats (Diffusers / Checkpoint / GGUF) with a clear error. - Loader (load/model_loaders/mistral_encoder.py) extracts the embedded Tekken tokenizer from the `tekken_model` U8 (safetensors) / fp16-per-byte (cow GGUF) tensor via mistral_common, falling back to the BFL HF tokenizer. Removes the INVOKEAI_MISTRAL_TOKENIZER_SOURCE env var. - Starter models: drop upstream Mistral 3.x entries, add Comfy-Org bf16/fp8/fp4 variants alongside the cow GGUFs. - MistralVariantType: drop Small3_1, keep only Cow. - pyproject.toml: add mistral-common dependency. Frontend recall: - Add Flux2DevVAEModel + Flux2DevMistralEncoderModel handlers, disambiguating Klein vs dev via presence of `mistral_encoder` / `qwen3_encoder` metadata fields (both bases are `flux2`). - Wire both into the Recall Parameters panel (hardcoded list was missing them). - Add `metadata.mistralEncoder` i18n key + colocated tests. * feat(flux2-dev): match ComfyUI's Mistral reference + accept 40-layer encoders After studying ComfyUI's `Flux2Tokenizer` / `Mistral3_24BModel` reference implementation, align the FLUX.2 [dev] text-encoder path with their setup: - Probing now accepts both 30-layer (cow distillation) and 40-layer (Mistral Small 3, BFL canonical / upstream) Mistrals. Re-adds `MistralVariantType.Mistral24B` alongside `Cow`. All three configs (Diffusers / Checkpoint / GGUF) updated. - Loaders strip `model.norm` (replace with Identity) when the loaded weights are the 30-layer cow distillation. Matches Comfy's `final_norm=False` for the pruned variant; for transformers' `MistralModel` the final RMSNorm is always built but the cow was trained against the raw post-layer-29 state. - 40-layer loads now log a clear warning that upstream Mistral 3.1 / 3.2 is NOT what FLUX.2's joint attention was trained against and recommends the Comfy-Org bf16/fp8/fp4 or gguf-org cow GGUF variants. BFL's canonical bundled text_encoder is also 40-layer so we don't hard-reject; the warning is opt-in self-discipline. - Text encoder invocation switches from `apply_chat_template(messages, ...)` to a raw text template `[SYSTEM_PROMPT]{sys}[/SYSTEM_PROMPT][INST]{prompt}[/INST]` fed straight to the tokenizer — byte-for-byte matches Comfy's `Flux2Tokenizer.llama_template.format(text)`. System prompt now includes the literal `\n` between "object" and "attribution" Comfy ships. - `_TekkenChatTemplateAdapter` renamed to `_TekkenRawTextAdapter` and exposes a `__call__(text, padding_side='left', ...)` interface that Tekken-encodes the raw string (BOS=1, no EOS) and left-pads with token id 11. Matches Comfy's `pad_left=True` / `pad_token=11` settings. Frontend types extended for the new `mistral3_24b` variant (zMistralVariantType, MODEL_VARIANT_TO_LONG_NAME, schema.ts). * fix(ui): remove unused exports flagged by knip on FLUX.2 [dev] branch Knip reported 6 unused exports. Each was dead code rather than incomplete wiring, verified against the actual consumers: - Drop the vestigial `flux2DevSourceModel` param end-to-end (state field, default, migration, reducer, action, selector, test). The FLUX graph builder auto-picks the diffusers source itself and never read this param; no UI set it. Mirrors how the Klein path already works. - Delete `selectIsFlux2Klein`; the graph builder computes this locally and only `selectIsFlux2Dev` is consumed. - Un-export `zMistralVariantType`; used only in the local `zAnyModelVariant` union, like `zQwenImageVariantType`. - Delete `selectMistralEncoderModels`; components use the `useMistralEncoderModels` hook instead. - Un-export `isFlux2DevMainModelConfig`; used only within types.ts, like its `isFluxDevMainModelConfig` / `isFlux2Klein9BMainModelConfig` siblings. * Chore OpenApi * Chore Ruff * chore(deps): lock mistral-common for FLUX.2 [dev] Mistral encoder * fix(flux2): disambiguate dev/Klein VAE recall by model variant The dev-vs-Klein VAE recall keyed off the presence of a mistral_encoder metadata field, but that field is only written when a standalone Mistral encoder is selected. A FLUX.2 [dev] image whose encoder came from a Diffusers source has a vae field but no mistral_encoder, so its VAE was silently recalled into the Klein slice. Resolve the image's own main model and check variant === 'dev' instead — the same signal the graph builder uses. Add regression coverage for the mistral_encoder-absent dev case, and add the missing modelManager.flux2Dev* i18n keys so the [dev] VAE/encoder labels are translatable. * fix(flux2): pass prompt as text= keyword to Mistral processor The diffusers FLUX.2-dev text encoder loads a PixtralProcessor, whose first positional __call__ parameter is `images`, not `text`. Passing the prompt positionally routed it into `images`, breaking the diffusers encoder path (only single-file/GGUF encoders, which use a text-first adapter, had been exercised). Pass text= explicitly. * fix(flux2): pass prompt as text= keyword to Mistral processor The diffusers FLUX.2-dev text encoder loads a PixtralProcessor, whose first positional __call__ parameter is `images`, not `text`. Passing the prompt positionally raised "Incorrect image source", breaking the diffusers encoder path entirely. Only single-file/GGUF encoders (text-first adapter) had been exercised. Verified against transformers 5.5.4. fix(flux2): emit Tekken special tokens in the embedded-tokenizer adapter _TekkenRawTextAdapter used mistral_common's raw Tekkenizer.encode, which runs with SpecialTokenPolicy.IGNORE and BPE-encodes the FLUX.2 markers ([SYSTEM_PROMPT], [/SYSTEM_PROMPT], [INST], [/INST]) as literal text — 54 tokens instead of 36, corrupting the prompt structure fed to FLUX.2 on the single-file and GGUF paths. Resolve the marker ids from the tokenizer's special vocab and splice them in; output is now byte-identical to the reference PixtralProcessor. * Add FLux2.dev to readme * fix(flux2-dev): address review — regional guidance, model classification, LoRA guards, encoder probes - Wire FLUX.2 [dev] regional guidance through addRegions instead of dropping it silently - Require pipeline layout for Main_Diffusers_Flux2_Config so transformer-only checkouts don't register as broken main models - Reject Klein<->dev LoRA cross-wiring on both frontend (variant filter) and backend (loaders raise) - Discriminate non-Mistral GGUFs via vocab-size floor; accept text_encoder.-prefixed encoder layouts at probe time - Dequantize fp8 checkpoints per-tensor to target dtype and drop lm_head before casting (avoid whole-dict fp32 peak) - Raise on unexpected Mistral layer count instead of inventing extraction indices - Fail Klein VAE recall closed when the main model is unresolvable - Add missing modelManager.mistralEncoder i18n key - Dedup: single-pass GGUF metadata read, consistent norm materialization, cat-based conditioning, drop redundant t() defaultValues * Feat: FLUX.2 [dev] review fixes, dedup, and shared-source refactors Address the PR #9234 review (correctness, install-probe gaps, polish) plus the deduplication follow-ups. Correctness - Wire FLUX.2 [dev] regional guidance through addRegions instead of silently dropping it (posCondCollect + flux2_dev_text_encoder handling case) - Require a full pipeline layout (model_index.json / transformer/) for Main_Diffusers_Flux2_Config so transformer-only checkouts don't register as broken main models and OSError mid-queue - Reject Klein<->dev LoRA cross-wiring on both ends: frontend filters LoRAs by variant in both graph builders; dev/Klein loaders raise instead of warn - Discriminate non-Mistral GGUFs via a vocab-size floor so Llama-2-13B and similar 5120-hidden/40-layer LMs no longer install as Mistral encoders - Accept text_encoder.-prefixed encoder layouts at install probe (matches the loader's prefix stripping) - Dequantize fp8 Mistral checkpoints per-tensor to the target dtype and drop lm_head before casting (avoid a whole-dict fp32 transient that can OOM) - Raise on an unexpected Mistral layer count instead of inventing extraction indices that silently degrade output - Fail Klein VAE recall closed when the image's main model is unresolvable - Add the missing modelManager.mistralEncoder i18n key Dedup / single source of truth - Consolidate the FLUX.2 dimension->variant tables (context/vec/hidden) into a shared configs/flux2_variant.py used by main.py and lora.py - Mistral loaders key the final-RMSNorm / warning decision on config.variant instead of re-deriving from num_hidden_layers==30 - Merge the separate Klein/dev VAE redux slots into one flux2VaeModel (slice migration v3->v4) and collapse the two metadata VAE handlers into one, removing the recall-disambiguation - Parameterize the near-identical dev/Klein canvas graph blocks into one shared addFlux2Features closure; add dev-path coverage to buildFLUXGraph.test.ts - Single-pass GGUF metadata read, consistent norm materialization, cat-based conditioning tensor, and drop redundant t() defaultValues in ParamFlux2DevModelSelect Tests: model_identification suite green; frontend parsing / graph / readiness / modelSelected suites green. * Fix: bump paramsSlice persist version to 4 for the shared FLUX.2 VAE slot The v3->v4 migration (Klein/dev VAE slots -> flux2VaeModel) bumped _version but left zParamsState._version at literal(3) and the initial state at 3, so migrate()'s final zParamsState.parse rejected with "expected 3". Bump the schema literal + initial state to 4 and add a v3->v4 migration test. * Fix: address FLUX.2 [dev] round-2 review (4 blockers + 6 cleanups) Blockers: - params migration: seed flux2DevMistralEncoderModel in the v3->v4 step so a genuine v3 blob passes zParamsState.parse() instead of wiping the whole params slice on upgrade; rebuild the migration test fixture as a field-accurate v3 object so it actually covers the regression. - guidance for [dev]: resolve the image's own model in the Guidance metadata parse gate and exempt variant === 'dev' so guidance is displayed/recalled for [dev] (still skipped for Klein); render the guidance slider for FLUX.2 [dev]. - source-model variant guard: require variant == Dev where the dev loader validates its Mistral/VAE source, and reject a [dev] source in the Klein loader — a mismatched pipeline otherwise fails with an opaque matmul error in denoise. - tokenizer offline load: drop the dead root-dir fallback + duplicated pre-try and add a root-directory AutoProcessor step to _load_tokenizer_for_model so processor files alongside the encoder weights load offline. Cleanups: - extract _reinit_inv_freq() with a rope_theta -> rope_parameters/rope_scaling fallback (fixes a latent AttributeError on pinned transformers 5.5, removes a verbatim duplicate). - flux2_dev_lora_collection_loader: replace the base assert with a ValueError that rejects non-FLUX.2 LoRAs, mirroring the Klein collection loader. - diffusers Mistral load: drop the never-run vision_tower/multi_modal_projector (~0.8GB) so they stay out of the cache and VRAM transfers. - clear flux2DevMistralEncoderModel on base switch and intra-flux2 variant switch. - pin mistral-common>=1.5.4,<2 (validated against 1.11.6). - fix contradictory 40-layer docstrings to match the taxonomy/loader story. * Chore openapi * fix(ui): bump params persist schema to v5 to resolve the dual-v4 collision main and this branch both shipped _version 4 with different new keys (PiD fields vs the flux2 VAE merge + Mistral encoder slot), so a v4 blob written by either parent would fail zParamsState.parse() after the merge and wipe the whole params slice. Keep main's v3->v4 step verbatim and move the flux2 slot merge + Mistral seed to a new v4->v5 step with conditional seeding for both v4 shapes. Also seed the five Wan component fields in v3->v4: they were added to the schema without a version bump while releases were still writing v3 blobs, so a genuine released-build (v6.13.x) v3 blob fails parse() on them today - same wipe, inherited from main. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Chore openapi * fix(flux2): scope the cross-variant source guard to encoder extraction, widen the tokenizer ladder The FLUX.2 loaders ran one validator on both the VAE- and the encoder-extraction call site, so the cross-variant check also rejected VAE-only sourcing. Klein and [dev] share the same 32-channel AutoencoderKLFlux2 and the linear UI relies on that -- buildFLUXGraph falls back to any FLUX.2 diffusers pipeline when only the VAE is needed, and readiness does not filter by variant. A Klein GGUF main plus a standalone Qwen3 encoder plus a [dev] pipeline as the only diffusers model therefore hit a ValueError behind an enabled Invoke button. Split the validator: format-only for the VAE path, format + variant for the encoder path. The Mistral tokenizer ladder's local-directory rungs tried AutoProcessor only. On transformers 5.5.4 that raises OSError for a mistral3 config.json without preprocessor_config.json -- exactly the BFL-style standalone-encoder layout the rungs target -- so the ladder fell through to the HF fetch and failed offline. Both rungs now loop (AutoProcessor, AutoTokenizer), with KeyError in the except tuple for tekken-only directories. * Chore openapi * feat(metadata): declare mistral_encoder on core_metadata, bump to 2.2.0 FLUX.2 [dev] recorded its Mistral text encoder as an undeclared extra key, relying on the node's `extra='allow'`, while the Klein counterpart `qwen3_encoder` is a proper field. Declare it so it lands in the OpenAPI schema and is typed in the frontend instead of `unknown`. Also bumps the node version, which was left at 2.1.0 across several model integrations that widened the node: `ideogram4_caption` (#9303) and the generation modes for FLUX.2, Anima, Qwen-Image, Ideogram 4, Wan, Krea-2 and Ernie. All changes are additive, so this is a minor bump - saved workflows carrying a core_metadata node now auto-update to the current field set on load rather than silently keeping a stale one. * fix(flux2): make flux2_dev_text_encoder idle-GPU-offloadable Main's #9428 marked every text-encoder node idle_gpu_offloadable and added a registry guard asserting the marker on all *_text_encoder nodes; the merge brought that guard onto this branch where flux2_dev_text_encoder (which neither parent knew about) fails it. The flag alone would be wrong: the marker's contract is that the saved conditioning is CPU-backed, because the borrowed GPU's pool lock is released the moment the node returns. Move the Mistral embeds to CPU before save (the placeholder clip_embeds follows their device), add the marker, bump to 1.0.1, and add the same output-device regression test the Klein encoder has. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(flux2): stop the Mistral tokenizer ladder from crashing and from silently mis-encoding Tekken Round-4 review blockers, both reproduced against transformers 5.5.4 with real files before fixing. 1. An AttributeError from a probe rung killed the whole load. `AutoTokenizer.from_pretrained` on a directory whose `tokenizer_config.json` names a `tokenizer_class` the installed transformers does not know resolves that class to None and dereferences it without a guard — exactly the layout `MistralCommonTokenizer.save_pretrained` writes. That is not in `_TOKENIZER_LOAD_ERRORS`, so it escaped `_try_load_tokenizer_from_dir` and crashed a load the HF rung would have completed. The probes now catch broadly; the expected-error tuple only selects the log level, so an unexpected failure is still logged loudly instead of being swallowed. 2. A root `tekken.json` next to `config.json` did not fail in `AutoTokenizer` — it resolved to a mistral-common-backed tokenizer that BPE-encodes `[SYSTEM_PROMPT]`/`[INST]` as literal text instead of splicing them as single Tekken ids. The encode "worked" and conditioning was silently off-distribution. Fixed on two independent paths: the ladder now reads a standalone `tekken.json` itself, ahead of the transformers probes, and any mistral-common-backed result is re-wrapped in `_TekkenRawTextAdapter` through its underlying `MistralTokenizer` rather than used as-is. The vocab is fine — only its `__call__` is wrong — so re-wrapping beats discarding, which would have traded silent corruption for an offline RuntimeError. Verified: the re-wrapped ids are identical to the reference adapter's. Also closes both non-blockers: - `_validate_encoder_source` in the Klein loader rejected only [dev], so a Klein 9B pipeline passed as `qwen3_source_model` for a Klein 4B transformer and hit the very matmul error the guard exists to prevent (the frontend and the standalone-encoder path both enforce the family match; the workflow editor's source field was the only way in). It is now an allowlist keyed on a shared `_KLEIN_TO_QWEN3_VARIANT` map — mirroring the frontend's `KLEIN_TO_QWEN3_VARIANT_MAP` — and checks the source's Qwen3 family against the main model, so a future third FLUX.2 variant fails closed on the Klein side too, not just on [dev]. `_validate_qwen3_encoder_variant` shares that map and now uses `getattr` instead of `hasattr`, which turned a None variant into an AttributeError in the error path rather than the intended ValueError. - The [dev] loader's `_validate_diffusers_format` docstring claimed the linear UI relies on the permissive VAE path. That holds for Klein, but the [dev] builder sources from dev-only pipelines and readiness gates on one, so there the cross-variant VAE case is reachable through the workflow editor only. The justification now states what actually holds: the 32-channel AutoencoderKLFlux2 is shared (the repo ships the Klein-sourced `flux2_vae` as a dependency of every [dev] GGUF starter), and `mistral_source_model` is not variant-filtered in the editor. Tests: a structurally valid Tekken fixture, so the ladder exercises the success path rather than only the raise path the previous fake produced; regression tests for both blockers on the directory and HF rungs; Klein-family coverage including same-family acceptance and the standalone-encoder guard's negative path, which had no coverage at all. All new tests mutation-verified — reverting the broad catch, the tekken rung, the re-wrap, the family check, or the allowlist each fails at least one. tests/app + tests/backend/model_manager: 3010 passed. The 9 failures are the pre-existing network-dependent ones in test_model_install / test_load_api / test_download_queue. --------- Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> | 29 天前 | |
Create SECURITY.md | 1 年前 | |
Global replace [ \t]+$, add "GB" (#1751) * "GB" * Replace [ \t]+$ global Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> | 3 年前 | |
update flake (#7032) Co-authored-by: psychedelicious <4822129+psychedelicious@users.noreply.github.com> | 1 年前 | |
update flake (#7032) Co-authored-by: psychedelicious <4822129+psychedelicious@users.noreply.github.com> | 1 年前 | |
feat: add native Intel XPU (torch.xpu) device support (#9401) * feat(backend): add Intel XPU (torch.xpu) device support Additive xpu branches only: device selection and normalization, float16 default, VRAM queries with a passthrough-VM fallback (missing SYCL free-memory aspect), fp8 layerwise casting via a runtime probe, VAE auto-tiling, partial loading, stats/OOM handling, multi-GPU parallel session execution (device enumeration, config/API validation, worker pinning, and the generation-device options endpoint), and the auxiliary image utilities (depth/SAM/DINO pipelines accept xpu instead of falling back to CPU; cache clearing is device-agnostic). CUDA (incl. ROCm), MPS, and CPU behavior unchanged. Verified end to end on Arc Pro B70 hardware, including dual-GPU worker startup. * test(backend): add XPU coverage for TorchDevice Mock-based, mirroring the CUDA/MPS suites: device choice, dtype, normalize, the xpu_mem_get_info fallback branches, and multi-GPU generation_devices resolution/validation/labeling on XPU. Also makes the auto-without-CUDA generation-devices test hermetic on XPU machines. * build: add [xpu] extra torch 2.7.1+xpu / torchvision 0.22.1+xpu / pytorch-triton-xpu 3.3.1 from the torch-xpu index, gated to linux-x86_64 and win_amd64; uv.lock regenerated. * feat(backend): extend idle-GPU text encoder offload to XPU The idle-device arbiter and the session processor's borrow path both gated on `device.type == "cuda"`, so on a multi-XPU system no device ever registered and `offload_text_encoders_to_idle_gpus` (enabled by default) silently did nothing: encoders kept churning the denoise model in and out of VRAM. Register and lend XPU devices alongside CUDA. MPS is deliberately excluded -- it is always a single shared device, so there is never another GPU to borrow. Verified on a dual Intel Arc Pro B70 host: a text encoder node now runs on the idle GPU while the session denoises on the other ("Running compel on idle device xpu:0 (session device xpu:1)"). * feat(ui): show the executing GPU for XPU sessions Queue items already persist the executing device generically (e.g. "xpu:1"), but both readers dropped it: the session event only forwarded devices starting with "cuda", and the frontend index parser only matched /^cuda:(\d+)$/. On a multi-XPU system the progress circle and queue-item badges were therefore always blank. Accept indexed XPU devices in both places, and correct the queue-item field description, which claimed the device is set only on CUDA. * fix(mm): gate Krea 2 fp8 encoder casting on fp8 storage support The Qwen3-VL encoder kept its fp8 storage only on CUDA, so elsewhere an fp8 checkpoint was loaded as full bf16 (~8.9GB instead of ~4.4GB) and thrashed partial loading when sharing a GPU with a large transformer. Reuse the existing cached `_device_supports_fp8_storage` probe, which already backs the layerwise-casting path. It returns True unconditionally on CUDA, so CUDA behaviour is unchanged. * chore: label XPU devices by index in load logs and fp8 help text Model load lines printed the device index only for CUDA, so every model on a multi-XPU host logged as a bare "xpu device", making it impossible to tell the GPUs apart. The FP8 Storage tooltip likewise claimed CUDA-only support. * test: cover XPU config validation, progress device and fp8 probe Three paths changed by this branch had no coverage: - The `device` field pattern was untested. `test_device_choice_xpu` looks like it covers it, but the config model does not enable `validate_assignment`, so assigning `config.device` skips validation entirely; only constructing the model exercises the pattern. Added constructor-based valid/invalid cases. - `generation_devices` validation was parametrized for cuda/cpu/mps only. - The progress event's device field, which now reports XPU sessions. Also cover `_device_supports_fp8_storage`, which gates FP8 storage in both the generic layerwise-casting path and the Krea 2 encoder: CUDA answers True without probing, CPU is rejected, and a failing XPU probe returns False instead of raising. Each new test was verified to fail when the corresponding fix is reverted. * fix(nodes): recognise XPU out-of-memory errors in the Anima VAE retry The Anima VAE decode catches OOM and retries once with tiling, which caps peak allocation. Detection matched `torch.cuda.OutOfMemoryError` or the words "out of memory" in the message, so it missed XPU entirely: torch's XPU backend does not raise a recoverable `torch.OutOfMemoryError` on exhaustion, it surfaces the Level Zero/UR result code as a plain RuntimeError -- and `UR_RESULT_ERROR_OUT_OF_DEVICE_MEMORY` contains no spaces, so the existing substring never matched. The decode therefore failed outright instead of retrying tiled. Match the `*_OUT_OF_DEVICE_MEMORY` / `*_OUT_OF_HOST_MEMORY` spellings (both UR and ZE prefixes) alongside the existing conditions, and fold the cuDNN/cuBLAS checks into the same case-insensitive comparison. Extends the existing parametrized retry test with the three XPU spellings; each was verified to fail before this change. Note the driver behaviour itself is not reproducible on the hardware used here -- this stack overcommits into host RAM and hangs rather than raising -- so the tests pin the classifier, not the driver. * style: wrap long vram_usage_gb ternary for ruff * fix: drop CUDA-only wording from progress device description Matches the committed openapi/schema artifacts, which already say "on a GPU". * docs: regenerate settings data for xpu device values * fix: stop xpu VRAM probe from reporting an unknown total as zero (0, 0) made the cache's available-VRAM arithmetic collapse to a constant -working_mem budget for the life of the process. Also widen the except: the failure type moves between torch releases (RuntimeError for the missing SYCL aspect, AssertionError from _lazy_init), and warn once when the blind estimate is in use. * fix: probe fp8 support on the target device, per device, without caching failures The probe allocated via an index-less "xpu", which resolves through the thread's current XPU device rather than the device being loaded onto -- so during idle-GPU encoder offload it measured the busy denoise GPU. It was also keyed on device type, letting one device decide for another, and memoised transient failures (it runs during a load, when the device may be momentarily full) with no way back but a restart. Also probe the bf16 upcast, which is the runtime path for Krea-2/FLUX. * fix: pin torch current device when borrowing an idle GPU Worker startup set both the session device and torch's per-thread current device; the offload borrow set only the former, leaving index-less allocations on the worker's own GPU. Extracted the shared helper and guarded it on backend availability. * fix: keep idle-GPU borrows within one device type generation_devices accepts a mixed list, so a cuda session could be handed an xpu device for its text encoder. * feat: detect Intel integrated GPUs via Level Zero torch exposes no is-integrated flag, but Level Zero does (ZE_DEVICE_PROPERTY_FLAG_INTEGRATED), and its loader already ships with the torch+xpu runtime -- so no new dependency and no compiled extension. Use it to keep iGPUs out of `generation_devices: auto` when a discrete GPU exists, and to stop budgeting them as dedicated VRAM (they share system RAM, like MPS). An unknown answer keeps the previous behaviour, an iGPU-only machine keeps its device, and an explicit device list can still opt one in. * feat: add xpu torch index to pins.json Gives the launcher an Intel install option instead of requiring a manual pip install of the extra. * fix: report VRAM diagnostics for the device in use All three sites dispatched on torch.cuda.is_available() first, so a mixed NVIDIA + Arc box running on xpu reported a constant 0.0 GB and logged "CUDA Memory Allocated" -- which would make XPU bug reports unactionable. * docs: record why xpu takes the CUDA VAE constants and keeps the broad OOM needle XPU SDPA was measured on Arc Pro B70 / torch 2.13+xpu: peak memory doubles when sequence length doubles (2.00x across 2048-16384; 2.0 MB at seq=16384 vs 512 MB for a materialised score matrix). So XPU is in CUDA's O(area) regime, not ROCm's math-attention regime, and the existing constants are correct rather than accidental. * fix: derive rand_device metadata from the backend's devices Was hardcoded to 'cuda' for any non-CPU noise, which is wrong on Arc. Falls back to 'cuda' when the device query has not resolved, so Nvidia metadata is unchanged. * docs: add Intel Arc install, driver and VRAM-reporting notes * fix: probe fp8 device support only when a model requests it The probe was the first statement in _should_use_fp8, so it allocated on the GPU during the first load of any model at all -- tokenizer, VAE, scheduler -- and on API/install threads it forced XPU lazy SYCL init on a thread that never generates. Moved below the exclusions. * fix: query Level Zero Sysman for driver-global free VRAM on xpu The blind estimate (total minus this process's reserved bytes) is what made _get_vram_available over-commit on a shared GPU: it feeds a formula that assumes a driver-global figure. Sysman's zesMemoryGetState reports that figure and is often available when the SYCL ext_intel_free_memory aspect is not, so try it before estimating. Measured on Arc Pro B70 with 16 GiB held by another process: Sysman reported 15.553 GiB free, the estimate 31.725 -- a 16.172 GiB error, exactly the foreign allocation. Sysman is not a guaranteed substitute (torch's query bottoms out in the same layer), so the estimate remains as a last resort. * fix: make the fp8 probe mirror the runtime cast path The storage cast happens on CPU while params are still CPU-resident, then the fp8 tensor is copied to the device and the pre-hook upcasts there. Probing all three steps on the device would pass on a build where the host->device fp8 copy or one upcast target fails, and break at forward time instead. Verified on Arc Pro B70 / torch 2.13+xpu: the full sequence works on both cards. * fix: degrade gracefully when a backend cannot name a device torch.xpu.get_device_name goes through _lazy_init, which raises AssertionError on a build without XPU. Naming is used only for labels and logs, so fall back to the device string rather than propagating. * fix: resolve an index-less device in the Sysman VRAM query Returning None for a device with no index would skip the driver-global query and fall through to the blind estimate with no visible symptom. Callers currently always pass a concrete device, so this is a latent hazard rather than a live bug. * fix: declare ctypes prototypes for the Level Zero calls Handles come back from (c_void_p * n)() as plain Python ints, and ctypes converts an undeclared int argument to a C int -- 32 bits. Any handle above 2**31 was being silently truncated; a direct test of that path segfaults. It happened to work on the B70 because the handles fit. Also: release the idle-GPU borrow if re-pinning raises (the setup was outside the try, so a failure there stranded the lock for the life of the process), and report a failing fp8 probe once per device instead of on every model load. * fix: drop the ZES_ENABLE_SYSMAN mutation from the Sysman probe Setting a process-wide environment variable from a read-only query leaks into child processes. It also bought nothing: the variable only gates Sysman on runtimes predating zesInit and must be set before Level Zero initialises, which torch has already done by then. Verified on Arc Pro B70 that zesInit succeeds with the variable unset. * refactor: tidy up the xpu additions after a cleanup review level_zero: cache the loader so it is opened and its prototypes configured once rather than twice, share the driver/device enumeration and its ordering guard between the two probes, and collapse the Sysman pair of globals into one nullable tuple. Also: fp8 support cache is a set (it only ever stored True), the pbr_maps empty_cache is routed through TorchDevice like the PR's other conversions, the shared-memory VRAM branch stops re-testing the device type it matched on, `_auto_generation_devices` partitions in one pass, and rand_device only answers when every generation device is the same accelerator. Merges three duplicate mem_get_info tests into one parametrized case and drops two fp8 probe tests fully subsumed by the cast-sequence test. * build: pin the xpu extra to torch 2.13.0 Intel's XPU backend matured considerably after 2.7.1: torch.xpu.mem_get_info() works on driver/kernel combinations where it previously raised, and the oneAPI user-space runtime ships with the wheel, so upgrading torch upgrades it too. Follows the rocm extra, which already pins ahead of cpu/cuda. pytorch-triton-xpu was renamed triton-xpu upstream. The darwin/aarch64 fallbacks stay on 2.7.1 to match the other extras and the project's torch<2.8.0 constraint on darwin. cpu/cuda/rocm exports are unchanged package-for-package (196/211/197); the only delta is a dropped "via pytorch-triton-xpu" comment annotation from the rename. * test: stub Sysman in the unknown-total xpu probe test Without it the test only passes where Level Zero cannot answer -- never on the Intel hardware the probe exists for, where Sysman returns before the tier under test is reached. * build: teach the pins check about the xpu index Its per-platform allowlist rejects anything unlisted, so pins.json's xpu entry fails it. PyTorch publishes XPU wheels for win32 and linux x86_64, matching the extra's markers. * fix: defer the xpu device pin like cuda's torch.xpu.set_device() brings up a SYCL context that holds VRAM in an otherwise idle process, the same reason the CUDA pin waits for the first claimed queue item. * docs: regenerate settings data on linux Regenerating on Windows flips two path defaults to backslashes, which the docs check rejects. * fix(mm): handle shared memory on integrated GPUs Their VRAM is system RAM, so a RAM copy doubles each model's footprint against the same pool. Drop it, letting a full load move weights rather than copy them. Keep partial loading on -- it is the only path that respects vram_available -- and raise a clean error when a full-load-only model cannot fit, instead of walking into an uncatchable OOM-kill. Warn once when a setting is overridden. Scoped to integrated XPU; CPU and MPS are unchanged. * docs: note intel device selection and integrated-GPU memory `auto` prefers CUDA on a mixed Nvidia/Arc box, and keep_ram_copy_of_weights is ignored on an integrated GPU. * chore(ui): typegen for the xpu device values * fix(mm): compare the integrated-GPU full-load guard against bytes still to move A resident model's weights occupy the same DRAM that vram_available is read from, so its total can exceed "available" precisely because it is loaded. lock() runs on every use and full_load_to_vram() is a no-op when resident; comparing the total refused the re-lock and evicted a healthy model on every other generation. Compare what full_load_to_vram() will actually move instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: run the integrated-GPU cache tests on CPU-only torch ModelCache.__init__ sizes the RAM cache from the device's total VRAM, which on an xpu execution device reads torch.xpu.get_device_properties() -- an AssertionError on the CPU-only builds CI runs, failing 9 of these tests before they reached their subject. Stub a fixed total during construction, and add a regression test for the resident-model re-lock guard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: LexiconCode <aaronwalker@protonmail.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> | 21 天前 | |
fix(api): stop synchronous route work from stalling the whole server (#9436) * fix(api): run gallery and search routes off the event loop The gallery list/name routes and the auth dependencies were declared `async def` while calling synchronous SQLite services, so their database work ran on the event loop. For its whole duration the process served no other request and delivered no socket.io event, which users experienced as the backend freezing mid-generation rather than as a slow gallery. Declaring them `def` hands them to Starlette's threadpool instead. Measured against a 200k-image, 1.7 GB database: the latency of an unrelated request issued while a gallery name query is in flight drops from 1662 ms to 4 ms (search) and from 2298 ms to 881 ms (no search). The queries themselves are unchanged; only the loop is freed. The residual 881 ms in the no-search case is response serialization of 202k items, which is tracked separately. Adds a regression test that stubs a blocking service call and asserts an unrelated route still answers during it, plus a contributor doc describing the rule. * perf(gallery): add a flat item-names endpoint and deprecate the legacy ones The name list that drives the virtualized gallery wrapped every entry in an object carrying a `kind` discriminator. Building those models cost 820ms of the 2225ms service call on a 200k-item library, and every consumer threw the field away — `itemRefsToNames` mapped it off immediately and each caller re-derived the kind from the file extension via `isVideoName`. Adds `GET /v1/gallery/item_names`, returning a flat name list in the same shape as the image-only `ImageNamesResult`. An optional `created_date` filter subsumes the separate by-date virtual-board route, so regular boards and virtual dates now share one endpoint, one cache and one query-args selector instead of a skipToken branch duplicated across the grid hook, range selection and both auto-select listeners. Measured on a 200k-image, 1.7 GB database: 2.51s -> 1.57s per request, 8.48 MB -> 3.85 MB of response, and the residual event-loop stall from serializing the response drops from 466ms to 102ms at p95. Existing integrations still call the old routes, so all five legacy name endpoints keep working and are marked `deprecated=True` with a pointer to the replacement. * perf(api): stop gzipping responses that are already compressed Starlette's GZipMiddleware compresses every response type except text/event-stream, so every image and video the gallery serves was being deflate-compressed a second time. Measured: a 1024x1024 PNG (3.00 MB) costs 52ms of event-loop time to gzip and comes back at 3.01 MB — larger than it went in; a 2048x2048 PNG costs 210ms for the same non-result. Compression runs on the event loop, so that time is a full stall of the process. With auto-switch enabled the UI fetches the full image after every generated image, so the cost lands repeatedly during a batch. Replaces it with a content-type-aware subclass that compresses an allowlist of text, JSON, XML and SVG responses and passes everything else through. The UI bundle and the API's JSON keep their compression unchanged. Lowering compresslevel is not an alternative for this case: on already-compressed input level 1 costs 51ms against level 9's 52ms, because deflate still scans the whole body. Making the level configurable is worthwhile for the *compressible* path and is tracked separately. Note for deployments: media responses no longer carry Content-Encoding: gzip. * feat(queue): add lightweight item summaries endpoint * build: unpin FastAPI and move to 0.141.1 The pin sat at 0.118.3 with a comment guessing the OpenAPI crash on 0.119 was "probably Invoke's [bug], because we are doing something unusual with AnyInvocation". It was not: fastapi/_compat/v2.py assumed every field mapping carries a `$ref` and raised KeyError otherwise. Upstream fixed it in 0.124.0 with no change needed here. Two later changes needed adapting to, both of which fail silently: - 0.130 emits `contentMediaType: application/octet-stream` instead of `format: binary` for file uploads. typegen.js mapped only the latter to `Blob`, so upload call sites would have started typing their `File` argument as `string`. It now maps both. - 0.141 keeps an included router as a single node in `app.routes` instead of copying its routes into it. The default-deny auth guard walked `app.routes` looking for APIRoute instances and found 2 of 197 — passing while inspecting almost nothing. It now walks `iter_route_contexts`, the traversal FastAPI's own OpenAPI generation uses, and asserts a floor on the route count so going blind fails loudly instead. Schema changes are limited to ValidationError gaining the optional `input`/`ctx` fields; upload fields still resolve to Blob. Starlette stays at 0.48.0. * perf(api): run every synchronous route handler off the event loop Package A converted the eight gallery and search routes that caused the reported multi-minute stalls. The same defect was present across the rest of the API: 167 route handlers were declared `async def` while awaiting nothing, so their synchronous service calls ran on the event loop. Each one stalls the entire process for its duration - no other request served, no socket.io event delivered - which is why the symptom looked like the application freezing rather than one slow endpoint. Candidates were identified by AST rather than by hand: `async def` route handlers with no `await`, `async with` or `async for` anywhere in the body, cross-checked for references to asyncio, anyio or the loop. Two flagged candidates were false positives (both the word "loop" in a comment). The diff is 167 signature lines plus one signature that ruff collapsed onto a single line once `async ` was removed. Adds tests/app/routers/test_no_blocking_async_routes.py, which enforces the rule for every handler including ones written later - a per-route test cannot cover a route that does not exist yet, and this failure mode is invisible until a user has a large enough library to notice. Two tests that invoked route handlers directly were updated to call them as the plain functions they now are. * Docs Changes * Chore openapi * fix(queue): bound and chunk the id list on the queue summary route `item_summaries_by_ids` expanded every client-supplied id into one SQLite bind parameter, with no limit on the route. Posting more ids than the per-statement variable limit (32766 on SQLite >= 3.32) raised `OperationalError: too many SQL variables`, which the route reported as a generic HTTP 500. Cap the request body at 1000 ids so oversized lists are rejected by validation before any database work starts, matching the existing MAX_VIDEO_BATCH_SIZE precedent. Independently, chunk the `IN (...)` expansion at 900 binds so no caller — including internal ones not covered by the route bound — can hit the ceiling; 900 stays under the 999 limit of pre-3.32 builds too. Both regression tests fail without the fix: the router test posts 32767 ids and gets 200 instead of 422, and the service test reproduces the OperationalError verbatim, sized off the limit the running SQLite build actually enforces. * fix(api): close the two check-then-act races the sync sweep opened Converting the routes from `async def` to `def` removed an implicit guarantee: a handler whose body contains no `await` could not be interleaved with another request, because the event loop had no point at which to switch. Two handlers relied on it. `POST /auth/setup` did has_admin() then create_admin() in separate transactions. Two concurrent requests both saw no admin and both created one, so the loser ended up with a persistent admin account instead of the intended 400. The condition now lives inside the INSERT's own transaction, behind BEGIN IMMEDIATE so a second process (invoke-useradd --admin) cannot slip a write in either. This mirrors what #9360 does for the update/delete last-admin invariant; create_admin was the one path it does not cover. Custom node install, uninstall and reload all mutate the same custom-nodes directory, sys.modules and invocation registry. Interleaved, a failed install's cleanup rmtree'd the directory a concurrent install had just cloned into. A module-level lock restores the exclusion; the install and uninstall bodies moved into helpers so the lock scope is visible rather than an 80-line reindent. Both regression tests fail without their fix: the admin one creates two administrators, the pack one loses the successful install's directory. perf(queue): render the queue list from summaries, with one sanitizer The list fetched full queue items for every visible row, each carrying its session graph and workflow — megabytes per screenful for fields no row draws. The rows now render from SessionQueueItemSummary and the full item is fetched only when a row is expanded, which is what the summary route added in this branch was for; until now nothing consumed it. The per-item summary query provides the same cache tags as getQueueItem, so every existing invalidation path covers the list rows with nothing to wire up, and the optimistic status write is mirrored so a row's status still flips without a round trip. The range hook batches at the backend's 1000-id limit, which a fast fling could otherwise exceed. Both sanitizers are now one generic function over a single redaction table: the summary and the full item are two projections of one row, and a field stripped from the list but left on the detail view is leaked anyway. A test walks the intersection of both models and asserts they redact it identically. `device` is deliberately not redacted in either. It names the instance's GPU rather than anything about the other user's work, and the list has always shown it — redacting it here would have quietly changed what non-admins see. parent_item_id joins the summary because the rows decide from it whether to offer a retry. * fix(api): close the sync-sweep races and wire up the queue summary route Two review follow-ups landed together here. Converting the routes from `async def` to `def` removed an implicit guarantee: a handler whose body contains no `await` could not be interleaved with another request, because the event loop had no point at which to switch. `POST /auth/setup` did has_admin() then create_admin() in separate transactions. Two concurrent requests both saw no admin and both created one, so the loser ended up with a persistent admin account instead of the intended 400. The condition now lives inside the INSERT's own transaction, behind BEGIN IMMEDIATE so a second process (invoke-useradd --admin) cannot slip a write in either. This mirrors what #9360 does for the update/delete last-admin invariant; create_admin is the one path it does not cover. Custom node install, uninstall and reload all mutate the same custom-nodes directory, sys.modules and invocation registry. Interleaved, a failed install's cleanup rmtree'd the directory a concurrent install had just cloned into. A module-level lock restores the exclusion; the install and uninstall bodies moved into helpers so the lock scope is visible rather than an 80-line reindent. Both regression tests fail without their fix: the admin one creates two administrators, the pack one loses the successful install's directory. The route added earlier in this branch had none — the list still fetched full queue items for every visible row, each carrying its session graph and workflow, so the claimed saving was not being realised. The rows now render from SessionQueueItemSummary and the full item is fetched only when a row is expanded. Measured against the previous commit, same backend and same 396-item queue, identical scroll (page load, queue tab, scroll to 60%): requests 62 -> 2 payload (gzip) 262 KB -> 1.3 KB (30 items) server time 60ms -> 4ms (30 items) The request count collapses because the old path was self-amplifying: the range hook re-asks which ids are uncached on every range event, and at ~60ms per response the cache had not filled yet, so overlapping fetches piled up. A side effect worth knowing: `items_by_ids` silently skips items it cannot deserialize, so a queue item whose graph references an unregistered node type left its row permanently blank. Summaries never touch the graph, so the row now renders and only the expanded detail is affected. The per-item summary query provides the same cache tags as getQueueItem, so every existing invalidation path covers the list rows with nothing to wire up; the optimistic status write is mirrored so a row still flips without a round trip. The range hook batches at the backend's 1000-id limit, which a fast fling could otherwise exceed. The summary and the full item are two projections of one row, and a field stripped from the list but left on the detail view is leaked anyway. Both now go through one generic function over a single redaction table; a test walks the intersection of the two models and asserts they redact it identically. `device` is deliberately not redacted in either. It names the instance's GPU rather than anything about the other user's work, and the list has always shown it — redacting it would have quietly changed what non-admins see. parent_item_id joins the summary because the rows decide from it whether to offer a retry. * fix(api): finish the review's non-blocking list Six follow-ups from @lstein's sweep that were left open. `require_admin` and `require_admin_or_default` go back to `async def`. They only read `is_admin` off already-resolved token data, so declaring them `def` bought a threadpool round-trip per admin request and nothing else. The `users.get` that can block lives in `get_current_user`, which stays synchronous — the docstrings now say why the two layers differ. The AST guard now inspects what it claims to. It walked only `tree.body`, so a handler registered from inside a factory function or an `if` block was never seen, and `_awaits_something` used `ast.walk`, which counts `await`s inside nested closures — a handler could have passed by defining an inner async helper it never awaits. Both are fixed and both now have their own tests, so the guard's behaviour is pinned rather than asserted in a comment. `convert_model` takes a lock non-blocking and answers 409 otherwise. Blocking would be wrong: a conversion runs for minutes, and for the same key the second caller reads a record the first is midway through replacing. Two conversions in flight also means two models resident at once, which nothing bounds. The body moved into `_convert_model` so the lock scope is visible. Tested for the 409 and for the lock surviving a failed conversion rather than wedging the endpoint for the process's lifetime. `do_hf_login` and `reset_hf_token` hold a lock across the write and the status read-back, which otherwise could report a status belonging to a different token than the one just written. The blocking-work doc gains the bound it was missing: anyio's thread limiter holds 40 tokens, so past forty concurrent blocking requests the stall moves rather than vanishes — and anything else needing a thread queues behind them, including the synchronous auth dependency that runs before a handler is reached. Noted there too that `test_event_loop_blocking.py` cannot show this, because its probe route has neither auth nor database access. `QueueItemDetail` tells a failed fetch apart from a pending one. A queue item the backend cannot serve — one whose graph references a node type this build no longer registers — previously read as "Loading" forever. Left alone deliberately: the stale `old_is_public` in the workflow-updated event, which is cosmetic and would need `workflow_records.update()` to return the previous row to fix properly; and the `delete_user` / `update_user` last-admin invariants, which belong to #9360. * chore: drop planning notes and scratch files from the branch These arrived via a merge of the fork's own branch, where they had been tracked since an earlier `git add -A`: ten *_PLAN.md files at the repo root, the `plans/` tree (fp8-compute, pid-porting, gzip-compresslevel) and `testscript.py`. None of them belong to this change — they are working notes for unrelated features — and they made up 21 of the 90 files a reviewer had to page past. The files stay on disk; only the index drops them. They are listed in .git/info/exclude locally rather than in .gitignore, so the repository carries no opinion about one contributor's notes. * fix(models): serialize the operations that share the models directory Follow-up to the conversion lock, which bounded conversions against each other but not against everything else that mutates a model now that those routes run in the threadpool too. `delete_model` and `bulk_delete_models` ran free alongside a conversion. Conversion is a read-modify-replace spanning many service calls — load, write a diffusers copy, rename the record, install the copy, delete the original — so a delete landing in the middle removes the record it is still working from. The conversion's own final delete then fails, and the copy it already installed survives: the admin is answered 204 and the model reappears under a new key. A per-key claim serializes operations on one model while leaving different models free to run in parallel; a global lock would have made every delete wait out an unrelated conversion. Bulk deletion claims each key separately and reports a busy one through its existing per-key `failed` list rather than aborting the request or racing the holder. Deletion never takes the conversion lock, so the two are always acquired in the same order. `DELETE /sync/orphaned` was the same collision from the other side. An orphan is defined as model files under the models root with no database record, which is also an exact description of a conversion in progress: it built its diffusers copy in a `TemporaryDirectory` directly under `models/`, so a scan taken during a conversion reported that working directory and the delete route would rmtree it mid-write. Fixed at the cause rather than with another lock — the copy is now built in `models/.convert_tmp`, still on the models volume so `install_path` moves rather than copies across a filesystem boundary, but named in `SKIP_DIRS`. The name lives next to that list as `CONVERSION_SCRATCH_DIRNAME` so writer and scanner cannot drift apart. All three regression tests fail without their fix: the delete reaches the installer mid-conversion, bulk deletion removes the busy key, and the scan reports `.convert_tmp` as an orphan. The scan test carries a control asserting a real orphan is still found, so a scan that has stopped finding anything cannot pass it. The same scan is equally blind to an in-flight install, but the installer has always run in its own worker thread — that race predates this branch and is left alone. * fix(models): extend the per-key claim to every operation on a model record Follow-up to 6c6e38e, which serialized conversion against deletion but left the operations that rewrite a record or its image running free beside it. The hazard is not the individual write — it is that conversion carries a snapshot. It reads the config before it starts and, minutes later, writes that snapshot's name, description, hash and source into the replacement record, then moves the model image over to the new key. Anything accepted on the old key in between is answered 200 and then silently discarded: a rename vanishes, a re-probe's findings vanish, an uploaded cover image is replaced by the one the conversion carried, a deleted one comes back. So `reidentify_model`, `update_model_record`, `update_model_image`, `delete_model_image` and `bulk_reidentify_models` now take the same per-key claim as conversion and deletion. Bulk reidentification claims each key separately and reports a busy one through its existing per-key `failed` list rather than aborting the request. `update_model_record` is the one `async def` among them; the claim only holds a threading lock across a set membership test, never across the `await`, so it cannot deadlock the loop. `reidentify_model`'s body moved into `_reidentify_model` so the bulk route can call it instead of carrying its own copy of the retain-these-fields logic — the two copies had already drifted to opposite `hasattr` orderings, and only one of them would have been updated the next time that list changes. Four new regression tests, each holding a conversion at a barrier and racing one operation against it; with the claim removed, all six of this file's race tests fail. * chore: typegen/openapi * Addressed PR comments * Addressed PR comments --------- Co-authored-by: JPPhoto <jpollack@jpollackphoto.com> Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com> | 18 天前 | |
fix(api): stop synchronous route work from stalling the whole server (#9436) * fix(api): run gallery and search routes off the event loop The gallery list/name routes and the auth dependencies were declared `async def` while calling synchronous SQLite services, so their database work ran on the event loop. For its whole duration the process served no other request and delivered no socket.io event, which users experienced as the backend freezing mid-generation rather than as a slow gallery. Declaring them `def` hands them to Starlette's threadpool instead. Measured against a 200k-image, 1.7 GB database: the latency of an unrelated request issued while a gallery name query is in flight drops from 1662 ms to 4 ms (search) and from 2298 ms to 881 ms (no search). The queries themselves are unchanged; only the loop is freed. The residual 881 ms in the no-search case is response serialization of 202k items, which is tracked separately. Adds a regression test that stubs a blocking service call and asserts an unrelated route still answers during it, plus a contributor doc describing the rule. * perf(gallery): add a flat item-names endpoint and deprecate the legacy ones The name list that drives the virtualized gallery wrapped every entry in an object carrying a `kind` discriminator. Building those models cost 820ms of the 2225ms service call on a 200k-item library, and every consumer threw the field away — `itemRefsToNames` mapped it off immediately and each caller re-derived the kind from the file extension via `isVideoName`. Adds `GET /v1/gallery/item_names`, returning a flat name list in the same shape as the image-only `ImageNamesResult`. An optional `created_date` filter subsumes the separate by-date virtual-board route, so regular boards and virtual dates now share one endpoint, one cache and one query-args selector instead of a skipToken branch duplicated across the grid hook, range selection and both auto-select listeners. Measured on a 200k-image, 1.7 GB database: 2.51s -> 1.57s per request, 8.48 MB -> 3.85 MB of response, and the residual event-loop stall from serializing the response drops from 466ms to 102ms at p95. Existing integrations still call the old routes, so all five legacy name endpoints keep working and are marked `deprecated=True` with a pointer to the replacement. * perf(api): stop gzipping responses that are already compressed Starlette's GZipMiddleware compresses every response type except text/event-stream, so every image and video the gallery serves was being deflate-compressed a second time. Measured: a 1024x1024 PNG (3.00 MB) costs 52ms of event-loop time to gzip and comes back at 3.01 MB — larger than it went in; a 2048x2048 PNG costs 210ms for the same non-result. Compression runs on the event loop, so that time is a full stall of the process. With auto-switch enabled the UI fetches the full image after every generated image, so the cost lands repeatedly during a batch. Replaces it with a content-type-aware subclass that compresses an allowlist of text, JSON, XML and SVG responses and passes everything else through. The UI bundle and the API's JSON keep their compression unchanged. Lowering compresslevel is not an alternative for this case: on already-compressed input level 1 costs 51ms against level 9's 52ms, because deflate still scans the whole body. Making the level configurable is worthwhile for the *compressible* path and is tracked separately. Note for deployments: media responses no longer carry Content-Encoding: gzip. * feat(queue): add lightweight item summaries endpoint * build: unpin FastAPI and move to 0.141.1 The pin sat at 0.118.3 with a comment guessing the OpenAPI crash on 0.119 was "probably Invoke's [bug], because we are doing something unusual with AnyInvocation". It was not: fastapi/_compat/v2.py assumed every field mapping carries a `$ref` and raised KeyError otherwise. Upstream fixed it in 0.124.0 with no change needed here. Two later changes needed adapting to, both of which fail silently: - 0.130 emits `contentMediaType: application/octet-stream` instead of `format: binary` for file uploads. typegen.js mapped only the latter to `Blob`, so upload call sites would have started typing their `File` argument as `string`. It now maps both. - 0.141 keeps an included router as a single node in `app.routes` instead of copying its routes into it. The default-deny auth guard walked `app.routes` looking for APIRoute instances and found 2 of 197 — passing while inspecting almost nothing. It now walks `iter_route_contexts`, the traversal FastAPI's own OpenAPI generation uses, and asserts a floor on the route count so going blind fails loudly instead. Schema changes are limited to ValidationError gaining the optional `input`/`ctx` fields; upload fields still resolve to Blob. Starlette stays at 0.48.0. * perf(api): run every synchronous route handler off the event loop Package A converted the eight gallery and search routes that caused the reported multi-minute stalls. The same defect was present across the rest of the API: 167 route handlers were declared `async def` while awaiting nothing, so their synchronous service calls ran on the event loop. Each one stalls the entire process for its duration - no other request served, no socket.io event delivered - which is why the symptom looked like the application freezing rather than one slow endpoint. Candidates were identified by AST rather than by hand: `async def` route handlers with no `await`, `async with` or `async for` anywhere in the body, cross-checked for references to asyncio, anyio or the loop. Two flagged candidates were false positives (both the word "loop" in a comment). The diff is 167 signature lines plus one signature that ruff collapsed onto a single line once `async ` was removed. Adds tests/app/routers/test_no_blocking_async_routes.py, which enforces the rule for every handler including ones written later - a per-route test cannot cover a route that does not exist yet, and this failure mode is invisible until a user has a large enough library to notice. Two tests that invoked route handlers directly were updated to call them as the plain functions they now are. * Docs Changes * Chore openapi * fix(queue): bound and chunk the id list on the queue summary route `item_summaries_by_ids` expanded every client-supplied id into one SQLite bind parameter, with no limit on the route. Posting more ids than the per-statement variable limit (32766 on SQLite >= 3.32) raised `OperationalError: too many SQL variables`, which the route reported as a generic HTTP 500. Cap the request body at 1000 ids so oversized lists are rejected by validation before any database work starts, matching the existing MAX_VIDEO_BATCH_SIZE precedent. Independently, chunk the `IN (...)` expansion at 900 binds so no caller — including internal ones not covered by the route bound — can hit the ceiling; 900 stays under the 999 limit of pre-3.32 builds too. Both regression tests fail without the fix: the router test posts 32767 ids and gets 200 instead of 422, and the service test reproduces the OperationalError verbatim, sized off the limit the running SQLite build actually enforces. * fix(api): close the two check-then-act races the sync sweep opened Converting the routes from `async def` to `def` removed an implicit guarantee: a handler whose body contains no `await` could not be interleaved with another request, because the event loop had no point at which to switch. Two handlers relied on it. `POST /auth/setup` did has_admin() then create_admin() in separate transactions. Two concurrent requests both saw no admin and both created one, so the loser ended up with a persistent admin account instead of the intended 400. The condition now lives inside the INSERT's own transaction, behind BEGIN IMMEDIATE so a second process (invoke-useradd --admin) cannot slip a write in either. This mirrors what #9360 does for the update/delete last-admin invariant; create_admin was the one path it does not cover. Custom node install, uninstall and reload all mutate the same custom-nodes directory, sys.modules and invocation registry. Interleaved, a failed install's cleanup rmtree'd the directory a concurrent install had just cloned into. A module-level lock restores the exclusion; the install and uninstall bodies moved into helpers so the lock scope is visible rather than an 80-line reindent. Both regression tests fail without their fix: the admin one creates two administrators, the pack one loses the successful install's directory. perf(queue): render the queue list from summaries, with one sanitizer The list fetched full queue items for every visible row, each carrying its session graph and workflow — megabytes per screenful for fields no row draws. The rows now render from SessionQueueItemSummary and the full item is fetched only when a row is expanded, which is what the summary route added in this branch was for; until now nothing consumed it. The per-item summary query provides the same cache tags as getQueueItem, so every existing invalidation path covers the list rows with nothing to wire up, and the optimistic status write is mirrored so a row's status still flips without a round trip. The range hook batches at the backend's 1000-id limit, which a fast fling could otherwise exceed. Both sanitizers are now one generic function over a single redaction table: the summary and the full item are two projections of one row, and a field stripped from the list but left on the detail view is leaked anyway. A test walks the intersection of both models and asserts they redact it identically. `device` is deliberately not redacted in either. It names the instance's GPU rather than anything about the other user's work, and the list has always shown it — redacting it here would have quietly changed what non-admins see. parent_item_id joins the summary because the rows decide from it whether to offer a retry. * fix(api): close the sync-sweep races and wire up the queue summary route Two review follow-ups landed together here. Converting the routes from `async def` to `def` removed an implicit guarantee: a handler whose body contains no `await` could not be interleaved with another request, because the event loop had no point at which to switch. `POST /auth/setup` did has_admin() then create_admin() in separate transactions. Two concurrent requests both saw no admin and both created one, so the loser ended up with a persistent admin account instead of the intended 400. The condition now lives inside the INSERT's own transaction, behind BEGIN IMMEDIATE so a second process (invoke-useradd --admin) cannot slip a write in either. This mirrors what #9360 does for the update/delete last-admin invariant; create_admin is the one path it does not cover. Custom node install, uninstall and reload all mutate the same custom-nodes directory, sys.modules and invocation registry. Interleaved, a failed install's cleanup rmtree'd the directory a concurrent install had just cloned into. A module-level lock restores the exclusion; the install and uninstall bodies moved into helpers so the lock scope is visible rather than an 80-line reindent. Both regression tests fail without their fix: the admin one creates two administrators, the pack one loses the successful install's directory. The route added earlier in this branch had none — the list still fetched full queue items for every visible row, each carrying its session graph and workflow, so the claimed saving was not being realised. The rows now render from SessionQueueItemSummary and the full item is fetched only when a row is expanded. Measured against the previous commit, same backend and same 396-item queue, identical scroll (page load, queue tab, scroll to 60%): requests 62 -> 2 payload (gzip) 262 KB -> 1.3 KB (30 items) server time 60ms -> 4ms (30 items) The request count collapses because the old path was self-amplifying: the range hook re-asks which ids are uncached on every range event, and at ~60ms per response the cache had not filled yet, so overlapping fetches piled up. A side effect worth knowing: `items_by_ids` silently skips items it cannot deserialize, so a queue item whose graph references an unregistered node type left its row permanently blank. Summaries never touch the graph, so the row now renders and only the expanded detail is affected. The per-item summary query provides the same cache tags as getQueueItem, so every existing invalidation path covers the list rows with nothing to wire up; the optimistic status write is mirrored so a row still flips without a round trip. The range hook batches at the backend's 1000-id limit, which a fast fling could otherwise exceed. The summary and the full item are two projections of one row, and a field stripped from the list but left on the detail view is leaked anyway. Both now go through one generic function over a single redaction table; a test walks the intersection of the two models and asserts they redact it identically. `device` is deliberately not redacted in either. It names the instance's GPU rather than anything about the other user's work, and the list has always shown it — redacting it would have quietly changed what non-admins see. parent_item_id joins the summary because the rows decide from it whether to offer a retry. * fix(api): finish the review's non-blocking list Six follow-ups from @lstein's sweep that were left open. `require_admin` and `require_admin_or_default` go back to `async def`. They only read `is_admin` off already-resolved token data, so declaring them `def` bought a threadpool round-trip per admin request and nothing else. The `users.get` that can block lives in `get_current_user`, which stays synchronous — the docstrings now say why the two layers differ. The AST guard now inspects what it claims to. It walked only `tree.body`, so a handler registered from inside a factory function or an `if` block was never seen, and `_awaits_something` used `ast.walk`, which counts `await`s inside nested closures — a handler could have passed by defining an inner async helper it never awaits. Both are fixed and both now have their own tests, so the guard's behaviour is pinned rather than asserted in a comment. `convert_model` takes a lock non-blocking and answers 409 otherwise. Blocking would be wrong: a conversion runs for minutes, and for the same key the second caller reads a record the first is midway through replacing. Two conversions in flight also means two models resident at once, which nothing bounds. The body moved into `_convert_model` so the lock scope is visible. Tested for the 409 and for the lock surviving a failed conversion rather than wedging the endpoint for the process's lifetime. `do_hf_login` and `reset_hf_token` hold a lock across the write and the status read-back, which otherwise could report a status belonging to a different token than the one just written. The blocking-work doc gains the bound it was missing: anyio's thread limiter holds 40 tokens, so past forty concurrent blocking requests the stall moves rather than vanishes — and anything else needing a thread queues behind them, including the synchronous auth dependency that runs before a handler is reached. Noted there too that `test_event_loop_blocking.py` cannot show this, because its probe route has neither auth nor database access. `QueueItemDetail` tells a failed fetch apart from a pending one. A queue item the backend cannot serve — one whose graph references a node type this build no longer registers — previously read as "Loading" forever. Left alone deliberately: the stale `old_is_public` in the workflow-updated event, which is cosmetic and would need `workflow_records.update()` to return the previous row to fix properly; and the `delete_user` / `update_user` last-admin invariants, which belong to #9360. * chore: drop planning notes and scratch files from the branch These arrived via a merge of the fork's own branch, where they had been tracked since an earlier `git add -A`: ten *_PLAN.md files at the repo root, the `plans/` tree (fp8-compute, pid-porting, gzip-compresslevel) and `testscript.py`. None of them belong to this change — they are working notes for unrelated features — and they made up 21 of the 90 files a reviewer had to page past. The files stay on disk; only the index drops them. They are listed in .git/info/exclude locally rather than in .gitignore, so the repository carries no opinion about one contributor's notes. * fix(models): serialize the operations that share the models directory Follow-up to the conversion lock, which bounded conversions against each other but not against everything else that mutates a model now that those routes run in the threadpool too. `delete_model` and `bulk_delete_models` ran free alongside a conversion. Conversion is a read-modify-replace spanning many service calls — load, write a diffusers copy, rename the record, install the copy, delete the original — so a delete landing in the middle removes the record it is still working from. The conversion's own final delete then fails, and the copy it already installed survives: the admin is answered 204 and the model reappears under a new key. A per-key claim serializes operations on one model while leaving different models free to run in parallel; a global lock would have made every delete wait out an unrelated conversion. Bulk deletion claims each key separately and reports a busy one through its existing per-key `failed` list rather than aborting the request or racing the holder. Deletion never takes the conversion lock, so the two are always acquired in the same order. `DELETE /sync/orphaned` was the same collision from the other side. An orphan is defined as model files under the models root with no database record, which is also an exact description of a conversion in progress: it built its diffusers copy in a `TemporaryDirectory` directly under `models/`, so a scan taken during a conversion reported that working directory and the delete route would rmtree it mid-write. Fixed at the cause rather than with another lock — the copy is now built in `models/.convert_tmp`, still on the models volume so `install_path` moves rather than copies across a filesystem boundary, but named in `SKIP_DIRS`. The name lives next to that list as `CONVERSION_SCRATCH_DIRNAME` so writer and scanner cannot drift apart. All three regression tests fail without their fix: the delete reaches the installer mid-conversion, bulk deletion removes the busy key, and the scan reports `.convert_tmp` as an orphan. The scan test carries a control asserting a real orphan is still found, so a scan that has stopped finding anything cannot pass it. The same scan is equally blind to an in-flight install, but the installer has always run in its own worker thread — that race predates this branch and is left alone. * fix(models): extend the per-key claim to every operation on a model record Follow-up to 6c6e38e, which serialized conversion against deletion but left the operations that rewrite a record or its image running free beside it. The hazard is not the individual write — it is that conversion carries a snapshot. It reads the config before it starts and, minutes later, writes that snapshot's name, description, hash and source into the replacement record, then moves the model image over to the new key. Anything accepted on the old key in between is answered 200 and then silently discarded: a rename vanishes, a re-probe's findings vanish, an uploaded cover image is replaced by the one the conversion carried, a deleted one comes back. So `reidentify_model`, `update_model_record`, `update_model_image`, `delete_model_image` and `bulk_reidentify_models` now take the same per-key claim as conversion and deletion. Bulk reidentification claims each key separately and reports a busy one through its existing per-key `failed` list rather than aborting the request. `update_model_record` is the one `async def` among them; the claim only holds a threading lock across a set membership test, never across the `await`, so it cannot deadlock the loop. `reidentify_model`'s body moved into `_reidentify_model` so the bulk route can call it instead of carrying its own copy of the retain-these-fields logic — the two copies had already drifted to opposite `hasattr` orderings, and only one of them would have been updated the next time that list changes. Four new regression tests, each holding a conversion at a barrier and racing one operation against it; with the claim removed, all six of this file's race tests fail. * chore: typegen/openapi * Addressed PR comments * Addressed PR comments --------- Co-authored-by: JPPhoto <jpollack@jpollackphoto.com> Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com> | 18 天前 |
以下内容由 AI 翻译，如有问题请 点此提交 issue 反馈
Invoke 是一款领先的创意引擎，旨在为专业人士和爱好者赋能。借助最新的人工智能驱动技术，生成并创作令人惊艳的视觉媒体。Invoke 提供业界领先的基于网络的用户界面，并作为多款商业产品的基础。
- 采用商业友好型许可，可免费使用
- 在兼容硬件上下载并安装
- 生成、优化、迭代图像及构建工作流
文档
| 快速链接 |
|---|
| 安装与更新 - 文档与教程 - 问题反馈 - 贡献指南 |
安装
要开始使用 Invoke，请下载启动器。
故障排除、常见问题与支持
请查看我们的常见问题，以获取常见安装问题及其他问题的解决方案。
如需更多帮助，请加入我们的 Discord。
功能特性
有关功能的完整详情，请参阅我们的文档。
Web 服务器与用户界面
Invoke 运行本地托管的 Web 服务器和 React 用户界面，提供业界领先的用户体验。
统一画布
统一画布是一个完全集成的画布实现，支持所有核心生成功能、图像补全/扩展、画笔工具等。这款创意工具让艺术家能够将 AI 作为创意协作者进行创作，可用于增强 AI 生成的图像、草图、摄影作品、渲染图等。
工作流与节点
Invoke 提供功能完备的工作流管理解决方案，使用户能够将基于节点的工作流的强大功能与用户界面的便捷性相结合。这允许开发可自定义的生成管道，并由希望创建特定工作流以支持其生产用例的用户共享。
画板与图库管理
Invoke 具备组织有序的图库系统，可在 Invoke 工作区中轻松存储、访问和重新混合您的内容。图像可以拖放到应用程序中任何基于图像的 UI 元素上，图像中丰富的元数据便于轻松调用工作流中使用的关键提示词或设置。
模型支持
- SD 1.5
- SD 2.0
- SDXL
- SD 3.5 Medium
- SD 3.5 Large
- CogView 4
- Flux.1 Dev
- Flux.1 Schnell
- Flux.1 Kontext
- Flux.1 Krea
- Flux Redux
- Flux Fill
- Flux.2 Dev
- Flux.2 Klein 4B
- Flux.2 Klein 9B
- Z-Image Turbo
- Z-Image Base
- Krea 2 Turbo
- Krea 2 Raw
- Anima
- Qwen Image
- Qwen Image Edit
- Ideogram 4
- ERNIE-Image
- ERNIE-Image-Turbo
- Nano Banana (仅 API)
- GPT Image (仅 API)
- Wan (仅 API)
其他功能
- 支持 ckpt、diffusers 以及部分 gguf 模型
- 图像放大工具
- 嵌入管理与支持
- 模型管理与支持
- 工作流创建与管理
- 基于节点的架构
- 对象分割与选择模型（SAM / SAM2）
贡献
我们非常鼓励任何希望为该项目做出贡献的人——无论是文档编写、功能开发、错误修复、代码清理、测试还是代码审查。
通过阅读我们的贡献文档、加入#dev-chat或 GitHub 讨论板，开始您的贡献之旅。
我们希望您使用 Invoke 时能获得与我们创建它时一样的乐趣，并希望您能选择成为我们社区的一员。
赞助商
Invoke 的开源开发工作由我们的赞助商提供支持。如果 Invoke 对您或您的企业有价值，请考虑赞助我们——您的支持将直接用于项目维护、新功能开发和社区支持。
我们衷心感谢以下赞助商：
支持者（每月 15 美元）
高级用户（每月 50 美元）
致谢
Invoke 是由来自全球充满热情且才华横溢的人们共同努力的成果。我们感谢他们投入的时间、辛勤工作和不懈努力。
本软件的原创部分版权 © 2024 归各贡献者所有。
项目介绍
Invoke is a leading creative engine for Stable Diffusion models, empowering professionals, artists, and enthusiasts to generate and create visual media using the latest AI-driven technologies. The solution offers an industry leading WebUI, and serves as the foundation for multiple commercial products.
Apache-2.0 Python19.15 K提交数ai-artartificial-intelligencegenerative-artimage-generationimg2imginpaintinglatent-diffusionlinuxmacosoutpaintingstable-diffusiontxt2imgwindows
定制我的领域