| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
fix(patches): support conv-shaped LoHa factors (#10048) | 3 天前 | |
fix(lora): map FLUX diffusers guidance-embedder layers onto guidance_in.in_layer/out_layer (#10045) | 4 天前 | |
fix(patches): return a cached LoRA patch to where patching found it A patch is the model cache's own object and BaseLayerPatch.to() rebinds its tensors in place, so anything patching does to its device or dtype outlives the patched scope inside a record the cache is still accounting for. Sidecar patching moved it onto the compute device and returned nothing: clear_patches() drops the module's reference, and the record is RAM-pinned, so cur_vram_bytes() reports zero and full_unload_from_vram() early-returns -- the cache cannot reclaim what it cannot see. Direct patching promoted it to float32 for the maths and returned only its device, doubling the record for the rest of the process while the cache billed the size it measured at put(). Measured on CUDA: sidecar cpu->cuda->cuda and direct bf16->fp32->fp32 before, both round-tripping unchanged after. | 1 个月前 | |
feat: multi-GPU parallel session execution (#9263) * feat(app): parallel multi-GPU session execution Run one generation session per configured GPU concurrently, with a tiled progress preview. Multi-user isolation is unchanged. Backed by five seams: - Per-thread device context (TorchDevice.set/get/clear_session_device); choose_torch_device() consults it first, so all device-selecting call sites resolve to the calling worker's GPU with no per-node changes. - Per-device model caches: build_model_manager builds one ModelCache per generation device; ModelLoadService.ram_cache resolves by current thread device; ram_caches fans out clear/drop/shutdown. - Atomic concurrent dequeue: a dequeue lock makes select+claim atomic so concurrent workers never claim the same item (works on FIFO; round-robin from #9086 slots in later). - Worker pool: one _SessionWorker per device, each pinning torch.cuda.set_device and its session device, with its own runner and cancel event; cancellation routes via an {item_id -> worker} lookup. Single-device installs keep the exact legacy single-worker behavior. Profiling disabled when >1 worker. - New config `generation_devices`; unset = legacy single-worker mode. Frontend: the canvas staging area already tiles per queue item; the main ImageViewer now tracks progress per session and renders a tile grid (ProgressImageTiles) when more than one session is active. Also adds a lock to ObjectSerializerForwardCache for concurrent access. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(tests): restore global device after multi-GPU cache routing test test_model_load_device_routing mutated the process-wide get_config() singleton (device = "cuda:0") to exercise the per-thread cache routing, but never restored it. The leaked CUDA device was then picked up by a later test (test_model_load::test_loading) via choose_torch_device(), which crashed with "Torch not compiled with CUDA enabled" on the CUDA-less CI runner. Add an autouse fixture to save/restore device and clear any pinned session device. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(ui): regenerate openapi schema and frontend types for generation_devices Regenerate openapi.json (make frontend-openapi) and the frontend schema.ts types (make frontend-typegen) so they include the new generation_devices config field, fixing the openapi-checks and typegen-checks CI jobs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(ui): regenerate openapi.json with uv to match CI generator `make frontend-openapi` used a bare `python` from a different environment that emitted the CacheStats @dataclass docstring as a schema description. CI generates the schema via `uv run`, which does not, so openapi-checks failed on the diff. Regenerate with the uv-locked environment to drop the stray description while keeping the generation_devices field. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(model-manager): serialize model construction against VRAM moves to prevent meta-device corruption Parallel multi-GPU session workers could intermittently crash with "unrecognized device meta" (denoise) or "Cannot copy out of meta tensor; no data!" (l2i), because model loading relies on process-global, non-thread-safe monkey-patches. accelerate.init_empty_weights() (used directly by the loaders and implicitly by diffusers' default low_cpu_mem_usage=True in from_pretrained) swaps torch.nn.Module.register_parameter globally for the duration of a load, routing every newly-registered parameter to the meta device. The model cache's VRAM load/unload runs nn.Module.load_state_dict(assign=True), whose assign path does setattr -> __setattr__ -> register_parameter. When one worker's VRAM move overlapped another worker's from_pretrained, the move's real weights got hijacked onto meta and blew up on the next .to(device). Introduce MODEL_LOAD_LOCK, a write-preferring readers-writer lock: - write lock = model construction (_load_and_cache, load_model_from_path), exclusive. - read lock = VRAM load/unload (ModelCache.lock(), repair_required_tensors_on_device). VRAM transfers across GPUs still overlap each other; they only block while a construction holds the write lock. The lock is always acquired before any per-cache lock to keep a consistent order and avoid an AB-BA deadlock with the writer's make_room/put. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): fix outpainting crash caused by model download collisions * fix(backend): make DiskImageFileStorage thread-safe for parallel sessions Image.open() is lazy: it reads the header but defers pixel decoding (and holds the file handle open) until the first .load()/.copy()/.convert(). The opened object was cached and the same object handed to every caller, so in multi-GPU parallel mode two session-processor worker threads could call .copy() on it concurrently and race on the shared file handle and decoder state. This surfaced as "broken data stream when reading image file" and "AssertionError: self.png is not None" during inpainting with batch >1. Force the decode (image.load()) before the object enters the cache so the cached object is safe for concurrent reads, and guard the cache structures (__cache / __cache_ids) with a lock since they are now mutated from multiple threads. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(ui): stack per-session progress bars during parallel generation The generation progress bars (under the Invoke button and the Viewer tab) both read a single global $lastProgressEvent atom, which every session overwrites. With parallel multi-GPU sessions this made the bar jump back and forth between sessions. Track progress per queue item id and render one bar per in-flight session, stacked vertically, each removed as its session reaches a terminal state. - stores.ts: add $progressEvents (map keyed by item_id), $activeProgressEvents (sorted), and set/clear helpers. - setEventListeners.tsx: populate per-item progress on invocation_progress; clear per item on terminal status; clear all on connect/disconnect/queue cleared. - ProgressBar.tsx: render a vertical stack of bars (one per active session) with a single-bar fallback for the idle / model-loading window; add containerProps so dockview tabs can position the stack. - Dockview tab call sites: move positioning into containerProps. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(ui): make $progressEvents module-local to satisfy knip $progressEvents is only referenced within stores.ts (via the $activeProgressEvents computed and the set/clear helpers), so exporting it tripped knip's unused-exports check. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(ui): cap stacked tab progress bars to fit below the tab label With 4 GPUs the stacked per-session progress bars grew past the bottom strip of the dockview tab and overlapped the "Viewer" label. Add a fitHeightPx prop: in fit mode the stack is capped to the available strip (10px below the ~40px tab's centered label) and the bars flex to share it, shrinking below their natural height only once they no longer fit. With 1-2 sessions the bars keep their familiar thin height; with 3+ they scale down to stay within the strip. The sidebar bar is unaffected and continues to stack at natural height (it has the vertical room). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(config): support "auto" generation_devices to use all GPUs by default generation_devices now accepts "auto" (the new default), which expands to every visible CUDA device — so multi-GPU parallel generation works out of the box without manually listing devices. On GPU-less systems "auto" resolves to the single cpu/mps device, preserving serial behavior. - config_default.py: type is now Union[Literal["auto"], list[str]], default "auto"; validator accepts "auto" or a list of device strings. - devices.py: add TorchDevice.get_generation_devices(), the single resolver that expands "auto", normalizes, and deduplicates. - session_processor / model_manager: both consumers use the resolver instead of iterating the raw config value (which would have iterated the characters of the "auto" string). - Regenerated docs/src/generated/settings.json. - Tests for the resolver (auto-with/without-CUDA, dedup, empty). An explicit single-device list (e.g. [cuda:0]) or an empty list opts out of parallelism. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(frontend): typegen+openapi * docs(multi-gpu): add configuration information * chore(frontend): typegen + openapi again * feat(settings): add Generation Devices selector to Settings dialog Add a badges UI in the Generation section of the Settings dialog for choosing which devices `generation_devices` should use, modeled on the Log Namespaces toggle UI. Backend: - New `GET /api/v1/app/generation_device_options` endpoint listing the selectable devices (cuda:N with GPU names, or the sole mps/cpu fallback). - Add `generation_devices` to the runtime-config update allowlist with validation rejecting invalid device strings and explicit nulls. Frontend: - New SettingsGenerationDevices component with active/inactive badges. "Auto (all GPUs)" is exclusive; removing the last explicit device reverts to auto. Admin/multiuser gated; notes restart requirement. - Wire into the Generation section; regenerate schema; add en strings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(settings): boldface the restart notice on Generation Devices Split the restart sentence into its own string and render it bold so users notice that device changes require restarting InvokeAI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(settings): show GPU name in Generation Devices badges Render device badges as "cuda:0 (RTX 3090 #1)" so identical cards can be told apart. Strips the "NVIDIA GeForce" vendor prefix and adds a 1-based "#N" suffix only when multiple cards share a name. The full device name remains available as the badge tooltip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(frontend): openapi * feat(multi-gpu): surface per-session GPU number in logs and UI Help users track which CUDA device is processing each session: - Model-load log: "Loaded model ... onto cuda device #N in ..s" - Denoise progress bars: "Denoising (#N)" across all architectures (SD1.5/SDXL, FLUX, FLUX2, Z-Image, Anima, SD3, CogView4) - Progress preview circle: GPU number centered in the ring, via a new `device` field on InvocationProgressEvent (resolved from the worker's thread-local session device) - Session Queue: new "GPU #" column between STATUS and TIME, backed by a `device` column on session_queue (migration_32) recorded when a worker claims an item Adds TorchDevice.get_session_device_label()/get_session_device_index() helpers and a frontend getCudaDeviceIndex() parser (with tests). Shows the number on CUDA only; CPU/MPS show nothing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(multi-gpu): show per-device names in startup log and progress circles - Startup log lists each generation device with its GPU number and id, e.g. "Using torch device: [AMD Radeon PRO W7900 #1 (cuda:0), ...]". Single-device setups keep the bare device name. - Canvas progress circles now show the CUDA device index in the center, matching the viewer panel. - Progress-circle tooltips show the device name and number on hover. - Both are hidden when only a single GPU is available. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(model-cache): share one CPU copy of model weights across per-GPU caches In multi-GPU mode the model manager builds one ModelCache per generation device, each with storage_device="cpu" and its own RAM-resident copy of every model. A model loaded on N GPUs therefore occupied N copies in RAM, and each cache sized itself against max_cache_ram_gb independently, so RAM use during the text/reference-image encoding phases skyrocketed and the system swapped — worst when two images rendered at once. This deduplicates the CPU-resident weights and makes RAM accounting global. - SharedCpuWeightsStore: process-/manager-global, refcounted store of one canonical CPU state_dict per model key. The first device to load a key registers its weights; subsequent devices adopt the canonical tensors and re-point their module's params at them (load_state_dict(assign=True)), freeing the duplicate. Weights live once in RAM regardless of GPU count; freed only when the last device releases. Per-device modules are kept (params are device-shuffled in place, so two GPUs need two modules), but their CPU-resident params alias the shared tensors. - RamBudget: single system-wide RAM authority. Splits RAM into shared (counted once via the store) and non-shared (per-instance). ModelCache eviction now runs against the global, deduplicated total and re-checks availability each iteration, since evicting a model another device still holds frees no RAM. build_model_manager wires one store + one budget into all device caches; the cap is max_cache_ram_gb as a true system-wide limit, else the sum of per-cache heuristics. Passing ram_budget=None preserves the prior local accounting. - LoRA/patch safety: direct LoRA patching did an in-place copy_ on the weight, which would corrupt the now-shared canonical tensor (and taint keep_ram_copy even with one GPU) when patching a CPU-resident weight. Switched to an out-of-place add (memory- equivalent) so the canonical tensor is never mutated; fixed the FluxControlLoRA expansion path to target the module's live parameter. Sidecar patching and FreeU/Seamless (which patch forward methods) were already safe. Validated on 2x AMD W7900 / ROCm: correct inference on both GPUs from one shared copy (full + partial load + Q8_0 GGUF quantized), concurrent load/unload without corruption, and LoRA isolation across devices. ~40 new tests; existing suites unchanged. Adds scripts/multigpu_ram_driver.py to drive concurrent dual-GPU generations via the queue API and measure peak RSS / leak drift. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(session-queue): cancel all in-progress items in bulk-cancel APIs (multi-GPU) With one session-processor worker per device, multiple queue items can be in_progress at once. cancel_by_batch_ids(), cancel_by_destination() and cancel_by_queue_id() excluded in_progress rows from their bulk UPDATE and then canceled only the single get_current() item (LIMIT 1), so on multi-GPU the other running items kept consuming a GPU and could still produce output after the user requested cancellation. Each running item must be canceled via _set_queue_item_status(), which emits the QueueItemStatusChangedEvent that the processor maps to the worker running that item_id and uses to set its cancel event. Add _cancel_in_progress_matching() to cancel every in-progress item matching the same filter (with user-id scoping preserved) and call it from all three bulk-cancel methods. The returned `canceled` count now includes canceled in-progress items. Adds regression tests that dequeue two items onto separate devices and assert every bulk cancel API moves all matching in_progress items to canceled and emits a cancel event for each (and that user-scoped cancel leaves another user's in-progress item running). Reported by JPPhoto in review of #9263. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(multi-gpu): address review findings (cancel race, bulk delete, device guards, refcount leak) Fixes from the code review of PR #9263: - Cancellation could be silently lost around dequeue: the per-iteration worker.cancel_event.clear() ran AFTER dequeue + gc.collect() + logging, so a cancel arriving in that window was set by the status handler and then wiped. Move the clear to before dequeue, and after claiming an item re-check (cancel_event + a fresh DB status read via _is_queue_item_terminal) and skip running if it is already terminal, closing both race windows. The runner's stale queue_item.status check could not catch this. - delete_by_destination only stopped one in-progress item (get_current) before deleting all matching rows, leaving other GPU workers running (and then failing to update a deleted row). Cancel every matching in-progress item via _cancel_in_progress_matching first. - generation_devices validation: a bare non-"auto" string (e.g. "cuda:0") was iterated character-by-character; an empty list silently fell back to one device. Reject both with a clear message. - get_generation_devices now fails fast on a CUDA device that does not exist (index past device_count, or CUDA unavailable) instead of starting a worker that errors cryptically at first allocation. - Shared-weights wrappers: if the canonical re-point (load_state_dict assign=True) threw after acquire(), the reference was leaked (the wrapper never entered the cache). Compute size metadata first, make acquire the last step, and release on failure. Adds tests for each: post-dequeue terminal guard, delete_by_destination cancellation, generation_devices validation, absent-device rejection, and acquire-released-on-repoint-failure. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): ruff format + make CPU-incompatible device test mock CUDA - Apply ruff 0.11.2 formatting to the files flagged by `ruff format --check`. - The new fail-fast guard in get_generation_devices() (reject a CUDA device that doesn't exist) made the pre-existing test_get_generation_devices_explicit_list_is_deduplicated fail on CPU-only CI runners, since it passes a cuda list with no CUDA present. Mock torch.cuda.is_available/device_count in that test (matching the existing pattern in this file) so it validates dedup on any runner. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(multi-gpu): stop RAM blowup/swapping during concurrent generations Three RAM fixes for multi-GPU (and one that helps single-GPU too), addressing transient spikes to ~100% RAM and swapping during text-encode/transformer loads: 1. Cap the global RAM-cache budget at a safe fraction of system RAM. When max_cache_ram_gb is unset, the budget was the *sum* of the per-device cache heuristics, so N GPUs each claiming ~50% of RAM summed to ~N*50% and starved the OS. Now clamp the sum to ModelCache.calc_system_ram_headroom_bytes() (50% of RAM - 2GB baseline, floored at 4GB). Promote the sizing magic numbers to named constants shared by the per-device heuristic and the global cap. 2. Adopt already-resident CPU weights across devices at load time. When a second device loads a model another device already holds, deep-copy a registered meta-weight structural clone and assign the shared canonical weights, instead of re-reading the model from disk and materializing a full transient second copy. Loader-agnostic (one mechanism in ModelLoader, no per-loader code): works for diffusers, single-file checkpoint, GGUF and transformers models, and preserves registered hooks (e.g. fp8 layerwise-cast). Best-effort with a meta-tensor self-check and fallback to a normal disk load on any failure. Skipped on single-device installs. 3. Dequantize FLUX.2 FP8 checkpoints straight to bf16. _dequantize_fp8_weights materialized the whole model in float32 (~36GB for 9B) before a later cast to bf16; now the multiply is done in float32 but stored bf16 per-weight, so the model is never held in float32. Numerically identical; halves the cold-load transient (helps single-GPU too). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(qwen-image): reserve VAE working memory so decode/encode don't OOM The Qwen Image VAE encode/decode invocations called model_on_device() without a working-memory estimate, unlike every other VAE family (SD/SDXL/SD3/CogView4/FLUX). So the model cache reserved only its small default working memory, never offloaded a large resident transformer (the VAE weights themselves are tiny), and the VAE's forward-pass activations then OOM'd VRAM — e.g. a ~40GB Qwen Image Edit transformer left ~1GB free while decode needed ~5GB. Reproduces single-GPU; unrelated to the multi-GPU RAM work. Add estimate_vae_working_memory_qwen_image() (same per-output-pixel scaling as the other estimators, handling the 5D Qwen latents) and pass it from both the i2l (encode, used for reference images in Image Edit) and l2i (decode) nodes, so the cache offloads the transformer before the VAE runs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(flux2): tile reference-image VAE encode to avoid VRAM OOM The FLUX.2 VAE encoder's mid-block self-attention scales quadratically with the input's spatial size, and on ROCm scaled_dot_product_attention falls back to a materialized attention matrix. Encoding a reference image (kontext) at full size therefore allocated ~15GB in a single attention call at 1024px — and hundreds of GB at the 2024px reference cap — OOMing VRAM regardless of how much other model memory was freed. Tile the reference-image encode to bound per-tile attention. The VAE's default tile size equals its sample_size (1024), whose per-tile attention still OOMs, so force a 512px tile (with a matching latent tile size derived from the config). Save/restore the VAE's tiling config since it is a shared, cached instance, so the final image decode does not inherit these settings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(multi-gpu): query execution device for VRAM-in-use accounting ModelCache._get_vram_in_use() called torch.cuda.memory_allocated() with no device argument, while _get_vram_available() reads memory_allocated(execution_device). The formula relies on those two canceling. In multi-GPU mode each worker calls torch.cuda.set_device for its own GPU, so the process-current device flips between workers; the no-argument call can then read a different (e.g. idle) GPU's allocation, breaking the cancellation and inflating "available" VRAM toward the card total. The cache then believes there is room and never offloads, so VRAM offloading effectively ignores device_working_mem_gb in multi-GPU. Single-GPU was unaffected (current device always equals the execution device). Query self._execution_device in both _get_vram_in_use() and the cache-state debug log. Add a regression test asserting the per-cache execution device is used. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(qwen-image): calibrate VAE working-memory estimate to the 3D-conv decode peak The Qwen Image VAE is a 3D-conv (video) VAE whose decode allocates large conv3d feature maps. A ~1MP decode was measured to peak at ~17 GiB of VRAM — far above what the generic 2200/1100 SD/FLUX constants reserved (~4.6 GiB), so the cache concluded the decode "fit" alongside the resident 20GB transformer + 15GB text encoder, never offloaded them, and OOMed. The offload only frees ~(working_mem - free) bytes, so the reservation must both cover the real peak and be large enough to trigger the offload of models the decode doesn't need. Raise the Qwen decode/encode constants (13000/6500) to match the measured peak. It's linear in output pixels, so it over-reserves past ~1.5MP (where the decode can exceed the card even after offloading) — that case is covered by force_tiled_decode. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(qwen-image): honor force_tiled_decode in the l2i node The Qwen Image latents-to-image node hardcoded vae.disable_tiling(), ignoring the global force_tiled_decode setting that the SD/SDXL l2i node honors. Wire it up the same way so users can opt into tiled VAE decode for very large outputs that exceed VRAM even after the transformer/text encoder are offloaded. Off by default, so normal-size decodes are unchanged (full-frame, no tile blending). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(ui): stop progress disk flashing during indeterminate phases The preview-panel progress circle re-renders on every InvocationProgressEvent. The parent passes a fresh progressEvent object each event, so the CircularProgress re-rendered constantly; during the indeterminate phases (everything except denoising) that restarted its CSS spin animation each time, which looked like the disk flashing. (Determinate denoising was unaffected because the value genuinely changes per step.) Split the circle into a memoized, ref-forwarding subcomponent keyed on its visual props (isIndeterminate, value, device label) so message-only updates no longer re-render it and the spin animation stays continuous. The Tooltip still anchors to it via the forwarded ref. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(multi-gpu): offload text encoders to idle GPUs Adds `offload_text_encoders_to_idle_gpus` (default on): when more than one generation device is configured and a GPU is idle, a session's text/prompt encoder runs on the idle GPU instead of the one running its denoise pipeline. This avoids evicting the denoise model from VRAM to make room for the encoder, and lets a cached encoder be reused across generations. Under full load (no idle GPU) behavior is unchanged. Mechanism: - New GENERATION_DEVICE_POOL arbiter (backend/util/device_pool.py) with a per-device exclusive-use lock. A native session blocking-acquires its own device's lock for the whole run; an encoder node try-borrows an idle device's lock for the duration of the node. This makes a borrowed encoder and a native session mutually exclusive on a GPU -- preventing the shared-encoder corruption that produced garbled images -- and is deadlock-free (borrows are non-blocking; a session only ever blocks on its own device). - DefaultSessionRunner re-pins the worker thread to the borrowed device for the whole encoder node; conditioning is stored on the CPU and the denoiser picks it up on its own GPU afterward. - Nodes opt in via @invocation(idle_gpu_offloadable=True), mirroring the existing `bottleneck` ClassVar marker. Applied to the text/prompt encoder nodes (compel + sdxl/refiner, flux, sd3, qwen-image, anima, cogview4, flux2 klein, z-image, flux_redux). Inspired by #9310; supersedes it. Tests: device-pool lock semantics, two concurrency regression tests asserting a session and a borrow never use a GPU at the same time, the runner offload context-manager behavior, and a marker-wiring check. Docs: invokeai-yaml.mdx (config setting) and creating-nodes.mdx (how to support the feature in a node). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(multi-gpu): adopt GGUF weights across devices to stop RAM spikes _build_meta_shell built meta placeholders with torch.empty_like, which GGMLTensor.__torch_dispatch__ rejects (NotImplemented for aten.empty_like). It threw on the first parameter, hit the silent except, and returned None — so GGUF models (e.g. a Q8_0 transformer) never registered a shell and the second GPU re-loaded the full model from disk, stacking a ~20GB transient on the retained copy and spiking RAM to ~70%. Fall back to a plain meta placeholder (logical shape/dtype) when empty_like isn't implemented by a tensor subclass; verified the adopted GGMLTensor shares the quantized storage, so it's one RAM copy across devices. Peak drops ~66→~46GB. Log shell-build failures at debug so a future un-adoptable family is diagnosable instead of silently double-loading. Also restore log_memory_usage's per-cold-load RAM logging (the capture method had no callers), slimmed to baseline→transient-peak process RAM. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(multi-gpu): tie device #N label to cuda index, not filtered position The backend device summary computed the disambiguating #N suffix by enumerating the filtered generation_devices list, so disabling a device (e.g. cuda:1) renumbered the survivors. The frontend labels over the full device set, so the two disagreed. Compute the suffix over all available devices instead, keeping the label stable and consistent with the frontend. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(multi-gpu): flash restart reminder when generation devices change Reword the Generation Devices caption to "Restart InvokeAI for changes to take effect." and flash that same warning as a toast on every successful change, so the restart requirement is hard to miss. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(queue): device-affinity dequeue to reduce model reload thrash on multi-GPU When a GPU worker dequeues, prefer — among the fairness-chosen user's equal-priority pending items — one whose models are already resident in that device's cache. Cross-device model reloads cost tens of seconds for large models; picking a warm item instead cuts thrash when a user queues a mix of models. Guardrails (from adversarial review): - Round-robin user choice and priority tiers are never overridden; the swap pool is limited to the candidate's user and priority. - The swap window is capped at AFFINITY_MAX_LOOKAHEAD past the candidate's item_id, bounding both cold-item deferral and per-dequeue scan cost. - Explicitly configured session_queue_mode=FIFO opts out of reordering. - Resident keys are snapshotted before the dequeue lock, and ModelCache.cached_model_keys() acquires its lock non-blockingly, so a long-running VRAM transfer can never stall other workers' dequeues. - Path-keyed cache entries (load_model_from_path) are excluded so a Windows drive letter can't poison substring scoring. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(qwen): restore legacy key remapping for single-file VL encoders under transformers 5.x The single-file Qwen2.5-VL encoder loader relied on Qwen2_5_VLForConditionalGeneration._checkpoint_conversion_mapping to translate ComfyUI's legacy key layout (visual.*, model.layers.*) to the modern one (model.visual.*, model.language_model.*). transformers 5.x ships that mapping empty — the conversion moved into from_pretrained's weight-converter machinery, which our manual load_state_dict path bypasses — so the vision tower was left on the meta device and loading failed with "Meta tensors remain". Fall back to the equivalent hardcoded mapping when the class attribute is empty or absent. Verified against qwen_2.5_vl_7b_fp8_scaled.safetensors: loads all 8.29B params with no meta tensors remaining. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: address multi-GPU review findings from PR #9263 review - Shared CPU weights: drop_model() now invalidates the model's canonical entries in SharedCpuWeightsStore, so a rebuild on another device can never adopt pre-settings-change weights still aliased by a locked (stale-marked) entry. release() is identity-checked so a stale holder's eviction cannot decrement a newly registered canonical. update_model_record holds MODEL_LOAD_LOCK.write_lock() (off the event loop) across the multi-cache drop to exclude in-flight loads. - Runtime config API: generation_devices is now fully validated at the route boundary — empty lists and unavailable devices (e.g. cuda:99) return 422 without mutating or persisting config, using the same TorchDevice resolution as startup. - Cache stats: /v2/models/stats aggregates per-device caches instead of reporting only the API thread's default cache. - Config/docs contract: session_queue_mode description now documents device-affinity reordering in single-user multi-GPU mode (and that explicit FIFO disables it), and that user rotation outranks priority across users in round_robin mode. Multi-GPU docs no longer claim generation_devices: [] is valid, and describe shared-RAM weight deduplication instead of per-GPU duplication. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: address multi-GPU review findings (RAM accounting, stats aggregation, MPS validation) - SharedCpuWeightsStore.invalidate() now retires still-referenced entries instead of dropping them from accounting, so RamBudget keeps counting retired weights until the last locked holder releases them. Prevents admitting models past max_cache_ram_gb while a replacement and a stale copy are both resident. - /models/stats aggregation takes max of cache_size and high_watermark across per-device caches (they share one global RamBudget, so summing over-reported an N-GPU system ~N times); event counters are still summed. - TorchDevice.get_generation_devices() rejects 'mps' when MPS is unavailable, so the runtime_config API 422s instead of persisting a device that fails at first tensor op. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(frontend): lint:prettier * fix: address JPPhoto's 2026-07-21 review (12 items) Backend: - layer_patcher: hold MODEL_LOAD_LOCK.read_lock() across patch application so FLUX Control LoRA shape expansion (register_parameter) cannot overlap a concurrent model construction's process-global init_empty_weights patch - flux_redux/flux_denoise: store Redux conditioning on CPU (it may be produced on a borrowed idle GPU) and assign the .to() result when consuming it - model_cache/ram_budget: coordinate eviction across device caches — when a cache's own stack is exhausted and the global budget is still short, peers evict their unlocked entries (non-blocking lock, deadlock-free), so max_cache_ram_gb holds even when RAM is retained only by an idle device - session_queue: 'except current' operations protect the workflow-call chain of EVERY in-progress item, not one arbitrary get_current() row - session_queue: _cancel_in_progress_matching tolerates rows deleted by a concurrent clear between its id SELECT and the per-item cancel - session_processor: the post-dequeue cancel guard cancels the freshly claimed item when skipping it (a stale cancel_event must not abandon it in_progress) - session_processor: _clone_session_runner refuses to downgrade DefaultSessionRunner subclasses or share custom runners across workers - session_processor: an offloaded encoder's cache activity is attributed to the running session's CacheStats (borrowed cache's stale stats pointer swapped for the borrow duration) - events: progress events report the queue item's persisted device, not the thread-local (temporarily borrowed) one - devices/config docs: generation_devices 'auto' defers to an explicitly pinned legacy 'device:' setting so upgrades don't start workers on every GPU Frontend: - ImageViewer context: a terminal status only clears the shared progress event/image globals when that item owns them (multi-GPU: canceling item A no longer blanks item B's live preview) - SettingsGenerationDevices: device tags are keyboard-operable (tabIndex + Enter/Space activation) Each fix has an exposure test per the review's suggestions. * chore: regenerate openapi.json (auth on get_generation_device_options) * fix(backend): avoid MODEL_LOAD_LOCK self-deadlock when patching a LoRA on a cold cache apply_smart_model_patches() held MODEL_LOAD_LOCK.read_lock() across its patch loop, but callers pass a lazy generator (e.g. flux_text_encoder._t5_lora_iterator) that constructs each LoRA via context.models.load() on demand. A cold-cache load takes MODEL_LOAD_LOCK.write_lock(); since the lock is non-reentrant and write-preferring, acquiring the write lock while this same thread already holds the read lock deadlocks (write waits for readers==0, but the consuming thread is that reader). The generation hung silently right after the encoder/tokenizer load, whenever a LoRA was applied and not already cached. Materialize the patch iterable before taking the read lock so every LoRA construction takes (and releases) the write lock first; the read lock then covers patch application only, which is its actual purpose (FLUX Control LoRA shape expansion calls register_parameter and must exclude concurrent construction). Compatible with wan_denoise's per-call iterator factory, and unrelated to the SD UNet path, which loads the LoRA before calling the singular patcher (no lock held). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: address JPPhoto's four merge blockers from the 2026-07-22 review 1. model_cache: a peer whose lock is contended during cross-cache eviction no longer leaves the shared RAM budget exceeded indefinitely. evict_unlocked_for_peer returns None on contention; the requester records a reconcile request on each skipped peer, and the synchronized-decorator hook honors it as soon as the peer's current operation releases the lock (outermost frame only — the RLock may be held reentrantly). The pending flag stays set until the budget is actually satisfied, so overshoot held by locked entries reconciles when their unlock releases the lock. 2. session_processor: a stale cancellation event from the previous item no longer cancels the freshly claimed, unrelated item. The post-dequeue guard now treats the DB status as the authority: a terminal row is skipped; a set cancel_event with a non-terminal row is a stale signal (a genuine cancel writes the row terminal BEFORE emitting) and is cleared, with a post-clear terminal re-check closing the clear's own race window. A shutdown-raced claim is still canceled so it isn't abandoned in_progress. 3. flux2_klein_text_encoder: conditioning is detached and moved to CPU before context.conditioning.save(), matching flux_text_encoder and flux_redux — the node is idle_gpu_offloadable, and GPU-resident embeddings would pin VRAM on a borrowed device after its pool lock is released. 4. session_queue clear: user-scoped clearing no longer assumes one current item. clear() cancels every in-progress item in scope via _cancel_in_progress_matching (each item's own status-changed event signals the worker running exactly that item) before deleting rows — same pattern as delete_by_destination; the router's arbitrary get_current() check (which could 403 the owner or cancel another user's item) is removed; and _on_queue_cleared honors the event's user_id so a scoped clear cannot stop other users' workers and abandon their rows. Each fix carries the regression test JPPhoto specified: contended-peer budget reconcile, stale-event-runs-item (plus the mid-clear race and shutdown cases), CPU-backed Klein conditioning, and Alice/Bob concurrent clear isolation at both the service and the event-handler layer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: regenerate OpenAPI schema for the clear endpoint docstring The merge-blocker fix 68edb02127 reworded the clear route's docstring, which is the OpenAPI operation description — openapi.json and schema.ts must follow. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(model cache): close lost-wakeup race in deferred RAM-budget reconcile The deferred reconcile request was recorded pre-admission and honored only by the peer's next lock release. Two interleavings could strand the shared RAM budget above its cap indefinitely: - Lost wakeup: the busy peer releases its lock (running its reconcile hook while the flag is still unset) before request_budget_reconcile() sets the flag; if the peer then stays idle, no future release honors the request. - Pre-admission clearing: a peer's reconcile could run between the request and the new model being counted, see the budget as satisfied, and clear the flag before the admission pushed usage over the cap. Fix both by (1) moving the reconcile request to the end of put(), after the new model is counted, so peers always evaluate the true budget state, and (2) having request_budget_reconcile() attempt the reconcile inline with a non-blocking lock acquire: either the peer's lock is free now and the reconcile runs immediately, or it is still held and the eventual release hook — which runs strictly after the flag is set — performs it. The prior regression test masked the race by touching cache_b.stats after the request; it now emulates the production release hook in the holder thread and asserts reconciliation with no subsequent cache access, and a new test forces the lost-wakeup interleaving by delaying the request until the peer's operation has fully finished. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(model cache): close remaining RAM-budget reconcile gaps Addresses the three lingering issues from review of the deferred budget-reconcile mechanism: 1. Manual lock releases bypass the reconcile hook. cached_model_keys() and evict_unlocked_for_peer() acquire/release _lock without the synchronized decorator, so a reconcile request whose inline attempt failed on their held lock was stranded when they released. Both now run the same reconcile hook after their manual release (non-blocking, preserving cached_model_keys' no-stall guarantee and avoiding the hold-A-block-on-B deadlock shape in evict_unlocked_for_peer). 2. clear() can wipe a concurrent request. A reconciler observing a satisfied budget could clear the pending flag just after a peer's admission (already counted, budget negative) set it, and the peer's inline attempt then saw the flag unset and returned — leaving the budget exceeded with no pending request. The reconcile now runs as a loop with a single guarded clear site: because admissions are counted before the flag is set, a negative budget re-check immediately after the clear proves a request may have been wiped; the flag is restored and reconciliation continues. This covers both former clear sites (satisfied early-out and post-eviction). 3. No reconcile trigger when the admitting cache itself holds the overshoot. put() requests reconciles from peers only, so when the exceeded budget was held by the admitting cache's own locked entry, no pending request existed anywhere and the eventual unlock ran its hook with the flag unset. unlock() now records a reconcile request on its own cache whenever it completes with the shared budget exceeded, so the entry that just became evictable triggers the reconcile. Supporting change: put() admitting a model while a peer's reconcile request is already pending must not let its own release hook evict the just-admitted entry before the loader's immediately-following get() (that would break the in-flight load with an IndexError). CacheRecord gains an awaiting_first_use grace flag, set on admission and cleared on first get()/lock(), which the asynchronous eviction paths (budget reconcile, peer-requested eviction) skip. The local make_room path ignores it: cold loads are serialized under MODEL_LOAD_LOCK, so it can never see another loader's entry inside the put()->get() window, and this bounds the flag's lifetime if a load errors out in between. Each new regression test was verified to fail against the previous implementation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(model cache): bound the admission grace and keep cached_model_keys stall-free Addresses the three issues from JPPhoto's 2026-07-27 review: 1. Prefetched submodels can no longer shield the budget forever. The SD single-file loader's proactive submodel put()s are now admitted with prefetch=True (no post-admission grace), since nothing ever get()s or lock()s them. As a backstop, put() sweeps stale grace flags from prior loads — cold loads are serialized under MODEL_LOAD_LOCK, so any flag still standing at the next admission belongs to a dead load (errored before get(), or LoadedModel dropped before lock()) and is cleared. 2. The grace now survives get() and ends at lock(). get() is synchronized, so clearing the flag inside it let get()'s own release hook run a pending reconcile and evict the very record it had just selected — detaching a live model from the cache and its RAM accounting before the caller could lock it. load_default also retrieves immediately after put() so no failure in between can orphan a graced record. 3. cached_model_keys()'s manual-release hook hands a pending reconcile to a short-lived background thread instead of running it inline: reconciliation evicts models and calls gc.collect(), which would break the method's no-stall contract and pause session dequeue. Each new regression test verified to fail with its mechanism reverted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(model cache): release abandoned admission grace * fix(model cache): keep grace release off the collecting thread release_first_use_grace() is invoked from a weakref.finalize callback, so it runs at an arbitrary decref/garbage-collection point in an arbitrary thread. Making it @synchronized therefore made ModelCache._lock — and, through the decorator's release hook, a full budget reconcile — reachable from anywhere. That inverts the lock order RamBudget documents as impossible. The hook's _reconcile_budget_if_pending reads RamBudget.available() -> SharedCpuWeightsStore.total_bytes_in_use(), both plain non-reentrant locks. A thread inside SharedCpuWeightsStore.acquire() holds the store lock while summing tensor sizes, an allocation loop that trips generational GC; if that collection reclaims an abandoned wrapper belonging to another device's cache, the release hook re-enters the store lock the thread is already holding and the thread deadlocks against itself, still holding it. Every other cache then blocks on its next _delete_cache_entry -> release_shared_weights(). Reproduced on a two-cache budget: the collecting thread wedges in total_bytes_in_use() and never returns. The same hook also ran evictions, gc.collect() and empty_cache() inline in whatever unrelated thread happened to drop the reference — including the API event loop — undoing the no-stall contract cached_model_keys() was just given. Do no locking work in the callback: hand the release to a short-lived background thread, exactly as cached_model_keys() does with its own pending reconcile. The thread may wait on the cache lock and do the slow work; the collecting thread returns immediately. The existing abandoned-wrapper test now polls for the (asynchronous) release. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(model cache): harden the deferred grace release Follow-ups from an adversarial review of 55a6fc496d: - Thread.start() can raise RuntimeError under thread/process limits. A weakref.finalize callback gets no retry (weakref retires it before invoking it) and its exceptions go to sys.unraisablehook, so the release was silently lost and the record kept shielding an idle cache. Fall back to clearing the flag inline under a non-blocking acquire, which takes no store or budget lock and so still cannot deadlock the collecting thread. No reconcile on that path by design: a pending request stays set for the next cache operation. - The regression test's outcome was a pure function of the ambient allocation count: nothing pinned the cycle between its creation and the collector thread, so an automatic gen-0 pass landing in the setup reclaimed it on the main thread and the test passed vacuously (or tripped its own setup assertions). Under an allocation-shifting plugin it failed at 6 of 12 offsets. Disable automatic gc across the setup so only the explicit collect reclaims the cycle; the same sweep is now 12 of 12 passing, and the test still fails against 7ffee4db04 with the expected re-entrancy report. - Correct the docstring: Thread.start() waits for the child to bootstrap, so the guarantee is "no lock waits", not "returns immediately". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(model cache): queue deferred cache work * fix(model cache): stop the deferred worker pinning records and dying silently Follow-ups from an adversarial review of c384ea187d, which replaced the per-release background threads with one long-lived worker per cache. - The worker's `work` local stays bound while it blocks in the next get(), so the last-processed CacheRecord — and transitively its model's CPU weights — was pinned until some unrelated item happened to be queued behind it. That is worse than an ordinary leak: _release_first_use_grace's release hook can evict that very record, removing it from the cache AND subtracting its bytes from the RamBudget, so the budget under-reported a model that was still resident and the next admission over-committed. Reproduced on a two-cache budget: after eviction plus an explicit gc.collect(), both the record and its module were still alive; queueing one more item freed them. The per-call threads this replaced did not have the bug — Thread._bootstrap_inner deletes _args on exit. Clear the reference in a finally before looping back. - `if self._deferred_work_thread.ident is None` is a "was it ever started" check, not a liveness check: ident is never cleared and a Thread cannot be restarted. A worker lost to an unexpected error (a logging handler that raises, os.fork(), or shutdown() before the first put(), which leaves its _DEFERRED_STOP queued for the thread that put() then starts) was gone for the life of the process, silently disabling every later grace release and budget reconcile — the failure this mechanism exists to prevent. Create a fresh thread whenever the previous one has exited, and never after shutdown(). - Only the worker drains the queue, but both dispatch sites enqueued unconditionally. cached_model_keys() runs on every dequeue (session_queue_sqlite._get_device_resident_model_keys), so an idle-device cache that never admitted a model — and so has no worker — accumulated one queued reconcile per dequeue forever; post-shutdown the same held for both sites, stranding CacheRecords in a queue nothing would drain. Route both through _dispatch_deferred, which drops the item when no worker is running. Dropping loses nothing: such a cache has nothing to evict, and put() re-runs a pending reconcile through the synchronized release hook when it admits one. - shutdown()'s early return meant a keep-alive timer re-armed by a post-shutdown put() was never cancelled by a later shutdown(). Don't arm timers on a shut-down cache. Tests: the two *_thread_start_failure_* tests installed their monkeypatch after put() had already started the worker, and c384ea187d removed the only Thread.start() from those paths — the patched raise was unreachable, so both passed without exercising their premise. Retargeted to what they actually verify (the finalizer and the lookup must not block, and the reconcile still happens). Four regression tests added; each fails against c384ea187d. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(model cache): stop the deferred worker outliving and pinning its cache Second round of adversarial-review follow-ups on the deferred-work thread. - The worker held a bound method, so a running thread — reachable from threading._active — kept its ModelCache alive, and with it every CacheRecord and every model's CPU weights. A cache released without shutdown() was therefore immortal, which is the opposite of what RamBudget's weakref registry is built for. Measured against the parent commit 7ffee4db04: five caches dropped without shutdown() left 5/5 caches and 5/5 models resident and five worker threads running, where the parent left 0/5 and no threads. The previous `ident is None` guard had accidentally bounded this (a cache whose worker died could never re-acquire a pinning thread); reviving the worker removed that bound, so the fix has to remove the strong reference itself. The worker is now a module-level function taking a weakref, and a weakref.finalize pushes _DEFERRED_STOP when the cache is collected so the parked thread exits instead of leaking one thread per abandoned cache. - Thread.start() is called from put(), which runs under both the cache lock and MODEL_LOAD_LOCK's write lock while completing a load. Under thread/pid exhaustion (RLIMIT_NPROC, a container's pids.max) its RuntimeError escaped and failed a generation whose model had already been fully constructed — to lose an optimization that _dispatch_deferred is explicitly designed to survive the absence of. Log and carry on; the next admission retries. - _dispatch_deferred justified dropping work with "a cache without a worker has never admitted a model". That was false: put() after shutdown() is reachable in production, because Invoker.stop() stops model_manager before session_processor, so an in-flight generation can admit a model after every cache has been shut down — and thread exhaustion reaches the same state without a teardown to bound it. Such a record was admitted with the first-use grace, then permanently shielded from both asynchronous eviction paths with its bytes still charged to the shared budget. Make the claim true instead of rewording it: put() grants the grace only when a worker is running to release it. lock() still clears it on the normal path, so nothing changes when the worker is healthy. Tests: the post-shutdown half of the drop test never set the pending flag, so cached_model_keys() short-circuited before reaching the dispatch guard and the assertion held unconditionally. Poll the budget in the pin test rather than reading it the instant the key disappears (_delete_cache_entry pops before it releases the weights). Two regression tests added; all seven of this series' new tests fail against c384ea187d. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(model cache): release abandoned RAM budget * fix(model cache): pin LoRA patches during use * fix(model cache): harden the LoRA pin path and close its review gaps Review fixes for the LoRA-pinning commit (1142430f87): - Fix the CI failure it shipped: test_krea2_text_encoder's fake LoadedModel lacked model_in_ram(), and the encoder's LoRA iterator now calls it. The fake now models the pin (with depth tracking), and the test asserts the patch spec carries a working pin. - Stop a keep-alive Timer.start() failure from leaking a permanent pin. @record_activity runs after lock_in_ram() has incremented the lock count but before model_in_ram()'s unlock-pairing try block is entered, so a RuntimeError under thread/pid exhaustion would pin the record (and its shared-budget bytes) for the life of the process. The timer is an optimization: log and continue instead. - Give lock_in_ram() the same already-dropped-record diagnostic as lock()/unlock(), so a pin on a detached record produces a matching lock-side message (issue 7513). - Pin the LoRA cache record in LoRAExt.patch_unet (the modular-denoise path) while its tensors are read during direct patching. This was the one remaining producer that dropped its LoadedModel handle at load time, leaving the record evictable by a peer cache mid-patch. - Close test vacuities: the pin-retention test in test_layer_patcher could not detect a dropped cache_pins.close() (the ExitStack would be collected silently), and the cache-side pin test only covered a warm record, leaving lock_in_ram's grace-clearing dead code under test. Added pin-release assertions for the normal, body-raise, restore-raise, and mid-materialization-raise paths (all verified to fail with close() neutered), a cold-record grace/finalizer test, and a Timer-failure regression test (verified to fail pre-fix). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(model cache): close the remaining raise-after-lock leak and widen the LoRAExt pin Self-review fixes for d63c1f37e0: - The synchronized-decorator's post-release reconcile hook runs inside the caller's frame after the method body, so a raise there (e.g. TorchDevice.empty_cache on a sick CUDA context after an eviction) escaped lock_in_ram()/lock() after the lock count was incremented — the same permanent-pin leak as the Timer.start() case, via a different path. The reconcile is deferrable housekeeping: swallow and log; the pending flag is only cleared once the budget is satisfied, so the next lock release retries. Regression test verified to fail pre-fix. - LoRAExt.patch_unet's pin now spans the yielded scope, not just the patch application: despite force_direct_patching=True, fp8-storage modules are routed to sidecar patching (float8 weights cannot be patched in place), which stores a live reference to the cached patch's layers inside the UNet for the whole denoise. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com> Co-authored-by: JPPhoto <jpollack@jpollackphoto.com> | 3 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 3 天前 | ||
| 4 天前 | ||
| 1 个月前 | ||
| 3 个月前 |