| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
Add build_line(...) util function. | 1 年前 | |
feat: add native Intel XPU (torch.xpu) device support (#9401) * feat(backend): add Intel XPU (torch.xpu) device support Additive xpu branches only: device selection and normalization, float16 default, VRAM queries with a passthrough-VM fallback (missing SYCL free-memory aspect), fp8 layerwise casting via a runtime probe, VAE auto-tiling, partial loading, stats/OOM handling, multi-GPU parallel session execution (device enumeration, config/API validation, worker pinning, and the generation-device options endpoint), and the auxiliary image utilities (depth/SAM/DINO pipelines accept xpu instead of falling back to CPU; cache clearing is device-agnostic). CUDA (incl. ROCm), MPS, and CPU behavior unchanged. Verified end to end on Arc Pro B70 hardware, including dual-GPU worker startup. * test(backend): add XPU coverage for TorchDevice Mock-based, mirroring the CUDA/MPS suites: device choice, dtype, normalize, the xpu_mem_get_info fallback branches, and multi-GPU generation_devices resolution/validation/labeling on XPU. Also makes the auto-without-CUDA generation-devices test hermetic on XPU machines. * build: add [xpu] extra torch 2.7.1+xpu / torchvision 0.22.1+xpu / pytorch-triton-xpu 3.3.1 from the torch-xpu index, gated to linux-x86_64 and win_amd64; uv.lock regenerated. * feat(backend): extend idle-GPU text encoder offload to XPU The idle-device arbiter and the session processor's borrow path both gated on `device.type == "cuda"`, so on a multi-XPU system no device ever registered and `offload_text_encoders_to_idle_gpus` (enabled by default) silently did nothing: encoders kept churning the denoise model in and out of VRAM. Register and lend XPU devices alongside CUDA. MPS is deliberately excluded -- it is always a single shared device, so there is never another GPU to borrow. Verified on a dual Intel Arc Pro B70 host: a text encoder node now runs on the idle GPU while the session denoises on the other ("Running compel on idle device xpu:0 (session device xpu:1)"). * feat(ui): show the executing GPU for XPU sessions Queue items already persist the executing device generically (e.g. "xpu:1"), but both readers dropped it: the session event only forwarded devices starting with "cuda", and the frontend index parser only matched /^cuda:(\d+)$/. On a multi-XPU system the progress circle and queue-item badges were therefore always blank. Accept indexed XPU devices in both places, and correct the queue-item field description, which claimed the device is set only on CUDA. * fix(mm): gate Krea 2 fp8 encoder casting on fp8 storage support The Qwen3-VL encoder kept its fp8 storage only on CUDA, so elsewhere an fp8 checkpoint was loaded as full bf16 (~8.9GB instead of ~4.4GB) and thrashed partial loading when sharing a GPU with a large transformer. Reuse the existing cached `_device_supports_fp8_storage` probe, which already backs the layerwise-casting path. It returns True unconditionally on CUDA, so CUDA behaviour is unchanged. * chore: label XPU devices by index in load logs and fp8 help text Model load lines printed the device index only for CUDA, so every model on a multi-XPU host logged as a bare "xpu device", making it impossible to tell the GPUs apart. The FP8 Storage tooltip likewise claimed CUDA-only support. * test: cover XPU config validation, progress device and fp8 probe Three paths changed by this branch had no coverage: - The `device` field pattern was untested. `test_device_choice_xpu` looks like it covers it, but the config model does not enable `validate_assignment`, so assigning `config.device` skips validation entirely; only constructing the model exercises the pattern. Added constructor-based valid/invalid cases. - `generation_devices` validation was parametrized for cuda/cpu/mps only. - The progress event's device field, which now reports XPU sessions. Also cover `_device_supports_fp8_storage`, which gates FP8 storage in both the generic layerwise-casting path and the Krea 2 encoder: CUDA answers True without probing, CPU is rejected, and a failing XPU probe returns False instead of raising. Each new test was verified to fail when the corresponding fix is reverted. * fix(nodes): recognise XPU out-of-memory errors in the Anima VAE retry The Anima VAE decode catches OOM and retries once with tiling, which caps peak allocation. Detection matched `torch.cuda.OutOfMemoryError` or the words "out of memory" in the message, so it missed XPU entirely: torch's XPU backend does not raise a recoverable `torch.OutOfMemoryError` on exhaustion, it surfaces the Level Zero/UR result code as a plain RuntimeError -- and `UR_RESULT_ERROR_OUT_OF_DEVICE_MEMORY` contains no spaces, so the existing substring never matched. The decode therefore failed outright instead of retrying tiled. Match the `*_OUT_OF_DEVICE_MEMORY` / `*_OUT_OF_HOST_MEMORY` spellings (both UR and ZE prefixes) alongside the existing conditions, and fold the cuDNN/cuBLAS checks into the same case-insensitive comparison. Extends the existing parametrized retry test with the three XPU spellings; each was verified to fail before this change. Note the driver behaviour itself is not reproducible on the hardware used here -- this stack overcommits into host RAM and hangs rather than raising -- so the tests pin the classifier, not the driver. * style: wrap long vram_usage_gb ternary for ruff * fix: drop CUDA-only wording from progress device description Matches the committed openapi/schema artifacts, which already say "on a GPU". * docs: regenerate settings data for xpu device values * fix: stop xpu VRAM probe from reporting an unknown total as zero (0, 0) made the cache's available-VRAM arithmetic collapse to a constant -working_mem budget for the life of the process. Also widen the except: the failure type moves between torch releases (RuntimeError for the missing SYCL aspect, AssertionError from _lazy_init), and warn once when the blind estimate is in use. * fix: probe fp8 support on the target device, per device, without caching failures The probe allocated via an index-less "xpu", which resolves through the thread's current XPU device rather than the device being loaded onto -- so during idle-GPU encoder offload it measured the busy denoise GPU. It was also keyed on device type, letting one device decide for another, and memoised transient failures (it runs during a load, when the device may be momentarily full) with no way back but a restart. Also probe the bf16 upcast, which is the runtime path for Krea-2/FLUX. * fix: pin torch current device when borrowing an idle GPU Worker startup set both the session device and torch's per-thread current device; the offload borrow set only the former, leaving index-less allocations on the worker's own GPU. Extracted the shared helper and guarded it on backend availability. * fix: keep idle-GPU borrows within one device type generation_devices accepts a mixed list, so a cuda session could be handed an xpu device for its text encoder. * feat: detect Intel integrated GPUs via Level Zero torch exposes no is-integrated flag, but Level Zero does (ZE_DEVICE_PROPERTY_FLAG_INTEGRATED), and its loader already ships with the torch+xpu runtime -- so no new dependency and no compiled extension. Use it to keep iGPUs out of `generation_devices: auto` when a discrete GPU exists, and to stop budgeting them as dedicated VRAM (they share system RAM, like MPS). An unknown answer keeps the previous behaviour, an iGPU-only machine keeps its device, and an explicit device list can still opt one in. * feat: add xpu torch index to pins.json Gives the launcher an Intel install option instead of requiring a manual pip install of the extra. * fix: report VRAM diagnostics for the device in use All three sites dispatched on torch.cuda.is_available() first, so a mixed NVIDIA + Arc box running on xpu reported a constant 0.0 GB and logged "CUDA Memory Allocated" -- which would make XPU bug reports unactionable. * docs: record why xpu takes the CUDA VAE constants and keeps the broad OOM needle XPU SDPA was measured on Arc Pro B70 / torch 2.13+xpu: peak memory doubles when sequence length doubles (2.00x across 2048-16384; 2.0 MB at seq=16384 vs 512 MB for a materialised score matrix). So XPU is in CUDA's O(area) regime, not ROCm's math-attention regime, and the existing constants are correct rather than accidental. * fix: derive rand_device metadata from the backend's devices Was hardcoded to 'cuda' for any non-CPU noise, which is wrong on Arc. Falls back to 'cuda' when the device query has not resolved, so Nvidia metadata is unchanged. * docs: add Intel Arc install, driver and VRAM-reporting notes * fix: probe fp8 device support only when a model requests it The probe was the first statement in _should_use_fp8, so it allocated on the GPU during the first load of any model at all -- tokenizer, VAE, scheduler -- and on API/install threads it forced XPU lazy SYCL init on a thread that never generates. Moved below the exclusions. * fix: query Level Zero Sysman for driver-global free VRAM on xpu The blind estimate (total minus this process's reserved bytes) is what made _get_vram_available over-commit on a shared GPU: it feeds a formula that assumes a driver-global figure. Sysman's zesMemoryGetState reports that figure and is often available when the SYCL ext_intel_free_memory aspect is not, so try it before estimating. Measured on Arc Pro B70 with 16 GiB held by another process: Sysman reported 15.553 GiB free, the estimate 31.725 -- a 16.172 GiB error, exactly the foreign allocation. Sysman is not a guaranteed substitute (torch's query bottoms out in the same layer), so the estimate remains as a last resort. * fix: make the fp8 probe mirror the runtime cast path The storage cast happens on CPU while params are still CPU-resident, then the fp8 tensor is copied to the device and the pre-hook upcasts there. Probing all three steps on the device would pass on a build where the host->device fp8 copy or one upcast target fails, and break at forward time instead. Verified on Arc Pro B70 / torch 2.13+xpu: the full sequence works on both cards. * fix: degrade gracefully when a backend cannot name a device torch.xpu.get_device_name goes through _lazy_init, which raises AssertionError on a build without XPU. Naming is used only for labels and logs, so fall back to the device string rather than propagating. * fix: resolve an index-less device in the Sysman VRAM query Returning None for a device with no index would skip the driver-global query and fall through to the blind estimate with no visible symptom. Callers currently always pass a concrete device, so this is a latent hazard rather than a live bug. * fix: declare ctypes prototypes for the Level Zero calls Handles come back from (c_void_p * n)() as plain Python ints, and ctypes converts an undeclared int argument to a C int -- 32 bits. Any handle above 2**31 was being silently truncated; a direct test of that path segfaults. It happened to work on the B70 because the handles fit. Also: release the idle-GPU borrow if re-pinning raises (the setup was outside the try, so a failure there stranded the lock for the life of the process), and report a failing fp8 probe once per device instead of on every model load. * fix: drop the ZES_ENABLE_SYSMAN mutation from the Sysman probe Setting a process-wide environment variable from a read-only query leaks into child processes. It also bought nothing: the variable only gates Sysman on runtimes predating zesInit and must be set before Level Zero initialises, which torch has already done by then. Verified on Arc Pro B70 that zesInit succeeds with the variable unset. * refactor: tidy up the xpu additions after a cleanup review level_zero: cache the loader so it is opened and its prototypes configured once rather than twice, share the driver/device enumeration and its ordering guard between the two probes, and collapse the Sysman pair of globals into one nullable tuple. Also: fp8 support cache is a set (it only ever stored True), the pbr_maps empty_cache is routed through TorchDevice like the PR's other conversions, the shared-memory VRAM branch stops re-testing the device type it matched on, `_auto_generation_devices` partitions in one pass, and rand_device only answers when every generation device is the same accelerator. Merges three duplicate mem_get_info tests into one parametrized case and drops two fp8 probe tests fully subsumed by the cast-sequence test. * build: pin the xpu extra to torch 2.13.0 Intel's XPU backend matured considerably after 2.7.1: torch.xpu.mem_get_info() works on driver/kernel combinations where it previously raised, and the oneAPI user-space runtime ships with the wheel, so upgrading torch upgrades it too. Follows the rocm extra, which already pins ahead of cpu/cuda. pytorch-triton-xpu was renamed triton-xpu upstream. The darwin/aarch64 fallbacks stay on 2.7.1 to match the other extras and the project's torch<2.8.0 constraint on darwin. cpu/cuda/rocm exports are unchanged package-for-package (196/211/197); the only delta is a dropped "via pytorch-triton-xpu" comment annotation from the rename. * test: stub Sysman in the unknown-total xpu probe test Without it the test only passes where Level Zero cannot answer -- never on the Intel hardware the probe exists for, where Sysman returns before the tier under test is reached. * build: teach the pins check about the xpu index Its per-platform allowlist rejects anything unlisted, so pins.json's xpu entry fails it. PyTorch publishes XPU wheels for win32 and linux x86_64, matching the extra's markers. * fix: defer the xpu device pin like cuda's torch.xpu.set_device() brings up a SYCL context that holds VRAM in an otherwise idle process, the same reason the CUDA pin waits for the first claimed queue item. * docs: regenerate settings data on linux Regenerating on Windows flips two path defaults to backslashes, which the docs check rejects. * fix(mm): handle shared memory on integrated GPUs Their VRAM is system RAM, so a RAM copy doubles each model's footprint against the same pool. Drop it, letting a full load move weights rather than copy them. Keep partial loading on -- it is the only path that respects vram_available -- and raise a clean error when a full-load-only model cannot fit, instead of walking into an uncatchable OOM-kill. Warn once when a setting is overridden. Scoped to integrated XPU; CPU and MPS are unchanged. * docs: note intel device selection and integrated-GPU memory `auto` prefers CUDA on a mixed Nvidia/Arc box, and keep_ram_copy_of_weights is ignored on an integrated GPU. * chore(ui): typegen for the xpu device values * fix(mm): compare the integrated-GPU full-load guard against bytes still to move A resident model's weights occupy the same DRAM that vram_available is read from, so its total can exceed "available" precisely because it is loaded. lock() runs on every use and full_load_to_vram() is a no-op when resident; comparing the total refused the re-lock and evicted a healthy model on every other generation. Compare what full_load_to_vram() will actually move instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: run the integrated-GPU cache tests on CPU-only torch ModelCache.__init__ sizes the RAM cache from the device's total VRAM, which on an xpu execution device reads torch.xpu.get_device_properties() -- an AssertionError on the CPU-only builds CI runs, failing 9 of these tests before they reached their subject. Stub a fixed total during construction, and add a regression test for the resident-model re-lock guard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: LexiconCode <aaronwalker@protonmail.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> | 25 天前 | |
feat: add native Intel XPU (torch.xpu) device support (#9401) * feat(backend): add Intel XPU (torch.xpu) device support Additive xpu branches only: device selection and normalization, float16 default, VRAM queries with a passthrough-VM fallback (missing SYCL free-memory aspect), fp8 layerwise casting via a runtime probe, VAE auto-tiling, partial loading, stats/OOM handling, multi-GPU parallel session execution (device enumeration, config/API validation, worker pinning, and the generation-device options endpoint), and the auxiliary image utilities (depth/SAM/DINO pipelines accept xpu instead of falling back to CPU; cache clearing is device-agnostic). CUDA (incl. ROCm), MPS, and CPU behavior unchanged. Verified end to end on Arc Pro B70 hardware, including dual-GPU worker startup. * test(backend): add XPU coverage for TorchDevice Mock-based, mirroring the CUDA/MPS suites: device choice, dtype, normalize, the xpu_mem_get_info fallback branches, and multi-GPU generation_devices resolution/validation/labeling on XPU. Also makes the auto-without-CUDA generation-devices test hermetic on XPU machines. * build: add [xpu] extra torch 2.7.1+xpu / torchvision 0.22.1+xpu / pytorch-triton-xpu 3.3.1 from the torch-xpu index, gated to linux-x86_64 and win_amd64; uv.lock regenerated. * feat(backend): extend idle-GPU text encoder offload to XPU The idle-device arbiter and the session processor's borrow path both gated on `device.type == "cuda"`, so on a multi-XPU system no device ever registered and `offload_text_encoders_to_idle_gpus` (enabled by default) silently did nothing: encoders kept churning the denoise model in and out of VRAM. Register and lend XPU devices alongside CUDA. MPS is deliberately excluded -- it is always a single shared device, so there is never another GPU to borrow. Verified on a dual Intel Arc Pro B70 host: a text encoder node now runs on the idle GPU while the session denoises on the other ("Running compel on idle device xpu:0 (session device xpu:1)"). * feat(ui): show the executing GPU for XPU sessions Queue items already persist the executing device generically (e.g. "xpu:1"), but both readers dropped it: the session event only forwarded devices starting with "cuda", and the frontend index parser only matched /^cuda:(\d+)$/. On a multi-XPU system the progress circle and queue-item badges were therefore always blank. Accept indexed XPU devices in both places, and correct the queue-item field description, which claimed the device is set only on CUDA. * fix(mm): gate Krea 2 fp8 encoder casting on fp8 storage support The Qwen3-VL encoder kept its fp8 storage only on CUDA, so elsewhere an fp8 checkpoint was loaded as full bf16 (~8.9GB instead of ~4.4GB) and thrashed partial loading when sharing a GPU with a large transformer. Reuse the existing cached `_device_supports_fp8_storage` probe, which already backs the layerwise-casting path. It returns True unconditionally on CUDA, so CUDA behaviour is unchanged. * chore: label XPU devices by index in load logs and fp8 help text Model load lines printed the device index only for CUDA, so every model on a multi-XPU host logged as a bare "xpu device", making it impossible to tell the GPUs apart. The FP8 Storage tooltip likewise claimed CUDA-only support. * test: cover XPU config validation, progress device and fp8 probe Three paths changed by this branch had no coverage: - The `device` field pattern was untested. `test_device_choice_xpu` looks like it covers it, but the config model does not enable `validate_assignment`, so assigning `config.device` skips validation entirely; only constructing the model exercises the pattern. Added constructor-based valid/invalid cases. - `generation_devices` validation was parametrized for cuda/cpu/mps only. - The progress event's device field, which now reports XPU sessions. Also cover `_device_supports_fp8_storage`, which gates FP8 storage in both the generic layerwise-casting path and the Krea 2 encoder: CUDA answers True without probing, CPU is rejected, and a failing XPU probe returns False instead of raising. Each new test was verified to fail when the corresponding fix is reverted. * fix(nodes): recognise XPU out-of-memory errors in the Anima VAE retry The Anima VAE decode catches OOM and retries once with tiling, which caps peak allocation. Detection matched `torch.cuda.OutOfMemoryError` or the words "out of memory" in the message, so it missed XPU entirely: torch's XPU backend does not raise a recoverable `torch.OutOfMemoryError` on exhaustion, it surfaces the Level Zero/UR result code as a plain RuntimeError -- and `UR_RESULT_ERROR_OUT_OF_DEVICE_MEMORY` contains no spaces, so the existing substring never matched. The decode therefore failed outright instead of retrying tiled. Match the `*_OUT_OF_DEVICE_MEMORY` / `*_OUT_OF_HOST_MEMORY` spellings (both UR and ZE prefixes) alongside the existing conditions, and fold the cuDNN/cuBLAS checks into the same case-insensitive comparison. Extends the existing parametrized retry test with the three XPU spellings; each was verified to fail before this change. Note the driver behaviour itself is not reproducible on the hardware used here -- this stack overcommits into host RAM and hangs rather than raising -- so the tests pin the classifier, not the driver. * style: wrap long vram_usage_gb ternary for ruff * fix: drop CUDA-only wording from progress device description Matches the committed openapi/schema artifacts, which already say "on a GPU". * docs: regenerate settings data for xpu device values * fix: stop xpu VRAM probe from reporting an unknown total as zero (0, 0) made the cache's available-VRAM arithmetic collapse to a constant -working_mem budget for the life of the process. Also widen the except: the failure type moves between torch releases (RuntimeError for the missing SYCL aspect, AssertionError from _lazy_init), and warn once when the blind estimate is in use. * fix: probe fp8 support on the target device, per device, without caching failures The probe allocated via an index-less "xpu", which resolves through the thread's current XPU device rather than the device being loaded onto -- so during idle-GPU encoder offload it measured the busy denoise GPU. It was also keyed on device type, letting one device decide for another, and memoised transient failures (it runs during a load, when the device may be momentarily full) with no way back but a restart. Also probe the bf16 upcast, which is the runtime path for Krea-2/FLUX. * fix: pin torch current device when borrowing an idle GPU Worker startup set both the session device and torch's per-thread current device; the offload borrow set only the former, leaving index-less allocations on the worker's own GPU. Extracted the shared helper and guarded it on backend availability. * fix: keep idle-GPU borrows within one device type generation_devices accepts a mixed list, so a cuda session could be handed an xpu device for its text encoder. * feat: detect Intel integrated GPUs via Level Zero torch exposes no is-integrated flag, but Level Zero does (ZE_DEVICE_PROPERTY_FLAG_INTEGRATED), and its loader already ships with the torch+xpu runtime -- so no new dependency and no compiled extension. Use it to keep iGPUs out of `generation_devices: auto` when a discrete GPU exists, and to stop budgeting them as dedicated VRAM (they share system RAM, like MPS). An unknown answer keeps the previous behaviour, an iGPU-only machine keeps its device, and an explicit device list can still opt one in. * feat: add xpu torch index to pins.json Gives the launcher an Intel install option instead of requiring a manual pip install of the extra. * fix: report VRAM diagnostics for the device in use All three sites dispatched on torch.cuda.is_available() first, so a mixed NVIDIA + Arc box running on xpu reported a constant 0.0 GB and logged "CUDA Memory Allocated" -- which would make XPU bug reports unactionable. * docs: record why xpu takes the CUDA VAE constants and keeps the broad OOM needle XPU SDPA was measured on Arc Pro B70 / torch 2.13+xpu: peak memory doubles when sequence length doubles (2.00x across 2048-16384; 2.0 MB at seq=16384 vs 512 MB for a materialised score matrix). So XPU is in CUDA's O(area) regime, not ROCm's math-attention regime, and the existing constants are correct rather than accidental. * fix: derive rand_device metadata from the backend's devices Was hardcoded to 'cuda' for any non-CPU noise, which is wrong on Arc. Falls back to 'cuda' when the device query has not resolved, so Nvidia metadata is unchanged. * docs: add Intel Arc install, driver and VRAM-reporting notes * fix: probe fp8 device support only when a model requests it The probe was the first statement in _should_use_fp8, so it allocated on the GPU during the first load of any model at all -- tokenizer, VAE, scheduler -- and on API/install threads it forced XPU lazy SYCL init on a thread that never generates. Moved below the exclusions. * fix: query Level Zero Sysman for driver-global free VRAM on xpu The blind estimate (total minus this process's reserved bytes) is what made _get_vram_available over-commit on a shared GPU: it feeds a formula that assumes a driver-global figure. Sysman's zesMemoryGetState reports that figure and is often available when the SYCL ext_intel_free_memory aspect is not, so try it before estimating. Measured on Arc Pro B70 with 16 GiB held by another process: Sysman reported 15.553 GiB free, the estimate 31.725 -- a 16.172 GiB error, exactly the foreign allocation. Sysman is not a guaranteed substitute (torch's query bottoms out in the same layer), so the estimate remains as a last resort. * fix: make the fp8 probe mirror the runtime cast path The storage cast happens on CPU while params are still CPU-resident, then the fp8 tensor is copied to the device and the pre-hook upcasts there. Probing all three steps on the device would pass on a build where the host->device fp8 copy or one upcast target fails, and break at forward time instead. Verified on Arc Pro B70 / torch 2.13+xpu: the full sequence works on both cards. * fix: degrade gracefully when a backend cannot name a device torch.xpu.get_device_name goes through _lazy_init, which raises AssertionError on a build without XPU. Naming is used only for labels and logs, so fall back to the device string rather than propagating. * fix: resolve an index-less device in the Sysman VRAM query Returning None for a device with no index would skip the driver-global query and fall through to the blind estimate with no visible symptom. Callers currently always pass a concrete device, so this is a latent hazard rather than a live bug. * fix: declare ctypes prototypes for the Level Zero calls Handles come back from (c_void_p * n)() as plain Python ints, and ctypes converts an undeclared int argument to a C int -- 32 bits. Any handle above 2**31 was being silently truncated; a direct test of that path segfaults. It happened to work on the B70 because the handles fit. Also: release the idle-GPU borrow if re-pinning raises (the setup was outside the try, so a failure there stranded the lock for the life of the process), and report a failing fp8 probe once per device instead of on every model load. * fix: drop the ZES_ENABLE_SYSMAN mutation from the Sysman probe Setting a process-wide environment variable from a read-only query leaks into child processes. It also bought nothing: the variable only gates Sysman on runtimes predating zesInit and must be set before Level Zero initialises, which torch has already done by then. Verified on Arc Pro B70 that zesInit succeeds with the variable unset. * refactor: tidy up the xpu additions after a cleanup review level_zero: cache the loader so it is opened and its prototypes configured once rather than twice, share the driver/device enumeration and its ordering guard between the two probes, and collapse the Sysman pair of globals into one nullable tuple. Also: fp8 support cache is a set (it only ever stored True), the pbr_maps empty_cache is routed through TorchDevice like the PR's other conversions, the shared-memory VRAM branch stops re-testing the device type it matched on, `_auto_generation_devices` partitions in one pass, and rand_device only answers when every generation device is the same accelerator. Merges three duplicate mem_get_info tests into one parametrized case and drops two fp8 probe tests fully subsumed by the cast-sequence test. * build: pin the xpu extra to torch 2.13.0 Intel's XPU backend matured considerably after 2.7.1: torch.xpu.mem_get_info() works on driver/kernel combinations where it previously raised, and the oneAPI user-space runtime ships with the wheel, so upgrading torch upgrades it too. Follows the rocm extra, which already pins ahead of cpu/cuda. pytorch-triton-xpu was renamed triton-xpu upstream. The darwin/aarch64 fallbacks stay on 2.7.1 to match the other extras and the project's torch<2.8.0 constraint on darwin. cpu/cuda/rocm exports are unchanged package-for-package (196/211/197); the only delta is a dropped "via pytorch-triton-xpu" comment annotation from the rename. * test: stub Sysman in the unknown-total xpu probe test Without it the test only passes where Level Zero cannot answer -- never on the Intel hardware the probe exists for, where Sysman returns before the tier under test is reached. * build: teach the pins check about the xpu index Its per-platform allowlist rejects anything unlisted, so pins.json's xpu entry fails it. PyTorch publishes XPU wheels for win32 and linux x86_64, matching the extra's markers. * fix: defer the xpu device pin like cuda's torch.xpu.set_device() brings up a SYCL context that holds VRAM in an otherwise idle process, the same reason the CUDA pin waits for the first claimed queue item. * docs: regenerate settings data on linux Regenerating on Windows flips two path defaults to backslashes, which the docs check rejects. * fix(mm): handle shared memory on integrated GPUs Their VRAM is system RAM, so a RAM copy doubles each model's footprint against the same pool. Drop it, letting a full load move weights rather than copy them. Keep partial loading on -- it is the only path that respects vram_available -- and raise a clean error when a full-load-only model cannot fit, instead of walking into an uncatchable OOM-kill. Warn once when a setting is overridden. Scoped to integrated XPU; CPU and MPS are unchanged. * docs: note intel device selection and integrated-GPU memory `auto` prefers CUDA on a mixed Nvidia/Arc box, and keep_ram_copy_of_weights is ignored on an integrated GPU. * chore(ui): typegen for the xpu device values * fix(mm): compare the integrated-GPU full-load guard against bytes still to move A resident model's weights occupy the same DRAM that vram_available is read from, so its total can exceed "available" precisely because it is loaded. lock() runs on every use and full_load_to_vram() is a no-op when resident; comparing the total refused the re-lock and evicted a healthy model on every other generation. Compare what full_load_to_vram() will actually move instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: run the integrated-GPU cache tests on CPU-only torch ModelCache.__init__ sizes the RAM cache from the device's total VRAM, which on an xpu execution device reads torch.xpu.get_device_properties() -- an AssertionError on the CPU-only builds CI runs, failing 9 of these tests before they reached their subject. Stub a fixed total during construction, and add a regression test for the resident-model re-lock guard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: LexiconCode <aaronwalker@protonmail.com> Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> | 25 天前 | |
fix(fp8): resolve compute dtype instead of reading model.dtype (#9412) * fix(fp8): resolve compute dtype instead of reading model.dtype SDXL with fp8_storage crashed before the UNet was ever called: NotImplementedError: "pow_cuda" not implemented for 'Float8_e4m3fn' After layerwise casting the UNet's weights are float8_e4m3fn, and diffusers derives `model.dtype` from the first parameter — so `unet.dtype` reports a storage-only dtype. The legacy SD/SDXL denoise path used it for every tensor it built, so the latents were created in float8 and the first bit of scheduler math (`sigma ** 2` in `add_noise`) blew up. torch has no arithmetic kernels for float8; it is only valid for weights that the forward hooks cast up per layer. Add `get_model_compute_dtype()`: returns `model.dtype` for normal models and the compute dtype for fp8 ones. The loader records the compute dtype on the model when it applies the cast; if the marker is missing (older cache entry, Krea2 encoder path) the resolver scans for the first non-fp8 float param, which works because the cast skips norm layers. Converted every site that derived a tensor dtype from a possibly-fp8 model: latents, noise, mask, masked_latents, conditioning, IP-Adapter and LoRA patch weights in denoise_latents and tiled_multi_diffusion_denoise_latents, plus the LoRA and T2I-Adapter extensions on the modular path. ControlNet and T2I-Adapter control images had the same latent bug — those configs expose an fp8_storage toggle too, so their control image would have been built in float8. Also point LayerPatcher at the shared FP8_STORAGE_DTYPES constant. Regression test covers the real loader path: `model.dtype` is float8 while the resolver returns fp16, and the resolved dtype survives the scheduler arithmetic that crashed. * fix(fp8): harden compute-dtype marker against double-cast poisoning Review follow-up on the compute-dtype resolver. `_apply_fp8_layerwise_casting` derives the compute dtype from the first parameter and is not idempotent. Called on an already-cast model, the first param is float8, so it would record float8 as the *compute* dtype — and `get_model_compute_dtype` trusts the marker, silently reintroducing the "pow_cuda" not implemented for 'Float8_e4m3fn' crash. Two guards close the class: `set_fp8_compute_dtype` rejects any storage-only dtype, and the cast early-returns when the marker is already present. Move the marker-setting into `_apply_fp8_to_nn_module` itself. It was duplicated at both call sites (load_default and krea2's text encoder), so a third caller would have to remember it — the exact failure the fallback scan exists to paper over. Log a warning when the last-resort fallback fires (fp8 storage, no marker, no non-fp8 float param): it returns the global torch dtype, which is wrong for a bf16-compute model and would otherwise surface as an unexplained mismatch deep in the forward pass. Note the same bug in the vendored HiDiffusion pipeline, which builds control images from `controlnet.dtype` at four sites. Dead code today — only apply_hidiffusion/remove_hidiffusion are imported — but it would reproduce the crash if ever wired up with an fp8 ControlNet. Tests: the float8-marker guard for both fp8 dtypes, the marker is set by the cast itself, and a double cast is a no-op (skipped norm layer stays in compute dtype, hooks registered once). The marker-missing fallback test now simulates a legacy model with delattr instead of locking in the old split. --------- Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com> | 1 个月前 | |
chore: ruff | 2 年前 | |
Add utility to_standard_float_mask(...) to convert various mask formats to a standardized format. | 2 年前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 1 年前 | ||
| 25 天前 | ||
| 25 天前 | ||
| 1 个月前 | ||
| 2 年前 | ||
| 2 年前 |