| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
feat(assets): split asset records from content (#16295) * review-stack 1/4: code (37 files, +3217/-3958) Review-and-land stack for synap5e/feat/asset-record-content-split, generated by review-stack.py. Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: path not under tests-unit/ or tests/ Question: Is the logic change right? Source tip: 7007d185825751a09f26bf7ba79f146dc2fbda74 Merge-base: 783545f689a0af730065994b46b382ae24844c99 * review-stack 2/4: tests-removed (24 files, +274/-8220) Review-and-land stack for synap5e/feat/asset-record-content-split, generated by review-stack.py. Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: test file deleted, or modified with deleted/(added+deleted) >= 0.9 Question: For each dropped assertion: obsolete by a ruling, or covered by a tests-new test? Source tip: 7007d185825751a09f26bf7ba79f146dc2fbda74 Merge-base: 783545f689a0af730065994b46b382ae24844c99 * review-stack 3/4: tests-changed (13 files, +1043/-1218) Review-and-land stack for synap5e/feat/asset-record-content-split, generated by review-stack.py. Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: remaining modified test files (incl. conftest.py / helpers) Question: Did the edits weaken an existing check? Source tip: 7007d185825751a09f26bf7ba79f146dc2fbda74 Merge-base: 783545f689a0af730065994b46b382ae24844c99 * review-stack 4/4: tests-new (46 files, +8601/-0) Review-and-land stack for synap5e/feat/asset-record-content-split, generated by review-stack.py. Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: test file added Question: Is the code layer well covered? Source tip: 7007d185825751a09f26bf7ba79f146dc2fbda74 Merge-base: 783545f689a0af730065994b46b382ae24844c99 * review-stack 5/6: code (13 files, +351/-104) Review-and-land stack for synap5e/feat/assets-di, generated by review-stack.py. Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: path not under tests-unit/ or tests/ Question: Is the logic change right? Source tip: eca2c74bffcbb4481e90f33d5b7b4efeeb05eb45 Merge-base: 20d59d2a5f8108d714a908b73a9c1754b9c36f2a * review-stack 6/6: tests (8 files, +753/-238) Review-and-land stack for synap5e/feat/assets-di, generated by review-stack.py. Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: every changed file under tests-unit/ or tests/ (added, modified, or deleted) Question: Is the code layer well covered, and did any edit weaken an existing check? Source tip: eca2c74bffcbb4481e90f33d5b7b4efeeb05eb45 Merge-base: 20d59d2a5f8108d714a908b73a9c1754b9c36f2a * review-stack 7/8: ported-fixes (42 files, +1361/-180) Review-and-land stack for synap5e/feat/assets-di-v2, generated by review-stack.py conventions (hand-built continuation layer; see the PR body). Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: the 11 base-branch fix/docs commits 595cd6e4..94d7185b cherry-picked across the DI refactor (7efdd1d7 excluded, superseded by layer 8) Question: was each base fix ported faithfully across the DI refactor? Source tip: 6841881069284803b902b4a9e33bdcda13126771 Merge-base: 7fdfb40f4b1d04c5b4b26ac7ab2328c83a8cb126 * review-stack 8/8: defensive-parity (4 files, +36/-3) Review-and-land stack for synap5e/feat/assets-di-v2, generated by review-stack.py conventions (hand-built continuation layer; see the PR body). Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: match-or-improve master's dependency defenses — NoAssets selection when DB deps unavailable (7efdd1d7's outcome via the DI seam), requirements warning before assets imports, blake3 in the guarded dependency set Question: does each degradation path now match or improve master's behavior? Source tip: ebc2cfeebc5a1ebae407d0cb975afb5f293b4111 Merge-base: 7fdfb40f4b1d04c5b4b26ac7ab2328c83a8cb126 * fix(assets): only discard content rows this operation actually inserted CR-9: Enumerated all six create_content call sites. Only scanner seeding and the three ingest registration paths track IDs for failure cleanup. * fix(assets): reject hash-only uploads with FEATURE_DISABLED when hashing is off CodeRabbit finding CR-2: reject hash-only multipart uploads before create_from_hash when hashing is disabled. * fix(assets): seed persists the stat it verified CR-7: persist the fresh seed-time restat instead of walk-time spec values. * fix(assets): route database lock failures to the lock guidance CR-16: route file-lock startup failures through the existing lock guidance and exit path. * fix(assets): drop the inaccurate temp-cleanup claim from the shutdown warning References CR-10. * fix(assets): walk the output root after execution so undeclared outputs register promptly Custom nodes that write files into the output directory without declaring them in output_ui only became assets when the next full walk happened - a frontend GET /object_info or a restart. Headless and API-only sessions never trigger either, so those files never converged into the asset database. The post-execution hook now requests a FULL scan of the output root instead of an enrich-only pass. The seeder's pending-request queue was generalised from enrich-specific to carrying a scan phase, so the request starts immediately when the seeder is idle and coalesces (escalating to FULL on a phase mismatch) when a scan is already running. queue_output_enrichment is renamed to queue_output_scan across the protocol, the NoAssets no-op and the call site. References FIX-6. * chore(assets): remove seeder paths orphaned by the output-scan change 45c2f96e rerouted both former enrich call sites to start()/enqueue_scan(), leaving two seeder methods that look live but are not. Review round F2 raised this along with four smaller items; the user's disposition was to fix all six here. - Delete start_enrich: zero callers repo-wide after 45c2f96e. - Delete enqueue_enrich: no production callers; its ~18 call sites in tests/test_asset_seeder.py move to enqueue_scan(phase=ScanPhase.ENRICH) with their semantics unchanged. The deletion forces the half-done class renames (TestEnqueueEnrich* -> TestEnqueueScan*, consistent with the already-renamed TestPendingScanDrain) and restores the module docstring that was dropped rather than reworded. - Document at manager.queue_output_scan that ScanPhase.FULL per debounce window is the deliberate, user-ratified trade, so it is not optimised back to ENRICH without revisiting the decision. - Document that SeedAssetSpec.size_bytes/mtime_ns are walk-time diagnostics only - production persists the seed-time restat since CR-7. - Export create_content_reporting_insert from the queries facade and fold scanner.py's direct-module import into the existing facade block. - Harden test_queue_output_scan_does_not_duplicate_declared_output against a vacuous pass: it now asserts the seeder finished without errors and that an undeclared sibling written into the same directory WAS registered by the same scan, proving the walk actually ran. No production behaviour changes beyond the two deletions. References F2-cleanup. * chore: comment cleanup Comment-Gate: 18 quarantined * fix(assets): preserve pause across the seeder's pending-scan drain pause() runs before every prompt, while pending-scan enqueue and resume only run inside the debounced gc-interval gate. If the active scan finishes just after the next prompt's pause, its finally block resets the seeder to idle and the pending drain starts a replacement with the run gate open, so resume becomes a no-op. Capture pausedness under the lock before resetting to idle, then start the drained scan already paused. Setting the state and gate before launching the thread avoids the start-then-reclear window and lets resume release the existing scan checkpoints. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * test(assets): pin job_id absence for scan-discovered assets Owner ruling, recorded 2026-09-03 in the stack-9-hardening planning notepad: scan-discovered assets — including undeclared outputs found by the post-execution walk — carry job_id = None, always; only emission-time registration (output_ui declaration) attributes a job; attributing walk finds to the most recent prompt would be a temporal-correlation guess that is wrong exactly when prompts interleave; None is honest provenance. Do NOT add proximity-based attribution heuristics to the scanner. Ratified against Jacob Segal's cross-job-attribution concern (2026-09-08 review meeting) — a wrongly-attributed asset could mean one user's cloud job sees another user's asset. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * [review-stack 10/10] assets-tests (#16218) * test(execution): run the battery with assets enabled and assert asset-system health at teardown * test(execution): cover list-shaped outputs registering assets Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> --------- Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * [review-stack 11/11] review-fixes (#16261) * fix(assets): only exit on database file-lock timeout when assets are enabled * test(assets): pin live_contents_under_prefixes path-filtering semantics * perf(assets): push live-content prefix filtering into SQL * test(assets): declare per-entry intent in the path-prefix corpus * test(assets): normalize POSIX-literal path expectations for Windows * test(assets): force observable stat changes and close-before-mutate on Windows-sensitive rewrites * test(assets): force an observable mtime change in the hash-mode split test --------- Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> Co-authored-by: guill <jacob.e.segal@gmail.com> | 21 天前 | |
feat(assets): split asset records from content (#16295) * review-stack 1/4: code (37 files, +3217/-3958) Review-and-land stack for synap5e/feat/asset-record-content-split, generated by review-stack.py. Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: path not under tests-unit/ or tests/ Question: Is the logic change right? Source tip: 7007d185825751a09f26bf7ba79f146dc2fbda74 Merge-base: 783545f689a0af730065994b46b382ae24844c99 * review-stack 2/4: tests-removed (24 files, +274/-8220) Review-and-land stack for synap5e/feat/asset-record-content-split, generated by review-stack.py. Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: test file deleted, or modified with deleted/(added+deleted) >= 0.9 Question: For each dropped assertion: obsolete by a ruling, or covered by a tests-new test? Source tip: 7007d185825751a09f26bf7ba79f146dc2fbda74 Merge-base: 783545f689a0af730065994b46b382ae24844c99 * review-stack 3/4: tests-changed (13 files, +1043/-1218) Review-and-land stack for synap5e/feat/asset-record-content-split, generated by review-stack.py. Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: remaining modified test files (incl. conftest.py / helpers) Question: Did the edits weaken an existing check? Source tip: 7007d185825751a09f26bf7ba79f146dc2fbda74 Merge-base: 783545f689a0af730065994b46b382ae24844c99 * review-stack 4/4: tests-new (46 files, +8601/-0) Review-and-land stack for synap5e/feat/asset-record-content-split, generated by review-stack.py. Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: test file added Question: Is the code layer well covered? Source tip: 7007d185825751a09f26bf7ba79f146dc2fbda74 Merge-base: 783545f689a0af730065994b46b382ae24844c99 * review-stack 5/6: code (13 files, +351/-104) Review-and-land stack for synap5e/feat/assets-di, generated by review-stack.py. Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: path not under tests-unit/ or tests/ Question: Is the logic change right? Source tip: eca2c74bffcbb4481e90f33d5b7b4efeeb05eb45 Merge-base: 20d59d2a5f8108d714a908b73a9c1754b9c36f2a * review-stack 6/6: tests (8 files, +753/-238) Review-and-land stack for synap5e/feat/assets-di, generated by review-stack.py. Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: every changed file under tests-unit/ or tests/ (added, modified, or deleted) Question: Is the code layer well covered, and did any edit weaken an existing check? Source tip: eca2c74bffcbb4481e90f33d5b7b4efeeb05eb45 Merge-base: 20d59d2a5f8108d714a908b73a9c1754b9c36f2a * review-stack 7/8: ported-fixes (42 files, +1361/-180) Review-and-land stack for synap5e/feat/assets-di-v2, generated by review-stack.py conventions (hand-built continuation layer; see the PR body). Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: the 11 base-branch fix/docs commits 595cd6e4..94d7185b cherry-picked across the DI refactor (7efdd1d7 excluded, superseded by layer 8) Question: was each base fix ported faithfully across the DI refactor? Source tip: 6841881069284803b902b4a9e33bdcda13126771 Merge-base: 7fdfb40f4b1d04c5b4b26ac7ab2328c83a8cb126 * review-stack 8/8: defensive-parity (4 files, +36/-3) Review-and-land stack for synap5e/feat/assets-di-v2, generated by review-stack.py conventions (hand-built continuation layer; see the PR body). Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: match-or-improve master's dependency defenses — NoAssets selection when DB deps unavailable (7efdd1d7's outcome via the DI seam), requirements warning before assets imports, blake3 in the guarded dependency set Question: does each degradation path now match or improve master's behavior? Source tip: ebc2cfeebc5a1ebae407d0cb975afb5f293b4111 Merge-base: 7fdfb40f4b1d04c5b4b26ac7ab2328c83a8cb126 * fix(assets): only discard content rows this operation actually inserted CR-9: Enumerated all six create_content call sites. Only scanner seeding and the three ingest registration paths track IDs for failure cleanup. * fix(assets): reject hash-only uploads with FEATURE_DISABLED when hashing is off CodeRabbit finding CR-2: reject hash-only multipart uploads before create_from_hash when hashing is disabled. * fix(assets): seed persists the stat it verified CR-7: persist the fresh seed-time restat instead of walk-time spec values. * fix(assets): route database lock failures to the lock guidance CR-16: route file-lock startup failures through the existing lock guidance and exit path. * fix(assets): drop the inaccurate temp-cleanup claim from the shutdown warning References CR-10. * fix(assets): walk the output root after execution so undeclared outputs register promptly Custom nodes that write files into the output directory without declaring them in output_ui only became assets when the next full walk happened - a frontend GET /object_info or a restart. Headless and API-only sessions never trigger either, so those files never converged into the asset database. The post-execution hook now requests a FULL scan of the output root instead of an enrich-only pass. The seeder's pending-request queue was generalised from enrich-specific to carrying a scan phase, so the request starts immediately when the seeder is idle and coalesces (escalating to FULL on a phase mismatch) when a scan is already running. queue_output_enrichment is renamed to queue_output_scan across the protocol, the NoAssets no-op and the call site. References FIX-6. * chore(assets): remove seeder paths orphaned by the output-scan change 45c2f96e rerouted both former enrich call sites to start()/enqueue_scan(), leaving two seeder methods that look live but are not. Review round F2 raised this along with four smaller items; the user's disposition was to fix all six here. - Delete start_enrich: zero callers repo-wide after 45c2f96e. - Delete enqueue_enrich: no production callers; its ~18 call sites in tests/test_asset_seeder.py move to enqueue_scan(phase=ScanPhase.ENRICH) with their semantics unchanged. The deletion forces the half-done class renames (TestEnqueueEnrich* -> TestEnqueueScan*, consistent with the already-renamed TestPendingScanDrain) and restores the module docstring that was dropped rather than reworded. - Document at manager.queue_output_scan that ScanPhase.FULL per debounce window is the deliberate, user-ratified trade, so it is not optimised back to ENRICH without revisiting the decision. - Document that SeedAssetSpec.size_bytes/mtime_ns are walk-time diagnostics only - production persists the seed-time restat since CR-7. - Export create_content_reporting_insert from the queries facade and fold scanner.py's direct-module import into the existing facade block. - Harden test_queue_output_scan_does_not_duplicate_declared_output against a vacuous pass: it now asserts the seeder finished without errors and that an undeclared sibling written into the same directory WAS registered by the same scan, proving the walk actually ran. No production behaviour changes beyond the two deletions. References F2-cleanup. * chore: comment cleanup Comment-Gate: 18 quarantined * fix(assets): preserve pause across the seeder's pending-scan drain pause() runs before every prompt, while pending-scan enqueue and resume only run inside the debounced gc-interval gate. If the active scan finishes just after the next prompt's pause, its finally block resets the seeder to idle and the pending drain starts a replacement with the run gate open, so resume becomes a no-op. Capture pausedness under the lock before resetting to idle, then start the drained scan already paused. Setting the state and gate before launching the thread avoids the start-then-reclear window and lets resume release the existing scan checkpoints. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * test(assets): pin job_id absence for scan-discovered assets Owner ruling, recorded 2026-09-03 in the stack-9-hardening planning notepad: scan-discovered assets — including undeclared outputs found by the post-execution walk — carry job_id = None, always; only emission-time registration (output_ui declaration) attributes a job; attributing walk finds to the most recent prompt would be a temporal-correlation guess that is wrong exactly when prompts interleave; None is honest provenance. Do NOT add proximity-based attribution heuristics to the scanner. Ratified against Jacob Segal's cross-job-attribution concern (2026-09-08 review meeting) — a wrongly-attributed asset could mean one user's cloud job sees another user's asset. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * [review-stack 10/10] assets-tests (#16218) * test(execution): run the battery with assets enabled and assert asset-system health at teardown * test(execution): cover list-shaped outputs registering assets Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> --------- Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * [review-stack 11/11] review-fixes (#16261) * fix(assets): only exit on database file-lock timeout when assets are enabled * test(assets): pin live_contents_under_prefixes path-filtering semantics * perf(assets): push live-content prefix filtering into SQL * test(assets): declare per-entry intent in the path-prefix corpus * test(assets): normalize POSIX-literal path expectations for Windows * test(assets): force observable stat changes and close-before-mutate on Windows-sensitive rewrites * test(assets): force an observable mtime change in the hash-mode split test --------- Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> Co-authored-by: guill <jacob.e.segal@gmail.com> | 21 天前 | |
feat(assets): align local API with cloud spec (#12863) * feat(assets): align local API with cloud spec Unify response models, add missing fields, and align input schemas with the cloud OpenAPI spec at cloud.comfy.org/openapi. - Replace AssetSummary/AssetDetail/AssetUpdated with single Asset model - Add is_immutable, metadata (system_metadata), prompt_id fields - Support mime_type and preview_id in update endpoint - Make CreateFromHashBody.name optional, add mime_type, require >=1 tag - Add id/mime_type/preview_id to upload, relax tags to optional - Rename total_tags → tags in tag add/remove responses - Add GET /api/assets/tags/refine histogram endpoint - Add DB migration for system_metadata and prompt_id columns Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix review issues: tags validation, size nullability, type annotation, hash mismatch check, and add tag histogram tests - Remove contradictory min_length=1 from CreateFromHashBody.tags default - Restore size field to int|None=None for proper null semantics - Add Union type annotation to _build_asset_response result param - Add hash mismatch validation on idempotent upload path (409 HASH_MISMATCH) - Add unit tests for list_tag_histogram service function Amp-Thread-ID: https://ampcode.com/threads/T-019cd993-f43c-704e-b3d7-6cfc3d4d4a80 Co-authored-by: Amp <amp@ampcode.com> * Add preview_url to /assets API response using /api/view endpoint For input and output assets, generate a preview_url pointing to the existing /api/view endpoint using the asset's filename and tag-derived type (input/output). Handles subdirectories via subfolder param and URL-encodes filenames with spaces, unicode, and special characters. This aligns the OSS backend response with the frontend AssetCard expectation for thumbnail rendering. Amp-Thread-ID: https://ampcode.com/threads/T-019cda3f-5c2c-751a-a906-ac6c9153ac5c Co-authored-by: Amp <amp@ampcode.com> * chore: remove unused imports from asset_reference queries Amp-Thread-ID: https://ampcode.com/threads/T-019cda7d-cb21-77b4-a51b-b965af60208c Co-authored-by: Amp <amp@ampcode.com> * feat: resolve blake3 hashes in /view endpoint via asset database Amp-Thread-ID: https://ampcode.com/threads/T-019cda7d-cb21-77b4-a51b-b965af60208c Co-authored-by: Amp <amp@ampcode.com> * Register uploaded images in asset database when --enable-assets is set Add register_file_in_place() service function to ingest module for registering already-saved files without moving them. Call it from the /upload/image endpoint to return asset metadata in the response. Amp-Thread-ID: https://ampcode.com/threads/T-019ce023-3384-7560-bacf-de40b0de0dd2 Co-authored-by: Amp <amp@ampcode.com> * Exclude None fields from asset API JSON responses Add exclude_none=True to model_dump() calls across asset routes to keep response payloads clean by omitting unset optional fields. Amp-Thread-ID: https://ampcode.com/threads/T-019ce023-3384-7560-bacf-de40b0de0dd2 Co-authored-by: Amp <amp@ampcode.com> * Add comment explaining why /view resolves blake3 hashes Amp-Thread-ID: https://ampcode.com/threads/T-019ce023-3384-7560-bacf-de40b0de0dd2 Co-authored-by: Amp <amp@ampcode.com> * Move blake3 hash resolution to asset_management service Extract resolve_hash_to_path() into asset_management.py and remove _resolve_blake3_to_path from server.py. Also revert loopback origin check to original logic. Amp-Thread-ID: https://ampcode.com/threads/T-019ce023-3384-7560-bacf-de40b0de0dd2 Co-authored-by: Amp <amp@ampcode.com> * Require at least one tag in UploadAssetSpec Enforce non-empty tags at the Pydantic validation layer so uploads with no tags are rejected with a 400 before reaching ingest. Adds test_upload_empty_tags_rejected to cover this case. Amp-Thread-ID: https://ampcode.com/threads/T-019ce377-8bde-7048-bc28-a9df063409f9 Co-authored-by: Amp <amp@ampcode.com> * Add owner_id check to resolve_hash_to_path Filter asset references by owner visibility so the /view endpoint only resolves hashes for assets the requesting user can access. Adds table-driven tests for owner visibility cases. Amp-Thread-ID: https://ampcode.com/threads/T-019ce377-8bde-7048-bc28-a9df063409f9 Co-authored-by: Amp <amp@ampcode.com> * Make ReferenceData.created_at and updated_at required Remove None defaults and type: ignore comments. Move fields before optional fields to satisfy dataclass ordering. Amp-Thread-ID: https://ampcode.com/threads/T-019ce377-8bde-7048-bc28-a9df063409f9 Co-authored-by: Amp <amp@ampcode.com> * Fix double commit in create_from_hash Move mime_type update into _register_existing_asset so it shares a single transaction with reference creation. Log a warning when the hash is not found instead of silently returning None. Amp-Thread-ID: https://ampcode.com/threads/T-019ce377-8bde-7048-bc28-a9df063409f9 Co-authored-by: Amp <amp@ampcode.com> * Add exclude_none=True to create/upload responses Align with get/update/list endpoints for consistent JSON output. Amp-Thread-ID: https://ampcode.com/threads/T-019ce377-8bde-7048-bc28-a9df063409f9 Co-authored-by: Amp <amp@ampcode.com> * Change preview_id to reference asset by reference ID, not content ID Clients receive preview_id in API responses but could not dereference it through public routes (which use reference IDs). Now preview_id is a self-referential FK to asset_references.id so the value is directly usable in the public API. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Filter soft-deleted and missing refs from visibility queries list_references_by_asset_id and list_tags_with_usage were not filtering out deleted_at/is_missing refs, allowing /view?filename=blake3:... to serve files through hidden references and inflating tag usage counts. Add list_all_file_paths_by_asset_id for orphan cleanup which intentionally needs unfiltered access. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Pass preview_id and mime_type through all asset creation fast paths The duplicate-content upload path and hash-based creation paths were silently dropping preview_id and mime_type. This wires both fields through _register_existing_asset, create_from_hash, and all route call sites so behavior is consistent regardless of whether the asset content already exists. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Remove unimplemented client-provided ID from upload API The `id` field on UploadAssetSpec was advertised for idempotent creation but never actually honored when creating new references. Remove it rather than implementing the feature. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Make asset mime_type immutable after first ingest Prevents cross-tenant metadata mutation when multiple references share the same content-addressed Asset row. mime_type can now only be set when NULL (first ingest); subsequent attempts to change it are silently ignored. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Use resolved content_type from asset lookup in /view endpoint The /view endpoint was discarding the content_type computed by resolve_hash_to_path() and re-guessing from the filename, which produced wrong results for extensionless files or mismatched extensions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Merge system+user metadata into filter projection Extract rebuild_metadata_projection() to build AssetReferenceMeta rows from {**system_metadata, **user_metadata}, so system-generated metadata is queryable via metadata_filter and user keys override system keys. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Standardize tag ordering to alphabetical across all endpoints Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Derive subfolder tags from path in register_file_in_place Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Reject client-provided id, fix preview URLs, rename tags→total_tags - Reject 'id' field in multipart upload with 400 UNSUPPORTED_FIELD instead of silently ignoring it - Build preview URL from the preview asset's own metadata rather than the parent asset's - Rename 'tags' to 'total_tags' in TagsAdd/TagsRemove response schemas for clarity Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: SQLite migration 0003 FK drop fails on file-backed DBs (MB-2) Add naming_convention to Base.metadata so Alembic batch-mode reflection can match unnamed FK constraints created by migration 0002. Pass naming_convention and render_as_batch=True through env.py online config. Add migration roundtrip tests (upgrade/downgrade/cycle from baseline). Amp-Thread-ID: https://ampcode.com/threads/T-019ce466-1683-7471-b6e1-bb078223cda0 Co-authored-by: Amp <amp@ampcode.com> * Fix missing tag count for is_missing references and update test for total_tags field - Allow is_missing=True references to be counted in list_tags_with_usage when the tag is 'missing', so the missing tag count reflects all references that have been tagged as missing - Add update_is_missing_by_asset_id query helper for bulk updates by asset - Update test_add_and_remove_tags to use 'total_tags' matching the API schema Amp-Thread-ID: https://ampcode.com/threads/T-019ce482-05e7-7324-a1b0-a56a929cc7ef Co-authored-by: Amp <amp@ampcode.com> * Remove unused imports in scanner.py Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Rename prompt_id to job_id on asset_references Rename the column in the DB model, migration, and service schemas. The API response emits both job_id and prompt_id (deprecated alias) for backward compatibility with the cloud API. Amp-Thread-ID: https://ampcode.com/threads/T-019cef41-60b0-752a-aa3c-ed7f20fda2f7 Co-authored-by: Amp <amp@ampcode.com> * Add index on asset_references.preview_id for FK cascade performance Amp-Thread-ID: https://ampcode.com/threads/T-019cef45-a4d2-7548-86d2-d46bcd3db419 Co-authored-by: Amp <amp@ampcode.com> * Add clarifying comments for Asset/AssetReference naming and preview_id Amp-Thread-ID: https://ampcode.com/threads/T-019cef49-f94e-7348-bf23-9a19ebf65e0d Co-authored-by: Amp <amp@ampcode.com> * Disallow all-null meta rows: add CHECK constraint, skip null values on write - convert_metadata_to_rows returns [] for None values instead of an all-null row - Remove dead None branch from _scalar_to_row - Simplify null filter in common.py to just check for row absence - Add CHECK constraint ck_asset_reference_meta_has_value to model and migration 0003 Amp-Thread-ID: https://ampcode.com/threads/T-019cef4e-5240-7749-bb25-1f17fcf9c09c Co-authored-by: Amp <amp@ampcode.com> * Remove dead None guards on result.asset in upload handler register_file_in_place guarantees a non-None asset, so the 'if result.asset else None' checks were unreachable. Amp-Thread-ID: https://ampcode.com/threads/T-019cef5b-4cf8-723c-8a98-8fb8f333c133 Co-authored-by: Amp <amp@ampcode.com> * Remove mime_type from asset update API Clients can no longer modify mime_type after asset creation via the PUT /api/assets/{id} endpoint. This reduces the risk of mime_type spoofing. The internal update_asset_hash_and_mime function remains available for server-side use (e.g., enrichment). Amp-Thread-ID: https://ampcode.com/threads/T-019cef5d-8d61-75cc-a1c6-2841ac395648 Co-authored-by: Amp <amp@ampcode.com> * Fix migration constraint naming double-prefix and NULL in mixed metadata lists - Use fully-rendered constraint names in migration 0003 to avoid the naming convention doubling the ck_ prefix on batch operations. - Add table_args to downgrade so SQLite batch mode can find the CHECK constraint (not exposed by SQLite reflection). - Fix model CheckConstraint name to use bare 'has_value' (convention auto-prefixes). - Skip None items when converting metadata lists to rows, preventing all-NULL rows that violate the has_value check constraint. Amp-Thread-ID: https://ampcode.com/threads/T-019cef87-94f9-7172-a6af-c6282290ce4f Co-authored-by: Amp <amp@ampcode.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Amp <amp@ampcode.com> | 6 个月前 | |
chore(assets): drop vestigial tags.tag_type column (#14248) tag_type was always "user" in practice — no code path ever set it to anything else (no system/seeded classification was wired up) and nothing queried it. The column, its ix_tags_tag_type index, and the TagUsage.type API field were dead weight, so they're removed. Adds alembic migration 0004 to drop the column and index. Verified: asset-seeder tests pass; migration applies cleanly on a fresh SQLite (tags retains only name; tag_type column + index dropped). Co-authored-by: guill <jacob.e.segal@gmail.com> | 3 个月前 | |
feat(assets): add namespaced model_type tags and align tag semantics (#14511) * feat(assets): add namespaced model type tags * fix(assets): mark path-derived upload tags automatic * fix(assets): merge duplicate scan specs * test(assets): make duplicate path normalization portable * feat(assets): add loader_path as the authoritative loader locator (#14796) * fix(assets): filter model_type tags by bucket extension sets Buckets sharing a base directory (e.g. diffusion_models and a custom unet_gguf) tagged every file in the directory regardless of whether the bucket could load it, so .safetensors files were tagged model_type:unet_gguf and vice versa. Carry each bucket's registered extension set through get_comfy_models_folders and only emit a model_type tag when the file extension matches, keeping the empty-set match-all convention from folder_paths.filter_files_extensions. Files under a model base matching no bucket now keep only the models tag instead of every directory-matching model_type tag. * feat(assets): replace response file_path with persisted loader_path The old file_path response field was a namespaced storage locator (models/checkpoints/foo.safetensors): not an absolute path, not unique identity, and not the value a loader consumes. Nothing needs that shape on the wire (hash/ID-based locating is the long-term direction), so it is dropped rather than renamed; the storage-root matching stays internal, powering display_name. What loaders DO need is the in-root loader path (category dropped: models/checkpoints/foo/bar.safetensors -> foo/bar.safetensors). Serve it as a first-class loader_path field, persisted on asset_references (migration 0006) and written by every ingest pipeline at insert, so responses read the column verbatim. Like the model_type tags, loader_path is a seed-time derivative of the model folder registry, maintained by the same scan lifecycle (new files seed fresh values, pruning retires rows whose bucket disappeared). Rows predating the column serve a null loader_path; databases from before this stack already need recreating for the base branch's tag changes. loader_path resolves every registered base including extra_model_paths entries; display_name only the canonical storage roots. A file can therefore be loadable with no display name (extra-path models) or the reverse (unregistered files under the models root), and loader_path is null exactly when no loader can resolve the file. * test(assets): lock loader_path matrix (asymmetry, null, persist/read) Cover the behaviour that has no production change but is easy to regress: the extra-path asymmetry (loadable but no storage namespace), null loader_path persistence for orphan files, and the response reading the stored column with a compute fallback for un-backfilled rows. * fix(assets): persist subfolder-qualified loader_path for ingested outputs ingest_existing_file built its seed spec with the file's basename, so outputs saved into a subfolder persisted loader_path (and the user_metadata filename that preview URLs split for their subfolder param) as just the basename: the served locator pointed at a file that does not exist at that path. Scanner and seeder specs already derive fname via compute_loader_path; use the same derivation here. * fix(assets): only extension-matching buckets contribute a loader_path The model-base match in get_asset_category_and_relative_path ignored each bucket's extension set, so a file inside a registered base whose extension the bucket cannot load (e.g. a .txt uploaded into model_type:checkpoints) advertised a loader_path that no loader list would ever resolve, while the tag side of the same stack already excluded it. Apply the extension check used for backend tags (empty set accepts any extension), keeping loader_path null exactly when no loader can resolve the file. * fix(assets): refresh loader_path when re-ingesting an existing reference upsert_reference only wrote loader_path on the INSERT branch, so re-ingesting an existing reference (an output overwritten in place, or a file re-registered after its loader_path derivation changed) kept the stale or NULL value forever. Write it on the UPDATE branch too, with a null-safe change guard so a loader_path difference alone is enough to trigger the update, and identical values stay a no-op. * fix(assets): repair semantic merge breakage from #14796 and master Two textually-clean but semantically-broken merges: - routes.py lost its folder_paths import when #14796's import block superseded the base's, while the content-type hardening added via the base's master merge still calls folder_paths.is_dangerous_content_type. - master's SVG download-hardening test uploads with the pre-namespacing bare checkpoints tag, which this branch's destination validation rejects; use model_type:checkpoints. --------- Co-authored-by: guill <jacob.e.segal@gmail.com> | 2 个月前 | |
feat(assets): add namespaced model_type tags and align tag semantics (#14511) * feat(assets): add namespaced model type tags * fix(assets): mark path-derived upload tags automatic * fix(assets): merge duplicate scan specs * test(assets): make duplicate path normalization portable * feat(assets): add loader_path as the authoritative loader locator (#14796) * fix(assets): filter model_type tags by bucket extension sets Buckets sharing a base directory (e.g. diffusion_models and a custom unet_gguf) tagged every file in the directory regardless of whether the bucket could load it, so .safetensors files were tagged model_type:unet_gguf and vice versa. Carry each bucket's registered extension set through get_comfy_models_folders and only emit a model_type tag when the file extension matches, keeping the empty-set match-all convention from folder_paths.filter_files_extensions. Files under a model base matching no bucket now keep only the models tag instead of every directory-matching model_type tag. * feat(assets): replace response file_path with persisted loader_path The old file_path response field was a namespaced storage locator (models/checkpoints/foo.safetensors): not an absolute path, not unique identity, and not the value a loader consumes. Nothing needs that shape on the wire (hash/ID-based locating is the long-term direction), so it is dropped rather than renamed; the storage-root matching stays internal, powering display_name. What loaders DO need is the in-root loader path (category dropped: models/checkpoints/foo/bar.safetensors -> foo/bar.safetensors). Serve it as a first-class loader_path field, persisted on asset_references (migration 0006) and written by every ingest pipeline at insert, so responses read the column verbatim. Like the model_type tags, loader_path is a seed-time derivative of the model folder registry, maintained by the same scan lifecycle (new files seed fresh values, pruning retires rows whose bucket disappeared). Rows predating the column serve a null loader_path; databases from before this stack already need recreating for the base branch's tag changes. loader_path resolves every registered base including extra_model_paths entries; display_name only the canonical storage roots. A file can therefore be loadable with no display name (extra-path models) or the reverse (unregistered files under the models root), and loader_path is null exactly when no loader can resolve the file. * test(assets): lock loader_path matrix (asymmetry, null, persist/read) Cover the behaviour that has no production change but is easy to regress: the extra-path asymmetry (loadable but no storage namespace), null loader_path persistence for orphan files, and the response reading the stored column with a compute fallback for un-backfilled rows. * fix(assets): persist subfolder-qualified loader_path for ingested outputs ingest_existing_file built its seed spec with the file's basename, so outputs saved into a subfolder persisted loader_path (and the user_metadata filename that preview URLs split for their subfolder param) as just the basename: the served locator pointed at a file that does not exist at that path. Scanner and seeder specs already derive fname via compute_loader_path; use the same derivation here. * fix(assets): only extension-matching buckets contribute a loader_path The model-base match in get_asset_category_and_relative_path ignored each bucket's extension set, so a file inside a registered base whose extension the bucket cannot load (e.g. a .txt uploaded into model_type:checkpoints) advertised a loader_path that no loader list would ever resolve, while the tag side of the same stack already excluded it. Apply the extension check used for backend tags (empty set accepts any extension), keeping loader_path null exactly when no loader can resolve the file. * fix(assets): refresh loader_path when re-ingesting an existing reference upsert_reference only wrote loader_path on the INSERT branch, so re-ingesting an existing reference (an output overwritten in place, or a file re-registered after its loader_path derivation changed) kept the stale or NULL value forever. Write it on the UPDATE branch too, with a null-safe change guard so a loader_path difference alone is enough to trigger the update, and identical values stay a no-op. * fix(assets): repair semantic merge breakage from #14796 and master Two textually-clean but semantically-broken merges: - routes.py lost its folder_paths import when #14796's import block superseded the base's, while the content-type hardening added via the base's master merge still calls folder_paths.is_dangerous_content_type. - master's SVG download-hardening test uploads with the pre-namespacing bare checkpoints tag, which this branch's destination validation rejects; use model_type:checkpoints. --------- Co-authored-by: guill <jacob.e.segal@gmail.com> | 2 个月前 | |
feat(assets): split asset records from content (#16295) * review-stack 1/4: code (37 files, +3217/-3958) Review-and-land stack for synap5e/feat/asset-record-content-split, generated by review-stack.py. Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: path not under tests-unit/ or tests/ Question: Is the logic change right? Source tip: 7007d185825751a09f26bf7ba79f146dc2fbda74 Merge-base: 783545f689a0af730065994b46b382ae24844c99 * review-stack 2/4: tests-removed (24 files, +274/-8220) Review-and-land stack for synap5e/feat/asset-record-content-split, generated by review-stack.py. Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: test file deleted, or modified with deleted/(added+deleted) >= 0.9 Question: For each dropped assertion: obsolete by a ruling, or covered by a tests-new test? Source tip: 7007d185825751a09f26bf7ba79f146dc2fbda74 Merge-base: 783545f689a0af730065994b46b382ae24844c99 * review-stack 3/4: tests-changed (13 files, +1043/-1218) Review-and-land stack for synap5e/feat/asset-record-content-split, generated by review-stack.py. Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: remaining modified test files (incl. conftest.py / helpers) Question: Did the edits weaken an existing check? Source tip: 7007d185825751a09f26bf7ba79f146dc2fbda74 Merge-base: 783545f689a0af730065994b46b382ae24844c99 * review-stack 4/4: tests-new (46 files, +8601/-0) Review-and-land stack for synap5e/feat/asset-record-content-split, generated by review-stack.py. Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: test file added Question: Is the code layer well covered? Source tip: 7007d185825751a09f26bf7ba79f146dc2fbda74 Merge-base: 783545f689a0af730065994b46b382ae24844c99 * review-stack 5/6: code (13 files, +351/-104) Review-and-land stack for synap5e/feat/assets-di, generated by review-stack.py. Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: path not under tests-unit/ or tests/ Question: Is the logic change right? Source tip: eca2c74bffcbb4481e90f33d5b7b4efeeb05eb45 Merge-base: 20d59d2a5f8108d714a908b73a9c1754b9c36f2a * review-stack 6/6: tests (8 files, +753/-238) Review-and-land stack for synap5e/feat/assets-di, generated by review-stack.py. Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: every changed file under tests-unit/ or tests/ (added, modified, or deleted) Question: Is the code layer well covered, and did any edit weaken an existing check? Source tip: eca2c74bffcbb4481e90f33d5b7b4efeeb05eb45 Merge-base: 20d59d2a5f8108d714a908b73a9c1754b9c36f2a * review-stack 7/8: ported-fixes (42 files, +1361/-180) Review-and-land stack for synap5e/feat/assets-di-v2, generated by review-stack.py conventions (hand-built continuation layer; see the PR body). Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: the 11 base-branch fix/docs commits 595cd6e4..94d7185b cherry-picked across the DI refactor (7efdd1d7 excluded, superseded by layer 8) Question: was each base fix ported faithfully across the DI refactor? Source tip: 6841881069284803b902b4a9e33bdcda13126771 Merge-base: 7fdfb40f4b1d04c5b4b26ac7ab2328c83a8cb126 * review-stack 8/8: defensive-parity (4 files, +36/-3) Review-and-land stack for synap5e/feat/assets-di-v2, generated by review-stack.py conventions (hand-built continuation layer; see the PR body). Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: match-or-improve master's dependency defenses — NoAssets selection when DB deps unavailable (7efdd1d7's outcome via the DI seam), requirements warning before assets imports, blake3 in the guarded dependency set Question: does each degradation path now match or improve master's behavior? Source tip: ebc2cfeebc5a1ebae407d0cb975afb5f293b4111 Merge-base: 7fdfb40f4b1d04c5b4b26ac7ab2328c83a8cb126 * fix(assets): only discard content rows this operation actually inserted CR-9: Enumerated all six create_content call sites. Only scanner seeding and the three ingest registration paths track IDs for failure cleanup. * fix(assets): reject hash-only uploads with FEATURE_DISABLED when hashing is off CodeRabbit finding CR-2: reject hash-only multipart uploads before create_from_hash when hashing is disabled. * fix(assets): seed persists the stat it verified CR-7: persist the fresh seed-time restat instead of walk-time spec values. * fix(assets): route database lock failures to the lock guidance CR-16: route file-lock startup failures through the existing lock guidance and exit path. * fix(assets): drop the inaccurate temp-cleanup claim from the shutdown warning References CR-10. * fix(assets): walk the output root after execution so undeclared outputs register promptly Custom nodes that write files into the output directory without declaring them in output_ui only became assets when the next full walk happened - a frontend GET /object_info or a restart. Headless and API-only sessions never trigger either, so those files never converged into the asset database. The post-execution hook now requests a FULL scan of the output root instead of an enrich-only pass. The seeder's pending-request queue was generalised from enrich-specific to carrying a scan phase, so the request starts immediately when the seeder is idle and coalesces (escalating to FULL on a phase mismatch) when a scan is already running. queue_output_enrichment is renamed to queue_output_scan across the protocol, the NoAssets no-op and the call site. References FIX-6. * chore(assets): remove seeder paths orphaned by the output-scan change 45c2f96e rerouted both former enrich call sites to start()/enqueue_scan(), leaving two seeder methods that look live but are not. Review round F2 raised this along with four smaller items; the user's disposition was to fix all six here. - Delete start_enrich: zero callers repo-wide after 45c2f96e. - Delete enqueue_enrich: no production callers; its ~18 call sites in tests/test_asset_seeder.py move to enqueue_scan(phase=ScanPhase.ENRICH) with their semantics unchanged. The deletion forces the half-done class renames (TestEnqueueEnrich* -> TestEnqueueScan*, consistent with the already-renamed TestPendingScanDrain) and restores the module docstring that was dropped rather than reworded. - Document at manager.queue_output_scan that ScanPhase.FULL per debounce window is the deliberate, user-ratified trade, so it is not optimised back to ENRICH without revisiting the decision. - Document that SeedAssetSpec.size_bytes/mtime_ns are walk-time diagnostics only - production persists the seed-time restat since CR-7. - Export create_content_reporting_insert from the queries facade and fold scanner.py's direct-module import into the existing facade block. - Harden test_queue_output_scan_does_not_duplicate_declared_output against a vacuous pass: it now asserts the seeder finished without errors and that an undeclared sibling written into the same directory WAS registered by the same scan, proving the walk actually ran. No production behaviour changes beyond the two deletions. References F2-cleanup. * chore: comment cleanup Comment-Gate: 18 quarantined * fix(assets): preserve pause across the seeder's pending-scan drain pause() runs before every prompt, while pending-scan enqueue and resume only run inside the debounced gc-interval gate. If the active scan finishes just after the next prompt's pause, its finally block resets the seeder to idle and the pending drain starts a replacement with the run gate open, so resume becomes a no-op. Capture pausedness under the lock before resetting to idle, then start the drained scan already paused. Setting the state and gate before launching the thread avoids the start-then-reclear window and lets resume release the existing scan checkpoints. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * test(assets): pin job_id absence for scan-discovered assets Owner ruling, recorded 2026-09-03 in the stack-9-hardening planning notepad: scan-discovered assets — including undeclared outputs found by the post-execution walk — carry job_id = None, always; only emission-time registration (output_ui declaration) attributes a job; attributing walk finds to the most recent prompt would be a temporal-correlation guess that is wrong exactly when prompts interleave; None is honest provenance. Do NOT add proximity-based attribution heuristics to the scanner. Ratified against Jacob Segal's cross-job-attribution concern (2026-09-08 review meeting) — a wrongly-attributed asset could mean one user's cloud job sees another user's asset. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * [review-stack 10/10] assets-tests (#16218) * test(execution): run the battery with assets enabled and assert asset-system health at teardown * test(execution): cover list-shaped outputs registering assets Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> --------- Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * [review-stack 11/11] review-fixes (#16261) * fix(assets): only exit on database file-lock timeout when assets are enabled * test(assets): pin live_contents_under_prefixes path-filtering semantics * perf(assets): push live-content prefix filtering into SQL * test(assets): declare per-entry intent in the path-prefix corpus * test(assets): normalize POSIX-literal path expectations for Windows * test(assets): force observable stat changes and close-before-mutate on Windows-sensitive rewrites * test(assets): force an observable mtime change in the hash-mode split test --------- Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> Co-authored-by: guill <jacob.e.segal@gmail.com> | 21 天前 | |
fix(assets): harden the asset catalogue's failure paths (#16393) * chore(assets): drop the unused asset_meta table from migration 0007 * fix(assets): drop asset_meta when downgrading a database that already created it * docs(assets): clarify asset schema docstring * test(assets): assert alembic and ORM index parity for the surviving asset tables * test(assets): cover asset system state index parity * refactor(tests): hoist migration-0007 test imports to module scope * fix(assets): guard the hashing dependency and chain the real import error * fix(assets): always resume background scanning when prompt handling fails * fix(api): derive the assets feature flag from the selected manager * fix(assets): paginate enrichment by id cursor so failures cannot starve or overflow the query * refactor(assets): extract prompt_worker so its resume contract is testable in-process * refactor(assets): test the blake3 import guard in-process instead of via subprocess * fix(assets): advance the enrichment cursor only past rows the batch attempted * fix(assets): track the scan pause across prompt worker iterations * chore(assets): address review follow-ups in the hashing guard, feature flags, and pagination pin * docs(assets): describe enrichment rows as attempted rather than selected The cursor holds at the last row a batch actually attempted, so a pause ends a batch early and the rows behind it are selected again when the scan resumes. Only the attempt is capped at once per pass. * test(api): derive the expected assets flag from the manager under test The assertion asked for the flag with no argument, so it read the parameter default rather than anything the manager reported - in a test whose subject is the two agreeing. Passing the manager's own state keeps it honest if the setup ever yields an enabled manager. * test(assets): let a broken prompt worker import fail instead of skipping The fixture wrapped importlib.import_module in a bare except that called pytest.skip, so a circular import, a missing dependency or a syntax error in app/prompt_worker.py would retire all four resume-contract tests while CI stayed green. The whole premise of extracting the module is that main.py can import it, so an import failure has to be a collection error. The CPU guard is genuinely load-bearing — comfy.model_management selects its device at import time and a CUDA build with no driver raises there — so it is kept, but as a precondition rather than an exception handler, matching the args.cpu-before-import convention already used by the comfy_test and comfy_api_nodes_test modules. Nothing is caught now. * docs(assets): name the unattempted rows instead of the ones behind the cursor The cursor moves forward through ascending ids, so 'the rows behind it' reads as the rows already passed - the opposite of what is selected again. * fix(assets): only absorb duplicate-path races when seeding scanned assets * fix(db): copy the legacy database inside the process lock * fix(assets): drop watch-list entries on stat errors instead of aborting the scan * fix(assets): clean up temp uploads on validation failures Multipart parsing writes the uploaded bytes to a temporary file before it validates the remaining form fields, so a request rejected after its file part had already been read left the temp file and its uuid directory on disk. Routing those removals through delete_temp_file_if_exists also changes the success path. The previous helper returned early when the temp file was already gone, so it never reached the parent rmdir; the shared helper attempts the rmdir unconditionally. Moving the upload to its destination leaves the temp path absent, so a successful upload now also discards its empty uuid directory, closing a pre-existing leak. * fix(assets): keep updated_at stable on no-op renames * docs(assets): make module docstrings and the rebuild warning truthful * fix(assets): correct event-log status snapshots and failure telemetry * test(assets): make the keyset tie-breaker and temp-exclusion tests falsifiable * chore: comment cleanup Comment-Gate: 3 quarantined * fix(assets): keep unreadable filesystem metadata from failing a whole scan batch * fix(assets): remove temporary uploads on non-UploadError failures * fix(assets): preserve successful specs when a scan batch fault propagates * test(db): drop the inert legacy-copy patch from the path preparation tests prepare_file_db_path no longer copies the legacy database - that moved inside the process lock in _init_file_db - so patching copy_legacy_default_db here did nothing. Leaving it implied a side effect the function does not have, and would have masked one if it were reintroduced. * fix(assets): report specs committed before a batch fault and preserve the fault itself * fix(assets): distinguish partial batch insert failures * refactor(assets): collapse the duplicated batch fault deferral into seed_asset_specs * fix(assets): reject duplicate file parts instead of stranding the first upload * refactor(assets): reap empty upload directories without importing the API layer * fix(assets): emit the invalid-mtime event once per scan Every other per-file emit on this scan path is gated -- mark_emitted( "stat_failed:enrich"), "hash_discarded_modified", "hash_failed", "enrich_failed" -- but scanner.invalid_mtime fired per file, so a restored archive or a FAT volume of pre-epoch mtimes put one structured event per file into the stream the closed vocabulary exists to keep parseable. Counted and emitted once, carrying the count. seed_asset_specs receives no _ScanProgress object and neither does insert_asset_specs above it, so routing this through mark_emitted would mean changing both signatures plus the seeder call site; the count form needs neither and the event is now strictly more informative than N identical fieldless lines. The per-file logging.warning is unchanged, and the emit stays inside seed_asset_specs so the static call-site manifest still matches. test_seed_skips_negative_fresh_mtime_with_warning_and_telemetry now pins the full list of invalid_mtime lines to exactly ["... count=1"] instead of asserting one such line exists -- a strictly stronger assertion, and the only change the new field required. * fix(assets): keep spec construction failures from wedging the watch list get_name_and_tags_from_asset_path raises ValueError by contract when a path stops resolving to a configured root, and it sat outside the guard, as did compute_loader_path and mimetypes.guess_type. An escape skipped the _WATCH_LIST[:] = remaining write at the end, so drained entries stayed on the list and were re-attempted every tick while entries past the fault never reached the increment _WATCH_SCAN_RETRIES needs to retire them. The list wedged permanently. Spec construction is now inside a guard that drops just the offending entry, and the list write moved into a finally so no future escape can skip it. The loop walks an iterator rather than the list, so the finally can put back the entries it never reached instead of discarding them. New event name rather than reusing one: scanner.watch_seed_failed is emitted only when seed_asset_specs returns an error, and widening it to also mean "never got as far as seeding" would make it lie -- a consumer treating it as a database-health signal would get false positives from what is really a path layout problem. scanner.watch_spec_failed is registered in ALLOWED_EVENTS and in the static call-site manifest. * refactor(assets): export the live-path conflict check as public API scanner.py reached past the package's own re-export surface to import _is_live_path_conflict directly out of records.py. The underscore said module-private while the import said otherwise, and records.py deliberately publishes its public names through app.assets.database.queries -- which the same import block three lines above was already using. The use is correct and unchanged; only the name and the route change. Renamed to is_live_path_conflict, listed in the package __init__ import and __all__ alongside its siblings, and scanner.py now takes it from the package like everything else it imports from there. * docs(db): restore the rationale for locking before migration Commit 1dbcdcd7 and the comment-cleanup pass 8205022f reduced this to "All database reads and writes, including the legacy import, run under the lock", dropping the part that did the work: upstream master locks after migrating and justifies it with "Alembic uses its own connection, so we must wait until it's done before locking -- otherwise our own lock blocks the migration". That is false, the lock is on a separate <db>.lock file, and the surviving sentence said nothing to stop a contributor "fixing" the ordering back. Restored and adapted rather than pasted: the legacy copy and the db_exists probe now happen inside the lock, which the original text predates, so both are named in the list of things the ordering makes mutually exclusive. * fix(assets): stop a scan on memory exhaustion instead of deferring it MemoryError is an Exception, so the per-spec and per-batch handlers stored it alongside ordinary faults and carried on - allocating for every remaining spec and then every remaining batch while the process was already out of memory. Both handlers now let it through, and the scan records a failure and stops. * docs(assets): document the prune failure response and its None result The route gained a 500 PRUNE_FAILED branch and the seeder method gained a None return, both so a prune that did not run cannot be reported as a clean one. Neither contract was written down. * test(assets): assert the surviving spec count after a propagated fault * docs(db): shorten the lock-ordering comment while keeping its rationale * docs(tests): drop the cross-module justification from the import-order comment * test(assets): restore the cpu flag after the guarded prompt worker import * test(assets): restore the cpu flag even when the prompt worker import fails * fix(assets): drop asset_meta in a new migration instead of editing 0007 0007 shipped in v0.36.0, v0.37.0 and v0.37.1, so editing it would leave two installs at that revision with different schemas depending on when they upgraded. Restore 0007 to its released form and drop the unused asset_meta table in 0008 instead. Nothing reads or writes asset_meta; asset metadata lives in the JSON columns on assets. 0008 downgrades by recreating the table and its four indexes exactly as 0007 creates them. * refactor(assets): keep prompt_worker in main.py Custom nodes may reference main.prompt_worker, and tests can already import main (test_db_init_locking does), so the function stays where it was. Its body keeps the resume-on-failure handling and the pause flag that persists across loop iterations; the tests now call main.prompt_worker. * fix(assets): report the selected manager through the existing feature flags * fix(assets): record a failed prune in the scan status A failed prune left the scan's error list empty, so the run looked clean. Record it instead and let discovery continue as before. * test(assets): test the feature flag API directly instead of copying the startup write Both parity tests wrote SERVER_FEATURE_FLAGS["assets"] themselves, so they passed whatever startup did. Pin the one real claim - a no-argument get_server_features() reports the flag - in the feature flags tests, and drop the disabled case, which default_asset_manager already covers. * Bring #16486's watch-list batching and seed logging into this branch The merge before this commit is `git merge -X ours origin/master`: in conflicting hunks it keeps this branch's side. This commit ports what #16486 changed in those hunks. - tick_watch_list takes no session. It collects settled entries and seeds them in one batch through insert_asset_specs, keeping this branch's per-entry stat and spec handling and the finally that always rewrites the watch list. A failed seed is reported once per batch, since the batch only returns its first error. - seed_asset_specs no longer warns again for a skipped spec; observe_asset_specs already logged why. - Tests call tick_watch_list() without a session, bind the write session where they seed for real, and fake insert_asset_specs with its (created, error) return. The mid-drain fault now comes from stat, because seeding runs after the loop. * Note where the assets core flag is set and why prompt_worker catches BaseException --------- Co-authored-by: guill <jacob.e.segal@gmail.com> | 9 天前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 21 天前 | ||
| 21 天前 | ||
| 6 个月前 | ||
| 3 个月前 | ||
| 2 个月前 | ||
| 2 个月前 | ||
| 21 天前 | ||
| 9 天前 |