[SESSION-06] Implement archive, storage tiering and audited deletion-sessionless-AtomGit

已关闭

[SESSION-06] Implement archive, storage tiering and audited deletion #25

urandon创建于 8月3日关闭于 24 天前

urandon成员

8月3日创建

Parent and architecture

Implementation epic: #6
Accepted architecture decision: #1
Depends on: SESSION-02 #21, SESSION-04 #23, SESSION-05 #24

Outcome

Implement reversible session archive state, explicit payload tiering and an audited destructive-retention procedure without allowing operational TTL cleanup to erase canonical history.

Estimate

5 SP / 4 engineering days
Risk: high

Scope

Implement idempotent archive and unarchive transitions with activity-index maintenance.
Keep archived session metadata/events/manifests in YDB with no archive TTL.
Define Object Storage lifecycle rules:
- recent payloads in STANDARD;
- eligible inactive immutable payloads in COLD;
- ICE only after an explicit age/economic threshold.
Keep event/snapshot manifests and digests in YDB across storage-class transitions.
Separate four procedures:
- user-visible archive/unarchive;
- ordinary retention expiry;
- tenant/account deletion;
- legal/audit retention hold/release.
Implement an audited deletion state machine that blocks writes, enumerates bounded manifests, deletes objects, removes/tombstones rows according to policy and records completion.
Add dry-run inventory and explicit confirmation for destructive cleanup.
Update retention, recovery and incident runbooks.

Non-goals

Session listing/history API.
Search or summary generation.
Automatically archiving the previous session on /new.
Using YDB row TTL to export payloads.

Verification

Archive/unarchive preserves ordered events, snapshots and artifacts.
Operational TTL tests cannot remove canonical session history.
Storage-class transition fixtures do not change object keys/digests or event ordering.
Deletion dry-run enumerates only the selected tenant/session prefixes.
Interrupted deletion resumes idempotently and cannot cross tenant boundaries.
A legal hold blocks destructive cleanup but not authorized read access.

Acceptance criteria

archived is a logical reversible state, not a retention deadline.
Destructive deletion is explicit, audited and separately authorized.
Object lifecycle policy is cost-aware and documents minimum-retention implications.
No recursive cleanup target can resolve to a broad bucket, root or unresolved variable.
Public documentation distinguishes archive, retention, deletion and audit/legal retention.

urandon成员

24 天前评论：

Implementation started on branch codex/session-06-archive-retention. Chosen safety model: reversible idempotent archive/unarchive stays distinct from destructive deletion; deletion is owner-authorized and persisted as requested -> deleting -> completed, with a durable tombstone and audit events; active legal hold blocks request/start/completion; request establishes a canonical write fence; dry-run builds a bounded deterministic inventory of exact BlobRefs and execution requires a typed inventory digest, deletes only those keys, and is resumable after interruption. Current work also adds session-scoped YDB indexes needed to avoid unbounded scans and Object Storage lifecycle tiering/runbook before opening the MR.

urandon成员

24 天前关联了pull request：SESSION-06: audited archive and retention lifecycle

urandon成员

24 天前评论：

Implementation is ready for review in !32: https://gitcode.com/urandon/sessionless/merge_requests/32 (commit abcbb0b). Delivered idempotent archive/unarchive, STANDARD -> COLD -> ICE lifecycle policy, durable legal holds and deletion tombstones/audit, owner authorization and write fencing, bounded exact-object dry-run with digest confirmation, resumable deletion, operator Make targets, migrations 00060-00064, and the public retention/recovery runbook. Verification passed: make test, make build, make integration, full make ydb-integration, make local-integration, make terraform-ci, clean-schema migration/repeatability, and make e2e-local on a clean stand. An earlier E2E attempt reused YDB-suite fixtures and failed only in the queue-outage recovery scenario; after the required clean-volume isolation, the full black-box suite passed.

urandon成员

24 天前评论：

Дополнение к MR !32 после self-review: commit a1918ea закрывает privacy-gap в legacy Telegram delivery state.

добавлен bounded reverse index telegram_deliveries_by_run (migration 00065), dual-write и immutable run ownership для delivery ID;
inline delivery text удаляется вместе с outbox/ready rows;
delivery BlobRef входит в точный session-scoped inventory и confirmation digest;
completion удаляет legacy/v2 ready rows, outbox и reverse-index rows транзакционно;
интеграционный lifecycle test покрывает inline и blob deliveries и подтверждает отсутствие строк после completion.

Проверки после дополнения: make test, make ydb-integration, make local-integration, make build, git diff --check — зелёные. MR: https://gitcode.com/urandon/sessionless/merge_requests/32

urandon成员

24 天前评论：

Implemented and merged via MR !32.

Final merge commit: cf7cfccb16e5d4bf9617ec267631ce078c9f074e (verified feature head 94480af84ecc30fe919a7555ed686d42fa4fcfe2).

Review cycle addressed all four findings: migration-safe reverse-index backfill/cutover, session-owned worker object paths with bounded legacy compatibility, durable non-TTL delivery/checkpoint ledgers, and a shared bounded deletion inventory.

Verification passed: make test, make build, make ci, clean make ydb-integration, make local-integration, make terraform-ci, clean make e2e-local, and GitHub exact-head Actions run 32137432646 (all six jobs successful).

urandon成员

24 天前关闭了 issue

urandon成员

17 天前添加了label：mvpsession

urandon成员

17 天前关联了里程碑：MVP — Core platform (#6)