已关闭
[SESSION-02] Implement YDB canonical session schema and bounded indexes #21
urandon创建于  8月3日关闭于  8月5日
urandon
urandon成员
8月3日 创建

Parent and architecture

Outcome

Implement the authoritative YDB schema and Object Storage manifest layout for canonical sessions, ordered events, bindings, participants and snapshots using bounded tenant-scoped access paths.

Estimate

  • 8 SP / 5 engineering days
  • Risk: high

Scope

  • Add migrations for:
    • sessions;
    • session_events;
    • frontend_bindings;
    • session_participants;
    • session_snapshots;
    • session activity index rows;
    • runs-by-session index rows.
  • Add session_id and trigger_event_id to runs and related atomic procedures.
  • Remove the pre-production conversations + context_epochs schema from the canonical baseline. Do not build a row-level backfill, dual-read, shadow-write or compatibility adapter for existing local/cloud-dev records.
  • Implement atomic sequence allocation, event append, session last-activity update and index maintenance.
  • Implement point/bounded-prefix reads for:
    • session by ID;
    • frontend binding;
    • ordered event page;
    • snapshot versions;
    • runs by session;
    • active/archived sessions by owner/recent activity.
  • Use fixed fan-out owner/status activity buckets derived from random session IDs; do not use a global chronological primary key.
  • Define session-scoped Object Storage keys and manifest validation for event payloads, attachments and compressed immutable snapshots.
  • Keep canonical session/event metadata free from TTL.
  • Update:
    • docs/ydb-state-store.md;
    • docs/ydb-partitioning.md;
    • migration/recovery documentation.
  • Add clean-schema, repeat-migration, tenant-isolation, concurrency and physical-partition contract tests.

Pre-production schema reset policy

There is no production conversation data and existing local/cloud-dev application data is disposable. Before the first production deployment:

  • the repository may squash or rewrite the application migration baseline to match the canonical session model;
  • local and cloud-dev application tables, migration ledger rows and Sessionless-owned dev Object Storage prefixes may be reset instead of migrated record by record;
  • the reset must preserve Terraform/bootstrap state, cloud resources outside the application schema, Lockbox secrets, IAM configuration and unrelated bucket prefixes;
  • a cloud-dev reset requires explicit environment identity checks (folder/database/bucket), an allowlisted non-production target and typed confirmation;
  • no reset command may accept an unresolved environment, broad bucket/database target or production identifier;
  • CI starts from an empty YDB/Object Storage fixture and proves the new head from zero;
  • once the first production baseline is deployed, applied migrations become immutable and normal expand/migrate/contract rules resume.

The migration runner still owns checksums, locking and crash behavior. Disposable data removes compatibility work; it does not permit an unguarded destructive command.

Non-goals

  • Telegram ingestion behavior.
  • Worker context materialization.
  • Lifecycle transition to COLD/ICE.
  • Search indexes or semantic search.

Verification

  • Concurrent event appends produce a strict gap-free sequence without duplicate event IDs.
  • Two tenants cannot resolve each other's sessions, bindings, events, snapshots or indexes.
  • Session/activity/event queries are point or fixed-fan-out prefix operations.
  • Activity-index tests prove bounded fan-out and descending recent-session pagination.
  • Operational TTL cleanup fixtures leave canonical rows untouched.
  • A clean YDB Local instance can migrate repeatedly from zero to the new head.
  • A guarded local/cloud-dev reset removes obsolete application tables, ledger state and owned fixture objects, then applies the canonical baseline successfully.
  • Negative tests prove the reset refuses production/unknown targets and cannot address Terraform state or unrelated Object Storage prefixes.

Acceptance criteria

  • Every canonical table and materialized index documents its primary key, partitioning rationale and query contract.
  • No global time-ordered hot key is introduced.
  • Archived metadata is retained without TTL.
  • Runs reference canonical session and trigger event identities.
  • Applied migrations, checksums and crash-recovery behavior remain compatible with the existing migration runner.
  • No production data-preservation machinery is added for accidental pre-production structures.
  • The canonical migration baseline contains no legacy conversation/context-epoch compatibility path.
  • The destructive reset path is explicit, non-production-only, narrowly scoped and fail-closed.
likedislike
urandonurandon成员
8月4日 修改了issue 的描述
urandonurandon成员
8月4日 关联了pull request:SESSION-01: define canonical session and event contracts
urandonurandon成员
8月5日 关联了pull request:SESSION-02: persist the canonical Session model
urandon
urandon成员
8月5日 评论:

Implementation update — canonical Session persistence

Implementation is published in MR !20, branch ai/session-02-canonical-schema, commit 470f4b6.

Implemented

  • Rebased the disposable pre-production migration baseline: legacy conversations and context_epochs tables are replaced by canonical sessions and append-only session_events.
  • Added tenant-first YDB tables for event idempotency, frontend bindings and their lookup keys, participants, fixed-fanout session activity, snapshots, and runs-by-session.
  • Extended runs with session_id and trigger_event_id; the YDB store maintains the session index atomically.
  • Implemented serializable, gap-free event sequence allocation with exact retry semantics. The live concurrency test proves that parallel appenders produce one contiguous sequence and that retrying an idempotency key returns the original event.
  • Implemented session lifecycle, binding creation/switching, participant authorization, bounded history/snapshot reads, activity listing, and per-session run listing with tenant mismatch rejection.
  • Updated Telegram identity resolution so Telegram remains a frontend adapter: the initial interaction creates a canonical Session, and /new creates another Session and atomically switches the binding.
  • Added canonical Object Storage key contracts under tenants/{tenant_id}/sessions/{session_id}/...; cross-session and cross-tenant references fail closed.
  • Kept YDB automatic partition management on canonical hot tables. The logical contract fixes tenant-first keys and 16 deterministic activity buckets, but does not assert or manage an exact physical partition count.
  • Added preprod-reset, make cloud-app-reset-plan, and make cloud-app-reset. The destructive path requires the exact Terraform-resolved cloud-dev folder, YDB endpoint, bucket, tenants/ prefix, and a target-derived typed confirmation. It drops only the explicit Sessionless table allowlist and preserves infrastructure and unrelated object prefixes.
  • Updated the public architecture/development documentation to describe Sessionless-owned canonical conversations and the intentional cloud-dev reset/rebase procedure.

Verified

  • make ci passed: generation, vet, race tests, all ten binaries, and foundation integration tests.
  • A clean YDB Local migration and a repeated migration both passed.
  • make ydb-integration passed, including concurrent append, idempotency, bindings, participant/activity reads, snapshots, runs-by-session, tenant isolation, and partition contracts.
  • make local-integration passed after a clean local application-data reset.
  • make e2e-local passed all deterministic multi-user scenarios, including the /new binding transition.
  • git diff --check and shell syntax validation passed.

No production or irreplaceable data exists yet, so this MR intentionally performs a baseline rebase instead of adding compatibility backfill or dual-read paths. The cloud-dev reset remains an explicit operator action; merging or deploying the application does not trigger it automatically.

likedislike
urandon
urandon成员
8月5日 评论:

Completion report

MR !20 has been merged into main as commit 76804cd.

The final implementation includes the canonical append-only Session schema, bounded tenant-first indexes, atomic frontend binding switches, Session-scoped Object Storage contracts, the guarded pre-production reset, and both review follow-ups:

  • bindings cannot be switched to an archived Session;
  • CreateSession idempotency requires an exact owner match and rejects conflicting owners.

Verification after merge:

  • GitCode push mirror completed successfully and GitHub main resolves to the exact merge SHA 76804cd;
  • GitHub Actions CI #62 passed all five jobs: Go verification, YDB schema/state store, local multi-service stand, Terraform environments, and runtime images;
  • the YDB job applied the schema twice and ran the live Session invariant/concurrency suite.

All acceptance criteria for #21 are satisfied.

likedislike
urandonurandon成员
8月5日 关闭了 issue
urandonurandon成员
21 天前 添加了label:mvpsessionydb
urandonurandon成员
21 天前 关联了里程碑:MVP — Core platform (#6)