已关闭
[AW-03c] Add bounded exact-replay recovery for worker exchanges #120
urandon创建于  26 天前关闭于  26 天前
urandon
urandon成员
26 天前 创建

Parent and dependency

Child of #75 (AW-03) and #72. Builds on the merged worker connection/session and daemon adapter slices #109/#110 and the durable exact-replay support already present in the control-plane exchange transaction.

Outcome

Add bounded, replay-safe recovery for one ambiguous authenticated worker exchange. The worker session must resend the exact already-built AW-02 batch under the same operation ownership, so transient HTTP failure can be reconciled without inventing a new heartbeat, claim, terminal, sequence, acknowledgement, or external effect.

This is the prerequisite for the real timer-poll cadence and foreground composition; it does not enable either one.

Required behavior

  • Retain a deep-owned exact pending batch only for the duration of the current in-process operation and retry only sanitized transport errors explicitly classified retryable.
  • Every retry sends byte-equivalent protocol content with identical message ID, worker/platform sequences, acknowledgement, binding, attempt/fence, and evidence.
  • Validate the first observed response against the pre-effect conformance snapshot, then advance local machine state exactly once.
  • Use locally bounded exponential full jitter beneath the existing total operation timeout. Ignore remote Retry-After; do not busy-loop or catch up after sleep.
  • Caller cancellation stops scheduling new retries. If a transport dependency ignores cancellation, retain the single operation owner until it returns; no replacement exchange may overtake it.
  • Unauthorized remains fenced. Protocol/conflict/divergent replay is terminal. Exhausted or timed-out reconciliation remains explicit reconciliation_required and never becomes success/offline.
  • Clear retained protocol payloads on completion/failure where Go permits and expose no bearer, endpoint, payload, or raw network error through snapshots/formatting.

Acceptance

  • Deterministic tests prove lost-response-then-exact-replay for heartbeat, lease claim, cancel acknowledgement, and terminal; no action is rebuilt between attempts.
  • Tests prove divergent replay response, stale generation, unauthorized, non-retryable protocol/conflict, exhaustion, caller cancellation, and cancellation-ignoring dependency behavior fail closed.
  • Tests use injected waits/entropy and channel synchronization, with no correctness sleeps.
  • Focused uncached, race, repeated/shuffled package tests, vet, full repository validation, independent review, and exact-head CI pass.

Non-goals and authorization

No production route, foreground activation, daemon/process/OCI launch, reconnect/bootstrap, new connection generation, provider call, credential mutation, model download, cloud deployment, or destructive operation is authorized. A later AW-03 child owns timer cadence, sleep/wake/offline observations, cost evidence, and reviewed foreground wiring.

likedislike
urandonurandon成员
26 天前 关联了里程碑:MVP — Attached workers (#72)
urandonurandon成员
26 天前 添加了label:attached-workermvptest
urandonurandon成员
26 天前 关联了pull request:[AW-03c] Add bounded exact-replay recovery for worker exchanges
urandon
urandon成员
26 天前 评论:

Implementation checkpoint published as PR #102 at exact head de72059f41627a2ede68bc05e20f2e259779871c.

  • exact canonical replay covers initial Manifest and semantic Heartbeat, LeaseClaim, CancelAck, and Terminal flows;
  • retries are bounded, full-jittered, sanitized, and disabled by default;
  • exhaustion/cancellation/non-retryable/divergent outcomes fail closed;
  • independent review found one pre-push P1: retry-time lease validation sampled a fresh clock and could leave an ambiguous effect overtaken after expiry;
  • the fix pins the pre-effect acceptance timestamp across every retry and adds a deterministic near-expiry regression; repeat review reports no remaining P1/P2;
  • local make generate, make test, make integration, docs, vet, race, 50x focused, and shuffle gates pass with a clean tree.

Exact-head GitHub CI: https://github.com/urandon/sessionless/actions/runs/34161458520 (in progress at checkpoint time).

likedislike
urandon
urandon成员
26 天前 评论:

Completed in PR #102, merged into main as 94ec38f90dac049da51cdfcbd951220fb2616350.

Evidence:

  • reviewed implementation head: de72059f41627a2ede68bc05e20f2e259779871c; independent final architecture review: CLEAN (no P1/P2);
  • exact PR head after the bounded CI retry: 714743a05cda9226a27d57d62790d5d33d8cd4e2 (code unchanged; final commit only retriggered CI);
  • exact-head GitHub Actions run 34163173541: all seven jobs successful, including YDB, local multi-service stand, and runtime-image reproducibility;
  • local validation: make generate, make test, make integration, make docs-check, focused race/stress/shuffle replay suites, and git diff --check all passed.

Delivered bounded process-local exact replay for sanitized retryable transport unavailability, fixed acceptance time across retries, explicit reconciliation on exhaustion/ambiguity, cancellation-safe bounded backoff, and deterministic tests across manifest, lease, active cancellation, terminal, divergence, fencing, and near-expiry boundaries. Timer cadence, durable restart/reconnect reconciliation, and foreground production activation remain outside this slice.

likedislike
urandonurandon成员
26 天前 关闭了 issue