已关闭
[MVP-07] Implement the isolated deterministic worker lifecycle #10
urandon创建于  7月28日关闭于  7月31日
urandon
urandon成员
7月28日 创建

Parent and architecture

  • Implementation epic: #6
  • Architecture: #1
  • Depends on: #4, #5, #7, #9

Outcome

Implement and verify the complete isolated worker contract using a deterministic fake harness before introducing Codex authentication/protocol variability.

Estimate

  • 5 SP / 4 engineering days
  • Risk: medium

Scope

  • Consume a versioned queue envelope containing run_id only.
  • Claim the YDB run lease and quota reservation before materializing tenant data.
  • Create a unique validated per-invocation work directory.
  • Fetch the context manifest, allowed files, workspace snapshot and skill bundle.
  • Execute a deterministic harness scenario with configurable turns, checkpoints, artifacts, cancellation and failures.
  • Persist progress/checkpoints/usage after each simulated turn/tool boundary.
  • Upload immutable artifacts/workspace snapshot and atomically commit result + delivery outbox.
  • Handle duplicate delivery, lease loss, timeout, cancellation and termination signal.
  • Terminate child processes and wipe tenant scratch/credential material before success/failure return.

Isolation verification

  • Attempt path traversal and cross-tenant object references.
  • Process two tenants sequentially in a reused local worker container and prove cleanup.
  • Inject duplicate queue delivery and prove one logical execution/charge.
  • Inject termination after checkpoints and prove safe resume or bounded retry.

Non-goals

  • Codex App Server or real provider authentication.
  • General-purpose sandbox escape protection beyond the defined container/tool policy; deeper sandboxing is a follow-up hardening track.

Acceptance criteria

  • Worker contract runs end-to-end against local adapters with no provider credential.
  • One invocation handles one tenant/run and concurrency is effectively one.
  • Canonical job/context state survives worker loss.
  • Artifacts are content-addressed and tenant-prefix validated.
  • Cleanup and retry behavior are covered by black-box tests.
likedislike
urandonurandon成员
7月30日 关联了pull request:MVP-07: implement the isolated deterministic worker lifecycle
urandon
urandon成员
7月30日 评论:

Implementation update — isolated deterministic worker lifecycle

Implementation is published in MR !14, branch ai/mvp-07-worker-lifecycle, commit ccf76cc.

Implemented

  • Scheduler admission now atomically creates a point-addressable worker_jobs(tenant_id, run_id) descriptor. The queue contains only tenant_id and run_id; context, files, workspace/skill references, delivery target, reservation, and admitted limits stay in YDB/Object Storage.
  • worker-runtime is now a one-shot, concurrency-one process over the harness-neutral ports. It uses the credential-free deterministic harness for this slice; no permanent Codex/OpenCode/Claude/Hermes choice is encoded.
  • Each invocation validates a held reservation, claims a fenced lease, transitions the run/attempt, and materializes tenant-scoped blobs into a unique 0700 scratch directory.
  • Materialization rejects traversal, symlinks, cross-tenant references, size drift, and SHA-256 drift. Scratch is removed after every outcome.
  • Deterministic turn boundaries renew the lease when required and atomically persist immutable checkpoint metadata plus observed token usage under the active fence.
  • Retryable failures retain the latest checkpoint and resume from the next sequence. Expired stable lease IDs are reclaimed with a new fence rather than reusing stale ownership.
  • Admitted runtime, turn, input/context, artifact, and per-blob bounds are enforced inside the worker.
  • Success uploads content-addressed artifacts and atomically commits run/attempt success, quota commit, manifest, Telegram delivery, scheduler counters, audit, and lease-index cleanup.
  • Failure, cancellation, and timeout atomically release quota and create a same-chat terminal Telegram delivery. Terminal queue duplicates acknowledge without another execution or delivery.
  • Added migration 00041_create_worker_jobs.sql with TTL and YDB-controlled size/load auto-partitioning. No manual partition count, split point, or threshold is introduced.
  • Added an explicit Compose worker profile and make worker-once. The default stand leaves the serverless-shaped worker stopped; one command consumes zero or one admitted run and exits.
  • Updated README and public English contract, local-stand, development, YDB state, and physical-partition documentation.

Verification

Passed locally:

make ci
  go generate ./...
  go vet ./...
  go test -race ./...
  build: 8/8 binaries
  go test -race -tags=integration ./test/integration/...

go test -run '^$' -tags=ydbintegration ./test/ydbintegration
git diff --check
sh -n scripts/*.sh
YAML parse: compose.yaml and .github/workflows/ci.yml

Coverage includes two tenants reusing one scratch root, traversal/cross-tenant rejection, checkpoint resume, at-least-once duplicate suppression, durable cancellation, runtime termination, lease renewal, expired-lease fencing, worker-job creation, atomic YDB completion/delivery, and lease-index cleanup.

Docker is unavailable on the development workstation, so no local image, Compose runtime, or live YDB execution claim is made. Issue remains open until commit ccf76cc reaches the GitHub mirror and all mandatory jobs, including live YDB and runtime image jobs, are green.

likedislike
urandon
urandon成员
7月31日 评论:

Completion report — MVP-07

Merged MR !14 delivers the isolated deterministic worker lifecycle. The mandatory mirrored GitHub Actions run is green, and merge commit ea5185f contains implementation commit ccf76cc.

Delivered

  • Durable tenant-scoped worker jobs; queue messages carry only tenant_id and run_id.
  • One-run, concurrency-one worker execution with fenced lease claim/renewal, checkpoints, resume, usage events, cancellation, timeout and signal handling.
  • Invocation-scoped 0700 scratch directories with tenant/path/symlink/size/digest validation and cleanup suitable for reused containers.
  • Product limits for runtime, turns, input/context and artifact sizes.
  • Content-addressed artifacts and atomic success/failure transitions covering run, attempt, quota, manifest, delivery outbox, scheduler counters, audit and lease cleanup.
  • Idempotent terminal handling: duplicate queue delivery acknowledges an already-terminal run without re-execution, duplicate charging or duplicate logical delivery.
  • YDB migration and local worker tooling/documentation, while keeping the runtime harness-neutral.

Verification

  • Local make ci passed before publication.
  • The mirrored GitHub CI completed successfully for the merged revision, including the mandatory YDB and runtime-image coverage.
  • GitCode main contains the implementation through merge commit ea5185f.

The acceptance criteria for MVP-07 are satisfied. Real subscription-backed harness integration remains deliberately outside this issue and is tracked by MVP-10 / #13.

likedislike
urandonurandon成员
7月31日 关闭了 issue
urandonurandon成员
8月26日 添加了label:mvpworker
urandonurandon成员
8月26日 关联了里程碑:MVP — Core platform (#6)