Implementation progress — protected Terraform bootstrap

The first MVP-09 slice is published on branch ai/mvp-09-yandex-cloud-dev, commit c12e7d3.

Implemented:

• A separate, non-destroyable Terraform bootstrap root under infra/terraform/bootstrap.
• A private, versioned, size-bounded Object Storage state bucket with noncurrent-version retention and prevent_destroy.
• A deletion-protected, throttled YDB Serverless deployment-lock database with a TTL-backed terraform_locks table and prevent_destroy.
• Terraform S3 backend lockfiles plus the separate YDB environment-lease boundary; credentials are injected only through provider/backend environment chains.
• Exact pins for Terraform 1.15.5 and Yandex provider 0.220.0, including the provider dependency lock file.
• Non-secret bootstrap/backend examples and documented state migration/recovery procedures.
• make terraform-ci and a mandatory GitHub Actions Terraform validation job.

Verified:

• terraform fmt -recursive -check -diff infra/terraform
• terraform init -backend=false -input=false
• terraform validate against Yandex provider 0.220.0
• make ci
• git diff --check

This is not the completion claim for #12. The destroyable cloud-dev root, least-privilege IAM, YDB/YMQ/Object Storage/runtime modules, immutable container revisions, API Gateway/canary, deploy/migrate/smoke/rollback procedures, and real cloud contract tests remain on the same branch.

Cloud-dev research decision: trigger-driven runtimes, IAM, secrets, canary, and cost controls

Research was checked against the current Yandex Cloud documentation and Terraform provider 0.220.0 before completing the environment root.

1. Runtime invocation model

Yandex Serverless Containers has two operation modes: an HTTP server and Commands (runtime.type = "task"). Triggers still invoke a container with an HTTP request. In HTTP mode, the application controls the response status; in Commands mode the platform returns HTTP 200 and exposes the process exit code through X-Task-Exit-Code.

For Sessionless, all trigger-facing components will therefore use HTTP mode:

• control-api serves API Gateway requests;
• worker-runtime receives the YMQ trigger event body and returns 2xx only after the idempotent state transition commits;
• reconciler and telegram-sender expose authenticated, bounded one-pass handlers invoked by timer triggers.

This gives the trigger an unambiguous success/failure signal. Local Compose keeps the existing polling/one-shot entrypoints, selected when the serverless HTTP mode is not enabled.

References: container operation modes, runtime environment, container invocation.

2. YMQ ownership and failure semantics

A YMQ trigger, not the worker, receives messages from the queue. It forwards a normalized JSON event, deletes source messages only after successful processing, and restores visibility after an invocation failure. A container that long-polls the same queue would race the trigger and break its acknowledgement contract.

The cloud worker will consequently process trigger-delivered envelopes directly. Initial batch_size is 1 for per-run failure isolation. The queue remains Standard because YMQ container triggers support Standard queues only, allow one trigger per queue, and require the queue and trigger to be in the same cloud. Retry/dead-letter behavior is owned by the queue redrive policy; application state transitions remain idempotent.

References: YMQ trigger contract and event format, Terraform trigger example, trigger retry semantics.

3. Isolation and IAM

The environment gets separate service accounts for deploy, API, scheduler, worker, Telegram delivery, and trigger invocation. Runtime accounts receive only the service roles needed by their component. Trigger invocation permission is attached to the specific target container; folder-wide invoker access is avoided.

The provider exposes an authoritative container IAM binding rather than a member resource. Because Terraform owns these private containers and their complete invoker policy, using that binding is acceptable here; external/manual grants to the same role are prohibited and checked during review.

Reference: container IAM binding warning.

4. Secrets and Terraform state

Terraform creates only Lockbox/KMS metadata and access bindings. Telegram credentials and identity keys are loaded as secret versions by the deployment procedure, outside Terraform; only non-secret version IDs are passed to container revisions. No payload is accepted in .tfvars, plans, command-line arguments, images, or repository files.

Reference: Lockbox delivery to Serverless Containers, Lockbox Terraform resources.

5. API Gateway and controlled promotion

Two private control-api slots are provisioned. API Gateway variables select the stable slot; its native canary block sends an explicit percentage to the candidate slot. Promotion changes variables/weight, not DNS. The custom domain and certificate remain stable through deploy and rollback.

Reference: API Gateway variables/canary/custom domains, custom-domain DNS behavior.

6. Cost guardrail limitation

YDB is configured with zero provisioned RUs plus explicit RU/s and storage ceilings; buckets have size/lifecycle bounds; containers use scale-to-zero defaults and bounded memory/timeouts.

The current Terraform provider does not expose a budget resource (the Billing Terraform reference only exposes cloud binding). Budget creation/verification will be an explicit deployment prerequisite through the Billing Budget API, scoped to the dev folder, and the deploy command will fail closed when the expected budget is absent. A budget is a notification/automation guardrail; reaching it does not itself stop resource consumption.

References: Terraform Billing coverage, Budget Create API, budget behavior.

Resulting event flow

flowchart LR
  TG["Telegram"] --> GW["API Gateway<br/>stable/canary variables"]
  GW --> API["private control-api slot"]
  API --> YDB["YDB Serverless"]
  API --> OBJ["Object Storage"]
  YDB --> REC["timer -> reconciler"]
  REC --> Q["YMQ dispatch queue"]
  Q --> TR["YMQ trigger<br/>batch size 1"]
  TR --> WRK["private worker HTTP handler"]
  WRK --> YDB
  WRK --> OBJ
  YDB --> SEND["timer -> telegram-sender"]
  SEND --> TG
  Q -. redrive .-> DLQ["dispatch DLQ"]

These decisions close the invocation ambiguity before Terraform binds real triggers to the runtime images.

Implementation update — cloud-dev infrastructure

Implementation is published in MR !16, branch ai/mvp-09-yandex-cloud-dev, commit 527bd34.

Implemented

• Reusable Terraform foundation, runtime, and edge modules pinned to Terraform 1.15.5 and Yandex provider 0.220.0.
• An isolated dev folder with resource-scoped runtime IAM, YDB Serverless, tenant-prefixed Object Storage, YMQ dispatch/delivery queues and DLQs, Container Registry retention, KMS/Lockbox metadata, and Cloud Logging.
• Private blue/green control containers behind API Gateway canary variables and a managed dev-api.<domain> certificate/DNS path.
• Timer-driven bounded reconciler/sender containers and a batch-size-one YMQ-triggered, concurrency-one worker container.
• Bounded HTTP trigger entrypoints and a strict Yandex YMQ event adapter. HTTP 2xx delegates acknowledgement to the trigger; non-2xx keeps retry/redrive ownership in YMQ.
• A repository-owned fenced YDB deployment lock. A lost fence cancels the child Terraform command; apply and destroy accept only a reviewed saved plan.
• Folder-first Billing Budget verification through the Billing API, immutable image push, secret payload streaming outside Terraform state/argv, private/public smoke checks, and Telegram webhook setup.
• A complete English runbook for first deployment, migrations, evidence, canary promotion, rollback, external Monitoring alerts, and protected destroy.
• README and CI updates; CI now validates both Terraform roots and builds the control, reconciler, sender, and worker images.

Locally verified

• make ci
• make test after the final trigger parser hardening
• make terraform-ci TERRAFORM=/tmp/sessionless-terraform-bin/terraform
• sh -n scripts/cloud-*.sh
• git diff --check

Both Terraform roots initialized and validated successfully with the pinned provider. Docker is not installed on the development workstation, so no local image build or real Yandex Cloud apply is claimed.

Remaining acceptance gates

Issue #12 remains open until:

1. mirrored GitHub Actions is green for commit 527bd34, including all four runtime images;
2. the first credentialed cloud-dev run follows docs/cloud-development.md and records budget scope, external alert IDs/test notification, schema head, IAM-only private invocation, timer/YMQ retry and DLQ behavior, custom-domain health, canary routing, and rollback evidence.

The provider limitation remains explicit: Billing Budget and Monitoring Alert resources are external guardrails, not silently represented as Terraform-managed resources.

CI evidence — commit 527bd34

The mirrored GitHub Actions run is green: CI #30634841746.

Verified against the exact GitCode MR head SHA 527bd34f9b0bcde6ddb5c9282e8680114004af22:

• Go verification: success
• Terraform environments: success
• YDB schema and state store: success
• Local multi-service stand, including restart/persistence: success
• Runtime images: success; control API, reconciler, Telegram sender, and worker images all built

The GitCode push mirror was triggered through gitcode-mcp and its terminal finished status was confirmed before reading the GitHub SHA. The remaining #12 acceptance boundary is the credentialed cloud-dev deployment and operational evidence listed in the previous comment.

Merge status

MR !16 was merged after GitHub Actions #30634841746 completed successfully for head 527bd34f9b0bcde6ddb5c9282e8680114004af22.

The infrastructure and deployment runbook are now on main. Issue #12 remains open only for the first credentialed cloud-dev deployment and the operational evidence already listed above; merge and CI acceptance are complete.

Cloud bootstrap research update: YMQ credentials and first-apply findings

The first real cloud-dev apply validated most of the planned topology, but exposed four provider/service contracts that were not visible in local validation.

1. YMQ cannot use the ephemeral key used by the Terraform S3 backend

Yandex Cloud documents that ephemeral access keys authenticate Object Storage only. YMQ's SQS-compatible tooling instead requires a static access key. This is also reflected by the Terraform provider: queue resources require YC_MESSAGE_QUEUE_ACCESS_KEY and YC_MESSAGE_QUEUE_SECRET_KEY.

Using the broad deployment identity's static key, storing a key in tfvars, or allowing the key payload into Terraform state would violate the issue's security contract.

Decision implemented in MR !17, commit 750f36d:

• create a dedicated queue-provisioner service account with only ymq.admin;
• create a separate scheduler access key for the existing ymq.writer service account;
• use Terraform output_to_lockbox so both generated key payloads go directly to custom-KMS Lockbox secrets; the provider documents that access_key and secret_key are not populated when this mode is used (resource reference);
• let the deployment wrapper resolve the provisioning key from Lockbox into the Terraform child process environment only;
• inject the scheduler writer key into the reconciler revision from its own Lockbox secret;
• do not share either key with the Telegram secret or any other runtime identity.

This requires one additional, explicitly confirmed targeted bootstrap after folder creation and budget verification:

CONFIRM_QUEUE_AUTH_BOOTSTRAP=sessionless-cloud-dev:queue-auth \
  ./scripts/cloud-terraform.sh queue-auth-bootstrap

Ordinary plan/apply/destroy remains saved-plan based and runs under the fenced YDB deployment lock.

2. Custom-KMS Lockbox consumers need two permissions

The first container revisions were rejected with PermissionDenied: Access to secret is denied. A runtime identity that reads a Lockbox secret encrypted with a customer-managed KMS key needs both:

• lockbox.payloadViewer on its specific secret;
• kms.keys.encrypterDecrypter on the backing key.

The fix grants KMS decrypt only to api, telegram-sender, and scheduler, while retaining secret-specific Lockbox bindings. The runtime module now explicitly depends on the completed foundation/IAM module so a first deployment cannot race secret grants.

3. API Gateway canary variable syntax

The provider accepted the HCL, but API Gateway rejected the OpenAPI document because control_container_id was declared and considered unused. Gateway variables use ${var.control_container_id}, not ${apigw.control_container_id}. The corrected form preserves the stable/candidate container-ID override used by weighted canary routing.

4. Container Registry lifecycle selection is mandatory

A lifecycle rule must select tagged or untagged images. The rule now uses tag_regexp = ".*", retaining the ten newest tagged images and expiring older matching images after 30 days.

Verification and deployment gate

Locally, commit 750f36d passes:

• make ci including Go race tests and integration tests;
• Terraform formatting and a clean terraform validate using pinned provider yandex-cloud/yandex 0.220.0;
• shell syntax and git diff --check.

The GitCode-to-GitHub push mirror has been triggered. No second cloud apply will be performed until GitHub Actions is green for the same commit SHA. After that gate, the next proof sequence is:

1. bootstrap the two YMQ keys into Lockbox;
2. review a new full saved plan and reject any unexpected destroy/replace action;
3. apply under the YDB deployment lock;
4. verify all schema migrations, certificate/DNS, private container invocation, YMQ queues/triggers, public health endpoints, and only then register the Telegram webhook.

The runbook and architecture diagram in docs/cloud-development.md were updated with this credential boundary and recovery procedure.

Cloud-dev bootstrap update: workload authentication and first healthy runtime

The first real Serverless Container invocations exposed two cloud-only authentication gaps that local YDB/MinIO tests cannot reproduce.

Findings

1. YDB environment credentials must select metadata explicitly.
   All five revisions had the correct service accounts and YDB IAM binding, but ydb-go-sdk-auth-environ falls back to anonymous credentials when no selector is set. Runtime logs therefore showed YDB discovery failing with Unauthenticated. The runtime module now sets YDB_METADATA_CREDENTIALS=1; the SDK obtains renewable IAM tokens from the revision service account through the Serverless Containers metadata service.

2. The artifact bucket deliberately rejects static-key authentication.
   The bucket has disabled_statickey_auth = true, while the existing S3 adapter used the AWS SDK signing path. This would allow command-only Telegram checks to pass but make every normal message or attachment fail before dispatch. The adapter now has a cloud mode selected by S3_IAM_METADATA_CREDENTIALS=true: it obtains the same metadata-issued IAM token and calls the Object Storage S3 HTTP API with Authorization: Bearer. Local MinIO remains on the AWS SDK/static development-credential path.

This keeps long-lived Object Storage keys out of Terraform state, Lockbox, revision environment, and the repository. Runtime access is still limited by the per-component service account and bucket IAM bindings.

Official constraints used for the decision:

• Yandex Serverless Containers runtime and metadata service
• YDB authentication through the metadata service
• Object Storage API authentication options
• AWS SDK for Go static/environment credential behavior

Implementation and verification

• Branch: ai/mvp-09a-cloud-bootstrap-readiness
• MR: !17
• Runtime-auth commits: 490c124 (YDB metadata credentials), ec87cd0 (Object Storage IAM bearer adapter)
• Published immutable runtime tag: ec87cd07c82b1d27c745ea4c1db88d08b298d46b, verified as linux/amd64
• Terraform apply: 0 added, 5 changed, 0 destroyed
• GitHub mirror CI: run 30821643003, success
• Private IAM invocation and public API Gateway checks pass for /healthz, /readyz, and /version
• Telegram webhook is registered at https://dev-api.sessionless.triborg.dev/telegram/webhook; Telegram reports zero pending updates
• YDB migrations 00001 through 00041 are applied
• Budget remains the existing folder-scoped monthly 100 RUB guardrail

The remaining acceptance proof is a live Telegram transaction: a command reply, a normal text workload through Object Storage -> YDB -> YMQ -> deterministic worker -> Telegram sender, and one small attachment. Monitoring alert creation/test and the documented blue/green canary/rollback exercise also remain before #12 can close.

Live Telegram ingress finding: Yandex public endpoints are not reachable reliably

Cloud-dev testing on 2026-08-03 found a network boundary that the original design did not cover.

Evidence

1. Independent IPv4 clients reach the deployed API Gateway custom domain successfully, complete TLS 1.2/1.3, and receive the expected webhook response.
2. Telegram Bot API delivery to the same API Gateway times out before any matching request reaches API Gateway or control-api logs.
3. The webhook was then moved to a public Yandex Workflows execution URL, following Yandex's documented Telegram-bot pattern. A direct synthetic POST to that URL returned an execution ID and the execution finished in 2.275 s.
4. Telegram still timed out against the native Workflows endpoint. At 2026-08-03T15:05:46Z, redacted getWebhookInfo metadata reported:
   • destination host: serverless-workflows.api.cloud.yandex.net;
   • destination IPv4: 84.201.181.26;
   • pending_update_count: 2;
   • last_error_message: Connection timed out.
5. Workflow history contained only the synthetic execution. The later /start and /compute status replies therefore prove that the cloud backend is runnable, but do not prove that the Workflows bridge accepted those updates; they can be delayed successes from Telegram's earlier delivery retries.

No bot token, webhook secret, workflow capability URL, Telegram identity, message body, or attachment content was logged during this diagnosis.

Conclusion

The failure is scoped to live reachability from Telegram's webhook network to the Yandex public edge. It is not explained by DNS delegation, the custom certificate, API Gateway routing, the Go handler, YDB, or the internal worker pipeline. Putting another Yandex-hosted public primitive in front of API Gateway does not cross that network boundary.

Revised cloud-dev ingress

Use a minimal Cloudflare Worker as the Telegram-facing edge, while retaining Yandex Workflows as the durable asynchronous handoff:

flowchart LR
    Telegram["Telegram Bot API"] -->|"POST + Telegram secret"| CF["Cloudflare Worker<br/>dev-api-sessionless.triborg.dev"]
    CF -->|"unchanged JSON<br/>private capability URL"| WF["Yandex Workflows"]
    WF -->|"trusted Lockbox header"| GW["Yandex API Gateway"]
    GW --> API["control-api"]

The Worker will:

• accept only POST /telegram/webhook;
• verify X-Telegram-Bot-Api-Secret-Token against a Cloudflare secret binding;
• enforce JSON and a small request-size ceiling;
• forward the unchanged body to the Yandex Workflows execution URL held in a second secret binding;
• return success only after Workflows returns an execution ID; otherwise return a retryable failure to Telegram;
• never log headers, request bodies, capability URLs, or secret values.

dev-api-sessionless.triborg.dev is deliberately a first-level hostname in the existing Cloudflare-managed triborg.dev zone, so Cloudflare manages its edge certificate without depending on the delegated sessionless.triborg.dev child zone.

Cost and deployment decision

Cloudflare Workers Free currently includes 100,000 requests/day and 10 ms CPU/request. Waiting for the outbound fetch() does not consume CPU time. This is comfortably inside the 100 RUB/month cloud-dev budget for rare Telegram traffic.

Keep Yandex resources in Terraform. Manage this thin external edge with pinned Wrangler plus a repository script:

• Worker source and non-secret custom-domain configuration are versioned;
• CLOUDFLARE_API_TOKEN is supplied only from the operator credential store;
• Telegram secret and Workflows capability URL are uploaded as Worker secret bindings and never enter Git, Terraform state, plans, or command-line arguments;
• CI runs local Worker tests and a dry-run bundle; cloud deployment remains an explicit operator action.

Acceptance delta for #12

In addition to the existing criteria, cloud-dev is not complete until:

• getWebhookInfo reports the Cloudflare hostname and zero pending updates;
• a fresh text update and an image update each create a Yandex Workflow execution;
• the full ingress -> YDB/YMQ -> worker -> delivery path is proven without inspecting payload contents;
• a Cloudflare outage or Yandex handoff failure produces a non-2xx response so Telegram retries;
• direct calls with a missing/wrong Telegram secret are rejected;
• canary and rollback are demonstrated after the edge is in place.

References:

• Yandex Workflows: execute a workflow via a public link
• Cloudflare Workers custom domains
• Cloudflare Workers pricing
• Cloudflare Workers limits

Scope correction after the live ingress research

The live test exposed three independent deliverables, and implementing all of them inside the bootstrap change would make #12 and MR !17 too broad. The work is now split as follows:

• #12 / MR !17: reproducible Yandex Cloud bootstrap, direct cloud contract smokes, and the documented Telegram-to-Yandex public-edge reachability finding.
• #17 — MVP-09b: Cloudflare-to-Workflows Telegram ingress, secret handling, bounded handoff retry, Wrangler deployment and live webhook reachability.
• #18 — MVP-09c: fresh text/image live E2E, opaque YDB/YMQ/Object Storage evidence, duplicate handling, canary and rollback.
• #19 — MVP-09d: operational alerts, sparse-traffic thresholds and a controlled notification test.

This supersedes the previous comment's suggestion to absorb the full revised ingress and all operational acceptance into #12. The research and redacted timeout evidence remain valid; implementation and proof move to their linked issues.

Final acceptance report — Yandex cloud-dev foundation

Implementation is ready to merge in MR !17, branch ai/mvp-09a-cloud-bootstrap-readiness, final head dfab76654063859962c6d847012bbf64698cce64.

Delivered

• protected Object Storage remote state and fenced YDB deployment locking;
• isolated cloud-dev folder, least-privilege IAM and a folder-scoped 100 RUB/month budget;
• Terraform-owned YDB, Object Storage, YMQ queues/DLQs, Container Registry, KMS/Lockbox, logging, private Serverless Containers/triggers, delegated DNS, managed certificate and API Gateway;
• YMQ-only static credentials generated directly into custom-KMS Lockbox without payload exposure in Terraform state;
• immutable linux/amd64 runtime publication and private image-pull authorization;
• workload metadata IAM authentication for YDB and Object Storage;
• migrations 00001 through 00041 applied;
• private IAM invocation plus direct API Gateway /healthz, /readyz and /version checks;
• saved-plan deployment, first-apply recovery, protected destroy and credential-lifecycle documentation.

Final scope correction

Live evidence proved that Telegram cannot reliably reach the tested native Yandex API Gateway or public Workflows endpoints. The runbook and webhook helper no longer instruct or permit the Terraform api_url to be used as the Telegram webhook destination.

The follow-up gates are explicit:

• #17: Cloudflare Telegram edge and Workflows handoff;
• #18: live text/image, duplicate, canary and rollback proof;
• #19: alerts and notification proof.

They depend on #12 but do not keep the Yandex foundation issue open.

Verification

Local final-branch evidence:

• make ci — success;
• pinned Terraform 1.15.5 in clean Linux containers: recursive format check, both roots init -backend=false and validate — success;
• sh -n scripts/*.sh and git diff --check — success;
• webhook helper negative guards — success.

Mirrored GitHub Actions run 30833889547 completed successfully on the exact final SHA. All five jobs passed: Go verification, Terraform environments, YDB schema/state store, local multi-service restart/persistence and runtime images.

No unresolved MR discussions remain. Issue #12 should close when !17 merges; no additional implementation or cloud action is required inside this scope.

Closure — !17 merged and main verified

MR !17 was squash-merged into main as 65667a0cd9916d71385d9ab1e847e6034780bf30.

The GitCode push mirror reached terminal finished state and GitHub main points at the same SHA. Mirrored GitHub Actions run 30835129381 completed successfully after the merge:

• Go verification — success;
• Terraform environments — success;
• YDB schema and state store — success;
• local multi-service restart/persistence — success;
• runtime images — success.

The Yandex cloud-dev foundation, workload authentication, protected deployment procedures and scoped runbook are now on main. The documented Telegram-to-native-Yandex reachability gap remains valid and is not hidden inside this closure.

Remaining work is tracked independently:

• #17: external Telegram edge and Workflows handoff;
• #18: live text/image, duplicate, canary and rollback proof;
• #19: alerts and notification proof.

Issue #12 is complete and can close.