Paired: GitHub #105 ↔ GitCode !85

github_pr: 105
head_repo: WenYuLuo/agent-dx
head_ref: feat/observability-qps-tiers-20261022
head_sha: 735d5706d5914b284da465563bf764abb4d1403a
base_repo: openJiuwen-ai/agent-dx
base_ref: master
github_repo: openJiuwen-ai/agent-dx
base_branch: master
base_sha: 6daa91984a10a8d76f55eabee194c02935986b0a
github_user: WenYuLuo

GitHub PR mirror for CodeArts CI.

Changes

Show Environment creation throughput and Paused inventory while reducing telemetry volume and the work performed for each instance during ordinary node reporting.

• Add request QPS, first-ready QPS, their peaks over the selected range, in-flight requests, outcomes, and latency. Peaks use the maximum one-minute average, with replica rates summed before taking the maximum.
• Show Running and Paused counts with matching cards, trends, node distribution, and inventory. Paused inventory is independent of resource samples.
• Aggregate long-term resource metrics by node and runtime class. Put per-instance details behind separate optional endpoints and a dedicated detail store. Examples configure separate scrape intervals and retention periods.
• Build normal node reconciliation catalogs from Coordinator memory. Load durable state during startup or recovery and update cached records, snapshots, and execution holds after committed writes. Ambiguous artifact writes invalidate the cache until recovery completes. Ordinary comparisons validate ownership and versions with creation admission left open. Fresh Pending claims remain owned by their create RPCs, and a missing snapshot entry cannot release an unresolved local reservation. Inspection requires a current authenticated session; startup and session recovery perform complete runtime adoption and orphan cleanup. Background wake-intent and dormant-expiry sweeps also select candidates from committed memory. Cancelled or ambiguous writes trigger authoritative recovery, and uncertain retirement cannot requeue a deleted Environment as a new create.
• Drive lifecycle handling through per-execution Execd state subscriptions, backend exit notifications, proxy activity deltas, and controller deadlines. Reconnects replay full state; stream loss invalidates observations and allows a recovery grace period before targeted backend confirmation. Preserve generation fencing, cleanup confirmation, restart limits, checkpoint publication ordering, and two-sided idle admission checks. Idempotent admission recovery handles lost barrier replies and node restart before routes are reopened. Ambiguous Start outcomes retain targeted cleanup retries with bounded backoff. Once a retry deadline is armed, passive reconciliation commands cannot postpone it and starve a pending restart. Pending workload checkpoint requests retain their operation identity and original time budget through transient commit or acknowledgement failures; retry deadlines resume the existing idempotent path without repeating a completed capture.
• Sample CPU and memory separately, every 15 seconds by default, with at most 32 concurrent reads. Sampling and cache writes bypass lifecycle queues. Writes atomically validate the controller registration, execution identity, generation, and eligibility; late samples cannot revive a retired or replaced execution. Log retention uses recovered controller records and preserves executions whose cleanup is unconfirmed.
• Record one environment.create total span for JSON, SSE, and embedded calls, aligned with the creation duration histogram and confirmed Running/publication result. Preserve SSE context without an extra stream span, record outcome and error status, and show total creation rows in Explore while preserving rejected HTTP requests that never enter the create service. End environment.queue at dequeue while preserving execution parentage. Inject W3C metadata when the RPC is sent so remote spans belong to the active client RPC span.
• Keep normal comparisons, Coordinator catalog inspections, and passive probes silent in tracing. Explicitly traced inspection requests retain their incoming context. Initial Pending comparisons are silent, while ownership conflicts and recovery from a controller fence retain their traces. Preserve traces for lifecycle changes, recovery, cleanup, and failed observations.
• Add a log detail dashboard with a default 15-minute window, manual refresh, a 500-line limit, runtime filters, literal keyword search, and on-demand HTTP trace details. Preserve backslashes, quotes, and backticks in filters. Overview queries use bounded short lookback windows.
• Add make python-check PYTHON_CHECK_BASE=<target-branch-ref> with pinned Ruff/Pylint rules, covering every changed Python file in the PR.

Deployment

adxlet requires runtime_control. Update the node configuration and the Execd binary in the runtime rootfs/bootstrap together, using a node IPv4 endpoint reachable from the runtime. Examples and E2E generators enable this configuration. metrics_interval_seconds defaults to 15 and is independent of node reporting.

See docs/testing/node-lifecycle.md, docs/deployment/runtime-control-stream.md, and docs/testing/observability-capacity.md for contracts and operational settings.

Validation

Latest-head standalone validation: 735d5706d5914b284da465563bf764abb4d1403a. Earlier component tests and the Firecracker/runsc matrix below were run at 72b5cd4af3c5b82f989b57ee70869355388dc04b; the subsequent three commits change only E2E fixtures, their tests, and testing documentation.

• Passed the complete native Linux amd64, logical two-node Docker standalone profile: all 11 required groups completed, 49 JUnit tests, zero failures, errors, or skips. Coverage includes SDK/data plane, lifecycle, authentication, capacity, placement, local-first creation, node failure, sandboxd restart, Adxlet restart, and supervised stop.

• Built and verified a clean latest-head native package and SDK candidate with matching commit provenance. The standalone backend is pinned sandboxd efc201531d7e2e9d69505da151eb66084b61eebf, using runc.

• Fixed a pre-existing restart-fixture readiness gap exposed by the initial run: recovered backend IDs were visible before sandboxd's first 35-second housekeeping cycle reported SERVING, while the following create had a 30-second scheduling deadline. The fixture now waits for both backend health and Coordinator node admission, preserves backend identities, and records both readiness conditions. The complete profile then verifies new creation and Adxlet recovery.

• Updated the trace assertion to validate the current recovery path: an execution may follow either its queue span or its environment.authority_reconcile span within the same trace. Missing and cross-trace parents still fail; at least one real queue/execution chain is required. Three readiness regressions and ten trace-collection tests pass, including tests demonstrated failing before each fixture fix. Eight live-output tests, the whole-PR Python precheck, and documentation checks also pass.

• Replaced the test OTLP backend's HTTP method override with a threaded standard-library WSGI application, addressing the platform naming finding. Five real-socket regressions cover log/trace separation, gzip, outage/recovery, credential rejection before persistence, unsupported requests, and concurrent JSON-line writes. The complete standalone profile was rerun with this receiver. The whole-PR Python precheck, all 26 affected tests, and documentation checks pass.

• Both nodes passed trace propagation and Execd HTTP context checks. Each recovered all 40 unique log probes after rotation, backend outage, and Collector restart. SDK lifecycle operations succeeded during the telemetry backend outage.

• Both node supervisors confirmed empty backend inventories on stop. Final audit found no task containers, networks, or cgroups; all seven pre-existing running containers remained running.

• Added regression coverage for creating during a blocked live catalog fetch, preserving an unresolved Claim reservation, deferring a durable Claim until its Create RPC arrives, and preserving a registered initial Pending controller. The relevant race assertions failed before the fixes.

• Startup recovery still retires uncertain old Pending executions. Session invalidation still immediately fences new allocation and drains accepted work before reopening. Old sessions, expired leases, and mismatched node certificates cannot inspect the catalog.

• Pending trace regressions verify silent healthy comparisons and retained conflict/recovery traces; the recovery assertion failed before its fix.

• Passed 485 native Linux API Server, adxlet, observability, and transport unit/integration tests, with zero failures or ignored tests.

• Passed the real Redis/mTLS control-RPC suite (36 tests), storage suite (52), complete Coordinator library tests (13), and snapshot tests (5). Two library cases are also included in control-RPC.

• Passed workspace formatting, all-target/all-feature Clippy with warnings denied, and documentation checks.

• Built and verified the complete native x86_64 Linux package and current SDK from the clean commit above. Its manifest records dirty=false; package, runtime kit, and harness hashes are retained.

• Passed both 12-case public-SDK lifecycle baselines on real Firecracker and runsc: backend exit/restart/exhaustion, pushed health failures, disconnected-state replay, active-request/idle protection, pause/resume, live-backend stream outages, checkpoint replay, node/relay restart recovery, checkpoint expiry, and zero steady catalog scans.

• Passed input-commit, result-commit, and acknowledgement fault injection on both runtimes. All six requests completed with one native checkpoint capture each, stable execution identity, and preserved workload state.

• Passed the runsc sampling case with 33 real instances. While 32 checkpoint requests were held by a test gate, the healthy 33rd instance completed six real Stats reads in six seconds; its latest sample was 0.386 seconds old. All held requests subsequently completed. Pause removed its samples, and resume exposed only its new execution.

• All four final runtime runs used the normal two-second node reporting interval, verified in their rendered configurations. Lifecycle baselines used a one-hour metrics interval to isolate event-driven behavior; retry/sampling cases used one-second metrics. The 32 runtime/scenario cases passed without changing node reporting cadence.

• Every run ended with an empty backend inventory and successful shutdown. Final host audit found no remaining task processes, mounts, or occupied task cgroups.

Earlier telemetry-specific checks, unchanged by this fix, passed Prometheus recording-rule/dashboard checks, Grafana/Loki literal filtering including backslashes, the whole-PR Python precheck (seven files), and a real Tempo 2.7.2 creation-row query regression. Earlier revisions also cover creation-span timing/parentage across JSON, SSE, and embedded calls.

The runtime matrix uses one native Linux host, isolated network/PID/mount namespaces, and local checkpoint storage. Network ACL enforcement is disabled. The runsc health fixture uses its default user; the explicit User=0:0 variant remains outside this matrix after an earlier entrypoint credential-switch EPERM. Conch, cross-host HA, production deployment, cross-host clock accuracy, and capacity load testing remain unverified.

CodeCheck

Exact-head static checks and Ruff are SUCCESS for 735d5706d5914b284da465563bf764abb4d1403a: report. The PR head matches, ci-successful is present, and ci-running is absent. The complete standalone profile and all 26 affected Python regressions passed for this same commit.