| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
fix(infra): bind forked workers to reserved ports (#1657) * fix(infra): bind forked workers to reserved ports Keep process and port ownership coupled across allocation, fork, health checks, and cleanup so failed starts and kills remain retryable. Migrate schedulers and service controllers to the owner-bound Guard protocol and roll back partial startup failures. * fix(infra): make Guard owner lifecycle atomic Serialize owner-bound allocation, fork, release, and kill state transitions so concurrent requests cannot orphan children or release ports from a newer process generation. Normalize worker indices across lifecycle endpoints and roll back partially acquired node port locks on allocation failure. | 4 天前 | |
feat(cli): add inference service cli (#1434) * feat(cli): inference service CLI — daemon + 4 verbs `areal inf` is an ollama-style operator console for the local inference service. One daemon per user, one OpenAI-compatible gateway endpoint, models registered against it. Verbs: inf run start daemon (gateway + router); inline --model registers an external (--api-url) or internal (--backend / --model-path) model in the same call inf ps list registered models inf status daemon health + model count inf stop SIGTERM gateway+router, grace period, then SIGKILL State is a single ~/.areal/inf/state.json (pid + url + admin key + started_at). Everything else lives in the gateway/router process memory; CLI is otherwise stateless. Layout under areal/experimental/cli/: main.py / state.py shared scaffold (areal_home, pid_alive, atomic_write_json), to be reused by `areal train` in a separate PR commands/inf/__init__.py all four verbs + register helpers in a single file (one place to scan what the user can do) commands/inf/state.py DaemonState dataclass + paths commands/inf/launcher.py subprocess spawn helpers, including base_gpu_id support for sglang dp>1 commands/inf/client.py urllib gateway + router HTTP client Heavy imports (sglang/vllm/torch via areal.api.cli_args) stay lazy — inside register helpers only — so `areal inf -h` and `areal -h` parse the click tree without paying for them. * fix(cli/inf): persist model worker pids in state.json so stop kills them `_register_internal` returns the list of sglang+data-proxy pids it spawned, but `_do_run` discarded the return value. state.json then only held gateway+router pids, so `areal inf stop` left sglang and data-proxy processes orphaned. Add a `worker_pids: list[int]` field to DaemonState, save the list right after register_internal succeeds, and include those pids in the stop kill set. Also fix _do_stop short-circuiting on a dead gateway pid — even when the daemon front-end is dead, model worker pids in state may still be alive (the original bug pattern this commit fixes). Always try to kill every pid we know about, then drop the state file. * fix(cli/inf): route startup messages through getLogger; align admin key default Startup-time prints (router/gateway pids, replica spawn lines, daemon ready, foreground / shutdown notices) were going through click.echo, so they bypassed the AReaL log formatter — no timestamps, no log level, no PascalCase tag. Switch all of these to `getLogger("InfCli").info(...)` so output looks like the rest of the project: 20260611-09:14:02.157 InfCli INFO: starting inference daemon ... 20260611-09:14:02.342 InfCli INFO: router pid=304116 http://... 20260611-09:14:02.910 InfCli INFO: gateway pid=304118 http://... Register InfCli in LOGGER_COLORS_EXACT under the launcher (blue) group, since `areal inf run` is conceptually a process launcher. Reads kept on click.echo on purpose: `inf ps` / `inf status` plain-text output is structured (tabular or single-field-per-line) and meant to be consumed by jq / awk in scripts; piping that through the colored log formatter would break it. Also flip `--admin-api-key` default from "areal-admin-key" to "admin-api-key" — matches the existing inference_service convention. * feat(cli/inf): standalone `register` and `deregister` verbs Phase 2 — model lifecycle separated from daemon lifecycle. areal inf register <name> [external | internal flags] Register a new model against a running daemon. Same flags as `inf run --model ...`, just attached to an existing service. areal inf deregister <name> [--grace] [--force] Drop the model from the router, unregister its proxy workers, and SIGTERM/SIGKILL the spawned sglang+data-proxy procs. DaemonState gains a `models: dict[str, ModelEntry]` mapping each registered model to its (pids, proxy_addrs). This replaces the previous flat `worker_pids: list[int]` so deregister can target one model's processes without touching the others. `inf stop` flattens across all entries and kills the whole set; the foreground / failure cleanup paths use the same flatten. `_register_internal` now returns `(pids, proxy_addrs)` rather than just pids — both are needed at deregister time (router unregister takes the proxy addr; SIGTERM takes the pid). External models also get a state entry now (empty pids/addrs) so `inf deregister` can find them and drop them from the router. phase 1 verbs unchanged. * feat(cli/inf): phase 3 — reward + collect verbs Two verbs that round out the RL data path: areal inf reward <session_api_key> <reward> [--model X] thin wrapper around POST /rl/set_reward. set_reward is the only thing that flips an active conversation into a ready trajectory, so any data-collection flow needs to call it (even with a dummy reward=0 just to "flush"). CLI verb is for users whose agent is in another language / shell / human raters; agent authors writing python are free to POST directly. areal inf collect <model> --batch-size N \ [--sessions-out FILE] [--output FILE] \ [--timeout T] [--poll-interval S] \ [--discount D] [--style individual|concat] client-side batch orchestrator. start_session(group_size=N) -> hand sessions to the agent (via --sessions-out FILE) -> poll /export_trajectories every poll-interval seconds, accumulating unique trajectories until N are collected or timeout fires -> one final export with remove_session=True for cleanup -> dump JSONL. Mirrors the controller's rollout_batch path but moves the wait loop to the client so gateway / router stay stateless. Agent lifecycle is intentionally NOT inside collect (no --agent-cmd): agent runs in its own process and just reads sessions_out. JSONL is the only output format on purpose -- the gateway already serializes trajectories to JSON at HTTP boundary, so .pt would mean re-decoding tensors only to re-encode them; trainers consuming the output can torch.tensor(x) when they need it. 8 verbs total (run / ps / status / stop / register / deregister / reward / collect). * feat(cli/inf): enrich ps/status with model kind / backend / addrs Phase 2.5 — bring `inf ps` and `inf status` to the design_inf.md fidelity (sections 11.5 / 11.6) without bringing back the multi-service concept. ModelEntry gains four fields: kind: 'internal' | 'external' backend: spec string ('sglang:tp=2,dp=2') for internal, '' for external api_url: external upstream URL, '' for internal inference_server_addrs: per-replica sglang/vllm URLs (internal) `_register_internal` returns (pids, proxy_addrs, inf_addrs); _do_run and _do_register both fill in the new ModelEntry fields. inf ps now reads state.models (CLI-side truth) instead of polling gateway /v1/models. Output: NAME KIND BACKEND WORKERS qwen3 internal sglang:tp=2,dp=2 2 gpt-4o external - - inf status switches to a multi-row table per design 11.5: COMPONENT STATUS ADDR DETAILS gateway ok http://127.0.0.1:8080 models=2 router ok http://127.0.0.1:.. qwen3 registered internal backend=sglang:tp=2 workers=2 gpt-4o registered external api_url=https://... JSON output of both verbs follows suit. Backwards-compat: ModelEntry's new fields all have defaults, so an old state.json still loads cleanly. * feat(cli/inf): add `inf models` verb `inf ps` currently lists registered models (table: NAME / KIND / BACKEND / WORKERS). Add `inf models` as a more explicit alias — docker-style "different verb for different resource". `ps` keeps its current behavior; both call the same _print_models helper so output is identical. This isn't part of the multi-service rollback (which we decided not to do). It's a small ergonomic addition for the single-daemon shape. * feat(cli/inf): add `logs` verb + TOML config support + help text for proxy/engine args Three related additions: 1. `inf logs --component NAME [-f] [-n LINES]` Tail a log under ~/.areal/inf/logs/. Component defaults to 'gateway'; can be 'router' or a full model log basename like 'qwen3-inf-0'. -f follows (tail -F), -n sets initial line count (default 200). Exec's tail directly for stream fidelity. 2. TOML config support (design 12) - Group-level option `areal inf --config FILE` merges FILE on top of ~/.areal/inf/config.toml (both optional). - config.py loads TOML via tomllib (py3.11+), maps [default] / [launch] / [register.internal] / [collect] sections to CLI option defaults via click's default_map mechanism. - Precedence: CLI flag > --config > ~/.areal/inf/config.toml > hardcoded default. 3. Detailed help for --engine-args and --proxy-args Users couldn't guess what to pass. Now both flags show inline hints (common sglang knobs / data-proxy flags + defaults). Help is a single paragraph so click's wrap_text handles terminal width. 10 verbs total (run / stop / ps / status / models / register / deregister / reward / collect / logs). * fix(cli/inf): align `collect` flags with design 11.9 Rename + add + drop options on `inf collect` so the surface matches the design spec exactly: rename --discount -> --turn-discount rename --style -> --export-style add --format json|jsonl (default jsonl) add --json progress-events flag (placeholder; not implemented) drop --task-id (always 'cli-collect' internally) drop --sessions-out (agents query gateway directly) drop --poll-interval (always 2.0s internally) JSON output (--format json) emits {tid: interaction, ...} pretty-printed. JSONL output (default) emits one trajectory per line, each prefixed with trajectory_id. config.toml [collect] keys renamed to match. * fix(cli/inf): three bugs (admin key, gpu collision, sglang request log) P1. --admin-api-key default reverts from "admin-api-key" to "areal-admin-key" — matches the v2 inference_service convention used everywhere else in the codebase. P2. Registering a second internal model collided with the first on GPUs 0..tp-1. Cause: base_gpu_id was always r * tp, computed only against the current model's dp index, ignoring GPUs already used by previously-registered models. Fix: track a monotonic `next_gpu_id` cursor on DaemonState and pass it into `_register_internal` as `base_gpu_id`. Each ModelEntry now records its (base_gpu_id, gpu_count) so deregister can roll back the cursor when removing the *last* registered model (preserves contiguous allocation; doesn't try to coalesce holes in the middle, which is fine for the v1 use case). P3. sglang server logs only had model-load output, no chat requests. Cause: SGLangConfig.log_requests defaults to False. Fix: spawn sglang with log_requests=True so /chat/completions traffic shows up under ~/.areal/inf/logs/<model>-inf-N.log. (data-proxy access logs are off via uvicorn config inside the data-proxy package itself; that's not under inf CLI's control.) * refactor(experimental): split inference CLI commands Move the inference service CLI out of the monolithic commands package and into command-specific modules. Align collect with the session/export flow by returning session keys, polling exports without revoking the router group, and cleaning up at the end. * feat(experimental): support multiple inference services * fix(experimental): harden inference CLI lifecycle Protect inference service state transitions so register and run do not race or leave orphaned processes behind. Key changes: - Track engine and proxy PIDs separately for phased shutdown - Lock model state during register and startup model setup - Clean up foreground services on SIGTERM and SIGHUP - Recover raw PIDs before forced service replacement - Align inference CLI model and session option names with the design * feat(cli/inf): add scheduler abstraction for worker placement Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(cli/inf): widen probe timeout, parallelize status, drop default_model Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(experimental): adopt cli scaffold v2 in inference service Rebase onto feat/experimental-cli-scaffold and replace the per-CLI duplicates with the shared base classes/utilities: - state.py keeps the inf-specific dataclasses and the two-file recover_pids_from_raw_state; RuntimeState now satisfies ServiceStateBase (gateway_alive + components + .load classmethod). - config.py / client.py / lifecycle.py shrink to thin subclasses of ConfigLoader / BaseHTTPClient / ServiceLifecycle. The old GatewayHTTPError / GatewayUnreachable names are kept as aliases of ServiceHTTPError / ServiceUnreachable so subcommands swap mechanically. - InferenceLifecycle overrides force_replace_slot to walk the inf raw-state helper (which knows about the secondary model-state file) and to remove both files on cleanup. - common.py drops scaffold-replaced helpers (running_state / load_running_state / refuse_if_running / wait_http_health / wait_client_health / print_services / print_models / probe_http_health); keeps backend-spec parsing, model registration, TaskHandle formatters, and terminate_runtime_state (data-flow order is inf-specific). - commands/run.py uses ServiceLifecycle for refuse / force-replace and ForegroundWatcher for the SIGINT/SIGTERM/SIGHUP handling. - commands/stop / status / ps / models / register / deregister / reward / collect route through inf_lifecycle; status.py emits via StatusReporter + ColumnSpec; ps/models via json_or_table. - launcher.py and scheduler/local.py replace pick_free_port with find_free_ports (non-ephemeral, no TOCTOU); LocalScheduler tracks allocated ports across submits. - commands/logs.py is removed; LogsCommand(lifecycle=inf_lifecycle) wires the verb in __init__.py. Net ~290 LOC dropped while every behavioral guarantee carries over. * docs(cli/inf): add inference service CLI guide Document the `areal inf` subcommand group: launching the gateway/router, registering models, RL session flow with rewards, trajectory collection, log management, and configuration file precedence. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * style(docs): apply mdformat to inference CLI guide * refactor(experimental): address PR review on inference CLI Address garrett4wade's review on #1435: 1+2) Drop the unfinished `collect` verb and the gateway RPCs it depended on. `commands/collect.py` is removed entirely; the gateway client no longer exposes `start_session` / `export_trajectories` (only `set_reward` remains, used by the surviving `reward` verb). The cli group, config bindings, cli_guide section, and parser tests are cleaned up accordingly. 3) Replace the bespoke `sglang:tp=2,dp=2` mini-DSL in `parse_backend_spec` with `ModelAllocation.from_str`, so the CLI accepts the same grammar as `InferenceEngineConfig.backend` in YAML configs (`sglang:d4`, `vllm:d2t4`). Help text, doc examples, and test fixtures are updated to the new form. 4) Collect the free functions in `state.py` (`models_dir`, `models_state_path`, `models_lock_path`, `locked_model_state`, `recover_pids_from_raw_state`) into an `InferenceStateStore` class and route every caller through a module-level `store` instance. The dataclasses now obtain paths via `store.<...>`, keeping on-disk-layout responsibilities in one place. Net: 13 files, +170 / -524 (mostly from dropping the collect verb). * style(tests): drop trailing blank lines after removed collect case * refactor(experimental): subclass scaffold NamespacedStateStore in inference scaffold's state.py promoted the namespace-aware free functions onto NamespacedStateStore. Update inference accordingly: - InferenceStateStore now subclasses NamespacedStateStore, gaining service_state_path / set_current_service / clear_current_service / current_service_path / resolve_service_name from the parent. It keeps the inf-specific models_*, lock_model_state, and overrides recover_pids_from_raw_state to walk both state files. - ServiceState.save / .remove route through ``store.set_current_service`` / ``store.clear_current_service`` instead of the deleted free functions; commands/run.py / register.py / tests resolve log paths via ``store.logs_dir`` and the service-state path via ``store.service_state_path``. Behavior unchanged. * style: sort v2 CLI imports * refactor(experimental): drop reward verb + RL session flow docs Address PR #1434 review: - Remove ``areal inf reward`` (commands/reward.py + the ``reward_cmd`` wiring in __init__.py) and the corresponding GatewayClient.set_reward RPC. The reward flow is server-side only for now; the CLI does not need to wrap it. - Drop the "RL session flow" and "Setting reward" sections from cli_guide.md plus the trailing "For plain inference there is no need to call /rl/start_session" note. - Drop the ``reward`` entry from the [default].service binding tuple in config.py and the reward-related cases in the parser test. --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> | 2 个月前 | |
fix(infra): bind forked workers to reserved ports (#1657) * fix(infra): bind forked workers to reserved ports Keep process and port ownership coupled across allocation, fork, health checks, and cleanup so failed starts and kills remain retryable. Migrate schedulers and service controllers to the owner-bound Guard protocol and roll back partial startup failures. * fix(infra): make Guard owner lifecycle atomic Serialize owner-bound allocation, fork, release, and kill state transitions so concurrent requests cannot orphan children or release ports from a newer process generation. Normalize worker indices across lifecycle endpoints and roll back partially acquired node port locks on allocation failure. | 4 天前 | |
perf(engine): stream Megatron microbatches from CPU (#1622) * perf(engine): stream Megatron microbatches from CPU Keep full training batches off the accelerator and transfer only the current microbatch before forward. This prevents packed and padded batch representations from scaling GPU residency with the global batch size. Key changes: - Broadcast Megatron batch RPC payloads through CPU process groups - Stage trainer and engine microbatch construction on CPU - Preserve tree, VLM, and loss-weight behavior during lazy transfer - Add CPU streaming and virtual-pipeline iterator regression tests * fix: keep streamed training payloads off accelerator Avoid staging complete v2 training payloads on the accelerator before Megatron microbatch streaming. Replace tree-count object collectives with a backend-compatible scalar MAX reduction and cover both placement paths with unit tests. * fix: reduce CPU-staged stats on compatible devices CPU-staged Megatron payloads can leave rollout statistics on CPU while the data-parallel process group uses NCCL or HCCL. Move only locally aggregated scalars to the backend-compatible device before collectives and keep empty reward-model placeholders on the engine device. Key changes: - add backend-aware scalar all-reduce handling - align empty reward-model stats with the engine device - cover CPU stats and empty train/eval batches Refs: #1622 * fix: harden streamed microbatch distributed coverage Fail fast when CPU-staged RPC methods lack their Gloo mirror and preserve tuple containers during recursive tensor broadcast. Extend distributed training coverage across TP1/PP1 and TP2/PP2 with multi-microbatch gradient-norm parity checks. --------- Co-authored-by: daihao <dh183333@antgroup.com> | 12 天前 | |
feat(awex): add separation AdamW delta weight transfer (#1623) * feat(config): add separation DTE topology gates * feat(awex): add separation AdamW delta weight transfer * docs(examples): add DTE separation GSM8K example * docs(examples): expand DTE GSM8K configuration * fix(awex): address separation DTE review feedback * fix(awex): harden separation delta transfer protocol * refactor(config): integrate delta transfer into train engine * refactor(config): simplify delta weight update switch * test(awex): isolate optional DTE dependency | 17 天前 | |
refactor: move 5 experimental modules into areal/v2 for 2.0 release (#1448) * refactor: move 5 experimental modules into areal/v2 for 2.0 release Move agent_service, inference_service, training_service, weight_update, and cli from areal/experimental/ to areal/v2/, and rewrite every reference (Python imports, `python -m` invocations, console-script entry points, CODEOWNERS, docs, review-pr signals) to the new path. - 5 directories migrated via `git mv` (history preserved) - 70 files modified across areal/, tests/, examples/, pyproject{,vllm}.toml, .github/CODEOWNERS, ROADMAP.md, and tooling docs - `areal/v2/__init__.py` added so v2 is an importable package - Build config (tool.uv.build-backend with module-root="") auto-discovers the new package — no pyproject changes beyond the entry point Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * style: apply pre-commit auto-fixes after areal/v2 move CI pre-commit job reformatted 54 files automatically: - .github/CODEOWNERS: realign owner columns (32-col) after shorter /areal/v2/ paths broke the previous 40-col alignment - areal/**, tests/**, examples/**, docs/**: ruff isort reorders `areal.v2.*` imports into their new alphabetical slot (between `areal.engine` and `areal.infra`) - markdown/yaml whitespace normalized by mdformat / ruff-format Pure formatting; no logic change. `pre-commit run --all-files` is now green locally. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(tests): point py<3.12 conftest stubs at areal/v2/ Two test conftest.py stubs (Python 3.10/3.11 compat) still had the `areal/experimental/{inference_service,weight_update}` path as the namespace package's __path__, broken by the v2 move. The sed pass missed them because the paths were comma-separated `os.path.join` args (`"areal", "experimental", "X"`), not slash-form path strings. Additionally insert an `areal.v2` stub between `areal` and the leaf package so the parent→child attribute wiring loop (which uses `name.rsplit(".", 1)`) can find a parent module in sys.modules. Without it `setattr(parent, child, ...)` silently no-ops and `unittest.mock.patch` traversal breaks on the new path. Spotted by gemini-code-assist on PR #1448. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor(tests): mirror areal/v2 move under tests/v2 Move agent_service, inference_service, training_service, and weight_update test directories from tests/experimental/ to tests/v2/ to mirror the source-tree layout. tests/experimental/ retains archon/ and openai/ (still-experimental modules). - 50 files migrated via `git mv` (history preserved) - tests/v2/__init__.py added - 9 files rewritten for the new dotted/slashed test paths: `tests.experimental.{4 modules}` → `tests.v2.{4 modules}` `tests/experimental/{4 modules}` → `tests/v2/{4 modules}` (covers integration_utils imports, fake_train_engine engine_class, pytest invocation strings in docstrings, and the tests/v2/weight_update/torchrun/run_nccl_weight_transfer.py path) conftest stubs (Python <3.12 namespace shims) keep working because _REPO_ROOT is computed via "..", "..", ".." from the conftest file — same depth under tests/v2/X/ as under tests/experimental/X/. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> | 2 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 4 天前 | ||
| 2 个月前 | ||
| 4 天前 | ||
| 12 天前 | ||
| 17 天前 | ||
| 2 个月前 |