| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
[fix] Guarantee even node distribution for simple storage units (#144) ## Motivation SimpleStorage fans experience data out across num_data_storage_units Ray actors. For memory and network bandwidth to be balanced across the cluster, those units must be spread evenly over the available Ray nodes — otherwise a subset of nodes absorbs a disproportionate share of the storage load and becomes a hotspot. The current placement relies on a placement group with the SPREAD strategy, which does not actually guarantee this. ## What was wrong 1. **SPREAD is best-effort, not a guarantee.** initialize_simple_storage created a placement group via get_placement_group(num_data_storage_units), which builds one CPU bundle per unit with ray.util.placement_group(..., strategy="SPREAD"). Ray's SPREAD is explicitly documented as best-effort: when scheduling is constrained (resource pressure, timing, node availability) it silently falls back to packing multiple bundles — and therefore multiple storage units — onto the same node. 2. **No control over the units-vs-nodes ratio.** SPREAD gives no even-split guarantee regardless of how num_data_storage_units compares to the node count, and its strict sibling STRICT_SPREAD isn't a usable alternative here because it *fails outright* once num_actors > num_nodes (the common case, since the recommended setting is ≥ 2× nodes). The result: uneven memory/bandwidth distribution and node hotspots that the "spread" was supposed to prevent. ## What this changes 1. **Explicit round-robin placement via NodeAffinitySchedulingStrategy.** New helper get_node_round_robin_scheduling_strategies(num_actors) in transfer_queue/utils/common.py enumerates all currently alive Ray nodes (ray.nodes() filtered by Alive, sorted by NodeID for deterministic ordering) and returns one strategy per actor, assigning actor i to alive_node_ids[i % len(alive_node_ids)] with soft=False (hard affinity). This guarantees each node receives floor(num_actors / num_nodes) or ceil(num_actors / num_nodes) units — an even split by construction, for **any** ratio of units to nodes. 2. **initialize_simple_storage now schedules per-unit.** It calls get_node_round_robin_scheduling_strategies(num_data_storage_units) and passes scheduling_strategy=strategies[rank] to each SimpleStorageUnit.options(...), replacing the previous placement_group + placement_group_bundle_index wiring. The creation log line now records the target node_id for each unit, making the distribution observable. 3. **Fail-fast on an empty cluster.** If no alive nodes are found, the helper raises RuntimeError("No alive Ray nodes found. Is Ray initialized?") instead of proceeding with an undefined placement. 4. **Config doc clarified.** config.yaml documents that units are round-robin scheduled across all alive nodes for an even per-node split, and keeps the "≥ 2× nodes" recommendation (now framed as "so each node hosts multiple units"). ## Tests Verified on a Ray cluster that num_data_storage_units units are distributed evenly across alive nodes for both units ≤ nodes and units > nodes cases, with each unit's target node confirmed via the new creation log line. Existing SimpleStorage unit and e2e lifecycle tests continue to pass. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> | 9 天前 | |
[feat] Add metrics exporter and dashboard for TransferQueue (#83) Expose Prometheus metrics from the controller and storage units so TransferQueue activity can be monitored end to end. Include a Grafana dashboard and tests so the observability workflow is easier to validate and adopt. --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> | 3 个月前 | |
[misc] refactor: simplify internal classes naming (#86) Rename internal classes by removing TransferQueue prefix: - TransferQueueStorageManager → StorageManager - TransferQueueStorageManagerFactory → StorageManagerFactory - StorageClientFactory → StorageKVClientFactory - TransferQueueRole → Role - move the factory class to base.py - rename simple_backend.py to simple_storage.py These classes are internal components not exposed as public API, so the shorter names improve readability without causing conflicts. Besides, enable modern type annotation style: Auto convert legacy Optional/Union type annotations to modern X | None syntax. cc @0oshowero0 --------- Signed-off-by: ji-huazhong <hzji210@gmail.com> Co-authored-by: ji-huazhong <hzji210@gmail.com> | 3 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 9 天前 | ||
| 3 个月前 | ||
| 3 个月前 |