master

文件	最后提交记录	最后更新时间
test_hccl_tensor_transport.py	[feat] Add HCCL tensor transport for RDT (#21) ## Description - Implement HCCLTensorTransport to integrate HCCL with RDT tensor transport flow and enable explicit registration - Add HCCL collective backend tests by using register_collective_backend API - Add tests covering HCCL tensor transport usage in RDT - Provide register_hccl_collective_backend, register_hccl_tensor_transport, and register_yr_tensor_transport API - Update the usage doc ## Self-Check Result - HCCL collective backend tests by using register_collective_backend API: bash [root@devserver-bms-54 ray-ascend]# pytest -v /home/hhc/ray_wp/ray-ascend/tests/collective/test_hccl_via_registry.py ====================================================================== test session starts ======================================================================= platform linux -- Python 3.10.16, pytest-8.3.4, pluggy-1.5.0 -- /usr/bin/python cachedir: .pytest_cache Test order randomisation NOT enabled. Enable with --random-order or --random-order-bucket=<bucket_type> rootdir: /home/hhc/ray_wp/ray-ascend configfile: pyproject.toml plugins: cov-6.2.1, hydra-core-1.3.2, jaxtyping-0.3.2, anyio-4.8.0, random-order-1.2.0, mock-3.14.1, typeguard-4.3.0 collected 6 items tests/collective/test_hccl_via_registry.py::test_allreduce PASSED [ 16%] tests/collective/test_hccl_via_registry.py::test_broadcast PASSED [ 33%] tests/collective/test_hccl_via_registry.py::test_allgather PASSED [ 50%] tests/collective/test_hccl_via_registry.py::test_reduce PASSED [ 66%] tests/collective/test_hccl_via_registry.py::test_reducescatter PASSED [ 83%] tests/collective/test_hccl_via_registry.py::test_send_recv PASSED [100%] ======================================================================== warnings summary ======================================================================== tests/collective/test_hccl_via_registry.py::test_allreduce /usr/local/python3.10.16/lib/python3.10/site-packages/ray/_private/worker.py:2052: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 warnings.warn( -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ========================================================================= tests coverage ========================================================================= ________________________________________________________ coverage: platform linux, python 3.10.16-final-0 ________________________________________________________ Name Stmts Miss Cover Missing --------------------------------------------------------------------------------------- ray_ascend/__init__.py 0 0 100% ray_ascend/_version.py 13 13 0% 4-34 ray_ascend/collective/__init__.py 2 0 100% ray_ascend/collective/hccl_collective_group.py 334 259 22% 30, 34-36, 65-83, 94-102, 118-119, 122-123, 126-130, 136-146, 150-162, 167, 173-181, 197-222, 241-274, 291-315, 326-333, 347-374, 393-426, 441-461, 476-496, 499-512, 524-554, 558-560, 572-585, 589-615, 625-636, 640-642, 647-653, 668-677 ray_ascend/direct_transport/__init__.py 3 3 0% 1-8 ray_ascend/direct_transport/hccl_tensor_transport.py 9 9 0% 1-22 ray_ascend/direct_transport/yr_tensor_transport.py 116 116 0% 1-253 ray_ascend/direct_transport/yr_tensor_transport_util.py 44 44 0% 1-91 --------------------------------------------------------------------------------------- TOTAL 521 444 15% Coverage HTML written to dir htmlcov Coverage XML written to file coverage.xml ============================================================ 6 passed, 1 warning in 60.06s (0:01:00) ============================================================= - HCCL tensor transport tests: bash [root@devserver-bms-54 ray-ascend]# pytest -v /home/hhc/ray_wp/ray-ascend/tests/direct_transport/test_hccl_tensor_transport.py ====================================================================== test session starts ======================================================================= platform linux -- Python 3.10.16, pytest-8.3.4, pluggy-1.5.0 -- /usr/bin/python cachedir: .pytest_cache Test order randomisation NOT enabled. Enable with --random-order or --random-order-bucket=<bucket_type> rootdir: /home/hhc/ray_wp/ray-ascend configfile: pyproject.toml plugins: cov-6.2.1, hydra-core-1.3.2, jaxtyping-0.3.2, anyio-4.8.0, random-order-1.2.0, mock-3.14.1, typeguard-4.3.0 collected 1 item tests/direct_transport/test_hccl_tensor_transport.py::test_hccl_tensor_transport PASSED [100%] ======================================================================== warnings summary ======================================================================== tests/direct_transport/test_hccl_tensor_transport.py::test_hccl_tensor_transport /usr/local/python3.10.16/lib/python3.10/site-packages/ray/_private/worker.py:2052: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 warnings.warn( -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ========================================================================= tests coverage ========================================================================= ________________________________________________________ coverage: platform linux, python 3.10.16-final-0 ________________________________________________________ Name Stmts Miss Cover Missing --------------------------------------------------------------------------------------- ray_ascend/__init__.py 0 0 100% ray_ascend/_version.py 13 13 0% 4-34 ray_ascend/collective/__init__.py 2 0 100% ray_ascend/collective/hccl_collective_group.py 334 259 22% 30, 34-36, 65-83, 94-102, 118-119, 122-123, 126-130, 136-146, 150-162, 167, 173-181, 197-222, 241-274, 291-315, 326-333, 347-374, 393-426, 441-461, 476-496, 499-512, 524-554, 558-560, 572-585, 589-615, 625-636, 640-642, 647-653, 668-677 ray_ascend/direct_transport/__init__.py 3 0 100% ray_ascend/direct_transport/hccl_tensor_transport.py 9 2 78% 21-22 ray_ascend/direct_transport/yr_tensor_transport.py 116 82 29% 47-49, 52, 56, 60, 66-104, 107-125, 140-152, 160-173, 185, 194-220, 238-248 ray_ascend/direct_transport/yr_tensor_transport_util.py 44 22 50% 9-10, 22-23, 46, 49, 53-55, 58-59, 62-63, 66, 71, 79, 82-83, 86-87, 90-91 --------------------------------------------------------------------------------------- TOTAL 521 378 27% Coverage HTML written to dir htmlcov Coverage XML written to file coverage.xml ============================================================ 1 passed, 1 warning in 68.25s (0:01:08) ============================================================= ## Related issues Closes #9 #3 --------- Signed-off-by: Haichuan Hu <kaisennhu@gmail.com>	2 个月前
test_yr_transport.py	[feat] Support auto-init YR backend based on metastore/etcd mode (#45) ## Description Add auto-initialization for YuanRong DS backend supporting both metastore and etcd modes. Metastore mode runs a built-in service on head node, eliminating external etcd dependency. Etcd mode supports existing etcd infrastructure. Both modes auto-start DS workers via Ray placement group across all cluster nodes. ## Changes - Core (ray_ascend/utils/yr_utils.py): - register_yr_tensor_transport(): unified public API for backend init + transport registration - YRBackendCoordinator: named actor for cluster-wide coordination, handles concurrent init requests - DataSystemActor: per-node actor managing DS worker lifecycle - start_datasystem_worker(): unified worker startup with init_mode parameter - Auto-init (ray_ascend/direct_transport/yr_tensor_transport.py): - get_ds_client(): auto-initializes backend when YR_DS_INIT_MODE is set - Docs: - Added initialization modes guide with environment variable configuration ## Related issues Fixes #44 #33 Signed-off-by: Haichuan Hu <kaisennhu@gmail.com>	3 个月前
test_yr_transport_util.py	[test] Refactored yr_test (#43) ## Description - Refactor _test_yr_transport_unit.py_ , _test_yr_transport.py_, isolating the tests of NPU and CPU. - Add more granular tests, such as the consistency of AdapterInterface. ## Self-check python ============================================================================================== tests coverage ============================================================================================== _____________________________________________________________________________ coverage: platform linux, python 3.11.9-final-0 ______________________________________________________________________________ Name Stmts Miss Cover Missing --------------------------------------------------------------------------------------- ray_ascend/__init__.py 4 0 100% ray_ascend/_version.py 2 0 100% ray_ascend/collective/__init__.py 2 2 0% 1-5 ray_ascend/collective/hccl_collective_group.py 311 311 0% 1-638 ray_ascend/direct_transport/__init__.py 2 0 100% ray_ascend/direct_transport/yr_tensor_transport.py 116 19 84% 72, 96-98, 111-123, 146-147, 162, 168, 185, 214-215, 247-248 ray_ascend/direct_transport/yr_tensor_transport_util.py 101 14 86% 11-14, 25-28, 76, 177-178, 195, 203, 209 ray_ascend/utils/__init__.py 2 0 100% ray_ascend/utils/serial_utils.py 35 25 29% 23-44, 63-81 ray_ascend/utils/yr_utils.py 63 47 25% 17, 22-24, 29-30, 42-108, 118-150 --------------------------------------------------------------------------------------- TOTAL 638 418 34% <img width="342" height="25" alt="image" src="https://github.com/user-attachments/assets/abe9553c-4c94-495f-9118-ae02ab9ae8e7" /> Low tests coverage of yr_utils.py is because there are too many codes about startup services. ## Related Issue - Implement #30 --------- Signed-off-by: dpj135 <958208521@qq.com>	3 个月前