| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
[MooncakeStore] Introduce MooncakeStore: A new Distributed Object Store for XpYd PD disaggregation (#126) Mooncake Store is a high-performance **distributed key-value (KV) cache storage engine** designed specifically for LLM inference scenarios. Unlike traditional caching systems such as Redis or Memcached, Mooncake Store is positioned as **a distributed KV cache rather than a generic caching system**. The key difference is that in the former, the key is derived from the value through hashing, so value is immutable after inserting (although the key/value pair may be garbage collected). Mooncake Store provides low-level object storage and management capabilities, while specific caching strategies (e.g., eviction policies) are left to upper-layer frameworks (like vLLM) or users for implementation, offering higher flexibility and customizability. Key features of Mooncake Store include: - **Object-level storage operations**: Mooncake Store provides simple and easy-to-use object-level APIs, including Put, Get, and Remove operations. - **Multi-replica support**: Mooncake Store supports storing multiple data replicas for the same object, effectively alleviating hotspots in access pressure. - **Eventual consistency**: Mooncake Store ensures that Get operations read complete and correct data, but does not guarantee the latest written data. This eventual consistency model ensures high performance while simplifying system design. - **High bandwidth utilization**: Mooncake Store supports striping and parallel I/O transfer of large objects, fully utilizing multi-NIC aggregated bandwidth for high-speed data reads and writes. - **Dynamic resource scaling**: Mooncake Store supports dynamically adding and removing nodes to flexibly handle changes in system load, achieving elastic resource management (to be refined in future versions). --------- Co-authored-by: Feng Ren <alogfans@gmail.com> Co-authored-by: Teng Ma <teng-ma@linux.alibaba.com> Co-authored-by: Shangming Cai <caishangming@linux.alibaba.com> Co-authored-by: susun <751080330@qq.com> Co-authored-by: liusy58 <liusy58@linux.alibaba.com> | 1 年前 | |
[Store] Check If Get Operation Completed Within Lease (#778) | 9 个月前 | |
[MooncakeStore] Introduce MooncakeStore: A new Distributed Object Store for XpYd PD disaggregation (#126) Mooncake Store is a high-performance **distributed key-value (KV) cache storage engine** designed specifically for LLM inference scenarios. Unlike traditional caching systems such as Redis or Memcached, Mooncake Store is positioned as **a distributed KV cache rather than a generic caching system**. The key difference is that in the former, the key is derived from the value through hashing, so value is immutable after inserting (although the key/value pair may be garbage collected). Mooncake Store provides low-level object storage and management capabilities, while specific caching strategies (e.g., eviction policies) are left to upper-layer frameworks (like vLLM) or users for implementation, offering higher flexibility and customizability. Key features of Mooncake Store include: - **Object-level storage operations**: Mooncake Store provides simple and easy-to-use object-level APIs, including Put, Get, and Remove operations. - **Multi-replica support**: Mooncake Store supports storing multiple data replicas for the same object, effectively alleviating hotspots in access pressure. - **Eventual consistency**: Mooncake Store ensures that Get operations read complete and correct data, but does not guarantee the latest written data. This eventual consistency model ensures high performance while simplifying system design. - **High bandwidth utilization**: Mooncake Store supports striping and parallel I/O transfer of large objects, fully utilizing multi-NIC aggregated bandwidth for high-speed data reads and writes. - **Dynamic resource scaling**: Mooncake Store supports dynamically adding and removing nodes to flexibly handle changes in system load, achieving elastic resource management (to be refined in future versions). --------- Co-authored-by: Feng Ren <alogfans@gmail.com> Co-authored-by: Teng Ma <teng-ma@linux.alibaba.com> Co-authored-by: Shangming Cai <caishangming@linux.alibaba.com> Co-authored-by: susun <751080330@qq.com> Co-authored-by: liusy58 <liusy58@linux.alibaba.com> | 1 年前 | |
revert: rollback v0.3.7-dev to 39bbe9c This commit reverts upstream/v0.3.7-dev back to 39bbe9cc10663f437f8b3a4487013e80f08ee2ba. | 5 个月前 | |
feat(store): support transfer engine p2phandshake (#852) * feat(allocator): Add transport endpoint support for segment transfers * refactor(Client): Replace segment_name with transport_endpoint in buffer descriptors and tests * refactor(test): switch to in-proc master and remove external service management * style: format code and adjust transfer task timeout * feat(Client::MountSegment): set te_endpoint conditionally based on metadata_connstring for P2P mode * Allow configuring default KV lease TTL via environment variable Read DEFAULT_KV_LEASE_TTL environment variable at server startup and parse as uint64_t value. This enables CI tests to override default configuration without code changes. Signed-off-by: xiaguan <751080330@qq.com> * style: format CI config and add [[nodiscard]] to methods Signed-off-by: xiaguan <751080330@qq.com> * refactor(tests): remove local_hostname parameter and improve test helpers The commit removes the unused local_hostname parameter from TransferSubmitter constructor, updates the associated initialization logic, and simplifies test code by introducing helper functions for segment creation. The changes improve code maintainability while preserving functionality. * refactor(TransferSubmitter): Remove unused local_hostname_ member and validation check * docs(store): update setup examples for p2p --------- Signed-off-by: xiaguan <751080330@qq.com> | 9 个月前 | |
refactor(store): Remove BufStatus and segment_name for AllocatedBuffer (#883) * Refactor(store): Remove BufStatus and segment_name for AllocatedBuffer - Removed BufStatus enum and related functionality from AllocatedBuffer. - Updated AllocatedBuffer constructor to eliminate segment_name parameter. - Enhanced getSegmentName method to retrieve segment name from the allocator. - Adjusted logging and tests to reflect changes in buffer status handling. * fix fmt * update comment | 9 个月前 | |
refactor(store): Remove BufStatus and segment_name for AllocatedBuffer (#883) * Refactor(store): Remove BufStatus and segment_name for AllocatedBuffer - Removed BufStatus enum and related functionality from AllocatedBuffer. - Updated AllocatedBuffer constructor to eliminate segment_name parameter. - Enhanced getSegmentName method to retrieve segment name from the allocator. - Adjusted logging and tests to reflect changes in buffer status handling. * fix fmt * update comment | 9 个月前 | |
[Test] fix(store): fix memory leak in client_integration_test.cpp (#881) | 9 个月前 | |
fix: initialize local hot cache with negative value | 7 个月前 | |
feat(client): Add client-side metrics for transfer and RPC operations (#733) * feat(client): Add RPC operation and tansfer metrics tracking * feat(client): Add client-side metrics for transfer and RPC operations This commit introduces a comprehensive metrics system for the client component, tracking transfer byte counts and operation latencies with both human-readable summaries and Prometheus-style serialization. Key features include: - New TransferMetric for tracking read/write bytes and latency histograms - MasterClientMetric for RPC call counting and latency tracking - Environment-controlled metrics reporting (MC_STORE_METRIC_REPORT) - Automatic periodic metrics collection thread - Enhanced test coverage for metrics validation - Unified metrics interface across all client operations The implementation provides detailed latency percentiles (P50/P95) and total byte tracking with automatic unit conversion (B/KB/MB/GB). * fix test * Update mooncake-store/src/client.cpp Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update mooncake-store/src/client.cpp Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update mooncake-store/src/client.cpp Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix format issue * refactor(client): make client metrics optional using ClientMetric class * fix lint --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> | 10 个月前 | |
[MooncakeStore] Introduce MooncakeStore: A new Distributed Object Store for XpYd PD disaggregation (#126) Mooncake Store is a high-performance **distributed key-value (KV) cache storage engine** designed specifically for LLM inference scenarios. Unlike traditional caching systems such as Redis or Memcached, Mooncake Store is positioned as **a distributed KV cache rather than a generic caching system**. The key difference is that in the former, the key is derived from the value through hashing, so value is immutable after inserting (although the key/value pair may be garbage collected). Mooncake Store provides low-level object storage and management capabilities, while specific caching strategies (e.g., eviction policies) are left to upper-layer frameworks (like vLLM) or users for implementation, offering higher flexibility and customizability. Key features of Mooncake Store include: - **Object-level storage operations**: Mooncake Store provides simple and easy-to-use object-level APIs, including Put, Get, and Remove operations. - **Multi-replica support**: Mooncake Store supports storing multiple data replicas for the same object, effectively alleviating hotspots in access pressure. - **Eventual consistency**: Mooncake Store ensures that Get operations read complete and correct data, but does not guarantee the latest written data. This eventual consistency model ensures high performance while simplifying system design. - **High bandwidth utilization**: Mooncake Store supports striping and parallel I/O transfer of large objects, fully utilizing multi-NIC aggregated bandwidth for high-speed data reads and writes. - **Dynamic resource scaling**: Mooncake Store supports dynamically adding and removing nodes to flexibly handle changes in system load, achieving elastic resource management (to be refined in future versions). --------- Co-authored-by: Feng Ren <alogfans@gmail.com> Co-authored-by: Teng Ma <teng-ma@linux.alibaba.com> Co-authored-by: Shangming Cai <caishangming@linux.alibaba.com> Co-authored-by: susun <751080330@qq.com> Co-authored-by: liusy58 <liusy58@linux.alibaba.com> | 1 年前 | |
code format & enable code format checking in ci (#677) Signed-off-by: doujiang24 <doujiang24@gmail.com> | 10 个月前 | |
[Store] Add initial support for master high availability failover (#451) * A temp version. Better to continue development after merging the latest main branch * Temp version to merge the latest main branch * Allow optional use HA mode, in default use non-HA mode. Fix a minor metrics bug. * Refactor the etcd_helper * refactor ha_helper * Add some unit tests. Refactor the code * Update cmakelists: build etcd_wrapper in default * Fix ci problems. Compile etcd wrapper only when use_etcd or with_store are set. * Update python config relating to mooncake-store client * make some blocking etcd helper function cancellable. bug fix: add string name of new errors that will be used in tostring. * Refactor etcd related code * Bug fix * Add basic masterviewhelper unit tests * In ci flow, install and start etcd to run HA feature unit test. * Fix a ci bug * Reuse master_server_address parameter and remove enable_ha parameter. * Format the code. Fix a minor bug. * Handle the error case: the coro server may fail to start or return internal error. | 1 年前 | |
revert: rollback v0.3.7-dev to 39bbe9c This commit reverts upstream/v0.3.7-dev back to 39bbe9cc10663f437f8b3a4487013e80f08ee2ba. | 5 个月前 | |
[Store] Check If Get Operation Completed Within Lease (#778) | 9 个月前 | |
refactor(store): Remove BufStatus and segment_name for AllocatedBuffer (#883) * Refactor(store): Remove BufStatus and segment_name for AllocatedBuffer - Removed BufStatus enum and related functionality from AllocatedBuffer. - Updated AllocatedBuffer constructor to eliminate segment_name parameter. - Enhanced getSegmentName method to retrieve segment name from the allocator. - Adjusted logging and tests to reflect changes in buffer status handling. * fix fmt * update comment | 9 个月前 | |
[Store] Check If Get Operation Completed Within Lease (#778) | 9 个月前 | |
[Chores] Offset Allocator Test Fix & Docs Fix (#806) * Fix unit test * fix docs | 9 个月前 | |
code format & enable code format checking in ci (#677) Signed-off-by: doujiang24 <doujiang24@gmail.com> | 10 个月前 | |
revert: rollback v0.3.7-dev to 39bbe9c This commit reverts upstream/v0.3.7-dev back to 39bbe9cc10663f437f8b3a4487013e80f08ee2ba. | 5 个月前 | |
refactor: introduce expected pattern for error handling in master service (#562) * refactor: introduce expected pattern for error handling in master service - Replace ErrorCode return types with tl::expected<T, ErrorCode> pattern - Improve error handling clarity by separating success values from error codes - Update MasterService methods to return expected<void, ErrorCode> or expected<T, ErrorCode> - Modify RPC service interfaces to support expected pattern - Update all related tests to handle new expected return types - Add necessary includes for ylt/util/expected.hpp This change makes error handling more explicit and type-safe: - Success cases can be accessed via .value() - Error cases can be accessed via .error() - Eliminates ambiguity between success and error states Future work: - Extend expected pattern to RPC response types - Enhance error code system for more comprehensive error handling * Refactor error handling to use expected<T,E> pattern instead of ErrorCode - Updated MasterService methods to replace ylt::expected with tl::expected for better error handling. - Modified BatchGetReplicaList, BatchPutStart, BatchPutEnd, and other methods to return tl::expected types. - Enhanced ClientIntegrationTest to handle expected results from Put, Get, Remove, and other operations using tl::expected. - Adjusted error handling in clientctl and master_metrics_test to utilize the new expected type. - Improved overall error reporting in tests to provide clearer feedback on operation failures. * fix test compile * refactor: update client implementation and remove master.proto - Enhanced client.h and client.cpp with new functionality - Removed obsolete master.proto file - Updated master_client.cpp and transfer_task.cpp - Improved integration and stress tests * fix: update Python integration to work with new batch API - Replace BatchObjectInfo with vector<vector<Replica::Descriptor>> - Update BatchPut to handle new return type vector<tl::expected<void, ErrorCode>> - Fix BatchQuery API usage to work with new expected pattern - Convert unordered_map to vector format for BatchPut parameter compatibility * refine master log and metric * fix(ci): should alloc first * merge main * fix tests | 11 个月前 | |
[Store] Serialize/Deserialize Offset Allocator (#760) Also add generic interfaces to serialize/deserialize other objects. | 10 个月前 | |
[Bench] fix: Update stress_cluster_benchmark (#791) Co-authored-by: 春明 <chunming.dc@alibaba-inc.com> | 10 个月前 | |
Refactor(store): Enable the transfer engine to autonomously detect (#817) * Refactor(store): Enable the transfer engine to autonomously detect topology * feat(client): support manual RDMA device configuration without auto-discovery | 9 个月前 | |
[Store] Check If Get Operation Completed Within Lease (#778) | 9 个月前 | |
code format & enable code format checking in ci (#677) Signed-off-by: doujiang24 <doujiang24@gmail.com> | 10 个月前 | |
feat(client): Abstract client-side data transmission for async and batch optimization (#455) * feat(client): add transfer submitter for optimized data transfer Signed-off-by: Jinyang Su <751080330@qq.com> * feat(store): implement async memcpy task execution with worker pool Add MemcpyWorkerPool to manage asynchronous execution of memcpy tasks. Refactor BatchGet and BatchPut methods for parallel execution and enhance logging for better traceability. * Squashed commit of the following: commit 38c435fcc6dc074cae6e505c6050a956c04297e5 Author: Feng Ren <alogfans@users.noreply.github.com> Date: Wed Jun 11 16:50:29 2025 +0800 Revert "[TransferEngine] Fix minor bugs in NVLink transport and benchmark (#468)" (#469) This reverts commit ffaad6aa189bfaa75c033af21b7b7f4734cccb80. commit 41b1df795425faa7f95f59eb5b3c46f809bd329f Author: ykwd <oneday117@qq.com> Date: Wed Jun 11 16:37:05 2025 +0800 [Store] Add initial support for master high availability failover (#451) * A temp version. Better to continue development after merging the latest main branch * Temp version to merge the latest main branch * Allow optional use HA mode, in default use non-HA mode. Fix a minor metrics bug. * Refactor the etcd_helper * refactor ha_helper * Add some unit tests. Refactor the code * Update cmakelists: build etcd_wrapper in default * Fix ci problems. Compile etcd wrapper only when use_etcd or with_store are set. * Update python config relating to mooncake-store client * make some blocking etcd helper function cancellable. bug fix: add string name of new errors that will be used in tostring. * Refactor etcd related code * Bug fix * Add basic masterviewhelper unit tests * In ci flow, install and start etcd to run HA feature unit test. * Fix a ci bug * Reuse master_server_address parameter and remove enable_ha parameter. * Format the code. Fix a minor bug. * Handle the error case: the coro server may fail to start or return internal error. commit ffaad6aa189bfaa75c033af21b7b7f4734cccb80 Author: Feng Ren <alogfans@users.noreply.github.com> Date: Wed Jun 11 16:02:41 2025 +0800 [TransferEngine] Fix minor bugs in NVLink transport and benchmark (#468) * [TransferEngine] Fix compilation bug in NVLink xport * [TransferEngine] Fix minor bugs in nvlink benchmark Signed-off-by: Jinyang Su <751080330@qq.com> --------- Signed-off-by: Jinyang Su <751080330@qq.com> | 1 年前 | |
Refactor(store): Enable the transfer engine to autonomously detect (#817) * Refactor(store): Enable the transfer engine to autonomously detect topology * feat(client): support manual RDMA device configuration without auto-discovery | 9 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 1 年前 | ||
| 9 个月前 | ||
| 1 年前 | ||
| 5 个月前 | ||
| 9 个月前 | ||
| 9 个月前 | ||
| 9 个月前 | ||
| 9 个月前 | ||
| 7 个月前 | ||
| 10 个月前 | ||
| 1 年前 | ||
| 10 个月前 | ||
| 1 年前 | ||
| 5 个月前 | ||
| 9 个月前 | ||
| 9 个月前 | ||
| 9 个月前 | ||
| 9 个月前 | ||
| 10 个月前 | ||
| 5 个月前 | ||
| 11 个月前 | ||
| 10 个月前 | ||
| 10 个月前 | ||
| 9 个月前 | ||
| 9 个月前 | ||
| 10 个月前 | ||
| 1 年前 | ||
| 9 个月前 |