memcache:基于MemFabric的高性能分布式KVCache存储引擎项目 - AtomGit

MemCache是针对AI推理场景设计的高性能分布式KVCache缓存，针对昇腾超节点、服务器的实现了精细化性能调优，提供较好的High Availability能力, 同时对接到vllm-ascend、sglang、mindie等开源推理框架

分支5 Tags7

Yyrewzjsx访问备份的map未加锁

86be28a1创建于 16 天前839次提交

文件	最后提交记录	最后更新时间
.gitcode	Add Pull Request Template Co-authored-by: y30060514<yangjianhong2@h-partners.com> # message auto-generated for no-merge-commit merge: !49 merge develop into develop Add Pull Request Template Created-by: CYangJH Commit-by: y30060514 Merged-by: chenyz6 Description: Add Pull Request Template See merge request: Ascend/memcache!49	5 个月前
3rdparty	fix: report error when memcache whl install failed Co-authored-by: shilinlee_com<836160610@qq.com> # message auto-generated for no-merge-commit merge: !205 merge mmc_install_failed_replace_version into develop fix: report error when memcache whl install failed Created-by: shilinlee_com Commit-by: shilinlee_com Merged-by: yrewzjsx Description: # Pull Request Template ## Description fix: report error when memcache whl install failed ## Related Issues <!-- Replace with actual issue numbers. Example: Closes #123, Related to #456 --> ## Type of Change - [x] Bug fix - [ ] New feature - [ ] Performance optimization - [ ] Documentation update - [ ] Other (please describe):_______________ ## Testing ![image.png](https://raw.gitcode.com/user-images/assets/7672915/d8b9c0ba-fdc9-46f2-bb42-89cf423157b9/image.png 'image.png') ![image.png](https://raw.gitcode.com/user-images/assets/7672915/bd3677b4-ed27-442a-9d54-4f0a6355c054/image.png 'image.png') ## Checklist - [ ] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. See merge request: Ascend/memcache!205	1 个月前
config	优化编译和配置文件 Co-authored-by: vaus<1078978268@qq.com> # message auto-generated for no-merge-commit merge: !230 merge develop into develop 优化编译和配置文件 Created-by: vaus Commit-by: vaus Merged-by: yrewzjsx Description: # Pull Request Template ## Description <!-- Provide a clear summary of the change, the problem it solves, and the technical approach. --> ## Related Issues <!-- Replace with actual issue numbers. Example: Closes #123, Related to #456 --> ## Type of Change - [ ] Bug fix - [ ] New feature - [ ] Performance optimization - [ ] Documentation update - [ ] Other (please describe):_______________ ## Testing <!-- Describe how the changes were tested. Include: - Unit/integration test coverage - Manual validation steps - Links to CI reports (e.g., CIDA) if available - Note any pending tests --> ## Checklist - [ ] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. See merge request: Ascend/memcache!230	26 天前
doc	[fix] 修改ock.mmc.client.read_thread_pool.size默认值为4 Co-authored-by: mrh1024<marunhua1@h-partners.com> # message auto-generated for no-merge-commit merge: !257 merge develop_fix into develop [fix] 修改ock.mmc.client.read_thread_pool.size默认值为4 Created-by: mrh1024 Commit-by: mrh1024 Merged-by: yrewzjsx Description: # Pull Request Template ## Description <!-- Provide a clear summary of the change, the problem it solves, and the technical approach. --> ## Related Issues <!-- Replace with actual issue numbers. Example: Closes #123, Related to #456 --> ## Type of Change - [ ] Bug fix - [ ] New feature - [ ] Performance optimization - [ ] Documentation update - [ ] Other (please describe):_______________ ## Testing <!-- Describe how the changes were tested. Include: - Unit/integration test coverage - Manual validation steps - Links to CI reports (e.g., CIDA) if available - Note any pending tests --> ## Checklist - [ ] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. See merge request: Ascend/memcache!257	18 天前
example	[fix] 解决测试脚本可能core的问题 Co-authored-by: mrh1024<marunhua1@h-partners.com> # message auto-generated for no-merge-commit merge: !232 merge develop_fix into develop [fix] 解决测试脚本可能core的问题 Created-by: mrh1024 Commit-by: mrh1024 Merged-by: yrewzjsx Description: # Pull Request Template ## Description <!-- Provide a clear summary of the change, the problem it solves, and the technical approach. --> ## Related Issues <!-- Replace with actual issue numbers. Example: Closes #123, Related to #456 --> ## Type of Change - [ ] Bug fix - [ ] New feature - [ ] Performance optimization - [ ] Documentation update - [ ] Other (please describe):_______________ ## Testing <!-- Describe how the changes were tested. Include: - Unit/integration test coverage - Manual validation steps - Links to CI reports (e.g., CIDA) if available - Note any pending tests --> ## Checklist - [ ] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. See merge request: Ascend/memcache!232	1 个月前
pre-commit	ci: add pre-commit script Co-authored-by: shilinlee<836160610@qq.com> # message auto-generated for no-merge-commit merge: !216 merge pre-commit into develop ci: add pre-commit script Created-by: shilinlee_com Commit-by: shilinlee Merged-by: yrewzjsx Description: # Pull Request Template ## Description <!-- Provide a clear summary of the change, the problem it solves, and the technical approach. --> ## Related Issues <!-- Replace with actual issue numbers. Example: Closes #123, Related to #456 --> ## Type of Change - [ ] Bug fix - [ ] New feature - [ ] Performance optimization - [ ] Documentation update - [ ] Other (please describe):_______________ ## Testing <!-- Describe how the changes were tested. Include: - Unit/integration test coverage - Manual validation steps - Links to CI reports (e.g., CIDA) if available - Note any pending tests --> ## Checklist - [ ] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. See merge request: Ascend/memcache!216	1 个月前
script	fix cpp example compile error Co-authored-by: wonder1121<wangdan@huawei.com> # message auto-generated for no-merge-commit merge: !250 merge develop into develop fix cpp example compile error Created-by: wonder1121 Commit-by: wonder1121 Merged-by: chenyz6 Description: # Pull Request Template ## Description <!-- Provide a clear summary of the change, the problem it solves, and the technical approach. --> ## Related Issues <!-- Replace with actual issue numbers. Example: Closes #123, Related to #456 --> ## Type of Change - [ ] Bug fix - [ ] New feature - [ ] Performance optimization - [ ] Documentation update - [ ] Other (please describe):_______________ ## Testing <!-- Describe how the changes were tested. Include: - Unit/integration test coverage - Manual validation steps - Links to CI reports (e.g., CIDA) if available - Note any pending tests --> ## Checklist - [ ] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. See merge request: Ascend/memcache!250	21 天前
src	访问备份的map未加锁 Co-authored-by: yrewzjsx<zhangjinshi2@huawei.com> # message auto-generated for no-merge-commit merge: !261 merge develop into develop 访问备份的map未加锁 Created-by: yrewzjsx Commit-by: yrewzjsx Merged-by: chenyz6 Description: 访问备份的map未加锁 See merge request: Ascend/memcache!261	16 天前
test	【同步代码】register_buffer增加返回值判断 Co-authored-by: p3rry<penghaiqing1@huawei.com> # message auto-generated for no-merge-commit merge: !254 merge develop_0514_sync_register into develop 【同步代码】register_buffer增加返回值判断 Created-by: p3rry Commit-by: p3rry Merged-by: yrewzjsx Description: # Pull Request Template ## Description <!-- Provide a clear summary of the change, the problem it solves, and the technical approach. --> ## Related Issues <!-- Replace with actual issue numbers. Example: Closes #123, Related to #456 --> https://gitcode.com/Ascend/memfabric_hybrid/issues/122 ## Type of Change - [ ] Bug fix - [ ] New feature - [ ] Performance optimization - [ ] Documentation update - [ ] Other (please describe):_______________ ## Testing <!-- Describe how the changes were tested. Include: - Unit/integration test coverage - Manual validation steps - Links to CI reports (e.g., CIDA) if available - Note any pending tests --> ## Checklist - [ ] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. See merge request: Ascend/memcache!254	19 天前
.clang-format	!824 【A2】GVM 支持 Device RDMA	8 个月前
.gitattributes	大幅度提高本地ut脚本速度（几分钟到几秒） Co-authored-by: j00808874<jiangchanghong3@huawei.com> # message auto-generated for no-merge-commit merge: !94 merge mr into develop 大幅度提高本地ut脚本速度（几分钟到几秒） Created-by: j00808874 Commit-by: j00808874 Merged-by: chenyz6 Description: # Pull Request Template ## Description <!-- Provide a clear summary of the change, the problem it solves, and the technical approach. --> ## Related Issues <!-- Replace with actual issue numbers. Example: Closes #123, Related to #456 --> ## Type of Change - [ ] Bug fix - [ ] New feature - [ ] Performance optimization - [ ] Documentation update - [ ] Other (please describe):_______________ ## Testing <!-- Describe how the changes were tested. Include: - Unit/integration test coverage - Manual validation steps - Links to CI reports (e.g., CIDA) if available - Note any pending tests --> ## Checklist - [ ] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. See merge request: Ascend/memcache!94	4 个月前
.gitignore	feat: support meta setup config api Co-authored-by: shilinlee<836160610@qq.com> # message auto-generated for no-merge-commit merge: !191 merge setup_meta into develop feat: support meta setup config api Created-by: shilinlee_com Commit-by: shilinlee Merged-by: chenyz6 Description: # Pull Request Template ## Description feat: support meta setup config api ## Related Issues <!-- Replace with actual issue numbers. Example: Closes #123, Related to #456 --> ## Type of Change - [ ] Bug fix - [x] New feature - [ ] Performance optimization - [ ] Documentation update - [ ] Other (please describe):_______________ ## Testing ```python from memcache_hybrid import MetaConfig, MetaService def main(): config = MetaConfig() config.meta_service_url = "tcp://127.0.0.1:5000" config.log_level = "error" print(config) meta_service = MetaService assert meta_service.setup(config) == 0, "Meta service setup failed" meta_service.main() if __name__ == "__main__": main() ``` ![image.png](https://raw.gitcode.com/user-images/assets/7672915/9bf16a31-53e6-4c3d-85fe-be642d4a22d7/image.png 'image.png') ## Checklist - [ ] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. See merge request: Ascend/memcache!191	1 个月前
.gitmodules	HTTP Service in metaservice Co-authored-by: hansanyue<hansanyue@h-partners.com> # message auto-generated for no-merge-commit merge: !39 merge dev_1 into develop HTTP Service in metaservice Created-by: han-sanyue Commit-by: hansanyue Merged-by: yrewzjsx Description: 接口查询效果参考 doc/memcache_restful_api.md See merge request: Ascend/memcache!39	5 个月前
.pre-commit-config.yaml	ci: add pre-commit script Co-authored-by: shilinlee<836160610@qq.com> # message auto-generated for no-merge-commit merge: !216 merge pre-commit into develop ci: add pre-commit script Created-by: shilinlee_com Commit-by: shilinlee Merged-by: yrewzjsx Description: # Pull Request Template ## Description <!-- Provide a clear summary of the change, the problem it solves, and the technical approach. --> ## Related Issues <!-- Replace with actual issue numbers. Example: Closes #123, Related to #456 --> ## Type of Change - [ ] Bug fix - [ ] New feature - [ ] Performance optimization - [ ] Documentation update - [ ] Other (please describe):_______________ ## Testing <!-- Describe how the changes were tested. Include: - Unit/integration test coverage - Manual validation steps - Links to CI reports (e.g., CIDA) if available - Note any pending tests --> ## Checklist - [ ] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. See merge request: Ascend/memcache!216	1 个月前
CMakeLists.txt	chore: cmake set CMAKE_SKIP_BUILD_RPATH true in RELEASE build Co-authored-by: shilinlee<836160610@qq.com> # message auto-generated for no-merge-commit merge: !204 merge mmc_cmake_rpath into develop chore: cmake set CMAKE_SKIP_BUILD_RPATH true in RELEASE build Created-by: shilinlee_com Commit-by: shilinlee Merged-by: yrewzjsx Description: # Pull Request Template ## Description chore: cmake set CMAKE_SKIP_BUILD_RPATH true in RELEASE build ## Related Issues <!-- Replace with actual issue numbers. Example: Closes #123, Related to #456 --> ## Type of Change - [ ] Bug fix - [ ] New feature - [ ] Performance optimization - [ ] Documentation update - [ ] Other (please describe):_______________ ## Testing <!-- Describe how the changes were tested. Include: - Unit/integration test coverage - Manual validation steps - Links to CI reports (e.g., CIDA) if available - Note any pending tests --> ## Checklist - [x] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. See merge request: Ascend/memcache!204	1 个月前
LICENSE	!1113 更新license	6 个月前
README.md	update installation doc Co-authored-by: wonder1121<wangdan@huawei.com> # message auto-generated for no-merge-commit merge: !239 merge develop into develop update installation doc Created-by: wonder1121 Commit-by: wonder1121 Merged-by: yrewzjsx Description: # Pull Request Template ## Description <!-- Provide a clear summary of the change, the problem it solves, and the technical approach. --> ## Related Issues <!-- Replace with actual issue numbers. Example: Closes #123, Related to #456 --> ## Type of Change - [ ] Bug fix - [ ] New feature - [ ] Performance optimization - [ ] Documentation update - [ ] Other (please describe):_______________ ## Testing <!-- Describe how the changes were tested. Include: - Unit/integration test coverage - Manual validation steps - Links to CI reports (e.g., CIDA) if available - Note any pending tests --> ## Checklist - [ ] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. See merge request: Ascend/memcache!239	22 天前
VERSION	更新版本号 1.2.0 Co-authored-by: p3rry<penghaiqing1@huawei.com> # message auto-generated for no-merge-commit merge: !259 merge develop_version_0516 into develop 更新版本号 1.2.0 Created-by: p3rry Commit-by: p3rry Merged-by: yrewzjsx Description: # Pull Request Template ## Description <!-- Provide a clear summary of the change, the problem it solves, and the technical approach. --> ## Related Issues <!-- Replace with actual issue numbers. Example: Closes #123, Related to #456 --> ## Type of Change - [ ] Bug fix - [ ] New feature - [ ] Performance optimization - [ ] Documentation update - [ ] Other (please describe):_______________ ## Testing <!-- Describe how the changes were tested. Include: - Unit/integration test coverage - Manual validation steps - Links to CI reports (e.g., CIDA) if available - Note any pending tests --> ## Checklist - [ ] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. See merge request: Ascend/memcache!259	18 天前

High-performance distributed key-value cache

🔄Latest News

[2025/12] MemCache已作为vllm-ascend backend使能大模型推理加速，详情查看vllm-ascend开源社区，使用示例
[2025/11] MemCache项目于2025年11月开源，开源社区地址为：https://gitcode.com/Ascend/memcache

🔜 Roadmap

MemCache roadmap和版本分支策略详见： Roadmap

🎉概述

MemCache是针对LLM推理、GR推理场景设计的高性能分布式KVCache存储引擎，其主要特性包括：

基于对象操作的API：支持批量和非批量的put/get/exist/remove操作，支持多层结构的KV Block读写接口。
支持多副本：单个对象支持多副本放置到不同的LocalService，默认是单副本，支持put接口指定副本数量。
高带宽低时延：使用 MemFabric 作为多级内存和异构网络传输的底座，在Ascend硬件上，基于device_rdma(A2)、device_sdma(A3)、host_rdma(A2/A3) 等路径提供OneCopy跨机跨介质数据直接访问能力，满足高带宽，低时延的读写性能述求。在鲲鹏硬件上，支持host_urma(K5)。支持host_shm实现同节点共享内存通信。
支持扩缩容：支持LocalService动态加入和移除
HA能力：在K8S集群中，MetaService支持多活能力，支持元数据恢复，提供尽力而为的HA能力。

🧩核心组件

MemCache包含LocalService和MetaService两大核心组件：

MetaService：
- 负责管理整个集群中内存池空间的分配和管理，处理LocalService的加入与退出。
- MetaService作为独立进程运行，提供两种启动方式：python API启动；二进制启动，详见 whl安装使用和 run安装使用
- MetaService支持两种部署形态： 1、单点模式：MetaService由单个进程组成，部署方式简单，但存在单点故障的问题。如果MetaService进程崩溃或无法访问，系统将无法继续提供服务，直至重新恢复为止。 2、HA模式：该模式基于K8S的的ClusterIP Service和Lease资源构建，部署较为复杂，该模式会部署多个MetaService进程实例，实现多活高可用。部署详见 MetaService HA
LocalService：负责承担如下功能：
- 客户端：作为客户端，以whl/so形式作为共享库被应用进程加载调用API
- 内存提供者：负责提供一段连续的内存区域作为内存池空间的一部分，其内存可以被其他LocalService实例基于地址直接访问。

🔥性能表现

MemCache核心能力是提供大容量内存池和高性能的H2D、D2H、D2RH、RH2D数据访问能力，由于MemCache以 MemFabric 作为池化底座，所以支持RH2D、D2RH等OneCopy跨机跨介质数据直接访问能力，下图为RH2D对比其他中转路径的对比示意图。

基于OneCopy跨机跨介质数据直接访问的能力，MemCache在A2/A3做了相关性能测试如下：模拟构造DeepSeek-R1模型KV大小的block，单个block size为：61x128K + 61x16K = 8784KB ≈ 8.57MB，共122个离散地址。

使用2个昇腾A2节点(每节点8张卡)组成双机内存池进行读写测试性能如下：

使用2个昇腾A3节点(每节点8张卡16Die)组成双机内存池进行读写测试性能如下：

🔍目录结构

├── LICENSE                                 # LICENSE
├── .clang-format                           # 格式化配置
├── .gitmodules                             # git配置
├── .gitignore                              # git忽视配置文件
├── CMakeLists.txt                          # 项目的CMakeList
├── doc                                     # 文档目录
├── example                                 # 样例
│  ├── cpp                                  # c++样例
│  └── python                               # python样例
├── script                                  # 构建脚本
│  ├── build_and_pack_run.sh                # 编译+打包脚本
│  ├── build.sh                             # 编译脚本
│  ├── run_ut.sh                            # 编译+运行ut脚本
├── test                                    # test目录
│  ├── python                               # python测试用例
│  ├── k8s_deploy                           # k8s ha样例脚本
│  └── ut                                   # 单元测试用例
├── src                                     # 源码
│  ├── memcache                             # MemCache 源码
├── config                                  # 配置目录
│  ├── mmc-local.conf                       # 本地服务配置文件模板
│  ├── mmc-meta.conf                        # meta服务配置文件模板
├── README.md