MemCache是针对AI推理场景设计的高性能分布式KVCache缓存，针对昇腾超节点、服务器的实现了精细化性能调优，提供较好的High Availability能力, 同时对接到vllm-ascend、sglang、mindie等开源推理框架
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 13 天前 | ||
【对接UBSIO】部分修改 Co-authored-by: gcw_qYJeyWK4<jiangbo97@huawei.com> # message auto-generated for no-merge-commit merge: !281 merge develop into develop 【对接UBSIO】部分修改 Created-by: gcw_qYJeyWK4 Commit-by: gcw_qYJeyWK4 Merged-by: yrewzjsx Description: # Pull Request Template ## Description 将 ubsIoEnable 全局替换为 localSsdSize 实现 SSD 自动启用； 新增 MmcSsdBlobAllocator 子类以容量计数替代地址树分配； 修复淘汰回调改用 MoveDown(srcMediaType) 建立正确的 HBM→DRAM→SSD→NONE 逐级下沉链； 并将所有 SSD 直接 I/O 从 MetaService 进程剥离、统一通过 RPC 委托 ## Related Issues <!-- Replace with actual issue numbers. Example: Closes #123, Related to #456 --> ## Type of Change - [ ] Bug fix - [x] New feature - [ ] Performance optimization - [ ] Documentation update - [ ] Other (please describe):_______________ ## Testing 原UT以及新增UT均跑过 <!-- Describe how the changes were tested. Include: - Unit/integration test coverage - Manual validation steps - Links to CI reports (e.g., CIDA) if available - Note any pending tests --> ## Checklist - [ ] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. See merge request: Ascend/memcache!281 | 10 天前 | |
feat: add GVA-to-VA conversion before UBS I/O and update config naming Co-authored-by: gcw_qYJeyWK4<jiangbo97@huawei.com> # message auto-generated for no-merge-commit merge: !317 merge feature/ssd-gva-convert into develop feat: add GVA-to-VA conversion before UBS I/O and update config naming Created-by: gcw_qYJeyWK4 Commit-by: gcw_qYJeyWK4 Merged-by: yrewzjsx Description: # Pull Request Template ## Description <!-- Provide a clear summary of the change, the problem it solves, and the technical approach. --> ## Related Issues <!-- Replace with actual issue numbers. Example: Closes #123, Related to #456 --> ## Type of Change - [ ] Bug fix - [x] New feature - [ ] Performance optimization - [ ] Documentation update - [ ] Other (please describe):_______________ ## Testing <!-- Describe how the changes were tested. Include: - Unit/integration test coverage - Manual validation steps - Links to CI reports (e.g., CIDA) if available - Note any pending tests --> ## Checklist - [ ] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. See merge request: Ascend/memcache!317 | 8 天前 | |
feat: add GVA-to-VA conversion before UBS I/O and update config naming Co-authored-by: gcw_qYJeyWK4<jiangbo97@huawei.com> # message auto-generated for no-merge-commit merge: !317 merge feature/ssd-gva-convert into develop feat: add GVA-to-VA conversion before UBS I/O and update config naming Created-by: gcw_qYJeyWK4 Commit-by: gcw_qYJeyWK4 Merged-by: yrewzjsx Description: # Pull Request Template ## Description <!-- Provide a clear summary of the change, the problem it solves, and the technical approach. --> ## Related Issues <!-- Replace with actual issue numbers. Example: Closes #123, Related to #456 --> ## Type of Change - [ ] Bug fix - [x] New feature - [ ] Performance optimization - [ ] Documentation update - [ ] Other (please describe):_______________ ## Testing <!-- Describe how the changes were tested. Include: - Unit/integration test coverage - Manual validation steps - Links to CI reports (e.g., CIDA) if available - Note any pending tests --> ## Checklist - [ ] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. See merge request: Ascend/memcache!317 | 8 天前 | |
update readme Co-authored-by: yrewzjsx<zhangjinshi2@huawei.com> # message auto-generated for no-merge-commit merge: !309 merge develop into develop update readme Created-by: yrewzjsx Commit-by: yrewzjsx Merged-by: chenyz6 Description: # Pull Request Template ## Description <!-- Provide a clear summary of the change, the problem it solves, and the technical approach. --> ## Related Issues <!-- Replace with actual issue numbers. Example: Closes #123, Related to #456 --> ## Type of Change - [ ] Bug fix - [ ] New feature - [ ] Performance optimization - [x] Documentation update - [ ] Other (please describe):_______________ ## Testing <!-- Describe how the changes were tested. Include: - Unit/integration test coverage - Manual validation steps - Links to CI reports (e.g., CIDA) if available - Note any pending tests --> ## Checklist - [ ] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. See merge request: Ascend/memcache!309 | 1 天前 | |
direct拷贝方向导出 Co-authored-by: Hututu-w<wangyibing13@huawei.com> # message auto-generated for no-merge-commit merge: !304 merge rfc into develop direct拷贝方向导出 Created-by: Hututu-w Commit-by: Hututu-w Merged-by: liu1103xwxw Description: ## 变更说明 在 memcache_hybrid 模块级新增 L2G=0 / G2L=1 / G2H=2 / H2G=3 / AUTO=9 五个常量，同步修改文档及示例测试文件中的调用方式。 变更文件： 1. __init__.py — 新增常量定义，加入 __all__ 2. doc/memcache_python_api.md — SMEMB_COPY_* 替换为 memcache_hybrid.* 3. test_mmc_malloc.py — 手写 0/1 替换为 L2G/G2L 4. mutil_process.py — 手写 0/1/2 替换为 L2G/G2L/G2H See merge request: Ascend/memcache!304 | 7 天前 | |
ci: add pre-commit script Co-authored-by: shilinlee<836160610@qq.com> # message auto-generated for no-merge-commit merge: !216 merge pre-commit into develop ci: add pre-commit script Created-by: shilinlee_com Commit-by: shilinlee Merged-by: yrewzjsx Description: # Pull Request Template ## Description <!-- Provide a clear summary of the change, the problem it solves, and the technical approach. --> ## Related Issues <!-- Replace with actual issue numbers. Example: Closes #123, Related to #456 --> ## Type of Change - [ ] Bug fix - [ ] New feature - [ ] Performance optimization - [ ] Documentation update - [ ] Other (please describe):_______________ ## Testing <!-- Describe how the changes were tested. Include: - Unit/integration test coverage - Manual validation steps - Links to CI reports (e.g., CIDA) if available - Note any pending tests --> ## Checklist - [ ] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. See merge request: Ascend/memcache!216 | 2 个月前 | |
当前修改仅涉及打包脚本，不涉及源码变更，所以没有ut的变更 Co-authored-by: gcw_qYJeyWK4<jiangbo97@huawei.com> # message auto-generated for no-merge-commit merge: !312 merge feature/ubsio-packaging into develop feature：打包带上ubsio Created-by: gcw_qYJeyWK4 Commit-by: gcw_qYJeyWK4 Merged-by: yrewzjsx Description: # Pull Request Template ## Description  ## Related Issues <!-- Replace with actual issue numbers. Example: Closes #123, Related to #456 --> ## Type of Change - [ ] Bug fix - [x] New feature - [ ] Performance optimization - [ ] Documentation update - [ ] Other (please describe):_______________ ## Testing <!-- Describe how the changes were tested. Include: - Unit/integration test coverage - Manual validation steps - Links to CI reports (e.g., CIDA) if available - Note any pending tests --> ## Checklist - [ ] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. See merge request: Ascend/memcache!312 | 8 天前 | |
fix: 修复local侧配置文件参数chunk size不生效的问题 Co-authored-by: t30062639<tanghongcheng1@h-partners.com> # message auto-generated for no-merge-commit merge: !325 merge 0623fix into develop fix: 修复local侧配置文件参数chunk size不生效的问题 Created-by: thc72549 Commit-by: t30062639 Merged-by: yrewzjsx Description: # Pull Request Template ## Description <!-- Provide a clear summary of the change, the problem it solves, and the technical approach. --> 配置项字段读取错误，修正为目标字段 ## Related Issues <!-- Replace with actual issue numbers. Example: Closes #123, Related to #456 --> ## Type of Change - [x] Bug fix - [ ] New feature - [ ] Performance optimization - [ ] Documentation update - [ ] Other (please describe):_______________ ## Testing <!-- Describe how the changes were tested. Include: - Unit/integration test coverage - Manual validation steps - Links to CI reports (e.g., CIDA) if available - Note any pending tests --> ## Checklist - [x] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. See merge request: Ascend/memcache!325 | 2 天前 | |
完善GVA直接读写场景下的租约管理 Co-authored-by: Oranbean258<tanghongcheng1@h-partners.com> # message auto-generated for no-merge-commit merge: !262 merge 0515fix_client_lease into develop 完善GVA直接读写场景下的租约管理 Created-by: thc72549 Commit-by: t30062639;Oranbean258 Merged-by: yrewzjsx Description: # Pull Request Template ## Description ### 变更说明 本次 PR 主要完善了memcache在 GVA 直读直写场景下的租约管理闭环，覆盖了 batch_alloc、batch_get_key_info、batch_copy 这条链路在 client 和 meta 两侧的状态同步、租约校验和过期清理能力。同时补充了 meta 侧租约 TTL 的配置化支持，以及对应的接口、协议和测试用例。 ### 主要变更 1. client 侧新增本地 GVA 租约跟踪能力。引入 LocalGvaBlobTracker/LocalGvaBlobInfo，在 BatchMalloc 后登记 GVA 信息，在 Query/BatchQuery 携带 MMC_QUERY_FLAG_GVA_READ_START 时刷新本地租约状态，在 BatchCopy 读路径中校验本地可读状态、租约是否过期以及未读 hole 区间，并在读完整个 blob 后上报 MMC_READ_FINISH。 2. client 侧新增巡检清理过期租约的周期任务。MmcClientDefault 启动后会创建后台线程，定期扫描本地 tracker 中已过期的读租约，尝试 claim READ_FINISH 并回调 meta；成功后移除本地记录，失败则释放 claim 以便后续重试。 3. client 侧补充租约过期校验。除了 GVA BatchCopy 读路径外，普通 Get/BatchGet 在数据拷贝完成后也会结合 meta 返回的剩余 TTL 做本地过期检查，避免“数据读到了但租约已失效”仍返回成功。 4. meta 侧新增基于 GVA 的反向索引。新增 MmcMetaGvaIndex，按 segment 维护 pendingWrite 索引，替代原来简单的 gva2updateMap_，用于支撑按 GVA 更新写状态、启动读租约和结束读租约。 5. meta 侧统一了GVA直接读写场景下按 GVA 的状态更新入口。新增 UpdateBlobActionByGva，统一处理 MMC_WRITE_OK、MMC_WRITE_FAIL、MMC_READ_FINISH，并在 Query/BatchQuery 带 flag 场景下返回剩余租约 TTL。 6. 新增 meta 租约 TTL 配置项。增加 ock.mmc.meta.lease_ttl_ms / leaseTtlMs，贯通默认配置、配置解析、meta 启动、Python 绑定和文档输出，meta 初始化时可将配置值传递到 MmcMetaManager，后续加租约/续租约使用该默认 TTL。 7. 扩展了协议和上层接口。QueryRequest/BatchQueryRequest 增加 operateId 和 flag，BatchUpdateBlobRequest 增加 operateIds，BatchUpdateResponse 增加 leaseTimeoutTtlMs 返回值；GetKeyInfo/BatchGetKeyInfo 也新增 flag 参数，支持从 Python/C++ 上层触发 GVA_READ_START。 ### 自验证 test\python\memcache\mock_server\test_mmc_malloc.py文件中增加多进程多client的gva直接读写测试用例，在A3上验证通过  <!-- Provide a clear summary of the change, the problem it solves, and the technical approach. --> ## Related Issues <!-- Replace with actual issue numbers. Example: Closes #123, Related to #456 --> ## Type of Change - [ ] Bug fix - [x] New feature - [ ] Performance optimization - [ ] Documentation update - [ ] Other (please describe):_______________ ## Testing <!-- Describe how the changes were tested. Include: - Unit/integration test coverage - Manual validation steps - Links to CI reports (e.g., CIDA) if available - Note any pending tests --> ## Checklist - [x] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. See merge request: Ascend/memcache!262 | 2 天前 | |
!824 【A2】GVM 支持 Device RDMA | 9 个月前 | |
大幅度提高本地ut脚本速度（几分钟到几秒） Co-authored-by: j00808874<jiangchanghong3@huawei.com> # message auto-generated for no-merge-commit merge: !94 merge mr into develop 大幅度提高本地ut脚本速度（几分钟到几秒） Created-by: j00808874 Commit-by: j00808874 Merged-by: chenyz6 Description: # Pull Request Template ## Description <!-- Provide a clear summary of the change, the problem it solves, and the technical approach. --> ## Related Issues <!-- Replace with actual issue numbers. Example: Closes #123, Related to #456 --> ## Type of Change - [ ] Bug fix - [ ] New feature - [ ] Performance optimization - [ ] Documentation update - [ ] Other (please describe):_______________ ## Testing <!-- Describe how the changes were tested. Include: - Unit/integration test coverage - Manual validation steps - Links to CI reports (e.g., CIDA) if available - Note any pending tests --> ## Checklist - [ ] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. See merge request: Ascend/memcache!94 | 5 个月前 | |
feat: support meta setup config api Co-authored-by: shilinlee<836160610@qq.com> # message auto-generated for no-merge-commit merge: !191 merge setup_meta into develop feat: support meta setup config api Created-by: shilinlee_com Commit-by: shilinlee Merged-by: chenyz6 Description: # Pull Request Template ## Description feat: support meta setup config api ## Related Issues <!-- Replace with actual issue numbers. Example: Closes #123, Related to #456 --> ## Type of Change - [ ] Bug fix - [x] New feature - [ ] Performance optimization - [ ] Documentation update - [ ] Other (please describe):_______________ ## Testing python from memcache_hybrid import MetaConfig, MetaService def main(): config = MetaConfig() config.meta_service_url = "tcp://127.0.0.1:5000" config.log_level = "error" print(config) meta_service = MetaService assert meta_service.setup(config) == 0, "Meta service setup failed" meta_service.main() if __name__ == "__main__": main()  ## Checklist - [ ] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. See merge request: Ascend/memcache!191 | 2 个月前 | |
当前修改仅涉及打包脚本，不涉及源码变更，所以没有ut的变更 Co-authored-by: gcw_qYJeyWK4<jiangbo97@huawei.com> # message auto-generated for no-merge-commit merge: !312 merge feature/ubsio-packaging into develop feature：打包带上ubsio Created-by: gcw_qYJeyWK4 Commit-by: gcw_qYJeyWK4 Merged-by: yrewzjsx Description: # Pull Request Template ## Description  ## Related Issues <!-- Replace with actual issue numbers. Example: Closes #123, Related to #456 --> ## Type of Change - [ ] Bug fix - [x] New feature - [ ] Performance optimization - [ ] Documentation update - [ ] Other (please describe):_______________ ## Testing <!-- Describe how the changes were tested. Include: - Unit/integration test coverage - Manual validation steps - Links to CI reports (e.g., CIDA) if available - Note any pending tests --> ## Checklist - [ ] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. See merge request: Ascend/memcache!312 | 8 天前 | |
ci: add pre-commit script Co-authored-by: shilinlee<836160610@qq.com> # message auto-generated for no-merge-commit merge: !216 merge pre-commit into develop ci: add pre-commit script Created-by: shilinlee_com Commit-by: shilinlee Merged-by: yrewzjsx Description: # Pull Request Template ## Description <!-- Provide a clear summary of the change, the problem it solves, and the technical approach. --> ## Related Issues <!-- Replace with actual issue numbers. Example: Closes #123, Related to #456 --> ## Type of Change - [ ] Bug fix - [ ] New feature - [ ] Performance optimization - [ ] Documentation update - [ ] Other (please describe):_______________ ## Testing <!-- Describe how the changes were tested. Include: - Unit/integration test coverage - Manual validation steps - Links to CI reports (e.g., CIDA) if available - Note any pending tests --> ## Checklist - [ ] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. See merge request: Ascend/memcache!216 | 2 个月前 | |
当前修改仅涉及打包脚本，不涉及源码变更，所以没有ut的变更 Co-authored-by: gcw_qYJeyWK4<jiangbo97@huawei.com> # message auto-generated for no-merge-commit merge: !312 merge feature/ubsio-packaging into develop feature：打包带上ubsio Created-by: gcw_qYJeyWK4 Commit-by: gcw_qYJeyWK4 Merged-by: yrewzjsx Description: # Pull Request Template ## Description  ## Related Issues <!-- Replace with actual issue numbers. Example: Closes #123, Related to #456 --> ## Type of Change - [ ] Bug fix - [x] New feature - [ ] Performance optimization - [ ] Documentation update - [ ] Other (please describe):_______________ ## Testing <!-- Describe how the changes were tested. Include: - Unit/integration test coverage - Manual validation steps - Links to CI reports (e.g., CIDA) if available - Note any pending tests --> ## Checklist - [ ] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. See merge request: Ascend/memcache!312 | 8 天前 | |
!1113 更新license | 6 个月前 | |
update readme Co-authored-by: yrewzjsx<zhangjinshi2@huawei.com> # message auto-generated for no-merge-commit merge: !309 merge develop into develop update readme Created-by: yrewzjsx Commit-by: yrewzjsx Merged-by: chenyz6 Description: # Pull Request Template ## Description <!-- Provide a clear summary of the change, the problem it solves, and the technical approach. --> ## Related Issues <!-- Replace with actual issue numbers. Example: Closes #123, Related to #456 --> ## Type of Change - [ ] Bug fix - [ ] New feature - [ ] Performance optimization - [x] Documentation update - [ ] Other (please describe):_______________ ## Testing <!-- Describe how the changes were tested. Include: - Unit/integration test coverage - Manual validation steps - Links to CI reports (e.g., CIDA) if available - Note any pending tests --> ## Checklist - [ ] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. See merge request: Ascend/memcache!309 | 1 天前 | |
更新版本号 1.2.0 Co-authored-by: p3rry<penghaiqing1@huawei.com> # message auto-generated for no-merge-commit merge: !259 merge develop_version_0516 into develop 更新版本号 1.2.0 Created-by: p3rry Commit-by: p3rry Merged-by: yrewzjsx Description: # Pull Request Template ## Description <!-- Provide a clear summary of the change, the problem it solves, and the technical approach. --> ## Related Issues <!-- Replace with actual issue numbers. Example: Closes #123, Related to #456 --> ## Type of Change - [ ] Bug fix - [ ] New feature - [ ] Performance optimization - [ ] Documentation update - [ ] Other (please describe):_______________ ## Testing <!-- Describe how the changes were tested. Include: - Unit/integration test coverage - Manual validation steps - Links to CI reports (e.g., CIDA) if available - Note any pending tests --> ## Checklist - [ ] I have performed a self-review of my own code. - [ ] I have updated the documentation. - [ ] I have added tests to prove my changes are effective. See merge request: Ascend/memcache!259 | 1 个月前 |
🔄Latest News
-
[2026/06] MemCache使能推理PrefixCache加速案例实践，请关注wiki主页案例库获取最新案例
-
[2025/12] MemCache已作为vllm-ascend backend使能大模型推理加速，详情查看vllm-ascend开源社区，使用示例
-
[2025/11] MemCache项目于2025年11月开源，开源社区地址为：https://gitcode.com/Ascend/memcache
🔜 Roadmap&发布策略
MemCache roadmap详见： Roadmap
MemCache 分支发布策略：分支发布策略
🎉概述
MemCache是针对LLM推理、GR推理场景设计的高性能分布式KVCache存储引擎，其主要特性包括：
- 基于对象操作的API：支持批量和非批量的put/get/exist/remove操作，支持多层结构的KV Block读写接口。
- 多层缓存池：支持HBM，DDR，SSD组成多级缓存池，多级之间通过淘汰和预取进行数据冷热交换。
- 高带宽低时延：使用 MemFabric 作为多级内存和异构网络传输的底座，在Ascend硬件上，基于device_rdma(A2)、device_sdma(A3)、host_rdma(A2/A3)、device_urma(A5) 等路径提供OneCopy跨机跨介质数据直接访问能力，满足高带宽，低时延的读写性能述求。在鲲鹏硬件上，支持host_urma(K5)。支持host_shm实现同节点共享内存通信。
- 支持扩缩容：支持LocalService动态加入和移除
- HA能力：在K8S集群中，MetaService支持多活能力，支持元数据恢复，提供尽力而为的HA能力。
🧩核心组件
MemCache包含LocalService和MetaService两大核心组件：
-
MetaService：
- 负责管理整个集群中内存池空间的分配和管理，处理LocalService的加入与退出。
- MetaService作为独立进程运行，提供两种启动方式：python API启动；二进制启动，详见 whl安装使用 和 run安装使用
- MetaService支持两种部署形态：
1、单点模式：MetaService由单个进程组成，部署方式简单，但存在单点故障的问题。如果MetaService进程崩溃或无法访问，系统将无法继续提供服务，直至重新恢复为止。
2、HA模式：该模式基于K8S的的ClusterIP Service和Lease资源构建，部署较为复杂，该模式会部署多个MetaService进程实例，实现多活高可用。部署详见怎么部署一个MemCache的HA集群
-
LocalService：负责承担如下功能：
- 客户端：作为客户端，以whl/so形式作为共享库被应用进程加载调用API
- 内存提供者：负责提供一段连续的内存区域作为内存池空间的一部分，其内存可以被其他LocalService实例基于地址直接访问。
🔥性能表现
MemCache核心能力是提供大容量内存池和高性能的H2D、D2H、D2RH、RH2D数据访问能力，由于MemCache以 MemFabric 作为池化底座，所以支持RH2D、D2RH等OneCopy跨机跨介质数据直接访问能力，下图为RH2D对比其他中转路径的对比示意图。
基于OneCopy跨机跨介质数据直接访问的能力，MemCache在A2/A3做了相关性能测试如下： 模拟构造DeepSeek-R1模型KV大小的block，单个block size为：61x128K + 61x16K = 8784KB ≈ 8.57MB，共122个离散地址。
- 使用2个昇腾A2节点(每节点8张卡)组成双机内存池进行读写测试性能如下：
- 使用2个昇腾A3节点(每节点8张卡16Die)组成双机内存池进行读写测试性能如下：
🚀快速入门
请访问以下文档获取简易教程。
- 安装使用：whl安装和使用（适用于Python用户），run编译、安装和使用（适用于C++用户）
- 配置文件：涉及MetaService、LocalService公共配置
- 样例执行：介绍如何端到端执行样例代码，包括C++和Python样例
📑学习教程
📦软件硬件配套说明
- MemCache软件依赖 MemFabric，相关配套与MemFabric相同
📌FAQ
常见问题请参考：FAQ