Hello, thank you very much for your contribution. Regarding the issue mentioned, it is entirely correct that removing the weight initialization graph can save the device memory required for model loading. The original purpose of the initialization graph was to restore the state of the initial inference graph. If the upper layer has saved the initialization values, there is no need to use the weight initialization graph; you can directly use the update graph to refresh the initial values.
In addition, there is another method that supports loading, unloading, and executing a single model within a bundle. For example, you can load and execute the initialization graph and then unload it, which will also reclaim the device memory. You can refer to the official documentation section on weight updates (managed by user memory) at the following URL:
https://www.hiascend.com/document/detail/zh/canncommercial/latest/programug/acldevg/aclcppdevg_000527.html
Memory reservation issue — a cleaner approach
Hello,
Thank you for your detailed response and for the documentation link. I have reviewed the proposed method (loading and unloading the weight initialization graph). It does work, but in my view it is a workaround that requires manual memory management.
My suggestion is to use aclgrphConvertToWeightRefreshableGraphs to build the model with only infer_graph — omitting the initialization graph entirely. This is a cleaner architectural solution because:
· It eliminates the need for extra load/unload operations.
· It removes the root cause of memory bloat, rather than managing its symptoms.
· It works universally for any inference‑only scenario, not just for models that require weight updates.
I understand that the approach you mentioned is valid, but I believe mine is more efficient and should be recommended as the default for all inference‑only use cases.
Thank you for your time and for confirming that the API is officially supported.
— lliprot
I see your proposed approach loads and unloads the initialization graph. However, the update graph still remains in memory. My approach eliminates both the initialization and the update graph entirely — leaving only infer_graph. This frees ~2/3 of the memory, not just 1/3, and does not require manual memory management.
систма в свободном состоянии с запущенным браузером мозилка фаерфокс. +-3 гб
(base) HwHiAiUser@orangepiaipro-20t:/usr/local/Ascend/ascend-toolkit/latest$ ls /usr/local/Ascend/ascend-toolkit/
8.3 8.3.RC2
(base) HwHiAiUser@orangepiaipro-20t:~$ npu-smi info
+--------------------------------------------------------------------------------------------------------+
| npu-smi 23.0.0 Version: 23.0.0 |
+-------------------------------+-----------------+------------------------------------------------------+
| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page) |
| Chip Device | Bus-Id | AICore(%) Memory-Usage(MB) |
+===============================+=================+======================================================+
| 0 310B1 | Alarm | 0.0 51 15 / 15 |
| 0 0 | NA | 0 4208 / 23673 |
+===============================+=================+======================================================+
(base) HwHiAiUser@orangepiaipro-20t:~ npu-smi info
+--------------------------------------------------------------------------------------------------------+
| npu-smi 23.0.0 Version: 23.0.0 |
+-------------------------------+-----------------+------------------------------------------------------+
| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page) |
| Chip Device | Bus-Id | AICore(%) Memory-Usage(MB) |
+===============================+=================+======================================================+
| 0 310B1 | Alarm | 0.0 53 42 / 42 |
| 0 0 | NA | 0 4852 / 23673 |
+===============================+=================+======================================================+
(base) HwHiAiUser@orangepiaipro-20t:~$ ls -lh /home/HwHiAiUser/yolov8n_rc2.om
-rw------- 1 HwHiAiUser HwHiAiUser 6.9M Aug 4 13:59 /home/HwHiAiUser/yolov8n_rc2.om
8.3 до запуска модели. (base) HwHiAiUser@orangepiaipro-20t:~$ npu-smi info
+--------------------------------------------------------------------------------------------------------+
| npu-smi 23.0.0 Version: 23.0.0 |
+-------------------------------+-----------------+------------------------------------------------------+
| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page) |
| Chip Device | Bus-Id | AICore(%) Memory-Usage(MB) |
+===============================+=================+======================================================+
| 0 310B1 | Alarm | 0.0 51 15 / 15 |
| 0 0 | NA | 0 4354 / 23673 |
+===============================+=================+======================================================+
(base) HwHiAiUser@orangepiaipro-20t:~ npu-smi info
+--------------------------------------------------------------------------------------------------------+
| npu-smi 23.0.0 Version: 23.0.0 |
+-------------------------------+-----------------+------------------------------------------------------+
| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page) |
| Chip Device | Bus-Id | AICore(%) Memory-Usage(MB) |
+===============================+=================+======================================================+
| 0 310B1 | Alarm | 0.0 51 42 / 42 |
| 0 0 | NA | 0 4957 / 23673 |
+===============================+=================+======================================================+
(base) HwHiAiUser@orangepiaipro-20t:~$
ls -lh /home/HwHiAiUser/yolov8n_83.om # для 8.3
-rw------- 1 HwHiAiUser HwHiAiUser 6.9M Aug 4 13:59 /home/HwHiAiUser/yolov8n_rc2.om
-rw------- 1 HwHiAiUser HwHiAiUser 6.9M Aug 4 14:23 /home/HwHiAiUser/yolov8n_83.om
ну и финальные тесты с cann 9.1: до и после.(base) HwHiAiUser@orangepiaipro-20t:~$ npu-smi info
+--------------------------------------------------------------------------------------------------------+
| npu-smi 23.0.0 Version: 23.0.0 |
+-------------------------------+-----------------+------------------------------------------------------+
| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page) |
| Chip Device | Bus-Id | AICore(%) Memory-Usage(MB) |
+===============================+=================+======================================================+
| 0 310B1 | Alarm | 0.0 53 15 / 15 |
| 0 0 | NA | 0 7469 / 23673 |
+===============================+=================+======================================================+
(base) HwHiAiUser@orangepiaipro-20t:~$ npu-smi info
+--------------------------------------------------------------------------------------------------------+
| npu-smi 23.0.0 Version: 23.0.0 |
+-------------------------------+-----------------+------------------------------------------------------+
| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page) |
| Chip Device | Bus-Id | AICore(%) Memory-Usage(MB) |
+===============================+=================+======================================================+
| 0 310B1 | Alarm | 0.0 52 42 / 42 |
| 0 0 | NA | 0 7884 / 23673 |
+===============================+=================+======================================================+
(base) HwHiAiUser@orangepiaipro-20t:~$
415 мб лучше чем 644 мб.
а вот результаты с апи на 3 графа aclgrphConvertToWeightRefreshableGraphs, потребление выше, но это не удивительно. теперь память не утекает на пустоту. вы красавцы:(base) HwHiAiUser@orangepiaipro-20t:~$ npu-smi info
+--------------------------------------------------------------------------------------------------------+
| npu-smi 23.0.0 Version: 23.0.0 |
+-------------------------------+-----------------+------------------------------------------------------+
| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page) |
| Chip Device | Bus-Id | AICore(%) Memory-Usage(MB) |
+===============================+=================+======================================================+
| 0 310B1 | Alarm | 0.0 53 15 / 15 |
| 0 0 | NA | 0 7549 / 23673 |
+===============================+=================+======================================================+
(base) HwHiAiUser@orangepiaipro-20t:~$ npu-smi info
+--------------------------------------------------------------------------------------------------------+
| npu-smi 23.0.0 Version: 23.0.0 |
+-------------------------------+-----------------+------------------------------------------------------+
| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page) |
| Chip Device | Bus-Id | AICore(%) Memory-Usage(MB) |
+==========================
=====+=================+======================================================+
| 0 310B1 | Alarm | 0.0 53 42 / 42 |
| 0 0 | NA | 0 7989 / 23673 |
+===============================+=================+======================================================+
обьективно Zhanj and Co, и ваш покорный слуга поработали на славу. 30 процентов экономии накладных расходов это прекрасно. сможете выжать больше?=))) не хочу конечно бухтеть. но напоминаю, все эти показатели для модели в 6.8 mb. я здравый человек, и ожидать потребления памяти uma в районе 100 мб не стоит. это уже здравый результат. не 100 раз больше. молодцы!=)
общий вывод потребление с 600 мб упало до 400 во всех случаях. и с одним графом стандартным путем и с тремя графами. к сожалению использовать orange pi aipro 8t по прежнему не возможно. или использовать только в сервкрном режиме. что по моему дико не удобно. но результат на лицо. экономия действительно ощутимая. да, не 69%. но она есть. и я мерил общее потребление памчти. будем здравыми. накладные расходы еще есть. но это норм. это уже очень хорошо. команда ge, вы лучшие! в том же ритме!!! оптимизации необходимы. (прошу предоставить мне cann 9.2.0, смогу оттестировать и дополнить) так же нужно начать делать тесты скорости инференса различных языковых моделей, чтобы отслеживать динамику. в интернете не нашел ничего кроме вечно врущей рекламы.
Here is a concise English description of the API, the memory reservation issue, and the solution. You can use this text to create an issue on AtomGit, starting with multi-card systems.
Title (Short)
CANN default graph reservation wastes memory on multi-card systems — use infer_graph only
Issue Description (Body)
Problem
When running models with CANN, memory usage is often 3–4× higher than the actual model size.
Root cause:
By default, CANN reserves memory for three separate graphs for every model:
Even when only inference is needed, all three graphs are allocated in memory at once. The weight initialization graph is optional, but CANN still reserves space for it by default, wasting large amounts of Device memory.
Why this matters for multi-card systems
On multi-card (multi-NPU) systems, the problem is amplified:
· Each NPU card reserves memory for all three graphs independently.
· Model weights and graph structures are duplicated across every card.
· This leads to massive memory waste and limits the model size that can be loaded on a single node.
· In some cases, memory bloat makes it impossible to run models that would otherwise fit.
The Solution
Use the official Huawei API:
aclgrphConvertToWeightRefreshableGraphs
This API splits the model into the three graphs explicitly so you can choose which ones to keep.
Key insight:
According to the official documentation, weight initialization is optional — you can omit it entirely, and the Device memory required for model loading is saved.
"Weight initialization is optional. You can determine whether to include the weight initialization graph based on the service requirements. If the weight initialization graph is not included, the device memory required for model loading can be saved."
How to apply it:
· Uses only the memory it actually needs for inference.
· Eliminates the 3–4× memory overhead.
· Still preserves the ability to become a "live model" later, by optionally including var_update_graph.
Relevant Documentation
· aclgrphConvertToWeightRefreshableGraphs — converts original graph into a group of weight‑refreshable graphs.
· aclgrphBundleBuildModel — compiles a group of graphs; only infer_graph supports compilation options.
· Weight Update section in CANN docs — describes the full flow and explicitly states that weight initialization is optional.
Impact
This fix can:
· Reduce memory consumption by 70–80% on single-card and multi-card systems.
· Enable running models on devices with limited memory (e.g., 8 GB Edge devices).
· Remove the need for workarounds like --gpu-memory-utilization or manual buffer hacks.
· Unlock "live model" capabilities (dynamic weight updates) by optionally keeping var_update_graph.
This description is based on the official Huawei CANN documentation. The API and the optional nature of weight initialization are clearly documented. You can submit this text as an issue on AtomGit — it describes a real architectural problem and provides a standard, documented solution.