结论: 1、convert本质上是一种新的quant_service,下称modelslim_convert,因此,命令行接口与msmodelslim quant保持一致,apiversion取modelslim_convert。 2、虚拟模型树、IR路由图是实现modelslim_convert的原子组件,位于modelslim_convert目录下。 3、egde的实现是Processor,因为没有Runner,需要手动构造BatchRequest,调用Processor.preprocess/process/postprocess。 4、保存通过IFormat,保存由于当前AscendV1Format缺失,可以暂时使用AsendV1Saver。 5、CompressedTensorFormatLoader实现modelslim_convert所提出的加载需求FormatLoaderInfra:1、将相应的目录加载为一个虚拟模型树。加载时使用包含若干个FormatLoadInfra实例的责任链,尝试加载模型,如[CompressedTensorFormatLoader, AscendV1FormatLoader(不急)]。
参考配置:
apiversion: modelslim_convert
spec: preprocess: - type: rename patterns: - from: ab* to: cd* - from: ef* to: gh* - type: convert source: - "model.language_model.layers..mlp.experts.gate_up_proj" target: - "model.language_model.layers..mlp.experts..gate_proj.weight" - "model.language_model.layers..mlp.experts.*.up_proj.weight" ops: - type: chunk dim: 0 - type: merge
linears: - match: - "model.language_model.layers.*.linear_attn.*" - "model.language_model.layers.*.linear_attn.*" - "model.language_model.layers.*.linear_attn.*" target: W8A8_MXFP8 route: - BF16 - W4A4_MXFP4 - BF16 - match: - "model.language_model.layers.*.linear_attn.*" - "model.language_model.layers.*.linear_attn.*" - "model.language_model.layers.*.linear_attn.*" target: W8A8_MXFP8 save: - type: ascend_v1 part_file_size: 4 parellel: workers: 8
结论:
1、convert本质上是一种新的quant_service,下称modelslim_convert,因此,命令行接口与msmodelslim quant保持一致,apiversion取modelslim_convert。
2、虚拟模型树、IR路由图是实现modelslim_convert的原子组件,位于modelslim_convert目录下。
3、egde的实现是Processor,因为没有Runner,需要手动构造BatchRequest,调用Processor.preprocess/process/postprocess。
4、保存通过IFormat,保存由于当前AscendV1Format缺失,可以暂时使用AsendV1Saver。
5、CompressedTensorFormatLoader实现modelslim_convert所提出的加载需求FormatLoaderInfra:1、将相应的目录加载为一个虚拟模型树。加载时使用包含若干个FormatLoadInfra实例的责任链,尝试加载模型,如[CompressedTensorFormatLoader, AscendV1FormatLoader(不急)]。
参考配置:
apiversion: modelslim_convert
spec:
preprocess:
- type: rename
patterns:
- from: ab*
to: cd*
- from: ef*
to: gh*
- type: convert
source:
- "model.language_model.layers..mlp.experts.gate_up_proj"
target:
- "model.language_model.layers..mlp.experts..gate_proj.weight"
- "model.language_model.layers..mlp.experts.*.up_proj.weight"
ops:
- type: chunk
dim: 0
- type: merge