已开启
设计 #22
caishengcheng创建于  6月3日
caishengcheng成员
6月3日 创建

结论:
1、convert本质上是一种新的quant_service,下称modelslim_convert,因此,命令行接口与msmodelslim quant保持一致,apiversion取modelslim_convert。
2、虚拟模型树、IR路由图是实现modelslim_convert的原子组件,位于modelslim_convert目录下。
3、egde的实现是Processor,因为没有Runner,需要手动构造BatchRequest,调用Processor.preprocess/process/postprocess。
4、保存通过IFormat,保存由于当前AscendV1Format缺失,可以暂时使用AsendV1Saver。
5、CompressedTensorFormatLoader实现modelslim_convert所提出的加载需求FormatLoaderInfra:1、将相应的目录加载为一个虚拟模型树。加载时使用包含若干个FormatLoadInfra实例的责任链,尝试加载模型,如[CompressedTensorFormatLoader, AscendV1FormatLoader(不急)]。

参考配置:

apiversion: modelslim_convert

spec:
preprocess:
- type: rename
patterns:
- from: ab*
to: cd*
- from: ef*
to: gh*
- type: convert
source:
- "model.language_model.layers..mlp.experts.gate_up_proj"
target:
- "model.language_model.layers.
.mlp.experts..gate_proj.weight"
- "model.language_model.layers.
.mlp.experts.*.up_proj.weight"
ops:
- type: chunk
dim: 0
- type: merge

linears:
  - match: 
		- "model.language_model.layers.*.linear_attn.*"
		- "model.language_model.layers.*.linear_attn.*"
		- "model.language_model.layers.*.linear_attn.*"
	target: W8A8_MXFP8
	route: 
		- BF16
		- W4A4_MXFP4
		- BF16
  - match: 
		- "model.language_model.layers.*.linear_attn.*"
		- "model.language_model.layers.*.linear_attn.*"
		- "model.language_model.layers.*.linear_attn.*"
	target: W8A8_MXFP8

save:
	- type: ascend_v1
	  part_file_size: 4


parellel:
	workers: 8
likedislike