Closed
[Requirement|需求建议]: mla_prolog算子增加torch_extension接口 #4542
zhengyuhao3Add on  Aug 24Closed at  Aug 24
zhengyuhao3Collaborators
Aug 24 Create

Thanks for sending an requirement! Please fill in the following template to help quickly solve your problem.

Backgroud(背景信息)

Kimi-K3模型MLA前置操作使用mla_prolog算子替代原有的小算子拼接,通过融合算子减少HBM搬运提升性能
通过torch_extension提供对外使用接口,用于算子调用

Benefit / Necessity (价值/作用)

价值:提升Kimi-k3模型在整网使用中的性能,穿刺性能达1.68x小算子拼接
应用场景:替换模型使用中对于mla前置处理的小算子拼接实现

Design(设计方案)

提供torch_extension接口,原型如下,通过传入ropeSin/ropeCos为空来控制是否执行ROPE计算操作
def mla_prolog_meta(
token_x,
weight_dq,
weight_uq_qr,
weight_uk,
weight_dkv_kr,
rmsnorm_gamma_cq,
rmsnorm_gamma_ckv,
kv_cache,
kr_cache,
*,
rope_sin=None,
rope_cos=None,
cache_index=None,
dequant_scale_x=None,
dequant_scale_w_dq=None,
dequant_scale_w_uq_qr=None,
dequant_scale_w_dkv_kr=None,
quant_scale_ckv=None,
quant_scale_ckr=None,
smooth_scales_cq=None,
actual_seq_len=None,
k_nope_clip_alpha=None,
rmsnorm_epsilon_cq=1e-05,
rmsnorm_epsilon_ckv=1e-05,
cache_mode=DEFAULT_CACHE_MODE,
query_norm_flag=False,
weight_quant_mode=0,
kv_cache_quant_mode=0,
query_quant_mode=0,
ckvkr_repo_mode=0,
quant_scale_repo_mode=0,
tile_size=128,
qc_qr_scale=1.0,
kc_scale=1.0,
token_x_dtype=None,
weight_dq_dtype=None,
weight_uq_qr_dtype=None,
weight_dkv_kr_dtype=None,
kv_cache_dtype=None,
)

likedislike
zhengyuhao3Collaborators
Aug 24 Comments:

/assign

likedislike
CANN-robotCANN-robotCollaborators
Aug 24 Assign zhengyuhao3 as responsible person
Zzhengyuhao3Collaborators
Aug 24 Associated pull request
CANN-robotCANN-robotCollaborators
Aug 24 Close issue
CANN-robotCANN-robotCollaborators
Aug 24 Added label: resolved