Closed
[Requirement|需求建议]: mla_prolog算子增加torch_extension接口 #4542
zhengyuhao3Add on Aug 24Closed at Aug 24
zhengyuhao3
Aug 24 Comments:
Aug 24 Comments:
/assign


Aug 24 Assign zhengyuhao3 as responsible person
Aug 24 Associated pull request
Aug 24 Close issue
Aug 24 Added label: resolved
Thanks for sending an requirement! Please fill in the following template to help quickly solve your problem.
Backgroud(背景信息)
Kimi-K3模型MLA前置操作使用mla_prolog算子替代原有的小算子拼接,通过融合算子减少HBM搬运提升性能
通过torch_extension提供对外使用接口,用于算子调用
Benefit / Necessity (价值/作用)
价值:提升Kimi-k3模型在整网使用中的性能,穿刺性能达1.68x小算子拼接
应用场景:替换模型使用中对于mla前置处理的小算子拼接实现
Design(设计方案)
提供torch_extension接口,原型如下,通过传入ropeSin/ropeCos为空来控制是否执行ROPE计算操作
def mla_prolog_meta(
token_x,
weight_dq,
weight_uq_qr,
weight_uk,
weight_dkv_kr,
rmsnorm_gamma_cq,
rmsnorm_gamma_ckv,
kv_cache,
kr_cache,
*,
rope_sin=None,
rope_cos=None,
cache_index=None,
dequant_scale_x=None,
dequant_scale_w_dq=None,
dequant_scale_w_uq_qr=None,
dequant_scale_w_dkv_kr=None,
quant_scale_ckv=None,
quant_scale_ckr=None,
smooth_scales_cq=None,
actual_seq_len=None,
k_nope_clip_alpha=None,
rmsnorm_epsilon_cq=1e-05,
rmsnorm_epsilon_ckv=1e-05,
cache_mode=DEFAULT_CACHE_MODE,
query_norm_flag=False,
weight_quant_mode=0,
kv_cache_quant_mode=0,
query_quant_mode=0,
ckvkr_repo_mode=0,
quant_scale_repo_mode=0,
tile_size=128,
qc_qr_scale=1.0,
kc_scale=1.0,
token_x_dtype=None,
weight_dq_dtype=None,
weight_uq_qr_dtype=None,
weight_dkv_kr_dtype=None,
kv_cache_dtype=None,
)