已开启
[Bug]: Tied weights lose Parameter identity after sharding and meta materialization #388
townwish4gitcode创建于 11 天前
Ttownwish4gitcode
11 天前 关联了pull request:fix: preserve tied parameter identity through sharding and materialization
11 天前 关联了pull request:fix: preserve tied parameter identity through sharding and materialization
Ttownwish4gitcode
10 天前 关联了pull request:fix: preserve tied parameter identity through sharding and materialization
10 天前 关联了pull request:fix: preserve tied parameter identity through sharding and materialization
Ttownwish4gitcode
9 天前 关联了pull request:fix: preserve tied parameter identity through sharding and materialization
9 天前 关联了pull request:fix: preserve tied parameter identity through sharding and materialization
Checklist
🐛 Describe the bug
问题描述
模型中的 tied weights 在参数分片或 meta device 物化后,可能只共享底层
storage,但不再是同一个
Parameter对象。例如 embedding 和 lm_head 完成分片后,原有逻辑只替换底层 tensor:
tensor_a = param_a.to_local() if isinstance(param_a, DTensor) else param_a.data if isinstance(param_b, DTensor): param_b._local_tensor = tensor_a else: param_b.data = tensor_a此时虽然两端可能共享数据,但仍然存在两个独立的参数对象:
model.embed_tokens.weight is not model.lm_head.weight这会影响 tied weights 的梯度聚合、优化器状态复用以及 HSDP 对共享参数
所有权的识别。
另外,meta 模型通过 to_empty 物化后,没有重新恢复 tied weights:
model.to_empty(device=device) return modelto_empty 可能为同一绑定参数的不同 FQN 创建独立对象,导致 checkpoint
加载前的参数发现看到错误的参数关系。
期望行为
Expected behavior
参数分片及 meta materialization 完成后:
model.embed_tokens.weight is model.lm_head.weight应继续成立。
对于 HSDP 模型,需要在 checkpoint target discovery 之前刷新 HSDP
参数状态并恢复共享参数;同时应校验绑定参数的 shape、dtype、DeviceMesh和 placements 是否一致,避免错误布局被静默绑定。
Additional context
Environment info
atlas a2/, torch 2.9.0
Thanks for contributing 🎉!