master

ascend-robot[feature] 量化配置yaml校验

e243b34d创建于 2 天前历史提交

文件	最后提交记录	最后更新时间
awq	[test] add test for awq support Co-authored-by: zhongzhoutan<1710115119@bjmu.edu.cn> # message auto-generated for no-merge-commit merge: !183 merge pr-awq-support-test into master [test] add test for awq support Created-by: tangxuanya Commit-by: zhongzhoutan Merged-by: ascend-robot Description: 1. 增加 ut 2. 增加 smoke test See merge request: Ascend/msmodelslim!183	2 个月前
common	[feature] add the awq algorithm support Co-authored-by: zhongzhoutan<1710115119@bjmu.edu.cn> # message auto-generated for no-merge-commit merge: !179 merge pr-awq-support-part1 into master [feature] add the awq algorithm support Created-by: tangxuanya Commit-by: zhongzhoutan Merged-by: ascend-robot Description: [feature] add the awq algorithm support 拆分pr, refer to https://gitcode.com/Ascend/msmodelslim/pull/169 1. 增加 awq 算法适配 1. 使用 IContxt 存储 stats 2. 在 runner 外增加 conntext manager 3. 移动 subgraph type 文件到 anti outlier 目录下 4. 增加 lca 自动寻找计算 loss 的父模块 2. qwen2 模型适配 subgraph config See merge request: Ascend/msmodelslim!179	2 个月前
default	【msmodelslim】【feature】支持默认结构的离群值抑制 Co-authored-by: anreywmh<18845895998@163.com> # message auto-generated for no-merge-commit merge: !142 merge w_dev_default_anti into master 【msmodelslim】【feature】支持默认结构的离群值抑制 Created-by: anreywmh Commit-by: anreywmh Merged-by: ascend-robot Description: 【msmodelslim】【feature】支持默认结构的离群值抑制 See merge request: Ascend/msmodelslim!142	3 个月前
flex_smooth	【msmodelslim】刷新文件头部声明和版权 Co-authored-by: caishengcheng<caishengcheng@huawei.com> # message auto-generated for no-merge-commit merge: !28 merge license into master 【msmodelslim】刷新文件头部声明和版权 Created-by: caishengcheng Commit-by: caishengcheng Merged-by: ascend-robot Description: 【msmodelslim】刷新文件头部声明和版权 See merge request: Ascend/msmodelslim!28	4 个月前
iter_smooth	[feature] add the awq algorithm support Co-authored-by: zhongzhoutan<1710115119@bjmu.edu.cn> # message auto-generated for no-merge-commit merge: !179 merge pr-awq-support-part1 into master [feature] add the awq algorithm support Created-by: tangxuanya Commit-by: zhongzhoutan Merged-by: ascend-robot Description: [feature] add the awq algorithm support 拆分pr, refer to https://gitcode.com/Ascend/msmodelslim/pull/169 1. 增加 awq 算法适配 1. 使用 IContxt 存储 stats 2. 在 runner 外增加 conntext manager 3. 移动 subgraph type 文件到 anti outlier 目录下 4. 增加 lca 自动寻找计算 loss 的父模块 2. qwen2 模型适配 subgraph config See merge request: Ascend/msmodelslim!179	2 个月前
smooth_quant	[feature] add the awq algorithm support Co-authored-by: zhongzhoutan<1710115119@bjmu.edu.cn> # message auto-generated for no-merge-commit merge: !179 merge pr-awq-support-part1 into master [feature] add the awq algorithm support Created-by: tangxuanya Commit-by: zhongzhoutan Merged-by: ascend-robot Description: [feature] add the awq algorithm support 拆分pr, refer to https://gitcode.com/Ascend/msmodelslim/pull/169 1. 增加 awq 算法适配 1. 使用 IContxt 存储 stats 2. 在 runner 外增加 conntext manager 3. 移动 subgraph type 文件到 anti outlier 目录下 4. 增加 lca 自动寻找计算 loss 的父模块 2. qwen2 模型适配 subgraph config See merge request: Ascend/msmodelslim!179	2 个月前
__init__.py	【msmodelslim】刷新文件头部声明和版权 Co-authored-by: caishengcheng<caishengcheng@huawei.com> # message auto-generated for no-merge-commit merge: !28 merge license into master 【msmodelslim】刷新文件头部声明和版权 Created-by: caishengcheng Commit-by: caishengcheng Merged-by: ascend-robot Description: 【msmodelslim】刷新文件头部声明和版权 See merge request: Ascend/msmodelslim!28	4 个月前
test_flex_smooth_non_fusion_api.py	[feature] add the awq algorithm support Co-authored-by: zhongzhoutan<1710115119@bjmu.edu.cn> # message auto-generated for no-merge-commit merge: !179 merge pr-awq-support-part1 into master [feature] add the awq algorithm support Created-by: tangxuanya Commit-by: zhongzhoutan Merged-by: ascend-robot Description: [feature] add the awq algorithm support 拆分pr, refer to https://gitcode.com/Ascend/msmodelslim/pull/169 1. 增加 awq 算法适配 1. 使用 IContxt 存储 stats 2. 在 runner 外增加 conntext manager 3. 移动 subgraph type 文件到 anti outlier 目录下 4. 增加 lca 自动寻找计算 loss 的父模块 2. qwen2 模型适配 subgraph config See merge request: Ascend/msmodelslim!179	2 个月前
test_flex_smooth_quant.py	[feature] add the awq algorithm support Co-authored-by: zhongzhoutan<1710115119@bjmu.edu.cn> # message auto-generated for no-merge-commit merge: !179 merge pr-awq-support-part1 into master [feature] add the awq algorithm support Created-by: tangxuanya Commit-by: zhongzhoutan Merged-by: ascend-robot Description: [feature] add the awq algorithm support 拆分pr, refer to https://gitcode.com/Ascend/msmodelslim/pull/169 1. 增加 awq 算法适配 1. 使用 IContxt 存储 stats 2. 在 runner 外增加 conntext manager 3. 移动 subgraph type 文件到 anti outlier 目录下 4. 增加 lca 自动寻找计算 loss 的父模块 2. qwen2 模型适配 subgraph config See merge request: Ascend/msmodelslim!179	2 个月前
test_flex_smooth_quant_processor.py	【msmodelslim】刷新文件头部声明和版权 Co-authored-by: caishengcheng<caishengcheng@huawei.com> # message auto-generated for no-merge-commit merge: !28 merge license into master 【msmodelslim】刷新文件头部声明和版权 Created-by: caishengcheng Commit-by: caishengcheng Merged-by: ascend-robot Description: 【msmodelslim】刷新文件头部声明和版权 See merge request: Ascend/msmodelslim!28	4 个月前
test_iter_smooth.py	[feature] add the awq algorithm support Co-authored-by: zhongzhoutan<1710115119@bjmu.edu.cn> # message auto-generated for no-merge-commit merge: !179 merge pr-awq-support-part1 into master [feature] add the awq algorithm support Created-by: tangxuanya Commit-by: zhongzhoutan Merged-by: ascend-robot Description: [feature] add the awq algorithm support 拆分pr, refer to https://gitcode.com/Ascend/msmodelslim/pull/169 1. 增加 awq 算法适配 1. 使用 IContxt 存储 stats 2. 在 runner 外增加 conntext manager 3. 移动 subgraph type 文件到 anti outlier 目录下 4. 增加 lca 自动寻找计算 loss 的父模块 2. qwen2 模型适配 subgraph config See merge request: Ascend/msmodelslim!179	2 个月前
test_iter_smooth_asym_processor.py	【msmodelslim】刷新文件头部声明和版权 Co-authored-by: caishengcheng<caishengcheng@huawei.com> # message auto-generated for no-merge-commit merge: !28 merge license into master 【msmodelslim】刷新文件头部声明和版权 Created-by: caishengcheng Commit-by: caishengcheng Merged-by: ascend-robot Description: 【msmodelslim】刷新文件头部声明和版权 See merge request: Ascend/msmodelslim!28	4 个月前
test_iter_smooth_non_fusion_api.py	[feature] add the awq algorithm support Co-authored-by: zhongzhoutan<1710115119@bjmu.edu.cn> # message auto-generated for no-merge-commit merge: !179 merge pr-awq-support-part1 into master [feature] add the awq algorithm support Created-by: tangxuanya Commit-by: zhongzhoutan Merged-by: ascend-robot Description: [feature] add the awq algorithm support 拆分pr, refer to https://gitcode.com/Ascend/msmodelslim/pull/169 1. 增加 awq 算法适配 1. 使用 IContxt 存储 stats 2. 在 runner 外增加 conntext manager 3. 移动 subgraph type 文件到 anti outlier 目录下 4. 增加 lca 自动寻找计算 loss 的父模块 2. qwen2 模型适配 subgraph config See merge request: Ascend/msmodelslim!179	2 个月前
test_iter_smooth_processor_config.py	[feature] 量化配置yaml校验 Co-authored-by: tanxiangyuu<tanxiangyu2@huawei.com> # message auto-generated for no-merge-commit merge: !380 merge dev into master [feature] 量化配置yaml校验 Created-by: tanxiangyuu Commit-by: tanxiangyuu Merged-by: ascend-robot Description: ## PR描述 (What this PR does / why we need it?) ### 背景 msmodelslim 的 YAML 量化配置解析链中，Metadata 模型缺少 extra='forbid' 设置，默认 extra='ignore' 导致字段名拼写错误（如 scroe→score、labll→label）、缩进错误（如 include/exclude 误入 metadata 层级）等低级失误被静默丢弃，用户无任何报错提示，量化流程以错误的默认参数继续执行。 此外，PyYAML 对重复键静默用后值覆盖前值，项目缺少相关检测。 ### 变更内容 1. Metadata extra='forbid' - msmodelslim/core/practice/interface.py：Metadata 添加字段验证器（validate_str_length、in_range），修复 verified_model_types 使用 dataclasses.field → pydantic.Field 的问题 - msmodelslim/utils/patch/pydantic.py：patch 后的 PatchedBaseModel 默认 extra='forbid'，所有子类自动继承；需要允许额外字段的模型（如 BaseQuantConfig）已显式设置 extra='allow'，不受影响 效果：Metadata 层级的拼写错误、缩进错误在配置加载阶段立即报错，不再静默丢弃。 2. YAML 重复键检测 - msmodelslim/utils/security/path.py：新增 _DuplicateKeyLoader（继承 yaml.SafeLoader），在 yaml_safe_load 中检测重复键，发现时抛出 SchemaValidateError 并提示行号 效果：config_id: a / config_id: b 重复键不再静默覆盖，立即报错。 3. Pydantic patch 优化 - msmodelslim/utils/patch/pydantic.py： - 错误信息包含完整路径（如 process.0.type 而非 value） - 一次性报出metadata或spec的所有校验错误，不截断 - __init__ 通过栈帧检测区分嵌套调用和直接调用，嵌套时不破坏 Pydantic 的错误收集 - model_config 默认 extra='forbid'，所有新建模型自动拒绝未知字段 4. 校验工具函数 - msmodelslim/utils/validation/value.py：新增 allow_empty_list、allow_empty_dict、at_least_one_key、in_range 等通用校验函数 - msmodelslim/utils/validation/pydantic.py：新增对应的 Pydantic AfterValidator 包装 5. 代码清理 - 移除各 processor 文件中冗余的 model_config = ConfigDict(extra="forbid")（由 patch 默认提供） - 统一代码格式（缩进、引号、行尾空格） - 修复 greater_than_zero 支持 Decimal 类型 - 移除未使用的 is_boolean 校验函数 ### 面向用户的变更 | 场景 | 修复前 | 修复后 | |------|--------|--------| | Metadata 字段拼写错误（scroe→score） | 静默丢弃，使用默认值 | 立即报错，提示完整路径 | | include/exclude 缩进到 metadata 下 | 静默丢弃，所有模块被量化 | 立即报错 | | YAML 重复键 | 静默覆盖 | 立即报错，提示行号 | | 任意 Pydantic 模型的未知字段 | 默认忽略 | 默认拒绝（extra='forbid'） | | 多个校验错误同时存在 | 只报第一个 | 一次性全部报出 | ### 错误信息示例 #### 示例1   #### 示例2   #### 示例3   ### 结构修改 | 文件 | 修改说明 | |------|---------| | 文件 | 修改说明 | | :--- | :--- | | msmodelslim/utils/patch/pydantic.py | 新增 _format_validation_error、_in_model_validate；默认 extra='forbid'；model_validate 合并 _additional_validation_errors | | msmodelslim/utils/validation/validation_state.py | 新增：管理校验上下文（路径前缀）和额外错误存储 | | msmodelslim/processor/base.py | validate_auto_processor_config_list 不再 raise，非 dict 项错误写入 validation_state；设置 spec. 路径前缀 | | msmodelslim/utils/security/path.py | 新增 _DuplicateKeyLoader；yaml_safe_load 使用自定义 Loader；添加 encoding='utf-8' | | msmodelslim/core/practice/interface.py | Metadata 添加验证器；verified_model_types 改用 pydantic.Field | | msmodelslim/utils/validation/value.py | 新增 allow_empty_list、allow_empty_dict、at_least_one_key、in_range | | msmodelslim/utils/validation/pydantic.py | 新增 _to_value_error 包装，Pydantic 能收集多个自定义验证器的错误 | | msmodelslim/processor/*.py | 移除冗余 model_config；统一代码格式 | See merge request: Ascend/msmodelslim!380	2 天前
test_iter_smooth_sym_processor.py	【msmodelslim】刷新文件头部声明和版权 Co-authored-by: caishengcheng<caishengcheng@huawei.com> # message auto-generated for no-merge-commit merge: !28 merge license into master 【msmodelslim】刷新文件头部声明和版权 Created-by: caishengcheng Commit-by: caishengcheng Merged-by: ascend-robot Description: 【msmodelslim】刷新文件头部声明和版权 See merge request: Ascend/msmodelslim!28	4 个月前
test_qkv_fusion.py	【msmodelslim】刷新文件头部声明和版权 Co-authored-by: caishengcheng<caishengcheng@huawei.com> # message auto-generated for no-merge-commit merge: !28 merge license into master 【msmodelslim】刷新文件头部声明和版权 Created-by: caishengcheng Commit-by: caishengcheng Merged-by: ascend-robot Description: 【msmodelslim】刷新文件头部声明和版权 See merge request: Ascend/msmodelslim!28	4 个月前
test_smooth_base.py	[feature] add the awq algorithm support Co-authored-by: zhongzhoutan<1710115119@bjmu.edu.cn> # message auto-generated for no-merge-commit merge: !179 merge pr-awq-support-part1 into master [feature] add the awq algorithm support Created-by: tangxuanya Commit-by: zhongzhoutan Merged-by: ascend-robot Description: [feature] add the awq algorithm support 拆分pr, refer to https://gitcode.com/Ascend/msmodelslim/pull/169 1. 增加 awq 算法适配 1. 使用 IContxt 存储 stats 2. 在 runner 外增加 conntext manager 3. 移动 subgraph type 文件到 anti outlier 目录下 4. 增加 lca 自动寻找计算 loss 的父模块 2. qwen2 模型适配 subgraph config See merge request: Ascend/msmodelslim!179	2 个月前
test_virtual_v_module.py	【msmodelslim】刷新文件头部声明和版权 Co-authored-by: caishengcheng<caishengcheng@huawei.com> # message auto-generated for no-merge-commit merge: !28 merge license into master 【msmodelslim】刷新文件头部声明和版权 Created-by: caishengcheng Commit-by: caishengcheng Merged-by: ascend-robot Description: 【msmodelslim】刷新文件头部声明和版权 See merge request: Ascend/msmodelslim!28	4 个月前
test_virtual_v_module_from_kv_fused.py	【msmodelslim】刷新文件头部声明和版权 Co-authored-by: caishengcheng<caishengcheng@huawei.com> # message auto-generated for no-merge-commit merge: !28 merge license into master 【msmodelslim】刷新文件头部声明和版权 Created-by: caishengcheng Commit-by: caishengcheng Merged-by: ascend-robot Description: 【msmodelslim】刷新文件头部声明和版权 See merge request: Ascend/msmodelslim!28	4 个月前