master

ascend-robot性能优化：throughput_optimizer 启用 dynamic shapes 复用编译图，消除并发搜索重复编译

3e30d69d创建于 3 小时前历史提交

文件	最后提交记录	最后更新时间
quantization	【Bugfix】修复deepseek-v4部分量化场景下的问题 Co-authored-by: ChenHuiwen<chenhuiwen7@huawei.com> # message auto-generated for no-merge-commit merge: !373 merge fix-ds-v4-quant into master 【Bugfix】修复deepseek-v4部分量化场景下的问题 Created-by: ChenHuiwen Commit-by: ChenHuiwen Merged-by: ascend-robot Description: # PR Template Thanks for your contribution; we appreciate it a lot. The following instructions will make your pull request healthier and help you get feedback more easily. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers. 感谢您的贡献，我们非常重视。以下说明将使您的拉取请求更健康，更易于获得反馈。如果您不理解某些项目，请不要担心，只需提交拉取请求并从维护人员那里寻求帮助即可。 **PR Type / PR类型** - [ ] Feature（功能新增） - [x] Bugfix（Bug 修复） - [ ] Docs（文档更新） - [ ] CI/CD（持续集成/持续部署） - [ ] Refactor（代码重构） - [ ] Perf（性能优化） - [x] Test-Cases（测试用例更新） - [ ] Other（其他） ## 🔍 Motivation / 变更动机 This PR fixes DeepSeek V4 quantization and tensor-parallel execution issues found in text_generate simulation. 此 PR 修复 DeepSeek V4 在 text_generate 仿真中与量化和张量并行相关的问题。 Specifically, the old backbone quantization override naming was ambiguous because the option is used to configure non-routed-expert linear layers such as attention projections, dense MLP layers, and shared experts. This PR renames it to non-expert to better match the actual behavior. 同时，DeepSeek V4 Flash/Pro 的 O projection path has model-specific grouped projection behavior. Under W4A8 quantization or high TP configurations, TensorCast previously could hit shape mismatches in wo_a / wo_b modeling. This PR keeps the modeled path aligned with the real DeepSeek V4 structure while avoiding invalid reshape or double-sharding behavior. ## 📝 Modification / 修改内容 - Rename quantization override terminology: - --quantize-backbone-linear-action -> --quantize-non-expert-linear-action - quantize_backbone_linear_action -> quantize_non_expert_linear_action - Update CLI help text, UserInputConfig, quantization config creation, and related regression tests. - Update non-expert quantization config patterns: - Rename _BACKBONE_LINEAR_PATTERNS to _NON_EXPERT_LINEAR_PATTERNS. - Keep routed MoE experts controlled by the broad --quantize-linear-action setting. - Keep attention, dense MLP, and shared-expert layers covered by the non-expert override. - Fix DeepSeek V4 W4A8 wo_a grouped projection: - When wo_a is quantized as W4A8, unpack int4 packed qweight back to its logical weight shape before the grouped einsum reshape. - Avoid using stale in_features/out_features values after TP wrapping. - Fix DeepSeek V4 O projection TP sharding: - Remove the duplicate V4-specific self_attn.o_proj TP rule. - Let the generic o_proj RowParallel rule handle wo_b/o_proj once. - Prevent o_proj from being sharded twice, which caused local input dim mismatch. - Add/update regression tests: - Cover W4A8 wo_a logical weight shape before grouped einsum. - Update DeepSeek V4 TP plan expectations. - Update quantization config and user config tests for the new non-expert terminology. ------ ## 📐 Associated Test Results / 关联测试结果 **Please provide the related test results, such as test reports, etc.** **请提供相关测试结果，例如测试报告等。**   ------ ## 🌟 Use cases (Optional) / 使用案例（可选） **If this PR introduces a new feature, it is better to list some use cases here and update the documentation.** **如果此拉取请求引入了新功能，最好在此处列出一些用例并更新文档。** ------ ## ✅ Checklist / 检查列表 **Before PR**: - [ ] Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests. / 修复的 Bug 已完全由单元测试覆盖，导致 Bug 的情况应在单元测试中添加。 - [ ] The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness. / 此拉取请求中的修改已完全由单元测试覆盖。如果不是，请添加更多单元测试以确保正确性。 - [ ] All relevant documentation (API docs, docstrings, example tutorials) has been updated to reflect these changes. / 所有相关文档（API 文档、文档字符串、示例教程）已更新以反映这些更改。 - [ ] Please ensure code files contain no Chinese comments. / 请保证代码文件中不含中文注释。 ------ See merge request: Ascend/msmodeling!373	19 天前
config_resolver.py	【同步】【非开发代码】代码从 develop 同步到 master Co-authored-by: yydyzr<liuyuncong1@huawei.com> Co-authored-by: gcw_61YBRfIt<chuzhenxing@huawei.com> Co-authored-by: 孔炳翔<1120200577@qq.com> Co-authored-by: zhengxinqian<qianzhengxin@huawei.com> Co-authored-by: hw_whx<wanghexiang7@huawei.com> Co-authored-by: jgong5<steven.gong@gmail.com> Co-authored-by: hw_whx<2952154980@qq.com> # message auto-generated for no-merge-commit merge: !330 merge master into master 【同步】【非开发代码】代码从 develop 同步到 master Created-by: AvadaKedavrua Commit-by: liujiawang;ascend-robot;AvadaKedavrua;lutean;Horacehxw;eveyin1;minghang_c;zwt__;tt0cool;elrond-g;jia_ya_nan;zhenyu_zhang;ChenHuiwen;wangshen001;Hudingyi;wendellX;Secluded_Ocean;jhon-117;yaohan404;jiangruitao;zhenghaojie;stormchasingg;panyj1993;cmh1056291129;yuyinkai1;sunguozhong;genius52;liu_jiaxu;HongMaoShuiGuai;zhengxinqian;weixin_43368449;jsez-li-bin;jgong5;wqh17101;w00609794;yydyzr;JieZhang679;sppedforcy;gcw_61YBRfIt;Jiong Gong;hw_whx;gongjiong;孔炳翔 Merged-by: ascend-robot Description: 代码从 develop 同步到 master，后续基于 master 演进，并支持打包 See merge request: Ascend/msmodeling!330	24 天前
input_generator.py	fix(deepseek-v4): account for context in ratio128 attention topk Co-authored-by: jia_ya_nan<jiayanan3@h-partners.com> # message auto-generated for no-merge-commit merge: !448 merge master into master fix(deepseek-v4): account for context in ratio128 attention topk Created-by: jia_ya_nan Commit-by: jia_ya_nan Merged-by: ascend-robot Description: # PR Template Thanks for your contribution; we appreciate it a lot. The following instructions will make your pull request healthier and help you get feedback more easily. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers. 感谢您的贡献，我们非常重视。以下说明将使您的拉取请求更健康，更易于获得反馈。如果您不理解某些项目，请不要担心，只需提交拉取请求并从维护人员那里寻求帮助即可。 **PR Type / PR类型** - [ ] Feature（功能新增） - [x] Bugfix（Bug 修复） - [ ] Docs（文档更新） - [ ] CI/CD（持续集成/持续部署） - [ ] Refactor（代码重构） - [ ] Perf（性能优化） - [ ] Test-Cases（测试用例更新） - [ ] Other（其他） ## 🔍 Motivation / 变更动机 **Please describe the motivation of this PR and the goal you want to achieve through this PR.** **请描述您的拉取请求的动机和您希望通过此拉取请求实现的目标。** DeepSeek-V4 在 ratio=128 的 compressed sparse attention 场景下，compressed topk 的宽度应基于请求的总序列长度计算，即包含历史 context 和当前 query tokens。原实现仅使用当前 query length 计算 compressed topk，导致长上下文 prefill 场景下生成的 attention topk shape 偏小，进而影响 attention shape dump 和性能建模结果的准确性。 本 PR 旨在修正 DeepSeek-V4 ratio=128 attention 在长上下文场景下的 compressed topk 计算逻辑，使其与真实总序列长度对齐。 ------ ## 📝 Modification / 修改内容 **Please briefly describe what modification is made in this PR.** **请简要描述此拉取请求中进行的修改。** - 在 attention metadata 中新增 max_seq_len 字段，用于记录当前 batch 的最大总序列长度。 - 在固定 batch 和 varlen 输入生成逻辑中填充 max_seq_len。 - 更新 DeepSeek-V4 get_compress_topk_idxs，支持使用 total sequence length 计算 compressed topk 宽度。 - 在 DeepSeek-V4 ratio=128 attention 路径中传入 attention_meta.max_seq_len，确保 compressed topk 包含历史 context chunks。 - 新增回归测试，覆盖 ratio=128 长上下文 topk shape 以及 helper 函数基于 total sequence length 计算的行为。 ------ ## 📐 Associated Test Results / 关联测试结果 **Please provide the related test results, such as test reports, etc.** **请提供相关测试结果，例如测试报告等。** 对于长上下文prefill场景：29s->45s，主要修复compressor ratio为128时attention耗时异常的问题 python -m cli.inference.text_generate deepseek-ai/DeepSeek-V4-Pro --device ATLAS_800_A3_560T_128G_DIE --num-devices 32 --num-queries 32 --query-length 102400 --context-length 921600 --compile --quantize-linear-action MXFP4 --quantize-non-expert-linear-action FP8 --quantize-attention-action FP8 --tp-size 1 --ep-size 32 --log-level info   ------ ## 🌟 Use cases (Optional) / 使用案例（可选） **If this PR introduces a new feature, it is better to list some use cases here and update the documentation.** **如果此拉取请求引入了新功能，最好在此处列出一些用例并更新文档。** ------ ## ✅ Checklist / 检查列表 **Before PR**: - [x] Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests. / 修复的 Bug 已完全由单元测试覆盖，导致 Bug 的情况应在单元测试中添加。 - [x] The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness. / 此拉取请求中的修改已完全由单元测试覆盖。如果不是，请添加更多单元测试以确保正确性。 - [x] All relevant documentation (API docs, docstrings, example tutorials) has been updated to reflect these changes. / 所有相关文档（API 文档、文档字符串、示例教程）已更新以反映这些更改。 - [x] Please ensure code files contain no Chinese comments. / 请保证代码文件中不含中文注释。 ------ See merge request: Ascend/msmodeling!448	6 天前
model_builder.py	性能优化：throughput_optimizer 启用 dynamic shapes 复用编译图，消除并发搜索重复编译 Co-authored-by: jhon-117<fangkai15@huawei.com> # message auto-generated for no-merge-commit merge: !457 merge feat/optimizer-reuse-graph-concurrency into master 性能优化：throughput_optimizer 启用 dynamic shapes 复用编译图，消除并发搜索重复编译 Created-by: jhon-117 Commit-by: jhon-117 Merged-by: ascend-robot Description: # PR Template Thanks for your contribution; we appreciate it a lot. The following instructions will make your pull request healthier and help you get feedback more easily. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers. **PR Type / PR类型** - [ ] Feature（功能新增） - [ ] Bugfix（Bug 修复） - [ ] Docs（文档更新） - [ ] CI/CD（持续集成/持续部署） - [ ] Refactor（代码重构） - [x] Perf（性能优化） - [ ] Test-Cases（测试用例更新） - [ ] Other（其他） ## 🔍 Motivation / 变更动机 **Please describe the motivation of this PR and the goal you want to achieve through this PR.** throughput_optimizer searches for the best concurrency (batch size) via binary search. Each probed concurrency point produces two model.forward calls (prefill + decode), and their tensor shapes vary with the batch size. This is the intended "build graph once, sweep many concurrencies" design — ModelRunner (and thus the compiled graph) is constructed once per parallel strategy, and the binary search reuses it. However, build_model called torch.compile(..., dynamic=False). With dynamic=False, Dynamo **specializes** the traced graph to the concrete input shapes (including the batch dimension). The first forward of each distinct shape triggers Dynamo's shape guards to fail, which forces a **recompile**. Concretely: - A single forward that triggers compilation costs ~2.26 s (vs ~0.019 s for a cache hit in the non-compile path). - Because every probed batch size yields a new shape, **recompiles grow linearly with the number of concurrency points**, dominating the optimizer wall time. **Goal:** let one compiled graph cover the entire concurrency range so the compile cost is paid once (per prefill/decode graph) instead of once per shape. ## 📝 Modification / 修改内容 **Please briefly describe what modification is made in this PR.** Single-line change in tensor_cast/core/model_builder.py: diff model = torch.compile( model, backend=get_backend(device_name=user_input.device), - dynamic=False, + dynamic=True, fullgraph=use_full_graph, ) **Principle.** torch.compile traces the eager model under Dynamo and compiles the resulting graph. dynamic controls how Dynamo treats integer tensor dimensions: - dynamic=False — every int dim is **specialized** to its concrete value at trace time. The compiled graph is shape-specific; any later call whose shapes differ from the guards causes a recompile. - dynamic=True — Dynamo marks int dims (e.g. the batch/concurrency dimension) as **symbolic/dynamic**. The compiled graph is shape-agnostic for those dims, so subsequent calls with different batch sizes reuse the same graph without guard failure or recompilation. Since the optimizer reuses a single ModelRunner/compiled model across the whole binary search, switching to dynamic=True means the prefill graph and the decode graph are each compiled once, and all concurrency points share them. The compile cost becomes a constant instead of scaling with the search range. Note: dynamo_new_graph does not drop to 1 because prefill and decode are structurally different graphs (different query_len/seq_len), and a few shape boundaries are not yet fully dynamic. The key win is that recompiles become a **stable constant (4)** that no longer grows with the number of concurrency points. Eliminating the residual recompiles via explicit torch._dynamo.mark_dynamic is left as a follow-up. ## 📐 Associated Test Results / 关联测试结果 **Please provide the related test results, such as test reports, etc.** Benchmark: Qwen3-32B, --compile, --device=TEST_DEVICE, --num-devices=1, --jobs=1, --input-length=64, --output-length=16. ruff lint passed (All checks passed!). | Concurrency range | Metric | Before (dynamic=False) | After (dynamic=True) | |---|---|---|---| | batch 1–8 | Dynamo recompiles | 10 | 4 | | batch 1–8 | Wall time | 28.14 s | 20.49 s (-27%) | | batch 1–16 | Dynamo recompiles | 12 | 4 (stable) | | batch 1–16 | Wall time | 32.97 s | 20.59 s (-38%) | Recompiles stop growing with the concurrency range after the change, confirming the graph is reused. The gain ratio increases with more concurrency points, as expected.  **Correctness.** Best-row results are bit-for-bit identical before/after, confirming dynamic=True does not alter the simulation numerics: | Range | Throughput (token/s) | TTFT (ms) | TPOT (ms) | concurrency | |---|---|---|---|---| | 1–8 (before / after) | 178.59 / 178.59 | 83.08 / 83.08 | 39.60 / 39.60 | 8 | | 1–16 (before / after) | 290.44 / 290.44 | 161.76 / 161.76 | 44.98 / 44.98 | 16 | ## 🌟 Use cases (Optional) / 使用案例（可选） N/A — no new feature; any throughput_optimizer / text_generate invocation with --compile benefits automatically. ## ✅ Checklist / 检查列表 **Before PR**: - [ ] Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests. — N/A (not a bug fix). - [ ] The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness. — Not checked: this is a compile-flag perf change with no behavioral change; correctness was verified by identical end-to-end optimizer results (see Test Results). A dedicated regression test is not added in this PR. - [ ] All relevant documentation (API docs, docstrings, example tutorials) has been updated to reflect these changes. — N/A (single flag change, no public API/doc impact). - [x] Please ensure code files contain no Chinese comments. — The changed line has no comments. ``` See merge request: Ascend/msmodeling!457	3 小时前
model_runner.py	feat: support mixed-batch variable input simulation (phase 1 disaggregation prefill) Co-authored-by: stormchasingg<sh_ding@zju.edu.cn> # message auto-generated for no-merge-commit merge: !438 merge bucket-token into master feat: support mixed-batch variable input simulation (phase 1 disaggregation prefill) Created-by: stormchasingg Commit-by: stormchasingg Merged-by: ascend-robot Description: # PR Template Thanks for your contribution; we appreciate it a lot. The following instructions will make your pull request healthier and help you get feedback more easily. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers. 感谢您的贡献，我们非常重视。以下说明将使您的拉取请求更健康，更易于获得反馈。如果您不理解某些项目，请不要担心，只需提交拉取请求并从维护人员那里寻求帮助即可。 **PR Type / PR类型** - [x] Feature（功能新增） - [ ] Bugfix（Bug 修复） - [x] Docs（文档更新） - [ ] Refactor（代码重构） - [ ] Perf（性能优化） - [x] Test-Cases（测试用例更新） - [ ] Other（其他） ## 🔍 Motivation / 变更动机 This PR adds mixed-batch variable-token throughput optimization support for throughput_optimizer, aligns the execution path and final summary output with the current implementation, and updates related regression coverage. The main goals are: - support variable-length request distribution modeling for disaggregation prefill optimization - allow --input-length to accept either a positive integer or a length-distribution YAML file - build heterogeneous request batches with real per-bin token composition - expose composition detail rows in the final result table - make the variable-length optimization path easier to validate and maintain 本 PR 为 throughput_optimizer 增加 mixed-batch 变长 token 寻优能力，并同步整理执行链路、最终结果展示和相关回归测试。 主要目标： - 支持 disaggregation prefill 场景下的变长请求分布建模 - 让 --input-length 同时支持正整数和 length-distribution YAML 文件路径 - 构造真实 heterogeneous request batch，按分布进行 mixed-batch 仿真 - 在最终结果表中展示 composition detail rows - 让变长寻优链路更容易验证和维护 ------ ## 📝 Modification / 修改内容 This PR mainly includes: 1. Mixed-batch variable-length input support - extend --input-length so it can accept: - a positive integer for fixed-length mode - an existing YAML file path for variable-length distribution mode - add check_positive_integer_and_string for argparse validation - add LengthBin / LengthDistribution - add length-distribution YAML loading and validation - add representative row generation and concurrency sample allocation in OptimizerData 2. Runtime argument flow refactor - keep args.input_length as: - int for fixed-length mode - str for length-distribution file mode - load LengthDistribution in ParallelRunner when args.input_length is a file path - set OptimizerData.input_length=None when OptimizerData.length_distribution is used 3. Mixed-batch inference path - add batched request construction in BaseThroughputOptimizer - build heterogeneous RequestInfo batches with: - num_input_tokens - query_len - samples - run mixed batches through generate_inputs_varlen 4. Disaggregation throughput integration - support variable-length prefill path in DisaggThroughputOptimizer - compute throughput using real batch token totals - emit one aggregate row plus multiple composition detail rows 5. Final summary/table refactor - extend OptimizerSummary to support batched mixed-batch final output - choose batched summary output when args.input_length is a distribution file path - expand composition_rows into the final table - add dedicated batched disaggregation table rendering - update Top numbering so only aggregate rows are counted as solutions and detail rows no longer consume ranking numbers 6. Runtime and compile option alignment - support compile_dynamic_shapes in model build - add peak-memory adjustment hook for heterogeneous mixed-batch memory feasibility checks 7. Tests and RFC updates - add/update regression tests for: - length distribution loading - --input-length integer/file argument handling - ParallelRunner loading distribution files into OptimizerData - optimizer summary batched rendering - base optimizer mixed-batch request construction - model runner peak-memory adjustment behavior - update mixed-batch variable-token RFC documents to match current implementation 本 PR 主要包含： 1. Mixed-batch 变长输入支持 - 扩展 --input-length，使其支持： - 正整数：固定输入长度模式 - 已存在的 YAML 文件路径：变长分布模式 - 增加 check_positive_integer_and_string 用于 argparse 校验 - 增加 LengthBin / LengthDistribution - 增加 length-distribution YAML 加载与校验 - 在 OptimizerData 中增加 representative row 生成和 sample 分配逻辑 2. Runtime 参数流重构 - args.input_length 保持为： - int：固定长度模式 - str：length-distribution 文件路径模式 - 在 ParallelRunner 中，当 args.input_length 是文件路径时加载 LengthDistribution - 使用 OptimizerData.length_distribution 时，将 OptimizerData.input_length 置为 None 3. Mixed-batch 推理路径 - 在 BaseThroughputOptimizer 中增加 batched request 构造 - 基于： - num_input_tokens - query_len - samples 构造 heterogeneous RequestInfo - 通过 generate_inputs_varlen 执行 mixed batch 4. Disaggregation 吞吐寻优接入 - 在 DisaggThroughputOptimizer 中接入变长 prefill 路径 - 基于 batch 真实 token 总量计算吞吐 - 输出 1 条 aggregate row 和多条 composition detail rows 5. 最终 summary / 表格重构 - 在 OptimizerSummary 中增加 batched mixed-batch 最终展示逻辑 - 当 args.input_length 是分布文件路径时，选择 batched summary 输出 - 在最终表格中展开 composition_rows - 增加 batched disaggregation 专用表格渲染 - 调整 Top 列语义：只对 aggregate row 编号，detail row 不再占用排名 6. Runtime 与 compile 选项对齐 - 在模型构建中支持 compile_dynamic_shapes - 增加 heterogeneous mixed-batch 场景的 peak memory feasibility adjustment hook ------ ## 📐 Associated Test Results / 关联测试结果 略。（超过10000字了） ------ ## 🌟 Use cases (Optional) / 使用案例（可选） Typical use cases: - mixed-batch variable-token prefill throughput optimization - validating disaggregation TTFT under heterogeneous request composition - inspecting aggregate result rows together with detailed composition rows 示例命令： bash python3 -m cli.inference.throughput_optimizer \ --input-length serving_cast/example/length_distribution.yaml \ --output-length 1 \ "$model_path" \ --device ATLAS_800_A3_752T_128G_DIE \ --num-devices 16 \ --compile \ --compile-dynamic-shapes \ --quantize-linear-action W8A8_STATIC \ --disagg \ --ttft-limits 2000 \ --tp-sizes 8 16 \ --batch-range 4 4 \ --reserved-memory-gb 10 \ --enable-shared-expert-tp \ --enable-dispatch-ffn-combine \ --word-embedding-tp row \ --log-level info \ 2>&1 | tee ./run_sc3_1.log 示例结果： Top 2 Disaggregation (Prefill) Configurations: +-----+-------------+------------------+---------------+---------+-------------+-----------+----------------------+---------------------------------------------------+------------+ | Top | num_devices | num_input_tokens | request_ratio | samples | concurrency | TTFT (ms) | Throughput (token/s) | parallel | batch_size | +-----+-------------+------------------+---------------+---------+-------------+-----------+----------------------+---------------------------------------------------+------------+ | 1 | 16 | all | 1.000 | 8 | 8 | 675.86 | 25153.15 | TP=8 | PP=1 | DP=2 | EP=16 | MOE-TP=1 | MOE-DP=1 | 4 | | - | 16 | 250 | 0.247 | 1 | 8 | - | - | TP=8 | PP=1 | DP=2 | EP=16 | MOE-TP=1 | MOE-DP=1 | 4 | | - | 16 | 2250 | 0.085 | 1 | 8 | - | - | TP=8 | PP=1 | DP=2 | EP=16 | MOE-TP=1 | MOE-DP=1 | 4 | | - | 16 | 2750 | 0.079 | 1 | 8 | - | - | TP=8 | PP=1 | DP=2 | EP=16 | MOE-TP=1 | MOE-DP=1 | 4 | | - | 16 | 3250 | 0.088 | 1 | 8 | - | - | TP=8 | PP=1 | DP=2 | EP=16 | MOE-TP=1 | MOE-DP=1 | 4 | | 2 | 16 | all | 1.000 | 4 | 4 | 572.65 | 14843.36 | TP=16 | PP=1 | DP=1 | EP=16 | MOE-TP=1 | MOE-DP=1 | 4 | | - | 16 | 250 | 0.247 | 1 | 4 | - | - | TP=16 | PP=1 | DP=1 | EP=16 | MOE-TP=1 | MOE-DP=1 | 4 | | - | 16 | 2250 | 0.085 | 1 | 4 | - | - | TP=16 | PP=1 | DP=1 | EP=16 | MOE-TP=1 | MOE-DP=1 | 4 | | - | 16 | 2750 | 0.079 | 1 | 4 | - | - | TP=16 | PP=1 | DP=1 | EP=16 | MOE-TP=1 | MOE-DP=1 | 4 | | - | 16 | 3250 | 0.088 | 1 | 4 | - | - | TP=16 | PP=1 | DP=1 | EP=16 | MOE-TP=1 | MOE-DP=1 | 4 | +-----+-------------+------------------+---------------+---------+-------------+-----------+----------------------+---------------------------------------------------+------------+ 典型场景： - mixed-batch 变长 token prefill 吞吐寻优 - heterogeneous request composition 下的 disaggregation TTFT 建模 - 同时查看 aggregate row 和 composition detail rows - 使用同一个 --input-length 参数在固定长度和分布文件模式之间切换 ------ ## ✅ Checklist / 检查列表 **Before PR**: - [x] Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests. / 修复的 Bug 已完全由单元测试覆盖，导致 Bug 的情况应在单元测试中添加。 - [x] The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness. / 此拉取请求中的修改已完全由单元测试覆盖。如果不是，请添加更多单元测试以确保正确性。 - [x] All relevant documentation (API docs, docstrings, example tutorials) has been updated to reflect these changes. / 所有相关文档（API 文档、文档字符串、示例教程）已更新以反映这些更改。 - [x] Please ensure code files contain no Chinese comments. / 请保证代码文件中不含中文注释。 ------ See merge request: Ascend/msmodeling!438	3 天前
model_source_security.py	fix(security): add model source safety checks Co-authored-by: jia_ya_nan<jiayanan3@h-partners.com> # message auto-generated for no-merge-commit merge: !385 merge fix/trust-remote-code-safety into master fix(security): add model source safety checks Created-by: jia_ya_nan Commit-by: jia_ya_nan Merged-by: ascend-robot Description: # PR Template Thanks for your contribution; we appreciate it a lot. The following instructions will make your pull request healthier and help you get feedback more easily. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers. 感谢您的贡献，我们非常重视。以下说明将使您的拉取请求更健康，更易于获得反馈。如果您不理解某些项目，请不要担心，只需提交拉取请求并从维护人员那里寻求帮助即可。 **PR Type / PR类型** - [ ] Feature（功能新增） - [x] Bugfix（Bug 修复） - [ ] Docs（文档更新） - [ ] CI/CD（持续集成/持续部署） - [ ] Refactor（代码重构） - [ ] Perf（性能优化） - [ ] Test-Cases（测试用例更新） - [x] Other（其他） ## 🔍 Motivation / 变更动机 **Please describe the motivation of this PR and the goal you want to achieve through this PR.** **请描述您的拉取请求的动机和您希望通过此拉取请求实现的目标。** 安全加固 ------ ## 📝 Modification / 修改内容 **Please briefly describe what modification is made in this PR.** **请简要描述此拉取请求中进行的修改。** 增加本地路径权限校验；增加日志风险提示 去掉不维护的老接口 ------ ## 📐 Associated Test Results / 关联测试结果 **Please provide the related test results, such as test reports, etc.** **请提供相关测试结果，例如测试报告等。**  ------ ## 🌟 Use cases (Optional) / 使用案例（可选） **If this PR introduces a new feature, it is better to list some use cases here and update the documentation.** **如果此拉取请求引入了新功能，最好在此处列出一些用例并更新文档。** ------ ## ✅ Checklist / 检查列表 **Before PR**: - [x] Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests. / 修复的 Bug 已完全由单元测试覆盖，导致 Bug 的情况应在单元测试中添加。 - [x] The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness. / 此拉取请求中的修改已完全由单元测试覆盖。如果不是，请添加更多单元测试以确保正确性。 - [x] All relevant documentation (API docs, docstrings, example tutorials) has been updated to reflect these changes. / 所有相关文档（API 文档、文档字符串、示例教程）已更新以反映这些更改。 - [x] Please ensure code files contain no Chinese comments. / 请保证代码文件中不含中文注释。 ------ See merge request: Ascend/msmodeling!385	12 天前
user_config.py	feat: support mixed-batch variable input simulation (phase 1 disaggregation prefill) Co-authored-by: stormchasingg<sh_ding@zju.edu.cn> # message auto-generated for no-merge-commit merge: !438 merge bucket-token into master feat: support mixed-batch variable input simulation (phase 1 disaggregation prefill) Created-by: stormchasingg Commit-by: stormchasingg Merged-by: ascend-robot Description: # PR Template Thanks for your contribution; we appreciate it a lot. The following instructions will make your pull request healthier and help you get feedback more easily. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers. 感谢您的贡献，我们非常重视。以下说明将使您的拉取请求更健康，更易于获得反馈。如果您不理解某些项目，请不要担心，只需提交拉取请求并从维护人员那里寻求帮助即可。 **PR Type / PR类型** - [x] Feature（功能新增） - [ ] Bugfix（Bug 修复） - [x] Docs（文档更新） - [ ] Refactor（代码重构） - [ ] Perf（性能优化） - [x] Test-Cases（测试用例更新） - [ ] Other（其他） ## 🔍 Motivation / 变更动机 This PR adds mixed-batch variable-token throughput optimization support for throughput_optimizer, aligns the execution path and final summary output with the current implementation, and updates related regression coverage. The main goals are: - support variable-length request distribution modeling for disaggregation prefill optimization - allow --input-length to accept either a positive integer or a length-distribution YAML file - build heterogeneous request batches with real per-bin token composition - expose composition detail rows in the final result table - make the variable-length optimization path easier to validate and maintain 本 PR 为 throughput_optimizer 增加 mixed-batch 变长 token 寻优能力，并同步整理执行链路、最终结果展示和相关回归测试。 主要目标： - 支持 disaggregation prefill 场景下的变长请求分布建模 - 让 --input-length 同时支持正整数和 length-distribution YAML 文件路径 - 构造真实 heterogeneous request batch，按分布进行 mixed-batch 仿真 - 在最终结果表中展示 composition detail rows - 让变长寻优链路更容易验证和维护 ------ ## 📝 Modification / 修改内容 This PR mainly includes: 1. Mixed-batch variable-length input support - extend --input-length so it can accept: - a positive integer for fixed-length mode - an existing YAML file path for variable-length distribution mode - add check_positive_integer_and_string for argparse validation - add LengthBin / LengthDistribution - add length-distribution YAML loading and validation - add representative row generation and concurrency sample allocation in OptimizerData 2. Runtime argument flow refactor - keep args.input_length as: - int for fixed-length mode - str for length-distribution file mode - load LengthDistribution in ParallelRunner when args.input_length is a file path - set OptimizerData.input_length=None when OptimizerData.length_distribution is used 3. Mixed-batch inference path - add batched request construction in BaseThroughputOptimizer - build heterogeneous RequestInfo batches with: - num_input_tokens - query_len - samples - run mixed batches through generate_inputs_varlen 4. Disaggregation throughput integration - support variable-length prefill path in DisaggThroughputOptimizer - compute throughput using real batch token totals - emit one aggregate row plus multiple composition detail rows 5. Final summary/table refactor - extend OptimizerSummary to support batched mixed-batch final output - choose batched summary output when args.input_length is a distribution file path - expand composition_rows into the final table - add dedicated batched disaggregation table rendering - update Top numbering so only aggregate rows are counted as solutions and detail rows no longer consume ranking numbers 6. Runtime and compile option alignment - support compile_dynamic_shapes in model build - add peak-memory adjustment hook for heterogeneous mixed-batch memory feasibility checks 7. Tests and RFC updates - add/update regression tests for: - length distribution loading - --input-length integer/file argument handling - ParallelRunner loading distribution files into OptimizerData - optimizer summary batched rendering - base optimizer mixed-batch request construction - model runner peak-memory adjustment behavior - update mixed-batch variable-token RFC documents to match current implementation 本 PR 主要包含： 1. Mixed-batch 变长输入支持 - 扩展 --input-length，使其支持： - 正整数：固定输入长度模式 - 已存在的 YAML 文件路径：变长分布模式 - 增加 check_positive_integer_and_string 用于 argparse 校验 - 增加 LengthBin / LengthDistribution - 增加 length-distribution YAML 加载与校验 - 在 OptimizerData 中增加 representative row 生成和 sample 分配逻辑 2. Runtime 参数流重构 - args.input_length 保持为： - int：固定长度模式 - str：length-distribution 文件路径模式 - 在 ParallelRunner 中，当 args.input_length 是文件路径时加载 LengthDistribution - 使用 OptimizerData.length_distribution 时，将 OptimizerData.input_length 置为 None 3. Mixed-batch 推理路径 - 在 BaseThroughputOptimizer 中增加 batched request 构造 - 基于： - num_input_tokens - query_len - samples 构造 heterogeneous RequestInfo - 通过 generate_inputs_varlen 执行 mixed batch 4. Disaggregation 吞吐寻优接入 - 在 DisaggThroughputOptimizer 中接入变长 prefill 路径 - 基于 batch 真实 token 总量计算吞吐 - 输出 1 条 aggregate row 和多条 composition detail rows 5. 最终 summary / 表格重构 - 在 OptimizerSummary 中增加 batched mixed-batch 最终展示逻辑 - 当 args.input_length 是分布文件路径时，选择 batched summary 输出 - 在最终表格中展开 composition_rows - 增加 batched disaggregation 专用表格渲染 - 调整 Top 列语义：只对 aggregate row 编号，detail row 不再占用排名 6. Runtime 与 compile 选项对齐 - 在模型构建中支持 compile_dynamic_shapes - 增加 heterogeneous mixed-batch 场景的 peak memory feasibility adjustment hook ------ ## 📐 Associated Test Results / 关联测试结果 略。（超过10000字了） ------ ## 🌟 Use cases (Optional) / 使用案例（可选） Typical use cases: - mixed-batch variable-token prefill throughput optimization - validating disaggregation TTFT under heterogeneous request composition - inspecting aggregate result rows together with detailed composition rows 示例命令： bash python3 -m cli.inference.throughput_optimizer \ --input-length serving_cast/example/length_distribution.yaml \ --output-length 1 \ "$model_path" \ --device ATLAS_800_A3_752T_128G_DIE \ --num-devices 16 \ --compile \ --compile-dynamic-shapes \ --quantize-linear-action W8A8_STATIC \ --disagg \ --ttft-limits 2000 \ --tp-sizes 8 16 \ --batch-range 4 4 \ --reserved-memory-gb 10 \ --enable-shared-expert-tp \ --enable-dispatch-ffn-combine \ --word-embedding-tp row \ --log-level info \ 2>&1 | tee ./run_sc3_1.log 示例结果： Top 2 Disaggregation (Prefill) Configurations: +-----+-------------+------------------+---------------+---------+-------------+-----------+----------------------+---------------------------------------------------+------------+ | Top | num_devices | num_input_tokens | request_ratio | samples | concurrency | TTFT (ms) | Throughput (token/s) | parallel | batch_size | +-----+-------------+------------------+---------------+---------+-------------+-----------+----------------------+---------------------------------------------------+------------+ | 1 | 16 | all | 1.000 | 8 | 8 | 675.86 | 25153.15 | TP=8 | PP=1 | DP=2 | EP=16 | MOE-TP=1 | MOE-DP=1 | 4 | | - | 16 | 250 | 0.247 | 1 | 8 | - | - | TP=8 | PP=1 | DP=2 | EP=16 | MOE-TP=1 | MOE-DP=1 | 4 | | - | 16 | 2250 | 0.085 | 1 | 8 | - | - | TP=8 | PP=1 | DP=2 | EP=16 | MOE-TP=1 | MOE-DP=1 | 4 | | - | 16 | 2750 | 0.079 | 1 | 8 | - | - | TP=8 | PP=1 | DP=2 | EP=16 | MOE-TP=1 | MOE-DP=1 | 4 | | - | 16 | 3250 | 0.088 | 1 | 8 | - | - | TP=8 | PP=1 | DP=2 | EP=16 | MOE-TP=1 | MOE-DP=1 | 4 | | 2 | 16 | all | 1.000 | 4 | 4 | 572.65 | 14843.36 | TP=16 | PP=1 | DP=1 | EP=16 | MOE-TP=1 | MOE-DP=1 | 4 | | - | 16 | 250 | 0.247 | 1 | 4 | - | - | TP=16 | PP=1 | DP=1 | EP=16 | MOE-TP=1 | MOE-DP=1 | 4 | | - | 16 | 2250 | 0.085 | 1 | 4 | - | - | TP=16 | PP=1 | DP=1 | EP=16 | MOE-TP=1 | MOE-DP=1 | 4 | | - | 16 | 2750 | 0.079 | 1 | 4 | - | - | TP=16 | PP=1 | DP=1 | EP=16 | MOE-TP=1 | MOE-DP=1 | 4 | | - | 16 | 3250 | 0.088 | 1 | 4 | - | - | TP=16 | PP=1 | DP=1 | EP=16 | MOE-TP=1 | MOE-DP=1 | 4 | +-----+-------------+------------------+---------------+---------+-------------+-----------+----------------------+---------------------------------------------------+------------+ 典型场景： - mixed-batch 变长 token prefill 吞吐寻优 - heterogeneous request composition 下的 disaggregation TTFT 建模 - 同时查看 aggregate row 和 composition detail rows - 使用同一个 --input-length 参数在固定长度和分布文件模式之间切换 ------ ## ✅ Checklist / 检查列表 **Before PR**: - [x] Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests. / 修复的 Bug 已完全由单元测试覆盖，导致 Bug 的情况应在单元测试中添加。 - [x] The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness. / 此拉取请求中的修改已完全由单元测试覆盖。如果不是，请添加更多单元测试以确保正确性。 - [x] All relevant documentation (API docs, docstrings, example tutorials) has been updated to reflect these changes. / 所有相关文档（API 文档、文档字符串、示例教程）已更新以反映这些更改。 - [x] Please ensure code files contain no Chinese comments. / 请保证代码文件中不含中文注释。 ------ See merge request: Ascend/msmodeling!438	3 天前
utils.py	【同步】【非开发代码】代码从 develop 同步到 master Co-authored-by: yydyzr<liuyuncong1@huawei.com> Co-authored-by: gcw_61YBRfIt<chuzhenxing@huawei.com> Co-authored-by: 孔炳翔<1120200577@qq.com> Co-authored-by: zhengxinqian<qianzhengxin@huawei.com> Co-authored-by: hw_whx<wanghexiang7@huawei.com> Co-authored-by: jgong5<steven.gong@gmail.com> Co-authored-by: hw_whx<2952154980@qq.com> # message auto-generated for no-merge-commit merge: !330 merge master into master 【同步】【非开发代码】代码从 develop 同步到 master Created-by: AvadaKedavrua Commit-by: liujiawang;ascend-robot;AvadaKedavrua;lutean;Horacehxw;eveyin1;minghang_c;zwt__;tt0cool;elrond-g;jia_ya_nan;zhenyu_zhang;ChenHuiwen;wangshen001;Hudingyi;wendellX;Secluded_Ocean;jhon-117;yaohan404;jiangruitao;zhenghaojie;stormchasingg;panyj1993;cmh1056291129;yuyinkai1;sunguozhong;genius52;liu_jiaxu;HongMaoShuiGuai;zhengxinqian;weixin_43368449;jsez-li-bin;jgong5;wqh17101;w00609794;yydyzr;JieZhang679;sppedforcy;gcw_61YBRfIt;Jiong Gong;hw_whx;gongjiong;孔炳翔 Merged-by: ascend-robot Description: 代码从 develop 同步到 master，后续基于 master 演进，并支持打包 See merge request: Ascend/msmodeling!330	24 天前