已开启
docs(examples): align table separator columns with header in EN guides #273
不喜欢雨天ༀ创建于 9 天前
docs(examples): align table separator columns with header in EN guides #273
已开启
共 8 个文件变更+154-154
| @@ -25,7 +25,7 @@ If you need to modify the detailed configuration, please refer to the documentat | |||
| 25 | The AWQ algorithm only supports weight quantization. The supported quantization types and quantization configurations are: | 25 | The AWQ algorithm only supports weight quantization. The supported quantization types and quantization configurations are: |
| 26 | 26 | ||
| 27 | | Field | Type | Description | Value Range | Notes | | 27 | | Field | Type | Description | Value Range | Notes | |
| 28 | -|:--| :-: | :-- | :-: | :-: | :-- | | 28 | +|:--|:-:|:--|:-:|:-:| |
| 29 | |batch_num|uint32|Number of batches used for quantization |1|/| | 29 | |batch_num|uint32|Number of batches used for quantization |1|/| |
| 30 | |skip_layers|str|Layers to skip quantization |/|Skip quantization layers support fuzzy matching. When the configured string is a layer name substring or matches the layer name, skip quantization for that layer and do not generate quantization configuration. The string must contain numbers or letters| | 30 | |skip_layers|str|Layers to skip quantization |/|Skip quantization layers support fuzzy matching. When the configured string is a layer name substring or matches the layer name, skip quantization for that layer and do not generate quantization configuration. The string must contain numbers or letters| |
| 31 | |weights.type|str|Quantized weight type|'int4'/'int8'/'float4_e2m1'/'mxfp4_e2m1'|/| | 31 | |weights.type|str|Quantized weight type|'int4'/'int8'/'float4_e2m1'/'mxfp4_e2m1'|/| |
| @@ -62,4 +62,4 @@ Where Score is the quantized model PPL. For specific values, refer to the follow | |||
| 62 | |LLAMA2-7B|pileval|wikitext2|5.472|5.550|5.725| | 62 | |LLAMA2-7B|pileval|wikitext2|5.472|5.550|5.725| |
| 63 | |QWEN2-7B|pileval|wikitext2|7.137|7.268|7.411| | 63 | |QWEN2-7B|pileval|wikitext2|7.137|7.268|7.411| |
| 64 | 64 | ||
| 65 | -After inference succeeds, a quantization log file ./amct_log/amct_pytorch.log is generated in the current directory | 65 | +After inference succeeds, a quantization log file ./amct_log/amct_pytorch.log is generated in the current directory |
| @@ -23,7 +23,7 @@ If you need to modify the detailed configuration, please refer to the documentat | |||
| 23 | The cast algorithm supports weight-only quantization and full quantization. The supported quantization types and quantization configurations are: | 23 | The cast algorithm supports weight-only quantization and full quantization. The supported quantization types and quantization configurations are: |
| 24 | 24 | ||
| 25 | | Field | Type | Description | Value Range | Notes | | 25 | | Field | Type | Description | Value Range | Notes | |
| 26 | -|:--| :-: | :-- | :-: | :-: | :-- | | 26 | +|:--|:-:|:--|:-:|:-:| |
| 27 | |batch_num|uint32|Number of batches used for quantization |1|/| | 27 | |batch_num|uint32|Number of batches used for quantization |1|/| |
| 28 | |skip_layers|str|Layers to skip quantization |/|Skip quantization layers support fuzzy matching. When the configured string is a layer name substring or matches the layer name, skip quantization for that layer and do not generate quantization configuration. The string must contain numbers or letters| | 28 | |skip_layers|str|Layers to skip quantization |/|Skip quantization layers support fuzzy matching. When the configured string is a layer name substring or matches the layer name, skip quantization for that layer and do not generate quantization configuration. The string must contain numbers or letters| |
| 29 | |weights.type|str|Quantized weight type|'hifloat8'|/| | 29 | |weights.type|str|Quantized weight type|'hifloat8'|/| |
| @@ -68,4 +68,4 @@ Where Score is the quantized model PPL. For specific values, refer to the follow | |||
| 68 | |QWEN2-7B|pileval|wikitext2|7.137|7.188| | 68 | |QWEN2-7B|pileval|wikitext2|7.137|7.188| |
| 69 | |QWEN3-8B|pileval|wikitext2|9.715|9.745| | 69 | |QWEN3-8B|pileval|wikitext2|9.715|9.745| |
| 70 | 70 | ||
| 71 | -After inference succeeds, a quantization log file ./amct_log/amct_pytorch.log is generated in the current directory | 71 | +After inference succeeds, a quantization log file ./amct_log/amct_pytorch.log is generated in the current directory |
| @@ -30,7 +30,7 @@ The FlatQuant algorithm supports the following partial quantization: | |||
| 30 | Supported quantization types and quantization configurations: | 30 | Supported quantization types and quantization configurations: |
| 31 | 31 | ||
| 32 | | Field | Type | Description | Value Range | Notes | | 32 | | Field | Type | Description | Value Range | Notes | |
| 33 | -|:--| :-: | :-- | :-: | :-: | :-- | | 33 | +|:--|:-:|:--|:-:|:-:| |
| 34 | |skip_layers|str|Layers to skip quantization |/|Skip quantization layers support fuzzy matching. When the configured string is a layer name substring or matches the layer name, skip quantization for that layer and do not generate quantization configuration. The string must contain numbers or letters| | 34 | |skip_layers|str|Layers to skip quantization |/|Skip quantization layers support fuzzy matching. When the configured string is a layer name substring or matches the layer name, skip quantization for that layer and do not generate quantization configuration. The string must contain numbers or letters| |
| 35 | |algorithm|dict|Quantization algorithm configuration used|{'flatquant'}|Refer to `INT4_FLAT_QUANT_CFG` example | 35 | |algorithm|dict|Quantization algorithm configuration used|{'flatquant'}|Refer to `INT4_FLAT_QUANT_CFG` example |
| 36 | 36 | ||
| @@ -97,4 +97,4 @@ The generation result after quantization is: | |||
| 97 | content: ============================================================================ | 97 | content: ============================================================================ |
| 98 | <> | 98 | <> |
| 99 | The Ascend Model Compression Toolkit (AMCT) is a powerful tool designed to ... | 99 | The Ascend Model Compression Toolkit (AMCT) is a powerful tool designed to ... |
| 100 | -``` | 100 | +``` |
| @@ -39,7 +39,7 @@ If you need to modify the detailed configuration, please refer to the documentat | |||
| 39 | The GPTQ algorithm only supports weight quantization. The supported quantization types and quantization configurations are: | 39 | The GPTQ algorithm only supports weight quantization. The supported quantization types and quantization configurations are: |
| 40 | 40 | ||
| 41 | | Field | Type | Description | Value Range | Notes | | 41 | | Field | Type | Description | Value Range | Notes | |
| 42 | -|:--| :-: | :-- | :-: | :-: | :-- | | 42 | +|:--|:-:|:--|:-:|:-:| |
| 43 | |batch_num|uint32|Number of batches used for quantization |1|/| | 43 | |batch_num|uint32|Number of batches used for quantization |1|/| |
| 44 | |skip_layers|str|Layers to skip quantization |/|Skip quantization layers support fuzzy matching. When the configured string is a layer name substring or matches the layer name, skip quantization for that layer and do not generate quantization configuration. The string must contain numbers or letters| | 44 | |skip_layers|str|Layers to skip quantization |/|Skip quantization layers support fuzzy matching. When the configured string is a layer name substring or matches the layer name, skip quantization for that layer and do not generate quantization configuration. The string must contain numbers or letters| |
| 45 | |weights.type|str|Quantized weight type|'int4'/'int8'/'float4_e2m1'/'mxfp4_e2m1'|/| | 45 | |weights.type|str|Quantized weight type|'int4'/'int8'/'float4_e2m1'/'mxfp4_e2m1'|/| |
| @@ -78,4 +78,4 @@ Where Score is the quantized model PPL. For specific values, refer to the follow | |||
| 78 | |QWEN2-7B|pileval|wikitext2|7.137|7.253|7.305 | 78 | |QWEN2-7B|pileval|wikitext2|7.137|7.253|7.305 |
| 79 | 79 | ||
| 80 | 80 | ||
| 81 | -After inference succeeds, a quantization log file ./amct_log/amct_pytorch.log is generated in the current directory | 81 | +After inference succeeds, a quantization log file ./amct_log/amct_pytorch.log is generated in the current directory |
| @@ -50,7 +50,7 @@ If you need to modify the detailed configuration, please refer to the documentat | |||
| 50 | The MINMAX algorithm supports weight-only quantization and full quantization. The supported quantization types and quantization configurations are: | 50 | The MINMAX algorithm supports weight-only quantization and full quantization. The supported quantization types and quantization configurations are: |
| 51 | 51 | ||
| 52 | | Field | Type | Description | Value Range | Notes | | 52 | | Field | Type | Description | Value Range | Notes | |
| 53 | -|:--| :-: | :-- | :-: | :-: | :-- | | 53 | +|:--|:-:|:--|:-:|:-:| |
| 54 | |batch_num|uint32|Number of batches used for quantization |1|/| | 54 | |batch_num|uint32|Number of batches used for quantization |1|/| |
| 55 | |skip_layers|str|Layers to skip quantization |/|Skip quantization layers support fuzzy matching. When the configured string is a layer name substring or matches the layer name, skip quantization for that layer and do not generate quantization configuration. The string must contain numbers or letters| | 55 | |skip_layers|str|Layers to skip quantization |/|Skip quantization layers support fuzzy matching. When the configured string is a layer name substring or matches the layer name, skip quantization for that layer and do not generate quantization configuration. The string must contain numbers or letters| |
| 56 | |weights.type|str|Quantized weight type|'int4'/'int8'/'float4_e2m1'|/| | 56 | |weights.type|str|Quantized weight type|'int4'/'int8'/'float4_e2m1'|/| |
| @@ -42,7 +42,7 @@ If you need to modify the detailed configuration, please refer to the documentat | |||
| 42 | The MXQUANT algorithm supports weight-only quantization and full quantization. The supported quantization types and quantization configurations are: | 42 | The MXQUANT algorithm supports weight-only quantization and full quantization. The supported quantization types and quantization configurations are: |
| 43 | 43 | ||
| 44 | | Field | Type | Description | Value Range | Notes | | 44 | | Field | Type | Description | Value Range | Notes | |
| 45 | -|:--| :-: | :-- | :-: | :-: | :-- | | 45 | +|:--|:-:|:--|:-:|:-:| |
| 46 | |batch_num|uint32|Number of batches used for quantization |1|/| | 46 | |batch_num|uint32|Number of batches used for quantization |1|/| |
| 47 | |skip_layers|str|Layers to skip quantization |/|Skip quantization layers support fuzzy matching. When the configured string is a layer name substring or matches the layer name, skip quantization for that layer and do not generate quantization configuration. The string must contain numbers or letters| | 47 | |skip_layers|str|Layers to skip quantization |/|Skip quantization layers support fuzzy matching. When the configured string is a layer name substring or matches the layer name, skip quantization for that layer and do not generate quantization configuration. The string must contain numbers or letters| |
| 48 | |weights.type|str|Quantized weight type|'mxfp8_e4m3fn'/'mxfp4_e2m1'|/| | 48 | |weights.type|str|Quantized weight type|'mxfp8_e4m3fn'/'mxfp4_e2m1'|/| |
| @@ -87,4 +87,4 @@ Where Score is the quantized model PPL. For specific values, refer to the follow | |||
| 87 | |QWEN3-8B|pileval|wikitext2|9.715|9.934| | 87 | |QWEN3-8B|pileval|wikitext2|9.715|9.934| |
| 88 | 88 | ||
| 89 | 89 | ||
| 90 | -After inference succeeds, a quantization log file ./amct_log/amct_pytorch.log is generated in the current directory | 90 | +After inference succeeds, a quantization log file ./amct_log/amct_pytorch.log is generated in the current directory |
| @@ -23,7 +23,7 @@ If you need to modify the detailed configuration, please refer to the documentat | |||
| 23 | The OFMR algorithm supports weight-only quantization and full quantization. The supported quantization types and quantization configurations are: | 23 | The OFMR algorithm supports weight-only quantization and full quantization. The supported quantization types and quantization configurations are: |
| 24 | 24 | ||
| 25 | | Field | Type | Description | Value Range | Notes | | 25 | | Field | Type | Description | Value Range | Notes | |
| 26 | -|:--| :-: | :-- | :-: | :-: | :-- | | 26 | +|:--|:-:|:--|:-:|:-:| |
| 27 | |batch_num|uint32|Number of batches used for quantization |1|/| | 27 | |batch_num|uint32|Number of batches used for quantization |1|/| |
| 28 | |skip_layers|str|Layers to skip quantization |/|Skip quantization layers support fuzzy matching. When the configured string is a layer name substring or matches the layer name, skip quantization for that layer and do not generate quantization configuration. The string must contain numbers or letters| | 28 | |skip_layers|str|Layers to skip quantization |/|Skip quantization layers support fuzzy matching. When the configured string is a layer name substring or matches the layer name, skip quantization for that layer and do not generate quantization configuration. The string must contain numbers or letters| |
| 29 | |weights.type|str|Quantized weight type|'float8_e4m3fn'/'hifloat8'|/| | 29 | |weights.type|str|Quantized weight type|'float8_e4m3fn'/'hifloat8'|/| |
| @@ -68,4 +68,4 @@ Where Score is the quantized model PPL. For specific values, refer to the follow | |||
| 68 | |QWEN2-7B|pileval|wikitext2|7.137|7.196| | 68 | |QWEN2-7B|pileval|wikitext2|7.137|7.196| |
| 69 | |QWEN3-8B|pileval|wikitext2|9.715|9.808| | 69 | |QWEN3-8B|pileval|wikitext2|9.715|9.808| |
| 70 | 70 | ||
| 71 | -After inference succeeds, a quantization log file ./amct_log/amct_pytorch.log is generated in the current directory | 71 | +After inference succeeds, a quantization log file ./amct_log/amct_pytorch.log is generated in the current directory |
| @@ -1,141 +1,141 @@ | |||
| 1 | -# AMCT Large Model SMOOTHQUANT Quantization | 1 | +# AMCT Large Model SMOOTHQUANT Quantization |
| 2 | - | 2 | + |
| 3 | -## 1 Quantization Prerequisites | 3 | +## 1 Quantization Prerequisites |
| 4 | - | 4 | + |
| 5 | -### 1.1 Install Dependencies | 5 | +### 1.1 Install Dependencies |
| 6 | - | 6 | + |
| 7 | -The dependency packages for this sample can be found in [requirements.txt](requirements.txt) | 7 | +The dependency packages for this sample can be found in [requirements.txt](requirements.txt) |
| 8 | - | 8 | + |
| 9 | -Note that the torch_npu package version needs to match the Python and torch package versions, and the CANN package needs to be installed | 9 | +Note that the torch_npu package version needs to match the Python and torch package versions, and the CANN package needs to be installed |
| 10 | - | 10 | + |
| 11 | -### 1.2 Model and Dataset Preparation | 11 | +### 1.2 Model and Dataset Preparation |
| 12 | - | 12 | + |
| 13 | -This sample uses Llama2-7b, qwen2-7b, and qwen3-8b models, pileval data, and wikitext2 dataset as examples. Data is loaded online, and models need to be downloaded by users themselves and the model path needs to be specified when executing the script. | 13 | +This sample uses Llama2-7b, qwen2-7b, and qwen3-8b models, pileval data, and wikitext2 dataset as examples. Data is loaded online, and models need to be downloaded by users themselves and the model path needs to be specified when executing the script. |
| 14 | - | 14 | + |
| 15 | -Note: The quantization data type combination float8_e4m3fn * float4_e2m1 only supports quantizing original data type torch.bfloat16. Please modify the data type when getting the model in the src/utils.py file. | 15 | +Note: The quantization data type combination float8_e4m3fn * float4_e2m1 only supports quantizing original data type torch.bfloat16. Please modify the data type when getting the model in the src/utils.py file. |
| 16 | - | 16 | + |
| 17 | -### 1.3 Simple Quantization Configuration | 17 | +### 1.3 Simple Quantization Configuration |
| 18 | -The quantization configuration used in this sample is built into the tool and can be obtained and used in the following ways: | 18 | +The quantization configuration used in this sample is built into the tool and can be obtained and used in the following ways: |
| 19 | - | 19 | + |
| 20 | -Quantization data type combination int8 * int8 configuration: | 20 | +Quantization data type combination int8 * int8 configuration: |
| 21 | -`from amct_pytorch import INT8_SMOOTHQUANT_CFG` | 21 | +`from amct_pytorch import INT8_SMOOTHQUANT_CFG` |
| 22 | -Quantization data type combination float8_e4m3fn * float4_e2m1 configuration: | 22 | +Quantization data type combination float8_e4m3fn * float4_e2m1 configuration: |
| 23 | -```python | 23 | +```python |
| 24 | -cfg = { | 24 | +cfg = { |
| 25 | - 'batch_num': 1, | 25 | + 'batch_num': 1, |
| 26 | - 'quant_cfg': { | 26 | + 'quant_cfg': { |
| 27 | - 'weights': { | 27 | + 'weights': { |
| 28 | - 'type': 'float4_e2m1', | 28 | + 'type': 'float4_e2m1', |
| 29 | - 'symmetric': True, | 29 | + 'symmetric': True, |
| 30 | - 'strategy': 'group', | 30 | + 'strategy': 'group', |
| 31 | - 'group_size': 32 | 31 | + 'group_size': 32 |
| 32 | - }, | 32 | + }, |
| 33 | - 'inputs': { | 33 | + 'inputs': { |
| 34 | - 'type': 'float8_e4m3fn', | 34 | + 'type': 'float8_e4m3fn', |
| 35 | - 'symmetric': True, | 35 | + 'symmetric': True, |
| 36 | - 'strategy': 'tensor', | 36 | + 'strategy': 'tensor', |
| 37 | - }, | 37 | + }, |
| 38 | - }, | 38 | + }, |
| 39 | - 'algorithm': {'smoothquant': {'smooth_strength': 0.77}}, | 39 | + 'algorithm': {'smoothquant': {'smooth_strength': 0.77}}, |
| 40 | - 'skip_layers': {'lm_head', 'down_proj'} | 40 | + 'skip_layers': {'lm_head', 'down_proj'} |
| 41 | -} | 41 | +} |
| 42 | -``` | 42 | +``` |
| 43 | -Quantization data type combination hifloat8 * hifloat8 configuration: | 43 | +Quantization data type combination hifloat8 * hifloat8 configuration: |
| 44 | -`from amct_pytorch import HIFP8_SMOOTHQUANT_CFG` | 44 | +`from amct_pytorch import HIFP8_SMOOTHQUANT_CFG` |
| 45 | - | 45 | + |
| 46 | -The base configuration is as follows, and `smooth_strength=0.5` and `0.8` can be tested respectively: | 46 | +The base configuration is as follows, and `smooth_strength=0.5` and `0.8` can be tested respectively: |
| 47 | -```python | 47 | +```python |
| 48 | -{ | 48 | +{ |
| 49 | - 'batch_num': 1, | 49 | + 'batch_num': 1, |
| 50 | - 'quant_cfg': { | 50 | + 'quant_cfg': { |
| 51 | - 'weights': { | 51 | + 'weights': { |
| 52 | - 'type': 'hifloat8', | 52 | + 'type': 'hifloat8', |
| 53 | - 'symmetric': True, | 53 | + 'symmetric': True, |
| 54 | - 'strategy': 'channel', | 54 | + 'strategy': 'channel', |
| 55 | - }, | 55 | + }, |
| 56 | - 'inputs': { | 56 | + 'inputs': { |
| 57 | - 'type': 'hifloat8', | 57 | + 'type': 'hifloat8', |
| 58 | - 'symmetric': True, | 58 | + 'symmetric': True, |
| 59 | - 'strategy': 'tensor', | 59 | + 'strategy': 'tensor', |
| 60 | - }, | 60 | + }, |
| 61 | - }, | 61 | + }, |
| 62 | - 'algorithm': {'smoothquant': {'smooth_strength': 0.5}}, | 62 | + 'algorithm': {'smoothquant': {'smooth_strength': 0.5}}, |
| 63 | - 'skip_layers': {'lm_head'} | 63 | + 'skip_layers': {'lm_head'} |
| 64 | -} | 64 | +} |
| 65 | -``` | 65 | +``` |
| 66 | - | 66 | + |
| 67 | -If you need to modify the detailed configuration, please refer to the documentation to construct the required quantization configuration dict. | 67 | +If you need to modify the detailed configuration, please refer to the documentation to construct the required quantization configuration dict. |
| 68 | - | 68 | + |
| 69 | -The SmoothQuant algorithm only supports full quantization. The supported quantization types and quantization configurations are: | 69 | +The SmoothQuant algorithm only supports full quantization. The supported quantization types and quantization configurations are: |
| 70 | - | 70 | + |
| 71 | -| Field | Type | Description | Value Range | Notes | | 71 | +| Field | Type | Description | Value Range | Notes | |
| 72 | -|:--| :-: | :-- | :-: | :-: | :-- | | 72 | +|:--|:-:|:--|:-:|:-:| |
| 73 | -|batch_num|uint32|Number of batches used for quantization |1|/| | 73 | +|batch_num|uint32|Number of batches used for quantization |1|/| |
| 74 | -|skip_layers|str|Layers to skip quantization |/|Skip quantization layers support fuzzy matching. When the configured string is a layer name substring or matches the layer name, skip quantization for that layer and do not generate quantization configuration. The string must contain numbers or letters| | 74 | +|skip_layers|str|Layers to skip quantization |/|Skip quantization layers support fuzzy matching. When the configured string is a layer name substring or matches the layer name, skip quantization for that layer and do not generate quantization configuration. The string must contain numbers or letters| |
| 75 | -|weights.type|str|Quantized weight type|'int8'/'float4_e2m1'/'hifloat8'|/| | 75 | +|weights.type|str|Quantized weight type|'int8'/'float4_e2m1'/'hifloat8'|/| |
| 76 | -|weights.symmetric|bool|Symmetric quantization|TRUE/FALSE|When quantization data type is float4_e2m1, only symmetric quantization is supported| | 76 | +|weights.symmetric|bool|Symmetric quantization|TRUE/FALSE|When quantization data type is float4_e2m1, only symmetric quantization is supported| |
| 77 | -|weights.strategy|str|Quantization granularity|'tensor'/'channel'/'group'|When quantization strategy is group, only quantization data type float4_e2m1 is supported, and float4_e2m1 only supports group| | 77 | +|weights.strategy|str|Quantization granularity|'tensor'/'channel'/'group'|When quantization strategy is group, only quantization data type float4_e2m1 is supported, and float4_e2m1 only supports group| |
| 78 | -|inputs.type|str|Quantized activation type|'int8'/'float8_e4m3fn'/'hifloat8'|/| | 78 | +|inputs.type|str|Quantized activation type|'int8'/'float8_e4m3fn'/'hifloat8'|/| |
| 79 | -|inputs.symmetric|bool|Symmetric quantization|TRUE/FALSE|When quantization strategy is token, asymmetric quantization is not supported, and per-group quantization is not supported; when quantization data type is float8_e4m3fn, only symmetric quantization is supported| | 79 | +|inputs.symmetric|bool|Symmetric quantization|TRUE/FALSE|When quantization strategy is token, asymmetric quantization is not supported, and per-group quantization is not supported; when quantization data type is float8_e4m3fn, only symmetric quantization is supported| |
| 80 | -|inputs.strategy|str|Quantization granularity|'tensor'/'token'|When quantization data type is float8_e4m3fn, only quantization strategy tensor is supported| | 80 | +|inputs.strategy|str|Quantization granularity|'tensor'/'token'|When quantization data type is float8_e4m3fn, only quantization strategy tensor is supported| |
| 81 | -|algorithm|dict|Quantization algorithm configuration used|{'smoothquant'}|/| | 81 | +|algorithm|dict|Quantization algorithm configuration used|{'smoothquant'}|/| |
| 82 | -|algorithm.smoothquant.smooth_strength|float|SmoothQuant algorithm parameter: migration strength|0-1|Does not include 0/1| | 82 | +|algorithm.smoothquant.smooth_strength|float|SmoothQuant algorithm parameter: migration strength|0-1|Does not include 0/1| |
| 83 | - | 83 | + |
| 84 | - | 84 | + |
| 85 | -## 2 Quantization Example | 85 | +## 2 Quantization Example |
| 86 | - | 86 | + |
| 87 | -### 2.1 Use Interface Method to Call | 87 | +### 2.1 Use Interface Method to Call |
| 88 | - | 88 | + |
| 89 | -**step 1.** Please execute the following command in the current directory to run the sample program. Users need to modify the model and dataset paths in the sample program according to actual conditions: | 89 | +**step 1.** Please execute the following command in the current directory to run the sample program. Users need to modify the model and dataset paths in the sample program according to actual conditions: |
| 90 | - | 90 | + |
| 91 | -```python | 91 | +```python |
| 92 | -python3 src/run_llama2_samples.py --model_path=/data/Llama2_7b_hf/ | 92 | +python3 src/run_llama2_samples.py --model_path=/data/Llama2_7b_hf/ |
| 93 | -``` | 93 | +``` |
| 94 | - | 94 | + |
| 95 | -```python | 95 | +```python |
| 96 | -python3 src/run_qwen_samples.py --model_path=/data/Qwen2-7b/ | 96 | +python3 src/run_qwen_samples.py --model_path=/data/Qwen2-7b/ |
| 97 | -``` | 97 | +``` |
| 98 | - | 98 | + |
| 99 | -```python | 99 | +```python |
| 100 | -python3 src/run_qwen_samples.py --model_path=/data/Qwen3-8b/ | 100 | +python3 src/run_qwen_samples.py --model_path=/data/Qwen3-8b/ |
| 101 | -``` | 101 | +``` |
| 102 | - | 102 | + |
| 103 | - | 103 | + |
| 104 | -If the following information appears, it indicates that quantization is successful: | 104 | +If the following information appears, it indicates that quantization is successful: |
| 105 | - | 105 | + |
| 106 | -```none | 106 | +```none |
| 107 | -Test time taken: 1.0 min 59.24865388870239 s | 107 | +Test time taken: 1.0 min 59.24865388870239 s |
| 108 | -Score: 5.477707 | 108 | +Score: 5.477707 |
| 109 | -``` | 109 | +``` |
| 110 | -Where Score is the quantized model PPL. For specific values, refer to the following table: | 110 | +Where Score is the quantized model PPL. For specific values, refer to the following table: |
| 111 | - | 111 | + |
| 112 | -| Model | Calibration Set | Dataset | Pre-quantization PPL | Post-int8*int8 quantization PPL | Post-float8_e4m3fn*float4_e2m1 quantization PPL | | 112 | +| Model | Calibration Set | Dataset | Pre-quantization PPL | Post-int8*int8 quantization PPL | Post-float8_e4m3fn*float4_e2m1 quantization PPL | |
| 113 | -| :-: | :-: | :-: | :-: | :-: | :-: | | 113 | +| :-: | :-: | :-: | :-: | :-: | :-: | |
| 114 | -|LLAMA2-7B|pileval|wikitext2|5.472|5.673|5.589| | 114 | +|LLAMA2-7B|pileval|wikitext2|5.472|5.673|5.589| |
| 115 | -|QWEN2-7B|pileval|wikitext2|7.137|7.155|7.252| | 115 | +|QWEN2-7B|pileval|wikitext2|7.137|7.155|7.252| |
| 116 | -|QWEN3-8B|pileval|wikitext2|9.715|9.861|9.931| | 116 | +|QWEN3-8B|pileval|wikitext2|9.715|9.861|9.931| |
| 117 | - | 117 | + |
| 118 | - | 118 | + |
| 119 | -### 2.2 hifloat8 * hifloat8 accuracy reference (Qwen3-8B + wikitext2) | 119 | +### 2.2 hifloat8 * hifloat8 accuracy reference (Qwen3-8B + wikitext2) |
| 120 | - | 120 | + |
| 121 | -Using `HIFP8_SMOOTHQUANT_CFG` (weights: hifloat8/channel, inputs: hifloat8/tensor, skip_layers=`{'lm_head'}`), | 121 | +Using `HIFP8_SMOOTHQUANT_CFG` (weights: hifloat8/channel, inputs: hifloat8/tensor, skip_layers=`{'lm_head'}`), |
| 122 | -full PPL evaluation on Qwen3-8B and wikitext2-raw-v1/test. Accuracy before and after quantization: | 122 | +full PPL evaluation on Qwen3-8B and wikitext2-raw-v1/test. Accuracy before and after quantization: |
| 123 | - | 123 | + |
| 124 | -Full 2048-token evaluation (146 segments, pre-quantization PPL 9.7252): | 124 | +Full 2048-token evaluation (146 segments, pre-quantization PPL 9.7252): |
| 125 | - | 125 | + |
| 126 | -| smooth_strength | Post-quantization PPL | PPL delta | Relative degradation | | 126 | +| smooth_strength | Post-quantization PPL | PPL delta | Relative degradation | |
| 127 | -| ---: | ---: | ---: | ---: | | 127 | +| ---: | ---: | ---: | ---: | |
| 128 | -| 0.5 | 9.7719 | +0.0467 | +0.4804% | | 128 | +| 0.5 | 9.7719 | +0.0467 | +0.4804% | |
| 129 | -| 0.8 | 9.7357 | +0.0105 | +0.1080% | | 129 | +| 0.8 | 9.7357 | +0.0105 | +0.1080% | |
| 130 | - | 130 | + |
| 131 | -Full 4096-token evaluation (73 segments, pre-quantization PPL 8.9975): | 131 | +Full 4096-token evaluation (73 segments, pre-quantization PPL 8.9975): |
| 132 | - | 132 | + |
| 133 | -| smooth_strength | Post-quantization PPL | PPL delta | Relative degradation | | 133 | +| smooth_strength | Post-quantization PPL | PPL delta | Relative degradation | |
| 134 | -| ---: | ---: | ---: | ---: | | 134 | +| ---: | ---: | ---: | ---: | |
| 135 | -| 0.5 | 9.0288 | +0.0313 | +0.3481% | | 135 | +| 0.5 | 9.0288 | +0.0313 | +0.3481% | |
| 136 | -| 0.8 | 9.0359 | +0.0385 | +0.4276% | | 136 | +| 0.8 | 9.0359 | +0.0385 | +0.4276% | |
| 137 | - | 137 | + |
| 138 | -Under 2048-token evaluation, `smooth_strength=0.8` is more accurate than the default `0.5`; under 4096-token evaluation, `smooth_strength=0.5` is slightly better than `0.8`. In practice, `smooth_strength` can be tuned according to sequence length. | 138 | +Under 2048-token evaluation, `smooth_strength=0.8` is more accurate than the default `0.5`; under 4096-token evaluation, `smooth_strength=0.5` is slightly better than `0.8`. In practice, `smooth_strength` can be tuned according to sequence length. |
| 139 | - | 139 | + |
| 140 | - | 140 | + |
| 141 | -After inference succeeds, a quantization log file ./amct_log/amct_pytorch.log is generated in the current directory | 141 | +After inference succeeds, a quantization log file ./amct_log/amct_pytorch.log is generated in the current directory |