[feat]新增一站式平台verl-torchtitan qwen1.7B SFT训练样例-cann-recipes-train-AtomGit

已合并

[feat]新增一站式平台verl-torchtitan qwen1.7B SFT训练样例 #158

horizon_C创建于 7月23日从horizon_C/cann-recipes-train： master 合入到 cann/cann-recipes-train： master

[feat]新增一站式平台verl-torchtitan qwen1.7B SFT训练样例 #158

已合并

horizon_C创建于 7月23日

从horizon_C/cann-recipes-train master合入到cann/cann-recipes-train master

共 5 个文件变更+628-1

MREADME.md+2-1

	@@ -37,8 +37,9 @@ cann-recipes-train 针对该平台在相应样例 README 中提供了简化的
37		37
38	\|实践\|简介\|	38	\|实践\|简介\|
39	\|-----\|-----\|	39	\|-----\|-----\|
40	-\|[Qwen3-1.7B SFT训练样例](llm_sft/qwen3/README.md#一站式平台快速启动sft训练示例) \|在一站式平台Atlas A2/A3环境中完成Qwen3-1.7B 单卡SFT训练。\|	40	+\|[Qwen3-1.7B 基于mindspeed SFT训练样例](llm_sft/qwen3/README.md#一站式平台快速启动sft训练示例) \|在一站式平台Atlas A2/A3环境中完成Qwen3-1.7B 单卡SFT训练。\|
41	\|[Qwen2.5-1.5B RL训练样例](llm_rl/qwen2_5/verl_npu_demo/README_single.md) \|在一站式平台Atlas A2/A3环境中基于verl框架完成Qwen2.5-1.5B-Instruct 单卡RL训练。\|	41	\|[Qwen2.5-1.5B RL训练样例](llm_rl/qwen2_5/verl_npu_demo/README_single.md) \|在一站式平台Atlas A2/A3环境中基于verl框架完成Qwen2.5-1.5B-Instruct 单卡RL训练。\|
		42	+\|[Qwen3-1.7B 基于torchtitan SFT训练样例](llm_sft/qwen3_1.7B_torchtitan/README.md) \|在一站式平台Atlas A2/A3环境中完成Qwen3-1.7B 单卡SFT训练。\|
42		43
43	## 💡 特性介绍	44	## 💡 特性介绍
44	本项目在探索最佳实践的过程中引入了如下特性：	45	本项目在探索最佳实践的过程中引入了如下特性：

Allm_sft/qwen3_1.7B_torchtitan/README.md+274-0

@@ -0,0 +1,274 @@
	1	+# Qwen3-1.7B SFT训练示例
	2	+
	3	+## 硬件要求
	4	+卡数：1张A2
	5	+
	6	+
	7	+## 一站式平台快速启动SFT训练示例
	8	+
	9	+### 环境要求
	10	+一站式平台模板选择：cann_master-py3.12-A2-arm-20260630
	11	+
	12	+### 项目及依赖构建
	13	+在当前目录下（cann-recipes-train/llm_sft/qwen3_1.7B_torchtitan/）执行:
	14	+```bash
	15	+# 下载本样例所在代码仓，并安装依赖
	16	+bash build_project_platform.sh
	17	+# 进入目录
	18	+cd torchtitan-npu
	19	+# 快速验证环境是否正常
	20	+NGPU=1 bash scripts/run_train.sh
	21	+```
	22	+
	23	+预计输出：
	24	+Running with configs: model and recipe resolved from the current master defaults
	25	+visible dies: 2
	26	+step: 1 loss: <finite> grad_norm: <finite>
	27	+Training completed
	28	+
	29	+
	30	+注：需要按照实际情况在以上脚本中修改 ascend-toolkit 路径, 一站式平台需要修改（以CANN 9.1.0为例）：
	31	+```
	32	+/home/developer/Ascend/cann-9.1.0/set_env.sh
	33	+```
	34	+
	35	+## 模型权重准备
	36	+本样例使用的Qwen3-1.7B模型权重准备方法如下：
	37	+```bash
	38	+# 从魔塔社区下载模型的基础文件，存放在当前目录的 ./assets/hf/Qwen3-1.7B 目录下
	39	+MODEL_DIR="./assets/hf/Qwen3-1.7B"
	40	+python3 -m pip install -U modelscope
	41	+modelscope download \
	42	+ --model PrimeIntellect/Qwen3-1.7B \
	43	+ --local_dir "$MODEL_DIR"
	44	+
	45	+pwd
	46	+ls ./assets/hf/Qwen3-1.7B
	47	+```
	48	+
	49	+## 数据集准备
	50	+本样例使用 willcb/V3-wordle 数据集：
	51	+```bash
	52	+# 1. 下载数据集（源文件已经是 Parquet 格式）
	53	+HF_HUB_DISABLE_XET=1 HF_ENDPOINT=https://hf-mirror.com hf download willcb/V3-wordle data/train-00000-of-00001.parquet --repo-type=dataset --local-dir ./assets/data/wordle_raw
	54	+
	55	+# 2. 复制到 torchtitan 约定的数据路径
	56	+mkdir -p ./assets/data/wordle
	57	+cp ./assets/data/wordle_raw/data/train-00000-of-00001.parquet ./assets/data/wordle
	58	+```
	59	+
	60	+Wordle 评测环境依赖 NLTK 语料库（words 和 averaged_perceptron_tagger）。以下脚本从 NLTK 官方源下载并自动解压到 $HOME/developer/nltk_data（约需 7 分钟）：
	61	+
	62	+```bash
	63	+pip install nltk
	64	+python3 -c "
	65	+import nltk
	66	+nltk.download('words')
	67	+nltk.download('averaged_perceptron_tagger_eng')
	68	+nltk.download('averaged_perceptron_tagger')
	69	+"
	70	+```
	71	+
	72	+查看数据集格式（从本地读取）：
	73	+
	74	+```bash
	75	+python3 -c "
	76	+from datasets import load_dataset
	77	+import pprint
	78	+ds = load_dataset('./assets/data/wordle', split='train')
	79	+pprint.pprint(next(iter(ds)), width=100, depth=3)
	80	+"
	81	+```
	82	+
	83	+`process_wordle_sample` processor 会将 `prompt` 和 `completion` 拼接为完整的 `[system, user, assistant, user, assistant, ...]` 消息列表。
	84	+
	85	+
	86	+## 基线 SFT 训练与 Wordle 评测
	87	+
	88	+使用 TorchTitan-NPU 完成 Qwen3-1.7B Wordle 数据集 SFT 训练，并通过 `vf-eval` 对训练后的模型进行交互式 Wordle 评测。
	89	+
	90	+### 1. 启动 SFT 训练
	91	+
	92	+使用 TorchTitan-NPU 对 Qwen3-1.7B 进行 Wordle 数据集 SFT：
	93	+
	94	+```bash
	95	+MODULE=torchtitan_npu.models.qwen3 \
	96	+CONFIG=sft_qwen3_1_7b_wordle \
	97	+NGPU=1 \
	98	+bash scripts/run_train.sh \
	99	+--hf_assets_path "assets/hf/Qwen3-1.7B" \
	100	+--checkpoint.folder checkpoint_wordle_sft \
	101	+--checkpoint.last_save_in_hf \
	102	+--checkpoint.enable \
	103	+--checkpoint.initial_load_in_hf \
	104	+dataloader:chat_data_loader_config \
	105	+--dataloader.dataset_path "assets/data/wordle"
	106	+```
	107	+
	108	+参数说明：
	109	+
	110	+\| 参数 \| 说明 \|
	111	+\|---\|---\|
	112	+\| `MODULE` \| 使用 Qwen3 TorchTitan-NPU 模型配置 \|
	113	+\| `CONFIG` \| Wordle SFT 配置文件 \|
	114	+\| `NGPU` \| 使用 GPU/NPU 数量 \|
	115	+\| `--checkpoint.folder` \| checkpoint 保存目录 \|
	116	+\| `--checkpoint.last_save_in_hf` \| 训练结束时保存 HuggingFace 格式 checkpoint \|
	117	+\| `--checkpoint.initial_load_in_hf` \| 从 HuggingFace 格式模型初始化 \|
	118	+\| `chat_data_loader_config` \| 使用对话格式数据加载器 \|
	119	+\| `--dataloader.dataset_path` \| Wordle SFT 数据集路径 \|
	120	+
	121	+训练完成后，checkpoint 默认保存于：
	122	+
	123	+```text
	124	+outputs/checkpoint_wordle_sft/
	125	+```
	126	+
	127	+### 2. 准备 HuggingFace 推理 checkpoint
	128	+
	129	+`infer_server.py` 使用 HuggingFace `AutoTokenizer` 加载 tokenizer。
	130	+
	131	+部分 TorchTitan checkpoint 主要保存模型权重和训练状态，不包含完整 tokenizer 文件，因此需要将原始 Qwen3 模型中的 tokenizer 配置补充到 checkpoint 目录。
	132	+
	133	+执行：
	134	+
	135	+```bash
	136	+src=assets/hf/Qwen3-1.7B
	137	+dst=outputs/checkpoint_wordle_sft/step-20
	138	+
	139	+test -d "$dst"
	140	+
	141	+for name in \
	142	+config.json \
	143	+tokenizer.json \
	144	+tokenizer_config.json \
	145	+special_tokens_map.json
	146	+do
	147	+ if test -f "$src/$name"; then
	148	+ cp "$src/$name" "$dst/$name"
	149	+ fi
	150	+done
	151	+
	152	+for name in \
	153	+generation_config.json \
	154	+vocab.json \
	155	+merges.txt
	156	+do
	157	+ if test -f "$src/$name"; then
	158	+ cp "$src/$name" "$dst/$name"
	159	+ fi
	160	+done
	161	+
	162	+echo "Loadable checkpoint:"
	163	+find "$dst" -maxdepth 1 -type f -printf '%f\n' \| sort
	164	+```
	165	+
	166	+### 3. 启动推理服务
	167	+```bash
	168	+source /home/developer/Ascend/cann-9.1.0/set_env.sh
	169	+# 关闭已有推理服务（可选）
	170	+pkill -f infer_server.py \|\| true
	171	+
	172	+# 安装推理依赖
	173	+pip install transformers
	174	+pip install torchvision==0.27.0+cpu \
	175	+--index-url https://download.pytorch.org/whl/cpu
	176	+# 启动 inference server
	177	+
	178	+python3 scripts/infer_server.py \
	179	+--model ./outputs/checkpoint_wordle_sft/step-20 \
	180	+--port 8000 \
	181	+> /tmp/infer_server.log 2>&1 &
	182	+
	183	+sleep 10
	184	+
	185	+curl http://localhost:8000/health
	186	+# {"status":"ok"}
	187	+```
	188	+
	189	+### 4. 配置 vf-eval 评测环境
	190	+
	191	+`vf-eval` 是基于 Prime-RL 的交互式评测工具，用于测试模型在 Wordle 环境中的推理能力。
	192	+
	193	+评测流程如下：
	194	+
	195	+1. 环境从预留评估词中选择秘密词；
	196	+2. 推理服务器接收系统提示和 Wordle 游戏规则；
	197	+3. 模型生成预测：
	198	+
	199	+```xml
	200	+<think>
	201	+...
	202	+</think>
	203	+<guess>[word]</guess>
	204	+```
	205	+
	206	+4. Wordle 环境解析 `<guess>` 内容；
	207	+5. 根据预测结果返回 G/Y/X 字母反馈；
	208	+6. 模型根据反馈继续预测；
	209	+7. 最多进行 6 轮交互；
	210	+8. 游戏结束后计算最终 reward。
	211	+
	212	+Reward 由以下部分组成：
	213	+
	214	+\| Reward \| 含义 \|
	215	+\|---\|---\|
	216	+\| `correct_answer` \| 是否猜中秘密词 \|
	217	+\| `partial_answer` \| 根据 green/yellow 字母数量提供部分奖励 \|
	218	+\| `length_bonus` \| 鼓励更少轮次完成任务 \|
	219	+\| `format_reward` \| 检查 `<guess>` 输出格式是否正确 \|
	220	+
	221	+初始化 vf-eval 环境：
	222	+
	223	+```bash
	224	+bash ../setup_prime_rl.sh
	225	+```
	226	+
	227	+### 5. 运行 Wordle 评测
	228	+```bash
	229	+# 激活 vf-eval 环境：
	230	+PRIMERL_DIR=./prime-rl
	231	+source "$PRIMERL_DIR/.venv-wordle-legacy/bin/activate"
	232	+
	233	+# 启动评测：
	234	+vf-eval wordle \
	235	+--num-examples 4 \
	236	+--rollouts-per-example 2 \
	237	+--api-base-url http://127.0.0.1:8000/v1 \
	238	+--max-concurrent 1 \
	239	+--verbose \
	240	+--temperature 0.6 \
	241	+--save-results
	242	+```
	243	+
	244	+参数说明：
	245	+
	246	+\| 参数 \| 说明 \|
	247	+\|---\|---\|
	248	+\| `--num-examples` \| 评测样本数量 \|
	249	+\| `--rollouts-per-example` \| 每个样本采样次数 \|
	250	+\| `--api-base-url` \| inference server 地址 \|
	251	+\| `--max-concurrent` \| 串行请求（避免单线程服务器过载）\|
	252	+\| `--temperature` \| 生成采样温度 \|
	253	+\| `--save-results` \| 保存评测结果 \|
	atomgit-bot7月23日 🟡 Medium Priority 在 `vf-eval wordle` 命令块的最后一行，`--save-results \` 包含一个尾部反斜杠 `\`，它是 shell 的续行符。当用户复制整个代码块并粘贴到终端时，shell 会将反斜杠解释为"下一行还有命令"，从而显示 `>` 提示符等待更多输入，导致命令无法正常执行。用户必须手动删除反斜杠或按 Ctrl-C 中断。该问题同时存在于中英文两个 README 文件中。建议：将 `--save-results \` 末尾的反斜杠去除，改为 `--save-results`。同时修复 README_EN.md 中相同的问题（第 251 行）。不准确？
	254	+
	255	+### 结果分析
	256	+
	257	+基线模型与 SFT 后的参考性能对比：
	258	+
	259	+\| 指标 \| Base Qwen3-1.7B \| Wordle SFT \| 变化 \|
	260	+\|------\|----------------\|------------\|------\|
	261	+\| 平均 format_reward \| 0.60 \| 1.0 \| +0.4 \|
	262	+\| 平均 correct_answer \| 0.00 \| 0.00 \| 持平 \|
	263	+\| 平均 partial_answer \| 0.00 \| 0.25 \| +0.25 \|
	264	+\| 平均 length_bonus \| 0.00 \| 0.00 \| 持平 \|
	265	+\| 平均 reward \| 0.22 \| 0.4 \| +0.18 \|
	266	+
	267	+格式正确性显著提升：format_reward 从 0.6 提升至 1.0，说明 SFT 阶段成功让模型学会了符合 Wordle 环境的 XML 交互格式，这与“SFT学习格式”的目标一致。
	268	+
	269	+部分正确性有所改善：partial_answer 从 0 提高到 0.25，表明模型开始能够生成包含正确字母（G/Y 反馈）的猜测，逐步掌握游戏逻辑，但猜中率仍然为 0（correct_answer 未变）。
	270	+
	271	+整体奖励上升：平均总奖励从 0.22 增至 0.4，增益主要来自格式和部分正确性的改进，但最终获胜率（length_bonus 仍为 0）未见提升，说明模型尚未能完整解出单词。
	272	+
	273	+与官方基线对比： Qwen3-1.7B 基线（0.22）接近官方报告的 ~0.2，SFT 后（0.4）也基本符合预期。预计在 RL 阶段后平均奖励可达 ~1.5 且胜率约 60%，表明当前 SFT 模型是 RL 的基座，需要 RL 提升其策略能力。
	274	+

Allm_sft/qwen3_1.7B_torchtitan/README_EN.md+283-0

@@ -0,0 +1,283 @@
	1	+# Qwen3-1.7B SFT Training Example
	2	+
	3	+## Hardware Requirements
	4	+
	5	+Number of cards: 1 A2
	6	+
	7	+## Quick Start for SFT Training on the One-Stop Platform
	8	+
	9	+### Environment Requirements
	10	+
	11	+Select the following one-stop platform template: `cann_master-py3.12-A2-arm-20260630`
	12	+
	13	+### Project and Dependency Setup
	14	+
	15	+Run the following commands in the current directory (`cann-recipes-train/llm_sft/qwen3_1.7B_torchtitan/`):
	16	+
	17	+```bash
	18	+# Download the repository containing this example and install its dependencies
	19	+bash build_project_platform.sh
	20	+# Enter the directory
	21	+cd torchtitan-npu
	22	+# Quickly verify that the environment is working correctly
	23	+NGPU=1 bash scripts/run_train.sh
	24	+```
	25	+
	26	+Expected output:
	27	+
	28	+Running with configs: model and recipe resolved from the current master defaults
	29	+visible dies: 2
	30	+step: 1 loss: <finite> grad_norm: <finite>
	31	+Training completed
	32	+
	33	+Note: Update the `ascend-toolkit` path in the scripts above according to your environment. On the one-stop platform, modify it as follows (using CANN 9.1.0 as an example):
	34	+
	35	+```
	36	+/home/developer/Ascend/cann-9.1.0/set_env.sh
	37	+```
	38	+
	39	+## Preparing Model Weights
	40	+
	41	+Prepare the Qwen3-1.7B model weights used in this example as follows:
	42	+
	43	+```bash
	44	+# Download the base model files from ModelScope and store them in
	45	+# ./assets/hf/Qwen3-1.7B under the current directory
	46	+MODEL_DIR="./assets/hf/Qwen3-1.7B"
	47	+python3 -m pip install -U modelscope
	48	+modelscope download \
	49	+ --model PrimeIntellect/Qwen3-1.7B \
	50	+ --local_dir "$MODEL_DIR"
	51	+
	52	+pwd
	53	+ls ./assets/hf/Qwen3-1.7B
	54	+```
	55	+
	56	+## Preparing the Dataset
	57	+
	58	+This example uses the `willcb/V3-wordle` dataset:
	59	+
	60	+```bash
	61	+# 1. Download the dataset (the source file is already in Parquet format)
	62	+HF_HUB_DISABLE_XET=1 HF_ENDPOINT=https://hf-mirror.com hf download willcb/V3-wordle data/train-00000-of-00001.parquet --repo-type=dataset --local-dir ./assets/data/wordle_raw
	63	+
	64	+# 2. Copy it to the data path expected by torchtitan
	65	+mkdir -p ./assets/data/wordle
	66	+cp ./assets/data/wordle_raw/data/train-00000-of-00001.parquet ./assets/data/wordle
	67	+```
	68	+
	69	+The Wordle evaluation environment depends on the NLTK corpora (`words` and `averaged_perceptron_tagger`). The following commands download them from the official NLTK source and automatically extract them to `$HOME/developer/nltk_data` (this takes approximately seven minutes):
	70	+
	71	+```bash
	72	+pip install nltk
	73	+python3 -c "
	74	+import nltk
	75	+nltk.download('words')
	76	+nltk.download('averaged_perceptron_tagger_eng')
	77	+nltk.download('averaged_perceptron_tagger')
	78	+"
	79	+```
	80	+
	81	+Inspect the dataset format by loading it locally:
	82	+
	83	+```bash
	84	+python3 -c "
	85	+from datasets import load_dataset
	86	+import pprint
	87	+ds = load_dataset('./assets/data/wordle', split='train')
	88	+pprint.pprint(next(iter(ds)), width=100, depth=3)
	89	+"
	90	+```
	91	+
	92	+The `process_wordle_sample` processor concatenates `prompt` and `completion` into a complete `[system, user, assistant, user, assistant, ...]` message list.
	93	+
	94	+## Baseline SFT Training and Wordle Evaluation
	95	+
	96	+Use TorchTitan-NPU to perform SFT on Qwen3-1.7B with the Wordle dataset, and then interactively evaluate the trained model on Wordle using `vf-eval`.
	97	+
	98	+### 1. Start SFT Training
	99	+
	100	+Use TorchTitan-NPU to perform SFT on Qwen3-1.7B with the Wordle dataset:
	101	+
	102	+```bash
	103	+MODULE=torchtitan_npu.models.qwen3 \
	104	+CONFIG=sft_qwen3_1_7b_wordle \
	105	+NGPU=1 \
	106	+bash scripts/run_train.sh \
	107	+--hf_assets_path "assets/hf/Qwen3-1.7B" \
	108	+--checkpoint.folder checkpoint_wordle_sft \
	109	+--checkpoint.last_save_in_hf \
	110	+--checkpoint.enable \
	111	+--checkpoint.initial_load_in_hf \
	112	+dataloader:chat_data_loader_config \
	113	+--dataloader.dataset_path "assets/data/wordle"
	114	+```
	115	+
	116	+Parameter descriptions:
	117	+
	118	+\| Parameter \| Description \|
	119	+\|---\|---\|
	120	+\| `MODULE` \| Uses the Qwen3 TorchTitan-NPU model configuration \|
	121	+\| `CONFIG` \| Wordle SFT configuration file \|
	122	+\| `NGPU` \| Number of GPUs/NPUs to use \|
	123	+\| `--checkpoint.folder` \| Directory in which to save checkpoints \|
	124	+\| `--checkpoint.last_save_in_hf` \| Saves the final checkpoint in Hugging Face format when training ends \|
	125	+\| `--checkpoint.initial_load_in_hf` \| Initializes from a model in Hugging Face format \|
	126	+\| `chat_data_loader_config` \| Uses the conversational-format data loader \|
	127	+\| `--dataloader.dataset_path` \| Path to the Wordle SFT dataset \|
	128	+
	129	+After training is complete, the checkpoint is saved by default to:
	130	+
	131	+```text
	132	+outputs/checkpoint_wordle_sft/
	133	+```
	134	+
	135	+### 2. Prepare the Hugging Face Inference Checkpoint
	136	+
	137	+`infer_server.py` uses Hugging Face `AutoTokenizer` to load the tokenizer.
	138	+
	139	+Some TorchTitan checkpoints primarily store model weights and training state and do not include the complete tokenizer files. Therefore, the tokenizer configuration from the original Qwen3 model must be added to the checkpoint directory.
	140	+
	141	+Run:
	142	+
	143	+```bash
	144	+src=assets/hf/Qwen3-1.7B
	145	+dst=outputs/checkpoint_wordle_sft/step-20
	146	+
	147	+test -d "$dst"
	148	+
	149	+for name in \
	150	+config.json \
	151	+tokenizer.json \
	152	+tokenizer_config.json \
	153	+special_tokens_map.json
	154	+do
	155	+ if test -f "$src/$name"; then
	156	+ cp "$src/$name" "$dst/$name"
	157	+ fi
	158	+done
	159	+
	160	+for name in \
	161	+generation_config.json \
	162	+vocab.json \
	163	+merges.txt
	164	+do
	165	+ if test -f "$src/$name"; then
	166	+ cp "$src/$name" "$dst/$name"
	167	+ fi
	168	+done
	169	+
	170	+echo "Loadable checkpoint:"
	171	+find "$dst" -maxdepth 1 -type f -printf '%f\n' \| sort
	172	+```
	173	+
	174	+### 3. Start the Inference Service
	175	+
	176	+```bash
	177	+source /home/developer/Ascend/cann-9.1.0/set_env.sh
	178	+# Stop an existing inference service (optional)
	179	+pkill -f infer_server.py \|\| true
	180	+
	181	+# Install inference dependencies
	182	+pip install transformers
	183	+pip install torchvision==0.27.0+cpu \
	184	+--index-url https://download.pytorch.org/whl/cpu
	185	+# Start the inference server
	186	+
	187	+python3 scripts/infer_server.py \
	188	+--model ./outputs/checkpoint_wordle_sft/step-20 \
	189	+--port 8000 \
	190	+> /tmp/infer_server.log 2>&1 &
	191	+
	192	+sleep 10
	193	+
	194	+curl http://localhost:8000/health
	195	+# {"status":"ok"}
	196	+```
	197	+
	198	+### 4. Configure the vf-eval Evaluation Environment
	199	+
	200	+`vf-eval` is an interactive evaluation tool based on Prime-RL. It is used to test the model's reasoning ability in the Wordle environment.
	201	+
	202	+The evaluation workflow is as follows:
	203	+
	204	+1. The environment selects a secret word from the reserved evaluation words.
	205	+2. The inference server receives the system prompt and the Wordle game rules.
	206	+3. The model generates a prediction:
	207	+
	208	+```xml
	209	+<think>
	210	+...
	211	+</think>
	212	+<guess>[word]</guess>
	213	+```
	214	+
	215	+4. The Wordle environment parses the content inside `<guess>`.
	216	+5. It returns G/Y/X letter feedback based on the prediction.
	217	+6. The model continues making predictions based on the feedback.
	218	+7. The interaction runs for at most six rounds.
	219	+8. The final reward is calculated when the game ends.
	220	+
	221	+The reward consists of the following components:
	222	+
	223	+\| Reward \| Meaning \|
	224	+\|---\|---\|
	225	+\| `correct_answer` \| Whether the secret word was guessed correctly \|
	226	+\| `partial_answer` \| Provides a partial reward based on the number of green/yellow letters \|
	227	+\| `length_bonus` \| Encourages completing the task in fewer rounds \|
	228	+\| `format_reward` \| Checks whether the `<guess>` output format is correct \|
	229	+
	230	+Initialize the vf-eval environment:
	231	+
	232	+```bash
	233	+bash ../setup_prime_rl.sh
	234	+```
	235	+
	236	+### 5. Run the Wordle Evaluation
	237	+
	238	+```bash
	239	+# Activate the vf-eval environment:
	240	+PRIMERL_DIR=./prime-rl
	241	+source "$PRIMERL_DIR/.venv-wordle-legacy/bin/activate"
	242	+
	243	+# Start the evaluation:
	244	+vf-eval wordle \
	245	+--num-examples 4 \
	246	+--rollouts-per-example 2 \
	247	+--api-base-url http://127.0.0.1:8000/v1 \
	248	+--max-concurrent 1 \
	249	+--verbose \
	250	+--temperature 0.6 \
	251	+--save-results
	252	+```
	253	+
	254	+Parameter descriptions:
	255	+
	256	+\| Parameter \| Description \|
	257	+\|---\|---\|
	258	+\| `--num-examples` \| Number of evaluation samples \|
	259	+\| `--rollouts-per-example` \| Number of samples generated for each example \|
	260	+\| `--api-base-url` \| Inference server address \|
	261	+\| `--max-concurrent` \| Sends requests serially to avoid overloading the single-threaded server \|
	262	+\| `--temperature` \| Generation sampling temperature \|
	263	+\| `--save-results` \| Saves the evaluation results \|
	264	+
	265	+### Results Analysis
	266	+
	267	+Reference performance comparison between the baseline model and the SFT model:
	268	+
	269	+\| Metric \| Base Qwen3-1.7B \| Wordle SFT \| Change \|
	270	+\|------\|----------------\|------------\|------\|
	271	+\| Average format_reward \| 0.60 \| 1.0 \| +0.4 \|
	272	+\| Average correct_answer \| 0.00 \| 0.00 \| No change \|
	273	+\| Average partial_answer \| 0.00 \| 0.25 \| +0.25 \|
	274	+\| Average length_bonus \| 0.00 \| 0.00 \| No change \|
	275	+\| Average reward \| 0.22 \| 0.4 \| +0.18 \|
	276	+
	277	+Format correctness improved significantly: `format_reward` increased from 0.6 to 1.0, indicating that the SFT stage successfully taught the model to follow the XML interaction format required by the Wordle environment. This is consistent with the goal of using SFT to learn the output format.
	278	+
	279	+Partial correctness improved: `partial_answer` increased from 0 to 0.25, indicating that the model began generating guesses containing correct letters (G/Y feedback) and gradually learned the game logic. However, the correct-guess rate remained at 0 (`correct_answer` was unchanged).
	280	+
	281	+Overall reward increased: The average total reward rose from 0.22 to 0.4. The gain came mainly from improvements in formatting and partial correctness, but the final win rate did not improve (`length_bonus` remained at 0), indicating that the model was still unable to solve the words completely.
	282	+
	283	+Comparison with the official baseline: The Qwen3-1.7B baseline score of 0.22 is close to the approximately 0.2 reported officially, and the post-SFT score of 0.4 is also broadly in line with expectations. After the RL stage, the average reward is expected to reach approximately 1.5 with a win rate of around 60%. This indicates that the current SFT model serves as the foundation for RL and that RL is needed to improve its strategic capabilities.

Allm_sft/qwen3_1.7B_torchtitan/build_project_platform.sh+31-0

@@ -0,0 +1,31 @@
	1	+set -euo pipefail
	2	+
	3	+# 可以用这个变量自定义下载路径。
	4	+repo=./torchtitan-npu
	5	+
	6	+# 固定 torchtitan-npu 版本，保证环境可复现
	7	+TORCHTITAN_NPU_REV=60f5a07
	8	+
	9	+if [ -d "$repo/.git" ]; then
	10	+ echo "Using existing repository: $repo"
	11	+else
	12	+ git clone https://gitcode.com/cann/torchtitan-npu.git "$repo"
	X X xuyujun7月24日上游主干并不稳定，建议当前锚定一个commit点？否则后续主线升级，当前脚本可能失效
	13	+fi
	14	+
	15	+cd "$repo"
	16	+
	17	+git checkout "$TORCHTITAN_NPU_REV"
	18	+
	19	+echo "Repo: $repo"
	20	+echo "Commit: $(git rev-parse HEAD)"
	21	+
	22	+source /home/developer/Ascend/cann-9.1.0/set_env.sh
	23	+
	24	+pip install -r requirements.txt
	25	+pip install -e .
	26	+
	27	+pip list \| grep -E 'torch\|torch_npu\|torchtitan\|triton-ascend\|safetensors'
	28	+
	29	+pip install --user --force-reinstall --no-deps pyarrow==21.0.0
	30	+
	31	+pip install -U uv openai mcp httpx tenacity textarena==0.7.4

Allm_sft/qwen3_1.7B_torchtitan/setup_prime_rl.sh+38-0

@@ -0,0 +1,38 @@
	1	+# 安装 Wordle 游戏环境 + vf-eval（NLTK 语料库已由 02.03 准备）
	2	+set -euo pipefail
	3	+
	4	+PRIMERL_DIR=./prime-rl
	5	+if [ -d "$PRIMERL_DIR/.git" ]; then
	6	+ echo "Using existing prime-rl at: $PRIMERL_DIR"
	7	+else
	8	+ git clone https://gitcode.com/gh_mirrors/pr/prime-rl.git "$PRIMERL_DIR"
	9	+fi
	10	+cd "$PRIMERL_DIR"
	11	+git checkout 188192ce64b2b7acf82e83ae36cfb0632bebde5b
	12	+
	13	+VERIFIERS_REV=d822f6aca7a967fc6698b1d595524c6278d84a5c
	14	+
	15	+if [ ! -e deps/verifiers/.git ] \|\|
	16	+ [ "$(git -C deps/verifiers rev-parse HEAD 2>/dev/null \|\| true)" != "$VERIFIERS_REV" ]; then
	17	+ rm -rf deps/verifiers
	18	+ git init deps/verifiers
	19	+ git -C deps/verifiers remote add origin \
	20	+ https://gitcode.com/GitHub_Trending/ver/verifiers.git
	21	+ git -C deps/verifiers fetch --depth=1 origin "$VERIFIERS_REV"
	22	+ git -C deps/verifiers checkout --detach FETCH_HEAD
	23	+fi
	24	+
	25	+export UV_CACHE_DIR=/tmp/uv-cache
	26	+mkdir -p "$UV_CACHE_DIR"
	27	+
	28	+# Prime-RL Wordle verifier 需要自己的 venv 环境。
	29	+uv venv .venv-wordle-legacy --clear
	30	+source .venv-wordle-legacy/bin/activate
	31	+
	32	+uv pip install -e deps/verifiers --index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
	33	+uv pip install -e deps/verifiers/environments/wordle --index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
	34	+
	35	+echo '=== 验证 ==='
	36	+echo "vf-eval: $(which vf-eval)"
	37	+echo "prime-rl revision: $(git rev-parse HEAD)"
	38	+echo "verifiers revision: $(git -C deps/verifiers rev-parse HEAD)"