变更摘要

此 PR 新增了生成式模型（Wan2.2）的数据处理工具模块，包含两个文件：data_transform.py 提供视频/图像张量的几何变换（裁剪、缩放、对齐等）和美学评分标注功能；utils.py 提供完整的数据读取管线，涵盖多格式文件解析、JSON 多进程加载、图像预处理（image2video/image2image 双模式）、文本清洗与分词等核心能力，不依赖 Megatron。

主要改动

• 新增 data_transform.py：提供视频张量的基础变换函数（to_tensor、crop、resize、center_crop_using_short_edge、longsideresize、shortsideresize、maxhwresize、calculate_centered_alignment）以及三个变换类 ToTensorVideo、CenterCropResizeVideo、ResizeVideo 和 TemporalRandomCrop，支持多种缩放策略（resize/longside/shortside/hxw）与时间维随机裁剪。

• 新增 data_transform.py 中的美学评分通知机制：定义了视频/图像高低美学评分的文本模板列表，并提供 add_aesthetic_notice_video 和 add_aesthetic_notice_image 函数，根据评分阈值（≤4.25 / ≥5.75）随机向 caption 中注入美学质量描述。

• 新增 utils.py 中的 DataFileReader：支持从 csv、json、jsonl、parquet、txt 格式加载数据，并内置 standard、combine、sorafeatured、vace 四种数据存储模式，可自动重定向文件路径、过滤保留键。

• 新增 utils.py 中的 JsonLoader：提供 JSON/JSONL 文件加载与后处理能力，支持通过 set_process_func 注册自定义处理函数，并实现了基于 multiprocessing.shared_memory 的多进程并行加载机制以避免大文件加载瓶颈。

• 新增 utils.py 中的 ImageProcesser 和 TextProcesser：ImageProcesser 支持 image2video（单图扩展为 T 帧视频）和 image2image 两种处理模式；TextProcesser 内置完整的英文文本清洗管线（URL/IP/HTML 标签/特殊标点/文件名去除等），并支持可配置的文本预处理方法链与 CFG（Classifier-Free Guidance）随机置空策略。

代码审查

Good, I've now read all the critical sections. Let me compile my findings and report them.

One more thing to check: TextProcesser.__call__ at line 425: when cfg=0.1 (default), there's a 10% chance that texts_info is replaced with [""]. But looking at the else branch (line 427), texts_info = texts — when enable_text_preprocessing is False, texts_info is the raw input texts. So when preprocessing is disabled, the tokenizer gets the original texts (which could be a single string or a list). When preprocessing is enabled and cfg drops out, texts_info = [""] — always a list with one empty string. This is consistent in that the tokenizer can handle both, but it's a bit asymmetric.

Also, the text_preprocessing function ignores use_clean_caption at line 462's recursive call — the parameter is not passed:

text = TextProcesser.text_preprocessing(text, text_preprocess_methods=text_preprocess_method)

But use_clean_caption and support_chinese are not passed. So in the recursive call, they default to use_clean_caption=True and support_chinese=False. This means if the original call had use_clean_caption=False or support_chinese=True, the recursive invocation would use the defaults instead. But actually, if text_preprocess_methods is a list of dicts, each recursive call processes one method, and the else branch (line 463) never reaches the use_clean_caption check (line 475). So this is fine — the use_clean_caption and support_chinese params are only used in the final fallback (line 474-478).

OK, I think I have enough. Let me now report all findings.

Let me also double-check the ResizeVideo init for the transform_size default:

def __init__(
    self,
    transform_size="auto",
    ...
):

When transform_size="auto" and mode != 'hxw' and mode in ["resize", "longside", "shortside"]:

self.transform_size = (transform_size["max

| 类型 | 数量 |
|------|-------|
| 🔴 阻塞 | 4 |
| 🟡 建议 | 9 |

⛔ 需要修改

Thanks for your pull-request.
The full list of commands accepted by me can be found at here。
You can get sig-info at here

PR Approval Progress

✅ Congratulations! All modules have met the lgtm and approve requirements.

Module Approval Details

💡 Tip:

• Committer can comment /approve or /lgtm
• Commenting /approve implies both code review (lgtm) and intent to merge (approve)

CLA Signature Pass

chenpeizhe, thanks for your pull request. All authors of the commits have signed the CLA. 👍

/approve