[feature] add torchnpugen packages-pytorch-AtomGit

已合并

[feature] add torchnpugen packages #30023

梁松伟创建于 1月26日从liangsongwei/pytorch_6590： v2.8.0 合入到 Ascend/pytorch： v2.8.0

[feature] add torchnpugen packages #30023

已合并

梁松伟创建于 1月26日

从liangsongwei/pytorch_6590 v2.8.0合入到Ascend/pytorch v2.8.0

共 31 个文件变更+844-844

Mbuild_libtorch_npu.py+1-1

	@@ -16,7 +16,7 @@ from distutils import file_util
16	# Disable autoloading before running 'import torch' to avoid circular dependencies	16	# Disable autoloading before running 'import torch' to avoid circular dependencies
17	os.environ["TORCH_DEVICE_BACKEND_AUTOLOAD"] = "0"	17	os.environ["TORCH_DEVICE_BACKEND_AUTOLOAD"] = "0"
18		18
19	-from codegen.utils import PathManager	19	+from torchnpugen.utils import PathManager
20		20
21	BASE_DIR = os.path.dirname(os.path.abspath(__file__))	21	BASE_DIR = os.path.dirname(os.path.abspath(__file__))
22	PathManager.check_directory_path_readable(os.path.join(BASE_DIR, "version.txt"))	22	PathManager.check_directory_path_readable(os.path.join(BASE_DIR, "version.txt"))

Mgenerate_code.sh+4-4

	@@ -23,19 +23,19 @@ testing_source_yaml="$CDIR/test/ops_unsupport_list.yaml"
23		23
24	op_plugin_functions_yaml_path="$op_plugin_config_path/npu_native_functions.yaml"	24	op_plugin_functions_yaml_path="$op_plugin_config_path/npu_native_functions.yaml"
25		25
26	-${python_execute} -m codegen.gen_backend_stubs \	26	+${python_execute} -m torchnpugen.gen_backend_stubs \
		AtlasAccount1月26日代码维护性: 代码中使用了硬编码的模块路径 `torchnpugen.gen_backend_stubs`。虽然从上下文看，这可能是从 `codegen.gen_backend_stubs` 重命名而来，但硬编码的模块名降低了脚本的灵活性。如果未来模块结构或命名再次发生变化，需要修改此脚本。结合第32行和第38行的类似改动，这属于批量重命名操作，但未考虑使用变量或参数来配置模块路径，使得脚本与特定的代码组织结构耦合。问题类型: 代码维护性文件路径: `generate_code.sh` 行号: 26 问题代码: `${python_execute} -m torchnpugen.gen_backend_stubs \` 修改建议: 建议将模块路径定义为变量，例如 `GEN_BACKEND_STUBS_MODULE="torchnpugen.gen_backend_stubs"`，然后在命令中引用该变量。这样，当模块路径需要调整时，只需修改变量的值，提高了脚本的可维护性和可配置性。此评论由代码审查工具自动生成
27	--output_dir="torch_npu/csrc/aten" \	27	--output_dir="torch_npu/csrc/aten" \
28	--source_yaml="$source_yaml" \	28	--source_yaml="$source_yaml" \
29	--impl_path="$CDIR/torch_npu/csrc/aten" \	29	--impl_path="$CDIR/torch_npu/csrc/aten" \
30	--op_plugin_impl_path="$CDIR/third_party/op-plugin/op_plugin/ops" \	30	--op_plugin_impl_path="$CDIR/third_party/op-plugin/op_plugin/ops" \
31	--op_plugin_yaml_path="$op_plugin_config_path/op_plugin_functions.yaml"	31	--op_plugin_yaml_path="$op_plugin_config_path/op_plugin_functions.yaml"
32		32
33	-${python_execute} -m codegen.autograd.gen_autograd \	33	+${python_execute} -m torchnpugen.autograd.gen_autograd \
		AtlasAccount1月26日代码维护性: 代码中使用了硬编码的模块路径 `torchnpugen.autograd.gen_autograd` 和目录路径 `$CDIR/torchnpugen/autograd`。与第26行的问题类似，这属于从 `codegen.autograd.gen_autograd` 和 `$CDIR/codegen/autograd` 重命名。硬编码的路径使得脚本依赖于特定的项目目录结构，如果未来目录结构调整，需要同步修改此脚本。虽然当前改动是统一的（从 `codegen` 改为 `torchnpugen`），但未采用更灵活的设计。问题类型: 代码维护性文件路径: `generate_code.sh` 行号: 33 问题代码: `${python_execute} -m torchnpugen.autograd.gen_autograd \ --out_dir="$CDIR/torch_npu/csrc/aten" \ --autograd_dir="$CDIR/torchnpugen/autograd" \` 修改建议: 建议将模块路径和目录路径定义为变量，例如 `GEN_AUTOGRAD_MODULE="torchnpugen.autograd.gen_autograd"` 和 `AUTOGRAD_DIR="$CDIR/torchnpugen/autograd"`。在命令中引用这些变量。这样，路径配置集中管理，便于维护和适应变化。此评论由代码审查工具自动生成
34	--out_dir="$CDIR/torch_npu/csrc/aten" \	34	--out_dir="$CDIR/torch_npu/csrc/aten" \
35	- --autograd_dir="$CDIR/codegen/autograd" \	35	+ --autograd_dir="$CDIR/torchnpugen/autograd" \
36	--npu_native_function_dir="$source_yaml"	36	--npu_native_function_dir="$source_yaml"
37		37
38	-${python_execute} -m codegen.codegen_ops_info	38	+${python_execute} -m torchnpugen.codegen_ops_info
		AtlasAccount1月26日代码维护性: 代码中使用了硬编码的模块路径 `torchnpugen.codegen_ops_info`。这是从 `codegen.codegen_ops_info` 重命名而来。同样的问题，硬编码模块名降低了脚本的灵活性。结合第26行和第33行的改动，这三次修改都是将 `codegen` 重命名为 `torchnpugen`，但脚本本身没有提供任何机制来适应这种重命名，而是直接写死新名称。问题类型: 代码维护性文件路径: `generate_code.sh` 行号: 38 问题代码: `${python_execute} -m torchnpugen.codegen_ops_info` 修改建议: 建议将模块路径定义为变量，例如 `CODEGEN_OPS_INFO_MODULE="torchnpugen.codegen_ops_info"`，然后在命令中引用。或者，考虑在脚本开头定义一个基础包名变量（如 `BASE_PKG="torchnpugen"`），用于拼接出完整的模块路径，这样重命名时只需修改一个地方。此评论由代码审查工具自动生成
39		39
40	if [ -f $CDIR/third_party/op-plugin/codegen/templates/_op_plugin_docs.py ]; then	40	if [ -f $CDIR/third_party/op-plugin/codegen/templates/_op_plugin_docs.py ]; then
41	if [ -f $CDIR/torch_npu/_op_plugin_docs.py ]; then	41	if [ -f $CDIR/torch_npu/_op_plugin_docs.py ]; then

Msetup.py+1-1

	@@ -29,7 +29,7 @@ from wheel.bdist_wheel import bdist_wheel
29	# Disable autoloading before running 'import torch' to avoid circular dependencies	29	# Disable autoloading before running 'import torch' to avoid circular dependencies
30	os.environ["TORCH_DEVICE_BACKEND_AUTOLOAD"] = "0"	30	os.environ["TORCH_DEVICE_BACKEND_AUTOLOAD"] = "0"
31		31
32	-from codegen.utils import PathManager	32	+from torchnpugen.utils import PathManager
33		33
34	BASE_DIR = os.path.dirname(os.path.realpath(__file__))	34	BASE_DIR = os.path.dirname(os.path.realpath(__file__))
35	THIRD_PARTY_PATH = os.path.join(BASE_DIR, "third_party")	35	THIRD_PARTY_PATH = os.path.join(BASE_DIR, "third_party")

Rcodegen/__init__.py→torchnpugen/__init__.py+27-27

	@@ -1,27 +1,27 @@
1	-import os	1	+import os
2	-import stat	2	+import stat
3	-	3	+
4	-import torchgen.gen	4	+import torchgen.gen
5	-from codegen.utils import PathManager	5	+from torchnpugen.utils import PathManager
6	-	6	+
7	-	7	+
8	-def _write_if_changed_security(self, filename: str, contents: str) -> None:	8	+def _write_if_changed_security(self, filename: str, contents: str) -> None:
9	- old_contents: Optional[str]	9	+ old_contents: Optional[str]
		AtlasAccount1月26日类型注解与代码可读性: 变量 `old_contents` 被声明为 `Optional[str]` 类型，但未从 `typing` 模块导入 `Optional`。虽然代码可能在其他地方已导入，但在此片段中缺少显式导入，会影响代码的清晰度和类型检查工具（如 mypy）的正确运行。问题类型: 类型注解与代码可读性文件路径: `torchnpugen/__init__.py` 行号: 9 问题代码: `old_contents: Optional[str]` 修改建议: 在函数内部或模块顶部添加 `from typing import Optional`，以确保类型注解的完整性和可移植性。此评论由代码审查工具自动生成
10	- filepath = os.path.realpath(filename)	10	+ filepath = os.path.realpath(filename)
11	- try:	11	+ try:
12	- with open(filepath, 'r') as f:	12	+ with open(filepath, 'r') as f:
13	- old_contents = f.read()	13	+ old_contents = f.read()
14	- except IOError:	14	+ except IOError:
15	- old_contents = None	15	+ old_contents = None
16	- if contents != old_contents:	16	+ if contents != old_contents:
17	- PathManager.remove_path_safety(filepath)	17	+ PathManager.remove_path_safety(filepath)
18	- with os.fdopen(os.open(filepath, os.O_RDWR \| os.O_CREAT, stat.S_IWUSR \| stat.S_IRUSR), "w") as f:	18	+ with os.fdopen(os.open(filepath, os.O_RDWR \| os.O_CREAT, stat.S_IWUSR \| stat.S_IRUSR), "w") as f:
19	- f.write(contents)	19	+ f.write(contents)
20	- os.chmod(filepath, stat.S_IRUSR \| stat.S_IEXEC \| stat.S_IRGRP \| stat.S_IXGRP)	20	+ os.chmod(filepath, stat.S_IRUSR \| stat.S_IEXEC \| stat.S_IRGRP \| stat.S_IXGRP)
21	-	21	+
22	-	22	+
23	-def apply_codegen_patches():	23	+def apply_codegen_patches():
24	- torchgen.gen.FileManager._write_if_changed = _write_if_changed_security	24	+ torchgen.gen.FileManager._write_if_changed = _write_if_changed_security
25	-	25	+
26	-	26	+
27	-apply_codegen_patches()	27	+apply_codegen_patches()

Rcodegen/autograd/__init__.py→torchnpugen/autograd/__init__.py+2-2

	@@ -20,8 +20,8 @@ from torchgen.model import (
20	TensorOptionsArguments	20	TensorOptionsArguments
21	)	21	)
22	from torchgen.api.types import Binding	22	from torchgen.api.types import Binding
23	-from codegen.gen_backend_stubs import parse_native_and_custom_yaml	23	+from torchnpugen.gen_backend_stubs import parse_native_and_custom_yaml
24	-from codegen.utils import CUSTOM_YAML_NAME	24	+from torchnpugen.utils import CUSTOM_YAML_NAME
25		25
26		26
27	def parse_native_and_custom_yaml_(args, *kwargs):	27	def parse_native_and_custom_yaml_(args, *kwargs):

Rcodegen/autograd/gen_autograd.py→torchnpugen/autograd/gen_autograd.py+3-3

Rcodegen/autograd/gen_autograd_functions.py→torchnpugen/autograd/gen_autograd_functions.py+0-0

文件重命名但无更改。

Rcodegen/autograd/gen_variable_factories.py→torchnpugen/autograd/gen_variable_factories.py+0-0

文件重命名但无更改。

Rcodegen/autograd/gen_variable_type.py→torchnpugen/autograd/gen_variable_type.py+0-0

文件重命名但无更改。

Rcodegen/autograd/templates/ADInplaceOrViewType.cpp→torchnpugen/autograd/templates/ADInplaceOrViewType.cpp+0-0

文件重命名但无更改。

Rcodegen/autograd/templates/Functions.cpp→torchnpugen/autograd/templates/Functions.cpp+0-0

文件重命名但无更改。

Rcodegen/autograd/templates/Functions.h→torchnpugen/autograd/templates/Functions.h+0-0

文件重命名但无更改。

Rcodegen/autograd/templates/VariableType.cpp→torchnpugen/autograd/templates/VariableType.cpp+0-0

文件重命名但无更改。

Rcodegen/autograd/templates/VariableType.h→torchnpugen/autograd/templates/VariableType.h+0-0

文件重命名但无更改。

Rcodegen/autograd/templates/python_functions.cpp→torchnpugen/autograd/templates/python_functions.cpp+0-0

文件重命名但无更改。

Rcodegen/autograd/templates/python_functions.h→torchnpugen/autograd/templates/python_functions.h+0-0

文件重命名但无更改。

Rcodegen/autograd/utils.py→torchnpugen/autograd/utils.py+2-2

	@@ -9,8 +9,8 @@ from torchgen.api.autograd import (
9	)	9	)
10	from torchgen.packaged.autograd.load_derivatives import load_derivatives	10	from torchgen.packaged.autograd.load_derivatives import load_derivatives
11		11
12	-from codegen.utils import get_torchgen_dir, CUSTOM_YAML_NAME, PathManager	12	+from torchnpugen.utils import get_torchgen_dir, CUSTOM_YAML_NAME, PathManager
13	-from codegen.gen_backend_stubs import parse_native_and_custom_yaml	13	+from torchnpugen.gen_backend_stubs import parse_native_and_custom_yaml
14		14
15		15
16	AUTOGRAD_BLACK_LIST = {'npu_format_cast.Tensor', 'npu_format_cast_', 'npu_format_cast_.acl_format'}	16	AUTOGRAD_BLACK_LIST = {'npu_format_cast.Tensor', 'npu_format_cast_', 'npu_format_cast_.acl_format'}

Rcodegen/codegen_ops_info.py→torchnpugen/codegen_ops_info.py+3-3

	@@ -9,8 +9,8 @@ import yaml
9	from torchgen.code_template import CodeTemplate	9	from torchgen.code_template import CodeTemplate
10	from torchgen.gen import FileManager	10	from torchgen.gen import FileManager
11		11
12	-from codegen.autograd.utils import VERSION_PART	12	+from torchnpugen.autograd.utils import VERSION_PART
13	-from codegen.utils import PathManager	13	+from torchnpugen.utils import PathManager
14		14
15	project_path = Path(os.path.dirname(__file__)).parent	15	project_path = Path(os.path.dirname(__file__)).parent
16	op_plugin_info_path = os.path.realpath(os.path.join(	16	op_plugin_info_path = os.path.realpath(os.path.join(
	@@ -96,7 +96,7 @@ def gen_ops_info(summary_dict):
96	skip_template = CodeTemplate(	96	skip_template = CodeTemplate(
97	"""\n'${op_name}': [${decorators}]"""	97	"""\n'${op_name}': [${decorators}]"""
98	)	98	)
99	- fm = FileManager(os.path.join("torch_npu", "testing"), os.path.join("codegen", "templates"), False)	99	+ fm = FileManager(os.path.join("torch_npu", "testing"), os.path.join("torchnpugen", "templates"), False)
100		100
101	fm.write_with_template(f"_npu_testing_utils.py", "npu_testing_utils.py", lambda:{	101	fm.write_with_template(f"_npu_testing_utils.py", "npu_testing_utils.py", lambda:{
102	"skip_detail": [skip_template.substitute(op_name=op, decorators=doc) for op, doc in skip_list.items()]	102	"skip_detail": [skip_template.substitute(op_name=op, decorators=doc) for op, doc in skip_list.items()]

Rcodegen/custom_functions.py→torchnpugen/custom_functions.py+1-1

	@@ -13,7 +13,7 @@ from torchgen.context import with_native_function, native_function_manager, meth
13	from torchgen.api.types import DispatcherSignature	13	from torchgen.api.types import DispatcherSignature
14	from torchgen.api import cpp	14	from torchgen.api import cpp
15	from torchgen.dest.register_dispatch_key import RegisterDispatchKey	15	from torchgen.dest.register_dispatch_key import RegisterDispatchKey
16	-from codegen.utils import (enable_opplugin, is_op_valid, field_tag, get_opplugin_wrap_name, parse_npu_yaml,	16	+from torchnpugen.utils import (enable_opplugin, is_op_valid, field_tag, get_opplugin_wrap_name, parse_npu_yaml,
17	gen_op_hook_post_code)	17	gen_op_hook_post_code)
18		18
19		19

Rcodegen/gen_backend_stubs.py→torchnpugen/gen_backend_stubs.py+799-799

Rcodegen/gen_functionalization_type.py→torchnpugen/gen_functionalization_type.py+0-0

文件重命名但无更改。

Rcodegen/templates/CustomFunctions.cpp→torchnpugen/templates/CustomFunctions.cpp+0-0

文件重命名但无更改。

Rcodegen/templates/CustomFunctions.h→torchnpugen/templates/CustomFunctions.h+0-0

文件重命名但无更改。

Rcodegen/templates/CustomRedispatch.cpp→torchnpugen/templates/CustomRedispatch.cpp+0-0

文件重命名但无更改。

Rcodegen/templates/CustomRedispatch.h→torchnpugen/templates/CustomRedispatch.h+0-0

文件重命名但无更改。

Rcodegen/templates/CustomRegisterSchema.cpp→torchnpugen/templates/CustomRegisterSchema.cpp+0-0

文件重命名但无更改。

Rcodegen/templates/ForeachRegister.cpp→torchnpugen/templates/ForeachRegister.cpp+0-0

文件重命名但无更改。

Rcodegen/templates/RegisterFunctionalization.cpp→torchnpugen/templates/RegisterFunctionalization.cpp+0-0

文件重命名但无更改。

Rcodegen/templates/custom_ops.py→torchnpugen/templates/custom_ops.py+0-0

文件重命名但无更改。

Rcodegen/templates/npu_testing_utils.py→torchnpugen/templates/npu_testing_utils.py+0-0

文件重命名但无更改。

Rcodegen/utils.py→torchnpugen/utils.py+1-1

	@@ -198,7 +198,7 @@ def filt_compositeimplicitautograd_api(native_yaml_path, npu_supported):
198	with open(native_yaml_path, 'r') as f:	198	with open(native_yaml_path, 'r') as f:
199	es = yaml.safe_load(f)	199	es = yaml.safe_load(f)
200		200
201	- from codegen.autograd.utils import TORCH_AUTOGRAD_FUNCTION	201	+ from torchnpugen.autograd.utils import TORCH_AUTOGRAD_FUNCTION
202	supported_autograd = []	202	supported_autograd = []
203	for e in es:	203	for e in es:
204	api_name = e['func'].split('(')[0]	204	api_name = e['func'].split('(')[0]

	@@ -2,10 +2,10 @@
2	To run this file by hand from the root of the PyTorch	2	To run this file by hand from the root of the PyTorch
3	repository, run:	3	repository, run:
4		4
5	-python -m codegen.autograd.gen_autograd \	5	+python -m torchnpugen.autograd.gen_autograd \
6	--npu_native_function_dir="./torch_npu/csrc/aten/npu_native_functions.yaml" \	6	--npu_native_function_dir="./torch_npu/csrc/aten/npu_native_functions.yaml" \
7	--out_dir=$OUTPUT_DIR \	7	--out_dir=$OUTPUT_DIR \
8	- --autograd_dir="./codegen/autograd/"	8	+ --autograd_dir="./torchnpugen/autograd/"
9		9
10	Where $OUTPUT_DIR is where you would like the files to be	10	Where $OUTPUT_DIR is where you would like the files to be
11	generated. In the full build system, OUTPUT_DIR is	11	generated. In the full build system, OUTPUT_DIR is
	@@ -24,7 +24,7 @@ import os
24	from torchgen.packaged.autograd.gen_inplace_or_view_type import gen_inplace_or_view_type	24	from torchgen.packaged.autograd.gen_inplace_or_view_type import gen_inplace_or_view_type
25	from torchgen.packaged.autograd.gen_autograd_functions import gen_autograd_functions_lib	25	from torchgen.packaged.autograd.gen_autograd_functions import gen_autograd_functions_lib
26		26
27	-from codegen.utils import get_torchgen_dir, gen_custom_yaml_path	27	+from torchnpugen.utils import get_torchgen_dir, gen_custom_yaml_path
28	from .gen_variable_type import (	28	from .gen_variable_type import (
29	gen_variable_type, gen_variable_type_head	29	gen_variable_type, gen_variable_type_head
30	)	30	)

类型注解与代码可读性: 变量 old_contents 被声明为 Optional[str] 类型，但未从 typing 模块导入 Optional。虽然代码可能在其他地方已导入，但在此片段中缺少显式导入，会影响代码的清晰度和类型检查工具（如 mypy）的正确运行。