已合并
docs: 修正dflow设计文档中模块路径和文件索引与代码不一致 #4307
lining23666创建于 26 天前
docs: 修正dflow设计文档中模块路径和文件索引与代码不一致 #4307
已合并
lining23666创建于 26 天前
2 个文件变更+30-30
@@ -8,7 +8,7 @@ This document is for dflow developers, describing the architecture design, core
8 8 
9### Scope9### Scope
10 10 
11-Covered modules: `flow_graph/`, `pydflow/`, `compiler/`, `base/`, `deployer/`, `executor/`, `udf/`. Does not cover the `llm_datadist` submodule (large model data distribution is an independent feature).11+Covered modules: `flow_graph/`, `pydflow/`, `runner/` (including `session/`, `compiler/`, `executor/`), `base/`, `deployer/`, `udf/`. Does not cover the `llm_datadist` submodule (large model data distribution is an independent feature).
12 12 
13Related documents:13Related documents:
14- [udf.md](udf.md) -- UDF submodule independent document14- [udf.md](udf.md) -- UDF submodule independent document
@@ -53,16 +53,16 @@ flowchart TD
53 FG["flow_graph/<br/>Graph construction core: FlowGraph/FlowNode/ProcessPoint"]53 FG["flow_graph/<br/>Graph construction core: FlowGraph/FlowNode/ProcessPoint"]
54 end54 end
55 subgraph session55 subgraph session
56- SES["session/<br/>DFlowSession API entry<br/>compilation/deployment/execution coordination"]56+ SES["runner/session/<br/>DFlowSession API entry<br/>compilation/deployment/execution coordination"]
57 end57 end
58 subgraph Compilation58 subgraph Compilation
59- CMP["compiler/<br/>FlowModelBuilder/PNE engine<br/>FlowGraph→FlowModel"]59+ CMP["runner/compiler/<br/>FlowModelBuilder/PNE engine<br/>FlowGraph→FlowModel"]
60 end60 end
61 subgraph Deployment61 subgraph Deployment
62 DEP["deployer/<br/>Multi-node deployment/cross-node communication<br/>fork executor processes"]62 DEP["deployer/<br/>Multi-node deployment/cross-node communication<br/>fork executor processes"]
63 end63 end
64 subgraph Execution64 subgraph Execution
65- EXEC["executor/<br/>Heterogeneous executors/data alignment<br/>Feed/Fetch"]65+ EXEC["runner/executor/<br/>Heterogeneous executors/data alignment<br/>Feed/Fetch"]
66 end66 end
67 subgraph UDF Submodule67 subgraph UDF Submodule
68 UDF["udf/<br/>User Defined Function framework<br/>(refer to udf.md)"]68 UDF["udf/<br/>User Defined Function framework<br/>(refer to udf.md)"]
@@ -80,11 +80,11 @@ flowchart TD
80|--------|---------------------|80|--------|---------------------|
81| `flow_graph/` | C++ graph construction API: FlowGraph/FlowNode/FlowData/ProcessPoint system |81| `flow_graph/` | C++ graph construction API: FlowGraph/FlowNode/FlowData/ProcessPoint system |
82| `pydflow/` | Python wrapper, @pyflow decorator, PyTorch integration, UDF project auto-generation |82| `pydflow/` | Python wrapper, @pyflow decorator, PyTorch integration, UDF project auto-generation |
83-| `session/` | DFlowSession API entry, coordination hub for compilation, deployment, and execution |83+| `runner/session/` | DFlowSession API entry, coordination hub for compilation, deployment, and execution |
84-| `compiler/` | FlowModelBuilder/PNE engine mechanism, graph optimization passes, compiles FlowGraph to FlowModel |84+| `runner/compiler/` | FlowModelBuilder/PNE engine mechanism, graph optimization passes, compiles FlowGraph to FlowModel |
85| `base/` | Model abstraction (FlowModel/GraphModel/PneModel), ModelRelation, deployment planning, OM serialization |85| `base/` | Model abstraction (FlowModel/GraphModel/PneModel), ModelRelation, deployment planning, OM serialization |
86| `deployer/` | Multi-node master-slave deployment, cross-node gRPC/memory queue communication, subprocess management |86| `deployer/` | Multi-node master-slave deployment, cross-node gRPC/memory queue communication, subprocess management |
87-| `executor/` | Heterogeneous executors, Feed/Fetch, data alignment, exception handling |87+| `runner/executor/` | Heterogeneous executors, Feed/Fetch, data alignment, exception handling |
88| `udf/` | UDF framework: SO loading/registration, state machine scheduling, message abstraction, built-in UDFs (refer to independent document) |88| `udf/` | UDF framework: SO loading/registration, state machine scheduling, message abstraction, built-in UDFs (refer to independent document) |
89 89 
90---90---
@@ -184,7 +184,7 @@ The three externally provided ProcessPoint types correspond to different computa
184 184 
185Additionally, `ModelPp` exists in the code (loads pre-compiled OM models, loads directly without compilation); it is an internal experimental feature and does not provide external interfaces.185Additionally, `ModelPp` exists in the code (loads pre-compiled OM models, loads directly without compilation); it is an internal experimental feature and does not provide external interfaces.
186 186 
187-During `FlowGraph` construction, `MultiThreadGraphBuilder` (8 threads parallel building GraphPp subgraphs) builds the FlowOperator list into a GE `Graph`, and sets `ATTR_NAME_IS_DATA_FLOW_GRAPH = true` to mark this graph as a dflow graph.187+During `FlowGraph` construction, the FlowOperator list is built into a GE `Graph`, and `ATTR_NAME_IS_DATA_FLOW_GRAPH = true` is set to mark this graph as a dflow graph.
188 188 
189### 3.2 C++ Runtime Interface189### 3.2 C++ Runtime Interface
190 190 
@@ -277,7 +277,7 @@ The compilation layer is located in `dflow/runner/compiler/`, adopting a four-la
277 277 
278```mermaid278```mermaid
279flowchart TD279flowchart TD
280- A["session/<br/>DFlowSession API entry<br/>+ lifecycle management"] --> B["model/<br/>FlowModelBuilder construction core<br/>+ FlowModelCache caching"]280+ A["runner/session/<br/>DFlowSession API entry<br/>+ lifecycle management"] --> B["model/<br/>FlowModelBuilder construction core<br/>+ FlowModelCache caching"]
281 B --> C["pne/<br/>ProcessNodeEngine engine abstraction<br/>+ UDF/CPU/NPU three engines"]281 B --> C["pne/<br/>ProcessNodeEngine engine abstraction<br/>+ UDF/CPU/NPU three engines"]
282 B --> D["data_flow_graph/<br/>Graph parsing + PP loading + compilation passes<br/>+ model relation construction + deployment planning"]282 B --> D["data_flow_graph/<br/>Graph parsing + PP loading + compilation passes<br/>+ model relation construction + deployment planning"]
283 C --> D283 C --> D
@@ -320,11 +320,11 @@ Engines are registered through `REGISTER_PROCESS_NODE_ENGINE` macro + SO plugin
320 320 
321**Multi-level caching** avoids repeated compilation: root model cache (graph_key index) + sub-model cache (SHA256 hash matching) + UDF cache (release_info matching, avoiding repeated cmake/make) + buildinfo cache.321**Multi-level caching** avoids repeated compilation: root model cache (graph_key index) + sub-model cache (SHA256 hash matching) + UDF cache (release_info matching, avoiding repeated cmake/make) + buildinfo cache.
322 322 
323-Subgraphs are compiled in parallel through `MultiThreadGraphBuilder` with multiple threads; FunctionPp uses async cmake/make compilation. The three-level caching coordination ensures incremental compilation efficiency.323+FunctionPp uses async cmake/make compilation. The three-level caching coordination ensures incremental compilation efficiency.
324 324 
325### 4.2 Model Abstraction Layer: FlowModel and ModelRelation325### 4.2 Model Abstraction Layer: FlowModel and ModelRelation
326 326 
327-`base/model/` defines the dflow model abstraction system, adopting a composition pattern inheritance hierarchy (`base/model/pne_model.h`):327+`base/model/` defines the dflow model abstraction system, adopting a composition pattern inheritance hierarchy (`inc/data_flow/model/pne_model.h`):
328 328 
329```mermaid329```mermaid
330classDiagram330classDiagram
@@ -546,7 +546,7 @@ For UDF Python development, refer to [udf.md](udf.md).
546 546 
547#### 4.6.1 How UDF Execution Location (host/device) Is Determined547#### 4.6.1 How UDF Execution Location (host/device) Is Determined
548 548 
549-The final UDF execution location is determined by the compilation-time attribute chain; the core logic is in `DataFlowGraphAutoDeployer::SelectResourceType` (`compiler/data_flow_graph/data_flow_graph_auto_deployer.cc`):549+The final UDF execution location is determined by the compilation-time attribute chain; the core logic is in `DataFlowGraphAutoDeployer::SelectResourceType` (`runner/compiler/data_flow_graph/data_flow_graph_auto_deployer.cc`):
550 550 
551| Attribute | Meaning | Setting Location |551| Attribute | Meaning | Setting Location |
552|-----------|---------|-----------------|552|-----------|---------|-----------------|
@@ -567,9 +567,9 @@ Although heavy_load UDF executes on the host CPU, it still needs to specify logi
567 567 
568#### 4.6.2 How Users Specify Deployment Location568#### 4.6.2 How Users Specify Deployment Location
569 569 
570-Users pass the deployment configuration JSON file through the compilation option `ge.experiment.data_flow_deploy_info_path` (`BuildModel` in `compiler/model/flow_model_builder.cc`). The configuration matches FlowNodes by node **deployment name** -- the deployment name preferentially uses the alias; if the node has no alias set, the original node name is used (`GetNodeDeployName` in `data_flow_graph_auto_deployer.cc`).570+Users pass the deployment configuration JSON file through the compilation option `ge.experiment.data_flow_deploy_info_path` (`BuildModel` in `runner/compiler/model/flow_model_builder.cc`). The configuration matches FlowNodes by node **deployment name** -- the deployment name preferentially uses the alias; if the node has no alias set, the original node name is used (`GetNodeDeployName` in `data_flow_graph_auto_deployer.cc`).
571 571 
572-**Deployment configuration JSON structure** (`ReadDeployInfoFromJsonFile` in `compiler/data_flow_graph/compile_config_json.cc`):572+**Deployment configuration JSON structure** (`ReadDeployInfoFromJsonFile` in `runner/compiler/data_flow_graph/compile_config_json.cc`):
573 573 
574```json574```json
575{575{
@@ -625,7 +625,7 @@ Using a complete Python user workflow as an example, the full chain from code to
625flowchart TD625flowchart TD
626 subgraph Graph Construction Phase626 subgraph Graph Construction Phase
627 A1["df.init(options)"] --> A2["df.FlowData() / @df.pyflow / GraphProcessPoint"]627 A1["df.init(options)"] --> A2["df.FlowData() / @df.pyflow / GraphProcessPoint"]
628- A2 --> A3["df.FlowGraph(outputs=[...])<br/>Reverse traversal extracts nodes+inputs<br/>MultiThreadGraphBuilder→ComputeGraph"]628+ A2 --> A3["df.FlowGraph(outputs=[...])<br/>Reverse traversal extracts nodes+inputs<br/>Build to ComputeGraph"]
629 end629 end
630 subgraph Compilation Phase630 subgraph Compilation Phase
631 B1["graph.feed_data() triggers lazy compilation"] --> B2["DFlowSession.CompileAndLoadGraph"]631 B1["graph.feed_data() triggers lazy compilation"] --> B2["DFlowSession.CompileAndLoadGraph"]
@@ -8,7 +8,7 @@
8 8 
9### 范围9### 范围
10 10 
11-涵盖模块:`flow_graph/`、`pydflow/`、`compiler/`、`base/`、`deployer/`、`executor/`、`udf/`。不涵盖 `llm_datadist` 子模块(大模型数据分发为独立特性)。11+涵盖模块:`flow_graph/`、`pydflow/`、`runner/`(含 `session/`、`compiler/`、`executor/`、`base/`、`deployer/`、`udf/`。不涵盖 `llm_datadist` 子模块(大模型数据分发为独立特性)。
12 12 
13相关文档:13相关文档:
14- [udf.md](udf.md) — UDF 子模块独立文档14- [udf.md](udf.md) — UDF 子模块独立文档
@@ -53,16 +53,16 @@ flowchart TD
53 FG["flow_graph/<br/>构图核心:FlowGraph/FlowNode/ProcessPoint"]53 FG["flow_graph/<br/>构图核心:FlowGraph/FlowNode/ProcessPoint"]
54 end54 end
55 subgraph session55 subgraph session
56- SES["session/<br/>DFlowSession API入口<br/>编译/部署/执行协调"]56+ SES["runner/session/<br/>DFlowSession API入口<br/>编译/部署/执行协调"]
57 end57 end
58 subgraph 编译58 subgraph 编译
59- CMP["compiler/<br/>FlowModelBuilder/PNE引擎<br/>FlowGraph→FlowModel"]59+ CMP["runner/compiler/<br/>FlowModelBuilder/PNE引擎<br/>FlowGraph→FlowModel"]
60 end60 end
61 subgraph 部署61 subgraph 部署
62 DEP["deployer/<br/>多节点部署/跨节点通信<br/>fork executor进程"]62 DEP["deployer/<br/>多节点部署/跨节点通信<br/>fork executor进程"]
63 end63 end
64 subgraph 执行64 subgraph 执行
65- EXEC["executor/<br/>异构执行器/数据对齐<br/>Feed/Fetch"]65+ EXEC["runner/executor/<br/>异构执行器/数据对齐<br/>Feed/Fetch"]
66 end66 end
67 subgraph UDF子模块67 subgraph UDF子模块
68 UDF["udf/<br/>用户自定义函数框架<br/>(详见 udf.md)"]68 UDF["udf/<br/>用户自定义函数框架<br/>(详见 udf.md)"]
@@ -80,11 +80,11 @@ flowchart TD
80|------|----------|80|------|----------|
81| `flow_graph/` | C++ 构图 API:FlowGraph/FlowNode/FlowData/ProcessPoint 体系 |81| `flow_graph/` | C++ 构图 API:FlowGraph/FlowNode/FlowData/ProcessPoint 体系 |
82| `pydflow/` | Python 封装、@pyflow 装饰器、PyTorch 集成、UDF 工程自动生成 |82| `pydflow/` | Python 封装、@pyflow 装饰器、PyTorch 集成、UDF 工程自动生成 |
83-| `session/` | DFlowSession API 入口,编译、部署、执行的协调中枢 |83+| `runner/session/` | DFlowSession API 入口,编译、部署、执行的协调中枢 |
84-| `compiler/` | FlowModelBuilder/PNE 引擎机制、图优化 pass,将 FlowGraph 编译为 FlowModel |84+| `runner/compiler/` | FlowModelBuilder/PNE 引擎机制、图优化 pass,将 FlowGraph 编译为 FlowModel |
85| `base/` | 模型抽象(FlowModel/GraphModel/PneModel)、ModelRelation、部署规划、OM 序列化 |85| `base/` | 模型抽象(FlowModel/GraphModel/PneModel)、ModelRelation、部署规划、OM 序列化 |
86| `deployer/` | 多节点主从部署、跨节点 gRPC/内存队列通信、子进程管理 |86| `deployer/` | 多节点主从部署、跨节点 gRPC/内存队列通信、子进程管理 |
87-| `executor/` | 异构执行器、Feed/Fetch、数据对齐、异常处理 |87+| `runner/executor/` | 异构执行器、Feed/Fetch、数据对齐、异常处理 |
88| `udf/` | UDF 框架:SO 加载注册、状态机调度、消息抽象、内置 UDF(详见独立文档) |88| `udf/` | UDF 框架:SO 加载注册、状态机调度、消息抽象、内置 UDF(详见独立文档) |
89 89 
90---90---
@@ -184,7 +184,7 @@ classDiagram
184 184 
185此外代码中还存在 `ModelPp`(加载预编译 OM 模型,直接加载不编译),它是内部实验性特性,未对外提供接口。185此外代码中还存在 `ModelPp`(加载预编译 OM 模型,直接加载不编译),它是内部实验性特性,未对外提供接口。
186 186 
187-`FlowGraph` 构建时通过 `MultiThreadGraphBuilder`(8 线程并行构建 GraphPp 子图)将 FlowOperator 列表构建为 GE `Graph`,并设置 `ATTR_NAME_IS_DATA_FLOW_GRAPH = true` 标记此图为 dflow 图。187+`FlowGraph` 构建时将 FlowOperator 列表构建为 GE `Graph`,并设置 `ATTR_NAME_IS_DATA_FLOW_GRAPH = true` 标记此图为 dflow 图。
188 188 
189### 3.2 C++ 运行接口189### 3.2 C++ 运行接口
190 190 
@@ -277,7 +277,7 @@ GraphPp 是否配置动态 shape 直接决定其走静态还是动态执行路
277 277 
278```mermaid278```mermaid
279flowchart TD279flowchart TD
280- A["session/<br/>DFlowSession API 入口<br/>+ 生命周期管理"] --> B["model/<br/>FlowModelBuilder 构建核心<br/>+ FlowModelCache 缓存"]280+ A["runner/session/<br/>DFlowSession API 入口<br/>+ 生命周期管理"] --> B["model/<br/>FlowModelBuilder 构建核心<br/>+ FlowModelCache 缓存"]
281 B --> C["pne/<br/>ProcessNodeEngine 引擎抽象<br/>+ UDF/CPU/NPU 三引擎"]281 B --> C["pne/<br/>ProcessNodeEngine 引擎抽象<br/>+ UDF/CPU/NPU 三引擎"]
282 B --> D["data_flow_graph/<br/>图解析 + PP加载 + 编译Pass<br/>+ 模型关系构建 + 部署规划"]282 B --> D["data_flow_graph/<br/>图解析 + PP加载 + 编译Pass<br/>+ 模型关系构建 + 部署规划"]
283 C --> D283 C --> D
@@ -320,11 +320,11 @@ CPU 引擎继承 NPU 引擎仅重写 `GetEngineName`,编译流程完全复用
320 320 
321**多级缓存**避免重复编译:root 模型缓存(graph_key 索引)+ 子模型缓存(SHA256 哈希匹配)+ UDF 缓存(release_info 匹配,避免重复 cmake/make)+ buildinfo 缓存。321**多级缓存**避免重复编译:root 模型缓存(graph_key 索引)+ 子模型缓存(SHA256 哈希匹配)+ UDF 缓存(release_info 匹配,避免重复 cmake/make)+ buildinfo 缓存。
322 322 
323-子图通过 `MultiThreadGraphBuilder` 多线程并行编译,FunctionPp 异步 cmake/make 编译,三级缓存协同保证增量编译效率。323+FunctionPp 异步 cmake/make 编译,三级缓存协同保证增量编译效率。
324 324 
325### 4.2 模型抽象层:FlowModel 与 ModelRelation325### 4.2 模型抽象层:FlowModel 与 ModelRelation
326 326 
327-`base/model/` 定义了 dflow 的模型抽象体系,采用组合模式的继承体系(`base/model/pne_model.h`):327+`base/model/` 定义了 dflow 的模型抽象体系,采用组合模式的继承体系(`inc/data_flow/model/pne_model.h`):
328 328 
329```mermaid329```mermaid
330classDiagram330classDiagram
@@ -546,7 +546,7 @@ UDF 的 Python 开发方式详见 [udf.md](udf.md)。
546 546 
547#### 4.6.1 UDF 执行位置(host/device)如何决定547#### 4.6.1 UDF 执行位置(host/device)如何决定
548 548 
549-UDF 的最终执行位置由编译期属性链路决定,核心逻辑在 `DataFlowGraphAutoDeployer::SelectResourceType`(`compiler/data_flow_graph/data_flow_graph_auto_deployer.cc`):549+UDF 的最终执行位置由编译期属性链路决定,核心逻辑在 `DataFlowGraphAutoDeployer::SelectResourceType`(`runner/compiler/data_flow_graph/data_flow_graph_auto_deployer.cc`):
550 550 
551| 属性 | 含义 | 设置位置 |551| 属性 | 含义 | 设置位置 |
552|------|------|----------|552|------|------|----------|
@@ -567,9 +567,9 @@ heavy_load UDF 虽然在 host CPU 执行,但仍需指定 logic_device_id——
567 567 
568#### 4.6.2 用户如何指定部署位置568#### 4.6.2 用户如何指定部署位置
569 569 
570-用户通过编译选项 `ge.experiment.data_flow_deploy_info_path` 传入部署配置 JSON 文件(`compiler/model/flow_model_builder.cc` 的 `BuildModel`)。配置以节点的**部署名**为 key 匹配 FlowNode——部署名优先取 alias,若节点未设置 alias 则用节点原名(`data_flow_graph_auto_deployer.cc` 的 `GetNodeDeployName`)。570+用户通过编译选项 `ge.experiment.data_flow_deploy_info_path` 传入部署配置 JSON 文件(`runner/compiler/model/flow_model_builder.cc` 的 `BuildModel`)。配置以节点的**部署名**为 key 匹配 FlowNode——部署名优先取 alias,若节点未设置 alias 则用节点原名(`data_flow_graph_auto_deployer.cc` 的 `GetNodeDeployName`)。
571 571 
572-**部署配置 JSON 结构**(`compiler/data_flow_graph/compile_config_json.cc` 的 `ReadDeployInfoFromJsonFile`):572+**部署配置 JSON 结构**(`runner/compiler/data_flow_graph/compile_config_json.cc` 的 `ReadDeployInfoFromJsonFile`):
573 573 
574```json574```json
575{575{
@@ -625,7 +625,7 @@ heavy_load UDF 虽然在 host CPU 执行,但仍需指定 logic_device_id——
625flowchart TD625flowchart TD
626 subgraph 构图阶段626 subgraph 构图阶段
627 A1["df.init(options)"] --> A2["df.FlowData() / @df.pyflow / GraphProcessPoint"]627 A1["df.init(options)"] --> A2["df.FlowData() / @df.pyflow / GraphProcessPoint"]
628- A2 --> A3["df.FlowGraph(outputs=[...])<br/>反向遍历提取节点+输入<br/>MultiThreadGraphBuilder→ComputeGraph"]628+ A2 --> A3["df.FlowGraph(outputs=[...])<br/>反向遍历提取节点+输入<br/>构建为ComputeGraph"]
629 end629 end
630 subgraph 编译阶段630 subgraph 编译阶段
631 B1["graph.feed_data() 触发惰性编译"] --> B2["DFlowSession.CompileAndLoadGraph"]631 B1["graph.feed_data() 触发惰性编译"] --> B2["DFlowSession.CompileAndLoadGraph"]