Backgroud（背景信息）

• Describe/Explain the status of the problem you wish to solve.
  hyper-paraller在训练过程中，无法获取其他卡上的layout信息
• Attach relevant issues if there is any.

Origin（信息来源）

• Explain which department/team made this request so that its priority can be given.

Benefit / Necessity （价值/作用）

• Describe/Explain the key value by fulfilling the request.
  使得在hyper-paraller的训练过程中，任意卡可以在想要获取其他卡上信息的时候拿到所有卡上的layout信息，并根据现状进行其他操作

Design（设计方案）

• Describe/Explain the general idea of the design. Pseudo-code is allowed
  step1 在hyper-parallel训练的某个阶段，获取当前卡上的layout数据，获取world_size，创建长度为world_size的列表
  step2 调用hyper-parallel的all_gather_object()接口，将同一个通信组内所有卡上的layout数据合并到一个大的列表，并按照rank id进行排序
  step3 将列表转换成dict
  最终的dict数据结构如下：
  {'0': {'weight': {'mesh_shape': (4, 2), 'tensor_map': (0, -1), 'interleaved_parallel': False, 'alias_name': ('dp', 'mp'), 'rank_list': (0, 1, 2, 3, 4, 5, 6, 7), 'type': 'Float32', 'full_shape': (32, 4)}}, '1': {'weight': {'mesh_shape': (4, 2), 'tensor_map': (0, -1), 'interleaved_parallel': False, 'alias_name': ('dp', 'mp'), 'rank_list': (0, 1, 2, 3, 4, 5, 6, 7), 'type': 'Float32', 'full_shape': (32, 4)}}, '2': {'weight': {'mesh_shape': (4, 2), 'tensor_map': (0, -1), 'interleaved_parallel': False, 'alias_name': ('dp', 'mp'), 'rank_list': (0, 1, 2, 3, 4, 5, 6, 7), 'type': 'Float32', 'full_shape': (32, 4)}}, '3': {'weight': {'mesh_shape': (4, 2), 'tensor_map': (0, -1), 'interleaved_parallel': False, 'alias_name': ('dp', 'mp'), 'rank_list': (0, 1, 2, 3, 4, 5, 6, 7), 'type': 'Float32', 'full_shape': (32, 4)}}, '4': {'weight': {'mesh_shape': (4, 2), 'tensor_map': (0, -1), 'interleaved_parallel': False, 'alias_name': ('dp', 'mp'), 'rank_list': (0, 1, 2, 3, 4, 5, 6, 7), 'type': 'Float32', 'full_shape': (32, 4)}}, '5': {'weight': {'mesh_shape': (4, 2), 'tensor_map': (0, -1), 'interleaved_parallel': False, 'alias_name': ('dp', 'mp'), 'rank_list': (0, 1, 2, 3, 4, 5, 6, 7), 'type': 'Float32', 'full_shape': (32, 4)}}, '6': {'weight': {'mesh_shape': (4, 2), 'tensor_map': (0, -1), 'interleaved_parallel': False, 'alias_name': ('dp', 'mp'), 'rank_list': (0, 1, 2, 3, 4, 5, 6, 7), 'type': 'Float32', 'full_shape': (32, 4)}}, '7': {'weight': {'mesh_shape': (4, 2), 'tensor_map': (0, -1), 'interleaved_parallel': False, 'alias_name': ('dp', 'mp'), 'rank_list': (0, 1, 2, 3, 4, 5, 6, 7), 'type': 'Float32', 'full_shape': (32, 4)}}}