| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
refactor(flotilla): Swordfish task builder (#5976) ## Changes Made Passing SubmittableTask<SwordfishTask> through the distributed pipeline nodes makes it difficult to layer and modify tasks, because they contain not only the plan templates but also things like plan context, node ids, input partitions, notify tokens, cancel tokens, etc etc. Currently the pipeline nodes have to deal with all that when all it really wants to do is add some instruction to the plan template. This PR introduces SwordfishTaskBuilder, which takes care of all the non-instruction related fields and exposes methods like 'map_plan' that the pipeline nodes can use to modify the plan, and it will automatically deal with the changes needed on the other fields, like adding node id. Other minor changes: - Task ids are created upon building a SwordfishTaskBuilder into a SubmittableSwordfishTask. ## Related Issues <!-- Link to related GitHub issues, e.g., "Closes #123" --> | 8 个月前 | |
refactor: Remove Unloaded MicroPartitions (#5710) ## Changes Made With the removal of the Ray runner, we no longer use unloaded MicroPartitions for reading ScanTasks, so we can remove that code path entirely. Couple of Notes: * With these changes, I could also update the Partition trait (for MicroPartition and RayPartitionRef) to never error. * I left some code paths to support MicroPartition.read_parquet even though the runners don't use them, only some tests do. If you think I should remove those as well, lmk | 9 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 8 个月前 | ||
| 9 个月前 |