| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
[ci] Upgrade to new runners and disable unsupported jobs. (#2818) Per the RFC and numerous conversations on Discord, this rebuilds the torch-mlir CI and discontinues the infra and coupling to the binary releases (https://discourse.llvm.org/t/rfc-discontinuing-pytorch-1-binary-releases/76371). I iterated on this to get latency back to about what it was with the old (much larger and non-ephemeral) runners: About 4m - 4.5m for an incremental change. Behind the scenes changes: * Uses a new runner pool operated by AMD. It is currently set to manual scaling and has two runners (32-core, 64GiB RAM) while we get some traction. We can either fiddle with some auto-scaling or use a schedule to give it an increase during certain high traffic hours. * Builds are now completely isolated and cannot have run-to-run interference like we were getting before (i.e. lock file/permissions stuff). * The GHA runner is installed directly into a manylinux 2.28 container with upgraded dev tools. This eliminates the need to do sub-invocations of docker on Linux in order to run on the same OS that is used to build wheels. * While not using it now, this setup was cloned from another project that posts the built artifacts to the job and fans out testing. Might be useful here later. * Uses a special git cache that lets us have ephemeral runners and still check out the repo and deps (incl. llvm) in ~13s. * Running in an Azure VM Scale Set. In-repo changes: * Disables (but does not yet delete): * Old buildAndTest.yml jobs * releaseSnapshotPackage.yml * Adds a new ci.yml pipeline and scripts the steps in build_tools/ci (by decomposing the existing build_linux_packages.sh for in-tree builds and modularizing it a bit better). * Test framework changes: * Adds a TORCH_MLIR_TEST_CONCURRENCY env var that can be used to bound the multiprocess concurrency. Ended up not using this in the final version but is useful to have as a knob. * Changes the default concurrency to nproc * 0.8 + 1 vs nproc * 1.1. We're running on systems with significantly less virtual memory and I did a bit of fiddling to find a good tradeoff. * Changed multiprocess mode to spawn instead of fork. Otherwise, I was getting instability (as discussed on discord). * Added MLIR configuration to disable multithreaded contexts globally for the project. Constantly spawning nproc * nproc threads (more than that actually) was OOM'ing. * Added a test timeout of 5 minutes. If a multiprocess worker crashes, the framework can get wedged indefinitely (and then will just be reaped after multiple hours). We should fix this, but this at least keeps the CI pool from wedging with stuck jobs. Functional changes needing followup: * No matter what I did, I couldn't get the LTC tests to work, and I'm not 100% sure they were being run in the old setup as the scripts were a bit twisty. I disabled them and left a comment. * Dropped out-of-tree build variants. These were not providing much signal and increase CI needs by 50%. * Dropped MacOS and Windows builds. Now that we are "just a library" and not building releases, there is less pressure to test these commit by commit. Further, since we bump torch-mlir to known good commits on these platforms, it has been a long time since either of these jobs have provided much signal (and they take ~an hour+ to run). We can add them back later post-submit if ever needed. | 2 年前 | |
Upstream the ONNX importer. (#2636) This is part 1 of 2, which will also include upstreaming the FX importer. I started with ONNX because it forces some project layout updates and is more self contained/easier as a first step. Deviating somewhat from the RFCs on project layout, I made the following decisions: * Locating the onnx_importer.py into torch_mlir.extras as Maks already has opened up that namespace and it seemed to fit. Better to have fewer things at that level. * Setup the build so that the root project only contains MLIR Python and pure Python deps (like the importers), but this can be augmented with the projects/ adding more depending on which features are enabled. * The default build continues to build everything whereas in TORCH_MLIR_ENABLE_ONLY_MLIR_PYTHON_BINDINGS=1 mode, it builds a torch-mlir-core wheel with the pure contents only. onnx_importer.py and importer_smoke_test.py are almost verbatim copies from SHARK-Turbine. I made some minor local alterations to adapt to paths and generalize the way they interact with the outer project. I expect I can copy these back to Turbine verbatim from here. I also updated the license boilerplate (they have the same license but slightly different project norms for the headers) but retained the correct copyright. Other updates: * Added the ONNX importer unit test (which also can generate test data) in lit, conditioned on the availability of the Python onnx package. In a followup once I know everything is stable, I'll add another env var that the CI can set to always enable this so we know conclusively if tests pass. * Moved the ONNX conversion readme to docs/. * Renamed CMake option TORCH_MLIR_ENABLE_ONLY_MLIR_PYTHON_BINDINGS -> TORCH_MLIR_ENABLE_PYTORCH_EXTENSIONS and inverted the sense. Made the JitIR importer and LTC options cmake_dependent_options for robustness. | 2 年前 | |
[TORCH] Modified fx_importer to support hop_while_loop (#4338) This PR adds support for emitting graphs for Pytorch HOPs, beginning with torch._higher_order_ops.while_loop. The proposed change is to modify the import_program to call function _import_all_child_modules, which recursively imports the stateless graph for all the children modules. Since HOP operator graphs are stateless graphs with no mutation, it is correct to import them as stateless graphs, although the method import_stateless_graph is marked as "deprecated". --------- Signed-off-by: Keshav Vinayak Jha <keshavvinayakjha@gmail.com> | 9 个月前 | |
[onnx_importer.py] Fix dim_value None not correctly processed and missing Float8E4M3FNUZType. (#4037) As per title. Changes tested on SHARK-TestSuite's alt_e2eshark. | 1 年前 | |
[python] Make module imports relative in fx.py and compiler_utils.py (#3925) This PR makes module imports relative in fx.py and compiler_utils.py. When torch-mlir python package is embedded into python package of other MLIR based project, there won't be torch_mlir top level package for absolute import. | 1 年前 | |
build: manually update PyTorch version (#4102) This commit sets the PyTorch and TorchVision versions to nightly release 2025-03-25. This commit also adds the strict flag (by default set to False) in fx's export_and_import method in accordance with the changes made here https://github.com/pytorch/pytorch/commit/ab45aaca977f8261b460c970a61bcd6162733428. It also updates the torch.export call from the _export_run method in fx_importer_backend by adding the strict flag. --------- Signed-off-by: Vivek Khandelwal <vivekkhandelwal1424@gmail.com> | 1 年前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 2 年前 | ||
| 2 年前 | ||
| 9 个月前 | ||
| 1 年前 | ||
| 1 年前 | ||
| 1 年前 |