TTom Stellard[MLIR] Fix checks for native arch
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
[mlir] Overhaul C/Python registration APIs to properly scope registration/loading activities. Since the very first commits, the Python and C MLIR APIs have had mis-placed registration/load functionality for dialects, extensions, etc. This was done pragmatically in order to get bootstrapped and then just grew in. Downstreams largely bypass and do their own thing by providing various APIs to register things they need. Meanwhile, the C++ APIs have stabilized around this and it would make sense to follow suit. The thing we have observed in canonical usage by downstreams is that each downstream tends to have native entry points that configure its installation to its preferences with one-stop APIs. This patch leans in to this approach with RegisterEverything.h and mlir._mlir_libs._mlirRegisterEverything being the one-stop entry points for the "upstream packages". The _mlir_libs.__init__.py now allows customization of the environment and Context by adding "initialization modules" to the _mlir_libs package. If present, _mlirRegisterEverything is treated as such a module. Others can be added by downstreams by adding a _site_initialize_{i}.py module, where '{i}' is a number starting with zero. The number will be incremented and corresponding module loaded until one is not found. Initialization modules can: * Perform load time customization to the global environment (i.e. registering passes, hooks, etc). * Define a register_dialects(registry: DialectRegistry) function that can extend the DialectRegistry that will be used to bootstrap the Context. * Define a context_init_hook(context: Context) function that will be added to a list of callbacks which will be invoked after dialect registration during Context initialization. Note that the MLIRPythonExtension.RegisterEverything is not included by default when building a downstream (its corresponding behavior was prior). For downstreams which need the default MLIR initialization to take place, they must add this back in to their Python CMake build just like they add their own components (i.e. to add_mlir_python_common_capi_library and add_mlir_python_modules). It is perfectly valid to not do this, in which case, only the things explicitly depended on and initialized by downstreams will be built/packaged. If the downstream has not been set up for this, it is recommended to simply add this back for the time being and pay the build time/package size cost. CMake changes: * MLIRCAPIRegistration -> MLIRCAPIRegisterEverything (renamed to signify what it does and force an evaluation: a number of places were incidentally linking this very expensive target) * MLIRPythonSoure.Passes removed (without replacement: just drop) * MLIRPythonExtension.AllPassesRegistration removed (without replacement: just drop) * MLIRPythonExtension.Conversions removed (without replacement: just drop) * MLIRPythonExtension.Transforms removed (without replacement: just drop) Header changes: * mlir-c/Registration.h is deleted. Dialect registration functionality is now in IR.h. Registration of upstream features are in mlir-c/RegisterEverything.h. When updating MLIR and a couple of downstreams, I found that proper usage was commingled so required making a choice vs just blind S&R. Python APIs removed: * mlir.transforms and mlir.conversions (previously only had an __init__.py which indirectly triggered mlirRegisterTransformsPasses() and mlirRegisterConversionPasses() respectively). Downstream impact: Remove these imports if present (they now happen as part of default initialization). * mlir._mlir_libs._all_passes_registration, mlir._mlir_libs._mlirTransforms, mlir._mlir_libs._mlirConversions. Downstream impact: None expected (these were internally used). C-APIs changed: * mlirRegisterAllDialects(MlirContext) now takes an MlirDialectRegistry instead. It also used to trigger loading of all dialects, which was already marked with a TODO to remove -- it no longer does, and for direct use, dialects must be explicitly loaded. Downstream impact: Direct C-API users must ensure that needed dialects are loaded or call mlirContextLoadAllAvailableDialects(MlirContext) to emulate the prior behavior. Also see the ir.c test case (e.g. mlirContextGetOrLoadDialect(ctx, mlirStringRefCreateFromCString("func"));). * mlirDialectHandle* APIs were moved from Registration.h (which now is restricted to just global/upstream registration) to IR.h, arguably where it should have been. Downstream impact: include correct header (likely already doing so). C-APIs added: * mlirContextLoadAllAvailableDialects(MlirContext): Corresponds to C++ API with the same purpose. Python APIs added: * mlir.ir.DialectRegistry: Mapping for an MlirDialectRegistry. * mlir.ir.Context.append_dialect_registry(MlirDialectRegistry) * mlir.ir.Context.load_all_available_dialects() * mlir._mlir_libs._mlirAllRegistration: New native extension that exposes a register_dialects(MlirDialectRegistry) entry point and performs all upstream pass/conversion/transforms registration on init. In this first step, we eagerly load this as part of the __init__.py and use it to monkey patch the Context to emulate prior behavior. * Type caster and capsule support for MlirDialectRegistry This should make it possible to build downstream Python dialects that only depend on a subset of MLIR. See: https://github.com/llvm/llvm-project/issues/56037 Here is an example PR, minimally adapting IREE to these changes: https://github.com/iree-org/iree/pull/9638/files In this situation, IREE is opting to not link everything, since it is already configuring the Context to its liking. For projects that would just like to not think about it and pull in everything, add MLIRPythonExtension.RegisterEverything to the list of Python sources getting built, and the old behavior will continue. Reviewed By: mehdi_amini, ftynse Differential Revision: https://reviews.llvm.org/D128593 | 3 年前 | |
[mlir] Add MLIR-C dylib. Per discussion on discord and various feature requests across bindings (Haskell and Rust bindings authors have asked me directly), we should be building a link-ready MLIR-C dylib which exports the C API and can be used without linking to anything else. This patch: * Adds a new MLIR-C aggregate shared library (libMLIR-C.so), which is similar in name and function to libLLVM-C.so. * It is guarded by the new CMake option MLIR_BUILD_MLIR_C_DYLIB, which has a similar purpose/name to the LLVM_BUILD_LLVM_C_DYLIB option. * On all platforms, this will work with both static, BUILD_SHARED_LIBS, and libMLIR builds, if supported: * In static builds: libMLIR-C.so will export the CAPI symbols and statically link all dependencies into itself. * In BUILD_SHARED_LIBS: libMLIR-C.so will export the CAPI symbols and have dynamic dependencies on implementation shared libraries. * In libMLIR.so mode: same as static. libMLIR.so was not finished for actual linking use within the project. An eventual relayering so that libMLIR-C.so depends on libMLIR.so is possible but requires first re-engineering the latter to use the aggregate facility. * On Linux, exported symbols are filtered to only the CAPI. On others (MacOS, Windows), all symbols are exported. A CMake status is printed unless if global visibility is hidden indicating that this has not yet been implemented. The library should still work, but it will be larger and more likely to conflict until fixed. Someone should look at lifting the corresponding support from libLLVM-C.so and adapting. Or, for special uses, just build with -DCMAKE_CXX_VISIBILITY_PRESET=hidden -DCMAKE_C_VISIBILITY_PRESET=hidden. * Includes fixes to execution engine symbol export macros to enable default visibility. Without this, the advice to use hidden visibility would have resulted in test failures and unusable execution engine support libraries. Differential Revision: https://reviews.llvm.org/D113731 | 4 年前 | |
[mlir] move SCF headers to SCF/{IR,Transforms} respectively This aligns the SCF dialect file layout with the majority of the dialects. Reviewed By: jpienaar Differential Revision: https://reviews.llvm.org/D128049 | 3 年前 | |
Reland [mlir] Remove uses of LLVM's legacy pass manager Use the new pass manager. This also removes the ability to run arbitrary sets of passes. Not sure if this functionality is used, but it doesn't seem to be tested. No need to initialize passes outside of constructing the PassBuilder with the new pass manager. Reland: Fixed custom calls to -lower-matrix-intrinsics in integration tests by replacing them with -O0 -enable-matrix. Reviewed By: mehdi_amini Differential Revision: https://reviews.llvm.org/D123425 | 4 年前 | |
[mlir] Refactor the Parser library in preparation for an MLIR binary format The current Parser library is solely focused on providing API for the textual MLIR format, but MLIR will soon also provide a binary format. This commit renames the current Parser library to AsmParser to better correspond to what the library is actually intended for. A new Parser library is added which will act as a unified parser interface between both text and binary formats. Most parser clients are unaffected, given that the unified interface is essentially the same as the current interface. Only clients that rely on utilizing the AsmParserState, or those that want to parse Attributes/Types need to be updated to point to the AsmParser library. Differential Revision: https://reviews.llvm.org/D129605 | 3 年前 | |
[mlir][capi] Unbreak Interfaces CAPI after 2387fadea3 No idea why check-mlir doesn't build this. | 4 年前 | |
[mlir] Overhaul C/Python registration APIs to properly scope registration/loading activities. Since the very first commits, the Python and C MLIR APIs have had mis-placed registration/load functionality for dialects, extensions, etc. This was done pragmatically in order to get bootstrapped and then just grew in. Downstreams largely bypass and do their own thing by providing various APIs to register things they need. Meanwhile, the C++ APIs have stabilized around this and it would make sense to follow suit. The thing we have observed in canonical usage by downstreams is that each downstream tends to have native entry points that configure its installation to its preferences with one-stop APIs. This patch leans in to this approach with RegisterEverything.h and mlir._mlir_libs._mlirRegisterEverything being the one-stop entry points for the "upstream packages". The _mlir_libs.__init__.py now allows customization of the environment and Context by adding "initialization modules" to the _mlir_libs package. If present, _mlirRegisterEverything is treated as such a module. Others can be added by downstreams by adding a _site_initialize_{i}.py module, where '{i}' is a number starting with zero. The number will be incremented and corresponding module loaded until one is not found. Initialization modules can: * Perform load time customization to the global environment (i.e. registering passes, hooks, etc). * Define a register_dialects(registry: DialectRegistry) function that can extend the DialectRegistry that will be used to bootstrap the Context. * Define a context_init_hook(context: Context) function that will be added to a list of callbacks which will be invoked after dialect registration during Context initialization. Note that the MLIRPythonExtension.RegisterEverything is not included by default when building a downstream (its corresponding behavior was prior). For downstreams which need the default MLIR initialization to take place, they must add this back in to their Python CMake build just like they add their own components (i.e. to add_mlir_python_common_capi_library and add_mlir_python_modules). It is perfectly valid to not do this, in which case, only the things explicitly depended on and initialized by downstreams will be built/packaged. If the downstream has not been set up for this, it is recommended to simply add this back for the time being and pay the build time/package size cost. CMake changes: * MLIRCAPIRegistration -> MLIRCAPIRegisterEverything (renamed to signify what it does and force an evaluation: a number of places were incidentally linking this very expensive target) * MLIRPythonSoure.Passes removed (without replacement: just drop) * MLIRPythonExtension.AllPassesRegistration removed (without replacement: just drop) * MLIRPythonExtension.Conversions removed (without replacement: just drop) * MLIRPythonExtension.Transforms removed (without replacement: just drop) Header changes: * mlir-c/Registration.h is deleted. Dialect registration functionality is now in IR.h. Registration of upstream features are in mlir-c/RegisterEverything.h. When updating MLIR and a couple of downstreams, I found that proper usage was commingled so required making a choice vs just blind S&R. Python APIs removed: * mlir.transforms and mlir.conversions (previously only had an __init__.py which indirectly triggered mlirRegisterTransformsPasses() and mlirRegisterConversionPasses() respectively). Downstream impact: Remove these imports if present (they now happen as part of default initialization). * mlir._mlir_libs._all_passes_registration, mlir._mlir_libs._mlirTransforms, mlir._mlir_libs._mlirConversions. Downstream impact: None expected (these were internally used). C-APIs changed: * mlirRegisterAllDialects(MlirContext) now takes an MlirDialectRegistry instead. It also used to trigger loading of all dialects, which was already marked with a TODO to remove -- it no longer does, and for direct use, dialects must be explicitly loaded. Downstream impact: Direct C-API users must ensure that needed dialects are loaded or call mlirContextLoadAllAvailableDialects(MlirContext) to emulate the prior behavior. Also see the ir.c test case (e.g. mlirContextGetOrLoadDialect(ctx, mlirStringRefCreateFromCString("func"));). * mlirDialectHandle* APIs were moved from Registration.h (which now is restricted to just global/upstream registration) to IR.h, arguably where it should have been. Downstream impact: include correct header (likely already doing so). C-APIs added: * mlirContextLoadAllAvailableDialects(MlirContext): Corresponds to C++ API with the same purpose. Python APIs added: * mlir.ir.DialectRegistry: Mapping for an MlirDialectRegistry. * mlir.ir.Context.append_dialect_registry(MlirDialectRegistry) * mlir.ir.Context.load_all_available_dialects() * mlir._mlir_libs._mlirAllRegistration: New native extension that exposes a register_dialects(MlirDialectRegistry) entry point and performs all upstream pass/conversion/transforms registration on init. In this first step, we eagerly load this as part of the __init__.py and use it to monkey patch the Context to emulate prior behavior. * Type caster and capsule support for MlirDialectRegistry This should make it possible to build downstream Python dialects that only depend on a subset of MLIR. See: https://github.com/llvm/llvm-project/issues/56037 Here is an example PR, minimally adapting IREE to these changes: https://github.com/iree-org/iree/pull/9638/files In this situation, IREE is opting to not link everything, since it is already configuring the Context to its liking. For projects that would just like to not think about it and pull in everything, add MLIRPythonExtension.RegisterEverything to the list of Python sources getting built, and the old behavior will continue. Reviewed By: mehdi_amini, ftynse Differential Revision: https://reviews.llvm.org/D128593 | 3 年前 | |
[mlir] Add MLIR-C dylib. Per discussion on discord and various feature requests across bindings (Haskell and Rust bindings authors have asked me directly), we should be building a link-ready MLIR-C dylib which exports the C API and can be used without linking to anything else. This patch: * Adds a new MLIR-C aggregate shared library (libMLIR-C.so), which is similar in name and function to libLLVM-C.so. * It is guarded by the new CMake option MLIR_BUILD_MLIR_C_DYLIB, which has a similar purpose/name to the LLVM_BUILD_LLVM_C_DYLIB option. * On all platforms, this will work with both static, BUILD_SHARED_LIBS, and libMLIR builds, if supported: * In static builds: libMLIR-C.so will export the CAPI symbols and statically link all dependencies into itself. * In BUILD_SHARED_LIBS: libMLIR-C.so will export the CAPI symbols and have dynamic dependencies on implementation shared libraries. * In libMLIR.so mode: same as static. libMLIR.so was not finished for actual linking use within the project. An eventual relayering so that libMLIR-C.so depends on libMLIR.so is possible but requires first re-engineering the latter to use the aggregate facility. * On Linux, exported symbols are filtered to only the CAPI. On others (MacOS, Windows), all symbols are exported. A CMake status is printed unless if global visibility is hidden indicating that this has not yet been implemented. The library should still work, but it will be larger and more likely to conflict until fixed. Someone should look at lifting the corresponding support from libLLVM-C.so and adapting. Or, for special uses, just build with -DCMAKE_CXX_VISIBILITY_PRESET=hidden -DCMAKE_C_VISIBILITY_PRESET=hidden. * Includes fixes to execution engine symbol export macros to enable default visibility. Without this, the advice to use hidden visibility would have resulted in test failures and unusable execution engine support libraries. Differential Revision: https://reviews.llvm.org/D113731 | 4 年前 | |
[MLIR] Fix checks for native arch Using if (TARGET ${LLVM_NATIVE_ARCH}) only works if MLIR is built together with LLVM, but not for standalone builds of MLIR. The correct way to check this is if (${LLVM_NATIVE_ARCH} IN_LIST LLVM_TARGETS_TO_BUILD), as the LLVM build system exports LLVM_TARGETS_TO_BUILD. To avoid repeating the same check many times, add a MLIR_ENABLE_EXECUTION_ENGINE variable. Differential Revision: https://reviews.llvm.org/D131071 (cherry picked from commit 57a9bccec7dea036dbfa1a78f1ec5e73ecf7a33c) | 3 年前 |