Open MPI main development repository
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
Remove V=1 from community Jenkins make invocations Commit 80cd3187c1 added both V=1 and VERBOSE=1 to MAKE_ARGS in the community Jenkins pr-builder.sh. V=1 turns off Automake's terse build summaries and instead echoes the full compiler/linker command lines for every source file. On the community Jenkins this blew up the size of the build logs enormously, making them unwieldy without a corresponding benefit for routine PR builds. Drop V=1 and keep only VERBOSE=1. VERBOSE=1 is the piece we actually want: it makes the Automake test harness cat each failing test's error log into stdout, so when a test fails its full output is captured in the CI logs and is available for diagnosis. Signed-off-by: Jeff Squyres <jeff@squyres.com> | 2 个月前 | |
Merge pull request #14314 from hppritcha/push_openpmix_prrte_shas_to_top_of_masters2 PMIX/PRRTE: advance shas to near head of masters | 18 天前 | |
Merge pull request #14314 from hppritcha/push_openpmix_prrte_shas_to_top_of_masters2 PMIX/PRRTE: advance shas to near head of masters | 18 天前 | |
opal/datatype: remove convertor checksum support The checksum-enabled convertor compiled pack and unpack a second time. No production code has used it for years, but it retained duplicate hot-path logic, convertor state, and a separate internal library. Remove checksum mode, its API and state, duplicate library, configure option, unused CRC utility, and dedicated test. Keep normal copies routed through cbmemcpy so accelerator handling is unchanged. With the checksum state gone, group immutable descriptor and dispatch fields separately from mutable runtime fields. Keep the frequently accessed conversion state on one cache line without changing the overall convertor size. Signed-off-by: George Bosilca <gbosilca@nvidia.com> | 22 天前 | |
test/datatype: add pack benchmarks, contrib tools, and resize coverage Keep to_self limited to public MPI datatype constructions suitable for release-to-release performance comparisons. Retain selectable operations, DDTBench 1.2.1 layouts, robust timing statistics, and datatype-specific by-hand baselines. Add pack_description_sweep for implementation-specific experiments that install exact optimized descriptors and control count, block length, loop grouping, and equivalent copy shapes. Exercise heterogeneous fragmentation separately: packing must not split a predefined element, while unpacking must assemble elements delivered one byte at a time. Retain per-trial timing samples in a trials.tsv sidecar so bootstrap confidence intervals can be computed from raw data after the run. Use independent derived-to-compact and compact-to-derived communication endpoints to isolate send-side and receive-side performance without conflating them in a single round-trip measurement. Add tools under contrib/datatype to build and run the public to_self benchmark across MPI installations, retain raw trials, compare pack, unpack, and communication operations, and generate per-datatype figures. Add an mpirun-driven architecture tuner, extend pack_description_sweep with unpack and fragmented-iovec controls, and add compare_pack_runs.py for point-by-point speedup with bootstrap confidence intervals. MPI_Type_create_resized installs explicit lower/upper bound markers on a type. Spell that bound-marker model out in opal_datatype_resize() and opal_datatype_add(), fix to_self's optimized_indexed_gap to stride identically for count > 1, and add resized_extent to pin the epsilon-suppression path. Signed-off-by: George Bosilca <gbosilca@nvidia.com> | 21 天前 | |
Merge pull request #14314 from hppritcha/push_openpmix_prrte_shas_to_top_of_masters2 PMIX/PRRTE: advance shas to near head of masters | 18 天前 | |
Remove the Open MPI Java MPI bindings The Java MPI bindings were always experimental, were never part of the MPI standard, and are no longer maintained. Remove them in their entirety. Deleted: - the ompi/mpi/java tree (Java sources and the JNI C glue) - the mpijavac wrapper compiler (mpijavac.pl.in) - the Java example programs (Hello/Ring/Connectivity.java) - the LANL macosx-dynamic-java contrib platform files, whose sole purpose was building the Java bindings Removed the Java build machinery: the --enable-mpi-java configure option, the ompi_setup_java / ompi_setup_mpi_java m4 macros, the OMPI_WANT_JAVA_BINDINGS automake conditional and preprocessor define, the libmpi_java shared-library versioning, and the related Makefile.am hooks. Removed the now-dead Java op-callback infrastructure from ompi/op (the java_data union member, the OMPI_OP_FLAGS_JAVA_FUNC flag, the ompi_op_set_java_callback() setter, and the reduction dispatch branch). The "Java bindings" line in ompi_info is retained but hard-coded to "no" so anything parsing that field keeps working. Updated all documentation to drop references to the Java bindings, and added a v6.0.0 changelog entry recording the removal. Signed-off-by: Jeff Squyres <jeff@squyres.com> | 2 个月前 | |
Merge pull request #14314 from hppritcha/push_openpmix_prrte_shas_to_top_of_masters2 PMIX/PRRTE: advance shas to near head of masters | 18 天前 | |
Merge pull request #14126 from bosilca/topic/improved_pack_unpack Improve derived datatype optimization and pack/unpack performance | 20 天前 | |
OSHMEM/SCOLL/UCC: Advertise ep_map field so teams are cacheable Set UCC_TEAM_PARAM_FIELD_EP_MAP in the team-create mask. The component already populates team_params.ep_map, but UCC keys optional fields off the mask, so without the bit every OpenSHMEM team was rejected as uncacheable by the UCC team cache. Signed-off-by: Bryce Westheimer <bwestheimer@nvidia.com> | 1 个月前 | |
Add the MPI standard ABI test suite and its make check targets Open MPI now builds a standard-ABI C library (libmpi_abi) alongside its traditional library, together with an mpicc_abi wrapper and a standard ABI header. That ABI layer is a distinct surface: it exposes the MPI Forum ABI header and constants, advertises ABI include and link paths through its wrapper, translates between standard ABI integer handle/sentinel values and Open MPI's internal handle representation, and forwards public MPI_* calls into the existing implementation. None of that was covered by Open MPI's general MPI correctness tests, which exercise the traditional library and assume the implementation beneath the ABI layer is already tested. This adds a dedicated test suite under ompi/test/mpi-abi/ that checks the ABI-facing surface from several directions -- metadata authority, installed artifacts, symbol reachability, handle translation, complete public-API call paths, callback conversion, and cross-implementation compatibility -- without re-testing the underlying MPI algorithms. Passing these tests does not prove every underlying MPI algorithm is correct; it proves the ABI surface is consistent with the standard ABI metadata derived from the MPI standard and can drive the already-tested Open MPI implementation through the ABI path. The runner is a Python program (mpi_abi_tests.py) split across sibling _abi_*.py modules for discovery, manifest, probe generation, fast checks, installed checks, cross-implementation checks, lookup tables, and reporting. Every module, template, and generated test case is listed in EXTRA_DIST so VPATH and distribution-tarball builds can import and run the suite from a read-only source tree; the runner is invoked with "python -B" so it never writes __pycache__ next to the modules in the source tree. Probe bodies are generated from .cbody.in and .prologue.in templates, and each logical probe is compiled into its own executable because MPI process state is undefined after many runtime failures. Reports are written as JSON and text into mode-specific build-tree directories. The suite is wired into Automake so its checks run in CI, and it remains Python 3.7 compatible like the rest of Open MPI's Python tooling. A new top-level requirements.txt unions the per-area docs/requirements.txt and ompi/test/mpi-abi/requirements.txt files, so installing that one file provides every Python package needed both to build the documentation and man pages with Sphinx and to run all of the MPI ABI checks. Three make targets drive the suite, each with different prerequisites and its own results directory: * "make check" runs the fast metadata, manifest, and source checks (the runner's check-fast mode, reached through check-local). These run entirely from the source and build trees and require neither an installed Open MPI nor mpicc_abi nor mpirun, so they are safe in any build environment and participate in the normal recursive make check. They compare the MPI-standard-derived ABI metadata under docs/ against the runner's manifest, classification rules, generated-source contracts, C header constants, and Fortran helper source contracts, catching drift between the ABI description and what the suite believes is implemented, skipped, or still uncovered before anything is installed or launched. Output goes to check-results/. * "make check-abi" runs the installed standard ABI checks against an installed Open MPI. It uses the installed mpicc_abi wrapper, the installed standard ABI header, and installed mpirun, expected on PATH unless overridden by the OMPI_ABI_TEST_* environment or make variables. It verifies that the wrapper advertises the ABI include and link paths, that the installed header declares exactly the implemented standard ABI C APIs with signatures matching the binding metadata (and does not declare non-ABI APIs), and that the ABI library exports the expected MPI_* / PMPI_* symbols. It then exercises the ABI helper conversion functions (MPI_Comm_toint / _fromint, MPI_Type_toint, and their PMPI forms) by round-tripping predefined, null, and dynamic handles, status sentinels, error classes, keyval sentinels, and configured datatype constants; runs real MPI programs built with mpicc_abi and launched with mpirun that validate return codes, output handles, statuses, counts, data movement, object state, request completion, RMA, and MPI-IO results through the ABI entry points; isolates callback and retained-lifetime probes so one callback failure cannot poison other probes; and runs Fortran binding regression checks. Open MPI does not yet provide an ABI-capable Fortran wrapper, so the Fortran checks deliberately record current behavior -- for example MPI_Abi_get_version reporting -1, -1 -- rather than claiming MPI-5 Fortran ABI coverage. That absence is an intentional wait-and-see decision whose rationale is documented in docs/building-apps/mpi-forum-abi.rst. Output goes to check-abi-results/. * "make check-abi-mpich" runs the optional cross-implementation compatibility checks against MPICH, and is the most demanding target. It requires both an installed Open MPI with standard ABI support and an installed MPICH built with MPI Forum ABI support (for MPICH 5.0.x, configured with --enable-mpi-abi so it installs mpicc_abi, mpi_abi.h, and libmpi_abi). MPICH's normal internal ABI is not the MPI Forum ABI, and neither implementation's plain mpicc is a substitute, so the runner discovers and classifies the MPI Forum ABI wrappers and launchers before selecting them; explicit MPICH_ABI_TEST_* and OMPI_ABI_TEST_* overrides are honored as operator intent and validated rather than silently falling back to another tool on PATH. Because invoking this target is an explicit request for compatibility results, missing or invalid prerequisites are reported as failures, not skips. The target records both ABI directions -- compile with MPICH and run against Open MPI's ABI runtime, and compile with Open MPI's mpicc_abi and run against MPICH -- after first compiling and launching a one-rank MPI_Init / MPI_Finalize sanity program with each implementation's own wrapper, launcher, and ABI library so that broken local launchers are not misreported as ABI mismatches. For each cross-direction executable it sanitizes the platform runtime library path (LD_LIBRARY_PATH on Linux; DYLD_LIBRARY_PATH plus rewriting the embedded ABI dylib load commands on macOS) so a binary compiled against one implementation cannot load a stale libmpi from the shell environment at run time, and it applies MPICH transport defaults (FI_PROVIDER=tcp with a non-loopback, non-tunnel IPv4 interface for ch4:ofi builds; UCX_TLS=self,sm for ch4:ucx builds) for local one- and two-rank jobs. It treats libmpi_abi as the sole MPI Forum ABI library per MPI-5.0 section 21.2.1 and validates the PMPI alternate entry points required by section 16.2.1 as symbols and as calls through libmpi_abi. Output goes to check-abi-mpich-results/. The check-abi and check-abi-mpich targets are also defined at the top of the tree, where they recurse into ompi/test/mpi-abi for the OMPI project and otherwise print a SKIP message. All three targets additionally skip cleanly when configure did not find a usable Python. Signed-off-by: Jeff Squyres <jeff@squyres.com> Co-authored-by: Howard Pritchard <howardp@lanl.gov> | 1 个月前 | |
Merge pull request #14325 from ocaisa/patch-1 Suspected (harmless) copy/paste error in `test/util/Makefile.am` | 20 天前 | |
Fixing .clang-format to allow it to work with v10 On RHEL 8.3, the version of clang-format that Redhat provides is: > clang-format --version clang-format version 10.0.1 (Red Hat 10.0.1-1.module+el8.3.0+7459+90c24896) > git --version git v2.27.0 I tried to fix up master PR #8747 (master version of 8723) with the command: `git clang-format -v HEAD~1` but ran into some issues with clang-format v10.0.1 being able to parse the .clang-format file. I did not spend much time trying to find viable alternatives, instead I would just remove the offending line in the .clang-format file and rince/repeat. Are these options that we could do without, so that we could run clang-format with both v10 and v11? Signed-off-by: Geoffrey Paulsen <gpaulsen@us.ibm.com> | 5 年前 | |
datatype: add opt_desc/desc equivalence test, shared corpus, and unpack oracle Introduce a reusable, public-MPI-only datatype corpus (ompi/test/datatype/datatype_corpus.[ch]) that builds and commits a curated set of committed datatypes, each paired with an independent by-hand pack/unpack reference and a set of shape traits. The corpus is consumer-agnostic: it describes each datatype but knows nothing about how a given test selects, drives, or reports on them, so more than one test can share a single definition of each construction. Add opt_desc_equiv, which drives every corpus entry through both the optimized descriptor and a generic walk over the canonical descriptor, across a sweep of fragment sizes, and asserts the two packed streams are byte-identical and both round-trip correctly. This catches optimizer rewrites that would change the on-the-wire layout. When the corpus supplies a by-hand pack reference the opt-path packed stream is also compared against it, catching early-completion bugs that would otherwise cause both opt and desc paths to agree on a short stream. The by-hand unpack reference is similarly used as an independent oracle against the opt-path unpacked buffer, catching common-mode bugs that affect both unpack interpreters identically. PAIR entries (distinct send/recv datatypes) are checked twice: once for the send type and once for the receive type. Refactor to_self to consume the corpus instead of carrying its own builders and by-hand baselines; it keeps its historical --data selection by matching each corpus entry's name against its own option table. The builders (including the DDTBench-derived shapes) move verbatim into the corpus, and a few adversarial shapes -- single-iteration, zero/negative extent, overlap, and mixed-type promotion -- are added to stress the optimizer and positioning edges. Cap --trials, --cycles, and --warmups at MAX_TRIALS (10000) to prevent stack overflows from the VLAs in print_result() sized by num_trials. Document that dump_datatypes is intentionally not broadcast to other ranks since datatype optimization is deterministic. Also fix OPAL_DATATYPE_SAFEGUARD_POINTER to bound negative-extent datatypes correctly. Instances may stride backward, so the region touched by COUNT copies is the union over all of them: fold a negative span into the lower bound and a positive span into the upper bound rather than assuming the last copy sits at the highest address. Without this the corpus's adversarial negative-extent shape trips false-positive debug warnings. Signed-off-by: George Bosilca <gbosilca@nvidia.com> | 21 天前 | |
docs: Render C/F90/F08 bindings in man pages Use the pympistandard Python module and associated MPI-4.1 MPI Standard JSON definition file to generate bindings -- regular and embiggened -- in the MPI man pages. In the generation code, had to handle a variety of cases: * When there is no C binding * When there is no mpif.h / use mpi binding * When there is no use mpi_f08 binding * When there are embiggened C and/or F08 bindings * When a single man page includes documentation for multiple MPI APIs * When we do not have man pages for MPI APIs described in the official MPI Forum API JSON Also, this commit adds a new submodule: the pympistandard Python module from the MPI Forum repository. Some other cleanups were also necessary as part of this commit: * Add some missing man pages * Fix some random RST formatting errors in existing man pages Signed-off-by: Jeff Squyres <jeff@squyres.com> | 1 年前 | |
mailmap: Add alternate email address for Brelle Emmanuel Signed-off-by: Brelle Emmanuel <emmanuel.brelle@bull.com> | 3 个月前 | |
Add secret exception for trufflehog scan This PR adds an allowlist configuration to exclude a false positive detection of a test private key in the libevent library. The file is part of the libevent test suite and is not a real secret. Signed-off-by: Arnaud Celermajer <acelermajer@nvidia.com> | 7 个月前 | |
docs: add LLM-friendly documentation artifacts Publish machine-readable, LLM-friendly documentation for the public Open MPI MPI API alongside the existing human-facing HTML and Unix man pages, so that LLMs, retrieval systems, and coding assistants can obtain concise, authoritative, version-correct API information without scraping themed HTML -- and without introducing a second, hand-maintained API reference that could drift from the real documentation. Everything is derived from the existing RST man pages and the MPI Forum binding metadata (docs/mpi-standard-apis.json, read via the embedded pympistandard library); none of the API content is independently authored. Generated artifacts. Written into a build-tree staging directory and carried inside the per-version HTML output tree, so they inherit the existing tarball, install, and Read the Docs publishing machinery: * llms.txt -- a short, self-describing index for each documentation version. * llms/openmpi-mpi-api.jsonl -- one structured JSON record per documented MPI procedure (standard, extension, deprecated, removed). * llms/openmpi-mpi-api.md and four per-interface corpora (C, mpif.h, use mpi, use mpi_f08). * llms/man-openmpi/man3/MPI_<name>.3.md -- one Markdown page per man3 page, 1:1 with the human man pages. * llms/man-openmpi/man1/<command>.1.md -- a Markdown corpus for the section-1 command man pages (mpirun, ompi_info, the wrapper compilers, ...); commands rather than APIs, so Markdown but no JSONL records. * llms/openmpi-docs-manifest.json -- the artifact inventory, and the only artifact that carries build identity. Curated, committed sources under docs/llms-src/: an interface-selection guide, a small examples corpus (reusing the top-level examples/ tree where it already covers a case), and the two JSON Schema files, which serve as both the published contract and the CI validator. Data model and invariants. * Two API populations. Standard procedures are metadata-driven (C, mpif.h / use mpi, and use mpi_f08 signatures, including large-count variants, from pympistandard). Extensions (MPIX_*, OMPI_*) are best-effort: signatures preserved verbatim from the RST, structured fields marked unknown. A "kind" field distinguishes them. * Each artifact's content hash is a pure function of its semantic content: build identity (git_commit, git_describe, generated_at) lives only in the manifest, so a per-artifact hash changes if and only if that artifact's documentation content changes. * Reproducible under the project's existing SOURCE_DATE_EPOCH model: the semantic artifacts are wall-clock-free and byte-identical across reruns at a given commit. As part of this, a pre-existing gap is closed by deriving the docs copyright year from SOURCE_DATE_EPOCH. * Link strategy follows the build type. A local git or release-tarball build emits links relative to each file (a self-contained, offline tree); a Read the Docs build emits absolute, slug-correct links. Because Read the Docs serves the version-neutral /llms.txt from the default version, llms.txt self-describes the .../en/VERSION_SLUG/ URL scheme so a consumer can reach any version. Build integration and shared infrastructure. * Shared metadata logic (pympistandard loading, C/Fortran binding rendering, VERSION parsing, ".. mpi-bindings:" directive parsing, and build-identity helpers) is factored into docs/ompi_docs_common.py and used by both the new generator and the existing man3 bindings generator. * docs/generate-llm-docs.py runs from a Makefile sentinel for "make" builds and from .readthedocs-pre-create-environment.sh for Read the Docs builds; a conf.py "build-finished" hook copies the staging tree into the Sphinx HTML output, so publication is identical under make and on Read the Docs. Validation. docs/validate-llm-docs.py runs as part of "make check" and verifies JSON Schema conformance of the catalog and manifest, cross-field invariants, that every record links back to the human docs, that the manifest inventories every artifact with matching hash and size, that the generated Markdown contains no unresolved RST, that the committed sample records match the generated catalog, and a determinism (no-diff) rerun at a fixed SOURCE_DATE_EPOCH. Two robustness fixes are included: "make check" now depends on the generation sentinel, so validation actually runs instead of being silently skipped on a clean tree; and a coverage gate fails the build if any MPI Forum procedure that Open MPI implements has no man page (detected from the public C header and the use-mpi-f08 sources), so an implemented MPI API can no longer be silently undocumented. Documentation and specification. The effort is documented for maintainers in docs/developers/llm-friendly-docs.rst -- intent, the llms.txt convention and external references, the design rationale, regeneration and the per-release manifest, and how to update the docs when MPI APIs change. The full design record, JSON Schemas, sample records, and task list live under specs/llms-friendly-docs/. A one-time feature release note is added to the changelog. Signed-off-by: Jeff Squyres <jeff@squyres.com> | 2 个月前 | |
ReadTheDocs CI builds updates Since RTD doesn't run autogen, configure, or make, we now have to manually copy a few RST files from the embedded PRRTE to the docs/ tree before RTD invokes Sphinx. Signed-off-by: Jeff Squyres <jeff@squyres.com> | 2 年前 | |
docs: warn agents about misuse of OMPI_HIDDEN Add guidance explaining when it is (and is not) safe to annotate a symbol with OMPI_HIDDEN, so that contributors -- and AI coding agents in particular -- do not reintroduce the class of build breakage that need to be fixed in the initial version of code present in PR #14317. As of Open MPI v6.0 the MPI interface is split across libmpi (Open MPI ABI) and libmpi_abi (standard MPI ABI), both of which link against the internal libopen_mpi. Many ompi_* symbols are defined in libopen_mpi but used from the bindings compiled into libmpi and libmpi_abi (for example, predefined handle objects like ompi_mpi_comm_parent and helpers like ompi_comm_split_type_hw_guided_support). Marking such a symbol OMPI_HIDDEN prevents it from being exported from libopen_mpi, so the links of both libmpi and libmpi_abi fail with unresolved symbols. Two files are updated: - AGENTS.md: add a "Golden rules" bullet describing the hazard and the rule of thumb -- if a symbol crosses a library boundary, use OMPI_DECLSPEC (or leave it un-annotated), never OMPI_HIDDEN. Only hide symbols that are certainly private to a single DSO. - docs/developers/source-code.rst: add a "Hiding symbols with OMPI_HIDDEN" subsection under Symbol Visibility that documents the v6.0 library structure and includes a warning admonition covering the unresolved-symbol failure mode and the same rule of thumb. These are documentation-only changes. CLAUDE.md is a symlink to AGENTS.md and is covered automatically. Signed-off-by: Howard Pritchard <howardp@lanl.gov> | 23 天前 | |
Sym link AGENTS.md to CLAUDE.md Apparently recent versions of Claude Code do not automatically read AGENTS.md. Make a sym link to CLAUDE.md so that that context is automatically read in when Claude Code starts. Signed-off-by: Jeff Squyres <jeff@squyres.com> | 2 个月前 | |
Changed the hyperlink in README.md and HACKING.md to correctly reflect the spinx webpage Signed-off-by: JTM-UofL <jtmarr2003@gmail.com> | 1 年前 | |
Add acoll collective component acoll is a collective component optimized for AMD "Zen"-based processors. It supports Bcast, Allreduce, Reduce, Barrier, Gather and Allgather APIs. Signed-off-by: Nithya V S <Nithya.VS@amd.com> | 2 年前 | |
Makefile.am: exclude test/ from the common-symbol scan The install-exec-hook runs config/find_common_syms to detect C common symbols (uninitialized globals) that should not appear in installed libraries. Test programs legitimately use uninitialized globals, and after the test suite was reorganised into sub-projects the scan started picking up object files under test/ that were never meant to be checked. Add --skipdir=test alongside the existing --skipdir=3rd-party so the scanner only inspects the production build artifacts it was designed for. Signed-off-by: George Bosilca <gbosilca@nvidia.com> | 23 天前 | |
Move the single-process MPI tests from test/mpi to ompi/test The MPI-layer tests that "make check" can run without a launcher were living under the top-level test/ tree, far from the OMPI code they exercise. Create ompi/test/ as their home, alongside that code, and move them there: test/mpi/t -> ompi/test/t (MPI_T category/events) test/mpi/file -> ompi/test/file (OMPIO MPI_File info hints) test/datatype -> ompi/test/datatype (MPI datatype tests only) All of these are single-process, self-checking programs that need no mpirun, so they remain wired into TESTS and continue to run under both a top-level "make check" and one run solely inside ompi/. Now that they live under ompi/, their "if PROJECT_OMPI" guards are vacuously true -- ompi/ only exists in an OMPI build -- so drop them. test/datatype was a mixed directory: thirteen programs that link libmpi and exercise the OMPI datatype engine and the MPI datatype API, plus to_self and reduce_local (built, never in TESTS), alongside three genuinely OPAL-level tests that link only libopen-pal. Split it along that line: the libmpi half moves to ompi/test/datatype, and opal_datatype_test, opal_datatype_bigcount, and unpack_hetero stay behind. The split is clean -- ddt_lib.c is used only by the four MPI tests that move, while the OPAL side has its own opal_ddt_lib.c -- so nothing needs to be shared across the two directories. reduce_local's check_op.sh driver moves with it. test/datatype keeps its Makefile.mca-dso-check include: all three remaining tests call opal_init(). Delete test/mpi/environment along the way. It is stale: chello.c links against $(top_builddir)/src/libmpi.la and src/libompi.la, a source layout that has not existed for two decades, and the directory was already excluded from SUBDIRS. Its csh run_tests driver, and the test/mpi/run_tests that only ever recursed into it, go with it. Nothing else remains under test/mpi, so the directory disappears. Finally, hoist test/Makefile.mca-dso-check to the top of the tree as Makefile.mca-dso-check. It is now included from two different subtrees (test/{datatype,threads} and ompi/test/{file,datatype}), so neither should own it; this mirrors the existing top-level Makefile.ompi-rules fragment. Automake distributes an included fragment automatically, so no EXTRA_DIST is needed. Update the riscv64-qemu-test GitHub Action to match. Its "Run MPI_OP" step cd's into the test directory and invokes reduce_local by hand under qemu; test/datatype still exists (it keeps the three OPAL-level tests), so the cd silently succeeded and the run failed on a missing executable. Point it at ompi/test/datatype. No other workflow references a moved path. Signed-off-by: Jeff Squyres <jeff@squyres.com> | 1 个月前 | |
build: regenerate the bindings when the generator changes The generated Fortran mpi_f08 bindings depended only on the templates they were generated from, not on the generator that generated them. Editing ompi/mpi/bindings/ and running "make" therefore silently reused the stale generated sources. That is worse than it first appears. It is not merely that a change to the generator appears to have no effect: a partial rebuild can link a library in which some bindings were emitted by the old generator and some by the new one, with no warning. Add OMPI_BINDINGS_GENERATOR to Makefile.ompi-rules, listing the generator's sources, and declare an explicit dependency on it everywhere bindings are generated. Automake suffix rules cannot carry extra prerequisites, hence the separate dependency lines. Also add ompi/mpi/bindings/ompi_bindings/parser.py to EXTRA_DIST in ompi/mpi/Makefile.am. It has never been included in distribution tarballs: it was omitted when it was introduced, even though both c.py and fortran.py import it. The omission went unnoticed because tarball builds never re-run the bindings generator: the generated sources shipped in the tarball are already up to date, so the missing file was never touched. Now that the generator's sources are prerequisites of the generated bindings, the missing file is a hard "make distcheck" failure: make cannot satisfy the parser.py prerequisite. Signed-off-by: Jeff Squyres <jeff@squyres.com> | 1 个月前 | |
Changed the hyperlink in README.md and HACKING.md to correctly reflect the spinx webpage Signed-off-by: JTM-UofL <jtmarr2003@gmail.com> | 1 年前 | |
ABI: switch to using MPI Standard ABI values This mod switches from using "synthetic" defined values and handles to the ones specified in the MPI 5.1 standard, and bumps the MPI standard version reported by Open MPI to 5.0. The python infrastructure included here generates a "canonical" MPI standard ABI compatible mpi.h using two json files: 1) mpi-standard-abi.json 2) mpi-standard-apis.json 2 is generated as part of building the MPI standard. We import that into our project for use in generating both the mpi.h as well as interface definitions in the man pages. 1 is generated using a separate script that processes the tables in Appendix A of the standard. Ideally this script will be merged into the MPI standard code base at some point. This script is currently at https://github.com/Joe-Downs/mpi-standard/tree/pr/handle-constant-tool/const-tool . It is used to generate the portion of mpi.h where defined values and handles are specified. This commit also enables generation of the abi interfaces and header files by default. Signed-off-by: Joseph Downs <joe.downs@lanl.gov> Co-authored-by: Howard Pritchard <howardp@lanl.gov> Signed-off-by: Jeff Squyres <jeff@squyres.com> | 1 个月前 | |
Convert more Perl build scripts to Python Continue the slow trend of removing Perl from our build process, and move to a more sustainable (at least by number of developers who know the language) language: Python. Use generative AI to convert 3 Perl scripts to Python: mpif-values.pl, gen-mpi-mangling.pl, and gen-mpi-sizeof.pl. Validate the results by code review+inspection and diff'ing the results of the Python-generated files with the original Perl-generated files. Signed-off-by: Jeff Squyres <jeff@squyres.com> | 1 年前 | |
Remove ORTE project Will be replaced by PRRTE. Ensure that OMPI and OPAL layers build without reference to ORTE. Setup opal/pmix framework to be static. Remove support for all PMI-1 and PMI-2 libraries. Add support for "external" pmix component as well as internal v4 one. remove orte: misc fixes - UCX fixes - VPATH issue - oshmem fixes - remove useless definition - Add PRRTE submodule - Get autogen.pl to traverse PRRTE submodule - Remove stale orcm reference - Configure embedded PRRTE - Correctly pass the prefix to PRRTE - Correctly set the OMPI_WANT_PRRTE am_conditional - Move prrte configuration to the end of OMPI's configure.ac - Make mpirun a symlink to prun, when available - Fix makedist with --no-orte/--no-prrte option - Add a `--no-prrte` option which is the same as the legacy `--no-orte` option. - Remove embedded PMIx tarball. Replace it with new submodule pointing to OpenPMIx master repo's master branch - Some cleanup in PRRTE integration and add config summary entry - Correctly set the hostname - Fix locality - Fix singleton operations - Fix support for "tune" and "am" options Signed-off-by: Ralph Castain <rhc@pmix.org> Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp> Signed-off-by: Joshua Hursey <jhursey@us.ibm.com> | 6 年前 | |
configure: Add symbol re-export for v5.x backward compatibility In Open MPI v6.0, 138 predefined MPI symbols (e.g., ompi_mpi_comm_world, ompi_mpi_comm_self, predefined datatypes, and operators) were relocated from libmpi to the new libopen_mpi library to support the MPI standard ABI implementation. This change broke backward compatibility on Darwin with applications compiled against Open MPI v5.x that directly reference these symbols through libmpi. This commit implements platform-specific symbol re-export mechanisms to restore backward compatibility: - macOS: Uses -Wl,-reexport-lopen_mpi to create LC_REEXPORT_DYLIB dependency, making libopen_mpi symbols available through libmpi at runtime via the dynamic linker. configure.ac now detects the platform and linker capabilities, setting Automake conditionals (OMPI_REEXPORT_MACOS and OMPI_REEXPORT_VERSION_SCRIPT) that control the linking strategy in ompi/Makefile.am. The -reexport_library probe runs only on Darwin. A new test under ompi/test/reexport/ verifies that symbols resolve correctly through libmpi at runtime; it validates the exact shared library produced by the current build (passed through the environment by the test harness) instead of hard-coding sonames. This allows v5.x-compiled applications to run with v6.0 libraries without recompilation, while maintaining the architectural benefits of the libopen_mpi separation. On macOS, libopen_mpi is deliberately NOT in libmpi's LIBADD: the -reexport-lopen_mpi flag both links and re-exports it, and if the library also appeared in LIBADD, ld64 would record an ordinary LC_LOAD_DYLIB reference instead of LC_REEXPORT_DYLIB. Because that flag is invisible to libtool, libmpi.la must list libopen-pal.la explicitly; otherwise libtool omits opal/.libs from the runtime search path of uninstalled programs that link only libmpi.la, and "make check" fails before "make install" (which also breaks the inner check run of "make distcheck"). Drop the symbol re-export machinery on ELF platforms The v5.x-compatibility re-export of libopen_mpi's predefined MPI handle symbols (ompi_mpi_comm_world and friends) through libmpi is only needed on macOS, where the Mach-O two-level namespace records which library each symbol was found in at link time. On ELF platforms the global symbol namespace is flat: at run time the dynamic linker resolves a v5.x binary's references from any loaded library, and libmpi's DT_NEEDED dependency on libopen_mpi guarantees libopen_mpi is loaded. No re-export mechanism is needed at all, and ompi/test/reexport's symbol-resolution test passes unchanged with the version script removed. The --version-script implementation was also actively harmful: it attached an OMPI_6.0 version tag to every symbol libmpi exports. Open MPI has never shipped versioned symbols, so this broke consumers that parse symbol tables (mpi4py's test_mpiapi.py fails on the "MPI_Send@OMPI_6.0" references of anything compiled against the library), and it meant that libraries built against a versioned libmpi could not link against an unversioned rebuild. Introducing symbol versioning would be an ABI policy decision in its own right and should not ride in as a side effect of a compatibility mechanism. Remove the version-script configure probe, the OMPI_REEXPORT_VERSION_SCRIPT conditional, and the libmpi_symbol_exports.ver script (whose curated symbol list had already been reduced to an export-everything placeholder). The macOS -Wl,-reexport-lopen_mpi mechanism is unchanged. Signed-off-by: Howard Pritchard <howardp@lanl.gov> Signed-off-by: Jeff Squyres <jeff@squyres.com> | 1 个月前 | |
Add the MPI standard ABI test suite and its make check targets Open MPI now builds a standard-ABI C library (libmpi_abi) alongside its traditional library, together with an mpicc_abi wrapper and a standard ABI header. That ABI layer is a distinct surface: it exposes the MPI Forum ABI header and constants, advertises ABI include and link paths through its wrapper, translates between standard ABI integer handle/sentinel values and Open MPI's internal handle representation, and forwards public MPI_* calls into the existing implementation. None of that was covered by Open MPI's general MPI correctness tests, which exercise the traditional library and assume the implementation beneath the ABI layer is already tested. This adds a dedicated test suite under ompi/test/mpi-abi/ that checks the ABI-facing surface from several directions -- metadata authority, installed artifacts, symbol reachability, handle translation, complete public-API call paths, callback conversion, and cross-implementation compatibility -- without re-testing the underlying MPI algorithms. Passing these tests does not prove every underlying MPI algorithm is correct; it proves the ABI surface is consistent with the standard ABI metadata derived from the MPI standard and can drive the already-tested Open MPI implementation through the ABI path. The runner is a Python program (mpi_abi_tests.py) split across sibling _abi_*.py modules for discovery, manifest, probe generation, fast checks, installed checks, cross-implementation checks, lookup tables, and reporting. Every module, template, and generated test case is listed in EXTRA_DIST so VPATH and distribution-tarball builds can import and run the suite from a read-only source tree; the runner is invoked with "python -B" so it never writes __pycache__ next to the modules in the source tree. Probe bodies are generated from .cbody.in and .prologue.in templates, and each logical probe is compiled into its own executable because MPI process state is undefined after many runtime failures. Reports are written as JSON and text into mode-specific build-tree directories. The suite is wired into Automake so its checks run in CI, and it remains Python 3.7 compatible like the rest of Open MPI's Python tooling. A new top-level requirements.txt unions the per-area docs/requirements.txt and ompi/test/mpi-abi/requirements.txt files, so installing that one file provides every Python package needed both to build the documentation and man pages with Sphinx and to run all of the MPI ABI checks. Three make targets drive the suite, each with different prerequisites and its own results directory: * "make check" runs the fast metadata, manifest, and source checks (the runner's check-fast mode, reached through check-local). These run entirely from the source and build trees and require neither an installed Open MPI nor mpicc_abi nor mpirun, so they are safe in any build environment and participate in the normal recursive make check. They compare the MPI-standard-derived ABI metadata under docs/ against the runner's manifest, classification rules, generated-source contracts, C header constants, and Fortran helper source contracts, catching drift between the ABI description and what the suite believes is implemented, skipped, or still uncovered before anything is installed or launched. Output goes to check-results/. * "make check-abi" runs the installed standard ABI checks against an installed Open MPI. It uses the installed mpicc_abi wrapper, the installed standard ABI header, and installed mpirun, expected on PATH unless overridden by the OMPI_ABI_TEST_* environment or make variables. It verifies that the wrapper advertises the ABI include and link paths, that the installed header declares exactly the implemented standard ABI C APIs with signatures matching the binding metadata (and does not declare non-ABI APIs), and that the ABI library exports the expected MPI_* / PMPI_* symbols. It then exercises the ABI helper conversion functions (MPI_Comm_toint / _fromint, MPI_Type_toint, and their PMPI forms) by round-tripping predefined, null, and dynamic handles, status sentinels, error classes, keyval sentinels, and configured datatype constants; runs real MPI programs built with mpicc_abi and launched with mpirun that validate return codes, output handles, statuses, counts, data movement, object state, request completion, RMA, and MPI-IO results through the ABI entry points; isolates callback and retained-lifetime probes so one callback failure cannot poison other probes; and runs Fortran binding regression checks. Open MPI does not yet provide an ABI-capable Fortran wrapper, so the Fortran checks deliberately record current behavior -- for example MPI_Abi_get_version reporting -1, -1 -- rather than claiming MPI-5 Fortran ABI coverage. That absence is an intentional wait-and-see decision whose rationale is documented in docs/building-apps/mpi-forum-abi.rst. Output goes to check-abi-results/. * "make check-abi-mpich" runs the optional cross-implementation compatibility checks against MPICH, and is the most demanding target. It requires both an installed Open MPI with standard ABI support and an installed MPICH built with MPI Forum ABI support (for MPICH 5.0.x, configured with --enable-mpi-abi so it installs mpicc_abi, mpi_abi.h, and libmpi_abi). MPICH's normal internal ABI is not the MPI Forum ABI, and neither implementation's plain mpicc is a substitute, so the runner discovers and classifies the MPI Forum ABI wrappers and launchers before selecting them; explicit MPICH_ABI_TEST_* and OMPI_ABI_TEST_* overrides are honored as operator intent and validated rather than silently falling back to another tool on PATH. Because invoking this target is an explicit request for compatibility results, missing or invalid prerequisites are reported as failures, not skips. The target records both ABI directions -- compile with MPICH and run against Open MPI's ABI runtime, and compile with Open MPI's mpicc_abi and run against MPICH -- after first compiling and launching a one-rank MPI_Init / MPI_Finalize sanity program with each implementation's own wrapper, launcher, and ABI library so that broken local launchers are not misreported as ABI mismatches. For each cross-direction executable it sanitizes the platform runtime library path (LD_LIBRARY_PATH on Linux; DYLD_LIBRARY_PATH plus rewriting the embedded ABI dylib load commands on macOS) so a binary compiled against one implementation cannot load a stale libmpi from the shell environment at run time, and it applies MPICH transport defaults (FI_PROVIDER=tcp with a non-loopback, non-tunnel IPv4 interface for ch4:ofi builds; UCX_TLS=self,sm for ch4:ucx builds) for local one- and two-rank jobs. It treats libmpi_abi as the sole MPI Forum ABI library per MPI-5.0 section 21.2.1 and validates the PMPI alternate entry points required by section 16.2.1 as symbols and as calls through libmpi_abi. Output goes to check-abi-mpich-results/. The check-abi and check-abi-mpich targets are also defined at the top of the tree, where they recurse into ompi/test/mpi-abi for the OMPI project and otherwise print a SKIP message. All three targets additionally skip cleanly when configure did not find a usable Python. Signed-off-by: Jeff Squyres <jeff@squyres.com> Co-authored-by: Howard Pritchard <howardp@lanl.gov> | 1 个月前 |
Open MPI
The Open MPI Project is an open source implementation of the Message Passing Interface (MPI) specification that is developed and maintained by a consortium of academic, research, and industry partners. Open MPI is therefore able to combine the expertise, technologies, and resources from all across the High Performance Computing community in order to build the best MPI library available. Open MPI offers advantages for system and software vendors, application developers and computer science researchers.
Official documentation
The Open MPI documentation can be viewed in the following ways:
- Online at https://docs.open-mpi.org/
- In self-contained (i.e., suitable for local viewing, without an
internet connection) in official distribution tarballs under
docs/_build/html/index.html.
Building the documentation locally
The source code for Open MPI's docs can be found in the Open MPI Git
repository under the docs folder.
Developers who clone the Open MPI Git repository will not have the HTML documentation and man pages by default; it must be built. Instructions for how to build the Open MPI documentation can be found here: https://docs.open-mpi.org/en/main/developers/prerequisites.html#sphinx-and-therefore-python.