JJeff Squyresdocs: add LLM-friendly documentation artifacts
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
docs: add LLM-friendly documentation artifacts Publish machine-readable, LLM-friendly documentation for the public Open MPI MPI API alongside the existing human-facing HTML and Unix man pages, so that LLMs, retrieval systems, and coding assistants can obtain concise, authoritative, version-correct API information without scraping themed HTML -- and without introducing a second, hand-maintained API reference that could drift from the real documentation. Everything is derived from the existing RST man pages and the MPI Forum binding metadata (docs/mpi-standard-apis.json, read via the embedded pympistandard library); none of the API content is independently authored. Generated artifacts. Written into a build-tree staging directory and carried inside the per-version HTML output tree, so they inherit the existing tarball, install, and Read the Docs publishing machinery: * llms.txt -- a short, self-describing index for each documentation version. * llms/openmpi-mpi-api.jsonl -- one structured JSON record per documented MPI procedure (standard, extension, deprecated, removed). * llms/openmpi-mpi-api.md and four per-interface corpora (C, mpif.h, use mpi, use mpi_f08). * llms/man-openmpi/man3/MPI_<name>.3.md -- one Markdown page per man3 page, 1:1 with the human man pages. * llms/man-openmpi/man1/<command>.1.md -- a Markdown corpus for the section-1 command man pages (mpirun, ompi_info, the wrapper compilers, ...); commands rather than APIs, so Markdown but no JSONL records. * llms/openmpi-docs-manifest.json -- the artifact inventory, and the only artifact that carries build identity. Curated, committed sources under docs/llms-src/: an interface-selection guide, a small examples corpus (reusing the top-level examples/ tree where it already covers a case), and the two JSON Schema files, which serve as both the published contract and the CI validator. Data model and invariants. * Two API populations. Standard procedures are metadata-driven (C, mpif.h / use mpi, and use mpi_f08 signatures, including large-count variants, from pympistandard). Extensions (MPIX_*, OMPI_*) are best-effort: signatures preserved verbatim from the RST, structured fields marked unknown. A "kind" field distinguishes them. * Each artifact's content hash is a pure function of its semantic content: build identity (git_commit, git_describe, generated_at) lives only in the manifest, so a per-artifact hash changes if and only if that artifact's documentation content changes. * Reproducible under the project's existing SOURCE_DATE_EPOCH model: the semantic artifacts are wall-clock-free and byte-identical across reruns at a given commit. As part of this, a pre-existing gap is closed by deriving the docs copyright year from SOURCE_DATE_EPOCH. * Link strategy follows the build type. A local git or release-tarball build emits links relative to each file (a self-contained, offline tree); a Read the Docs build emits absolute, slug-correct links. Because Read the Docs serves the version-neutral /llms.txt from the default version, llms.txt self-describes the .../en/VERSION_SLUG/ URL scheme so a consumer can reach any version. Build integration and shared infrastructure. * Shared metadata logic (pympistandard loading, C/Fortran binding rendering, VERSION parsing, ".. mpi-bindings:" directive parsing, and build-identity helpers) is factored into docs/ompi_docs_common.py and used by both the new generator and the existing man3 bindings generator. * docs/generate-llm-docs.py runs from a Makefile sentinel for "make" builds and from .readthedocs-pre-create-environment.sh for Read the Docs builds; a conf.py "build-finished" hook copies the staging tree into the Sphinx HTML output, so publication is identical under make and on Read the Docs. Validation. docs/validate-llm-docs.py runs as part of "make check" and verifies JSON Schema conformance of the catalog and manifest, cross-field invariants, that every record links back to the human docs, that the manifest inventories every artifact with matching hash and size, that the generated Markdown contains no unresolved RST, that the committed sample records match the generated catalog, and a determinism (no-diff) rerun at a fixed SOURCE_DATE_EPOCH. Two robustness fixes are included: "make check" now depends on the generation sentinel, so validation actually runs instead of being silently skipped on a clean tree; and a coverage gate fails the build if any MPI Forum procedure that Open MPI implements has no man page (detected from the public C header and the use-mpi-f08 sources), so an implemented MPI API can no longer be silently undocumented. Documentation and specification. The effort is documented for maintainers in docs/developers/llm-friendly-docs.rst -- intent, the llms.txt convention and external references, the design rationale, regeneration and the per-release manifest, and how to update the docs when MPI APIs change. The full design record, JSON Schemas, sample records, and task list live under specs/llms-friendly-docs/. A one-time feature release note is added to the changelog. Signed-off-by: Jeff Squyres <jeff@squyres.com> | 3 个月前 | |
docs: add LLM-friendly documentation artifacts Publish machine-readable, LLM-friendly documentation for the public Open MPI MPI API alongside the existing human-facing HTML and Unix man pages, so that LLMs, retrieval systems, and coding assistants can obtain concise, authoritative, version-correct API information without scraping themed HTML -- and without introducing a second, hand-maintained API reference that could drift from the real documentation. Everything is derived from the existing RST man pages and the MPI Forum binding metadata (docs/mpi-standard-apis.json, read via the embedded pympistandard library); none of the API content is independently authored. Generated artifacts. Written into a build-tree staging directory and carried inside the per-version HTML output tree, so they inherit the existing tarball, install, and Read the Docs publishing machinery: * llms.txt -- a short, self-describing index for each documentation version. * llms/openmpi-mpi-api.jsonl -- one structured JSON record per documented MPI procedure (standard, extension, deprecated, removed). * llms/openmpi-mpi-api.md and four per-interface corpora (C, mpif.h, use mpi, use mpi_f08). * llms/man-openmpi/man3/MPI_<name>.3.md -- one Markdown page per man3 page, 1:1 with the human man pages. * llms/man-openmpi/man1/<command>.1.md -- a Markdown corpus for the section-1 command man pages (mpirun, ompi_info, the wrapper compilers, ...); commands rather than APIs, so Markdown but no JSONL records. * llms/openmpi-docs-manifest.json -- the artifact inventory, and the only artifact that carries build identity. Curated, committed sources under docs/llms-src/: an interface-selection guide, a small examples corpus (reusing the top-level examples/ tree where it already covers a case), and the two JSON Schema files, which serve as both the published contract and the CI validator. Data model and invariants. * Two API populations. Standard procedures are metadata-driven (C, mpif.h / use mpi, and use mpi_f08 signatures, including large-count variants, from pympistandard). Extensions (MPIX_*, OMPI_*) are best-effort: signatures preserved verbatim from the RST, structured fields marked unknown. A "kind" field distinguishes them. * Each artifact's content hash is a pure function of its semantic content: build identity (git_commit, git_describe, generated_at) lives only in the manifest, so a per-artifact hash changes if and only if that artifact's documentation content changes. * Reproducible under the project's existing SOURCE_DATE_EPOCH model: the semantic artifacts are wall-clock-free and byte-identical across reruns at a given commit. As part of this, a pre-existing gap is closed by deriving the docs copyright year from SOURCE_DATE_EPOCH. * Link strategy follows the build type. A local git or release-tarball build emits links relative to each file (a self-contained, offline tree); a Read the Docs build emits absolute, slug-correct links. Because Read the Docs serves the version-neutral /llms.txt from the default version, llms.txt self-describes the .../en/VERSION_SLUG/ URL scheme so a consumer can reach any version. Build integration and shared infrastructure. * Shared metadata logic (pympistandard loading, C/Fortran binding rendering, VERSION parsing, ".. mpi-bindings:" directive parsing, and build-identity helpers) is factored into docs/ompi_docs_common.py and used by both the new generator and the existing man3 bindings generator. * docs/generate-llm-docs.py runs from a Makefile sentinel for "make" builds and from .readthedocs-pre-create-environment.sh for Read the Docs builds; a conf.py "build-finished" hook copies the staging tree into the Sphinx HTML output, so publication is identical under make and on Read the Docs. Validation. docs/validate-llm-docs.py runs as part of "make check" and verifies JSON Schema conformance of the catalog and manifest, cross-field invariants, that every record links back to the human docs, that the manifest inventories every artifact with matching hash and size, that the generated Markdown contains no unresolved RST, that the committed sample records match the generated catalog, and a determinism (no-diff) rerun at a fixed SOURCE_DATE_EPOCH. Two robustness fixes are included: "make check" now depends on the generation sentinel, so validation actually runs instead of being silently skipped on a clean tree; and a coverage gate fails the build if any MPI Forum procedure that Open MPI implements has no man page (detected from the public C header and the use-mpi-f08 sources), so an implemented MPI API can no longer be silently undocumented. Documentation and specification. The effort is documented for maintainers in docs/developers/llm-friendly-docs.rst -- intent, the llms.txt convention and external references, the design rationale, regeneration and the per-release manifest, and how to update the docs when MPI APIs change. The full design record, JSON Schemas, sample records, and task list live under specs/llms-friendly-docs/. A one-time feature release note is added to the changelog. Signed-off-by: Jeff Squyres <jeff@squyres.com> | 3 个月前 | |
docs: add LLM-friendly documentation artifacts Publish machine-readable, LLM-friendly documentation for the public Open MPI MPI API alongside the existing human-facing HTML and Unix man pages, so that LLMs, retrieval systems, and coding assistants can obtain concise, authoritative, version-correct API information without scraping themed HTML -- and without introducing a second, hand-maintained API reference that could drift from the real documentation. Everything is derived from the existing RST man pages and the MPI Forum binding metadata (docs/mpi-standard-apis.json, read via the embedded pympistandard library); none of the API content is independently authored. Generated artifacts. Written into a build-tree staging directory and carried inside the per-version HTML output tree, so they inherit the existing tarball, install, and Read the Docs publishing machinery: * llms.txt -- a short, self-describing index for each documentation version. * llms/openmpi-mpi-api.jsonl -- one structured JSON record per documented MPI procedure (standard, extension, deprecated, removed). * llms/openmpi-mpi-api.md and four per-interface corpora (C, mpif.h, use mpi, use mpi_f08). * llms/man-openmpi/man3/MPI_<name>.3.md -- one Markdown page per man3 page, 1:1 with the human man pages. * llms/man-openmpi/man1/<command>.1.md -- a Markdown corpus for the section-1 command man pages (mpirun, ompi_info, the wrapper compilers, ...); commands rather than APIs, so Markdown but no JSONL records. * llms/openmpi-docs-manifest.json -- the artifact inventory, and the only artifact that carries build identity. Curated, committed sources under docs/llms-src/: an interface-selection guide, a small examples corpus (reusing the top-level examples/ tree where it already covers a case), and the two JSON Schema files, which serve as both the published contract and the CI validator. Data model and invariants. * Two API populations. Standard procedures are metadata-driven (C, mpif.h / use mpi, and use mpi_f08 signatures, including large-count variants, from pympistandard). Extensions (MPIX_*, OMPI_*) are best-effort: signatures preserved verbatim from the RST, structured fields marked unknown. A "kind" field distinguishes them. * Each artifact's content hash is a pure function of its semantic content: build identity (git_commit, git_describe, generated_at) lives only in the manifest, so a per-artifact hash changes if and only if that artifact's documentation content changes. * Reproducible under the project's existing SOURCE_DATE_EPOCH model: the semantic artifacts are wall-clock-free and byte-identical across reruns at a given commit. As part of this, a pre-existing gap is closed by deriving the docs copyright year from SOURCE_DATE_EPOCH. * Link strategy follows the build type. A local git or release-tarball build emits links relative to each file (a self-contained, offline tree); a Read the Docs build emits absolute, slug-correct links. Because Read the Docs serves the version-neutral /llms.txt from the default version, llms.txt self-describes the .../en/VERSION_SLUG/ URL scheme so a consumer can reach any version. Build integration and shared infrastructure. * Shared metadata logic (pympistandard loading, C/Fortran binding rendering, VERSION parsing, ".. mpi-bindings:" directive parsing, and build-identity helpers) is factored into docs/ompi_docs_common.py and used by both the new generator and the existing man3 bindings generator. * docs/generate-llm-docs.py runs from a Makefile sentinel for "make" builds and from .readthedocs-pre-create-environment.sh for Read the Docs builds; a conf.py "build-finished" hook copies the staging tree into the Sphinx HTML output, so publication is identical under make and on Read the Docs. Validation. docs/validate-llm-docs.py runs as part of "make check" and verifies JSON Schema conformance of the catalog and manifest, cross-field invariants, that every record links back to the human docs, that the manifest inventories every artifact with matching hash and size, that the generated Markdown contains no unresolved RST, that the committed sample records match the generated catalog, and a determinism (no-diff) rerun at a fixed SOURCE_DATE_EPOCH. Two robustness fixes are included: "make check" now depends on the generation sentinel, so validation actually runs instead of being silently skipped on a clean tree; and a coverage gate fails the build if any MPI Forum procedure that Open MPI implements has no man page (detected from the public C header and the use-mpi-f08 sources), so an implemented MPI API can no longer be silently undocumented. Documentation and specification. The effort is documented for maintainers in docs/developers/llm-friendly-docs.rst -- intent, the llms.txt convention and external references, the design rationale, regeneration and the per-release manifest, and how to update the docs when MPI APIs change. The full design record, JSON Schemas, sample records, and task list live under specs/llms-friendly-docs/. A one-time feature release note is added to the changelog. Signed-off-by: Jeff Squyres <jeff@squyres.com> | 3 个月前 | |
docs: add LLM-friendly documentation artifacts Publish machine-readable, LLM-friendly documentation for the public Open MPI MPI API alongside the existing human-facing HTML and Unix man pages, so that LLMs, retrieval systems, and coding assistants can obtain concise, authoritative, version-correct API information without scraping themed HTML -- and without introducing a second, hand-maintained API reference that could drift from the real documentation. Everything is derived from the existing RST man pages and the MPI Forum binding metadata (docs/mpi-standard-apis.json, read via the embedded pympistandard library); none of the API content is independently authored. Generated artifacts. Written into a build-tree staging directory and carried inside the per-version HTML output tree, so they inherit the existing tarball, install, and Read the Docs publishing machinery: * llms.txt -- a short, self-describing index for each documentation version. * llms/openmpi-mpi-api.jsonl -- one structured JSON record per documented MPI procedure (standard, extension, deprecated, removed). * llms/openmpi-mpi-api.md and four per-interface corpora (C, mpif.h, use mpi, use mpi_f08). * llms/man-openmpi/man3/MPI_<name>.3.md -- one Markdown page per man3 page, 1:1 with the human man pages. * llms/man-openmpi/man1/<command>.1.md -- a Markdown corpus for the section-1 command man pages (mpirun, ompi_info, the wrapper compilers, ...); commands rather than APIs, so Markdown but no JSONL records. * llms/openmpi-docs-manifest.json -- the artifact inventory, and the only artifact that carries build identity. Curated, committed sources under docs/llms-src/: an interface-selection guide, a small examples corpus (reusing the top-level examples/ tree where it already covers a case), and the two JSON Schema files, which serve as both the published contract and the CI validator. Data model and invariants. * Two API populations. Standard procedures are metadata-driven (C, mpif.h / use mpi, and use mpi_f08 signatures, including large-count variants, from pympistandard). Extensions (MPIX_*, OMPI_*) are best-effort: signatures preserved verbatim from the RST, structured fields marked unknown. A "kind" field distinguishes them. * Each artifact's content hash is a pure function of its semantic content: build identity (git_commit, git_describe, generated_at) lives only in the manifest, so a per-artifact hash changes if and only if that artifact's documentation content changes. * Reproducible under the project's existing SOURCE_DATE_EPOCH model: the semantic artifacts are wall-clock-free and byte-identical across reruns at a given commit. As part of this, a pre-existing gap is closed by deriving the docs copyright year from SOURCE_DATE_EPOCH. * Link strategy follows the build type. A local git or release-tarball build emits links relative to each file (a self-contained, offline tree); a Read the Docs build emits absolute, slug-correct links. Because Read the Docs serves the version-neutral /llms.txt from the default version, llms.txt self-describes the .../en/VERSION_SLUG/ URL scheme so a consumer can reach any version. Build integration and shared infrastructure. * Shared metadata logic (pympistandard loading, C/Fortran binding rendering, VERSION parsing, ".. mpi-bindings:" directive parsing, and build-identity helpers) is factored into docs/ompi_docs_common.py and used by both the new generator and the existing man3 bindings generator. * docs/generate-llm-docs.py runs from a Makefile sentinel for "make" builds and from .readthedocs-pre-create-environment.sh for Read the Docs builds; a conf.py "build-finished" hook copies the staging tree into the Sphinx HTML output, so publication is identical under make and on Read the Docs. Validation. docs/validate-llm-docs.py runs as part of "make check" and verifies JSON Schema conformance of the catalog and manifest, cross-field invariants, that every record links back to the human docs, that the manifest inventories every artifact with matching hash and size, that the generated Markdown contains no unresolved RST, that the committed sample records match the generated catalog, and a determinism (no-diff) rerun at a fixed SOURCE_DATE_EPOCH. Two robustness fixes are included: "make check" now depends on the generation sentinel, so validation actually runs instead of being silently skipped on a clean tree; and a coverage gate fails the build if any MPI Forum procedure that Open MPI implements has no man page (detected from the public C header and the use-mpi-f08 sources), so an implemented MPI API can no longer be silently undocumented. Documentation and specification. The effort is documented for maintainers in docs/developers/llm-friendly-docs.rst -- intent, the llms.txt convention and external references, the design rationale, regeneration and the per-release manifest, and how to update the docs when MPI APIs change. The full design record, JSON Schemas, sample records, and task list live under specs/llms-friendly-docs/. A one-time feature release note is added to the changelog. Signed-off-by: Jeff Squyres <jeff@squyres.com> | 3 个月前 | |
docs: add LLM-friendly documentation artifacts Publish machine-readable, LLM-friendly documentation for the public Open MPI MPI API alongside the existing human-facing HTML and Unix man pages, so that LLMs, retrieval systems, and coding assistants can obtain concise, authoritative, version-correct API information without scraping themed HTML -- and without introducing a second, hand-maintained API reference that could drift from the real documentation. Everything is derived from the existing RST man pages and the MPI Forum binding metadata (docs/mpi-standard-apis.json, read via the embedded pympistandard library); none of the API content is independently authored. Generated artifacts. Written into a build-tree staging directory and carried inside the per-version HTML output tree, so they inherit the existing tarball, install, and Read the Docs publishing machinery: * llms.txt -- a short, self-describing index for each documentation version. * llms/openmpi-mpi-api.jsonl -- one structured JSON record per documented MPI procedure (standard, extension, deprecated, removed). * llms/openmpi-mpi-api.md and four per-interface corpora (C, mpif.h, use mpi, use mpi_f08). * llms/man-openmpi/man3/MPI_<name>.3.md -- one Markdown page per man3 page, 1:1 with the human man pages. * llms/man-openmpi/man1/<command>.1.md -- a Markdown corpus for the section-1 command man pages (mpirun, ompi_info, the wrapper compilers, ...); commands rather than APIs, so Markdown but no JSONL records. * llms/openmpi-docs-manifest.json -- the artifact inventory, and the only artifact that carries build identity. Curated, committed sources under docs/llms-src/: an interface-selection guide, a small examples corpus (reusing the top-level examples/ tree where it already covers a case), and the two JSON Schema files, which serve as both the published contract and the CI validator. Data model and invariants. * Two API populations. Standard procedures are metadata-driven (C, mpif.h / use mpi, and use mpi_f08 signatures, including large-count variants, from pympistandard). Extensions (MPIX_*, OMPI_*) are best-effort: signatures preserved verbatim from the RST, structured fields marked unknown. A "kind" field distinguishes them. * Each artifact's content hash is a pure function of its semantic content: build identity (git_commit, git_describe, generated_at) lives only in the manifest, so a per-artifact hash changes if and only if that artifact's documentation content changes. * Reproducible under the project's existing SOURCE_DATE_EPOCH model: the semantic artifacts are wall-clock-free and byte-identical across reruns at a given commit. As part of this, a pre-existing gap is closed by deriving the docs copyright year from SOURCE_DATE_EPOCH. * Link strategy follows the build type. A local git or release-tarball build emits links relative to each file (a self-contained, offline tree); a Read the Docs build emits absolute, slug-correct links. Because Read the Docs serves the version-neutral /llms.txt from the default version, llms.txt self-describes the .../en/VERSION_SLUG/ URL scheme so a consumer can reach any version. Build integration and shared infrastructure. * Shared metadata logic (pympistandard loading, C/Fortran binding rendering, VERSION parsing, ".. mpi-bindings:" directive parsing, and build-identity helpers) is factored into docs/ompi_docs_common.py and used by both the new generator and the existing man3 bindings generator. * docs/generate-llm-docs.py runs from a Makefile sentinel for "make" builds and from .readthedocs-pre-create-environment.sh for Read the Docs builds; a conf.py "build-finished" hook copies the staging tree into the Sphinx HTML output, so publication is identical under make and on Read the Docs. Validation. docs/validate-llm-docs.py runs as part of "make check" and verifies JSON Schema conformance of the catalog and manifest, cross-field invariants, that every record links back to the human docs, that the manifest inventories every artifact with matching hash and size, that the generated Markdown contains no unresolved RST, that the committed sample records match the generated catalog, and a determinism (no-diff) rerun at a fixed SOURCE_DATE_EPOCH. Two robustness fixes are included: "make check" now depends on the generation sentinel, so validation actually runs instead of being silently skipped on a clean tree; and a coverage gate fails the build if any MPI Forum procedure that Open MPI implements has no man page (detected from the public C header and the use-mpi-f08 sources), so an implemented MPI API can no longer be silently undocumented. Documentation and specification. The effort is documented for maintainers in docs/developers/llm-friendly-docs.rst -- intent, the llms.txt convention and external references, the design rationale, regeneration and the per-release manifest, and how to update the docs when MPI APIs change. The full design record, JSON Schemas, sample records, and task list live under specs/llms-friendly-docs/. A one-time feature release note is added to the changelog. Signed-off-by: Jeff Squyres <jeff@squyres.com> | 3 个月前 |