Feature #3288

Task #2045: API design and language bindings

Feature #3152: Infrastructure and patterns for expressing public interfaces

Use build system infrastructure instead of custom scripts to manage API levels.

Added by Eric Irrgang about 1 year ago. Updated 11 months ago.

build system


GROMACS currently uses source validation scripts and extra macros in various build system tools to reconcile documentation organization and visibility, and internal header usage in terms of declared API levels.

The heuristics are not particularly easy to resolve during development and are not enforced at the most useful times.

This is a proposal to use a combination of repository layout and basic CMake functionality to improve the clarity, maintainability, and rigor of API / header scoping. As a result, we will need less custom infrastructure (source tree checker scripts).

Side benefits include decoupling from Doxygen in several ways that should make it easier to update our Doxygen version requirements in the future.

Proposal: Define API level and dependency policies via CMake

Module dependencies are explicit in the CMakeLists.txt. Dependence on an INTERFACE target allows resolution of headers in the installed API. Dependence on the PUBLIC target allows includes to resolve headers for the build-tree / library-internal module public API.

This necessarily means that the sources and headers of all API levels cannot stay mixed together the way they are. The ability to #include "gromacs/moduleA_installed_header.h" should not convey the ability to #include "gromacs/moduleB_detail.h" at build time. moduleB_detail.h would only be available as moduleB_detail.h to sources in the moduleB target. moduleB_public.h would only be available to moduleA as gromacs/moduleB_public.h by depending on moduleA PUBLIC in CMakeLists.txt.

src/api/cpp and src/api/cpp/include/gmxapi demonstrate how this can work. However, this confuses the current include sorter script, and the headers are accidentally exposed because all of the modules in src/gromacs/moduleX have src/ in their include path. The five headers in src/ could easily be moved to a subdirectory.

Module API level:

Option 1: API level is determined at the scope of a module. A module's public interface is either in the installed headers / public API or the module is available in the build-tree only. Doxygen visibility directives are not necessary, because the modules in the two APIs are curated.

Option 1.1: The installed API is provided by one (or a few) module(s) explicitly as a distinct API and C++ namespace.

Option 2: Each module may have installed headers and library-internal headers. Two completely separate sets of API documentation are generated, or at least C++ namespace should help alert library API doc viewers / module maintainers when a symbol is actually part of the installed API.

Documentation for API levels becomes scoped by file and location. The list(s) of installed headers (necessary for the CMake target INSTALL command) is the complete list of files for Doxygen to process for the public/installed API. The library-internal / modules API can be documented largely as before, but should link to the public API documentation for entities that are no longer in scope.

Distinguish between API documentation scope and API documentation verbosity.

  • It may be appropriate or unavoidable in some cases for public API headers to include notes, details, or references beyond the scope of the public API (such as a developer note explaining the need for a forward reference from the library API, or a necessary detail that is explicitly excluded from the compatibility guarantee of the public API specification). Such details can be hidden (e.g. @\internal) to be excluded from the standard public API documentation, but revealed in developer builds of the public API documentation. Ref #3402.
  • The library-internal API may similarly have standard documentation for normative use cases, plus internal implementation details or other more verbose documentation revealed by a more verbose documentation build target. Public API documentation accessible from the library-internal API documentation should have multiple visual reminders of the API distinction.
  • When browsing the library documentation, it should be obvious to the developer of new library code whether a dependency they are considering is in the public API or library-internal API, and easy to identify module details that are not part of the library-internal API. Doxygen doesn't have great mechanisms for identifying which content was rendered conditionally (\cond or \internal), so it may be appropriate to use a clearly distinct documentation scope for module-internal docs, so that the scope of the library-internal API is not obscured when browsing non-public module details.
    The easiest way to make these distinctions is to use distinct Doxygen groups for different API levels. In other words, if we want to continue to define "modules" such that a module expresses in multiple documented API levels, documentation clarity and maintainability can be improved by using multiple Doxygen groups per module instead of one group per module.
  • Should the public API documentation link to library API documentation or provide separate local docs for e.g. forward references or types / functionality that is opaque at that API level?

Further Questions

Is there a preference for whether library or module interface headers are necessarily nested in single subdirectory trees versus separated into parallel directory trees? E.g. $PROJECT_SOURCE_DIR/src/gromacs/module/private.h plus $PROJECT_SOURCE_DIR/src/gromacs/module/include/gromacs/module/private.h, versus $PROJECT_SOURCE_DIR/include/gromacs/module plus $PROJECT_SOURCE_DIR/src/gromacs/module.

A handful of "convenience headers" in src/gromacs aggregate "include"s of module headers, providing multiple ways to access the same API without a clear indication of what is normative. If we want module APIs to be expressed in gromacs/module.h headers, or to distinguish between gromacs/module.h and gromacs/module/feature.h, we should clarify the meaning of these paths. Is there a compelling reason not to just get rid of these headers?

Is more necessary to decouple from specific Doxygen versions (constraints on Doxygen version?)

Other checks to reimplement

Can gmxpre.h go away?

Check for a precompiler define instead of #include "gmxpre.h", or replace "gmxpre.h" with a preprocessor define provided by CMake?

sort includes with clang-format instead of

Use Doxygen directly instead of script acting on Doxygen XML for dependency graphs.

reconsider ${PROJECT_BINARY_DIR}/src/gromacs/installed-headers.txt

Document doxygen groups and directories in module headers instead of misc.cpp and directories.cpp

Consider freezing API and implementation versions.

Use monotonically versioned implementation namespaces (for (a) supporting multiple API versions such as file format revisions, and (b) clarifying API versions at compile time without worrying about API compatibility guarantees or versioning API. Very limited use may be allowed for (c) feature development or API migration across multiple commits, with tracking and approval of the release manager)

Associated revisions

Revision 39591239 (diff)
Added by Eric Irrgang 6 months ago

Begin moving installed interface to `api/`

Create `api/` directory outside of `src/` to decouple the public
interface from the internal and build tree infrastructure.

Move the `gmxapi` installed headers to `api/include/gmxapi`.
The public header maintenance (once managed with custom CMake functions)
is now reduced to filesystem layout and `install` commands in

Deferred to future changes:
  • Split public and developer doxygen builds.
  • Move remaining libgmxapi public interface details to `api/gmxapi`
  • Re-expose essential public facilities from libgromacs to allow
    (re-)deprecation of ENABLE_LEGACY_API.

Refs #3288

Revision 39132a7e (diff)
Added by Eric Irrgang 5 months ago

Allow more granularity for public interfaces in build tree.

Move the gmxapi installed headers from api/include/gmxapi to
api/gmxapi/include/gmxapi, making room for api/nblib/include installed
headers or others. This allows CMake targets in the build tree to
restrict access to public headers as well as private headers such that
the `gmxapi` target allows `#include "gmxapi/someheader.h"` but does not
inadvertently allow `#include "nblib/anotherheader.h"` without an
explicit `target_link_libraries`.

Fix some disordered header inclusions.

Refs #3288

Revision 09dd1e79 (diff)
Added by M. Eric Irrgang 3 months ago

Establish `api/` as the home for installed headers.

Prepare to remove `src/` from global build tree include path.
  • Create `api/legacy` for the new `legacy_api` target.
  • Move gmxapi and nblib public header dependencies to `api/legacy`.

This change is orthogonal to the clarification of the public API.
The purpose is to normalize the way we manage installed headers.
gmxapi and nblib already provide their public interfaces through files
and CMake targets in `api/`. For the remaining installed libgromacs
headers, the `legacy_api` target provides a transitional home as the
modules are individually updated in terms of CMake machinery and as the
legacy API functionality is reexamined.

Refs #3288

Revision 9c71edb2 (diff)
Added by M. Eric Irrgang 2 months ago

Provide common library headers through new CMake target.

Prepare to remove `src/` from global build tree include path.
Create `src/include` and `common` target for library-level headers.

Ref #3288

Revision 312e89c3 (diff)
Added by M. Eric Irrgang 2 months ago

Provide a catch-all target for library modules.

Prepare to remove `src/` from global build tree include path.
Existing module headers are accessible through the new `legacy_modules`

Ref #3288

Revision f310554a (diff)
Added by Eric Irrgang 2 months ago

Be more explicit about build-tree-only dependencies.

Use the BUILD_INTERFACE CMake generator expression to improve
self-documentation and to prevent build-tree-only dependencies from
propagating to exported targets. When libgromacs is a STATIC target,
even private dependencies can affect the exported target through
transitive usage requirements (e.g. compiler/linker flags). However,
we currently only use the `common` and `legacy_modules` targets for
build time header paths.

Ref #3288

Revision 9f8e11f3 (diff)
Added by M. Eric Irrgang 2 months ago

Separate the testutils module.

Create `testutils` target and separate public/private headers.
Remove `src/` from global build tree include path.

Apply some tidying demanded by clang-tidy.

Ref #3288

Revision 5d1f7c83 (diff)
Added by M. Eric Irrgang 2 months ago

Create CMake targets for each of the modules.

Establish a template for further work towards using
CMake to manage module dependencies through targets
and the include directories they provide.

There should be no significant behavioral differences.
Actual relocation of the headers is deferred.

Refs #3288

Revision 49aa0700 (diff)
Added by Paul Bauer 2 months ago

Fix missing copyright changes

Was overlooked during previous changes.

Refs #3288


#1 Updated by Eric Irrgang about 1 year ago

  • Description updated (diff)

#2 Updated by Eric Irrgang about 1 year ago

  • Description updated (diff)

#3 Updated by Eric Irrgang about 1 year ago

  • Description updated (diff)

#4 Updated by Eric Irrgang 11 months ago

  • Description updated (diff)
  • Private changed from Yes to No

Also available in: Atom PDF