Task #2998

Update Python detection

Added by Eric Irrgang 4 months ago. Updated about 2 months ago.

In Progress
build system


GROMACS CMake Python detection affects documentation builds, validation testing, in-build-tree configuration of the gmxapi Python package, and in-build-tree configuration of the sample_restraint sample MD plugin code. These consumers have different requirements of a found Python installation.

In the case of the Python packages with C++ extension code, Python libraries and headers are also needed that match the Python interpreter. Prior to CMake 3.12, the CMake tools for this are fairly poor. Starting in 3.12, the old find_package(PythonInterp) is deprecated and the new FindPython3 tools are incompatible, but far superior.


1. Unify Python detection in GROMACS CMake configuration to avoid confusing conflicts.
2. Migrate away from FindPythonInterp for future compatibility.
3. Improve robustness of Python development environment discovery.

CMake 3.12 was released mid-2018, so it will be a while before GROMACS requires it. It would be a shame to pass up on the more robust Python detection where available, though. It is particularly helpful for people trying to build gmxapi in environments with multiple Python installations where some have the development files and others do not (as is common with module systems on HPC clusters). Maybe we should back-port the FindPythonInterp from the CMake project. Alternatively, the pybind11 library detection is more robust than CMake < 3.12, and compatible enough that we could use it instead, but we would need to add_subdirectory(...pybind) before doing other Python detection.


This task may span multiple GROMACS release cycles, but will have immediately relevant changes.

Associated revisions

Revision e2a13eb2 (diff)
Added by Eric Irrgang 3 months ago

Python detection consolidation.

  • Move Python discovery up one level to the project root.
  • Remove a TODO and clean up comments for validation test.
  • Allow Sphinx discovery to appear in CMake output on first config.

Refs #2998

Change-Id: I16a406b2f21c517042729f3d970e0b5a79d40f33

Revision 592b6c20 (diff)
Added by Mark Abraham 3 months ago

Fix hwloc and sphinx detection

These were too noisy and were not implemented efficiently (e.g. not
caching the result of execute process). If an environment change is
needed to detect something (like loading a module) then we expect
developers to be able to unset a cache variable (or just regenerate a
new cache).

Refs #3024, #3011, #2998

Change-Id: I84dfb03856b9f900fb21004bc676e4ff2647a4b4


#1 Updated by Eric Irrgang 4 months ago

One transitional point to consider: The hinting for FindPythonInterp and FindPython3 is changing. Though PYTHON_EXECUTABLE was never documented as a hint to FindPythonInterp, it is common to use. We could explicitly discourage its use, document an alternative GMX variable, avoid the issue, or embrace and document the new style of hinting, which prefers the naming of a python installation root rather than an executable location. Of those, I think it is a bad idea to have a GROMACS-specific way of hinting a well-known CMake facility. I think we should probably just use pybind's detection in the near term. However, pybind's detection might evolve out of synch with what GROMACS wants to expose to its users, in which case back-porting the CMake 3.12+ detection would seem preferable.

#2 Updated by Eric Irrgang 2 months ago

  • Status changed from New to In Progress

Minimum requirements for GROMACS 2020 have been met. Some additional documentation may be helpful if we do not intend to further modernize CMake Python detection and whether or not we replace our use of CMake's Python finding stuff with that of pybind.

Note 1: pybind's FindPythonLibsNew.cmake is more robust than CMake's, and since we have pybind embedded in the gmxapi Python package, we may prefer to make pybind more global in the project and load it earlier.

Note 2: It seems that CMAKE_PREFIX_PATH may be the most robust way to hint the Python installation to use in the face of potential weirdness with virtual environments, Pyenv, and the like. For example, -DCMAKE_PREFIX_PATH=$HOME/.pyenv/versions/gmx-py35/ works better than specifying PYTHON_EXECUTABLE and/or SPHINX_EXECUTABLE, which are common but non-normative workarounds.

#3 Updated by Mark Abraham about 2 months ago

  • Target version changed from 2020-infrastructure-stable to 2020-infrastructure-update-post-beta1

Also available in: Atom PDF