Task #2998

Update Python detection

Added by Eric Irrgang over 1 year ago. Updated 9 months ago.

build system


GROMACS CMake Python detection affects documentation builds, validation testing, in-build-tree configuration of the gmxapi Python package, and in-build-tree configuration of the sample_restraint sample MD plugin code. These consumers have different requirements of a found Python installation.

In the case of the Python packages with C++ extension code, Python libraries and headers are also needed that match the Python interpreter. Prior to CMake 3.12, the CMake tools for this are fairly poor. Starting in 3.12, the old find_package(PythonInterp) is deprecated and the new FindPython3 tools are incompatible, but far superior.


1. Unify Python detection in GROMACS CMake configuration to avoid confusing conflicts.
2. Migrate away from FindPythonInterp for future compatibility.
3. Improve robustness of Python development environment discovery.

CMake 3.12 was released mid-2018, so it will be a while before GROMACS requires it. It would be a shame to pass up on the more robust Python detection where available, though. It is particularly helpful for people trying to build gmxapi in environments with multiple Python installations where some have the development files and others do not (as is common with module systems on HPC clusters). Maybe we should back-port the FindPythonInterp from the CMake project. Alternatively, the pybind11 library detection is more robust than CMake < 3.12, and compatible enough that we could use it instead, but we would need to add_subdirectory(...pybind) before doing other Python detection.


This task may span multiple GROMACS release cycles, but will have immediately relevant changes.

Associated revisions

Revision e2a13eb2 (diff)
Added by Eric Irrgang over 1 year ago

Python detection consolidation.

  • Move Python discovery up one level to the project root.
  • Remove a TODO and clean up comments for validation test.
  • Allow Sphinx discovery to appear in CMake output on first config.

Refs #2998

Change-Id: I16a406b2f21c517042729f3d970e0b5a79d40f33

Revision 592b6c20 (diff)
Added by Mark Abraham over 1 year ago

Fix hwloc and sphinx detection

These were too noisy and were not implemented efficiently (e.g. not
caching the result of execute process). If an environment change is
needed to detect something (like loading a module) then we expect
developers to be able to unset a cache variable (or just regenerate a
new cache).

Refs #3024, #3011, #2998

Change-Id: I84dfb03856b9f900fb21004bc676e4ff2647a4b4

Revision bb2dd483 (diff)
Added by M. Eric Irrgang 9 months ago

Use CMake 3.12+ facilities for detecting Python.

Consolidate Python detection in the main CMakeLists.txt before the
first usage in a subdirectory. Use FindPython3.cmake instead of
FindPythonInterp.cmake. Populate the legacy PYTHON_EXECUTABLE variable
for compatibility.

Fixes #2998


#1 Updated by Eric Irrgang over 1 year ago

One transitional point to consider: The hinting for FindPythonInterp and FindPython3 is changing. Though PYTHON_EXECUTABLE was never documented as a hint to FindPythonInterp, it is common to use. We could explicitly discourage its use, document an alternative GMX variable, avoid the issue, or embrace and document the new style of hinting, which prefers the naming of a python installation root rather than an executable location. Of those, I think it is a bad idea to have a GROMACS-specific way of hinting a well-known CMake facility. I think we should probably just use pybind's detection in the near term. However, pybind's detection might evolve out of synch with what GROMACS wants to expose to its users, in which case back-porting the CMake 3.12+ detection would seem preferable.

#2 Updated by Eric Irrgang over 1 year ago

  • Status changed from New to In Progress

Minimum requirements for GROMACS 2020 have been met. Some additional documentation may be helpful if we do not intend to further modernize CMake Python detection and whether or not we replace our use of CMake's Python finding stuff with that of pybind.

Note 1: pybind's FindPythonLibsNew.cmake is more robust than CMake's, and since we have pybind embedded in the gmxapi Python package, we may prefer to make pybind more global in the project and load it earlier.

Note 2: It seems that CMAKE_PREFIX_PATH may be the most robust way to hint the Python installation to use in the face of potential weirdness with virtual environments, Pyenv, and the like. For example, -DCMAKE_PREFIX_PATH=$HOME/.pyenv/versions/gmx-py35/ works better than specifying PYTHON_EXECUTABLE and/or SPHINX_EXECUTABLE, which are common but non-normative workarounds.

#3 Updated by Mark Abraham over 1 year ago

  • Target version changed from 2020-infrastructure-stable to 2020-infrastructure-update-post-beta1

#4 Updated by Paul Bauer about 1 year ago

  • Target version changed from 2020-infrastructure-update-post-beta1 to 2020-beta3

not going to happen before beta3

#5 Updated by Eric Irrgang about 1 year ago

Do we want to take any more action on this before 2020 release or should we update the target to 2021?

#6 Updated by Paul Bauer about 1 year ago

  • Target version changed from 2020-beta3 to 2021-infrastructure-stable

I'll bump this now to 2021 if no one is going to work on this for 2020

#7 Updated by Mark Abraham about 1 year ago

If there's something broken, then we can treat it as a bug and fix it. But the long term aspects are best considered after the release.

#8 Updated by Anonymous 9 months ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF