Update Python detection
GROMACS CMake Python detection affects documentation builds, validation testing, in-build-tree configuration of the gmxapi Python package, and in-build-tree configuration of the sample_restraint sample MD plugin code. These consumers have different requirements of a found Python installation.
In the case of the Python packages with C++ extension code, Python libraries and headers are also needed that match the Python interpreter. Prior to CMake 3.12, the CMake tools for this are fairly poor. Starting in 3.12, the old
find_package(PythonInterp) is deprecated and the new FindPython3 tools are incompatible, but far superior.
1. Unify Python detection in GROMACS CMake configuration to avoid confusing conflicts.
2. Migrate away from FindPythonInterp for future compatibility.
3. Improve robustness of Python development environment discovery.
CMake 3.12 was released mid-2018, so it will be a while before GROMACS requires it. It would be a shame to pass up on the more robust Python detection where available, though. It is particularly helpful for people trying to build gmxapi in environments with multiple Python installations where some have the development files and others do not (as is common with module systems on HPC clusters). Maybe we should back-port the FindPythonInterp from the CMake project. Alternatively, the pybind11 library detection is more robust than CMake < 3.12, and compatible enough that we could use it instead, but we would need to add_subdirectory(...pybind) before doing other Python detection.
This task may span multiple GROMACS release cycles, but will have immediately relevant changes.
Python detection consolidation.
- Move Python discovery up one level to the project root.
- Remove a TODO and clean up comments for validation test.
- Allow Sphinx discovery to appear in CMake output on first config.
Fix hwloc and sphinx detection
These were too noisy and were not implemented efficiently (e.g. not
caching the result of execute process). If an environment change is
needed to detect something (like loading a module) then we expect
developers to be able to unset a cache variable (or just regenerate a
Use CMake 3.12+ facilities for detecting Python.
Consolidate Python detection in the main CMakeLists.txt before the
first usage in a subdirectory. Use FindPython3.cmake instead of
FindPythonInterp.cmake. Populate the legacy PYTHON_EXECUTABLE variable
#1 Updated by Eric Irrgang over 1 year ago
One transitional point to consider: The hinting for FindPythonInterp and FindPython3 is changing. Though PYTHON_EXECUTABLE was never documented as a hint to FindPythonInterp, it is common to use. We could explicitly discourage its use, document an alternative GMX variable, avoid the issue, or embrace and document the new style of hinting, which prefers the naming of a python installation root rather than an executable location. Of those, I think it is a bad idea to have a GROMACS-specific way of hinting a well-known CMake facility. I think we should probably just use pybind's detection in the near term. However, pybind's detection might evolve out of synch with what GROMACS wants to expose to its users, in which case back-porting the CMake 3.12+ detection would seem preferable.
#2 Updated by Eric Irrgang over 1 year ago
- Status changed from New to In Progress
Minimum requirements for GROMACS 2020 have been met. Some additional documentation may be helpful if we do not intend to further modernize CMake Python detection and whether or not we replace our use of CMake's Python finding stuff with that of pybind.
Note 1: pybind's FindPythonLibsNew.cmake is more robust than CMake's, and since we have pybind embedded in the gmxapi Python package, we may prefer to make pybind more global in the project and load it earlier.
Note 2: It seems that
CMAKE_PREFIX_PATH may be the most robust way to hint the Python installation to use in the face of potential weirdness with virtual environments, Pyenv, and the like. For example,
-DCMAKE_PREFIX_PATH=$HOME/.pyenv/versions/gmx-py35/ works better than specifying PYTHON_EXECUTABLE and/or SPHINX_EXECUTABLE, which are common but non-normative workarounds.