Project

General

Profile

Feature #2961

Task #2045: API design and language bindings

Feature #2896: Python packaging

How should Python package find GROMACS resources under various circumstances?

Added by Eric Irrgang 6 months ago. Updated 2 days ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
Difficulty:
uncategorized
Close

Description

Under different scenarios (see parent issue) during or after configuration, build, and installation, a Python package built from the GROMACS repo needs to know where to find GROMACS resources.

This issue does not yet enumerate all scenarios.

Client code (including the Python package, when build with Python packaging tools) should use the same C++ standard library when processing public headers. It could be helpful to write a CMakeToolchain.txt file with the GROMACS installation so that a dependent CMake configuration can be launched with effective hints before find_package() is run.

Early implementations need to know where the gmx wrapper binary is. We could configure a gmxrc.py module at the same time GMXRC files are configured. If we want to run tests from the build tree before installation, though, we need an alternative. Another option would be to attach properties to a CMake target that are different for the build tree target or an imported target.

If there are any other data that need to be provided to libgromacs through the posix environment, like GMXDATA, then that also needs to be addressed.


Related issues

Related to GROMACS - Bug #3111: sample_restraint testing should not download filesClosed
Related to GROMACS - Task #3133: Cookiecutter for sample_restraintNew
Related to GROMACS - Feature #3177: Spack package management supportNew

Associated revisions

Revision 6ec0f1f1 (diff)
Added by Eric Irrgang 4 months ago

Allow gmxapi Python package tests to be run from GROMACS build tree.

  • Configure gmxapi._gmxapi to build with the ability to find libgmxapi.
  • Add a gmxapi_pytest custom CMake target to invoke pytest.

Refs #2961
Refs #2756

Change-Id: I64ca82afbf37da3c37759ec302c2ac9cc1666de2

Revision dcc396af (diff)
Added by Eric Irrgang 3 months ago

Make gmxcli pytest fixture more robust.

Look for multiple possible `gmx` command line executables.

This change may be followed up, amended, replaced, or superseded by
change Icefe89e97009110be55dc8e1f3db5726ec1fe53a but is necessary
now to support child changes under review.

Refs #2961

Change-Id: I7d9d830c1b16fbd2af86aded3f1fea5f3a63307f

Revision 1913e5be (diff)
Added by Eric Irrgang 3 months ago

Make gmxcli pytest fixture more robust.

Look for multiple possible `gmx` command line executables.

This change may be followed up, amended, replaced, or superseded by
change Icefe89e97009110be55dc8e1f3db5726ec1fe53a but is necessary
now to support child changes under review.

Refs #2961

Change-Id: I7d9d830c1b16fbd2af86aded3f1fea5f3a63307f

Revision 5e5da871 (diff)
Added by Paul Bauer 3 months ago

Basic toolchain to facilitate GMXAPI build

Adds a basic toolchain file and install directives for it.
This is aimed at having one point for the API to look up how
a native GROMACS installation was build and allow the bindings for the
API to be build in the same way.

Added hint for directory to GMXRC so that it will be easier to find.

Refs #2961

Change-Id: Icefe89e97009110be55dc8e1f3db5726ec1fe53a

Revision c27d7f42 (diff)
Added by Eric Irrgang 3 months ago

Use GROMACS CMake toolchain file for gmxapi Python package build.

Refs #2961

Change-Id: If085425879f00a8a8573627dbec197afa39c2ab0

Revision 3ef3736c (diff)
Added by Eric Irrgang about 1 month ago

Restore CMake convenience targets for sample_restraint.

When building GROMACS with GMX_PYTHON_PACKAGE=ON, the Python and C++
tests in python_packaging/sample_restraint are configured for use in
the GROMACS build tree. Let the tests in
python_packaging/sample_restraint use input data from
src/testutils/simulationdatabase.

Refs #2961

Change-Id: Ida104dc224e7a6af3874bd0604439033fe4f72a1

History

#1 Updated by Eric Irrgang 6 months ago

  • Description updated (diff)

#2 Updated by Eric Irrgang 2 months ago

For issues such as helping users to build mpi4py, how should we determine whether GROMACS was built with MPI or not? The CMake toolchain file that is now generated sets the compiler to the MPI compiler wrapper, if appropriate, but there is no explicit annotation in this or other CMake support files, as far as I can see. Is there a CMake test we should run on the compiler in python_package/src/CMakeLists.txt? Should we add a CMake variable to one of the exported helper listings?

#3 Updated by Eric Irrgang 2 months ago

The gmx_mpi command line tool may be inappropriate for wrapping in an MPI-enabled gmxapi script. This would affect how we determine valid executables for use by the command line wrapper or purest test fixtures.

I would appreciate feedback from people with more familiarity with MPI-enabled command line tools regarding subprocesses of processes launched by (e.g.) mpiexec. See also #3086

#4 Updated by Eric Irrgang 2 months ago

Eric Irrgang wrote:

For issues such as helping users to build mpi4py...

Note that mpi4py installs an mpi.cfg that could be useful for either checking or hinting a compatible MPI compiler wrapper. I haven't checked yet whether this is retrievable through the Python pkg resources tools, but its existence should at least be noted...

#5 Updated by Eric Irrgang about 2 months ago

  • Related to Bug #3111: sample_restraint testing should not download files added

#6 Updated by Eric Irrgang about 2 months ago

  • Related to Task #3133: Cookiecutter for sample_restraint added

#7 Updated by Eric Irrgang about 2 months ago

  • Related to Feature #3177: Spack package management support added

#8 Updated by Joe Jordan 2 days ago

Eric Irrgang wrote:

Eric Irrgang wrote:

For issues such as helping users to build mpi4py...

Note that mpi4py installs an mpi.cfg that could be useful for either checking or hinting a compatible MPI compiler wrapper. I haven't checked yet whether this is retrievable through the Python pkg resources tools, but its existence should at least be noted...

The mpi.cfg file can be a bit dangerous. Even when using a venv python pip has a global cache, which means that if the user has more than one python environment configured with different mpi libraries, the mpi.cfg file will have the same contents unless the user installed mpi4py with the command

pip install --no-cache-dir mpi4py

This issue may eventually be solved by the introduction of pip cache https://github.com/pypa/pip/issues/4685 but we should not wait for that to solve this issue.

It is possible to querry the values set in mpi.cfg within python:

import mpi4py
mpi4py.get_config()

or alternately

from mpi4py import MPI
MPI.Get_library_version()

and this value will always be the "correct" one that is the mpi backend that is used by python, but the user may be expecting a different mpi version. I think we can get away with just having a small python script that runs during cmake time which returns the mpi version mpi4py is using. This can be compared with the cmake mpi library path. Perhaps there is a more elegant solution but I think this will meet our needs for the time being.

Also available in: Atom PDF