Project

General

Profile

Task #2756

Feature #2585: Infrastructure supporting external API

gmxapi integration testing

Added by Eric Irrgang 11 months ago. Updated about 2 months ago.

Status:
In Progress
Priority:
Normal
Assignee:
-
Category:
analysis tools
Difficulty:
uncategorized
Close

Description

Supports gmxapi milestone 3 at parent issue.

Integration testing

GROMACS 2019 defines gmxapi 0.0.7. The kassonlab/gmxapi and kassonlab/sample_restraint repositories are the principal client software packages to test against a gmxapi compatible GROMACS installation.

Goals

  • Gmxapi interfaces should continue functioning with unchanged semantics for other GROMACS changes, or API level needs to be incremented according to semantic versioning.
  • External projects need to be tested outside of the gromacs build tree to validate external interfaces of installation. Suggested external projects: Python package, sample_restraint, validation test suite.
  • Tests should be clear about the API version they are testing, and we should test all versions that aren’t unsupported (though we need a policy in this regard) and we can note whether new API updates are backwards compatible.
  • Forward-compatibility testing: we should at least know whether we are breaking old client code and include release notes, regardless of policy
  • ABI compatibility testing TBD. (should we test mismatched compilers and such?)
  • Example code in documentation should be tested, where possible.

Tasks

Set up testing matrix elements on Jenkins.

Jenkins build hosts will have Python virtual environments preconfigured for the integration tests, with necessary dependencies installed and archives of the 0.0.7 repositories for https://github.com/kassonlab/gmxapi and https://github.com/kassonlab/sample_restraint
After Jenkins installs GROMACS, these packages can be built and installed and their CI scripts run.


Related issues

Related to GROMACS - Task #2912: C++ extension module for Python bindingsResolved
Related to GROMACS - Task #3033: Clean up and modernize googletest bundling and usageIn Progress
Related to GROMACS - Bug #3111: sample_restraint testing should not download filesNew
Related to GROMACS - Task #3133: Cookiecutter for sample_restraintNew
Blocks GROMACS - Bug #2722: gmxapi may over-manage RPATHNew

Associated revisions

Revision 3fd14b5f (diff)
Added by Paul Bauer 10 months ago

Disable gmxapi by default

Due to outstanding issues with the integration testing and tests failing
with large number of ranks, the gmxapi default has been changed to not
be build. In Jenkins, all supported builds still are still set to build
with GMXAPI enabled.

Refs #2765, #2722, #2756

Change-Id: I2cc42c461edc206aaa30be6cac3db0a52ccae991

Revision 4ed5d8ed (diff)
Added by Eric Irrgang 6 months ago

Read GMXRC in gmxapi docker test environment.

Allows expected behavior for gmxapi/ci-... docker containers and ensures
that GROMACS components available during image build are available when
running.

Allows `gmx` binary to be easily discovered by Python wrappers.

Ref: #2756

Gerrit patch set 9390/4

Revision 354da806 (diff)
Added by Eric Irrgang 6 months ago

More Python testing infrastructure for gmxapi.

  • Add and improve test fixtures.
  • Generate MD input files for tests.

Refs: #2756
Refs: #2896

Gerrit patch set 9392/2

Revision 929343d1 (diff)
Added by Eric Irrgang 5 months ago

More Python testing infrastructure for gmxapi.

  • Add and improve test fixtures.
  • Generate MD input files for tests.

Refs: #2756
Refs: #2896

Change-Id: I15ac8e44844cbf699cfe362e2fa443d07a355b3d

Revision b67801d7 (diff)
Added by Eric Irrgang 4 months ago

Updates to gmxapi docker entry points.

  • Read GMXRC in gmxapi docker test environment. Allows `gmx` binary to
    be easily discovered by Python wrappers. Supports expected behavior
    for gmxapi/ci-... docker containers and ensures that GROMACS
    components available during image build are available when running.
  • Allow more granular testing. Rename and reorganize entry points for
    the docker test containers.

Ref: #2756

updates from new patch set to change 9390

Revision e5a8e153 (diff)
Added by Eric Irrgang 4 months ago

Updates to gmxapi docker entry points.

  • Read GMXRC in gmxapi docker test environment. Allows `gmx` binary to
    be easily discovered by Python wrappers. Supports expected behavior
    for gmxapi/ci-... docker containers and ensures that GROMACS
    components available during image build are available when running.
  • Allow more granular testing. Rename and reorganize entry points for
    the docker test containers.

Ref: #2756

Change-Id: I5e781fae4709e4ccb718a8295c3fbe68dd59de44

Revision 42adebaf (diff)
Added by Eric Irrgang 3 months ago

Improve gmxapi docker testing environment.

  • Generalize venv directory.
  • Add some usage notes.
  • Better support alternative users in executing containers.
  • Remove stale TODO.
  • Allow for passing additional arguments to pytest.

Refs #2756

Change-Id: I9f67d46aa93fb789825cb78cc2a95b24f7dfccaa

Revision 7cf28864 (diff)
Added by Eric Irrgang 3 months ago

Clarify use of Python venv in gmxapi docker images.

Refs #2756

Change-Id: I182c83be424fcf134bc0f3ce7c41a58c6fb1f07e

Revision 6ec0f1f1 (diff)
Added by Eric Irrgang 2 months ago

Allow gmxapi Python package tests to be run from GROMACS build tree.

  • Configure gmxapi._gmxapi to build with the ability to find libgmxapi.
  • Add a gmxapi_pytest custom CMake target to invoke pytest.

Refs #2961
Refs #2756

Change-Id: I64ca82afbf37da3c37759ec302c2ac9cc1666de2

Revision 4217d8fc (diff)
Added by Eric Irrgang about 2 months ago

Enable GMXAPI by default.

Change default CMake configuration to enable the build and install of
libgmxapi and associated headers.

Refs #2756
Refs #3059

Change-Id: I5688a1ad6b524882090201014fc34cb34c1e3748

History

#1 Updated by Eric Irrgang 11 months ago

  • Tracker changed from Feature to Task

#2 Updated by Eric Irrgang 11 months ago

  • Blocks Bug #2722: gmxapi may over-manage RPATH added

#3 Updated by Gerrit Code Review Bot 10 months ago

Gerrit received a related patchset '7' for Issue #2756.
Uploader: Paul Bauer ()
Change-Id: gromacs~release-2019~I9ad2995263c7798c2c4569377be701d4a8ec7291
Gerrit URL: https://gerrit.gromacs.org/8667

#4 Updated by Paul Bauer 10 months ago

I spend some time on this now, and while I have a setup that works on my machine it seemed to time out on the Jenkins agent I was running it on.
I also have some reservations to the CMake setup, as we don't need to test e.g. the documentation generation during our Jenkins builds.
Additionally, I see an issue with race conditions when two different builds try to build the tests at the same time and one of them spontaneously decides to uninstall the installed version why another one is running tests at the moment.

#5 Updated by Paul Bauer 10 months ago

Also, the CI scripts don't seem to support versioned builds, making it difficult to have proper version testing in Jenkins

#6 Updated by Gerrit Code Review Bot 10 months ago

Gerrit received a related DRAFT patchset '1' for Issue #2756.
Uploader: Paul Bauer ()
Change-Id: gromacs~release-2019~I2cc42c461edc206aaa30be6cac3db0a52ccae991
Gerrit URL: https://gerrit.gromacs.org/8801

#7 Updated by Paul Bauer 10 months ago

  • Target version changed from 2019 to 2020

More likely to be done in 2020

#8 Updated by Szilárd Páll 10 months ago

  • Category set to analysis tools

As noted on gerrit, I think such proposals are better suited for an earlier release stage rather than in the middle of the betas.

Secondly, I'm not certain that testing GROMACS' own API with/against external software is the best choice. Why not have API integration tests implemented in the source repo?

#9 Updated by Erik Lindahl 10 months ago

Agree about testing location. If the API is part of the source tree, the full integration tests should also live there so anybody can test it while developing, not in a separate external repo.

#10 Updated by Eric Irrgang 10 months ago

better suited for an earlier release stage

To be clear, this has been part of ongoing discussion since before #2585. I created this task issue to collect notes from private conversations and to have something to link in the Gerrit changes that started appearing.

If the API is part of the source tree, the full integration tests should also live there so anybody can test it while developing

The API is tested in the source repo as much as possible, but the C++ API is limited enough right now that it is hard to write thorough tests without developing an entire application. A big part of the test space targeted here is the extensibility.

The current GROMACS testing infrastructure gives access to the GROMACS build tree and libgromacs, so it is not suited to testing client software. I think we should work on infrastructure in the GROMACS repo to allow more isolated testing, but I think that is somewhat of a separate issue.

For historic reasons, it is hard to tell whether there are behaviors in GROMACS that might change unexpectedly and violate an assumption that was based on observation rather than specification.

This issue (#2756) was intended to be about using a client software package whose behavior we control and understand to check for surprises when built against a GROMACS installation. Currently, the only (known) client software are the two packages I've proposed using.

I'm not sure it is well documented yet, but recent discussion indicates that we should plan to test gmxapi 0.0.7 code against GROMACS 2019, though gmxapi 0.0.8 client code will likely build against both GROMACS 2019 and later. We may want to fork the discussion to address (a) tests on the 2019 release branch, and (b) tests on master and 2020.

#11 Updated by Paul Bauer 10 months ago

Here is the follow-up from the dev teleconference

Things to consider for getting integration tests running during Jenkins verification
  • Rework build script to have gmx-integration build target that only builds what is necessary for tests to run and complete
  • Consider requirement for C++14 in gmxapi build (just remembered this as well)
  • Find way to reuse virtual envrionment -> Is it possible to archive a previously constructed venv and unpack it?
  • Work around requirement to know C compiler version used for mpi4py (as instructed in build script: CC holds the compiler that is also associated with the `mpicc` used to build the mpi4py Python package.)
  • Lay out plan to bind current gmxapi closer to general GROMACS API planning and integration to remove need for separate repository (can still be upstream, but should be included in main tree)

#12 Updated by Eric Irrgang 8 months ago

This testing is slated to migrate to Docker-based CI infrastructure. Until that is set up on Jenkins, external relevant CI testing is visible at https://travis-ci.org/kassonlab/gromacs-gmxapi/branches

#13 Updated by Gerrit Code Review Bot 7 months ago

Gerrit received a related DRAFT patchset '4' for Issue #2756.
Uploader: M. Eric Irrgang ()
Change-Id: gromacs~master~Ic74ee9abc70f9d1be3e1d94619477c65ee093cb0
Gerrit URL: https://gerrit.gromacs.org/9336

#14 Updated by Eric Irrgang 7 months ago

  • Related to Task #2912: C++ extension module for Python bindings added

#15 Updated by Eric Irrgang 7 months ago

A note to consider test data availability: src/testutils/simulationdatabase will ought to be sufficient for C++ API tests. Presumably client code of the C++ API (such as the Python package) should also use the standardized test data from this source. Should the data be installable with GROMACS? Available through the C++ API? Copied into the Python packaging area by the top-level CMake project? I could imagine CMake flags like "INSTALL_TEST_DATA" or pytest arguments like "--test-data-directory".

#17 Updated by Gerrit Code Review Bot 6 months ago

Gerrit received a related DRAFT patchset '1' for Issue #2756.
Uploader: M. Eric Irrgang ()
Change-Id: gromacs~master~I5e781fae4709e4ccb718a8295c3fbe68dd59de44
Gerrit URL: https://gerrit.gromacs.org/9390

#18 Updated by Gerrit Code Review Bot 6 months ago

Gerrit received a related DRAFT patchset '1' for Issue #2756.
Uploader: M. Eric Irrgang ()
Change-Id: gromacs~master~I15ac8e44844cbf699cfe362e2fa443d07a355b3d
Gerrit URL: https://gerrit.gromacs.org/9392

#19 Updated by Eric Irrgang 5 months ago

Updates from discussion with Mark this morning regarding new Jenkins pipelines implemented as Kubernetes pods.

Tool chains

New testing infrastructure will provision build hosts with single toolchains so that compiler hinting is not necessary. This means that we won't have to do any extra checking to make sure that mpicc discovered by CMake or pip are compatible with the default compilers or with those used by different components (libgromacs vs. mpi4py, etc.).

Driving the builds

For convenience, all building and testing will be driven by CMake and CTest. We can build the package in the build tree with CMake and then invoke pytest from a CTest target with an appropriate PYTHONPATH.

Processing test results consists of evaluating the return value of the CTest and, optionally, scripted processing of console output or JUnit XML output as a filesystem artifact tracked by Jenkins.

Coverage and optimization

To better simulate real-world use cases, we should have a CMake build target that either builds a virtualenv or receives one as an argument, then runs pip install for the package.
We can configure the Docker images for the build hosts to have a Python virtualenv pre-configured for the jenkins user with the minimum versions of the Python package dependencies. This virtualenv could be activated as part of the environment set up for the Jenkins agent or passed as a special CMake flag. We can also/instead test --user installs for system Python installations or non-virtualenv installs for user space Python installations.

In any case, with the environment pre-configured, we can run pip with --no-index to build and install the package with baseline dependencies.

We should also test up-to-date Python packages (at least occasionally) either by periodically preparing an environment with then-current packages or by having a Jenkins test that allows PyPI to resolve the package dependencies. Note that we can also stash a bunch of downloaded packages in a directory to reference with the pip install --find-links option.

#20 Updated by Eric Irrgang 3 months ago

  • Status changed from New to In Progress
  • Target version changed from 2020 to 2020-infrastructure-stable

Note that, in addition to figuring out how to give the sample_restraint tree access to pybind11 and googletest sources, we need to tell it how to find the gmxapi package in the build tree (${CMAKE_BINARY_DIR}/python_packaging/src/gmxapi_staging) when running the Python-based tests. We also need CTest wrappers for both packages.

#21 Updated by Eric Irrgang 3 months ago

  • Related to Task #3033: Clean up and modernize googletest bundling and usage added

#22 Updated by Mark Abraham about 2 months ago

  • Target version changed from 2020-infrastructure-stable to 2020-infrastructure-update-post-beta1

#23 Updated by Eric Irrgang 1 day ago

  • Related to Bug #3111: sample_restraint testing should not download files added

#24 Updated by Eric Irrgang 1 day ago

  • Related to Task #3133: Cookiecutter for sample_restraint added

Also available in: Atom PDF