Feature #2585: Infrastructure supporting external API
gmxapi integration testing
Supports gmxapi milestone 3 at parent issue.
GROMACS 2019 defines gmxapi 0.0.7. The kassonlab/gmxapi and kassonlab/sample_restraint repositories are the principal client software packages to test against a gmxapi compatible GROMACS installation.
- Gmxapi interfaces should continue functioning with unchanged semantics for other GROMACS changes, or API level needs to be incremented according to semantic versioning.
- External projects need to be tested outside of the gromacs build tree to validate external interfaces of installation. Suggested external projects: Python package, sample_restraint, validation test suite.
- Tests should be clear about the API version they are testing, and we should test all versions that aren’t unsupported (though we need a policy in this regard) and we can note whether new API updates are backwards compatible.
- Forward-compatibility testing: we should at least know whether we are breaking old client code and include release notes, regardless of policy
- ABI compatibility testing TBD. (should we test mismatched compilers and such?)
- Example code in documentation should be tested, where possible.
Set up testing matrix elements on Jenkins.
Jenkins build hosts will have Python virtual environments preconfigured for the integration tests, with necessary dependencies installed and archives of the 0.0.7 repositories for https://github.com/kassonlab/gmxapi and https://github.com/kassonlab/sample_restraint
After Jenkins installs GROMACS, these packages can be built and installed and their CI scripts run.
Disable gmxapi by default
Due to outstanding issues with the integration testing and tests failing
with large number of ranks, the gmxapi default has been changed to not
be build. In Jenkins, all supported builds still are still set to build
with GMXAPI enabled.
Read GMXRC in gmxapi docker test environment.
Allows expected behavior for gmxapi/ci-... docker containers and ensures
that GROMACS components available during image build are available when
Allows `gmx` binary to be easily discovered by Python wrappers.
Gerrit patch set 9390/4
#4 Updated by Paul Bauer 6 months ago
I spend some time on this now, and while I have a setup that works on my machine it seemed to time out on the Jenkins agent I was running it on.
I also have some reservations to the CMake setup, as we don't need to test e.g. the documentation generation during our Jenkins builds.
Additionally, I see an issue with race conditions when two different builds try to build the tests at the same time and one of them spontaneously decides to uninstall the installed version why another one is running tests at the moment.
#8 Updated by Szilárd Páll 6 months ago
- Category set to analysis tools
As noted on gerrit, I think such proposals are better suited for an earlier release stage rather than in the middle of the betas.
Secondly, I'm not certain that testing GROMACS' own API with/against external software is the best choice. Why not have API integration tests implemented in the source repo?
#10 Updated by Eric Irrgang 6 months ago
better suited for an earlier release stage
To be clear, this has been part of ongoing discussion since before #2585. I created this task issue to collect notes from private conversations and to have something to link in the Gerrit changes that started appearing.
If the API is part of the source tree, the full integration tests should also live there so anybody can test it while developing
The API is tested in the source repo as much as possible, but the C++ API is limited enough right now that it is hard to write thorough tests without developing an entire application. A big part of the test space targeted here is the extensibility.
The current GROMACS testing infrastructure gives access to the GROMACS build tree and libgromacs, so it is not suited to testing client software. I think we should work on infrastructure in the GROMACS repo to allow more isolated testing, but I think that is somewhat of a separate issue.
For historic reasons, it is hard to tell whether there are behaviors in GROMACS that might change unexpectedly and violate an assumption that was based on observation rather than specification.
This issue (#2756) was intended to be about using a client software package whose behavior we control and understand to check for surprises when built against a GROMACS installation. Currently, the only (known) client software are the two packages I've proposed using.
I'm not sure it is well documented yet, but recent discussion indicates that we should plan to test gmxapi 0.0.7 code against GROMACS 2019, though gmxapi 0.0.8 client code will likely build against both GROMACS 2019 and later. We may want to fork the discussion to address (a) tests on the 2019 release branch, and (b) tests on master and 2020.
#11 Updated by Paul Bauer 6 months ago
Here is the follow-up from the dev teleconferenceThings to consider for getting integration tests running during Jenkins verification
- Rework build script to have gmx-integration build target that only builds what is necessary for tests to run and complete
- Consider requirement for C++14 in gmxapi build (just remembered this as well)
- Find way to reuse virtual envrionment -> Is it possible to archive a previously constructed venv and unpack it?
- Work around requirement to know C compiler version used for mpi4py (as instructed in build script: CC holds the compiler that is also associated with the `mpicc` used to build the mpi4py Python package.)
- Lay out plan to bind current gmxapi closer to general GROMACS API planning and integration to remove need for separate repository (can still be upstream, but should be included in main tree)
#12 Updated by Eric Irrgang 4 months ago
This testing is slated to migrate to Docker-based CI infrastructure. Until that is set up on Jenkins, external relevant CI testing is visible at https://travis-ci.org/kassonlab/gromacs-gmxapi/branches
#15 Updated by Eric Irrgang 3 months ago
A note to consider test data availability:
src/testutils/simulationdatabase will ought to be sufficient for C++ API tests. Presumably client code of the C++ API (such as the Python package) should also use the standardized test data from this source. Should the data be installable with GROMACS? Available through the C++ API? Copied into the Python packaging area by the top-level CMake project? I could imagine CMake flags like "INSTALL_TEST_DATA" or pytest arguments like "--test-data-directory".
#19 Updated by Eric Irrgang about 1 month ago
Updates from discussion with Mark this morning regarding new Jenkins pipelines implemented as Kubernetes pods.
New testing infrastructure will provision build hosts with single toolchains so that compiler hinting is not necessary. This means that we won't have to do any extra checking to make sure that
mpicc discovered by CMake or
pip are compatible with the default compilers or with those used by different components (libgromacs vs. mpi4py, etc.).
Driving the builds¶
For convenience, all building and testing will be driven by CMake and CTest. We can build the package in the build tree with CMake and then invoke pytest from a CTest target with an appropriate PYTHONPATH.
Processing test results consists of evaluating the return value of the CTest and, optionally, scripted processing of console output or JUnit XML output as a filesystem artifact tracked by Jenkins.
Coverage and optimization¶
To better simulate real-world use cases, we should have a CMake build target that either builds a virtualenv or receives one as an argument, then runs
pip install for the package.
We can configure the Docker images for the build hosts to have a Python virtualenv pre-configured for the jenkins user with the minimum versions of the Python package dependencies. This virtualenv could be activated as part of the environment set up for the Jenkins agent or passed as a special CMake flag. We can also/instead test
--user installs for system Python installations or non-virtualenv installs for user space Python installations.
In any case, with the environment pre-configured, we can run
--no-index to build and install the package with baseline dependencies.
We should also test up-to-date Python packages (at least occasionally) either by periodically preparing an environment with then-current packages or by having a Jenkins test that allows PyPI to resolve the package dependencies. Note that we can also stash a bunch of downloaded packages in a directory to reference with the
pip install --find-links option.