Feature #2585: Infrastructure supporting external API
gmxapi integration testing
Supports gmxapi milestone 3 at parent issue.
GROMACS 2019 defines gmxapi 0.0.7. The kassonlab/gmxapi and kassonlab/sample_restraint repositories are the principal client software packages to test against a gmxapi compatible GROMACS installation.
- Gmxapi interfaces should continue functioning with unchanged semantics for other GROMACS changes, or API level needs to be incremented according to semantic versioning.
- External projects need to be tested outside of the gromacs build tree to validate external interfaces of installation. Suggested external projects: Python package, sample_restraint, validation test suite.
- Tests should be clear about the API version they are testing, and we should test all versions that aren’t unsupported (though we need a policy in this regard) and we can note whether new API updates are backwards compatible.
- Forward-compatibility testing: we should at least know whether we are breaking old client code and include release notes, regardless of policy
- ABI compatibility testing TBD. (should we test mismatched compilers and such?)
- Example code in documentation should be tested, where possible.
Set up testing matrix elements on Jenkins.
Jenkins build hosts will have Python virtual environments preconfigured for the integration tests, with necessary dependencies installed and archives of the 0.0.7 repositories for https://github.com/kassonlab/gmxapi and https://github.com/kassonlab/sample_restraint
After Jenkins installs GROMACS, these packages can be built and installed and their CI scripts run.
Disable gmxapi by default
Due to outstanding issues with the integration testing and tests failing
with large number of ranks, the gmxapi default has been changed to not
be build. In Jenkins, all supported builds still are still set to build
with GMXAPI enabled.
#4 Updated by Paul Bauer 4 months ago
I spend some time on this now, and while I have a setup that works on my machine it seemed to time out on the Jenkins agent I was running it on.
I also have some reservations to the CMake setup, as we don't need to test e.g. the documentation generation during our Jenkins builds.
Additionally, I see an issue with race conditions when two different builds try to build the tests at the same time and one of them spontaneously decides to uninstall the installed version why another one is running tests at the moment.
#8 Updated by Szilárd Páll 4 months ago
- Category set to analysis tools
As noted on gerrit, I think such proposals are better suited for an earlier release stage rather than in the middle of the betas.
Secondly, I'm not certain that testing GROMACS' own API with/against external software is the best choice. Why not have API integration tests implemented in the source repo?
#10 Updated by Eric Irrgang 4 months ago
better suited for an earlier release stage
To be clear, this has been part of ongoing discussion since before #2585. I created this task issue to collect notes from private conversations and to have something to link in the Gerrit changes that started appearing.
If the API is part of the source tree, the full integration tests should also live there so anybody can test it while developing
The API is tested in the source repo as much as possible, but the C++ API is limited enough right now that it is hard to write thorough tests without developing an entire application. A big part of the test space targeted here is the extensibility.
The current GROMACS testing infrastructure gives access to the GROMACS build tree and libgromacs, so it is not suited to testing client software. I think we should work on infrastructure in the GROMACS repo to allow more isolated testing, but I think that is somewhat of a separate issue.
For historic reasons, it is hard to tell whether there are behaviors in GROMACS that might change unexpectedly and violate an assumption that was based on observation rather than specification.
This issue (#2756) was intended to be about using a client software package whose behavior we control and understand to check for surprises when built against a GROMACS installation. Currently, the only (known) client software are the two packages I've proposed using.
I'm not sure it is well documented yet, but recent discussion indicates that we should plan to test gmxapi 0.0.7 code against GROMACS 2019, though gmxapi 0.0.8 client code will likely build against both GROMACS 2019 and later. We may want to fork the discussion to address (a) tests on the 2019 release branch, and (b) tests on master and 2020.
#11 Updated by Paul Bauer 4 months ago
Here is the follow-up from the dev teleconferenceThings to consider for getting integration tests running during Jenkins verification
- Rework build script to have gmx-integration build target that only builds what is necessary for tests to run and complete
- Consider requirement for C++14 in gmxapi build (just remembered this as well)
- Find way to reuse virtual envrionment -> Is it possible to archive a previously constructed venv and unpack it?
- Work around requirement to know C compiler version used for mpi4py (as instructed in build script: CC holds the compiler that is also associated with the `mpicc` used to build the mpi4py Python package.)
- Lay out plan to bind current gmxapi closer to general GROMACS API planning and integration to remove need for separate repository (can still be upstream, but should be included in main tree)
#12 Updated by Eric Irrgang about 2 months ago
This testing is slated to migrate to Docker-based CI infrastructure. Until that is set up on Jenkins, external relevant CI testing is visible at https://travis-ci.org/kassonlab/gromacs-gmxapi/branches
#15 Updated by Eric Irrgang 20 days ago
A note to consider test data availability:
src/testutils/simulationdatabase will ought to be sufficient for C++ API tests. Presumably client code of the C++ API (such as the Python package) should also use the standardized test data from this source. Should the data be installable with GROMACS? Available through the C++ API? Copied into the Python packaging area by the top-level CMake project? I could imagine CMake flags like "INSTALL_TEST_DATA" or pytest arguments like "--test-data-directory".