Task #2756

Updated by Eric Irrgang about 1 year ago

Supports gmxapi milestone 3 at parent issue.

h1. Integration testing

GROMACS 2019 defines gmxapi 0.0.7. The kassonlab/gmxapi and kassonlab/sample_restraint repositories are the principal client software packages to test against a gmxapi compatible GROMACS installation. These packages are probably compatible with GROMACS 2020, but are not likely to be tested in the GROMACS project infrastructure.

GROMACS 2020 coincides with the definition of gmxapi 0.1, which is not fully compatible with GROMACS 2019, and probably will not be.

GROMACS infrastructure will include full gmxapi testing on GitLab Runner, but not on Jenkins.

h2. Goals

* Gmxapi interfaces should continue functioning with unchanged semantics for other GROMACS changes, or API level needs to be incremented according to semantic versioning.
* External projects need to be tested outside of the gromacs build tree to validate external interfaces of installation. Suggested external projects: Python package, sample_restraint, validation test suite.
* Tests should be clear about the API version they are testing, and we should test all versions that aren’t unsupported (though we need a policy in this regard) and we can note whether new API updates are backwards compatible.
* Forward-compatibility testing: we should at least know whether we are breaking old client code and include release notes, regardless of policy
* ABI compatibility testing TBD. (should we test mismatched compilers and such?)
* Example code in documentation should be tested, where possible.

h2. Matrix

Matrix axes contain the following.

* Python versions 3.5, 3.6, 3.7, 3.8
* MPI implementations mpich, mvapich, openmpi
* C++ compilers (all supported clang, gcc, intel...)
* GROMACS with MPI, thread-MPI, and no MPI
* GPU domain decomposition (relevant mostly for MD plugin testing, but possibly more)
* GROMACS floating point precision
* GROMACS branches master, release-2020, and beyond
* gmxapi 0.1, 0.2, and beyond

h2. Versions tested

h3. Nightly

For version information not specified in the matrix description, I think it is appropriate to test the most recent minor version or dependency version.

h3. pre-commit

* Python patch version: default or most recent at time of configuration.
* Python package dependencies: minimum version described in package requirements.
* C++ dependencies (pybind, googletest, Eigen?): minimum version described in package requirements.
* cross verification (from master) with a release branch: oldest known compatible tag.
* cross verification (from release branch patch) with master: tip.


- [ ] Define Set up testing matrices for the various schedules (nightly, pre-commit, others?). Choose the sparse matrix elements to test.
- [ ] Add "nightly-test" stage?
- [ ] Set up nightly rebuild of Python "venvs"s. Tagged artifacts? Some other caching mechanism?
- [ ] Generate initial pre-commit venvs (Keys: Python version, MPI version, and C++ toolchain.)
- [ ] Document update procedure for pre-commit venvs.
- [ ] Is there a GROMACS installation artifact already generated?
- [ ] What stage should client software be built and tested in?
- [ ] Extend .test-script-template for pytest with JUnit XML output.

h2. Job dependencies

Apologies for the poor ASCII art...

build GROMACS -> install GROMACS -> build sample_restraint -> C++ unit tests
>-> pytest tests (MPI does not need as full coverage; could depend
on other variables)
\-> install gmxapi -> non-MPI pytest tests
\-> MPI pytest tests
\-> acceptance tests(?)


The most expensive part here is the GROMACS
Jenkins build and install. The client software builds take a few seconds. The pytest unit tests take well under a minute. These timings may be comparable to the container launch or artifact handling, so it is probably sufficient to hosts will have a single stage to

1. install gmxapi
2. build (and install) sample_restraint
3-7. run
Python virtual environments preconfigured for the various tests.

The choice also depends whether there is a reason to have more stages versus wider stages, because we should plan to cross-check versions of the gmxapi Python package
integration tests, with necessary dependencies installed and versions archives of the sample code, at least until we are confident that we have well-characterized 0.0.7 repositories for and acceptable compatibility guarantee
After Jenkins installs GROMACS, these packages can be built
and versioning API.
installed and their CI scripts run.