Task #3272: Port complete CI testing to Gitlab
Document Gitlab CI instead of Jenkins and releng
To complete the switch to Gitlab, the documentation currently found in the releng repository needs to be ported and adapted to match the Gitlab build and test system.
Establish stubs for documenting GitLab configuration.
Insert an `infrastructure` document under the `tools` node to own the
`jenkins` doc and a new `gitlab` doc.
Reorganize release templates.
Rename "release" config file to "archive" to collect jobs related to
preparing archives for export. "release" jobs are just special cases of
jobs that already fit in other categories.
Move several jobs between the "lint," "documentation," "testing-matrix,"
and "archive" groupings. Sort job definitions within files in terms of
execution dependency, definition dependency, and toolchain version.
In the future, we can probably reduce the distinction between jobs that
are 'release' or not (jobs that always run may choose a code path
according to to $GROMACS_RELEASE) so I'll leave the 'release' jobs
clustered with the non-'release' jobs.
#2 Updated by Erik Lindahl 4 months ago
My plan is that we should stick entirely to Gitlab's default docs for the overall infrastructure (which is far better than ours), and only document our deviations from their typical flow.
We might also need to document a few things about how we create images, but here too our first task is to learn to keep it simple. Once we've had things running smoothly for 6 months we can start to think about customisation :-)
#3 Updated by Eric Irrgang 4 months ago
The biggest challenge for me right now is in understanding the naming schemes and many sources of job composition. It is not clear where standard parameters are supposed to come from.
Note that the depth and breadth of
extends usage makes it harder to understand the GROMACS YAML configuration in terms of standard GitLab documentation, and might be considered deviant from typical flow.
Other deviations include the large number of build stages and extensive use of "needs" instead of "dependencies".
Local details to document include the various stages defined, internally relevant variables, reusable artifacts, and conditional job logic (both suggested heuristics and which configuration mechanics should be used, particularly since we aren't using the newer "rules" syntax).
#4 Updated by Paul Bauer 4 months ago
Standard parameters are defined in the variable template and the build/test script templates and are also printed to the output for the jobs that do the actual CMake configuration.
I tried to make the individual included parts individually small so that we are able to compose the actual final job according to a combination of different extends template, something I also saw recommended somewhere in the Gitlab documentation.
Needs is recommended to create non-blocking dependecies, so that different test jobs only need to wait for their build job to be done and not all of them. I think waiting on a complete stage to finish is counterproductive for our kinds of testing, as we have many different build and test jobs that don't depend on others and can fire their dependent jobs before all the jobs in the same stage are done.
#5 Updated by Eric Irrgang 4 months ago
I added some provisional notes about variables and added a question about needs to https://gerrit.gromacs.org/c/gromacs/+/15880
I removed the "private" marker from https://gerrit.gromacs.org/c/gromacs/+/15881 to help identify conventions that should also be documented (and to try to reduce inheritance complexity).
Erik Lindahl wrote:
we should stick entirely to Gitlab's default docs for the overall infrastructure
It would be great if we could just list the job parameters and such that we use, link to GitLab docs for what they mean, and document how/why we choose the values we use. I don't think that the way the current jobs are composed fits such a documentation approach, though. At a high level, the main thing missing is a set of guidelines for composing the jobs.