Project

General

Profile

Bug #3263

Change dockerfile organization to have all builds in one file

Added by Paul Bauer 3 months ago. Updated about 1 month ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
build system
Target version:
Affected version - extra info:
Affected version:
Difficulty:
uncategorized
Close

Description

Currently all ci docker containers have their own build file in a separate folder under admin/dockerfiles/ci-{type-of-docker-build}/Dockerfile

It will be beneficial to combine all of those files into one central Dockerfile, as this will allow reusing the build artefacts from other docker builds, as well as reduce code duplication.

Note, this will only affect CI builds.


Related issues

Related to GROMACS - Feature #3395: Consider scripted composition of DockerfilesFeedback wanted

Associated revisions

Revision 1093dfa9 (diff)
Added by Eric Irrgang about 1 month ago

Simplify GitLab CI Dockerfiles.

Consolidate some docker build instructions into distinct images for
the base dependencies, the various toolchains, and the docs build
environment. Combine Dockerfiles that only differ in toolchain version.

Total image size is prioritized over total build time by making all
apt-get layers include update, install, and cleaning in single RUN
commands, where intermediate layers are part of the tagged images.

Made images both for clang and gcc builds of the documentation.

Refs #3263

Change-Id: I55d6a128067f78214193223756db94a1964c491d

History

#1 Updated by Eric Irrgang 3 months ago

If the intention is to combine Dockerfiles that produce a single tagged image, it sounds like a good idea.

If the intention is to use docker build --target to select the stage that produces the image to be tagged, that is non-standard enough that I would be skeptical unless there is some other constraint, like if it is only feasible to issue a docker build instruction at one point in the pipeline, or if layer cache is being reused when it needs to be invalidated.

If the intention is to have a single Dockerfile that does very different things based on build arguments, I think that would be very nonstandard, hard to read/maintain, and hard to implement with effective build layer cache.

For the tip of release-2020 as of now, it looks like the compiler toolchain variants would make sense as build-time variations on a single Dockerfile recipe. The "docs" Dockerfile looks like it should remain separate, but be shortened to FROM one of the compiler toolchain images.

What is the mechanism by which these images are built? It looks like they should need to be rebuilt very rarely, verging on "never" as far as the CI infrastructure is concerned. Is there a nice place in the GitLab Runner pipeline to build/rebuild these images, or might they just exist on dockerhub?

#2 Updated by Mark Abraham 3 months ago

Eric Irrgang wrote:

If the intention is to combine Dockerfiles that produce a single tagged image, it sounds like a good idea.

Yes we should avoid obvious repetition.

If the intention is to use docker build --target to select the stage that produces the image to be tagged, that is non-standard enough that I would be skeptical unless there is some other constraint, like if it is only feasible to issue a docker build instruction at one point in the pipeline, or if layer cache is being reused when it needs to be invalidated.

We'll use that command for clarity at the point of call, but we don't need a single Dockerfile to rule them all.

If the intention is to have a single Dockerfile that does very different things based on build arguments, I think that would be very nonstandard, hard to read/maintain, and hard to implement with effective build layer cache.

Agreed

For the tip of release-2020 as of now, it looks like the compiler toolchain variants would make sense as build-time variations on a single Dockerfile recipe. The "docs" Dockerfile looks like it should remain separate, but be shortened to FROM one of the compiler toolchain images.

Yes. Eric's https://gerrit.gromacs.org/#/c/15017 looks good

What is the mechanism by which these images are built? It looks like they should need to be rebuilt very rarely, verging on "never" as far as the CI infrastructure is concerned. Is there a nice place in the GitLab Runner pipeline to build/rebuild these images, or might they just exist on dockerhub?

Indeed they are only built rarely and we should run the builds manually until that proves to be a human bottleneck.

#3 Updated by Paul Bauer about 1 month ago

  • Status changed from New to Resolved

I think we can resolve this for now and have the discussions in the other related open issues

#4 Updated by Eric Irrgang about 1 month ago

  • Related to Feature #3395: Consider scripted composition of Dockerfiles added

#5 Updated by Paul Bauer about 1 month ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF