Fixing Dockerfile image build consistency
When Docker emerged in 2013 wasn’t a new technology in containerization space.
The container history started 1979 with Unix V7 Chroot and continued, 2000 FreeBSD Jails, 2001 Linux VServer, 2004 Solaris Containers, 2005 Open VZ, 2006 Process Containers, 2008 LXC, etc.
dotCloud now Docker popularized containers by adding simplicity in packaging, managing, and layered dependency based on Linux Kernel technologies like cgroups, namespaces, and Linux Kernel Capabilities (cap_*)
Context
Docker had a significant impact on development, application packaging, and deployment.
The most significant contribution, I believe, was in encapsulating all dependencies in deployable units that simplified the job of the DevOps/SRE teams significantly.
No technology is perfect neither is Docker containers.
Docker doesn’t solve all problems in development and deployment. This article will cover a few potential options that can significantly reduce dependency, consistency, and management of the deployable units.
The Problem
Dockerfile is the leading contender for the problem to be solved.
Dockerfile maintenance
Due to the imperative nature of the Dockerfile, it is effortless to start, but once you start using it extensively, it creates more problems than it solves.
Base image
We always try to minimize the size of the base image, and we have multiple options to select from, starting with Alpine Linux that is base on musl-libc and BusyBox or Ubuntu and Debian, Fedora, etc. based distribution optimized for containers. All these Base Images bring package management dependency that adds to the storage. We can select minimal Base Images like FROM scratch or Google distroless best works with statically linked binaries or GO base binaries.
Avoid as well building golden images as it is difficult to inspect and manage security.
Manual layer management
We all heard that you should minimize the docker image layers if you want to build an optimized image. But the tradeoff is by squishing the nr. of layers, and we remove the possibility of reusing the cash.
Every line in Dockerfile like RUN, CMD, COPY ADD, etc. adds a new layer.
You can combine the layers;
From
FROM alpine
RUN apk update && apk upgrade
RUN apk add nodejsTo
FROM alpine
RUN apk update && apk upgrade && apk add nodejsBut you will lose the possibility of reusing the cash.
Even if you opt for a multilayer approach, you will not be able to reuse intermediate layers. Docker images are composable that stack on each other, and not an easy way to manage cached layers.
Note: You may consider utilizing the Multi-Stage build.
Image Test and inspection
One of the problems with Dockerfile you can not test dependency efficiently and code before building the image. Details inspection of the components is also tough to achieve. You can test and inspect the image only once it is built.
Solution
We initially decided to use Nix as a functional expression language and NixOS as our development environment on GCP.
The main reason for the decision was Nix excellent support for multiple programming languages and consistently reproducible and highly reliable
dependencies management.
Nix provides a reliable function buildImage and buildLayeredImage to build Docker images, and this offers an excellent opportunity to keep tooling consistent in the development environment.
The below examples base on an excellent Video presentation by Rok Garbas from Tweag Nix + Docker, a match made in heaven
We will try to build Redis Docker container image.
Single-layer Docker Images
To build a single-layer Redis Docker Image, we will use the inbuilt nix function buildImage
redis.nix
{ pkgs ? import <nixpkgs>{}
}:let
inherit (pkgs.dockerTools) buildImage;in buildImage {
name = "nix-redis";
tag = "latest"; contents = with pkgs; [
redis
]; config = {
Cmd = [ "/bin/redis-server" ];
WorkingDir = "/data";
Volumes = {
"/data" = {};
};
};
created = "now";
}
As in above, you can see similarities with Dockerfile. Writing nix expression is straightforward and maps directly with Docker image build expressions.
Nest step, we will build nix expression with
~> nix-build redis.nix
/nix/store/j2i6lk1fgjb8f3ds5k508zjr7ymw3m88-docker-image-nix-redis.tar.gzThe above command will create a result symlink to *tar.gz file that contains all information needed to import into the Docker as an image.
~> ls -l result
lrwxrwxrwx 73 mudrii 23 Sep 19:05 result -> /nix/store/j2i6lk1fgjb8f3ds5k508zjr7ymw3m88-docker-image-nix-redis.tar.gzNow we can import *.tar.gz file into Docker.
~> docker load < result
Loaded image: nix-redis:latest
~> docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
nix-redis latest 13ea7dccae63 24 hours ago 156MB
We can run the container from nix-redis.
~> docker run --rm -ti nix-redis
1:C 23 Sep 2021 11:11:05.339 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 23 Sep 2021 11:11:05.339 # Redis version=6.2.5, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 23 Sep 2021 11:11:05.339 # Warning: no config file specified, using the default config. In order to specify a config file use /bin/redis-server /path/to/redis.conf
1:M 23 Sep 2021 11:11:05.340 * monotonic clock: POSIX clock_gettime
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 6.2.5 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in standalone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
| `-._ `._ / _.-' | PID: 1
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | https://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-' 1:M 23 Sep 2021 11:11:05.340 # Server initialized
1:M 23 Sep 2021 11:11:05.340 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:M 23 Sep 2021 11:11:05.341 * Ready to accept connections
Note: If you do not specify in redis.nix options for
tag = "latest";Docker will generate and attach SHA256 build tag to the imported image.Note: If you omit from the file
created = "now";Nix will use Docker Image created time as UNIX epoch (00:00:00, January 1st, 1970 in UTC) for the purpose of reproducibility.
~> docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
nix-redis 798k6n4arzcdl82kz26v9xg5pk01554d f1a8b6a12f08 51 years ago 156MBMulti-layer Docker Images
For building a multilayered Redis Docker Image, we will use the inbuilt nix function buildLayeredImage
Building a multilayered option is preferred. By using buildLayeredImage will allow us to cache every layer separately. We have granular control over layers in the image, and every single layer can be reused from the cache.
redis-multi.nix
{ pkgs ? import <nixpkgs>{}
, name ? "nix-redis-layered"
, redis ? pkgs.redis
}:let
inherit (pkgs.dockerTools) buildLayeredImage;in buildLayeredImage {
inherit name;
tag = "latest"; contents = with pkgs; [
redis
]; config = {
Cmd = [ "/bin/redis-server" ];
WorkingDir = "/data";
Volumes = {
"/data" = {};
};
};
created = "now";
maxLayers = 100;
}
As you can notice are minimal differences from a single layer. The only addon is to limit nr. of layers per Docker Image.
Note: Base on Docker documentation, allowed a maximum of 128 layers limits for overlay2 FS. From what I understood, this is due to a restriction of Linux Kernel not accepting more arguments length to a syscall, but is configurable.
Check your Docker Storage Driver by
~> docker info | grep "Storage Driver:"
Storage Driver: overlay2Let’s build out a multilayered container.
~> nix-build redis-multi.nix
/nix/store/c37yd8b30sh5y9yr1900zlhrxmdimvfc-nix-redis-layered.tar.gzLike in the 1st exercise, we should have a result symlink pointing to the nix store where the tar.gz archived image resides.
We can have a pick inside the archive before importing an image into the Docker.
~> ls -la result
.rw-r--r-- 431 mudrii users 23 Sep 19:58 redis-multi.nix
.rw-r--r-- 320 mudrii users 23 Sep 19:58 redis.nix
lrwxrwxrwx 68 mudrii users 23 Sep 19:58 result -> /nix/store/c37yd8b30sh5y9yr1900zlhrxmdimvfc-nix-redis-layered.tar.gz~> tar --list --file result
a412bdc048f52d573ad646355e118e7c237b59ea5cc7390f45a9ae932eaefc23/layer.tar
c1c544388655af4ae8177c16f0c1b14c76a8c231094a29942666be76773bffc2/layer.tar
c30d7ab5125d37157dc5826d9c2ae09890e22a5c8e298112e7461992397e3130/layer.tar
720d79ac643ac0faa43cbe0e5835233e67352e1f1bd17df124bd5b7d827cc73e/layer.tar
...
..
.
e9768307ff114da4db594040f966b4d9ff7b12373ccef7a4b40d4c1dc5b4bbd2/layer.tar
7b5c7d75aa7aea149ca73fe1d4aa33b65569395ccd2e7e69c3e82e4eb98389a6/layer.tar
ad8e5c65f5bf76e14b1a1e0a2a1f3b5fdb217b1ef2fdc6aaa41c4fc85236174e/layer.tar
0c2b166ddb88e06bae5382b5dba592a7ffded79f5f502141fc46a76230f86816/layer.tar
9b2bc95625a01f8de32f62dc0eb4357bd5782909d714fbdb03a9a87caa7e9c71/layer.tar
492a73edeac89ee536058f505aef22a0b0a42a11af73e8988b5304d02fa5c619.json
manifest.json
~> tar --list --file result | wc -l
75
The newly created multilayered image gives us 74 layers.
Now we are ready to import the archive into Docker as an image.
~> docker load < result
a412bdc048f5: Loading layer [==================================================>] 1.649MB/1.649MB
c1c544388655: Loading layer [==================================================>] 276.5kB/276.5kB
c30d7ab5125d: Loading layer [==================================================>] 31.6MB/31.6MB
720d79ac643a: Loading layer [==================================================>] 1.362MB/1.362MB
461d9b12152c: Loading layer [==================================================>] 6.031MB/6.031MB
c871541b5aae: Loading layer [==================================================>] 112.6kB/112.6kB
fdb9fbdf000e: Loading layer [==================================================>] 3.543MB/3.543MB
b1cae4033a7f: Loading layer [==================================================>] 133.1kB/133.1kB
d3333d590067: Loading layer [==================================================>] 2.038MB/2.038MB
3c49a3a82d73: Loading layer [==================================================>] 133.1kB/133.1kB
eadc6d642670: Loading layer [==================================================>] 512kB/512kB
0905d3dfdcae: Loading layer [==================================================>] 4.915MB/4.915MB
a4c0f2c42da2: Loading layer [==================================================>] 532.5kB/532.5kB
b9462ca7218c: Loading layer [==================================================>] 92.16kB/92.16kB
...
..
.
8cb49edd4f95: Loading layer [==================================================>] 1.106MB/1.106MB
6d53ab07cf25: Loading layer [==================================================>] 2.97MB/2.97MB
7fc9f06b9563: Loading layer [==================================================>] 2.519MB/2.519MB
52830d6f69e1: Loading layer [==================================================>] 491.5kB/491.5kB
62f481182e43: Loading layer [==================================================>] 174.1kB/174.1kB
e9768307ff11: Loading layer [==================================================>] 225.3kB/225.3kB
7b5c7d75aa7a: Loading layer [==================================================>] 1.812MB/1.812MB
ad8e5c65f5bf: Loading layer [==================================================>] 28.44MB/28.44MB
0c2b166ddb88: Loading layer [==================================================>] 4.26MB/4.26MB
9b2bc95625a0: Loading layer [==================================================>] 10.24kB/10.24kB
Loaded image: nix-redis-layered:latestReview imported layers
~> docker history nix-redis-layered:latest
IMAGE CREATED CREATED BY SIZE COMMENT
492a73edeac8 25 hours ago 252B store paths: ['/nix/store/vbsalcy8rvgcvlgdc4zfip4hvrbivf4y-nix-redis-layered-customisation-layer']
<missing> 25 hours ago 4.25MB store paths: ['/nix/store/a81n9nqr8imc36rf34kk9qjps7i62c90-redis-6.2.5']
<missing> 25 hours ago 27.7MB store paths: ['/nix/store/n5j5fjn60nhck658j9ab84k8n9z24n1r-systemd-247.6']
<missing> 25 hours ago 1.8MB store paths: ['/nix/store/0di899y5p0j8qx07pfvd3wb0iblkhr1b-pcre2-10.36']
<missing> 25 hours ago 215kB store paths: ['/nix/store/0b62nmnadx1xw2wz6hib3p6gzlylbdjc-lz4-1.9.3']
<missing> 25 hours ago 167kB store paths: ['/nix/store/rzmy26qy8afia865dbn06g6hfrrqiwvg-libmicrohttpd-0.9.71']
<missing> 25 hours ago 447kB store paths: ['/nix/store/002d6xvw29my0a8ayj8wb77wzvyy7vgv-libapparmor-3.0.1']
<missing> 25 hours ago 2.38MB store paths: ['/nix/store/d7fx1xwjdsvrypivnmp0pd2s2invphz8-iptables-1.8.7']
<missing> 25 hours ago 2.88MB store paths: ['/nix/store/n2dprmax869lb76gwpc2ygaqwbdgq7r0-gnutar-1.34']
<missing> 25 hours ago 1.09MB store paths: ['/nix/store/wi87l1pm7dfy9gn5kqh2ddzid7pn1x5s-gnupg-2.2.27']
<missing> 25 hours ago 744kB store paths: ['/nix/store/hh5ai1v2hhizkv5n4fr4hg72l1a5z1iw-curl-7.76.1']
<missing> 25 hours ago 2.54MB store paths: ['/nix/store/jrkrc4v0anvgi6axszy6lwqqcnlzjkq2-cryptsetup-2.3.5']
<missing> 25 hours ago 153kB store paths: ['/nix/store/1mai0bv7ch11nljavsyckcxlpixy6cg2-popt-1.18']
<missing> 25 hours ago 213kB store paths: ['/nix/store/n9va4cqy8r026jcjkxbnzhqbd33kl3nm-nghttp2-1.43.0-lib']
<missing> 25 hours ago 412kB store paths: ['/nix/store/1a9gbnqccqxjg86n5q8556ggi7f43psz-lvm2-2.03.12-lib']
<missing> 25 hours ago 293kB store paths: ['/nix/store/5r0a92rsppim1dljzr0rhpxwdsmnab9l-libssh2-1.9.0']
<missing> 25 hours ago 1.09MB store paths: ['/nix/store/5i7z7iidkhiwdb760p18b8px8g686rha-libpcap-1.10.0']
...
..
.
<missing> 25 hours ago 5.98MB store paths: ['/nix/store/50msfhkz5wbyk8i78pjv3y9lxdrp7dlm-gcc-10.3.0-lib']
<missing> 25 hours ago 1.33MB store paths: ['/nix/store/wv35g5lff84rray15zlzarcqi9fxzz84-bash-4.4-p23']
<missing> 25 hours ago 30.9MB store paths: ['/nix/store/jsp3h3wpzc842j0rz61m5ly71ak6qgdn-glibc-2.32-54']
<missing> 25 hours ago 227kB store paths: ['/nix/store/ckb0qa2yrxrpp0piffgjq9id38gc5z9v-libidn2-2.3.1']
<missing> 25 hours ago 1.63MB store paths: ['/nix/store/5d821pjgzb90lw4zbg6xwxs7llm335wr-libunistring-0.9.10']Another excellent tool that gives us insight into image details and can analyze the image from an optimization standpoint is dive
You can also configure dive and integrate with your CI tools.
~> CI=true dive nix-redis-layered
Using default CI config
Image Source: docker://nix-redis-layered
Fetching image... (this can take a while for large images)
Analyzing image...
efficiency: 100.0000 %
wastedBytes: 0 bytes (0 B)
userWastedPercent: 0.0000 %
Inefficient Files:
Count Wasted Space File Path
None
Results:
PASS: highestUserWastedPercent
SKIP: highestWastedBytes: rule disabled
PASS: lowestEfficiency
Result:PASS [Total:3] [Passed:2] [Failed:0] [Warn:0] [Skipped:1]Nix Docker Image optimization
Let’s compare the nix build images with available official Redis images.
~> docker pull redis
Using default tag: latest
latest: Pulling from library/redis
a330b6cecb98: Pull complete
14bfbab96d75: Pull complete
8b3e2d14a955: Pull complete
5da5e1b21a2f: Pull complete
6af3a5ca4596: Pull complete
4f9efe5b47a5: Pull complete
Digest: sha256:e595e79c05c7690f50ef0136acc9d932d65d8b2ce7915d26a68ca3fb41a7db61
Status: Downloaded newer image for redis:latest
docker.io/library/redis:latest
~> docker pull redis:alpine
alpine: Pulling from library/redis
a0d0a0d46f8b: Already exists
a04b0375051e: Pull complete
cdc2bb0f9590: Pull complete
8f19735ec10c: Pull complete
ac5156a4c6ca: Pull complete
7b7e1b3fdb00: Pull complete
Digest: sha256:fa785f9bd167b94a6b30210ae32422469f4b0f805f4df12733c2f177f500d1ba
Status: Downloaded newer image for redis:alpine
docker.io/library/redis:alpine
Let’s compare
~> docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
nix-redis-layered latest 492a73edeac8 27 hours ago 156MB
nix-redis latest 13ea7dccae63 27 hours ago 156MB
redis latest 02c7f2054405 2 weeks ago 105MB
redis alpine f6f2296798e9 3 weeks ago 32.3MBAt the 1st glance, we can see nix build is significantly bigger than official images, nix-redis:latest and nix-redis-layered:latest images is 156MB compare with redis:latest based on debian:buster-slim 105MB and redis:alpine based on alpine 32.3MB.
Official Images are highly optimized with statically linked libraries.
Now we can try and see if we can optimize our nix Redis image.
We will use a two-stage build that will rely on the previous redis-multi.nix, and we need to have one more nix file redis-multi-mini.nix to optimize the image further.
redis-multi-mini.nix
{ pkgs ? import <nixpkgs> {}
}:import ./redis-multi.nix {
inherit pkgs; name = "nix-redis-layered-minimal"; redis = pkgs.redis.overrideAttrs (old: {
makeFlags = old.makeFlags ++ ["USE_SYSTEMD=no"];
preBuild = ''
set -x
makeFlagsArray=(PREFIX="$out"
CC="${pkgs.musl.dev}/bin/musl-gcc"
CFLAGS="-I${pkgs.musl.dev}/include"
LDFLAGS="-L${pkgs.musl.dev}/lib")
'';
postInstall = "rm -f $out/bin/redis-{benchmark,check-*,cli}";
});
}
Let’s Build a new image.
~> nix-build redis-multi-mini.nix
/nix/store/k2q3nk68gvr25v825nh46s8mfdryca4y-nix-redis-layered-minimal.tar.gzThe new image is optimized with fewer layers.
~> tar --list --file result
2a894fdfe28bd1ceee40ddeea14823d89d52c50f4d5bb005c7eac74244e8331e/layer.tar
4c40801bc9f0d6275846f9d6768701ee4a87c3cd199c21fbd27d28529d9d7fb0/layer.tar
0d9152ec11246d30cc6f6935e793a28f46db2cd5c7e1ef8bda1125e228526d2f/layer.tar
ee626d23550b2811c38be9c7a5cf6eeb3a62425f1bf5363758b75ed0f8b55c7f/layer.tar
8023da80f76ff1c01ed438b5581533c4b4584b430218c15ee77f51bd388faa8c/layer.tar
1c5ba5caa619e901adde8f8062a324370f5a11e12fdfe325e04684293b3a112c/layer.tar
36bf402fa650e105532a17d08191e8b489dba14141df3c784a0b5db641bd3cb1.jsonImporting archive into the Docker as a new image
~> docker load < result
2a894fdfe28b: Loading layer [==================================================>] 1.649MB/1.649MB
4c40801bc9f0: Loading layer [==================================================>] 276.5kB/276.5kB
0d9152ec1124: Loading layer [==================================================>] 31.6MB/31.6MB
ee626d23550b: Loading layer [==================================================>] 4.198MB/4.198MB
8023da80f76f: Loading layer [==================================================>] 2.304MB/2.304MB
1c5ba5caa619: Loading layer [==================================================>] 10.24kB/10.24kB
Loaded image: nix-redis-layered-minimal:latestLet’s check the size.
~> docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
nix-redis-layered-minimal latest 36bf402fa650 25 hours ago 39.2MB
nix-redis-layered latest 492a73edeac8 28 hours ago 156MB
nix-redis latest 13ea7dccae63 28 hours ago 156MB
redis latest 02c7f2054405 2 weeks ago 105MB
redis alpine f6f2296798e9 3 weeks ago 32.3MBWe Managed to squeeze the image from 156MB to 39.2MB.
We can statically link with CC="${pkgs.musl.dev}/bin/musl-gcc -static"(currently is broken in Redis build) that can reduce our image to merely 2MB
Fin
Building Docker images with nix expressions solve many inherited issue presented with Dockerfile. This solution is not for everybody as we already selected Nix as our configuration management language was easy to choose.
We can find an extensive list of Nix expressions examples
The code used in the article you can find on my GitHub repository