chroot as docker alternative for building

As mentioned in the previous article, docker does not fit my needs.

It is too complex to maintain, has a high-performance cost, and all provided features (apart for filesystem isolation) are roughly useless.

So the solution seemed to compile and package every version of compiler and toolchain by myself. Using a normal binary on the host has many advantages, but it is a maintenance burden.

So I checked If I could use chroot, which is one of the many older tools which docker is based on.

The nice thing about this approach is that I can probably reuse containers, even if I have to inspect how they work. On the other hand, this is one of the issues that I wanted to avoid.

As one of the things I dislike when using docker is running programs as an administrator, in order not to redo the same error, I’m going to use schroot (hopefully available on most GNU/Linux distributions).

Creating the chroot environment is fortunately very easy:

GCCVER=9
docker create --name temp-container --rm gcc:"$GCCVER"
sudo docker cp temp-container:/ /opt/gcc-"$GCCVER"
docker rm temp-container
sudo chroot /opt/gcc-"$GCCVER"  gcc --version # test if it seems to work with chroot

printf '[gcc-%s]\ndescription=gcc compiler\ntype=directory\ndirectory=%s\nusers=%s\n' "$GCCVER" /opt/gcc-"$GCCVER" "$USER" | sudo tee /etc/schroot/chroot.d/gcc-"$GCCVER">/dev/null
schroot --chroot gcc-"$GCCVER" -- gcc --version # test if it seems to work with schroot

The first thing I tested was the possible performance hit.

time schroot --chroot gcc-"$GCCVER" -- gcc --version >/dev/null

real    0.460
user    0.280
sys     0.109

Which, unfortunately, is high. Even higher than docker exec, so it does not solve the problem I had.

It felt wrong that the performance hit was so high, so I looked again in the man page and found the following section:

Session actions:
  --automatic-session             Begin, run and end a session automatically
                                  (default)
  -b [ --begin-session ]          Begin a session; returns a session ID
  --recover-session               Recover an existing session
  -r [ --run-session ]            Run an existing session
  -e [ --end-session ]            End an existing session

TIL that schroot has sessions never noticed them before.

Thus, similarly to docker, it is possible to reuse them. (It would be more correct to say that docker, like schroot, has sessions, but let’s keep it this way as of today probably most people know docker but not schroot)

schroot --begin-session --chroot gcc-"$GCCVER" --session-name gcc-"$GCCVER"-session
time schroot --run-session --chroot gcc-"$GCCVER"-session gcc --version >/dev/null
real    0.050
user    0.037
sys     0.010

There is still overhead, but it is significantly smaller, this might be actually negligible when compiling, even if I have some doubt about it.

Apparently, there is little to no documentation to performance consideration, as the world seems to agree that chroot, like docker should not affect performance.

My gcc.sh now looks like

#!/bin/sh

exec schroot --run-session --chroot gcc-9-session --directory "$PWD" -- gcc "$@";

And lets build libressl again:

schroot --begin-session --chroot gcc-9 --session-name gcc-9-session;
rm -rf /path/to/libressl/build.chroot;
cmake -S /path/to/libressl/ -B build.docker.shell -DCMAKE_TOOLCHAIN_FILE=toolchain-docker-shell.cmake --debug-trycompile;
time cmake --build /path/to/libressl/build.chroot
time to build libressl
real    1:07.97
user    3:03.81
sys     42.810
schroot --end-session --chroot gcc-9-session

Hurra. That’s a factor of "only" 1.5 (approx).

Notice that schroot does not come for free. I did, for example, not mount my home directory (actually I forgot it), yet the project compiled successfully. This is because schroot, by default, setups an environment where some locations are already mounted, like my home directory (where I checked out the project).

There are optimization possibilities, because when using chroot directly

time sudo chroot /opt/gcc-"$GCCVER"  gcc --version >/dev/null
real    0.016
user    0.015
sys     0.000

The performance hit decreased further.

To recapitulate.

By using (s)chroot we can reuse the docker images.

It means we might have fewer maintenance issues, as we can use the images directly of the GCC team, and integrate them with the host environment. The overhead when compiling is much smaller compared to docker, making it a possible viable solution (this might depend on the compile times of the project and other properties, for example how long is the meantime of a single GCC invocation).

The docker image is not a black box as before, as we took it apart. Granted no changes where necessary, but this might not be true for all images.

schroot has some automation, for example, it mounts the home directory, adds the user of the host system and so on.

On the one hand, adding the user avoids possible issues with file permissions, on the other hand, leveraging on automatically added directories might cause issues if the source code is not located under the home directory.

I did not find a runtime option for mounting a directory, something similar to docker --volume, so, for now, I’m happy with letting schroot automatically mount my home directory.

Conclusion: a good (not ideal) local solution, but if you want to use some cloud provider, it won’t help.

But I did never suggest there would have been a silver-bullet.

I believe the easiest/stablest solution is still to package the toolchain, and eventually use those packages in (s)chroot, docker or other environments.