chroot as docker alternative for building
As mentioned in the previous article, docker does not fit my needs.
It is too complex to maintain, has a high-performance cost, and all provided features (apart for filesystem isolation) are roughly useless.
So the solution seemed to compile and package every version of compiler and toolchain by myself. Using a normal binary on the host has many advantages, but it is a maintenance burden.
So I checked If I could use chroot, which is one of the many older tools which docker is based on.
The nice thing about this approach is that I can probably reuse containers, even if I have to inspect how they work. On the other hand, this is one of the issues that I wanted to avoid.
As one of the things I dislike when using docker is running programs as an administrator, in order not to redo the same error, I’m going to use schroot (hopefully available on most GNU/Linux distributions).
Creating the chroot environment is fortunately very easy:
GCCVER=9 docker create --name temp-container --rm gcc:"$GCCVER" sudo docker cp temp-container:/ /opt/gcc-"$GCCVER" docker rm temp-container sudo chroot /opt/gcc-"$GCCVER" gcc --version # test if it seems to work with chroot printf '[gcc-%s]\ndescription=gcc compiler\ntype=directory\ndirectory=%s\nusers=%s\n' "$GCCVER" /opt/gcc-"$GCCVER" "$USER" | sudo tee /etc/schroot/chroot.d/gcc-"$GCCVER">/dev/null schroot --chroot gcc-"$GCCVER" -- gcc --version # test if it seems to work with schroot
The first thing I tested was the possible performance hit.
time schroot --chroot gcc-"$GCCVER" -- gcc --version >/dev/null real 0.460 user 0.280 sys 0.109
Which, unfortunately, is high. Even higher than
docker exec, so it does not solve the problem I had.
It felt wrong that the performance hit was so high, so I looked again in the man page and found the following section:
Session actions: --automatic-session Begin, run and end a session automatically (default) -b [ --begin-session ] Begin a session; returns a session ID --recover-session Recover an existing session -r [ --run-session ] Run an existing session -e [ --end-session ] End an existing session
TIL that schroot has sessions never noticed them before.
Thus, similarly to docker, it is possible to reuse them. (It would be more correct to say that docker, like schroot, has sessions, but let’s keep it this way as of today probably most people know docker but not schroot)
schroot --begin-session --chroot gcc-"$GCCVER" --session-name gcc-"$GCCVER"-session time schroot --run-session --chroot gcc-"$GCCVER"-session gcc --version >/dev/null real 0.050 user 0.037 sys 0.010
There is still overhead, but it is significantly smaller, this might be actually negligible when compiling, even if I have some doubt about it.
Apparently, there is little to no documentation to performance consideration, as the world seems to agree that
docker should not affect performance.
gcc.sh now looks like
#!/bin/sh exec schroot --run-session --chroot gcc-9-session --directory "$PWD" -- gcc "$@";
And lets build libressl again:
schroot --begin-session --chroot gcc-9 --session-name gcc-9-session; rm -rf /path/to/libressl/build.chroot; cmake -S /path/to/libressl/ -B build.docker.shell -DCMAKE_TOOLCHAIN_FILE=toolchain-docker-shell.cmake --debug-trycompile; time cmake --build /path/to/libressl/build.chroot time to build libressl real 1:07.97 user 3:03.81 sys 42.810 schroot --end-session --chroot gcc-9-session
Hurra. That’s a factor of "only" 1.5 (approx).
Notice that schroot does not come for free. I did, for example, not mount my home directory (actually I forgot it), yet the project compiled successfully. This is because schroot, by default, setups an environment where some locations are already mounted, like my home directory (where I checked out the project).
There are optimization possibilities, because when using
time sudo chroot /opt/gcc-"$GCCVER" gcc --version >/dev/null real 0.016 user 0.015 sys 0.000
The performance hit decreased further.
By using (s)chroot we can reuse the docker images.
It means we might have fewer maintenance issues, as we can use the images directly of the GCC team, and integrate them with the host environment. The overhead when compiling is much smaller compared to docker, making it a possible viable solution (this might depend on the compile times of the project and other properties, for example how long is the meantime of a single GCC invocation).
The docker image is not a black box as before, as we took it apart. Granted no changes where necessary, but this might not be true for all images.
schroot has some automation, for example, it mounts the home directory, adds the user of the host system and so on.
On the one hand, adding the user avoids possible issues with file permissions, on the other hand, leveraging on automatically added directories might cause issues if the source code is not located under the home directory.
I did not find a runtime option for mounting a directory, something similar to docker
--volume, so, for now, I’m happy with letting
schroot automatically mount my home directory.
Conclusion: a good (not ideal) local solution, but if you want to use some cloud provider, it won’t help.
But I did never suggest there would have been a silver-bullet.
I believe the easiest/stablest solution is still to package the toolchain, and eventually use those packages in (s)chroot, docker or other environments.
Do you want to share your opinion? Or is there an error, some parts that are not clear enough?
You can contact me here.