The C logo, under public domain.

Botstrapping and cross-compiling

6 - 8 minutes read, 1514 words
Categories: c
Keywords: arm c cross-compiling embedded linux

Maybe it’s me, but I like to know the environment I work in and add facilities for helping in my daily workflows. I have more than 50 executables (mostly shell script) under ~/bin, some are used daily, others once or twice a year. The main advantage is that for both cases, shell scripts provide very good documentation of how I do accomplish specific tasks.

This is one of the main reasons for using a computer: automating tasks, reduce repetitive work. If not, I tend to prefer human interaction, as generally human beings tend to be friendlier than a robot.

So when I get a new device, a new environment, or when I have to work from a different workstation, one of the first things I do is looking around what’s available and eventually add those little programs I find so useful.

I even have a USB stick with appimages and scripts for GNU/Linux and portable applications for windows.

In some environments, there are policies that do not allow installing new programs, downloading executables, and inserting SUB sticks.

Fortunately, both Windows and POSIX-based systems have shells, and I am still allowed to open a text editor and save textual files.

While the language given by the shell is not always ideal, it is possible to write many utilities with it. If lucky it is possible to let do the hard work to external programs and glue them together in your script.

This is coincidentally part of the Unix philosophy; provides little specialized tools that can be put together easily.

Normally I could always leverage programs available on the operating systems, most scripts just call some programs and glue them together. Of course, being able to install third-party programs makes everything easier, why reinvent the wheel if there is no enough time?

But this year I had to work in a (for me) new environment. A Linux distribution, missing most of the coreutils executables (not even cut and sort where available!), and with no package manager.

The /bin/sh directory is painfully empty. Only the absolute minimum is available: an old busybox and half a dozen of other executables.

Also, the programs that are available (like kill and chmod), are crippled, missing many options, and of course, other useful programs, like grep or awk, were missing completely.

Oh, and because there is no bash, there is no autocompletion and no history.

While it might seem like a very improbable scenario to encounter such a device, nearly everyone has a device with approximately the same limitations: android phones.

It is possible to enable access to the shell, but it is a very stripped-down environment. If you can install programs, termux provides a package manager and nearly all programs one can expect on a GNU/Linux distribution. Unless you have an older phone, as termux does not support versions previous to Android 7.

Maybe it’s just me, but the idea of working in such an environment is horrible.

The only possibility to work efficiently, without changing the environment, is offloading as much work as possible on a separate PC. But as tools like ssh are missing too, even copying a file is a process difficult to automate (and of course tar and other archiving utilities are missing).

So either I could do everything by hand, or try to offload as much as possible.

I know by experience that by doing everything by hand I will do the most improbable and dumbest errors, and burn a lot of time and energy solving non-issues multiple times. I tried offloading as much work as possible, but just synchronizing content and trying to debug the issue is such a hurdle that I decided I had to come up with something else.

Bootstrapping my toolbox on that machine.

The downside is that it will cost some time to get something useful.

The upside is that I do need to do it only once.

If I do everything by hand, I will need more time for every task. And every time, if something goes wrong, there will be the doubt if I had done something differently.

Cross-compiling grep

The first tool I noticed I would very like to have is grep. Searching in text files or piping output is such a common operation that I decided I would try to compile it for the platform.

The first issue is that this platform has a different architecture: 32-bit arm.

Thus I need to cross-compile.

Fortunately, both clang and GCC can compile for arm, so I installed arm-linux-gnueabi-gcc package on my GNU/Linux machine, and after extracting the sources of grep I’ve executed .configure and make.

> ./configure CC=arm-linux-gnueabi-gcc --host=arm-none-linux-gnueabi && make
# configure and make output ...
> file src/grep
src/grep: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/, BuildID[sha1]=384ab6445e534210b6552592459dc745ea098a8c, for GNU/Linux 3.2.0, not stripped

I copied the binary on the device, but it did not start correctly. Turns out, the glibc on the device is too old for the binary I just compiled.

Thus I tried to compile statically

> ./configure CC=arm-linux-gnueabi-gcc --host=arm-none-linux-gnueabi CFLAGS="-static" && make
# configure and make output ...
> file src/grep
src/grep: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, BuildID[sha1]=590ab26ed4124bbe2592f1f963ebfc06d37151e4, for GNU/Linux 3.2.0, not stripped

And copied again the compiled binary to the device. Again, it would not start. The kernel on the device, which cannot be updated, is too old.

I could not find any option for creating binaries for older kernels in GCC, apparently, I need to recompile glibc with an the appropriate --enable-kernel switch.

Fortunately, I found a toolchain for this device. I’ve "installed" (copied under /opt) it on my PC, and could use it for building grep

./configure --host=arm-none-linux-gnueabi CC=/opt/my-toolchain/bin/arm-none-linux-gnueabi-gcc CFLAGS="-w -static" && make

And finally, I’ve got a working binary.

I also tried using clang with the toolchain, but with no luck, as it seems that configure uses compiler flags not supported by clang when cross-compiling (or probably I’m doing something wrong and still have not figured it out):

./configure --host=arm-none-linux-gnueabi CC=clang CFLAGS="-w -static --target=arm-none-linux-gnueabi --gcc-toolchain=/opt/mytoolchain --sysroot=/opt/my-toolchain/arm-none-linux-gnueabi/libc" && make

Notice that grep has some optional dependencies, like PCRE. While C (and C++) are ubiquitous, unfortunately, there is no standard practice or convention for handling dependencies (actually not even for building, even less for cross-compiling).

Thus to keep things simple, at least at the beginning, I’ll try to avoid binaries with dependencies.

Crosscompiling GNU coreutils

After grepping happily on my crippled device, I decided that it would be great to cut some text, and noted that cut is missing too!

As I’ve managed to get grep working, I’ve decided to see how difficult it is to download the GNU coreutils package.

Turns out it is a little more complex. First I needed to download some dependencies for building, then I could ./configure

> sudo aptitude install automake autopoint bison gperf rsync
> ./bootstrap
> ./configure CC=arm-linux-gnueabi-gcc --host=arm-none-linux-gnueabi && make

Unfortunately, configure for the target platform does not work. Apparently, it tries to execute a compiled file: src/make-prime-list: cannot execute binary file: Exec format error.

The solution, found by trial-and-error (not great, I know…​), is to compile first for the host platform, and afterward for the target.

> ./bootstrap
> ./configure && make
> make clean
> ./configure CC=arm-linux-gnueabi-gcc --host=arm-none-linux-gnueabi && make

I decided to report it as a bug, but then noticed it is a known bug from 2012.


Now I have cut, du, sort, md5sum, a non-crippled chown, and a lot more utilities, some of them I have never used and probably do not even need.

With those basic tools, I could finally create some useful scripts for automating manual tasks, like copying some part of an output and pass it to another program, monitoring some activities, and so on.

As the POSIX shell is Turing complete it would have been possible to accomplish the same task without cross-compiling anything. It would only have been much more difficult to write what would have been a simple script.

Sure I would love to have more complex programs like ssh and tmux, I’ll see what will be the next step.

Now I can work more efficiently, and some tasks that were tedious and error-prone can be done fast and correctly anytime with no effort, just by providing some building blocks available in every sane GNU/Linux distribution.

It is always worth improving the working environment, it is the common denominator for most tasks. Improving, and especially adding tools available in other environments makes it easier to reuse programs (or use different programs with similar interfaces), no matter how small or big, and work more efficiently.