# My Adventure on cross-compilation to arm musl static binary My adventure starts simply with a request from one of my friends. They were running some Python scripts on a smart network switch to monitor the traffic. However, it was a very limiting environment as only the packages curated by the company were available for use. Although they were given full control of the device, unable to being run any code or software did harm them a lot. For instance, we wanted to run a monitoring code for [Phidgets22](https://www.phidgets.com/docs/Phidget22). Unfortunately, the libraries for this hadn't been vetted by the company so they were unable to use it. After hearing about their situation, I thought of cross-compilation. I have used C/C++ for some time in the past and was well aware of the toolchains involved. Also, I am aware of cross-compilation which I use quite often in languages like Golang and dotnet. However, I had never got to cross-compiling on the grandfather language yet. So I simply suggested to my friend that he try C and leave me with the cross-compilation step. The first step of having a C code that interfaces with Phidget22 to get sensor data was actually the easiest part and it took us barely an hour to get working on our regular laptops. After this came the `monumental` task of cross-compiling. We started with figuring out the architecture of the Network Switch. I was aware that the switch used some sort of ARM CPU. For me as a starting step and a method to test the architecture of the CPU, I thought of starting by compiling the basic `helloworld.c` program. ```c #include int main() { printf("Hello, World!\n"); } ``` The first step to cross-compiling was setting up the toolchain in the system. I could have searched for the toolchain on the package manager for my distro but I was also aware that building binary for C is not just compiling. We also have to deal with linking the binary with the correct tool and the correct library compiled for that architecture. Based on all of this and the fact that I am quite familiar with docker and container for such isolation, I searched for a docker image with my requirements and found [dockcross](https://github.com/dockcross/dockcross). Given that the project has quite recent activities, and my adventure this time was just to get the cross-compilation I decided to use their provided images. Perhaps configuring my own image would be a possible future adventure. After consulting the docs and understanding my requirements I choose to compile using `armhf musl`. The reason for choosing `armhf` was simple as I heard we previously did try to compile using a Raspberry Pi which was running `aarch64` and that didn't seem to work. As for why `musl`, from my previous experience, I knew `gnu` and its networking libraries made use of dynamic `dopen` calls in its code which meant that using it for static linking was not the best idea. Even most modern languages like Golang seem to prefer `musl` when creating a static binary. So, I was off to work with it. ```bash $ docker run --rm dockcross/linux-armv7l-musl > ./dockcross-linux-armv7l-musl $ chmod u+x ./dockcross-linux-armv7l-musl $ ./dockcross-linux-armv7l-musl bash ``` The first step was to compile the hello world program, I tried creating both the dynamic and statically linked binary as, ```bash $ armv7l-linux-musleabihf-gcc helloworld.c -o helloworld $ armv7l-linux-musleabihf-gc -static helloworld.c -o helloworld-static ``` Now running `file` on generated binaries, we see the following output. ```bash $ file helloworld helloworld: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-armhf.so.1, not stripped $ file helloworld-static helloworld-static: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), static-pie linked, with debug_info, not stripped ``` The moment of truth. We transfer the files and run them on the switch. Starting with the dynamically linked binary. ```bash $ ./helloworld error: no such file or directory ``` No luck on this, then the statically linked binary, and ..... ```bash $ ./helloworld-static Hello, World! ``` Whew, at least this worked. So off to compiling the actual Phidget22 code. Since I had already compiled the Phidget22 code previously for running on our machines, I knew that it had a dependency on `libphidget22` library. While we were able to find an `armhf` build of the library, it wasn't built for `musl`. So I had to start by building `libphidget22` for `musl`. The build tool used by `libphidget22` was `autoconf`. So on obtaining the source code, I got started with, ```bash $ autoreconf --force --install ``` This ensured that the `autoconf` was properly setup with the build environment. Then after a quick look into the options available using `./configure --help` I was off to compilation with, ```bash $ export TOOL=armv7l-linux-musleabihf $ export PREFIX=/usr/xcc/armv7l-linux-musleabihf/armv7l-linux-musleabihf/ $ ./configure --enable-static --disable-shared --host=$TOOL --prefix=$PREFIX ``` Here the options, `--enable-static --disable-shared` allow me to generate libraries to be used for static linking. Similarly `--host` specifies the host system where the final binary will run, which is in contrast to the build system where the binary is compiled. This is always specified by a triplet. The triplet here specifies that we are compiling for `armv7` architecture running `linux` system, with `musl abi`. There is also `hf` tag that is `arm` specific. The triplet for the build system is not specified here but would have been, `x86_64-linux-gnu` . As my system is running a 64-bit x86 processor with GNULinux. Unfortunately, not everything was sunshine and rainbow as the compilation failed with an error message, ```bash configure: error: Missing libusb! ``` Turns out Phidget22 is also capable of interfacing using USB, and while that is a feature that isn't used by us it wasn't possible to disable it during the compilation process using autoconf's defines. So, we had to compile `libusb` too. In hindsight, I could have visited Alpine Linux's package repository and got the library from there. But at the time, I was at that time too invested in cross-compiling by myself that I didn't think ahead. After getting the source code for `libusb` I was happy to find that they also used `autoconf` as a build system and I could employ similar techniques. So once again I restarted the process with, ```bash $ autoreconf --force --install $ ./configure --enable-static --disable-shared --host=$TOOL --prefix=$PREFIX ``` ..... and a failure once again. ```bash configure: error: udev support requested byt libudev header not installed ``` Ok, so we are missing `libudev` now? Another headhunt later I found out that `libudev` has already been merged with `systemd` and that if I wanted to compile the missing libraries I might have to build `systemd`. Now, this is a task too big just for running Phidgets on a network switch. I spent hours trying to figure out ways of only compiling `udev` or finding the libraries to no avail. The next day, I went through the options for `libusb` again. Lo and behold, there, under optional features, what do I find? ```bash Optional Features: .... --enable-udev use udev for device enumeration and hotplug support (recommnded) [default=yes] .... ``` well.... turns out I didn't have to use `udev` after all. All the hours I spent searching for answers could have been better resolved by looking better once again. *Sigh* So now armed with this knowledge I built and installed `libusb` , ```bash $ ./configure --enable-static --disable-shared --disable-udev --host=$TOOL --prefix=$PREFIX $ make -j8 $ sudo make install ``` And ... done! Now I can finally build `libphidget22` as, ```bash $ ./configure --enable-static --disable-shared --host=$TOOL --prefix=$PREFIX $ make -j8 $ sudo make install ``` Now, with all the dependencies ready, it was just a matter of running, ```bash $ armv7l-linux-musleabihf-gcc -ggdb3 -Wall -s -static -o phidget_motion_static main.c -lphidget22 -lusb-1.0 ``` and? ```bash $ file phidget_motion_static phidget_motion_static: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), static-pie linked, stripped ``` Finally, just what we needed. This entire binary is also only `1.6MB`. Running it on the Switch also gave us the required sensor values. **Perfect**. With this my adventure is complete.