RPi chroot and cross compiling (Part 1)

RPis are fantastic machines, but compared to using my PC, they’re a fraction of the performance, obviously. So, it’s convenient to be able to develop locally with a much faster compiler with much more RAM, then deploy on the Pi.

The Pi-3 is remarkably fast with it’s 4 processors, but with only 1G of RAM, I’ve had some not huge C++ programs with very extensive debugging enabled require more RAM than the machine has to compile. Trying to compile 4 things at once is a recipe for disaster.

So without further ado…

Actually there’s quite a lot of ado. This isn’t so much a guide as a brain dump of stuff I learned as I went along while skipping out the really tedious dead ends. You might want to skim read and skip around a bit.

Running Raspian in a QEMU emulator.

Couldn’t get it to work, sorry! It’s also the least interesting option.

Running an RPi-like chroot

This uses linux’s binfmt facility, to run dispatch non “native” code to various executors. Cargo cult instructions on Ubuntu are:

sudo apt-get install qemu binfmt-support qemu-user-static

Now mount the disk image. The shell transcript should look like:

$ sudo kpartx -av 2016-05-27-raspbian-jessie-lite.img
add map loop0p1 (252:0): 0 129024 linear /dev/loop0 8192
add map loop0p2 (252:1): 0 2572288 linear /dev/loop0 137216
$ sudo mount /dev/mapper/loop0p2 /mnt/

Note that kpartx reads the partition table and sets up a loopback for each partition. Then you mount the right partition on /mnt, for example to get at it.

Now, you need one more bit of setup to be able to chroot: you have to put the ARM emulator in the chroot so it can be accessed. But first, run:

update-binfmts --display

and examine the output. In there should be:

qemu-arm (enabled):
     package = qemu-user-static
        type = magic
      offset = 0
       magic = \x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00
        mask = \xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff
 interpreter = /usr/bin/qemu-arm-static
    detector =

That pretty much says that when the kernel sees the right header, it should execute a the program using the “/usr/bin/qemu-arm-static” binary. So, set that up:

sudo cp /usr/bin/qemu-arm-static /mnt/usr/bin

If you don’t set it up, you’ll get a “no such file or directory” error when you try to chroot, which is caused by the kernel failing to find qemu-arm-static. Note that they have rather sensibly compiled it as fully static, so you don’t need to bring the entire dynamic runtime into the chroot.

So, now execute (my machine is called sheepfrog by the way) and see the following transcript:

$ sudo chroot /mnt
root@sheepfrog:/# g++ --version
g++ (Raspbian 4.9.2-10) 4.9.2
Copyright (C) 2014 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO


like… woah. Now, a benchmark

From bash:

time for ((i=0; i < 10000; i++)); do : ; done

My RPi3 gives 0.4 seconds, my i7 Q820 @ 1.73GHz (yes, it is 2017) running the emulator gives 0.942 seconds, which is a fair bit slower. If you have a half way modern machine then you’ll be neck and neck, probably faster than native. Of course, even my old laptop has much more RAM and vastly faster storage, which makes a huge difference.

Natively my machine takes, er, 2.37 seconds! That’s because I’ve set up bash to change the xterm title, so it’s flogging the x server in that loop. Removing that gives 0.065 seconds. That’s a bit more sensible.

Running the chroot and installing things

Anyway, this isn’t the real reason for doing this. The main point of setting up the chroot is to be able to chroot into is and run apt-get to install rpi packages. Then eventually one can install packages etc and use them with a native x86 cross compiler. So first, let’s get going!

~ #chroot /mnt
root@sheepfrog:/# apt-get update
qemu: uncaught target signal 4 (Illegal instruction) - core dumped
Illegal instruction (core dumped)


Turns out the solution isn’t so bad. The cargo-cult hacking is to comment out the only line in /mnt/etc/ld.so.preload so the file looks like this:


Now, apt-get update works fine and you can install the all the -dev packages IN THE WORLD!

The eagle-eyed might wonder HOW apt-get works, since I never copied over the resolv.conf file, so how on earth can it fetch anything? The answer is that Rapspian provides a default resolv.conf that points at google’s handy nameservers!

¬†Wait that all sounds unnecessary. It’s unnecessary, right?

Yes, that is all indeed unnecessary. apt and dpkg is flexible enough that it can run as if it were in a chroot without actually being in a chroot. That means you can use the native apt-get to manipulate the contents of /mnt. If you’re not running a Debian derived distro, then you’ll have to somehow install apt-get. Probably the easiest way is to debootstrap a native installation and chroot into it.

Anyway apparently the magic option is -o Dir=something, so:

~ #sudo apt-get -o Dir=/mnt/   update
Hit http://archive.raspberrypi.org jessie InRelease
Hit http://mirrordirector.raspbian.org jessie InRelease
W: Failed to fetch http://mirrordirector.raspbian.org/raspbian/dists/jessie/InRelease  Unable to find expected entry 'main/binary-amd64/Packages' in Release file (Wrong sources.list entry or malformed file)

W: Failed to fetch http://archive.raspberrypi.org/debian/dists/jessie/InRelease  Unable to find expected entry 'main/binary-amd64/Packages' in Release file (Wrong sources.list entry or malformed file)

E: Some index files failed to download. They have been ignored, or old ones used instead.

Can you hear the “bwaaa bwaaaaa” trombone noise? I can hear the trombone noise.

Apparently despite being in a subtree set up as ARM, it’s looking for packages based on the host architecture, not some config file somewhere. The key is obviously going to tell it that we really so want ARM.

I’ve not managed to figure it out yet. Adding:

 -o apt::architecture=armhf -o DPkg::Options::='--admindir=/mnt/var/lib/dpkg'

kind of helps. This is supposed to tell dpkg to look in the right place too., apt is a manager which wraps dpkg, so you have to configure both of them. But then I get huge numbers of dependency errors when I try to install anything. I suspect it’s still looking in the wrong places. And sure enough, strace verifies that it’s still reading files from /etc, but not anything terribly important from /var. Strange.

Cross compiling.

That comes in part 2!