Enhanced constexpr

In an earlier post I wrote about using constexpr to generate compile time lookup tables. Unfortunately it turns out that that was dependent on a GCC extension. In actual C++14, pretty much nothing in <cmath> is constexpr.

My brother and I have written a proposal, currently P0533R0 to introduce a number of constexpr functions into <cmath>. Making functions constexpr is slightly tricky because some of them may set the global variable errno and/or floating point exception flags (effectively global variables),  and so it needs to be a language solution, not a pure library one. However, the basic arithmetic operators also fall into similar traps, so it’s not that bad.

The initial paper is to constexpr-ify all “simple” functions which don’t have pointer arguments. In this case, simple is defined as “closed over the extended rationals”, which includes the standard arithmetic operators, comparison operators and things like floor() and ceil(). It turns out there are actually rather a lot of these including many I’d never heard of before ploughing through <cmath>.

I think it should be straightforward to implement since GCC and LLVM both already do compile time evaluation of even non-simple functions (e.g. sqrt), the former actually making them constexpr as an extension of C++14, the latter only using them in the optimizer.




Overwrite a file but only if it exists (in BASH)

Imagine you have a line in a script:

cat /some/image.img > /dev/mmcblk0

which dumps a disk image on to an SD card. I have such a line as part of the setup. Actually, a more realistic one is:

pv /some/image.img | sudo 'bash -c cat >/dev/mmcblk0'

which displays a progress bar and doesn’t require having the whole script being run as root. Either way, there’s a problem: if the SD card doesn’t happen to be in when that line runs, it will create /dev/mmcblk0. Then all subsequent writes will go really fast (at the speed of the main disk), and you will get confused and sad when none of the changes are reflected on the SD card. You might even reboot which will magically fix the problem (/dev gets nuked). That happened to me 😦

The weird, out of place dd tool offers a nice fix:

pv /some/image.img | sudo dd conv=nocreat of=/dev/mmcblk0

You can specify a pseudo-conversion, which tells it to not create the file if it doesn’t already exist. It also serves the second purpose as the “sudo tee” idiom but without dumping everything to stdout.

A simple hack but a useful one. You can do similar things like append, but only if it exists, too. The incantation is:

dd conv=nocreat,notrunc oflag=append of=/file/to/append/to

That is, don’t create the file, don’t truncate it on opening and append. If you allow it to truncate, then it will truncate then append, which is entirely equivalent to overwriting but nonetheless you can still specify append without notrunc.

Warn or exit on failure in a shell script

Make has a handy feature where when a rule fails, it will stop whatever it’s doing. Often though you simply want a linear list of commands to be run in sequence (i.e. a shell script) with the same feature.

You can more or less hack that feature with BASH using the DEBUG trap. The trap executes a hook before every command is run, so you can use it to test the result of the previous command. That of course leaves the last command dangling, so you can put the same hook on the EXIT trap which runs after the last command finishes.

Here’s the snippet and example which warns (rather than exits) on failure:

function test_failure(){
  #Save the exit code, since anything else will trash it.
  if [ $v != 0 ]
    echo -e Line $LINE command "\e[31m$COM\e[0m" failed with code $v
  #This trap runs before a command, so we use it to
  #test the previous command run. So, save the details for
  #next time.

#Set up the traps
trap test_failure EXIT
trap test_failure DEBUG

#Some misc stuff to test.
echo hello
sleep 2 ; bash -c 'exit 3'

echo world

echo what > /this/is/not/writable

echo the end

Running it produces:

$ bash errors.bash 
Line 21 command bash -c 'exit 3' failed with code 3
Line 25 command false failed with code 1
errors.bash: line 27: /this/is/not/writable: No such file or directory
Line 27 command echo what > /this/is/not/writable failed with code 1
the end

RPi chroot and cross compiling (Part 1)

RPis are fantastic machines, but compared to using my PC, they’re a fraction of the performance, obviously. So, it’s convenient to be able to develop locally with a much faster compiler with much more RAM, then deploy on the Pi.

The Pi-3 is remarkably fast with it’s 4 processors, but with only 1G of RAM, I’ve had some not huge C++ programs with very extensive debugging enabled require more RAM than the machine has to compile. Trying to compile 4 things at once is a recipe for disaster.

So without further ado…

Actually there’s quite a lot of ado. This isn’t so much a guide as a brain dump of stuff I learned as I went along while skipping out the really tedious dead ends. You might want to skim read and skip around a bit.

Running Raspian in a QEMU emulator.

Couldn’t get it to work, sorry! It’s also the least interesting option.

Running an RPi-like chroot

This uses linux’s binfmt facility, to run dispatch non “native” code to various executors. Cargo cult instructions on Ubuntu are:

sudo apt-get install qemu binfmt-support qemu-user-static

Now mount the disk image. The shell transcript should look like:

$ sudo kpartx -av 2016-05-27-raspbian-jessie-lite.img
add map loop0p1 (252:0): 0 129024 linear /dev/loop0 8192
add map loop0p2 (252:1): 0 2572288 linear /dev/loop0 137216
$ sudo mount /dev/mapper/loop0p2 /mnt/

Note that kpartx reads the partition table and sets up a loopback for each partition. Then you mount the right partition on /mnt, for example to get at it.

Now, you need one more bit of setup to be able to chroot: you have to put the ARM emulator in the chroot so it can be accessed. But first, run:

update-binfmts --display

and examine the output. In there should be:

qemu-arm (enabled):
     package = qemu-user-static
        type = magic
      offset = 0
       magic = \x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00
        mask = \xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff
 interpreter = /usr/bin/qemu-arm-static
    detector =

That pretty much says that when the kernel sees the right header, it should execute a the program using the “/usr/bin/qemu-arm-static” binary. So, set that up:

sudo cp /usr/bin/qemu-arm-static /mnt/usr/bin

If you don’t set it up, you’ll get a “no such file or directory” error when you try to chroot, which is caused by the kernel failing to find qemu-arm-static. Note that they have rather sensibly compiled it as fully static, so you don’t need to bring the entire dynamic runtime into the chroot.

So, now execute (my machine is called sheepfrog by the way) and see the following transcript:

$ sudo chroot /mnt
root@sheepfrog:/# g++ --version
g++ (Raspbian 4.9.2-10) 4.9.2
Copyright (C) 2014 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO


like… woah. Now, a benchmark

From bash:

time for ((i=0; i < 10000; i++)); do : ; done

My RPi3 gives 0.4 seconds, my i7 Q820 @ 1.73GHz (yes, it is 2017) running the emulator gives 0.942 seconds, which is a fair bit slower. If you have a half way modern machine then you’ll be neck and neck, probably faster than native. Of course, even my old laptop has much more RAM and vastly faster storage, which makes a huge difference.

Natively my machine takes, er, 2.37 seconds! That’s because I’ve set up bash to change the xterm title, so it’s flogging the x server in that loop. Removing that gives 0.065 seconds. That’s a bit more sensible.

Running the chroot and installing things

Anyway, this isn’t the real reason for doing this. The main point of setting up the chroot is to be able to chroot into is and run apt-get to install rpi packages. Then eventually one can install packages etc and use them with a native x86 cross compiler. So first, let’s get going!

~ #chroot /mnt
root@sheepfrog:/# apt-get update
qemu: uncaught target signal 4 (Illegal instruction) - core dumped
Illegal instruction (core dumped)


Turns out the solution isn’t so bad. The cargo-cult hacking is to comment out the only line in /mnt/etc/ld.so.preload so the file looks like this:


Now, apt-get update works fine and you can install the all the -dev packages IN THE WORLD!

The eagle-eyed might wonder HOW apt-get works, since I never copied over the resolv.conf file, so how on earth can it fetch anything? The answer is that Rapspian provides a default resolv.conf that points at google’s handy nameservers!

 Wait that all sounds unnecessary. It’s unnecessary, right?

Yes, that is all indeed unnecessary. apt and dpkg is flexible enough that it can run as if it were in a chroot without actually being in a chroot. That means you can use the native apt-get to manipulate the contents of /mnt. If you’re not running a Debian derived distro, then you’ll have to somehow install apt-get. Probably the easiest way is to debootstrap a native installation and chroot into it.

Anyway apparently the magic option is -o Dir=something, so:

~ #sudo apt-get -o Dir=/mnt/   update
Hit http://archive.raspberrypi.org jessie InRelease
Hit http://mirrordirector.raspbian.org jessie InRelease
W: Failed to fetch http://mirrordirector.raspbian.org/raspbian/dists/jessie/InRelease  Unable to find expected entry 'main/binary-amd64/Packages' in Release file (Wrong sources.list entry or malformed file)

W: Failed to fetch http://archive.raspberrypi.org/debian/dists/jessie/InRelease  Unable to find expected entry 'main/binary-amd64/Packages' in Release file (Wrong sources.list entry or malformed file)

E: Some index files failed to download. They have been ignored, or old ones used instead.

Can you hear the “bwaaa bwaaaaa” trombone noise? I can hear the trombone noise.

Apparently despite being in a subtree set up as ARM, it’s looking for packages based on the host architecture, not some config file somewhere. The key is obviously going to tell it that we really so want ARM.

I’ve not managed to figure it out yet. Adding:

 -o apt::architecture=armhf -o DPkg::Options::='--admindir=/mnt/var/lib/dpkg'

kind of helps. This is supposed to tell dpkg to look in the right place too., apt is a manager which wraps dpkg, so you have to configure both of them. But then I get huge numbers of dependency errors when I try to install anything. I suspect it’s still looking in the wrong places. And sure enough, strace verifies that it’s still reading files from /etc, but not anything terribly important from /var. Strange.

Cross compiling.

That comes in part 2!


Can it be true? That I hold here in my mortal hands, a splat of purest crud?

Today, a semi-successful experiment. I tried to make a small arc furnace using an arc welder, a graphite crucible and some inanimate carbon gouging rods. The goal was to melt aluminium successfully enough to do some casting. The idea behind using the arc welder is that it’s accessible and doesn’t require faffing around with fire, and getting the consumables brought in (gas, for example). The whole thing ought to be less messy and quicker to set up and tear down.

The furnace consists of a solid, 4kg sized graphite crucible (remarkably inexpensive) sitting in a badly welded, but very stable steel holder steel holder:


An arc furnace. The white stuff on the inside is alumina fumes which settled on the side.

The idea was to strike an arc with the crucible and a copper clad carbon rod (a gouging electrode). It kinda worked, but the arc was pretty unreliable and surprisingly weedy even on the highest setting on the welder. I had much better luck striking the arc between two carbon rods and moving that around as a heat source.


The rods are very much a consumable!

That kinda worked, and I was certainly able to get some melting (as you can see in the splat). Enough to prove the principle but not enough to actually do some casting. The summary is kind of:

  • Someone stole my flux (own brand lo-salt), so I got a lot of aluminium oxide for my troubles.
  • The welder doesn’t like the 16A breaker for the outdoor power socket. Works fine on the other 16A breaker in the basement but keeps tripping out.
  • Not enough insulation, a rather large crucible, repeated cutouts and cold weather meant I couldn’t retain enough heat to make a pour.

The thing to do now it appears is to make the furnace by hollowing out an alumina firebrick. They’re very porous and so excellent insulators (one video has the person picking up his brick with bare hands with a pool of molten aluminium in the central hole).

Nonetheless, it proves the principle. The 400 and 600A (one of each) crocodile clip style rod holders hold the carbon rods well. The rods work, strike an arc and provide aluminium melting heat.


Adventures in gluing

The thing I’m working on has some metal parts glued into a plastic case. The plastic is smooth (the mould is finished to the “Society of Plastics Engineers” 600 paper spec), and the metal is shiny chrome plated brass. I never considered either of those to be especially hard to glue, so I went for a pretty straightforward epoxy, Loctite 5 minute instant mix clear. It’s a decent enough go-to, much like any other random 5 minute clear epoxy. The short curing time is great for the impatient (me) and the clear finish is great because you can be a bit lax about the cleanup, which is especially important given the short curing time.

So, I splurged some on the metal part, shoved it in the hole and wiped off any egregious amount of excess with white spirit. A week later, I was fiddling with the part seeing how strong it was and was kind of surprised/distressed about how easily the joint broke. Like, really far too easily. And incidentally always the same way: the glue detaching from the plastic.

Then it got me thinking. I had a vague recollection that 5 minute epoxies kind of suck and come to think of it, so do clear ones. In a sort of ad-hoc way I think I’d come up with this:

Ed’s  rule of glue

Every feature added to glue makes it worse.

Fast curing? worse. Non, tinted transparent? That’ll cost you. Conductive? [sucks air through teeth]. It kind of makes sense, you can’t optimize everything at once.

That’s not to say you should never use that glue, it’s still perfectly fine for a lot of stuff, but slower curing and/or those ones that cure a nasty brown colour will almost certainly perform better. So, anyway, I tried  some polyurethane glue I had lying around and it seemed to be a bit better.

Time to experiment!

So,  which one to use? So first, the selection. There’s roughly an infinite number of glues out there, give or take a few. Some have wild and wacky properties, but even if you stick to the middle ground of generally gluing stuff to other stuff, there’s still a large selection.

So, I picked some somewhat arbitrary criteria and selected some glues that generally fitted:

  1. Specced to glue metal to plastic, bonus points for having ABS in the list.
  2. Available from local shops or next day delivery from RS
  3. Reasonable curing time to handling strength.
  4. Auto mixer if possible where applicable (better for large volumes).
  5. Not wildly expensive.
  6. Decent brands and available in the US (I’d normally be happy with RS pro stuff, but I need to be able to get it abroad easily).

The brands are mostly big, well known ones. 3M is always good. Loctite likewise seems fine. The other one, Araldite, I have a particular soft spot for. For years in the home gamer space it was pretty much synonymous with epoxy. My old DT teacher was a huge fan (rightly so) because you could fix all manner of sins with it. Countless projects were rescued with Araldite at my school.

And so, I came up with the following list:

  1. Loctite 5 minute instant mix. That’s the baseline, and my gut feeling says that other clear, fast setting epoxies won’t differ wildly.
  2. Araldite Standard. Bog-standard DIY shop glue. Slow setting (pot time of an hour, high strength in 24), dries kind of opaque yellowish.
  3. Araldite 2011. One of about a billion varieties. Similar to 2, but dries clear yellow. Allegedly slightly stronger on ABS. Also a pro glue, so comes in pro packaging which means you need one of those applicator guns and a clip on mixing nozzle for the auto mixer.
  4. Loctite 330. It’s an acrylic glue which I didn’t even know was a thing before today. You put glue on one surface an spray the activator on the other surface and put them together. Datasheet specifies 7387 or 7386 activator, but Loctite actually sell it packaged with 7388 of course. Fast cure time (30 mins to half strength) , and almost indefinite pot time, which is handy, since you don’t mix it with the activator. Not much by the way of instructions though.
  5. Araldite 2028. OK, I lied about not bothering with other fast curing clear epoxies. This one claims excellent performance on ABS, not so much in the datasheet which is not anywhere official I could find, but in this handy selection guide, which has a mysteriously chosen subset of the glues on offer. Also pro packaging with an auto-mixer.
  6. 3M ScotchWeld DP8005. Now I’ve heard about acrylic adhesives, I’m on a roll. Also RS had a relatively small selection of 3M and I felt things wouldn’t be complete without one. Claims to work well on ABS. Has an annoyingly short pot life (3 minutes and they’re not messing around).
  7. Everbuild gator glue. Bog standard polyurethane glue (basically the same as Gorilla

    I love the logo too!

    Glue). Generally considered OK for metal and plastic. Reasonable pot life and cure time. Single part, so no mixing needed. Also expands as it cures. This is not (as is popularly thought) useful for filling gaps since it expands as foam when the pockets of glue get too large. However, it does slightly splurge out so if you miss small areas, they get filled from the nearby ones. If you’re not familiar with British knock off brands, Everbuild is basically an adhesive company which makes cheaper versions of the well known stuff. So Gator glue/Gorilla glue, One Strike Filler/Pollyfilla, Stixall/No More Nails, etc etc. They also sell the delightfully named ASBO brand anti climb paint and anti graffiti painting. That makes me chuckle every time.

  8. Polycaprolacetone. I remember that sticking to everything really hard when I tried to use it. Applying it as an acetone based slurry, then reflowing it thermally after it’s dried. This is basically me messing around.


    White crud on the two snaps on the right hand side. No glue was applied there.

  9. Cyanoacrylate (generic brand). Another baseline, known to be strong and effective, but outgasses this white crud everywhere which is electrically insulating and has to be cleaned off.

Oh and the surface prep. Did I mention I forgot the surface prep before?

  1. Clean both surfaces with IPA (isopropyl alcohol).
  2. The full monty: clean with IPA, sand with 400 grit paper (3M as it happens), then clean again with IPA.
  3. As-is, with whatever crud and release agent is left on by the moulding processes.

And it’s arrived!

Swag! Lots of glue arrived


Check out the swag, including a spiffy (and necessary) applicator gun, because as you can see, the pro packages don’t have full syringes. And 3 hours of cleaning, sanding, mixing, gluing and wiping off the excess, I present:


That’s a lot of gluing and there’s more to come.

So now, I need some way to measure the force required to eject the metal bits from a case. Naturally, since I’m doing the whole thing at short notice and in rather a hurry,  I had to cobble together something with whatever was lying around or could buy locally. My solution is to go to a local DIY shop and enter a kind of trance until a solution based on the weird selection on offer presents itself. The result was this:


After a few iterations, I settled on pushing down on a set of old kitchen scales held down with cable ties and string. The lever is 4x and the scales go to 5Kg, and the extra weights on the end are 2.5Kg at 5x. Maximum force it can apply is 32.5Kgf.

It’s basically a parallel action mechanism with a long lever. I balanced it to make it neutral and marked graduations on the lever so I can put a weight at a known position to apply a known force. You can’t see the M4 screw on the underside which is embedded into the pressing bit with some brass threaded inserts. I used some 5 minute epoxy (surprise!) to glue them in, but not a clear one and the horrendous brown colour made me think I should have tried it too. Though being nasty coloured and fast curing (hard to clean off) is not a great combo. The big M10 studding isn’t going into bare pine. The holes are lined with some sort of plastic tubing which felt a lot like HDPE. Naturally it was 15mm, and they only had 14 and 16mm drills, so a lot of sanding was involved (later I found just hacking off the outer layer with a Stanley knife was faster. Oh well). It’s also quite loose (inner diameter 11mm), which is good because I can’t drill straight apparently.

Oh, also, I glued on the main pillar upside-down first time (look: it’s not symmetric) at the other end with the Gator glue (scar is visible just by the pliers). I realised after about 20 minutes and that was enough that one of the end grain to long grain joints was already stronger than the wood. Even if I find doesn’t measure up here, I’ve become a fan of the speed of polyurethane glue and strength on end grain for woodworking

Oh and check out how straight the pillar is 8-). There’s nothing to keep it straight except for the square planed end. I’m excessively pleased about that.

RTFM, n00b.

So the spare DP8005 didn’t cure in a nice solid lump, at least in not in a few hours, and it’s meant to have a pot time of about 3 minutes. I think it was a mixing problem. I’m not 100% sure I got the right nozzles and since it’s a 10:1 mix, that really matters. So I mixed some by hand, too (strongly not recommended) and tried that. But since mixing is slow and it has a short pot time, I didn’t have a chance to glue up all variants. When it’s properly mixed, this stuff sets FAST.

The first results.

Thoughts so far during gluing:

  1. The Loctite 330 activator is this yellow gunk that has to be cleaned off. It’s pretty nasty and it’s more or less impossible to clean it off interestingly shaped surfaces.
  2. Scotchweld DP8005 is made by 3M and being 3M they’ve really put the effort into the packaging. It reseals really well by plugging up the holes (unlike the Araldite which has a simple cap), with a keyed cap, so you don’t mix the two sides. Also it’s gritty! That’s apparently to keep the surfaces apart by the minimum spacing. 3M think of everything.
  3. Cyanoacrylate is and always will be a pain in the neck.
  4. My lab now smells of glue and solvents. Niiiice.

Anyway, I left them curing for 36 hours in my lab. Then I spent I don’t know how long pressing snaps out of cases and writing down the results. And then some awk code to collate and prettyprint the results of course.


                        Surface prep||     IPA clean     |clean, sand, clean |       none        
Glue                                ||    ave     min    |    ave     min    |    ave     min    
Loctite instant mix 5 minuted epoxy ||    14.7    8.8    |    12.1    9.2    |     9.1    8.0    
Araldite standard                   ||*    7.9    5.6    |*   11.2   10.0    |*    4.8    4.0    
Araldite 2011                       ||*    6.5    4.8    |    11.5   11.2    |     5.4    5.2    
Loctite 330                         ||*   12.3    7.2    |*   16.3   14.4    |*   13.6   10.4    
Araldite 2028                       ||     7.1    5.6    |     6.9    6.4    |     5.9    5.2    
Scotchweld DP8005 badly mixed       ||*    4.4    3.6    |*    4.1    4.0    |*    4.5    4.4    
Scotchweld DP8005                   ||    22.1   20.0    |                   |    17.5   13.6    
Gator glue (no water)               ||*   11.7   11.2    |*    7.5    4.8    |*    9.7    8.8    
PCA                                 ||     3.2    2.8    |     5.1    3.6    |     3.0    2.0    
Cyanoacrylate                       ||    21.0   15.7    |    22.6   18.4    |    21.6   19.7    

* = Clean off excess glue with IPA


And… ehhhhhh.

It’s a bit mediocre.

Well OK. I’ve established that it’s really important to clean the parts before gluing. Sanding seems to help a bit, mostly, but to be honest I didn’t do a great job of it.

The 5 minute epoxy came out pretty strong comparatively and although it seems strong enough it’s not exactly decisive. The CA is strong (as expected), but has outgassing problems. The DP8005 is excellent if done properly, but I think it cures a bit fast to be useful for this application. The Loctite 330 is decent, but that spray is just too horrible for this application.

My ad-hoc experiments earlier had Gator Glue doing better relative to the Loctite epoxy. I think something might be wrong there.

And wow, Araldite, you kinda suck 😦 Is this the end of my fond childhood memories?

No, seriously, RTFM, n00b.

OK, so I forgot a bunch of stuff. I forgot to dampen one of the surfaces with the polyurethane glue (Gator Glue). That might explain the so-so results. Time to repeat the experiment. This time, I also switched to genuine Gorilla glue, not because I expect it to be better, but because the results are going to be used in the US and Gator glue isn’t available there.

Also, the lumps of left over araldite 2011 seemed kinda not fully cured even after 36 hours. I mean they’re sort of hard, but not really solid like I’d expect. The data sheet claims it cures at 10 degrees C (colder than the room I used) but rather sneakily, the strengths are not listed for low curing temperatures.

Double also, in the DP8005 datasheet, it mentions that you should discard the first bit out of the nozzle because it won’t be properly mixed. Not only did I not do that either, but it seems like a really good idea for the other auto mixers as well.

Round 2. Fight!

I’m only re-testing the ones which (a) didn’t seem to come out quite as expected (b) won’t cause massive wastage and (c)  don’t have other issues. So, I’m doing Araldite 2011 with a bake at 80 degrees C for an hour (far more than enough) and Gorilla glue with a water spray. I’m also going to be a bit more generous with the Araldite and apply it to both surfaces. I chose 80 based on a kind of gut feeling compromise: the datasheet listed curing times up to 100, but 80 still cured fast and the plastic starts to soften slightly at 85.

For surface prep I’m only doing the IPA clean since it works well, but not the sanding since it’s variable, hard to do and possibly too awkward to be good in production.

The only thing I am varying consistently is whether or not  to wipe off any excess with IPA.


                        Surface prep||     IPA clean     
Glue                                ||    ave     min    
2011, both sides, bake              ||*   20.7   16.8    
2011, both sides, bake              ||    26.9   19.7    
Gorilla Glue, wipe after            ||*   19.9   17.6    
Gorilla Glue                        ||    17.2   15.6    
Scotchweld DP8005                   ||    22.1   20.0    

* = Clean off excess glue with IPA

Well, that’s a bit better.  The Araldite 2011 is the clear winner in strength terms. One measurement actually hit the maximum 32.5Kg the default setup of the press could reach and tore a chunk of the chrome plating off the underlying brass of the snaps . And the Gorilla Glue puts in a very respectable showing. Not quite as strong, but the worst cases are pretty similar. I’m pretty confident now that either of these will do in the final application with the correct surface prep.






Occasional ENOSYS with l2cap connect() on the RPi 3.

I’ve been working a lot with bluetooth and my library has recently been giving me occasional errors. The error is when calling connect() on an l2cap socket (this is the only bluetooth low energy socket and is packet oriented with retries) I occasionally and at random get ENOSYS.

That means “Function not implemented” or “System call not implemented”. Very strange. Even stranger is that it was not on connect(), but on return getsockopt. This is an async call, so connect() returns either EINPROGRESS (if all is well) or another error. Connection errors (i.e ETIMEOUT, EREFUSED, or if you’re unlucky EIO or ENOMEM) are then collected when you come back later and pick them up with getsockopt.

My first thought was my program had bugs (of the form of pointer related memory errors) and I was somehow corrupting my system calls. Strace revealed that actually my system calls were precisely as expected and identical and so it wasn’t (maybe) my fault.

I then came across https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=145144&p=962612, which seemed to imply that this was a problem on the RPi 3, and not entirely unique to me. I was beginning to really suspect that it wasn’t my fault and something else was causing it. I then started (unfairly as it transpires) cursing the kernel devs responsible in my head because that should never happen: either the syscall is implemented or it is not. There’s no dynamic behaviour there.

Anyway it’s clearly a problem with bluetooth, so it was time to break out btmon, a bluez tool which gets copies of all HCI packets, parses them and pretty prints them. And I was getting this:

 HCI Event: LE Meta Event (0x3e) plen 12                                                                                                                                                   [hci0] 461.313621
      LE Read Remote Used Features (0x04)
        Status: Connection Failed to be Established (0x3e)
        Handle: 64
        Features: 0x1f 0x00 0x00 0x00 0x00 0x00 0x00 0x00
          LE Encryption
          Connection Parameter Request Procedure
          Extended Reject Indication
          Slave-initiated Features Exchange
          LE Ping

It had me going a while because you’ll see that the error code (0x3e ) is by irritating coincidence the same number as the code indicating it’s a BLE related event. To cut a rather long and winding story short, I eventually ended up digging into the kernel sources to find where bluetooth errors got translated into system call errors. And I found this:


The rather handily named “bt_to_errno()” function. Now 0x3e was missing from the list. Checking with the bluetooth 4 spec, we eventually find in table 1.1 in  Volume 4, Part D, Section 1.1 the list of error codes. And it corresponds to “Connection Failed to be Established”. There’s no real explanation and this code seems to mean “something went wrong”.

As I mentioned, that was missing from bt_to_errno(), I’m guessing because it’s rare in the wild, and possibly no hardware until recently actually ever returned the code. I’m generally in  favour of the idea of never writing code to handle a condition you’ve never seen, since it’s awfully hard to test.

And flipping to the end you can see that if a code arrives and no code has been written to handle it, the function returns ENOSYS. And you know that’s kind of sensible. The list of errors is not very rich, and there isn’t really anything more suitable.

Of course now we know this happens and seems to correspond to a sporadic error from the hardware, I think the correct choice is to return EAGAIN, which is more or less “try again: it might work, fail again or fail with a new error”. I’ll see if the kernel bluetooth people agree.

Edit: they don’t: EAGAIN and EINPROGRESS are the same error code! Time to figure out a better code.