Server Send Events

Server send events are a way of getting data pushed over an http connection to a browser. The JavaScript interface is very simple. Here’s some useful info:

I thought it would be a neat way of debugging some GIS code in C++: all I had to do was write the data to a socket, and have a page that collected it in JavaScript and plugged it into Google maps. It was that simple, sort of, except that it was actually incredibly awkward getting it up and running since web browsers are both finicky and don’t provide much error information.

All you have to do is open a socket, write the right HTTP headers and send the correct data. I eventually ended up sending it in chunked encoding, which means each message is essentially preceded by a length so the browser knows how much data to accept and put together into a message.  The alternative is to use Content-Length and have fixed length messages (like the w3schools example), but I couldn’t manage to get my browser (Firefox) to accept more than one message before closing the connection due to an error. No idea why, but the chunked encoding is much more flexible anyway.

Probably the biggest hurdle is that my HTLM page was just a file, but the server send events were from a (local) server. Either way that meant it was cross domain and so Firefox would block the request because of CORS. Turns out the fix was a simple headre tweak but that had me going for a while!

Anyway, here’s the code:

#include <iostream>
#include <cstring>
#include <cerrno>
#include <vector>
#include <iomanip>


#include <unistd.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netinet/ip.h>


using namespace std;

class VeryBasicServer
{
	private:
		int sockfd=-1; 
		int connection_fd{-1};
	
	public:

		VeryBasicServer()
		{
			//Incantation to create a socket 
			//Address famile: AF_INET (that is internet, aka IPv4)
			//Type is a reliable 2 way stream, aka TCP
			//Note this just creates a socket handle, it doesn't do anything yet.
			//The 0 is the socket type which is pretty redundant for SOCK_STREAM (only one option)
			sockfd = socket(AF_INET, SOCK_STREAM, 0);
			if (sockfd < 0)
				throw runtime_error("Error opening socket:"s + strerror(errno));

			
			//Set options at the socket level (a socket has a stack of levels, and 
			//all of them have options). This one allows reuse of the address, so that
			//if the program crashes, we don't have to wait for the OS to reclaim it, before
			//we can use this socket again. Useful for debugging!
			int true_ = 1; 
			setsockopt(sockfd,SOL_SOCKET,SO_REUSEADDR,&true_,sizeof(int));
			
			
			//Binding a socket is what sets the port number for a particular
			//socket.
			sockaddr_in serv_addr = {};
			serv_addr.sin_family = AF_INET;          //internet address family
			serv_addr.sin_addr.s_addr = INADDR_ANY;  //allow connections from any address
			serv_addr.sin_port = htons(6502);        //Still fighting the 80s CPU wars. 6502 > 8080
			if (bind(sockfd, (struct sockaddr *) &serv_addr, sizeof(serv_addr)) < 0) 
				throw runtime_error("Error binding socket:"s + strerror(errno));
			
			//This is necessary for stream sockets (e.g. TCP). When a client_ connects
			//a new socket will be created just for that connection on a high port.
			listen(sockfd,5);
		}
		

		void read_ignore_and_dump_incoming_data()
		{
			vector<char> buf(65536, 0);
			int n = read(connection_fd,buf.data(),buf.size());
			if (n < 0) 
				throw runtime_error("Error reading from socket:"s + strerror(errno));
			cout << "Message reads: ";
			cout.write(buf.data(), n);
		}

		void accept()
		{
			//Listen blocks until a cnnection is made, then hands over the newly created
			//socket for the connection
			connection_fd = ::accept(sockfd, nullptr, nullptr);

			if (connection_fd < 0) 
				throw runtime_error("Error accepting:"s + strerror(errno));
			
			//We can actually ignore the HTTP header just to get up
			//and running. For ServerSendEvents, there's only one accepted
			//MIME type, and uncompressed is always a valid compression option
			//even if the client_ doesn't request it.
			read_ignore_and_dump_incoming_data();


			//Construct a valid working header.
			//
			//Two important things to note. One is the access control. Since
			//this server isn't serving anything except SSEs, the web page
			//which is using them ust come from elsewhere. Unless we allow 
			//connections from places other than the originating server, then
			//the web browser will block the attempt for security.
			//
			//The other point is the chunked encodeing. The browser connecting
			//has to know when we've finished sending an event in the stream
			//of data. Chunked encoding allows us to send data blocks along with a 
			//length so the server knows when a block is finished. The other option
			//is to have a fixed Content-Length instead. I never got it working,
			//but it's much less flexible so I didn't try hard.
			//
			//Note also the \r\n line terminations, not just \n. Part of the HTTP spec.
			write("HTTP/1.1 200 OK\r\n"                          //Standard HTTP header line
			      "Content-Type: text/event-stream\r\n"          //This is the only allowed MIME type for SSE
                  "Transfer-Encoding: chunked\r\n"               //Chunked encoding lets it know when an event is done without knowing packet boundaries.
				  "Access-Control-Allow-Origin: *\r\n"           //Because the HTML comes from a file, not this server, we have to allow access
			      "\r\n");                                       //End of header indicator

		}
		
		//Write data with less than the minimal amount of error checking.
		void write(const string& str)
		{
			int n = ::write(connection_fd, str.data(), str.size());

			if(n < 0)
				throw runtime_error("Error writing:"s + strerror(errno));
		}

		void write_chunk(const string& str)
		{
			//Chunked encoding is the size in hex followed by the data. Note that both
			//the size and data fields are terminated by HTTP line terminations (\r\n)
			ostringstream o;
			o << hex << str.size() << "\r\n" << str << "\r\n";
			write(o.str());
		}

		~VeryBasicServer()
		{
			cerr << "closing\n";
			close(connection_fd);
			close(sockfd);
		}
};


int main()
{
	VeryBasicServer hax;

	hax.accept();
	
	cout << "Press enter to send an event" << endl;
	for(int i=1;;i++)
	{
		if(cin.get() == EOF)
			break;
		
		//Build up an event in server-send-event format. The message consists of 
		//one or more fields of the form:
		//field: value\n
		//Followed by an empty line.
		//
		//The main tag is "data:" which carries the data payload.
		//See https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events
		//for more info (e.g. different message types and dispatch)
		string sse;
		sse += "data: " + to_string(i) + "\n";  //Send the current number as the data value
		sse += "\n";                            //Empty field ends the message.

		//Send the message data using chunked encoding
		hax.write_chunk(sse);

	}

}

And the corresponding HTML page is:

<!DOCTYPE html>
<html>
<head>  <meta charset="UTF-8"> </head>
<body>

<h1>Getting server updates</h1>
<div id="result"></div>

<script>
if(typeof(EventSource) !== "undefined") {
    var source = new EventSource("http://127.0.0.1:6502");
    source.onmessage = function(event) {
        document.getElementById("result").innerHTML += "Data: " + event.data + "<br>";
    };
    source.onerror = function(event) {
        document.getElementById("result").innerHTML += "Connection failed<br>";
    };
} else {
    document.getElementById("result").innerHTML = "Sorry, your browser does not support server-sent events...";
}
</script>

</body>
</html>

And that’s basically it. It’s a handy and rather silly chunk you can put in a C++ file. It’s a lot of very poor practices and does all the wrong things for scalability, security, generality and good sense, but it’s handy for debugging hacks.

Code is on github.

Advertisements

Overwrite a file but only if it exists (in BASH)

Imagine you have a line in a script:

cat /some/image.img > /dev/mmcblk0

which dumps a disk image on to an SD card. I have such a line as part of the setup. Actually, a more realistic one is:

pv /some/image.img | sudo 'bash -c cat >/dev/mmcblk0'

which displays a progress bar and doesn’t require having the whole script being run as root. Either way, there’s a problem: if the SD card doesn’t happen to be in when that line runs, it will create /dev/mmcblk0. Then all subsequent writes will go really fast (at the speed of the main disk), and you will get confused and sad when none of the changes are reflected on the SD card. You might even reboot which will magically fix the problem (/dev gets nuked). That happened to me 😦

The weird, out of place dd tool offers a nice fix:

pv /some/image.img | sudo dd conv=nocreat of=/dev/mmcblk0

You can specify a pseudo-conversion, which tells it to not create the file if it doesn’t already exist. It also serves the second purpose as the “sudo tee” idiom but without dumping everything to stdout.

A simple hack but a useful one. You can do similar things like append, but only if it exists, too. The incantation is:

dd conv=nocreat,notrunc oflag=append of=/file/to/append/to

That is, don’t create the file, don’t truncate it on opening and append. If you allow it to truncate, then it will truncate then append, which is entirely equivalent to overwriting but nonetheless you can still specify append without notrunc.

Warn or exit on failure in a shell script

Make has a handy feature where when a rule fails, it will stop whatever it’s doing. Often though you simply want a linear list of commands to be run in sequence (i.e. a shell script) with the same feature.

You can more or less hack that feature with BASH using the DEBUG trap. The trap executes a hook before every command is run, so you can use it to test the result of the previous command. That of course leaves the last command dangling, so you can put the same hook on the EXIT trap which runs after the last command finishes.

Here’s the snippet and example which warns (rather than exits) on failure:

function test_failure(){
  #Save the exit code, since anything else will trash it.
  v=$?
  if [ $v != 0 ]
  then
    echo -e Line $LINE command "\e[31m$COM\e[0m" failed with code $v
  fi
  #This trap runs before a command, so we use it to
  #test the previous command run. So, save the details for
  #next time.
  COM=${BASH_COMMAND}
  LINE=${BASH_LINENO}
}

#Set up the traps
trap test_failure EXIT
trap test_failure DEBUG

#Some misc stuff to test.
echo hello
sleep 2 ; bash -c 'exit 3'

echo world
false

echo what > /this/is/not/writable

echo the end

Running it produces:

$ bash errors.bash 
hello
Line 21 command bash -c 'exit 3' failed with code 3
world
Line 25 command false failed with code 1
errors.bash: line 27: /this/is/not/writable: No such file or directory
Line 27 command echo what > /this/is/not/writable failed with code 1
the end
$

RPi chroot and cross compiling (Part 1)

RPis are fantastic machines, but compared to using my PC, they’re a fraction of the performance, obviously. So, it’s convenient to be able to develop locally with a much faster compiler with much more RAM, then deploy on the Pi.

The Pi-3 is remarkably fast with it’s 4 processors, but with only 1G of RAM, I’ve had some not huge C++ programs with very extensive debugging enabled require more RAM than the machine has to compile. Trying to compile 4 things at once is a recipe for disaster.

So without further ado…

Actually there’s quite a lot of ado. This isn’t so much a guide as a brain dump of stuff I learned as I went along while skipping out the really tedious dead ends. You might want to skim read and skip around a bit.

Running Raspian in a QEMU emulator.

Couldn’t get it to work, sorry! It’s also the least interesting option.

Running an RPi-like chroot

This uses linux’s binfmt facility, to run dispatch non “native” code to various executors. Cargo cult instructions on Ubuntu are:

sudo apt-get install qemu binfmt-support qemu-user-static

Now mount the disk image. The shell transcript should look like:

$ sudo kpartx -av 2016-05-27-raspbian-jessie-lite.img
add map loop0p1 (252:0): 0 129024 linear /dev/loop0 8192
add map loop0p2 (252:1): 0 2572288 linear /dev/loop0 137216
$ sudo mount /dev/mapper/loop0p2 /mnt/

Note that kpartx reads the partition table and sets up a loopback for each partition. Then you mount the right partition on /mnt, for example to get at it.

Now, you need one more bit of setup to be able to chroot: you have to put the ARM emulator in the chroot so it can be accessed. But first, run:

update-binfmts --display

and examine the output. In there should be:

qemu-arm (enabled):
     package = qemu-user-static
        type = magic
      offset = 0
       magic = \x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00
        mask = \xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff
 interpreter = /usr/bin/qemu-arm-static
    detector =

That pretty much says that when the kernel sees the right header, it should execute a the program using the “/usr/bin/qemu-arm-static” binary. So, set that up:

sudo cp /usr/bin/qemu-arm-static /mnt/usr/bin

If you don’t set it up, you’ll get a “no such file or directory” error when you try to chroot, which is caused by the kernel failing to find qemu-arm-static. Note that they have rather sensibly compiled it as fully static, so you don’t need to bring the entire dynamic runtime into the chroot.

So, now execute (my machine is called sheepfrog by the way) and see the following transcript:

$ sudo chroot /mnt
root@sheepfrog:/# g++ --version
g++ (Raspbian 4.9.2-10) 4.9.2
Copyright (C) 2014 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

root@sheepfrog:/#

like… woah. Now, a benchmark

From bash:


time for ((i=0; i < 10000; i++)); do : ; done

My RPi3 gives 0.4 seconds, my i7 Q820 @ 1.73GHz (yes, it is 2017) running the emulator gives 0.942 seconds, which is a fair bit slower. If you have a half way modern machine then you’ll be neck and neck, probably faster than native. Of course, even my old laptop has much more RAM and vastly faster storage, which makes a huge difference.

Natively my machine takes, er, 2.37 seconds! That’s because I’ve set up bash to change the xterm title, so it’s flogging the x server in that loop. Removing that gives 0.065 seconds. That’s a bit more sensible.

Running the chroot and installing things

Anyway, this isn’t the real reason for doing this. The main point of setting up the chroot is to be able to chroot into is and run apt-get to install rpi packages. Then eventually one can install packages etc and use them with a native x86 cross compiler. So first, let’s get going!

~ #chroot /mnt
root@sheepfrog:/# apt-get update
qemu: uncaught target signal 4 (Illegal instruction) - core dumped
Illegal instruction (core dumped)
root@sheepfrog:/#

vader

Turns out the solution isn’t so bad. The cargo-cult hacking is to comment out the only line in /mnt/etc/ld.so.preload so the file looks like this:

#/usr/lib/arm-linux-gnueabihf/libarmmem.so

Now, apt-get update works fine and you can install the all the -dev packages IN THE WORLD!

The eagle-eyed might wonder HOW apt-get works, since I never copied over the resolv.conf file, so how on earth can it fetch anything? The answer is that Rapspian provides a default resolv.conf that points at google’s handy 8.8.8.8 nameservers!

 Wait that all sounds unnecessary. It’s unnecessary, right?

Yes, that is all indeed unnecessary. apt and dpkg is flexible enough that it can run as if it were in a chroot without actually being in a chroot. That means you can use the native apt-get to manipulate the contents of /mnt. If you’re not running a Debian derived distro, then you’ll have to somehow install apt-get. Probably the easiest way is to debootstrap a native installation and chroot into it.

Anyway apparently the magic option is -o Dir=something, so:


~ #sudo apt-get -o Dir=/mnt/   update
Hit http://archive.raspberrypi.org jessie InRelease
Hit http://mirrordirector.raspbian.org jessie InRelease
W: Failed to fetch http://mirrordirector.raspbian.org/raspbian/dists/jessie/InRelease  Unable to find expected entry 'main/binary-amd64/Packages' in Release file (Wrong sources.list entry or malformed file)

W: Failed to fetch http://archive.raspberrypi.org/debian/dists/jessie/InRelease  Unable to find expected entry 'main/binary-amd64/Packages' in Release file (Wrong sources.list entry or malformed file)

E: Some index files failed to download. They have been ignored, or old ones used instead.

Can you hear the “bwaaa bwaaaaa” trombone noise? I can hear the trombone noise.

Apparently despite being in a subtree set up as ARM, it’s looking for packages based on the host architecture, not some config file somewhere. The key is obviously going to tell it that we really so want ARM.

I’ve not managed to figure it out yet. Adding:

 -o apt::architecture=armhf -o DPkg::Options::='--admindir=/mnt/var/lib/dpkg'

kind of helps. This is supposed to tell dpkg to look in the right place too., apt is a manager which wraps dpkg, so you have to configure both of them. But then I get huge numbers of dependency errors when I try to install anything. I suspect it’s still looking in the wrong places. And sure enough, strace verifies that it’s still reading files from /etc, but not anything terribly important from /var. Strange.

Cross compiling.

That comes in part 2!

 

Occasional ENOSYS with l2cap connect() on the RPi 3.

I’ve been working a lot with bluetooth and my library has recently been giving me occasional errors. The error is when calling connect() on an l2cap socket (this is the only bluetooth low energy socket and is packet oriented with retries) I occasionally and at random get ENOSYS.

That means “Function not implemented” or “System call not implemented”. Very strange. Even stranger is that it was not on connect(), but on return getsockopt. This is an async call, so connect() returns either EINPROGRESS (if all is well) or another error. Connection errors (i.e ETIMEOUT, EREFUSED, or if you’re unlucky EIO or ENOMEM) are then collected when you come back later and pick them up with getsockopt.

My first thought was my program had bugs (of the form of pointer related memory errors) and I was somehow corrupting my system calls. Strace revealed that actually my system calls were precisely as expected and identical and so it wasn’t (maybe) my fault.

I then came across https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=145144&p=962612, which seemed to imply that this was a problem on the RPi 3, and not entirely unique to me. I was beginning to really suspect that it wasn’t my fault and something else was causing it. I then started (unfairly as it transpires) cursing the kernel devs responsible in my head because that should never happen: either the syscall is implemented or it is not. There’s no dynamic behaviour there.

Anyway it’s clearly a problem with bluetooth, so it was time to break out btmon, a bluez tool which gets copies of all HCI packets, parses them and pretty prints them. And I was getting this:

 HCI Event: LE Meta Event (0x3e) plen 12                                                                                                                                                   [hci0] 461.313621
      LE Read Remote Used Features (0x04)
        Status: Connection Failed to be Established (0x3e)
        Handle: 64
        Features: 0x1f 0x00 0x00 0x00 0x00 0x00 0x00 0x00
          LE Encryption
          Connection Parameter Request Procedure
          Extended Reject Indication
          Slave-initiated Features Exchange
          LE Ping

It had me going a while because you’ll see that the error code (0x3e ) is by irritating coincidence the same number as the code indicating it’s a BLE related event. To cut a rather long and winding story short, I eventually ended up digging into the kernel sources to find where bluetooth errors got translated into system call errors. And I found this:

http://lxr.free-electrons.com/source/net/bluetooth/lib.c#L45

The rather handily named “bt_to_errno()” function. Now 0x3e was missing from the list. Checking with the bluetooth 4 spec, we eventually find in table 1.1 in  Volume 4, Part D, Section 1.1 the list of error codes. And it corresponds to “Connection Failed to be Established”. There’s no real explanation and this code seems to mean “something went wrong”.

As I mentioned, that was missing from bt_to_errno(), I’m guessing because it’s rare in the wild, and possibly no hardware until recently actually ever returned the code. I’m generally in  favour of the idea of never writing code to handle a condition you’ve never seen, since it’s awfully hard to test.

And flipping to the end you can see that if a code arrives and no code has been written to handle it, the function returns ENOSYS. And you know that’s kind of sensible. The list of errors is not very rich, and there isn’t really anything more suitable.

Of course now we know this happens and seems to correspond to a sporadic error from the hardware, I think the correct choice is to return EAGAIN, which is more or less “try again: it might work, fail again or fail with a new error”. I’ll see if the kernel bluetooth people agree.

Edit: they don’t: EAGAIN and EINPROGRESS are the same error code! Time to figure out a better code.

 

 

 

 

 

The best of both worlds

I currently run Ubuntu LTS (14.04 at the moment) on my work laptop and Arch on my home laptop. I like the stability of Ubuntu, and very much like that I  can run security updates without breaking anything. However, the old versions of programs are annoying in some cases. I love running Arch because I always have the latest versions. This bit me today since the version of gerbv I was running was too old to support the high resolution drill placements in Eagle 7 and so my gerbers looked bad.

Give me Archbuntu!

First, get arch-bootstrap.sh and run it in (say) /other_OS to make /other_OS/ARCH_x64. You now have a bootstrapped arch installation. You can chroot into it and it looks arch-like but won’t work properly.

The main reason is that none of the special files are there. Essentially you want everything to mirror your main OS, except for the the OS itself. This can be done with bind mounts. Add the following to /etc/fstab and run mount -a

/proc          /other_OS/ARCH_x64/proc          none defaults,bind 0 0
/sys           /other_OS/ARCH_x64/sys           none defaults,bind 0 0
/dev           /other_OS/ARCH_x64/dev           none defaults,bind 0 0
/tmp           /other_OS/ARCH_x64/tmp           none defaults,bind 0 0
/home          /other_OS/ARCH_x64/home          none defaults,bind 0 0
/etc/passwd    /other_OS/ARCH_x64/etc/passwd    none defaults,bind 0 0
/etc/shadow    /other_OS/ARCH_x64/etc/shadow    none defaults,bind 0 0
/etc/group     /other_OS/ARCH_x64/etc/group     none defaults,bind 0 0
/etc/sudoers   /other_OS/ARCH_x64/etc/sudoers   none defaults,bind 0 0
/etc/sudoers.d /other_OS/ARCH_x64/etc/sudoers.d none defaults,bind 0 0

Now you can chroot into /other_OS/ARCH_x64 and run pacman. Note that /tmp is shared partly to avoid duplication, but also because that’s where the X11 socket it. Without it, you’ll need to use xhost to allow chroot’d programs to connect. And the password related files are duplicated to so sudo still works.

While you can sudo, chroot, then su – to yourself then run programs, that’s a little inconvenient. chroot is root only for security reasons, but since the chroot image is legitimate, we can chroot into that safely as anyone. The following code essentially chroots into /other_OS/ARCH_x64, drops root and then executes its arguments.

#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>

int main(int argc, char** argv)
{
	int r = chroot("/other_OS/ARCH_x64/");

	if(argc == 1)
		return 0;
	
	if(r == -1)
	{
		fprintf(stderr, "Error chrooting: %s\n", strerror(errno));
		return 1;
	}
	
	r = setuid(getuid());

	if(r == -1)
	{
		fprintf(stderr, "Error dropping root: %s\n", strerror(errno));
		return 1;
	}

	r = setgid(getgid());

	if(r == -1)
	{
		fprintf(stderr, "Error dropping root: %s\n", strerror(errno));
		return 1;
	}

	execvp(argv[1], argv+1);

	fprintf(stderr, "Error: %s\n", strerror(errno));
}

Compiling this as ae (think arch exec) and putting it in the path allows you to do “ae gerbv” instead of “gerbv” to get the Arch version, for example. Note that unlike the commandline version, the chroot function doesn’t alter the CWD, so it all works just fine.

Now I have the best of both worlds!