What’s all this new-fangled DBus rubbish?

The 1990s called…

DBus has been one of the larger changes that swept through Linux in the last 20 years or so. I mostly (with the exception of a small amount of whinging) ignored it. Usually I’ve encountered it when it hasn’t worked or has got in the way with something or other. Naturally when things worked, I was generally unaware of it.

It’s basically:

  • Async RPC system with call, response, exception and asynchronus push.
  • Authentication (useful for talking to daemons running as root)
  • Introspection (the set of methods, arguments etc are exposed via DBus)
  • Central point for services to register objects and methods: you open a dbus connection as opposed to a random socket

Various tools exist to send and receive messages, e.g. python libraries, shell commands via ‘dbus-send’ and so on. I’ve nothing against RPC in general, but it replaces essentially shell scripts triggered by the kernel etc with programs exchanging messages. You can do more with the latter (though often that isn’t necessary), but it replaces easily discoverable scripts with reading documentation, something the programming community is not well known for and as such it has a rather higher barrier to entry.

It has also has been used to replace simply libraries with relatively complex IPC requiring complex setup. But I won’t blame a table saw for injuries it causes through misuse. Anyway this post is a nearly linear stream of me learning DBus with the mistakes and confusion removed.

The tools

I will be relying on the following:

  • python dbus module import dbus
  • The basic dbus commandline program: dbus-send
  • QT’s qdbus since it provides in many cases a nicer interface and introspection
  • Gnome’s gdbus for more verbose introspection

The basics

I’m interested in manipulating the system, so I will be working with the system bus. There’s usually a system bus (for talking to OS related daemons) and a session bus (for all the programs in a logged in session). You can make more if you like but no one does.

In order to make an RPC call, you need:

  1. A program to talk to, e.g. NetworkManager (this is called the bus name) and usually has a name like org.freedesktop.NetworkManager
  2. An object in the program. Each program exports a tree of objects, rooted at / and separated with forward slashes. Something like: /org/freedesktop/NetworkManager. Freedesktop likes redundant naming because it’s redundant and repeats things redundantly. They could equally well have exported /.
  3. The interface and method name. Just like any OO system each object can present zero or more interfaces with methods.

We can examine this. First, qdbus will show the names present on the system bus. For example, on my system (I don’t know what the numbers prefixed with colons are) I have:

qdbus --system  | grep -v :
 org.freedesktop.systemd1
 org.freedesktop.resolve1
 fi.epitest.hostap.WPASupplicant
 fi.w1.wpa_supplicant1
 org.freedesktop.NetworkManager
 org.gnome.DisplayManager
 org.freedesktop.ColorManager
 org.freedesktop.Avahi
 org.bluez
 org.freedesktop.UPower
 org.freedesktop.Accounts
 org.freedesktop.login1
 org.freedesktop.RealtimeKit1
 org.freedesktop.UDisks2
 org.freedesktop.ModemManager1
 org.freedesktop.bolt
 org.freedesktop.PackageKit
 org.freedesktop.PolicyKit1
org.freedesktop.DBus

There’s a few one might recognise, Udisks for hotplug disks, NetworkManager for wifi control etc, ModemManager1 because this machine actually has a real actual physical 56k modem, and a few other well established ones like systemd. You can go further and query what’s on the bus. For example, I can query systemd using gdbus, which as you may recall is the verbose, detailed one. I’m going to query the root object to see what’s there:

$gdbus introspect  --system --dest org.freedesktop.systemd1   --object-path / 
node / {
  interface org.freedesktop.DBus.Peer {
    methods:
      Ping();
      GetMachineId(out s machine_uuid);
    signals:
    properties:
  };
  interface org.freedesktop.DBus.Introspectable {
    methods:
      Introspect(out s data);
    signals:
    properties:
  };
  interface org.freedesktop.DBus.Properties {
    methods:
      Get(in  s interface,
          in  s property,
          out v value);
      GetAll(in  s interface,
             out a{sv} properties);
      Set(in  s interface,
          in  s property,
          in  v value);
    signals:
      PropertiesChanged(s interface,
                        a{sv} changed_properties,
                        as invalidated_properties);
    properties:
  };
  node org {
  };
};

If you’ve ever read IDLs of any sort this will look vaguely familiar. Systemd is exporting an object which presents three interfaces:

  • org.freedesktop.DBus.Peer
  • org.freedesktop.DBus.Introspectable
  • org.freedesktop.DBus.Properties

and a subobject, org. Since gdbus doesn’t recurse by default, that’s all that’s displayed. The first of those objects has two methods neither of which have any arguments, which makes them easy to call. So I shall ping dbus:

$dbus-send --print-reply --system --dest=org.freedesktop.systemd1 / org.freedesktop.DBus.Peer.GetMachineId
method return time=1632074720.055870 sender=:1.0 -> destination=:1.3006 serial=54374 reply_serial=2
   string "73f867af39124a3583c288e620019332"

and it responds. See how the bus name (dest), path (/) and interface.object (org.freedesktop.DBus.Peer.GerMachineId) are used. I can examine another common one. when given a bus, but no object, qdbus will recurse showing the object tree:

$qdbus --literal --system org.freedesktop.NetworkManager 
/
/org
/org/freedesktop
/org/freedesktop/NetworkManager
/org/freedesktop/NetworkManager/DnsManager
/org/freedesktop/NetworkManager/DHCP4Config
/org/freedesktop/NetworkManager/DHCP4Config/70
/org/freedesktop/NetworkManager/ActiveConnection
/org/freedesktop/NetworkManager/ActiveConnection/2
/org/freedesktop/NetworkManager/ActiveConnection/72
/org/freedesktop/NetworkManager/AccessPoint
/org/freedesktop/NetworkManager/AccessPoint/2686
/org/freedesktop/NetworkManager/Devices
/org/freedesktop/NetworkManager/Devices/3
/org/freedesktop/NetworkManager/Devices/2
/org/freedesktop/NetworkManager/Devices/1
/org/freedesktop/NetworkManager/Devices/25
/org/freedesktop/NetworkManager/Devices/4
/org/freedesktop/NetworkManager/AgentManager
/org/freedesktop/NetworkManager/Settings
/org/freedesktop/NetworkManager/Settings/8
/org/freedesktop/NetworkManager/Settings/7
/org/freedesktop/NetworkManager/Settings/6
/org/freedesktop/NetworkManager/Settings/5
/org/freedesktop/NetworkManager/Settings/4
/org/freedesktop/NetworkManager/Settings/3
/org/freedesktop/NetworkManager/Settings/2
/org/freedesktop/NetworkManager/Settings/1
/org/freedesktop/NetworkManager/Settings/24
/org/freedesktop/NetworkManager/Settings/9
/org/freedesktop/NetworkManager/IP6Config
/org/freedesktop/NetworkManager/IP6Config/3
/org/freedesktop/NetworkManager/IP6Config/204
/org/freedesktop/NetworkManager/IP6Config/203
/org/freedesktop/NetworkManager/IP6Config/6
/org/freedesktop/NetworkManager/IP4Config
/org/freedesktop/NetworkManager/IP4Config/3
/org/freedesktop/NetworkManager/IP4Config/204
/org/freedesktop/NetworkManager/IP4Config/203
/org/freedesktop/NetworkManager/IP4Config/6

For reference, gdbus (non recursive; recursive is too verbose for this blog post) gives:

$gdbus introspect  --system --dest org.freedesktop.NetworkManager   --object-path /
node / {
  node org {
  };
};

The root node isn’t very interesting, it just has the child org and nothing else. qdbus will also give a more compact method view, for example, I can query one of the devices:

$qdbus --literal --system org.freedesktop.NetworkManager /org/freedesktop/NetworkManager/Devices/3 
method QDBusVariant org.freedesktop.DBus.Properties.Get(QString interface_name, QString property_name)
method QVariantMap org.freedesktop.DBus.Properties.GetAll(QString interface_name)
signal void org.freedesktop.DBus.Properties.PropertiesChanged(QString interface_name, QVariantMap changed_properties, QStringList invalidated_properties)
method void org.freedesktop.DBus.Properties.Set(QString interface_name, QString property_name, QDBusVariant value)
method QString org.freedesktop.DBus.Introspectable.Introspect()
method QString org.freedesktop.DBus.Peer.GetMachineId()
method void org.freedesktop.DBus.Peer.Ping()
property read QDBusObjectPath org.freedesktop.NetworkManager.Device.ActiveConnection
property readwrite bool org.freedesktop.NetworkManager.Device.Autoconnect
property read QList<QDBusObjectPath> org.freedesktop.NetworkManager.Device.AvailableConnections
property read uint org.freedesktop.NetworkManager.Device.Capabilities
property read uint org.freedesktop.NetworkManager.Device.DeviceType
property read QDBusObjectPath org.freedesktop.NetworkManager.Device.Dhcp4Config
property read QDBusObjectPath org.freedesktop.NetworkManager.Device.Dhcp6Config
property read QString org.freedesktop.NetworkManager.Device.Driver
property read QString org.freedesktop.NetworkManager.Device.DriverVersion
property read bool org.freedesktop.NetworkManager.Device.FirmwareMissing
property read QString org.freedesktop.NetworkManager.Device.FirmwareVersion
property read QString org.freedesktop.NetworkManager.Device.Interface
property read uint org.freedesktop.NetworkManager.Device.Ip4Address
property read QDBusObjectPath org.freedesktop.NetworkManager.Device.Ip4Config
property read QDBusObjectPath org.freedesktop.NetworkManager.Device.Ip6Config
property read QString org.freedesktop.NetworkManager.Device.IpInterface
property read QDBusRawType::aa{sv} org.freedesktop.NetworkManager.Device.LldpNeighbors
property readwrite bool org.freedesktop.NetworkManager.Device.Managed
property read uint org.freedesktop.NetworkManager.Device.Metered
property read uint org.freedesktop.NetworkManager.Device.Mtu
property read bool org.freedesktop.NetworkManager.Device.NmPluginMissing
property read QString org.freedesktop.NetworkManager.Device.PhysicalPortId
property read bool org.freedesktop.NetworkManager.Device.Real
property read uint org.freedesktop.NetworkManager.Device.State
property read QDBusRawType::(uu) org.freedesktop.NetworkManager.Device.StateReason
property read QString org.freedesktop.NetworkManager.Device.Udi
method void org.freedesktop.NetworkManager.Device.Delete()
method void org.freedesktop.NetworkManager.Device.Disconnect()
method QDBusRawType::a{sa{sv}} org.freedesktop.NetworkManager.Device.GetAppliedConnection(uint flags, qulonglong& version_id)
method void org.freedesktop.NetworkManager.Device.Reapply(QDBusRawType::a{sa{sv}} connection, qulonglong version_id, uint flags)
signal void org.freedesktop.NetworkManager.Device.StateChanged(uint new_state, uint old_state, uint reason)
property read QList<QDBusObjectPath> org.freedesktop.NetworkManager.Device.Wireless.AccessPoints
property read QDBusObjectPath org.freedesktop.NetworkManager.Device.Wireless.ActiveAccessPoint
property read uint org.freedesktop.NetworkManager.Device.Wireless.Bitrate
property read QString org.freedesktop.NetworkManager.Device.Wireless.HwAddress
property read uint org.freedesktop.NetworkManager.Device.Wireless.Mode
property read QString org.freedesktop.NetworkManager.Device.Wireless.PermHwAddress
property read uint org.freedesktop.NetworkManager.Device.Wireless.WirelessCapabilities
signal void org.freedesktop.NetworkManager.Device.Wireless.AccessPointAdded(QDBusObjectPath access_point)
signal void org.freedesktop.NetworkManager.Device.Wireless.AccessPointRemoved(QDBusObjectPath access_point)
method QList<QDBusObjectPath> org.freedesktop.NetworkManager.Device.Wireless.GetAccessPoints()
method QList<QDBusObjectPath> org.freedesktop.NetworkManager.Device.Wireless.GetAllAccessPoints()
signal void org.freedesktop.NetworkManager.Device.Wireless.PropertiesChanged(QVariantMap properties)
method void org.freedesktop.NetworkManager.Device.Wireless.RequestScan(QVariantMap options)
property readwrite uint org.freedesktop.NetworkManager.Device.Statistics.RefreshRateMs
property read qulonglong org.freedesktop.NetworkManager.Device.Statistics.RxBytes
property read qulonglong org.freedesktop.NetworkManager.Device.Statistics.TxBytes
signal void org.freedesktop.NetworkManager.Device.Statistics.PropertiesChanged(QVariantMap properties)

Yikes! There’s a lot there. Actually gdbus has the nice thing where it also queries properties for you. I recommend trying that. Anyway if you carefully read through, you can see the GetAccessPoints method. If I call it, I get a list of accesspoints:

$dbus-send --print-reply --system --dest=org.freedesktop.NetworkManager /org/freedesktop/NetworkManager/Devices/3 org.freedesktop.NetworkManager.Device.Wireless.GetAccessPoints
method return time=1632075788.446527 sender=:1.12 -> destination=:1.3041 serial=431030 reply_serial=2
   array [
      object path "/org/freedesktop/NetworkManager/AccessPoint/2686"
      object path "/org/freedesktop/NetworkManager/AccessPoint/2699"
      object path "/org/freedesktop/NetworkManager/AccessPoint/2700"
      object path "/org/freedesktop/NetworkManager/AccessPoint/2701"
      object path "/org/freedesktop/NetworkManager/AccessPoint/2702"
      object path "/org/freedesktop/NetworkManager/AccessPoint/2703"
      object path "/org/freedesktop/NetworkManager/AccessPoint/2704"
      object path "/org/freedesktop/NetworkManager/AccessPoint/2706"
      object path "/org/freedesktop/NetworkManager/AccessPoint/2708"
      object path "/org/freedesktop/NetworkManager/AccessPoint/2710"
      object path "/org/freedesktop/NetworkManager/AccessPoint/2711"
      object path "/org/freedesktop/NetworkManager/AccessPoint/2712"
      object path "/org/freedesktop/NetworkManager/AccessPoint/2713"
      object path "/org/freedesktop/NetworkManager/AccessPoint/2714"
      object path "/org/freedesktop/NetworkManager/AccessPoint/2715"
      object path "/org/freedesktop/NetworkManager/AccessPoint/2716"
      object path "/org/freedesktop/NetworkManager/AccessPoint/2717"
   ]

This is not very shell friendly, but I can persist. For example, if I examine one of the APs, I get:

$gdbus introspect  --system --dest org.freedesktop.NetworkManager   --object-path /org/freedesktop/NetworkManager/AccessPoint/2714
node /org/freedesktop/NetworkManager/AccessPoint/2714 {
  interface org.freedesktop.DBus.Properties {
    methods:
      Get(in  s interface_name,
          in  s property_name,
          out v value);
      GetAll(in  s interface_name,
             out a{sv} properties);
      Set(in  s interface_name,
          in  s property_name,
          in  v value);
    signals:
      PropertiesChanged(s interface_name,
                        a{sv} changed_properties,
                        as invalidated_properties);
    properties:
  };
  interface org.freedesktop.DBus.Introspectable {
    methods:
      Introspect(out s xml_data);
    signals:
    properties:
  };
  interface org.freedesktop.DBus.Peer {
    methods:
      Ping();
      GetMachineId(out s machine_uuid);
    signals:
    properties:
  };
  interface org.freedesktop.NetworkManager.AccessPoint {
    methods:
    signals:
      PropertiesChanged(a{sv} properties);
    properties:
      readonly u Flags = 0;
      readonly u WpaFlags = 0;
      readonly u RsnFlags = 0;
      readonly ay Ssid = [0x48, 0x50, 0x2d, 0x50, 0x72, 0x69, 0x6e, 0x74, 0x2d, 0x33, 0x31, 0x2d, 0x4f, 0x66, 0x66, 0x69, 0x63, 0x65, 0x6a, 0x65, 0x74, 0x20, 0x36, 0x36, 0x30, 0x30];
      readonly u Frequency = 2412;
      readonly s HwAddress = 'xx:xx:xx:xx:xx:xx';
      readonly u Mode = 2;
      readonly u MaxBitrate = 54000;
      readonly y Strength = 0x31;
      readonly i LastSeen = 2602413;
  };
};

That looks interesting. The ssid, which is a byte array is a property. The way to read properties is with the property get method which is present, so you have to call that. Note it takes two arguments, both strings, one which is the interface name, the other being the method, so you have to specify those. And querying AP number 2714 gives:

 $dbus-send --print-reply --system --dest=org.freedesktop.NetworkManager /org/freedesktop/NetworkManager/AccessPoint/2714 org.freedesktop.DBus.Properties.Get string:org.freedesktop.NetworkManager.AccessPoint string:Ssid
method return time=1632076552.174223 sender=:1.12 -> destination=:1.3082 serial=431816 reply_serial=2
   variant       array of bytes "HP-Print-31-Officejet 6600"

Oh look, one of my neighbours has one of the worse printers ever made. I had one of those printers.

Python provides a reasonable library for doing such things. Here’s some code which iterates over all available network interfaces and prints the SSID of whichever access points it finds:

import dbus
bus = dbus.SystemBus()

obj = bus.get_object('org.freedesktop.NetworkManager', '/org/freedesktop/NetworkManager')
network_manager = dbus.Interface(obj, 'org.freedesktop.NetworkManager')

#Iterate over all devices
for device in network_manager.GetDevices():

    #Get the wireless interface for each device
    obj = bus.get_object('org.freedesktop.NetworkManager', device)
    wlan =  dbus.Interface(obj, 'org.freedesktop.NetworkManager.Device.Wireless')
    
    #Note we don't get an error until we attempt to use the interface
    #I suspect there is a better way
    try:
        for ap_path in  wlan.GetAccessPoints():
            # Read the SSID property
            obj =  bus.get_object('org.freedesktop.NetworkManager', ap_path)
            ap_props =   dbus.Interface(obj, 'org.freedesktop.DBus.Properties')
            ssid = ap_props.Get('org.freedesktop.NetworkManager.AccessPoint', 'Ssid')
            print(''.join([str(v) for v in ssid]))

    except dbus.exceptions.DBusException as e:
        pass

So that’s it for the basics. There’s a whole introspection API as well which I believe returns the structure as XML.

But to call methods, introspection isn’t needed.

Writing a service

Writing a service is pretty easy in Python. Here’s an example where dbus doesn’t steal the entire main loop. Note that this runs in the sesson bus, not the system one, because of security.

from gi.repository import GLib
import time

class Service(dbus.service.Object):
   def __init__(self):
      #Register dbus with GLib's main loop
      dbus.mainloop.glib.DBusGMainLoop(set_as_default=True)
      
      #Bus name 
      bus_name = dbus.service.BusName("com.hello.helloworld", dbus.SessionBus())

      #Object to export
      dbus.service.Object.__init__(self, bus_name, "/")

      self._count=0

   #Register two methods to the one interface
   @dbus.service.method("com.hello.helloworld.Message", in_signature='', out_signature='s')
   def get_message(self):
      self._count+=1
      return "Hello, world " + str(self._count)
    
   @dbus.service.method("com.hello.helloworld.Message", in_signature='i')
   def set_counter(self, i):
      self._count = i
    
   #A way of polling the main loop so that GLib doesn't 
   #steal the program's main loop
   def poll(self):
      loop = GLib.MainLoop()
      def quit():
          loop.quit()
      GLib.idle_add(quit)
      loop.run()

#Instantiate the service
service = Service()

#Poll it
while True:
    service.poll()
    time.sleep(.1)

And it works:

~ $qdbus --session com.hello.helloworld / get_message
Hello, world 1
~ $qdbus --session com.hello.helloworld / get_message
Hello, world 2
~ $qdbus --session com.hello.helloworld / get_message
Hello, world 3
~ $qdbus --session com.hello.helloworld / get_message
Hello, world 4
~ $qdbus --session com.hello.helloworld / get_message
Hello, world 5
~ $qdbus --session com.hello.helloworld / set_counter 0

~ $qdbus --session com.hello.helloworld / get_message
Hello, world 1
~ $qdbus --session com.hello.helloworld / get_message
Hello, world 2

qdbus is often a lot less verbose to use!

Writing a system service

If you try to run with the system bus, it will fail, even if you run as root. This is because of the security policy in place. The policies allow non-root daemons to run as system services, and no one special cased it to make root ones always allowed, which is fine.

The policies are in /etc/dbus-1/system.d/ and they’re sort of understandable, but only sort of. It is documented: essentially it’s default deny with allow/deny rules applied top to bottom based on a matching scheme. In order to run the service as root, the following file works:

<!DOCTYPE busconfig PUBLIC
 "-//freedesktop//DTD D-BUS Bus Configuration 1.0//EN"
 "http://www.freedesktop.org/standards/dbus/1.0/busconfig.dtd">
<busconfig>

  <!-- Only root can own the service -->
  <policy user="root">
    <allow own="com.hello.helloworld"/>
  </policy>

  <policy context="default">
    <allow send_destination="com.hello.helloworld"/>
  </policy>
</busconfig>

essentially this allows only root to own the service, but allows anyone (the default context) to send messages. One the file is created, you then need to reload DBus which can be done with SIGHUP or with systemctl reload debus on systemd based systems.

Starting a system service with systemd

I could do it manually, but perhaps I should bend to the winds of change.

This is simple, for this rather basic service. First, add #!/usr/bin/env python3 to the first line of the script and make it executable. Then copy it to /opt/helloservice/service.py. Then add a very simple unit file to /etc/systemd/system/hello.service:

[Unit]
Description=Hello world service

[Service]
ExecStart=/opt/helloservice/service.py

[Install]
WantedBy=multi-user.target

Is it me or is it weird that the old Windows 3 style .INI files have become a perverse sort of standard?

The future..

Anyway, now a simple:

sudo systemctl daemon-reload
sudo systemctl start hello

starts the service. And to enable it on boot:

sudo systemctl enable hello

which apparently symlinks it in a directory.

And that’s it

A new system service, running as root controllable by a user. The purpose is to have some NeoPixels on an Raspberry Pi controlled as a user (the pixels can only run as root due to the need to access low level hardware).

Make hacks: embedded version info, detecting non timestamp change and automatically generated dependencies

For the purposes of traceability you may wish to embed the version of a program into it. If you’re using full on CI, you very much shouldn’t need this. If your CI system doesn’t record such things, then you need to fix it now.

But if you’re hacking around locally, especially for research it can be really useful to know where that executable came from or more specifically where some results came from, because it’s easy to lose track of that. An easy way to do this is to embed the git hash of the repository into the executable so it can be written alongside the data.

Essentially if git status --porcelain prints nothing then the repository is entirely clean. With that in mind, here’s a short script which generates a C++ source file with the git hash if the repository is clean, and prints a loud, red warning if it is not clean. Here is get_version.sh:

git_hash=`git rev-parse HEAD`

if [[ "$(git status --porcelain)" != "" ]]
then
	echo -e "\033[31mThere are uncommitted changes. This means that the build" 1>&2
	echo -e "Will not represent a traceable version.\033[0m" 1>&2
	time=`date +%s`
	version="${git_hash}-${time}"
else
	version="${git_hash}"
fi

cat <<FOO
namespace version{
	const char* version_string = "$version";
}
FOO

It’s easy enough to use from make:

.PHONY: FORCE
versioninfo.cc: FORCE
	bash get_version.sh > versioninfo.cc

and of course make your program depend on versioninfo.o. But it’s not very clean; this will rebuild and then re-link every single time. The key is to make the FORCE dependency depend on whether anything has changed.

This script (get_version_target.sh), reruns get_version.sh and compares it to the existing versioninfo.cc. If there’s a change, it prints the target (FORCE), otherwise it prints nothing.

if ! diff -q versioninfo.cc <( bash get_version.sh 2> /dev/null ) > /dev/null 2>&1
then
        echo FORCE
fi

You then need to plumb this into make using the shell command:

version_target=$(shell bash get_version_target.sh)
.PHONY: FORCE 
versioninfo.cc: $(version_target)
	bash get_version.sh > versioninfo.cc

This will now only re-generate versioninfo.cc (and hence the .o and the final executables) if the git hash changes.

With the basics in place, you can make the version info as detailed as you like, for example you could record any tags and branch names, so you could record those and if it’s an actual point release etc. The downside of the shell commands is that they run every time make is called so you will want to make them fast otherwise incremental rebuilds will become annoyingly slow.

Using this mechanism, make can be much more malleable than expected. This is an immensely powerful feature. But remember:

Today’s hack

Let’s say you’re automating a git workflow for a variety of good and bad reasons. Commits are fine, you can just do a:

commit -am 'a message'

and it goes through non interactively. Now let’s say you have a merge and you try:

git merge

It pop open an editor window to let you type a message. There’s no merge -m option to let you provide a message. If you look closely, you’ll see that the editor window already has a message filled in saying it’s a merge. This means if you save without exiting, git sees a message and decides all is OK and it can proceed. So all you have to do is provide an “editor” which quits without saving every time. The program which does nothing except exit successfully is /true so you can simply do:

EDITOR=true git merge

for a non interactive merge.

I don’t know whether to be proud or ashamed, but it works.

 

Small hack of the day

Two things:

  1. syntax highlighting is nice
  2. wouldn’t it be nice to copy /paste it

The main problem is many terminals don’t for no especially good reason allow copy/paste of formatted text. There are three tools which help:

  1. xclip -t text/html this eats standard input and allows pasting of it as HTML, so it can include formatting and color and so on. By default, xclip does plain text, so you have to specify that it’s HTML.
  2. aha this takes ANSI formatted text (i.e. the formatting scheme used by terminals and turns it into HTML).
  3.  unbuffer many tools will only write color output to a terminal, not a pipe. Through the magic of PTYs (pseudoterminals) this program fools other programs into thinking they’re running on a terminal when they’re not.

Now all you have to do is pipe them together. So, take a silly C++ program like this:

void silly;

And compile it like this:

unbuffer g++-7 error.cc -std=c++1z  | aha | xclip -t text/html

Then a simple paste into a box which accepts HTML (e.g. email, etc) gives:

error.cc:2:6: error: variable or field 'silly' declared void
 void silly;
      ^~~~~

 

Overwrite a file but only if it exists (in BASH)

Imagine you have a line in a script:

cat /some/image.img > /dev/mmcblk0

which dumps a disk image on to an SD card. I have such a line as part of the setup. Actually, a more realistic one is:

pv /some/image.img | sudo 'bash -c cat >/dev/mmcblk0'

which displays a progress bar and doesn’t require having the whole script being run as root. Either way, there’s a problem: if the SD card doesn’t happen to be in when that line runs, it will create /dev/mmcblk0. Then all subsequent writes will go really fast (at the speed of the main disk), and you will get confused and sad when none of the changes are reflected on the SD card. You might even reboot which will magically fix the problem (/dev gets nuked). That happened to me 😦

The weird, out of place dd tool offers a nice fix:

pv /some/image.img | sudo dd conv=nocreat of=/dev/mmcblk0

You can specify a pseudo-conversion, which tells it to not create the file if it doesn’t already exist. It also serves the second purpose as the “sudo tee” idiom but without dumping everything to stdout.

A simple hack but a useful one. You can do similar things like append, but only if it exists, too. The incantation is:

dd conv=nocreat,notrunc oflag=append of=/file/to/append/to

That is, don’t create the file, don’t truncate it on opening and append. If you allow it to truncate, then it will truncate then append, which is entirely equivalent to overwriting but nonetheless you can still specify append without notrunc.

Warn or exit on failure in a shell script

Make has a handy feature where when a rule fails, it will stop whatever it’s doing. Often though you simply want a linear list of commands to be run in sequence (i.e. a shell script) with the same feature.

You can more or less hack that feature with BASH using the DEBUG trap. The trap executes a hook before every command is run, so you can use it to test the result of the previous command. That of course leaves the last command dangling, so you can put the same hook on the EXIT trap which runs after the last command finishes.

Here’s the snippet and example which warns (rather than exits) on failure:

function test_failure(){
  #Save the exit code, since anything else will trash it.
  v=$?
  if [ $v != 0 ]
  then
    echo -e Line $LINE command "\e[31m$COM\e[0m" failed with code $v
  fi
  #This trap runs before a command, so we use it to
  #test the previous command run. So, save the details for
  #next time.
  COM=${BASH_COMMAND}
  LINE=${BASH_LINENO}
}

#Set up the traps
trap test_failure EXIT
trap test_failure DEBUG

#Some misc stuff to test.
echo hello
sleep 2 ; bash -c 'exit 3'

echo world
false

echo what > /this/is/not/writable

echo the end

Running it produces:

$ bash errors.bash 
hello
Line 21 command bash -c 'exit 3' failed with code 3
world
Line 25 command false failed with code 1
errors.bash: line 27: /this/is/not/writable: No such file or directory
Line 27 command echo what > /this/is/not/writable failed with code 1
the end
$