An introduction to MQTT

A few years ago, I was asked to put temperature monitoring in a customer's server room and to integrate it with their existing monitoring and notification software. We ended up buying a rack-mountable temperature monitor, for nearly £200, that ran its own web server for propagating temperature data. Although the device ostensibly published data in XML, that turned out to be so painful to parse that we ended up screen-scraping the human-readable web pages to get the data. Temperature sensors are fairly cheap, but by the time you've wrapped them in a case with a power supply, an Ethernet port, a web server, enough of an OS to drive the above, and volatile and non-volatile storage for the same, they get expensive. I was sure that somewhere there must be physically-lightweight sensors with simple power, simple networking, and a lightweight protocol that allowed them to squirt their data down the network with a minimum of overhead. So my interest was piqued when Jan-Piet Mens spoke at FLOSS UK's Spring Conference on "Small Things for Monitoring". Once he started passing working demonstration systems around the room without interrupting the demonstration, it was clear that this was what I'd been looking for.

The lightweight protocol that he'd chosen is MQTT. This is an OASIS-standard (though not yet RFC) protocol for pub/sub (publish/subscribe) messaging and its transport on unreliable networks, the name of which, Mens says, no longer actually stands for anything. The server normally runs on TCP port 1883; the protocol has some idea of security, supporting TLS, authentication, ACLs, and the encryption of payloads, which can be up to 256MB in size. Mens said, though, that the use of payloads approaching that limit is "a little bit theoretical". A simple quality of service (QoS) flag is supported; setting this to 0 gives "fire and forget", or "delivery at most once"; 1 gives assured delivery, or "delivery at least once"; and 2 gives "delivery once only". Higher values, of course, impose higher overheads.

The protocol uses topic names to distinguish one message from another, though the arrangement of the namespace is entirely up to each implementer. These UTF-8 strings are hierarchical, looking a little like Unix filenames, and both single depth (+) and arbitrary depth (#) wildcards are supported. Examples he gave were temperature/room/living, devices/# (likely meaning "all devices" and including both devices/usb/scanner/0 and devices/scsi/disc/12) and finance/+/eur/rate (perhaps meaning "the euro rate from all providers"). Publishers publish data on the topic of their choice, and subscribers who have subscribed to that part of the namespace are informed of it.

The server software that enables all this is properly called the broker. There are a number of broker implementations, though the one Mens focused on is Mosquitto (the double-t is deliberate, and reflects the protocol's name). Well-behaved brokers, which include Mosquitto, support the concept of bridging; when two brokers are bridged, messages received by one are passed on to the other, so that the other broker's clients may subscribe to them if they choose.

Client-side, things are lightweight. Under Mosquitto, client-side software is limited to mosquitto_sub, for subscribing, and mosquitto_pub, for publishing. There are client implementations in Lua, Python, C, JavaScript, Perl, Ruby, Java, and many others (Mens thinks COBOL doesn't have one, but that's a rare exception). He gave some small (but functional) Python examples, using the Paho MQTT library. The publishing code was one substantive line long, while the subscription example was made eight lines longer by the asynchronous nature of subscription: you know when you're going to publish, but you don't know when somebody else is going to publish to a topic you're subscribed to, so callbacks to handle incoming data must be defined before entering the event loop. So the subscription example looked like:

    import paho.mqtt.client as paho
    def on_connect(mosq, userdata, rc):
         mqttc.subscribe("conf/+", 0)
    def on_message(mosq, userdata, msg):
         print "%s %s" % (msg.topic, str(msg.payload))
    mqttc = paho.Client(userdata=None)
    mqttc.on_connect = on_connect
    mqttc.on_message = on_message
    mqttc.connect("localhost", 1883, 60)
    mqttc.loop_forever()

There's nothing implicitly small or low-volume about MQTT — Mens mentioned that IBM's broker software can handle 15 million messages per second. However, it's seeing a lot of use in the Internet of Things (IoT). Mens here briefly but trenchantly expressed his opinion that IoT devices are best kept off the public internet, encouraging us to deploy in "walled garden" intranets unless we were sure we knew what we were doing.

At this point the demonstration began. Mens first passed around a GL-Inet AR150, a WiFi-enabled "pocket" router costing around €25, powered by an attached USB battery, and running OpenWrt. On top of that, it was running a Mosquitto broker. Mens explained that this demonstration required a broker, so he chose a small and lightweight platform to show that it could be done. But his real interest was in even smaller, cheaper devices that could act as MQTT clients.

To this end he introduced the ESP8266, a low-cost microcontroller with a full TCP stack. He described it as "like an Arduino" — you can use the same IDE for it, and it can be programmed in Lua, MicroPython, and C, among other languages — but it has integrated WiFi and it's really cheap. He showed a slide of several ESP8266-based boards, many of which came from Wemos, and all of which were significantly less than €5. He also showed a slide of an Electrodragon, an ESP8266-based board with a couple of 220V relays inside the package. The applications for home automation, or remote rebooting, are obvious, though he said the quality was on par with its €6 price tag - "the German TÜV would get heart attacks if they saw this". The Sonoff is another ESP8266-based appliance switch he's seen.

The Wemos D1 Mini is an ESP8266-based board costing around €5, that can take a variety of comparably cheap shields to add physical functionality, including a temperature/humidity sensor, and a button. He placed on the podium a Lego enclosure containing a D1 Mini running a counter and driving an Adafruit quadruple seven-segment display, and passed around a D1 Mini with a button shield and a USB battery pack. The latter D1 Mini was running about ten substantive lines of C code from the Homie-ESP8266 project; each button press and each release was published to the broker as a separate MQTT message, on a specific button-related topic. In turn, the ESP8266 driving the counter was subscribing to that topic, and on receiving notification of each new button press, incremented the counter. There was a little sleight-of-hand involving an injunction not to press the button more than once, on pain of buying Mens a gin-and-tonic, plus some code that double-counted each fourth button press, but this generated little protest from the crowd and I am optimistic that Mens later received several drinks.

The ESP8266 also supports over-the-air updates, and Mens has written a lightweight web application to help Homie users organize and, where necessary, update their sensors.

Returning to MQTT, Mens noted that if you have a sensor which reports on an MQTT topic every half-hour, a client that subscribes to that topic may not wish to wait up to 30 minutes to get its first data. MQTT addresses this through "retained" messages; when a client publishes a message to a topic, it can specify that the message is to be retained. The broker will send the most recent retained message on any topic to any subscribing client as soon as the client subscribes to that topic. This can be used to inform new clients both of most-recent readings, and of persistent data such as a sensor's location or name.

Similarly, the protocol supports a "last will and testament" message, which a publishing client can give to a broker to be published in a given topic at some future time, if the publishing client goes silent. If the broker doesn't hear from that client within the specified time, and the last communication with it didn't include a proper disconnection message, then the broker will publish the "last will and testament" message. The protocol also includes a keepalive message that the client can use to reassure the broker that all remains well, even if the client has no actual data payload to be published.

Mens found that, when processing data collected via MQTT, he was continually writing little utilities to pass the data into other systems. He finally decided to save time and write mqttwarn, a general-purpose tool for redistributing MQTT data into other systems; currently it supports more than 60 such systems, including Facebook, Twitter, and Slack, as well as via more-traditional channels such as ssh, SMTP, and XMPP. There is also check-mqtt, which allows a NAGIOS/ICINGA system to subscribe to a topic and extract data therefrom.

Although Mens has done a lot of work in this space, MQTT is by no means his pet project. It is used widely in the field by other free-software projects (for example, GitHub can publish to MQTT whenever your project receives a pull request or a new issue is opened) and in commercial endeavors (the electricity metering companies Flukso and RemakeElectric use MQTT). It seems to be gaining a lot of traction in spaces including alerting, metering, logging, location awareness, tracking, automation and control, and host monitoring. And probably some time in the next few weeks, it will also be used to monitor the temperature of the air going in and out of my co-located server.

[Thanks to the Linux Foundation, LWN's travel sponsor, for supporting my travel to the event.]

Index entries for this article
GuestArticles	Yates, Tom
Conference	FLOSS UK/2018

---

to post comments