Thing Minder

Once upon a time, IBM developed the lightweight, TCP-based protocol MQTT. The name was an acronym for "Message Queuing Telemetry Transport"; however, the long form of the name disappeared with the ratification of MQTT as an official OASIS standard at the end of 2014.

MQTT is a pub/sub protocol. Clients connect to a server known as a broker and, if they are authorized, they can issue messages or wait for messages to arrive. By default, MQTT uses TCP port 1883 and TLS-TCP port 8883.

Unlike HTTP, MQTT does not require polling by the client; instead, the broker informs waiting subscribers as soon as a message arrives and communicates the message to them. This approach saves bandwidth and allows communication with lower latency.

The broker doesn't know what the clients do with the messages; the broker's job is over and done with as soon as the message is delivered. MQTT enables a decoupled 1-to-n or 1-to-1 relationship between different devices or programs that do not specifically need to know each other in order to communicate. The entities just need to understand each other's data.

MQTT clients, both subscribers and publishers, can come and go at any time. New clients can connect to the server at any time. However, an MQTT broker usually has access control lists (ACLs) that govern access to certain topics.

Topic, a Name String

Each message is associated with a topic. Topics are hierarchically structured name strings in UTF-8 with a length up to 64KB, and they resemble Unix paths. Unlike path names, topics do not start with a leading forward slash. For example – staying with the Internet of Things (IoT) theme – a lamp could publish its state via a lamp/kitchen/state topic, while the lamp/living_room/switch topic switches a lamp in the living room on or off. On the same broker, a fan can publish its state via air_conditioning/fan/number/1.

If certain clients want to find out the state of the lamp in the living room, they can request messages using lamp/living_room/state. If they use lamp/+/state instead, they will find out as soon as someone switches any lamp on or off. Clients can receive all activities for all lamps using lamp/#.

The special character + stands for a topic level and can occur multiple times (lamp/+/+). The # character, however, can only occur once, and always at the end of the topic; it means "this level and all underlying topic levels." Simply specifying # will show a client everything. Clients don't actually know whether they really see all the messages that arrive at the broker; the broker ACLS decide this.

Messages transport payloads; payload data can also be binary and be up to 256MB in size. This restriction is more theoretical because microcontrollers, which only have a few kilobytes of memory, also often speak MQTT and it isn't possible for them to handle such a volume of data. Payloads are often simple strings (23, to indicate a temperature) or data packed in JSON ({"celsius": 23.0 }).

Last Will and Testament

If a message is published with a topic, the broker distributes it to all subscribers following the topic. If nobody is currently following that topic, the message is lost – it isn't cached because there is no queue. To prevent this behavior, users can set a retained flag when publishing. This flag prompts the broker to store the message and to offer it to future subscribers. If, for example, the outside temperature is published with temp/terrace and a client subscribes to this topic hours later, this client is immediately sent this recently published temperature value (last-known value) if the retained flag is set.

What happens if a client unexpectedly fails? LWT (Last Will and Testament) is an important feature of MQTT. A client tells the broker during the connection setup that: "In case of my death, please send the following message with this topic …" If the broker then detects that this client is no longer alive, it does as it was told and publishes the message with the desired topic. MQTT thus has monitoring installed: It's easy to write to clients that respond to the passing of other clients, such as by sending alerts.

LWT messages are often used together with the retained flag in order to permanently describe the state of a client: When starting, a client will, for example, publish a status message with a retained flag with the clients/id/status topic. If this client deceases, the broker would distribute the dead message with a retained flag and the same topic on its behalf. A query on clients/id/status thus always reveals whether the client is online or offline.

Think of an MQTT broker as a large pot into which messages fall. Anyone can help themselves from this pot at any time. It then processes the data and publishes it again under certain circumstances.

You could, for example, imagine that a temperature sensor periodically publishes readings in Fahrenheit with the temp/fahrenheit topic. A trivial program subscribes to this topic, reads the values, converts them into degrees Celsius and publishes them again (throws them back into the pot), this time with the temp/celsius topic. The program is then simultaneously a subscriber and publisher.

Quality Control

MQTT is based on TCP and benefits from its high transmission reliability. Nevertheless, MQTT uses three service qualities to make sure messages are transmitted. These guarantees range from no assurance (Level 0) to assurance that the message at least arrive once (Level 1), to the promise that a message will arrive exactly once (Level 2).

As you can see in Figure 1, the three Quality of Service (QoS) levels are increasingly communicative: QoS Level 2 requires a fourway handshake between the client and broker. QoS 0 is usually very reliable in LAN. QoS 0 is usually enough if telemetry data is transmitted. It's not so bad if the outside temperature published every minute fails now and then – it'll be sent again a minute later anyway (QoS 0 is fine). Users might want to transmit the position of a mobile device with a bit more reliability (QoS 1 or 2), although the network traffic and costs rise, too. The higher the specified QoS level, the more network traffic you create.

Brokers

MQTT brokers occupy a broad range from open source software to expensive appliances. The open source Mosquitto [1] is a lightweight, but powerful, broker developed in C with ACLs, with authentication plugins written in C and TLS (Transport Layer Security). Mosquitto also supports client authentication with TLS client certificates. Although you will find it can be found in most repositories, we recommend the packages from the project site.

A new candidate called Vernemq [2] was written in Erlang and supports clustering. Hivemq [3], written in Java, has proved its worth in the commercial environment: Hivemq also comes with clustering and a plugin system. Mosquitto, Vernemq, and Hivemq all support WebSockets, even if Mosquitto users have to compile it in.

Mosquitto's default configuration is ideal for experimenting. Users need to adapt it first if they want to add authentication and TLS. However, you don't need to preconfigure either topics or client. Mosquitto's command-line tools are useful for initial experiments (Listing 1). Subscribing watches all topics ( mosquitto_sub ), and publishing (mosquitto_pub) publishes a message from the second terminal that arrives at the subscriber.

01 # in the first terminal
02 mosquitto_sub -v -t '#'
03
04 # in the second terminal
05 mosquitto_pub -t linux/magazine -m "Hello world"
06
07 # with Retain Flag
08 mosquitto_pub -r -t linux/magazine -m "Hello World"

If the subscriber is simply restarted, it doesn't receive the old messages – it only receives the message if it was started before the publisher. However, if users use the retained flag in the final example from Listing 1, MQTT treats its future subscriber to the linux/magazine topic. The older message is submitted later in this case. If the subscriber remains connected, it will see the new message on this topic from now on. If the subscriber reconnects, it will receive the retained message and then all new messages.

Choosing a broker depends on your personal requirements: Is clustering support required or is a unique solution with HA proxy available? Is Java reliable on the system? Is commercial support needed? Can your own people master the necessary programming languages (C, Erlang, Java) to develop their own broker plugins?

Bridges

Not all client APIs, and thus not all devices that communicate via MQTT, have all protocol features. Microcontrollers, for example, are typically not capable of using TLS because of the meager hardware equipment.

If, however, you want to safely use such devices in the network, you should use a bridge for help. In this scenario, place MQTT clients in the vicinity of a broker, which then takes on the tasks of sending a configurable list of topics to remote brokers and receiving lists from remote brokers to distribute to local clients. Without TLS support, the clients are only exposed on the short network section to the nearest broker.

A bridge does not necessarily have to transport all topics – instead, you can define which topics it should accept and which it should send again. This design allows finely tuned configurations that don't just blurt out all data. In Figure 2, the P and S clients are capable of exchanging messages amongst themselves about both brokers, and system administrators determine which of the two brokers address the other first (keyword: firewall).

It gets more difficult if redundancy comes into play – for example, if a second broker is to step in when the first one fails. This isn't possible in Mosquitto, but Vernemq and Hivemq are both equipped with clustering capabilities. Whether a client in a two-node cluster publishes via node A or B is irrelevant – the nodes try to act as brokers.

Sys Admin

MQTT can render useful services for system administrators in many situations. If a system administrator has started a long running shell script and wants to know sometime later whether it is still carrying out the job, the admin can simply check whether a few instructions like mosquitto_pub appear now and then in the script.

These instructions are signs of life that a smaller broker following localhost can easily receive. The trivial function

progname=${progname:=$(basename $0)}
topic="processes/${progname}"
  mqtt_opts="--quiet -h localhost -p 1883"
function mqtt() {
   # Ignore PUB errors
   mosquitto_pub ${mqtt_opts} -t "${topic}" -m "$*" || true
}

ensures that you can look for the rights with a subscribe on the broker (see Listing 2).

01 #!/bin/sh
02
03 source mqttfuncs.sh
04
05 mqtt Start
06
07 # for ... # long loop
08 mqtt "process now xyz `date`"
09 # done
10
11 mqtt "So, now it is the end"

For Programmers

The mosquitto_* utilities are essential for tests or for embedding in shell scripts. MQTT APIs exist for many languages, from Java and JavaScript, to C, C#, Objective-C, Lua, Perl, and Python. The subscriber written in Python initially arranges a callback to be invoked when a message arrives (Listing 3).

01 #!/usr/bin/env python
02
03 import paho.mqtt.client as paho   # pip install paho-mqtt
04
05 def on_message(mosq, userdata, msg):
06     print "%s (qos=%s, r=%s) %s" % (msg.topic, str(msg.qos), msg.retain, str(msg.payload))
07
08 mqtt = paho.Client()
09 mqtt.on_message = on_message
10
11 mqtt.connect("localhost", 1883, 60)
12 mqtt.subscribe("linux/+", 0)
13
14 mqtt.loop_forever()

The client now connects to the broker and goes into a continuous loop in a controlled manner. If a message arrives from the broker with the right topic, the callback is executed, which outputs the message and stores it in a database.

Anyone wanting to monitor whether the client is still alive can insert a last will and testament into the script from Listing 3:

# Arrange LWT
mqtt.will_set('alerts/attention', payload="Died: %s" % sys.argv[0], qos=0, retain=False)

The user now receives information as soon as the client terminates (with Ctrl+C).

The project mqttwarn [4] provides a Python tool with many plugins for such monitoring (Figure 3). Mqttwarn links topics that it follows with services it invokes. The whole thing is controlled using a configuration file, as shown in Listing 4.

01 [defaults]
02 launch   = log, pushover
03
04 [config:log]
05 targets = {
06     'info'   : [ 'info' ],
07   }
08
09 [config:pushover]
10 targets = {
11     'linux'  : ['ml1234567890..', 'buzQx...'],
12   }
13
14 [alerts/attention]
15 targets = pushover:linux, log:info
16 title: Linux Magazine
17 format = {payload}

WebSockets

Modern web browsers can use MQTT with the help of WebSockets. The Paho project [5] provides a JavaScript module for MQTT. All brokers have WebSockets support on freely configurable ports. Note that communication via WebSockets does not run via the standard MQTT port 1883.

The need to perform HTTP polling and refreshes is eliminated when you have WebSockets in the browser: The browser receives messages as soon as MQTT clients publish them via the broker. This process provides interactive applications: With just a bit of skill in HTML and JavaScript, you can develop interesting monitoring panels, such as a panel for representing sensor or position data for mobile devices in real time.

In Real Life

MQTT is used outside of the IoT in various products. An MQTT plugin was recently developed for the statistics daemon Collectd [6] and for the logging system Graylog [7]. The open source project OwnTracks [8] supplies iOS and Android clients that provide positioning data to the MQTT broker via MQTT; friends and family members can find each other (see Figure 4).

Anyone who deals with home automation and openHAB can also benefit from MQTT [9]: You'll discover a variety of techniques for controlling openHAB using MQTT or obtaining openHAB status information via MQTT. MQTT is also in use with all the major companies: Facebook used MQTT for its messenger service due to the lower latency [10].