MQTT (Message Queue Telemetry Transport) is a messaging wire communications protocol for machine to machine (M2M) and software component integration.
Use cases:
- Sensors / IOT devices
- Robotics, Industrial Monitoring & Remote Control
- Automation – Smart Home
- Mobile, Web Browser UI clients, Cloud Services
- Integration – local & long range communication between software & services
In contrast to world wide web, where browsers pull web pages from internet servers (request/response), MQTT implements a Publish / Subscribe (PubSub) and a push messaging model.
Common with HTTP at transport OSI network layer MQTT extends TCP/IP.
MQTT clients connect and transmit to brokers (servers), who register subscribers to named topics (channels).
Message data payload can contain arbitrary binary or UTF-8 Unicode encoded text data (up to 256 MB per message).
Internet of Things (IOT) devices deploy MQTT as a lightweight email (SMTP) like solution for data exchange suitable for integrating software, sensors, devices and cloud services.
An OASIS standard currently at version 5x ( http://mqtt.org/ ) defines capabilities including:
- Distributed, bi-directional message flows
- Quality of Service (QOS) delivery levels: at least once, only once, guaranteed message order
- Security: Authentication, Encryption – TLS, OpenSSL, OAuth
- Low power consumption / bandwidth
- Push-store-forward including to offline clients
- Browser client integration with WebSockets (RFC 6455)
Lets take a closer look at Mosquitto ( https://mosquitto.org/ ) an open source MQTT broker implementation sponsored by Apache Software Foundation .
Mosquitto MQTT Setup under Linux Ubuntu
Mosquitto can be installed on Ubuntu with package manager:
sudo apt-get install mosquitto mosquitto-clients -y
From a binary package ( https://mosquitto.org/files/source/ ) or compiled from source:
Download source:
mkdir mqtt cd mqtt git clone https://github.com/eclipse/mosquitto
Install Dependencies:
sudo apt-get install git cmake libc-ares-dev uuid-dev libssl-dev zlib1g-dev xsltproc docbook-xsl RFC6455 WebSockets sudo apt-get install libwebsockets-dev Unit Test Framework sudo apt-get install libcunit1 libcunit1-doc libcunit1-dev
Edit build config (eg add websockets)
vi config.mk
Build / Install
make clean make test make make install
Mosquitto File Paths / Commands:
Config File/etc/mosquitto/conf.d/default.conf
Logs/var/log/mosquitto/mosquitto.log
Start/stop/restart
sudo /etc/init.d/mosquitto restart [ ok ] Restarting mosquitto (via systemctl): mosquitto.service. 1594324669: Config loaded from /etc/mosquitto/mosquitto.conf. 1594324669: Opening ipv6 listen socket on port 1883. 1594324669: Opening ipv4 listen socket on port 1883. 1594324669: Opening ipv4 listen socket on port 8883. 1594331039: mosquitto version 1.6.10 starting
Mosquitto Password Authentication
Mosquitto user/password authentication can be required on a per listener basis.
Add broker authentication per listener in default.conf
listener 1883 localhost allow_anonymous false password_file /etc/mosquitto/passwd
Generate per user password credential –
sudo mosquitto_passwd -c /etc/mosquitto/passwd <user>
To prevent disclosure of password in shell command history an environment variable can be set in user profile –
MOSQUITTO_PASSWORD=<password>
Command line connect string references environment password variable
mosquitto_pub -h localhost -t test.out -m "test message #3" -u mqtt -P $MOSQUITTO_PASSWORD
Mosquitto SSL setup
Mosquitto man page has information on SSL and TLS setup.
Firewall port 8883 is opened (production deployment should restrict connection to known clients)
sudo ufw allow from any to any port 8883 proto tcp
To use self signed SSL certificates we create a Certificate Authority (CA) key –
sudo openssl req -new -x509 -days 360 -extensions v3_ca -keyout ca.key -out ca.crt
Next we generate and sign certificate for MQTT server, noting that key encryption is not supported –
sudo openssl genrsa -out server.key 2048 sudo openssl req -out server.csr -key server.key -new sudo openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 360
Finally we create a certificate for MQTT client –
sudo openssl genrsa -des3 -out client.key 2048 sudo openssl req -out client.csr -key client.key -new sudo openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out client.crt -days 360
Key Points –
- Server key must not use key encryption or Mosquitto raises error:
1594334396: Error: Unable to load server key file "/etc/mosquitto/certs/server.key". Check keyfile. 1594334396: OpenSSL Error[0]: error:2807106B:UI routines:UI_process:processing error 1594334396: OpenSSL Error[1]: error:0906406D:PEM routines:PEM_def_callback:problems getting password 1594334396: OpenSSL Error[2]: error:0906A068:PEM routines:PEM_do_header:bad password read 1594334396: OpenSSL Error[3]: error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib
- Common Name (CN) must be different across each of CA, Server and Client certs for example –
Common Name (e.g. server FQDN or YOUR name) []:
CA: selfcert.local
Server: mqtt.steveio.com
Client: ESP32
Otherwise following errors occur –
Server: 1594327583: OpenSSL Error: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca Client: stevee@ideapad-530S:~/ssl_cert$ sudo mosquitto_pub --insecure --cafile /etc/mosquitto/ca_certificates/ca.crt --cert /etc/mosquitto/certs/client.crt --key /etc/mosquitto/certs/client.key -h mqtt.steveio.com -p 8883 -q 1 -t "test" -i anyclientID --tls-version tlsv1.3 -m "Hello" -d Enter PEM pass phrase: Client anyclientID sending CONNECT OpenSSL Error[0]: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed Error: A TLS error occurred.
Mosquitto Broker SSL Configuration
Mosquitto Broker SSL default.conf Listener Configuration
listener 8883 mqtt.steveio.com cafile /etc/mosquitto/ca_certificates/ca.crt keyfile /etc/mosquitto/certs/server.key certfile /etc/mosquitto/certs/server.crt
Restart Mosquitto service and check logs
sudo service mosquitto restart
sudo cat /var/log/mosquitto/mosquitto.lo 1596376328: mosquitto version 1.6.10 starting 1596376328: Config loaded from /etc/mosquitto/mosquitto.conf. 1596376328: Opening ipv6 listen socket on port 1883. 1596376328: Opening ipv4 listen socket on port 1883. 1596376328: Opening ipv4 listen socket on port 8883.
Testing SSL connection with openSSL client
# use openssl client to test connection sudo openssl s_client -connect mqtt.steveio.com:8883 -CAfile /etc/mosquitto/ca_certificates/ca.crt
SSL Publish using Mosquitto broker command line
sudo mosquitto_pub --cafile /etc/mosquitto/ca_certificates/ca.crt --cert /etc/mosquitto/certs/client.crt --key /etc/mosquitto/certs/client.key -h mqtt.steveio.com -p 8883 -q 1 -t "test" -i anyclientID --tls-version tlsv1.3 -m "Hello" -d -u mqtt -P $MOSQUITTO_PASSWORD Enter PEM pass phrase: Client anyclientID sending CONNECT Client anyclientID received CONNACK (0) Client anyclientID sending PUBLISH (d0, q1, r0, m1, 'test', ... (5 bytes)) Client anyclientID received PUBACK (Mid: 1, RC:0) Client anyclientID sending DISCONNECT
SSL Subscribe using Mosquitto broker
sudo mosquitto_sub --cafile /etc/mosquitto/ca_certificates/ca.crt --cert /etc/mosquitto/certs/client.crt --key /etc/mosquitto/certs/client.key -h mqtt.steveio.com -p 8883 -q 1 -t "test" -i anyclientID --tls-version tlsv1.3 -d -u mqtt -P $MOSQUITTO_PASSWORD
MQTT Offline Clients – Store / Persist / Forward
A feature within MQTT specification is store / forward, enabling message delivery to offline or clients connecting only on a scheduled basis. Normally, messages are delivered immediately to all topic subscribers. When clients setup a persistent session, messages with QOS level 1 or 2 are queued for delivery in the event they are offline.
Requirements:
- QoS level: 1 / 2
- Fixed client ID
- Always connect with clean_session=False
- Subscriptions must be made with QoS>0
- Messages published must have QoS>0
Example Store / Forward Message Flow
Subscriber specifies qos level 1, client identity (-i), clean flag (-c)
mosquitto_sub -h localhost -t test.out -u mqtt -P $MOSQUITTO_PASSWORD -q 1 -c -i broker Client -c5646 sending CONNECT Client -c5646 received CONNACK Client -c5646 sending SUBSCRIBE (Mid: 1, Topic: test.out, QoS: 1) Client -c5646 received SUBACK Subscribed (mid: 1): 1
Publisher sends msg #1 to online client
mosquitto_pub -h localhost -t test.out -m "test message #1" -u mqtt -P $MOSQUITTO_PASSWORD -d -q 1 -i client Client -c8212 sending CONNECT Client -c8212 received CONNACK Client -c8212 sending PUBLISH (d0, q1, r0, m1, 'test.out', ... (15 bytes)) Client -c8212 received PUBACK (Mid: 1) Client -c8212 sending DISCONNECT
Subscriber (online) receives message #1
Client -c8186 received PUBLISH (d0, q1, r0, m1, 'test.out', ... (15 bytes)) Client -c8186 sending PUBACK (Mid: 1) test message #1
Subscriber disconnects, publisher sends message #2
mosquitto_pub -h localhost -t test.out -m "test message #2" -u mqtt -P $MOSQUITTO_PASSWORD -d -q 1 -i client
Subscriber connects, receiving message sent during offline
mosquitto_sub -h localhost -t test.out -u mqtt -P $MOSQUITTO_PASSWORD -d -q 1 -c -i broker Client broker sending CONNECT Client broker received CONNACK Client broker sending SUBSCRIBE (Mid: 1, Topic: test.out, QoS: 1) Client broker received PUBLISH (d0, q1, r0, m2, 'test.out', ... (15 bytes)) Client broker sending PUBACK (Mid: 2) test message #2 Client broker received SUBACK Subscribed (mid: 1): 1
There is a more detailed examination of persistence and store / forward here.
As MQTT matures it would be useful to see tools like Postfix qshape emerge for bottleneck and message queue administration.
