Categories
arduino C/C++ esp8266 Internet of Things Python Software WebSockets

WebSocket Binary – ESP8266 to Web Browser

Binary wire protocols are long established for embedded machine to machine (M2M) communication, network applications and wireless radio data transmission.

Internet of Things (IOT) devices, real time sensors, robotics, smart home of industrial machine control data also demand efficient, low latency & lightweight data communications.

WebSockets ( RFC6455 ) protocol brings native support for binary framed messaging to web browser clients, offering a compact lightweight format for fast and efficient endpoint messaging.

Why use binary format data messaging?

Compared to serialisation of more complex text based wire formats, binary is lightweight and requires minimal storage / bandwidth and processing.

Taking an example key/value command data message:

// JSON encoding
{"cmd":101,"value":180}
23 * 2 = 46 bytes

// CSV plain text encoding
101,180\n
9 * 2 = 18 bytes 

// Binary
101 180
int (4 bytes) + int(4 bytes)
4+4 = 8 bytes

In case of high performance applications supporting a large number of clients or very high frequency of data exchange, minimising data size, bandwidth and processing becomes an important priority.

Binary wire protocols are long established for embedded M2M messaging

Taking as a simple example an embedded ESP8266 WiFi device, message gateway and web browser client, data serialisation and bidirectional binary framed WebSocket data exchange are demonstrated.

ESP8266 Byte Array Serialisation

Internally data is represented in embedded microcontrollers as ones and zeros, sequences of bits arranged in addressable memory.

Higher level programming language abstraction provides human readable textual labels and in case of C/C++ associated type information.

Lets define a mixed type data structure that could be some kind of sensor or message data payload –

    // define mixed type data struct
    struct Data
    {
        int id;
        float v1;
        float v2;
        unsigned long v3;
        char v4[20];
    };

    struct Data data;

    // populate data values
    data.id = 67;
    data.v1 = 3.14157;
    data.v2 = -7.123;

    unsigned long ts = millis();
    data.v3 = ts;

    char c[20] = "N NE E SE S SW W NW";
    strncpy(data.v4, c, 20);

To access underlying bytes, a pointer to data structure address is created –

    uint8_t * bytePtr = (uint8_t*) &data;    
    webSocket.sendBIN(bytePtr, sizeof(data));

Data pointer and length are passed to WebSocket send method “webSocket.sendBIN()”, byte range is read, packaged (framed) according to protocol specification and written to TCP/IP network socket.

Hexidecimal and Binary text representation of in memory data structure can also be displayed –

void printBytes(const void *object, size_t size)
{
    const uint8_t * byte;
    for ( byte = (uint8_t *) object; size--; ++byte )
    {
        Serial.print(*byte, HEX);
        Serial.print("\t");
        Serial.println(*byte, BIN);
    }
    Serial.println('\n');
}

Python WebSocket Server

A Python3 middleware hosts WebSocket server and acts as a message relay gateway.

Binary WebSocket messages can be decoded in Python, the struct module performs conversions between Python data types and C structs –

async def wsApi(websocket, path):
    try:
        async for message in websocket:
            print('User-Agent: '+ websocket.request_headers['User-Agent'])
            print('Sec-WebSocket-Key: '+websocket.request_headers['Sec-WebSocket-Key'])
            print('MessageType: '+str(type(message)))
            print(message);
            print('Hex: '+message.hex());

            if isinstance(message, (bytes, bytearray)):

                i = message[:4];
                print(i);
                tuple_of_data = struct.unpack("i", i)
                print(tuple_of_data)

                tuple_of_data = struct.unpack_from("f", message, 4)
                print(tuple_of_data)

                tuple_of_data = struct.unpack_from("f", message, 8)
                print(tuple_of_data)

                tuple_of_data = struct.unpack_from("i", message, 12)
                print(tuple_of_data)

                tuple_of_data = struct.unpack_from("20s", message, 16)
                print(tuple_of_data[0])

                ## forward message
                await asyncio.wait([user.send(message) for user in USERS])

To index into byte array and read a number of bytes according to data type being unpacked Python’s array slice method “i = message[:4]” can be used where [<from>:<to>] specifies start/end positions.

Method struct.unpack_from() is another approach, taking as parameters a format character specifying data type (“i” – integer, “f” – float), data buffer and an index (in bytes) to read from.

Here is decoded binary message output including some WebSocket headers –

User-Agent: arduino-WebSocket-Client
Sec-WebSocket-Key: zoJ0aR/5XunSvEKKcUkWfQ==
MessageType: <class 'bytes'>
b'C\x00\x00\x00|\x0fI@\x9e\xef\xe3\xc0\xb9\x17\x00\x00N NE E SE S SW W NW\x00'
Hex: 430000007c0f49409eefe3c0b91700004e204e45204520534520532053572057204e5700
b'C\x00\x00\x00'
(67,)
(3.1415700912475586,)
(-7.123000144958496,)
(6073,)
b'N NE E SE S SW W NW\x00'

Web Browser – Binary Encode/Decode in JavaScript

In web browser, JavaScript primitives Blob, ArrayBuffer and TypedArray perform a similar conversion.

Firstly, received WebSocket messages (event object) can be debugged to console –

 websocket.onmessage = function (event) {
    console.log(event);

Binary framed data payload is reported as type “Blob” (raw data) of length 36 bytes –

Chrome Developer Tools console log for WebSocket Binary message receieve event

To de-serialise message, raw data Blob is converted asynchronously using FileReader API to ArrayBuffer, a generic fixed length binary data buffer –

    if (event.data instanceof Blob)  // Binary Frame
    {
      // convert Blob to ArrayBuffer
      var arrayPromise = new Promise(function(resolve) {
          var reader = new FileReader();

          reader.onloadend = function() {
              resolve(reader.result);
          };

          reader.readAsArrayBuffer(event.data);
      });

When promise is fulfilled, ArrayBuffer can be read using typed views (Uint32Array, Uint32Array) for integer (including long) and float types, TextDecoder API is used to decode character array –

arrayPromise.then(function(buffer) {

          // Decoding Binary Packed Data

          // int (4 bytes)
          var arrInt = new Uint32Array(buffer);
          var id = arrInt[0];
          console.log("id:"+id);

          // 2x float (4 bytes)
          var arrFloat = new Uint32Array(buffer,4);
          var v1 = arrFloat[0];
          var v2 = arrFloat[1];
          console.log("v1: "+v1);
          console.log("v2: "+v2);

          // long (4 bytes)
          var v3 = arrInt[3];
          console.log("v3:"+v3);

          // character data (20 bytes)
          var uint8Array = new Uint8Array(buffer,16);
          var string = new TextDecoder("utf-8").decode(uint8Array);
          console.log(string);
      });

JavaScript Binary Data Encoding

Binary data can also be encoded from native JavaScript. TypedArrays created for each data type – integer, float, long and character array are populated and packed into an ArrayBuffer suitable for use as WebSocket data payload –

console.log("Binary Encode example");

// Binary Encode example
var buffer = new ArrayBuffer(36)
var arrInt =   new Uint32Array(buffer, 0, 1);
arrInt[0] = 67;
var arrFloat = new Float32Array(buffer, 4, 2);
arrFloat[0] = 3.14157;
arrFloat[1] = -7.123;

var arrInt2 =   new Uint32Array(buffer, 12, 1);
arrInt2[0] = Date.now();

var uint8Array = new Uint8Array(buffer,16);
var charBuffer = new TextEncoder("utf-8").encode("N NE E SE S SW W NW");

for(var i = 0; i<charBuffer.length; i++)
{
  uint8Array[i] = charBuffer[i];
}

// send binary data
websocket.send(buffer);

At message gateway, logs demonstrate parity between data packed by embedded device and those sent from web browser client –

User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36
Sec-WebSocket-Key: 1TD9Zp71cMTivUbj+QSx5w==
MessageType: <class 'bytes'>
b'C\x00\x00\x00|\x0fI@\x9e\xef\xe3\xc0\x07\x1c\xff\x91N NE E SE S SW W NW\x00'
Hex: 430000007c0f49409eefe3c0071cff914e204e45204520534520532053572057204e5700
b'C\x00\x00\x00'
(67,)
(3.1415700912475586,)
(-7.123000144958496,)
(-1845552121,)
b'N NE E SE S SW W NW\x00'

Limitations / Drawbacks

Compared to UTF-8 text formats (XML, JSON) packed binary data has significant disadvantages –

  • legibility – text based key/value formats are easy to read, manipulate and maintain
  • fixed frame boundaries – using positional byte sequence indexes means even small changes to message structure, size or field position require updates to consumer client code
  • endianess / alignment / padding must be maintained consistently, compiler and platform implementation differences may occur

Security

WebSockets Secure (WSS) offers transport layer security (TLS) to encrypt data streams. An authentication and authorisation strategy (challenge/response password, token or certificate based) for client identification should also be deployed. Cryptographic message digest signing or encryption might also be used as extra protection for critical data.

Categories
C/C++ esp8266 Internet of Things microcontrollers WebSockets

ESP8266 IOT Microservo WebSocket

Internet of things devices can be networked wirelessly over internet and integrate easily with cloud, mobile and web applications.

WebSockets ( RFC6455 ) run on a variety of platforms including embedded systems and web browsers enabling low latency bidirectional binary socket communication over TCP/IP.

ESP8266 is a low cost Wi-Fi micro-controller compatible with Arduino open-source electronics framework designed for networked sensor, robotics and micro-electronic control applications.

A microservo controlled by ESP8266 with web & mobile interface

WebSocket MicroServo

In a simple example a micro-servo reports position data and is controlled via a potentiometer (rotary encoder), web and mobile interfaces.

This tutorial demonstrates:

  • Microservo / Potentiometer GPIO control
  • ESP8266 WebSocket Client
  • Binary format Messaging
  • Python WebSocket Library to synchronise state between clients
  • Responsive HTML5 / Javascript Browser Control Interface

Circuit Wiring

  • 5v regulated DC power supply
  • Microservo connected to ESP8266 GPIO 15
  • 100k Potentiometer – CLK -> GPIO 12, DT -> GPIO 13

Microservo Setup

Microservo power is provided by an external regulated 5v DC power supply sharing a common ground connection to microcontroller.

Servo library initialisation and global variables –

#include <Servo.h>
int servoPin = 15;
int angle; // current angle (degrees)
int servoStartAngle = 90; // initial position
int limit = 90; // range of servo in degrees
Servo Servo1;

// in setup()
Servo1.attach(servoPin);
rotateServo(0);

Servo position is set in a function by passing angle in degrees as a parameter –

void rotateServo(int angle)
{
  Serial.print("RotateServo: ");
  Serial.println(angle);
  Servo1.write(angle);
}

Potentiometer / Rotary Encoder Setup

Potentiometer, a type of incremental rotary encoder based on variable resistance, allows relative rotary motion to be tracked, 2 out-of-phase output channels indicate direction of travel.

#define outputA 12 // Rotary Encoder #1 CLK
#define outputB 13 // Rotary Encoder #2 DT

int counter = 0; // rotary encoder incremental position
int aState; // rotary encoder state comparator
int aLastState;  

void setup() {
   pinMode(outputA,INPUT);
   pinMode(outputB,INPUT);
       
   aLastState = digitalRead(outputA);
}

Rotary encoder CLK and DT pins are compared to determine direction of rotation. Calling map() converts counter (relative position) to angle –

void readRotaryEncoder()
{  

  aState = digitalRead(outputA);

   if (aState != aLastState){     
     // If the outputB state is different to the outputA state, that means the encoder is rotating clockwise
    if (digitalRead(outputB) != aState) { 
      if (counter < limit)
      {
        counter ++;
        angle = map(counter, -90, 90, 0, 180);
        rotateServo(angle);
      }
    } else {
      if (counter + limit > 0)
      {
        counter --;
        angle = map(counter, -90, 90, 0, 180);
        rotateServo(angle);
      }
    }
  } 
  aLastState = aState;
}

ESP8266 microcontroller, Potentiometer (Rotary Encoder), Microservo, 5v DC power supply

Websocket Binary Message Framing

WebSocket protocol natively supports binary framed messaging, offering a compact lightweight format for fast and efficient endpoint messaging.

To report and update microservo position command messages are passed between control interfaces and microcontroller as binary data.

A Python service running on server creates WebSocket and maintains and synchronises shared application state between connected clients.

In web browser user interface, Javascript also has native support for (un)packing binary data.

Data Serialisation

A C struct data_t encapsulates two fields – “cmd” (unit8_t) and “value” (int) – two commands are defined, one to report servo position and another to set a new position, value represents an angle.

To assist serialisation data_t is wrapped in a packet union –

// data message
typedef struct data_t
{
  uint8_t cmd;
  int value;
};

// message packaging / envelope
typedef union packet_t {
 struct data_t data;
 uint8_t packet[sizeof(struct data_t)];
};

#define PACKET_SIZE sizeof(struct data_t)

#define CMD_SERVO_ANGLE 12 // command to report servo position
#define CMD_SERVO_ROTATE 13 // move servo to a specified position

Instances of data structs are created representing send and receive messages.

// send / receive msg data structures
union packet_t sendMsg;
union packet_t receiveMsg;

// messaging function prototypes
void readByteArray(uint8_t * byteArray);
void writeByteArray();
void printByteArray();

// buffer 
uint8_t byteArray[PACKET_SIZE];

Methods allow struct data to be written and read from byte array buffer –

// read bytes from buffer
void readByteArray(uint8_t * byteArray)
{
  for (int i=0; i < PACKET_SIZE; i++)
  {
    receiveMsg.packet[i] = byteArray[i];
  }
}

// write data to buffer
void writeByteArray()
{
  for(int i=0; i<PACKET_SIZE; i++)
  {
    // msg into byte array
    byteArray[i] = sendMsg.packet[i]; 
  }
}

ESP8266 Wifi / Websocket Setup

Details for setting up ESP8266 wifi can be found here.

ESP8266 Websocket library is added by including headers, defining server IP address / port / path and creating a WebSocketsClient class instance.

#include <WebSocketsClient.h>

WebSocketsClient webSocket;

const char* ws_server = "192.168.1.127";
const uint16_t ws_port = 6789;
const char* ws_path = "/";

In setup WebSocket library is initialised –

void setup() {
    ...

    // server address, port and URL
    webSocket.begin(ws_server, ws_port, ws_path);
  
    // event handler
    webSocket.onEvent(webSocketEvent);
  
    // use HTTP Basic Authorization (optional)
    //webSocket.setAuthorization("user", "Password");
  
    // try again if connection has failed
    webSocket.setReconnectInterval(5000);

WebSocket – Message Receive

Event of type “WStype_BIN” defines handling when a binary format message is received, size is reported and hexdump() displays message contents –

void webSocketEvent(WStype_t type, uint8_t * payload, size_t length) {

  switch(type) {
    ...
    case WStype_BIN:
      Serial.printf("[WSc] get binary length: %u\n", length);
      hexdump(payload, length);
      setServoPosition(payload);
      break;

In setServoPosition() received byte array is de-serialised into message data structure. Angle field is used to update servo and potentiometer position.

void setServoPosition(uint8_t * byteArray)
{
  readByteArray(byteArray);
  printByteArray(receiveMsg);

  int angle = (int) receiveMsg.data.value;
  rotateServo(angle);

  counter = map(angle, 0, 180, -90, 90); // update rotary encoder
}

WebSocket – Message Send

Servo position is reported by populating sendMsg data structure, converting to byte array and calling webSocket.sendBIN() passing a pointer payload data and size.

// send servo position to Websocket server
void sendServoPosition()
{
  sendMsg.data.cmd = CMD_SERVO_ANGLE;
  sendMsg.data.value = angle;

  // write message to buffer
  writeByteArray();
  
  webSocket.sendBIN(byteArray, PACKET_SIZE);
}

Server Side – Python WebSocket (WS) Library

A simple WebSocket server implemented in Python v3x is tasked with text / binary message exchange, tracking connected clients and maintaining shared application state.

WebSocket server event loop is started by passing function/method name, IP address and port –

start_server = websockets.serve(wsApi, "192.168.1.127", 6789)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()

In event loop new clients are registered. Clients can be identified by checking protocol headers –

async def wsApi(websocket, path):
    # register(websocket) sends user_event() to websocket
    await register(websocket)

...
async def register(websocket):
    USERS.add(websocket)
    await notify_users()
    print(websocket.request_headers);
...

### debug output
Host: 192.168.1.127:6789
Connection: Upgrade
Upgrade: websocket
Sec-WebSocket-Version: 13
Sec-WebSocket-Key: LnOM1uj7n3gE4cFGqJ1yFg==
Sec-WebSocket-Protocol: arduino
Origin: file://
User-Agent: arduino-WebSocket-Client

Received message and headers are printed and handling is defined for binary (byte array) format messages.

### <class 'bytes'>
### b'\x0c\x00\x00\x00@\x00\x00\x00'
### (12, 64)

Python struct library allows packed binary bytes representing C structs to be unpacked as native python data types.

Format specifier “Ii” represents a message containing an int and unsigned int.

    try:
        async for message in websocket:
            print('Sec-WebSocket-Key: '+websocket.request_headers['Sec-WebSocket-Key'])
            print('MessageType: '+str(type(message)))
            print(message);
            if isinstance(message, (bytes, bytearray)):

                tuple_of_data = struct.unpack("Ii", message)
                cmd = tuple_of_data[0]
                value = tuple_of_data[1]
                STATE["value"] = value
                await notify_state()
    except Exception as e:
        print(e);
    finally:
        await unregister(websocket)

Connected WebSocket clients are push notified (synchronised) when position data (state) is updated –

async def notify_state():
    if USERS: 
        binary_data = struct.pack("Ii", 12, STATE['value'])
        await asyncio.wait([user.send(binary_data) for user in USERS])

WebSocket Browser Client – Binary Messaging in JavaScript

Web Interface is responsive and runs in mobile and web browser clients.

A radial D3.js radial gauge displays current angle. HTML5 buttons (divs) and slider control allow position to be updated.

Web / Mobile Interface provisioned by WebSocket (RFC6455)

JavaScript WebSocket onmessage function handles decoding of binary framed data.

A FileReader object is used to convert Blob to ByteArray in an asynchronous function, with flow control provided by a promise (future). TypedArray Uint8Array is used to extract 4 byte int and unsigned int command and value data fields –

websocket.onmessage = function (event) {
    if (event.data instanceof Blob)  // Binary Frame
    {
      async function readBinaryData(blob) {
          let promise = new Promise((res, rej) => {

            var fileReader = new FileReader();
            fileReader.onload = function(event) {
                var arrayBuffer = event.target.result;
                res(arrayBuffer);
            };
            fileReader.readAsArrayBuffer(blob);
          });

          // wait until the promise returns us a value
          let arrayBuffer = await promise;

          var v = new Uint8Array(arrayBuffer);
          // v[0] = cmd, v[4] = value
          //console.log(v[0] + " " + v[4]);

          // update UI elements
          value.textContent = v[4];
          angleSlider.value = v[4];
          gauges.forEach(function(g) {
              g.gauge.update(v[4]);
          });
      };

      readBinaryData(event.data);

Conclusion

WebSockets offer significant advantages over HTTP request/response polling techniques for real time data exchange, principally overhead of opening connection should occur only once per client.

Message push notify model, lightweight protocol framing, client libraries for embedded devices and native support in modern web browsers make this protocol well suited to real time Internet of Things data message exchange.

While underlying TCP provides message ordering and re-transmission (of failed packets), higher level application abstractions: guaranteed message delivery (at least once, at most once), message acknowledgements and queue / persist / forward (to offline clients) are not specified by WebSocket specification.

Similarly, TLS SSL can be used at network layer to encrypt data transmission (WebSocket Secure WSS), but client authentication / authorisation is not handled by WebSocket protocol, meaning a strategy for token or key based client identification (OAuth for example) must be considered for secure use cases.

Event driven implementations supporting asynchronous non-blocking IO result in efficient, well structured and modular code with handlers dedicated to specific tasks.

Factors influencing choice between binary framing and text format messages (field delimeted, JSON) include legibility, convenience, compactness and parsing overhead. Binary format introduces complexity due to differences in compiler / platform / network architectures and data type implementations between programming languages.

Full Source Code on GitHub –

Web UI –
https://github.com/steveio/arduino/blob/master/python/wsESP8266RotaryEncoderServo.html

Python WebSocket Server –
https://github.com/steveio/arduino/blob/master/python/wsESP8266RotaryEncoderServo.html

ESP8266 WebSocket MicroServo Sketch –
https://github.com/steveio/arduino/blob/master/ESP8266RotaryEncoderServo/ESP8266RotaryEncoderServo.ino

Categories
circuits Internet of Things microcontrollers sensors Software weather station

Weather Station Wind Vane

What types of sensor can be used for a weather vane? How to track angular position using a rotary encoder? How easy is calibration? What coding considerations for a weather station wind direction project?

Mesopotamian base-60 number system resulted in our idea of 360° in a full circle. Early compasses described 32 points and eight cardinal directions of wind, serving as navigational aids for maritime exploration.

References recorded in ancient China as early as 139 BC described “wind observing fan”. In classical Greece astronomer Andronicus constructed a weather vane at “tower of winds” in Athens. Weather vanes were known in many places of antiquity.

The word “vane” derives from Old English “fane” (Germanic Fahne) signifying “cloth, banner, flag” all of which can be deployed as visual wind direction indicators.

In modern times, absolute and incremental encoders are sensor devices measuring rotary position (angle) and motion. Resolution, precision and accuracy have distinct meaning.

Absolute encoders maintain position during power off or device reset. Incremental motion encoder data is relative, sensors of this type require “homing” (passing a known position) to calibrate.

Lets consider some types of rotary encoder

  • magnetic rotary encoder
  • 360° Potentiometer
  • optical encoder
  • magnetic sensor array

Magnetic Rotary Encoder

Contactless magnetic encoders track a dipole magnet attached to a rotating shaft above sensor, recording rotational angle and direction through a full turn of 360° with high resolution and precision.

Internally hall sensors measure angular position of a magnetic field presented at surface, converting this to a voltage.

On chip digital signal processing (DSP) amplifies and filters planar field data before conversion by Analogue to Digital conversion (ADC).

Having no mechanical friction leads to long expected life span.

A wide operating temperature range (-40 Deg.C to 150 Deg.C) and environmental tolerances (~80% humidity) allow for a wide potential application range.

2/3 wire I2C/SPI programmable interfaces provide standardised micro-controller connectivity and control.

AS5600 Datasheet
MLX90316 Datasheet

Potentiometer 360 degree

Several commercial wind vanes targeted at maritime applications deploy a 360° potentiometer connected directly to vane shaft.

Having a compact, space efficient design, high resolution (degrees of direction) can be tracked.

Detent (stops or clicks) add rotational resistance and a fixed set of positions but increase friction.

Electro-mechanical contacts are subject to mechanical wear and surface corrosion of contact track impacting accuracy, durability and longevity.

Optical Encoder

Optical incremental encoders – IR LED / Sensor pair with a spinning disk interrupter are accurate at very high RPM rates with low sensor latency (rise time).

Resolution is determined by interrupt light “chopper” disk design and relative position is measured by counting rotational sensor ticks.

Quadrature or two channel encoding, with a phase offset, is employed to determine rotational direction.

Calibration, including between device reset/power off is a challenge – sensor pulse counting during rotation must be relative to a fixed/known initial position.

Magnetic Sensor Array

Early compasses recorded 32 points to indicate winds as a navigational aid to sailors.

Wiring 32 sensors together requires considerable soldering & assembly skill. If 4 or 8 bit resolution is sufficient, magnetic linear hall or reed switches might be used – both are contactless, low cost and widely available.

Sensors arranged in a ring array activated by a rotating magnet allow a micro-controller to track position changes.

One approach is to use polling and a GPIO pin per sensor. Pin change interrupts can also be used for state notification.

An analogue multiplexer (CD4051) reduces number of required input pins to 4 (3 address pins, 1 data), optionally a common interrupt enables this to work with an event (interrupt) driven model.

Sensor Implementation – Polling vs Event Model

Polling, reading position at a set frequency (interval), provides consistency and allows simple computation analysis – roll-up averages for example. Higher frequency sampling results in higher precision.

In event model – an interrupt is triggered when sensor state (position) changes.

Recording position data only when direction changes is a low power consumption approach, extending operating duration of a battery powered device, especially on windless days.

To implement event driven design with a multiplexer poses a challenge, at circuit level a common interrupt line wired with isolating diodes to each sensor is required.

A change to any individual sensor triggers an interrupt, micro-controller can then check each multiplexer channel to determine position.

Calibration – how to determine magnetic north?

A compass bearing is required to determine direction relative to cardinal directions.

Wind vanes in a fixed position are manually calibrated. Electronic sensor devices can resolve orientation relative to magnetic compass.

Wind direction is defined by World Geodetic System (WGS) as direction from which wind blows and is measured clockwise from geographical north, namely, true north (meteorology) or in aviation reporting relative to magentic north. 

Visualisation – Wind Rose and Polar Distribution Charts

Wind roses, a type of polar bar chart provide a visualisation of wind distribution: direction and magnitude (velocity) frequency at a location over a given time interval.

Categories
arduino circuits Coding Internet of Things microcontrollers sensors Software

Weather Vane – Magnetic Sensor Rotary Encoder

A ring of 8 magnetic digital hall sensors (one per cardinal direction) are activated by a rotating neodymium magnet attached to a shaft, creating a simple rotary encoder.

Hall Effect Magnetic Sensor array connected to Arduino UNO microcontroller.

Input Pull-Up Resistors

Each hall effect sensor is wired to a digital micro-controller pin.

To prevent “floating”, input pin state is biased HIGH using pull-up resistors .

External pull-up 10k resistors are connected between hall effect sensor 5v+ and digital out pins.

If no external resistors are present GPIO pins should be setup as INPUT_PULLUP activating microcontroller internal 20k pull up resistor.

Polling for Active Pin

Each iteration of loop() reads input pins to determine active sensor.

// current and previous active sensor pin
int active = NULL;
int lastActive = NULL;

void loop() {

  int v;
  active = 0;

  for(int i = 3; i <= 10; i++)
  {
    v = digitalRead(i);

    if (v == 0)
    {
      active = i;
    }
  }
  if (active == 0) // magnet between sensor positions
  {
      active = lastActive;      
  }
  if (active != lastActive)
  {
    Serial.print(active);
    Serial.print("\t");
    Serial.println(directionLabel[active-3]);
  }

  lastActive = active;
}

Variables are maintained to track current and previous activation, direction is updated on position change.

If magnet is between sensor positions and no pin is active, last active position is reported.

Compass Direction Labels

Finally pin number is translated to direction (“N”, “NE”, “E” etc) by indexing into an ordered character pointer array.

// pin order direction labels
char d0[] = "NE";
char d1[] = "SE";
char d2[] = "E";
char d3[] = "S";
char d4[] = "N";
char d5[] = "W";
char d6[] = "NW";
char d7[] = "SW";

char * directionLabel[] = { d0, d1, d2, d3, d4, d5, d6, d7 };

...
// i == active sensor pin number 3 - 10 
Serial.println(directionLabel[i-3]);

Interrupts – Event Driven

Instead of polling (reading sensors on each loop() iteration) we can minimise processing and power consumption by updating direction only when magnet position changes.

Less power is consumed reading current position from a variable in flash memory compared to reading each sensor input pin – decoupling logic to maintain position from code reporting current value increases efficiency.

On Arduino (Uno, Nano etc) by default specific pins trigger external interrupts. Any GPIO pin can be used as an interrupt trigger with pin change interrupts.

To setup pin-change interrupts for digital pins 3 – 10 :

volatile int irqState = 0;
unsigned long lastIrq;
int irqDelay = 100; // millisecs

ISR (PCINT0_vect) 
{
  irqState = 1; 
}

ISR(PCINT2_vect)
{
  irqState = 1; 
}

void setupPinChangeInterrupt()
{
  cli();

  // 1 – Turn on Pin Change Interrupts
  PCICR |= 0b00000001;      // turn on port b (PCINT0 – PCINT7) pins D8 - D13
  PCICR |= 0b00000100;      // turn on port d (PCINT16 – PCINT23) pins D0 - D7

  // 2 – Choose Which Pins to Interrupt ( 3 mask registers correspond to 3 INT ports )
  PCMSK0 |= 0b00000111;    // turn on pins D8,D9,D10
  PCMSK2 |= 0b11111000;    // turn on pins D3 - D7 (PCINT19 - 23)

  sei();                     // turn on interrupts
}

A full example of setting up Arduino pin change interrupts, checking state and reading pins from data register can be found on github and there’s a useful guide here.

Now in loop() we can check for active pin only when interrupt event occurs, software de-bounce timeout prevents multiple repeat activations:

void loop() {

  if (irqState == 1 &amp;&amp; (millis() - lastIrq > irqDelay))
  {

    // check for active pin...

    lastIrq = millis();
    irqState = 0;
  }
}

Hardware Common Interrupt

A more portable solution can be implemented in hardware by adding a common interrupt line from each Hall Sensor input, isolating switch input with a diode which conducts only in one direction.

Now a change to any sensor input causes common interrupt (pin D2) to go LOW, signalling to micro-controller to check and update active magnet position.

1N4148 High Speed Signal Diode isolate common interrupt line

A single external interrupt can be handled by Arduino Uno/Nano pin D2

attachInterrupt(0, pin2IRQ, FALLING);

Power consumption can be reduced further by implementing deep sleep between sensor change interrupts, waking only to update state or transmit position data at intervals.

Arduino Nano v3 micro-controller tracking interrupt triggered magnetic switch position

Real Time Wind Compass Web Interface

D3.js Wind Compass UI has a design inspired by Dieter Rams who worked for Braun and is single HTML file adapted from a simple clock.

Units range is changed to 360 divided into sub-divisions of 10 and 45 (8 compass directions).

User Interface (UI) Data Provisioning

A finished product might transmit data wirelessly using LORA, Wifi, Bluetooth or 433mhz RF.

For prototype testing we can use serialToWebsocket.py a script based on Python’s PySerial library to capture serial console output and relay this to a websocket.

python3 serialToWebsocket.py 
connected to: /dev/ttyUSB0

3	S
5	SW
4	W
6	NW

We can use Python to run a simple webserver to develop and test our interface –

python -m SimpleHTTPServer 3001

Wind Compass can now be loaded in a browser –

http://127.0.0.1:3001/wsWindCompass.html

UI demo and source code can be found below –

See it in action –

Full source code can on github:

Arduino Wind Vane Sketch:
https://github.com/steveio/arduino/tree/master/WindVane8HallSensor
Wind Compass D3.js Web UI:
https://github.com/steveio/mqttWebSocket/blob/master/wsWindCompass.html
Serial to Websocket Python Script
https://github.com/steveio/arduino/blob/master/python/serialToWebsocket.py

Categories
Coding Internet of Things Python Software Uncategorized

Arduino Serial to Websocket in Python

What if we would like to publish data transmitted over RS232 Serial from an embedded Arduino device to a WebSocket browser client?

When prototyping a cable serial connection is very convenient as RF or networking modules may not yet be implemented. How do we get serial data to provision a cloud API service or web browser interface?

We can achieve this easily with Python and PySerial library:

#!/usr/bin/python

import serial
import asyncio
import datetime
import random
import websockets

ser = serial.Serial(
port='/dev/ttyUSB0',\
baudrate=115200,\
parity=serial.PARITY_NONE,\
stopbits=serial.STOPBITS_ONE,\
bytesize=serial.EIGHTBITS,\
timeout=0)

print("connected to: " + ser.portstr)


async def tx(websocket, path):
    line = []
    while True:
        for i in ser.read():
            c = chr(i)
            line.append(c)
            if c == '\n':
                print(''.join(line))
                await websocket.send(''.join(line))
                line = []
                break

start_server = websockets.serve(tx, "127.0.0.1", 5678)

asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()

ser.close()

Here is a more detailed example of reading serial port data in C language on Linux Platform –

https://blog.mbedded.ninja/programming/operating-systems/linux/linux-serial-ports-using-c-cpp/

Categories
arduino Coding Internet of Things nodejs Python Software

MQTT to Websockets with ESP32, NodeJS and D3.js

MQTT is a lightweight messaging protocol suitable for embedded and IOT devices.

Websockets ( RFC6455 – https://tools.ietf.org/html/rfc6455 ) is socket programming for internet, an evolution of browser / web server HTTP enabling real-time bidirectional data exchange and binary messaging.

How do we interface a MQTT enabled IOT sensor device with a web browser interface displaying a real time graph chart?

A real time web based barometric air pressure chart interface created with WebSockets and D3.js

Introduction to WebSockets – Real Time TCP Sockets for Internet

With modern browser engines and responsive web UI technologies built on HTML5, SVG and JavaScript frameworks, sophisticated visualisation, display and dashboard reporting capabilities have emerged.

Responsive Web UI runs in any browser installed device – laptop, dekstop, tablet or mobile, without need to install additional software or prepare application code for specific device architectures.

HTTP browser clients essentially implement a polling request/response technique for retrieving and updating HTML format webpages.

Due to need to establish a connection for each new request, HTTP is not well suited to real time or high volume messaging, charting or visualisation applications.

Although AJAX (asynchronous JavaScipt XML) and REST, SOAP API programming overcome this to a certain extent these methods are relatively inefficient for some use cases due to protocol overhead.

With Websockets, TCP network socket programming becomes possible in a browser client application.

Clients can establish a network socket connection, this channel remains open and two-way data exchange including binary messaging formats takes place.

Sockets are well established in UNIX and Windows OS client/server programming, but are relatively new to the web.

Arduino ESP32 Barometer Sensor MQTT Device

An ESP32 microcontroller with BMP280 environmental sensor and OLED LCD display

An environmental sensor based on an Expressif ESP32 micro-controller and BMP280 Bosch sensor reads air pressure, temperature and altitude –

#include <Adafruit_BMP280.h>

Adafruit_BMP280 bmp;

void setup() {

  if (!bmp.begin()) {
    Serial.println(F("Could not find a valid BMP280 sensor, check wiring!"));
    while (1);
  }
  
  /* Default settings from datasheet. */
  bmp.setSampling(Adafruit_BMP280::MODE_NORMAL,     /* Operating Mode. */
                  Adafruit_BMP280::SAMPLING_X2,     /* Temp. oversampling */
                  Adafruit_BMP280::SAMPLING_X16,    /* Pressure oversampling */
                  Adafruit_BMP280::FILTER_X16,      /* Filtering. */
                  Adafruit_BMP280::STANDBY_MS_500); /* Standby time. */  

}

void loop() {

  Serial.print(F("Temperature = "));
  Serial.print(bmp.readTemperature());
  Serial.println(" *C");

  Serial.print(F("Pressure = "));
  Serial.print(bmp.readPressure()/100); //displaying the Pressure in hPa, you can change the unit
  Serial.println(" hPa");

  Serial.print(F("Approx altitude = "));
  Serial.print(bmp.readAltitude(1019.66)); //The "1019.66" is the pressure(hPa) at sea level in day in your region
  Serial.println(" m");                    //If you don't know it, modify it until you get your current altitude

  display.clearDisplay();
  float t = bmp.readTemperature();           //Read temperature in C
  float p = bmp.readPressure()/100;         //Read Pressure in Pa and conversion to hPa
  float a = bmp.readAltitude(1019.66);      //Calculating the Altitude, the "1019.66" is the pressure in (hPa) at sea level at day in your region

  delay(2000);
}

Data is communicated over Wifi to an MQTT messaging server.

On Arduino we can use PubSub MQTT Library ( https://github.com/knolleary/pubsubclient ).

To set this up, first we define Wifi and MQTT server credentials and topic id (channels) and define a transmit data buffer:

const char* ssid = "__SSID__";
const char* pass = "__PASS__";

IPAddress mqtt_server(192, 168, 1, 127); // local LAN Address
const char* mqtt_user = "mqtt";
const char* mqtt_password = "__mqttpassword__";

const char* mqtt_channel_pub = "esp32.out";
const char* mqtt_channel_sub = "esp32.in";

WiFiClient wifi;
PubSubClient mqtt(wifi);

#define MSG_BUFFER_SIZE (128)
char msg[MSG_BUFFER_SIZE];

Then we setup Wifi connection:

  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }

  Serial.println("");
  Serial.println("WiFi connected");
  Serial.print("Connecting to ");
  Serial.println(ssid);
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());

And attempt to connect to MQTT broker, sending a hello message:

void loop() {
  
  while (!mqtt.connected()) {
    Serial.print("Attempting MQTT connection...");
    // Create a random client ID
    String clientId = "ESP8266Client-";
    clientId += String(random(0xffff), HEX);
    // Attempt to connect
    if (mqtt.connect(clientId.c_str(), mqtt_user, mqtt_password)) {
      Serial.println("connected");
      // Once connected, publish an announcement...
      mqtt.publish(mqtt_channel_pub, "hello ESP32");
      // ... and resubscribe
      mqtt.subscribe(mqtt_channel_sub);
    } else {
      Serial.print("failed, rc=");
      Serial.print(mqtt.state());
      Serial.println(" try again in 5 seconds");
      // Wait 5 seconds before retrying
      delay(5000);
    }
  }

  mqtt.loop();
}

After reading, we can transmit sensor values (Temperature T, Pressure P, Altitude A) on ESP32:

  snprintf(msg, MSG_BUFFER_SIZE, "%.2f,%.2f,%.2f", t, p, a);

  Serial.print("Publish message: ");
  Serial.println(msg);
  mqtt.publish(mqtt_channel_pub, msg);

MQTT to WebSocket RFC6455 – Node.JS Relay

On server we require a relay to subscribe for MQTT messages on sensor device channel, establish a WebSocket and write data to connected browser clients.

An implementation in NodeJS requires WS, MQTT and events libraries:

// setup Websocket Server
const WebSocket = require('ws');
var ws_host = "192.168.1.127";
var ws_port = "8080";
const wss = new WebSocket.Server({ host: ws_host, port: ws_port });
var ws = null;

// Setup MQTT Client
// mqtt[s]://[username][:password]@host.domain[:port]
var mqtt = require('mqtt'), url = require('url');
var mqtt_url = url.parse(process.env.MQTT_URL || 'mqtt://192.168.1.127:1883');
var auth = (mqtt_url.auth || ':').split(':');
var url = "mqtt://" + mqtt_url.host;
var mqtt_channel_in = "esp8266.in";
var mqtt_channel_out = "esp8266.out";

var options = {
    port: mqtt_url.port,
    clientId: 'mqttjs_' + Math.random().toString(16).substr(2, 8),
    username: 'mqtt',
    password: '__mqtt_password__',
    keepalive: 60,
    reconnectPeriod: 1000,
    protocolId: 'MQIsdp',
    protocolVersion: 3,
    clean: true,
    encoding: 'utf8'
};

NodeJS is event based, when an MQTT message is received it can be forwarded to all connected WebSocket clients:

mqttClient.on('message', function sendMsg(topic, message, packet) {

  console.log(topic + ": " + message);

  var eventListeners = require('events').EventEmitter.listenerCount(mqttClient,'message');
  console.log(eventListeners + " Listner(s) listening to mqttClient message event");
  console.log(mqttClient.rawListeners('message'));

  wss.clients.forEach(function each(ws) {
    if (ws.isAlive === false) return ws.terminate();
    console.log(data);
    ws.send(data+" ");
  });

});

MQTT allows many subscribers to receive topic messages.

Python Eclipse Paho MQTT client with Mongo DB

A client based on Eclipse Paho ( https://www.eclipse.org/paho/ ) developed in Python might add persistence by writing to a Mongo DB datastore:

### Python MQTT client
### Subscribes to an MQTT topic receiving JSON format messages in format:
###    [{"ts":1586815920,"temp":22.3,"pressure":102583,"alt":76}]
###
### Writes receieved JSON data to a mongo DB collection
###
import paho.mqtt.client as mqtt
import json
import pymongo

mqtt_server = "192.168.1.127"
mqtt_port = 1883
mqtt_keepalive = 60
mqtt_channel_out = "esp8266.out"
mqtt_channel_in = "esp8266.in"

mongo_server = "mongodb://localhost:27017/"
mongo_db = "weather"
mongo_collection = "sensorData"

def on_connect(client,userdata,flags,rc):
    print("Connected with result code:"+str(rc))
    print ("MQTT server: "+mqtt_server+", port: "+str(mqtt_port));
    print ("MQTT topic: "+mqtt_channel_out);
    client.subscribe(mqtt_channel_out)

def on_message(client, userdata, msg):
    print(msg.payload)
    parsed_json = (json.loads(msg.payload))
    res = sensorData.insert_one(parsed_json[0])

mongoClient = pymongo.MongoClient(mongo_server)
mydb = mongoClient[mongo_db]
sensorData = mydb[mongo_collection]

mqttClient = mqtt.Client()
mqttClient.connect(mqtt_server,mqtt_port,mqtt_keepalive);

mqttClient.on_connect = on_connect
mqttClient.on_message = on_message

mqttClient.loop_forever()

Web Browser Client – D3.js WebSocket Real Time Chart

WebSockets are supported natively in JavaScript by modern browser clients.

Setting up a WebSocket client, we consider re-connect attempts and parse received message data (JSON format in this example):

      <script type="text/javascript">

        var dataArr = [];

        var ws = null
        var maxReconnectAttemps = 10;
        var reconnectAttempts = 0;

        // setup WebSocket
        function setupWebSocket()
        {

          reconnectAttempts = 0;

          ws = new WebSocket('ws://192.168.1.127:8080',[]);

          ws.onopen = function () {
            console.log('WebSocket Open');
          };

          ws.onerror = function (error) {
            console.log('WebSocket Error ' + error);
          };

          ws.onmessage = function (e) {
            var rawData = e.data;
            if(rawData.trim().length > 1 &amp;&amp; rawData.trim() != "undefined")
            {
              try {

                var jsonObj = JSON.parse(rawData);
                jsonObj[0]['t'] = jsonObj[0]['t']; // temperature
                jsonObj[0]['p'] = jsonObj[0]['pressure']; // air pressure
                jsonObj[0]['a'] = jsonObj[0]['alt']; // altitude

                dataArr.push(jsonObj);

              } catch(e) {
                  console.log("Invalid JSON:"+rawData.toString());
              }
            }
          };
        }

        // check connection status every 60sec, upto maxReconnectAttemps, try reconnect
        const interval = setInterval(function checkConnectStatus() {
          if (reconnectAttempts++ < maxReconnectAttemps)
          {
            if (ws.readyState !== ws.OPEN) {
               console.log("WS connection closed - try re-connect");
               setupWebSocket();
            }
          }
        }, 60000);

        document.addEventListener("DOMContentLoaded", function() {
            setupWebSocket();
        });

Finally D3.js ( https://d3js.org/ ) is used to render chart as HTML5 / SVG.

D3.js real time barometer chart with WebSocket data provisioning

Source code and documentation can be found here:

Barometer D3.js:
http://bl.ocks.org/steveio/d549b0610fd489e6a09df8f2aa805ad3
https://gist.github.com/steveio/d549b0610fd489e6a09df8f2aa805ad3

ESP32 wifi Arduino MQTT sensor Client:
https://github.com/steveio/arduino/blob/master/ESP32SensorOLEDWifiMQTT/ESP32SensorOLEDWifiMQTT.ino

MQTT to WebSocket NodeJS relay:
https://github.com/steveio/mqttWebSocket

Categories
Internet of Things

MQTT – Internet of Things Messaging with Mosquitto

MQTT (Message Queue Telemetry Transport) is a messaging wire communications protocol for machine to machine (M2M) and software component integration.

Use cases:

  • Sensors / IOT devices
  • Robotics, Industrial Monitoring & Remote Control
  • Automation – Smart Home
  • Mobile, Web Browser UI clients, Cloud Services
  • Integration – local & long range communication between software & services

In contrast to world wide web, where browsers pull web pages from internet servers (request/response), MQTT implements a Publish / Subscribe (PubSub) and a push messaging model.

Common with HTTP at transport OSI network layer MQTT extends TCP/IP.

MQTT clients connect and transmit to brokers (servers), who register subscribers to named topics (channels).

A WiFi enabled ESP8266 device and Mobile (Android) interface

Message data payload can contain arbitrary binary or UTF-8 Unicode encoded text data (up to 256 MB per message).

Internet of Things (IOT) devices deploy MQTT as a lightweight email (SMTP) like solution for data exchange suitable for integrating software, sensors, devices and cloud services.

An OASIS standard currently at version 5x ( http://mqtt.org/ ) defines capabilities including:

  • Distributed, bi-directional message flows
  • Quality of Service (QOS) delivery levels: at least once, only once, guaranteed message order
  • Security: Authentication, Encryption – TLS, OpenSSL, OAuth
  • Low power consumption / bandwidth
  • Push-store-forward including to offline clients
  • Browser client integration with WebSockets (RFC 6455)

Lets take a closer look at Mosquitto ( https://mosquitto.org/ ) an open source MQTT broker implementation sponsored by Apache Software Foundation .

Mosquitto MQTT Setup under Linux Ubuntu

Mosquitto can be installed on Ubuntu with package manager:

sudo apt-get install mosquitto mosquitto-clients -y  

From a binary package ( https://mosquitto.org/files/source/ ) or compiled from source:

Download source:

mkdir mqtt
cd mqtt
git clone https://github.com/eclipse/mosquitto

Install Dependencies:

sudo apt-get install git cmake libc-ares-dev uuid-dev libssl-dev zlib1g-dev xsltproc docbook-xsl
RFC6455 WebSockets
sudo apt-get install libwebsockets-dev
Unit Test Framework
sudo apt-get install libcunit1 libcunit1-doc libcunit1-dev

Edit build config (eg add websockets)

vi config.mk

Build / Install

make clean
make test
make
make install

Mosquitto File Paths / Commands:

Config File
/etc/mosquitto/conf.d/default.conf

Logs
/var/log/mosquitto/mosquitto.log

Start/stop/restart

sudo /etc/init.d/mosquitto restart
[ ok ] Restarting mosquitto (via systemctl): mosquitto.service.
1594324669: Config loaded from /etc/mosquitto/mosquitto.conf.
1594324669: Opening ipv6 listen socket on port 1883.
1594324669: Opening ipv4 listen socket on port 1883.
1594324669: Opening ipv4 listen socket on port 8883.
1594331039: mosquitto version 1.6.10 starting

Mosquitto Password Authentication

Mosquitto user/password authentication can be required on a per listener basis.

Add broker authentication per listener in default.conf

listener 1883 localhost
allow_anonymous false
password_file /etc/mosquitto/passwd

Generate per user password credential –

sudo mosquitto_passwd -c /etc/mosquitto/passwd <user>

To prevent disclosure of password in shell command history an environment variable can be set in user profile –

MOSQUITTO_PASSWORD=<password>

Command line connect string references environment password variable

mosquitto_pub -h localhost -t test.out -m "test message #3" -u mqtt -P $MOSQUITTO_PASSWORD

Mosquitto SSL setup

Mosquitto man page has information on SSL and TLS setup.

Firewall port 8883 is opened (production deployment should restrict connection to known clients)

sudo ufw allow from any to any port 8883 proto tcp

To use self signed SSL certificates we create a Certificate Authority (CA) key –

sudo openssl req -new -x509 -days 360 -extensions v3_ca -keyout ca.key -out ca.crt

Next we generate and sign certificate for MQTT server, noting that key encryption is not supported –

sudo openssl genrsa -out server.key 2048
sudo openssl req -out server.csr -key server.key -new
sudo openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 360

Finally we create a certificate for MQTT client –

sudo openssl genrsa -des3 -out client.key 2048
sudo openssl req -out client.csr -key client.key -new
sudo openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out  client.crt -days 360

Key Points –

  • Server key must not use key encryption or Mosquitto raises error:
	1594334396: Error: Unable to load server key file "/etc/mosquitto/certs/server.key". Check keyfile.
	1594334396: OpenSSL Error[0]: error:2807106B:UI routines:UI_process:processing error
	1594334396: OpenSSL Error[1]: error:0906406D:PEM routines:PEM_def_callback:problems getting password
	1594334396: OpenSSL Error[2]: error:0906A068:PEM routines:PEM_do_header:bad password read
	1594334396: OpenSSL Error[3]: error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib

  • Common Name (CN) must be different across each of CA, Server and Client certs for example –

Common Name (e.g. server FQDN or YOUR name) []:
CA: selfcert.local
Server: mqtt.steveio.com
Client: ESP32

Otherwise following errors occur –

Server:
1594327583: OpenSSL Error: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca

Client:
stevee@ideapad-530S:~/ssl_cert$ sudo mosquitto_pub --insecure --cafile /etc/mosquitto/ca_certificates/ca.crt --cert /etc/mosquitto/certs/client.crt --key /etc/mosquitto/certs/client.key -h mqtt.steveio.com -p 8883 -q 1 -t "test" -i anyclientID --tls-version tlsv1.3 -m "Hello" -d
Enter PEM pass phrase:
Client anyclientID sending CONNECT
OpenSSL Error[0]: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
Error: A TLS error occurred.

Mosquitto Broker SSL Configuration

Mosquitto Broker SSL default.conf Listener Configuration

listener 8883 mqtt.steveio.com
cafile /etc/mosquitto/ca_certificates/ca.crt
keyfile /etc/mosquitto/certs/server.key
certfile /etc/mosquitto/certs/server.crt

Restart Mosquitto service and check logs

sudo service mosquitto restart
sudo cat /var/log/mosquitto/mosquitto.lo

1596376328: mosquitto version 1.6.10 starting
1596376328: Config loaded from /etc/mosquitto/mosquitto.conf.
1596376328: Opening ipv6 listen socket on port 1883.
1596376328: Opening ipv4 listen socket on port 1883.
1596376328: Opening ipv4 listen socket on port 8883.

Testing SSL connection with openSSL client

# use openssl client to test connection
sudo openssl s_client -connect mqtt.steveio.com:8883 -CAfile /etc/mosquitto/ca_certificates/ca.crt

SSL Publish using Mosquitto broker command line

sudo mosquitto_pub --cafile /etc/mosquitto/ca_certificates/ca.crt --cert /etc/mosquitto/certs/client.crt --key /etc/mosquitto/certs/client.key -h mqtt.steveio.com -p 8883 -q 1 -t "test" -i anyclientID --tls-version tlsv1.3 -m "Hello" -d -u mqtt -P $MOSQUITTO_PASSWORD 

Enter PEM pass phrase:
Client anyclientID sending CONNECT
Client anyclientID received CONNACK (0)
Client anyclientID sending PUBLISH (d0, q1, r0, m1, 'test', ... (5 bytes))
Client anyclientID received PUBACK (Mid: 1, RC:0)
Client anyclientID sending DISCONNECT

SSL Subscribe using Mosquitto broker

sudo mosquitto_sub --cafile /etc/mosquitto/ca_certificates/ca.crt --cert /etc/mosquitto/certs/client.crt --key /etc/mosquitto/certs/client.key -h mqtt.steveio.com -p 8883 -q 1 -t "test" -i anyclientID --tls-version tlsv1.3 -d -u mqtt -P $MOSQUITTO_PASSWORD

MQTT Offline Clients – Store / Persist / Forward

A feature within MQTT specification is store / forward, enabling message delivery to offline or clients connecting only on a scheduled basis. Normally, messages are delivered immediately to all topic subscribers. When clients setup a persistent session, messages with QOS level 1 or 2 are queued for delivery in the event they are offline.

Requirements:

  • QoS level: 1 / 2
  • Fixed client ID
  • Always connect with clean_session=False
  • Subscriptions must be made with QoS>0
  • Messages published must have QoS>0

Example Store / Forward Message Flow

Subscriber specifies qos level 1, client identity (-i), clean flag (-c)

mosquitto_sub -h localhost -t test.out -u mqtt -P $MOSQUITTO_PASSWORD -q 1 -c -i broker

Client -c5646 sending CONNECT
Client -c5646 received CONNACK
Client -c5646 sending SUBSCRIBE (Mid: 1, Topic: test.out, QoS: 1)
Client -c5646 received SUBACK
Subscribed (mid: 1): 1

Publisher sends msg #1 to online client

mosquitto_pub -h localhost -t test.out -m "test message #1" -u mqtt -P $MOSQUITTO_PASSWORD -d -q 1 -i client

Client -c8212 sending CONNECT
Client -c8212 received CONNACK
Client -c8212 sending PUBLISH (d0, q1, r0, m1, 'test.out', ... (15 bytes))
Client -c8212 received PUBACK (Mid: 1)
Client -c8212 sending DISCONNECT

Subscriber (online) receives message #1

Client -c8186 received PUBLISH (d0, q1, r0, m1, 'test.out', ... (15 bytes))
Client -c8186 sending PUBACK (Mid: 1)
test message #1

Subscriber disconnects, publisher sends message #2

mosquitto_pub -h localhost -t test.out -m "test message #2" -u mqtt -P $MOSQUITTO_PASSWORD -d -q 1 -i client

Subscriber connects, receiving message sent during offline

mosquitto_sub -h localhost -t test.out -u mqtt -P $MOSQUITTO_PASSWORD -d -q 1 -c -i broker

Client broker sending CONNECT
Client broker received CONNACK
Client broker sending SUBSCRIBE (Mid: 1, Topic: test.out, QoS: 1)
Client broker received PUBLISH (d0, q1, r0, m2, 'test.out', ... (15 bytes))
Client broker sending PUBACK (Mid: 2)
test message #2
Client broker received SUBACK
Subscribed (mid: 1): 1

There is a more detailed examination of persistence and store / forward here.

As MQTT matures it would be useful to see tools like Postfix qshape emerge for bottleneck and message queue administration.