Whats the right Messaging Protocol for me?

TL;DR Recap on messaging tools + using the PicoW x MQTT x DHT22

Some of these are typical on telecom, like Kafka

Others, on Healthcare, like RMQ

Intro

You might be playing around with your Pi:

• https://jalcocert.github.io/RPi/posts/rpi-mqtt/

Superset Setup

IoT Pi Project with Superset

Or with your PicoW, like I was doing recently:

PicoW Post

MQTT + HomeAssistant

PicoW Scripts Repo

Source Code on Github

Or maybe you are an architect withing D&A and need some clarity on messaging protocols.

As seen here:

1. MQTT
2. Kafka
3. RabitMQ

See https://www.geeksforgeeks.org/kafka-vs-rabbitmq/

• WebSockets are best for real-time communication where low latency is crucial.
• MQTT excels in low-bandwidth environments and is optimized for IoT applications.
• RabbitMQ is suitable for scenarios requiring reliable message delivery and complex routing logic.

There is also Web RTC (Real Time Communication).

MQTT

MQTT (Message Queuing Telemetry Transport) is a lightweight, open-source messaging protocol designed for efficient communication in situations where network bandwidth and power are limited.

MQTT (Message Queuing Telemetry Transport):

MQTT is a lightweight, publish-subscribe messaging protocol designed for constrained devices and low-bandwidth, high-latency or unreliable networks.

It provides a flexible and efficient mechanism for asynchronous, real-time communication between clients and servers.

MQTT is commonly used in IoT (Internet of Things) applications, telemetry systems, and messaging applications where real-time data streams need to be transmitted reliably and efficiently.

While MQTT can be used for real-time communication in various scenarios, it may not be as widely supported or as easy to integrate as REST APIs or WebSockets in certain contexts.

It’s primarily used in IoT (Internet of Things) systems, where devices communicate over unreliable or low-bandwidth networks.

MQTT works on a publish-subscribe model, where devices can publish messages to a topic and subscribe to topics to receive messages.

This decouples the sender from the receiver, making it ideal for scenarios where multiple devices need to communicate without needing direct knowledge of each other.

You can use MQTT with a Pi: https://jalcocert.github.io/RPi/posts/rpi-mqtt/

MQTT Use Case

Monitor a Car Battery Remotely using ESP8266

https://www.youtube.com/watch?v=VnGRFwDrLHo

RabbitMQ

• https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-base.rabbitmq/

RabbitMQ is an open-source message broker that facilitates communication between applications, services, or microservices in a distributed system.

It supports multiple messaging protocols, including AMQP (Advanced Message Queuing Protocol), which is the default protocol used by RabbitMQ for managing and routing messages.

RabbitMQ operates on a queue-based model, where messages are placed in queues by producers and consumed by consumers.

It ensures reliable delivery and helps decouple systems, allowing for asynchronous and scalable communication between different components.

Tools for MQTT

1. ThingBoards https://github.com/thingsboard/thingsboard

2. MQTTX

• mosquito mqtt
• mqttx

flatpak install flathub com.emqx.MQTTX

3. MQTT Explorer

4. MQTT with Python: https://www.emqx.com/en/blog/how-to-use-mqtt-in-python

Apache Kafka

https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-base.kafka/

Apache Kafka is a distributed event streaming platform that is used for building real-time data pipelines and streaming applications.

Kafka is designed to handle large volumes of high-throughput, fault-tolerant, and distributed data streams in a reliable and scalable manner.

Kafka works on a publish-subscribe model, where producers write data (events) to topics, and consumers read from those topics.

Kafka stores streams of records in topics and allows real-time processing of these records.

It is particularly known for its ability to handle massive amounts of data with low latency.

Conclusions

See also https://bunkeriot.github.io/BunkerM/

These tools will be useful:

1. VSCodium
2. THonny
3. Arduino IDE

MQTT x PicoW x DHT22

For the Pi4 it was very simple to get the pin schematics:

pinout 
#sudo apt install python3-gpiozero

So I got to know about Schemdraw and Fritzing.

uv pip show schemdraw

See the PicoW datasheet: https://datasheets.raspberrypi.com/picow/pico-w-datasheet.pdf

And pinout: https://datasheets.raspberrypi.com/picow/PicoW-A4-Pinout.pdf

As per this video

Which has this related code

• DHT22 data to GP15
• Possitive (VCC) to VBUS from the PicoW
• Negative to ground (GND) of the PicoW

After you upload the DHT22.py and the main…

We can finally ready DHT22 temp and humidity data!

So now, lets just combine this knowledge with the PicoW MQTT setup:

Making some adjustments, we will be pushing now DHT22 info and the Internal temp to MQTT:

Lets connect back to EMQX to verify: http://192.168.1.11:18083 via admin/public or your configured pwd.

You will see couple of clients: the PicoW and your HA, if you have just followed this post.

• http://192.168.1.11:18083/#/clients

Go back to http://192.168.1.11:18083/#/websocket to connect

And now you should be able to subscribe to:

1. pico/temperature/internal
2. pico/humidity/dht22
3. pico/temperature/dht22

Here is a quick OBS video + ffmpeg on how to make the HA setup:

for f in *.mkv; do ffmpeg -i "$f" -c copy "${f%.mkv}.mp4"; done

I made the video edit very quick!

Outro

If you want to get into electronics, see these:

1. Arduino IDE
2. LibrePCB

REST API (HTTP): RESTful APIs are widely used for communication between clients and servers over HTTP. They are well-suited for request-response interactions and are easy to understand and implement. However, they may not be the best choice for real-time communication as they are based on the request-response model, which can introduce latency and overhead for real-time updates. REST APIs are suitable for scenarios where real-time updates are not critical or where the frequency of updates is low.

WebSockets: WebSockets provide full-duplex communication channels over a single TCP connection, enabling real-time, bidirectional communication between clients and servers. They offer low-latency, high-performance communication and are well-suited for applications requiring real-time updates, such as chat applications, live dashboards, and multiplayer games. WebSockets can be more complex to implement compared to REST APIs, but they offer significant benefits for real-time applications.

MQTT (Message Queuing Telemetry Transport): MQTT is a lightweight, publish-subscribe messaging protocol designed for constrained devices and low-bandwidth, high-latency or unreliable networks. It provides a flexible and efficient mechanism for asynchronous, real-time communication between clients and servers. MQTT is commonly used in IoT (Internet of Things) applications, telemetry systems, and messaging applications where real-time data streams need to be transmitted reliably and efficiently. While MQTT can be used for real-time communication in various scenarios, it may not be as widely supported or as easy to integrate as REST APIs or WebSockets in certain contexts.

Redis as a message broker

Redis can be used as a message broker, and in the context of publish/subscribe (Pub/Sub) functionality, it shares similarities with message brokers like MQTT and RabbitMQ.

Redis as a Message Broker:

• Pub/Sub Mechanism: Redis has built-in Pub/Sub capabilities. Clients can subscribe to channels, and when a message is published to a channel, all subscribed clients receive it. This allows for a decoupled communication model where publishers don’t need to know about specific subscribers.

• Data Structures for Queuing: Besides Pub/Sub, Redis’s list data structure can also be used to implement basic message queues. Producers can LPUSH (left push) messages onto a list, and consumers can BRPOP (blocking right pop) to retrieve and process messages. Redis Streams, introduced later, offer more advanced features for message streaming, including persistence, consumer groups, and acknowledgments.

• Simplicity and Performance: Redis is known for its speed and simplicity. Its in-memory nature allows for very low latency message delivery. Setting up and using Redis for basic messaging is generally straightforward.

• “At-Most-Once” Delivery (for basic Pub/Sub): By default, Redis Pub/Sub offers “at-most-once” delivery semantics. This means that if a subscriber is disconnected when a message is published, it will not receive that message upon reconnection. There’s no built-in mechanism for message persistence or guaranteed delivery in the basic Pub/Sub. Redis Streams offer more reliable delivery options.

MQTT (Message Queuing Telemetry Transport):

• Specifically Designed for Messaging: MQTT is a lightweight messaging protocol specifically designed for constrained devices and low-bandwidth, unreliable networks, often used in IoT (Internet of Things) scenarios.

RabbitMQ:

• Full-Featured Message Broker: RabbitMQ is a robust and feature-rich message broker that supports multiple messaging protocols (including AMQP, MQTT, STOMP).

In the context of a Flask SSE application:

Redis can be a good choice as a message broker to decouple your event producers from your Flask SSE endpoint, especially if you have multiple sources of real-time data or if you anticipate scaling your application.

You would typically use Redis’s Pub/Sub or Streams to distribute events to your Flask application, which then formats them as SSE events and sends them to connected browsers.

While Redis offers messaging capabilities, it’s not a direct replacement for all use cases of MQTT or RabbitMQ, especially those requiring strong delivery guarantees, complex routing, or protocol-specific features.

The best choice depends on the specific requirements and complexity of your application.

For simpler real-time updates and scenarios where some message loss is acceptable (or handled at the application level), Redis can be a performant and easy-to-manage option.

Cloud for IoT

• https://cloud.google.com/free
  • PUB/SUB https://cloud.google.com/free/docs/free-cloud-features#pub-sub

• AWS IoT - https://www.youtube.com/watch?v=hgQ-Ewrm48c

Others

AMQ vs RMQ

ActiveMQ is a powerful, open-source message broker written in Java.

It acts as a middleman for communication between different applications, allowing them to send and receive messages asynchronously.

This means that a sender doesn’t need the receiver to be online and available at the exact same time; the messages are stored in ActiveMQ and delivered when the receiver is ready.

This is a key concept in “message-oriented middleware” (MOM) and is essential for building reliable, scalable systems.

RMQ is a common abbreviation for RabbitMQ.

ActiveMQ and RabbitMQ (RMQ) are similar in that they are both message brokers that perform similar functions.

However, they have distinct differences in their architecture, features, and ideal use cases.

Here’s a breakdown of the similarities and differences:

Similarities

• Both are Message Brokers: They both act as a central hub for messages, storing them in queues or topics until a consumer application is ready to process them.
• Asynchronous Communication: They enable decoupling between applications, allowing them to communicate without being directly connected in real time.
• Support for Multiple Protocols: Both support various messaging protocols like AMQP, MQTT, and STOMP, which allows them to be used with a wide range of clients and programming languages.
• High Availability and Scalability: Both offer features for creating clusters of brokers to ensure high availability and distribute the message load.

Key Differences

In summary, while both ActiveMQ and RabbitMQ serve the same general purpose of being a message broker, their underlying architecture and strengths differ.

Your choice between the two would depend on your specific project needs, especially concerning the programming language you’re using (ActiveMQ for a heavy Java environment, for example), performance requirements, and the complexity of your message routing logic.