RPi IoT Project - Temperature Data with MongoDB, Databricks & Metabase

Raspberry Pi - Mongo and Databricks

Raspberry Pi Project to collect and then send IoT Sensor Data to Mongo and use Analyze it with Databricks.

sequenceDiagram
    participant Job
    participant PythonScript as The Python Script
    participant MongoDBCloud as MongoDB Cloud Setup
    participant Databricks

    Job ->> PythonScript: Description
    Job ->> MongoDBCloud: Description
    Job ->> Databricks: Description

This diagram is generated with MermaidJS and FontAwsome

ToDo list

We have already used Mongo with a DHT sensor.

How about pushing the data to the cloud and analyze it there with Databricks?

• Job
  • The Python Script - Pushing Data from Python to Mongo is Done already - From Project: Py to Mongo To Metabase
  • MongoDB Cloud Setup
  • Databricks

MongoDB Cloud Setup

Get a Cloud VM running (up to you to choose the provider) and get Docker and Docker-Compose installed.

You can Install Portainer as well and use the Stack I prepared for Mongo, or just execute this from the CLI to spin a MongoDB container.

docker run -d \
  --name mongodb \
  -e MONGO_INITDB_ROOT_USERNAME=yourusername \
  -e MONGO_INITDB_ROOT_PASSWORD=yourpassword \
  -e MONGO_INITDB_DATABASE=test \
  -v mongodb_data:/data/db \
  -p 27017:27017 \
  --restart always \
  mongo:latest

Get the hostname with:

curl -sS https://ipinfo.io/json #the command to use

Make sure that the MongoDB port is ready to accept connections, by default is 27017 TCP.

The Python Script

The related source code 👇

Make your Raspberry Pi run this container:

version: '3'
services:
  dht_sensor_mongo:
    image: dht_sensor_mongo:latest  # Use the name of your custom image
    container_name: dht_sensor_mongo
    privileged: true
    environment:
      MONGODB_HOST: 192.168.3.200 #a GCP Compute Engine that is running MongoDB
      MONGODB_PORT: 27017  # Specify the MongoDB port
      MONGO_INITDB_ROOT_USERNAME: yourusername  # Specify the MongoDB root username
      MONGO_INITDB_ROOT_PASSWORD: yourpassword  # Specify the MongoDB root password
      MONGO_DB_NAME: sensor_data  # Specify the MongoDB database name
      MONGO_COLLECTION_NAME: dht_sensor  # Specify the MongoDB collection name
      DHT_SENSOR_TYPE: DHT22  # Set the DHT sensor type here (DHT11 or DHT22)
      DHT_PIN: 4  # Set the DHT sensor pin here      

Checking that MongoDB works

Get an interactive terminal of your MongoDB container and login to the Mongo Shell:

docker exec -it mongodb sh
mongosh --username yourusername --password yourpassword --authenticationDatabase admin

show dbs

# Switch to the our DB
use sensor_data

# List all collections in the "sensor_data" database
show collections

# And to get the latest ones
db.dht_sensor.find().sort({ timestamp: -1 }).limit(10);


db.dht_sensor.findOne();

#db.dht_sensor.explain().find({});

• Job
  • The Python Script
  • MongoDB Cloud Setup
  • Databricks

Databricks

The Unified Analytics Platform - Databricks: https://community.cloud.databricks.com/

#connectionString='mongodb+srv://CONNECTION_STRING_HERE/ #mongodb://username:password@host:port/database
mongodb://yourusername:yourpassword@host:27017/test
mongodb://yourusername:yourpassword@34.27.181.245:27017/sensor_data
database="sample_supplies"
collection="sales"

db.dht_sensor.findOne();
{
  _id: ObjectId("65324697e3578aaa6690a953"),
  timestamp: '2023-10-20T09:21:27Z',
  temperature: 21.700000762939453,
  humidity: 52.599998474121094
}