BI Tools for the AI era

1. Data Connection: Redash allows users to connect to a wide range of data sources, including relational databases (e.g., PostgreSQL, MySQL), NoSQL databases (e.g., MongoDB), data warehouses (e.g., Amazon Redshift, Google BigQuery), REST APIs, and more. This versatility enables data analysts to work with data from multiple sources in a unified platform.

2. Querying: Users can create SQL queries and run them within Redash to extract and transform data. The tool provides a user-friendly query builder interface, making it accessible to users with varying levels of SQL expertise.

3. Visualization: Redash offers a variety of visualization options, including charts, graphs, and pivot tables, to help users explore and understand their data. It supports popular charting libraries like Chart.js, Plotly, and more, allowing for customizable and interactive visualizations.

4. Dashboards: Users can combine multiple visualizations and queries into interactive dashboards. These dashboards can be customized and shared with others, making it easy to communicate insights and monitor key performance indicators (KPIs).

5. Collaboration: Redash supports collaboration among team members by allowing them to share queries, dashboards, and insights. This promotes knowledge sharing and facilitates data-driven decision-making within organizations.

6. Scheduled Reports: Redash enables users to schedule and automate the generation of reports and dashboards, which can be sent via email or other channels. This feature is useful for regularly distributing updated data and insights to stakeholders.

7. Embedding: Redash provides options for embedding visualizations and dashboards into other applications or websites, allowing for seamless integration of data analytics capabilities into existing workflows.

8. Data Governance: Redash provides role-based access control and data source integration, helping organizations maintain data security and governance standards.

Redash is a versatile data analytics tool that simplifies the process of querying, visualizing, and sharing data from various sources.

If you have ever recorded a VBA macro to see the general structure and then tweak it, this is going to be familiar.

When you will find the use case that several similar pannels have to be created: for example for anomaly detection on 100 servers, or 50 different apps you have to monitor…

…you can define the first panel via Grafana UI, see how one panel look and then make Python do the rest for you.

IMPORTANT: you need to keep different panel id and positions across the dashboard

# Main Execution
file_path = "Grafana-Initial.json"  # Replace with your file path

iterate_data = extract_site_info_from_file(file_path)

# if iterate_data:
#     panels = []
#     for site, panel_id, grid_x, grid_y in iterate_data:
#         panel_json = create_elasticsearch_stat_panel_with_filter(site, grid_x, grid_y, panel_id)
#         panels.append(json.dumps(panel_json))

#     output_json = "[" + ",".join(panels) + "]"
#     print(output_json)
# else:
#     print("Failed to extract site information.")

Then, you will take the JSON and carefully add it respecting other dashboard areas you might have (this includes the very end of it, where you have the dashboard uid and so on)

Ever wondered how programmers build software? If builders built houses the way programmers build software, the first woodpecker to come along would destroy civilization.

In this video, Fireship, a popular YouTube creator, takes us on a journey to instrument, collect, and visualize telemetry data from our server using open telemetry and the LGTM stack.

The LGTM stack consists of five technologies:

• Open Telemetry: An industry-standard framework that collects telemetry data from software.
• Grafana: The frontend UI for visualizing data.
• Prometheus: A time-series database for storing metrics.
• Tempo: A database for storing traces.
• Loki: A database for storing logs.

Setting Up the LGTM Stack

To get started:

1. Server Setup: Fireship recommends using Hostinger, which offers a variety of hosting services, including a Linux virtual private server (VPS) suitable for this project.
2. Deploying LGTM Backend: The LGTM backend can be deployed using a Docker image provided by Graphon.
3. Server Access: After setting up your server on Hostinger, you can access the management dashboard and connect via SSH.

Running Grafana

1. Verify Docker: Use a Docker command to confirm the Docker daemon is running.
2. Run Docker Image: Pull the Docker image for OTel LGTM from the cloud, which may take a minute.
3. Access Grafana: Open the browser and navigate to the IP address on port 3000 to log into Grafana as admin.

Connecting to a Data Source

Grafana alone is not useful unless connected to a data source.

Fireship demonstrates how to connect Loki, Prometheus, and Tempo, and then populate them with data using a sample application built with Dino, a framework with built-in telemetry support.

Analyzing Data with Grafana

With the LGTM stack up and running:

• Use Grafana to explore logs, metrics, and traces.
• Use Prometheus’ query language to analyze and dive deeper into your data to identify errors and anomalies.

Takeaways

• The LGTM stack provides a holistic approach to software observability, enabling you to instrument, collect, and visualize telemetry data from your server.
• With the right tools, you can identify errors and anomalies in your application, allowing for data-driven decisions to improve your software.
• Security is crucial when deploying a telemetry backend. Services like Hostinger offer built-in protections against harmful traffic and threats.