BI Tools for the AI era

1. Data Connection: Redash allows users to connect to a wide range of data sources, including relational databases (e.g., PostgreSQL, MySQL), NoSQL databases (e.g., MongoDB), data warehouses (e.g., Amazon Redshift, Google BigQuery), REST APIs, and more. This versatility enables data analysts to work with data from multiple sources in a unified platform.

2. Querying: Users can create SQL queries and run them within Redash to extract and transform data. The tool provides a user-friendly query builder interface, making it accessible to users with varying levels of SQL expertise.

3. Visualization: Redash offers a variety of visualization options, including charts, graphs, and pivot tables, to help users explore and understand their data. It supports popular charting libraries like Chart.js, Plotly, and more, allowing for customizable and interactive visualizations.

4. Dashboards: Users can combine multiple visualizations and queries into interactive dashboards. These dashboards can be customized and shared with others, making it easy to communicate insights and monitor key performance indicators (KPIs).

5. Collaboration: Redash supports collaboration among team members by allowing them to share queries, dashboards, and insights. This promotes knowledge sharing and facilitates data-driven decision-making within organizations.

6. Scheduled Reports: Redash enables users to schedule and automate the generation of reports and dashboards, which can be sent via email or other channels. This feature is useful for regularly distributing updated data and insights to stakeholders.

7. Embedding: Redash provides options for embedding visualizations and dashboards into other applications or websites, allowing for seamless integration of data analytics capabilities into existing workflows.

8. Data Governance: Redash provides role-based access control and data source integration, helping organizations maintain data security and governance standards.

Redash is a versatile data analytics tool that simplifies the process of querying, visualizing, and sharing data from various sources.

If you have ever recorded a VBA macro to see the general structure and then tweak it, this is going to be familiar.

When you will find the use case that several similar pannels have to be created: for example for anomaly detection on 100 servers, or 50 different apps you have to monitor…

…you can define the first panel via Grafana UI, see how one panel look and then make Python do the rest for you.

IMPORTANT: you need to keep different panel id and positions across the dashboard

# Main Execution
file_path = "Grafana-Initial.json"  # Replace with your file path

iterate_data = extract_site_info_from_file(file_path)

# if iterate_data:
#     panels = []
#     for site, panel_id, grid_x, grid_y in iterate_data:
#         panel_json = create_elasticsearch_stat_panel_with_filter(site, grid_x, grid_y, panel_id)
#         panels.append(json.dumps(panel_json))

#     output_json = "[" + ",".join(panels) + "]"
#     print(output_json)
# else:
#     print("Failed to extract site information.")

Then, you will take the JSON and carefully add it respecting other dashboard areas you might have (this includes the very end of it, where you have the dashboard uid and so on)

Ever wondered how programmers build software? If builders built houses the way programmers build software, the first woodpecker to come along would destroy civilization.

In this video, Fireship, a popular YouTube creator, takes us on a journey to instrument, collect, and visualize telemetry data from our server using open telemetry and the LGTM stack.

The LGTM stack consists of five technologies:

• Open Telemetry: An industry-standard framework that collects telemetry data from software.
• Grafana: The frontend UI for visualizing data.
• Prometheus: A time-series database for storing metrics.
• Tempo: A database for storing traces.
• Loki: A database for storing logs.

Setting Up the LGTM Stack

To get started:

1. Server Setup: Fireship recommends using Hostinger, which offers a variety of hosting services, including a Linux virtual private server (VPS) suitable for this project.
2. Deploying LGTM Backend: The LGTM backend can be deployed using a Docker image provided by Graphon.
3. Server Access: After setting up your server on Hostinger, you can access the management dashboard and connect via SSH.

Running Grafana

1. Verify Docker: Use a Docker command to confirm the Docker daemon is running.
2. Run Docker Image: Pull the Docker image for OTel LGTM from the cloud, which may take a minute.
3. Access Grafana: Open the browser and navigate to the IP address on port 3000 to log into Grafana as admin.

Connecting to a Data Source

Grafana alone is not useful unless connected to a data source.

Fireship demonstrates how to connect Loki, Prometheus, and Tempo, and then populate them with data using a sample application built with Dino, a framework with built-in telemetry support.

Analyzing Data with Grafana

With the LGTM stack up and running:

• Use Grafana to explore logs, metrics, and traces.
• Use Prometheus’ query language to analyze and dive deeper into your data to identify errors and anomalies.

Takeaways

• The LGTM stack provides a holistic approach to software observability, enabling you to instrument, collect, and visualize telemetry data from your server.
• With the right tools, you can identify errors and anomalies in your application, allowing for data-driven decisions to improve your software.
• Security is crucial when deploying a telemetry backend. Services like Hostinger offer built-in protections against harmful traffic and threats.

Understanding Kibana Query Languages

• Lucene Query Syntax:

  • Lucene is a powerful full-text search engine library. Elasticsearch, which Kibana uses, is built on top of Lucene.
  • Lucene’s query syntax is quite low-level and can be complex, involving boolean operators, field names, wildcards, and more.
  • Directly writing Lucene queries can be cumbersome for many users.

• Kibana Query Language (KQL):

  • KQL is a user-friendly query language designed specifically for Kibana. It simplifies the process of searching and filtering data in Elasticsearch.
  • It provides a more intuitive syntax compared to raw Lucene, making it easier for users to construct queries.
  • KQL is ultimately translated into Lucene queries behind the scenes.
  • KQL is designed to be very human readable.

• Elasticsearch Domain-Specific Language (DSL):

  • Elasticsearch DSL is a JSON-based query language that provides a more flexible and powerful way to interact with Elasticsearch.
  • It allows for complex queries, aggregations, and other operations that might be difficult or impossible to express using KQL.
  • DSL is the raw Json that is sent to elasticsearch.
  • While KQL is converted to a Lucene query, that lucene query is then wrapped in a DSL Json object.

Relationship to Lucene Queries

• Both KQL and DSL are ultimately translated into Lucene queries.
• KQL acts as a higher-level abstraction, simplifying the process for users.
• DSL provides the most control, as it is the direct Json that is sent to elasticsearch.

Translating Your KQL Query to DSL

Your KQL query is:

viewerID : * and site : "AD04" and (HttpPlayerPlaybackEndEvent_assetType : * or HttpPlayerStartEvent_assetType : * )

Here’s how to translate it into Elasticsearch DSL:

{
  "query": {
    "bool": {
      "must": [
        {
          "wildcard": {
            "viewerID": {
              "value": "*"
            }
          }
        },
        {
          "term": {
            "site.keyword": {
              "value": "AD04"
            }
          }
        },
        {
          "bool": {
            "should": [
              {
                "wildcard": {
                  "HttpPlayerPlaybackEndEvent_assetType": {
                    "value": "*"
                  }
                }
              },
              {
                "wildcard": {
                  "HttpPlayerStartEvent_assetType": {
                    "value": "*"
                  }
                }
              }
            ]
          }
        }
      ]
    }
  }
}

Explanation of the DSL:

• "query": {"bool": {"must": [...]}}: This defines a boolean query, where all must clauses must match.
• "wildcard": {"viewerID": {"value": "*"}}: This matches any value in the viewerID field.
• "term": {"site.keyword": {"value": "AD04"}}: This matches the exact term “AD04” in the site.keyword field. Note that if the site field is not set as a keyword, then the term query would be “site” instead of “site.keyword”.
• "bool": {"should": [...]}: This defines a boolean query, where at least one should clause must match.
• "wildcard": {"HttpPlayerPlaybackEndEvent_assetType": {"value": "*"}}: This matches any value in the HttpPlayerPlaybackEndEvent_assetType field.
• "wildcard": {"HttpPlayerStartEvent_assetType": {"value": "*"}}: This matches any value in the HttpPlayerStartEvent_assetType field.

Key Points:

• When dealing with exact string matches in Elasticsearch, it’s often best to use the .keyword sub-field if it exists. This ensures that the entire string is matched as a single term.
• Wildcard queries can be expensive, so try to be as specific as possible.
• DSL provides much more control, and is required for more complex queries.

Imagine you’re planning a big trip with your friends.

Before you pack your bags and hit the road, you’d probably want to agree on a few key things, right?

A project charter is like that agreement for a project at work or school.

In simple terms, it’s a short, important document that clearly says:

• What exactly are we trying to achieve with this project? (The goal)
• Why are we doing this project? (The reason and benefits)
• Who are the main people involved? (The key players)
• What are the main things we need to do? (The high-level plan)
• What are the big limits or rules we need to follow? (Like budget or deadlines)
• Who has the authority to say “yes, let’s do this”? (The person who approves the project)

Think of it as the project’s “birth certificate” and “permission slip.” It makes sure everyone is on the same page from the very beginning and knows what the project is all about before a lot of time and effort is spent.

1. Project purpose: Why is this project important, and what business need does it fulfill?
2. Project objectives: What do you want to achieve by the end of your project, and what metrics will you use to measure project success?
3. Project scope and deliverables: What specific deliverables will you complete, and what will you NOT work on?
4. Required resources or project needs: What do you need to complete this work? This includes people, tools, and budget.
5. Project stakeholders: Who are your project team members? Who needs to approve the project charter, who’s the project sponsor, and who should be looped into key decisions?