Skip to main content
All projects
Infrastructure & DevOps
/projects/observability-platform

Observability & Monitoring Platform

project.observability-platform.local
Observability & Monitoring Platform project screenshot

  1. Problem:

    Incidents took hours to diagnose due to fragmented monitoring.

  2. Solution:

    Implemented service-level observability with Prometheus and Grafana, unified SLI tracking.

  3. Impact:

    MTTD reduced by 60%, on-call burden decreased significantly.

Overview

Service-level golden signals and dashboards that match how engineers actually triage—not vanity charts.

Results & metrics

  • 60% reduction in Mean Time To Detect (MTTD) across production incidents
  • Unified Golden Signal telemetry (latency, traffic, errors, saturation) across all microservices
  • Eliminated alert fatigue by triaging alerts into Teams/Slack vs Email based on severity
  • Self-managed Prometheus & Grafana architecture saving significant monitoring costs

Before / after

  • Incident Detection Time

    Before: Hours of tribal debugging across disparate logs to locate failures

    After: Under 10 minutes (60% faster MTTD) with real-time SLI dashboards

  • Alert Quality

    Before: Noisy spam alerts routing everything to noisy developer channels

    After: Contextual severity-based routing: informational to Slack, critical to Pager/Email

  • System Health Visibility

    Before: Fragmented infrastructure metrics with no service-level visibility

    After: Unified single pane of glass showing golden signals for each production service

What I built

Scrape, store, visualize

Prometheus scrapes the services; Grafana shows latency, traffic, errors and saturation.

Alerts by severity

Alertmanager sends standard alerts to Teams/Slack and critical ones to Email.

Faster detection

Service-level observability cut incident detection time (MTTD) by 60%.

Stack

  • Prometheus, Time-Series Metrics
  • Grafana, Telemetry Dashboards
  • Alertmanager, Severity Routing
  • CloudWatch Logs, Log Correlation
  • Node & Blackbox Exporters

Decisions

Key trade-offs and design calls that shaped the final delivery.

  • Prometheus + Grafana over CloudWatch alone

    Why
    Context
    CloudWatch dashboards were limited and expensive for custom metrics
    Decision
    Self-managed observability stack with better visualization and lower cost for high-cardinality metrics

  • Tiered alerting

    Why
    Context
    Alert fatigue from non-critical notifications going to same channel as incidents
    Decision
    Route standard alerts to Teams/Slack, critical alerts to Email—context-aware routing reduces noise

Architecture

The primary system boundaries, runtime pieces, and how the project was structured in production.

Diagram, 9 connections:
  1. Production Microservices → Node & Blackbox Exporters
  2. Node & Blackbox Exporters → Prometheus Scrape Server
  3. Prometheus Scrape Server → Time-Series Storage: Retention
  4. Time-Series Storage: Retention → SLO & Error Budget Rules
  5. SLO & Error Budget Rules → Alertmanager Engine
  6. Alertmanager Engine → Slack / Teams Channel: Standard (Warning / P3)
  7. Alertmanager Engine → Email & Pager: Critical P1 (Breach / P1)
  8. Time-Series Storage: Retention → Grafana Golden Signal Dashboards
  9. CloudWatch Application Logs → Grafana Golden Signal Dashboards (Log Correlation)
Groups: Workloads & Infrastructure: Production Microservices, Node & Blackbox Exporters, CloudWatch Application Logs; Prometheus TSDB Engine: Prometheus Scrape Server, Time-Series Storage: Retention, SLO & Error Budget Rules; Triage & Escalation: Alertmanager Engine, Slack / Teams Channel: Standard, Email & Pager: Critical P1; Single Pane of Glass: Grafana Golden Signal Dashboards.
  • Prometheus

    Metrics Collection

    How

    Scrapes service metrics, golden signals, and deployment health indicators from applications and infrastructure.

  • Grafana Dashboards

    Visualization

    How

    Service-level views tied to SLOs—latency, traffic, errors, saturation—for faster triage and diagnosis.

  • CloudWatch Logs

    Log Aggregation

    How

    Centralized logging for AWS services and application logs—correlates with metrics for root cause analysis.

  • Alertmanager

    Alert Routing

    How

    Multi-channel alerts: Teams/Slack for standard alerts, Email for critical—routes based on severity with context.

Mermaid source

Paste into mermaid.live to edit the diagram.

flowchart TB
  subgraph Targets [Workloads & Infrastructure]
    APP[Production Microservices]
    EXP[Node & Blackbox Exporters]
    CW[CloudWatch Application Logs]
  end
  subgraph Metrics [Prometheus TSDB Engine]
    PROM[Prometheus Scrape Server]
    TSDB[(Time-Series Storage: Retention)]
    RULES[SLO & Error Budget Rules]
  end
  subgraph Alerting [Triage & Escalation]
    AM[Alertmanager Engine]
    SLACK[Slack / Teams Channel: Standard]
    PAGE[Email & Pager: Critical P1]
  end
  subgraph Viz [Single Pane of Glass]
    GRAF[Grafana Golden Signal Dashboards]
  end
  APP --> EXP
  EXP --> PROM
  PROM --> TSDB
  TSDB --> RULES
  RULES --> AM
  AM -->|Warning / P3| SLACK
  AM -->|Breach / P1| PAGE
  TSDB --> GRAF
  CW -.->|Log Correlation| GRAF

Pipeline

How changes moved from development through validation and deployment.

Select a step for its detail

Incidents

Operational failures, rehearsals, or recovery moments that changed how the system was run.

  • P3

    Prometheus scrape targets flapping

    Resolution

    Increased scrape timeout and implemented service discovery health checks

    Lesson: Default timeouts don't work for all services—tune scrape intervals per target

  • P2

    Grafana dashboard permissions too broad

    Resolution

    Implemented role-based access control with team-specific folder permissions

    Lesson: Dashboard access should follow least privilege—not everyone needs edit access

  • P2

    Alertmanager routing misconfiguration

    Resolution

    Fixed routing tree configuration to ensure critical alerts reach email while standard alerts go to Slack

    Lesson: Test alert routing with simulated alerts—misconfigurations silence critical notifications