Metrics Monitoring

System Design · HLD Series

Design a Metrics Monitoring & Alerting System (Prometheus-Style)

A pull-based observability backbone that scrapes instrumented targets on an interval, stores dense time-series data efficiently, evaluates alerting rules continuously, and routes incidents to the right on-call human — without ever depending on the very systems it is watching.

Jump to HLD Deep architecture Sequence flows Data model

20MActive time series

15sScrape interval

1.3M/sSamples ingested

1. Requirements scope 2. Capacity numbers 3. High-level design boxes 4. Deep architecture scale 5. Sequence flows scrape & alert 6. Data model control plane 7. Deep dives interview probes 8. Summary recap

1. Requirements

Before drawing boxes, pin down what "monitoring system" actually has to guarantee — including the uncomfortable one: it must keep working precisely when everything else is on fire.

Functional

Pull-based scrapingPeriodically fetch metric samples from instrumented targets over HTTP on a fixed interval.

Time-series storagePersist labeled samples efficiently, with configurable downsampling and retention over time.

Alert evaluationContinuously evaluate rule expressions and fire notifications when conditions hold for a defined duration.

Query & dashboardsSupport ad-hoc and dashboard querying via a purpose-built query language over the stored series.

Non-functional

Independent availabilityMust stay up and keep collecting even when the systems it monitors are actively failing — no shared fate.

Cardinality resilienceHandle very high dimensionality (millions of label combinations) without degrading ingest or query.

Low detection latencyAn incident should surface within roughly one scrape interval, not minutes later.

Cost-bounded storageCompress and downsample aging data so multi-year retention doesn't require multi-petabyte disks.

Explicitly out of scope Log aggregation and full-text search (that's a separate log-indexing system, e.g. an ELK/Loki-style design) and distributed request tracing (a separate tracing/span-collection system). This post is purely metrics: numeric time series, alerting on them, and dashboards over them.

2. Capacity Estimation

Rough numbers for a mid-size organization running its own metrics fleet across several hundred services.

Dimension	Assumption	Resulting load
Scrape fan-out	50,000 targets, 15s scrape interval	50,000 / 15 ≈ 3,333 scrapes/sec across the fleet
Active cardinality	20M active time series, ≈2 bytes/sample (compressed chunks)	20M / 15s ≈ 1.33M samples/sec ingested ≈ 230 GB/day raw
Alert rules	8,000 rules, 30s evaluation cycle	8,000 / 30 ≈ 267 rule evaluations/sec
Retention	15 days raw @ 15s resolution, 13 months downsampled @ 5m resolution	≈3.4 TB raw + ≈0.9 TB downsampled ≈ 4.3 TB total (vs ≈80 TB if raw kept for 13 months)

Why this matters for the design The gap between "80 TB if naive" and "4.3 TB with downsampling" is not an optimization — it's the reason a long-term-storage tier with compaction (Thanos/Cortex/Mimir style) exists at all. Storage strategy is a first-class requirement, not an afterthought.

3. High-Level Design

One Prometheus server per cluster/team pulls metrics, evaluates rules locally, and hands firing alerts to a separate routing tier. Dashboards read through a query layer that can span multiple servers.

scrape path rule evaluation alert & notify long-term storage query/read path

What each box owns

Prometheus server

Owns the scrape scheduler (who to poll, on what interval), the parser for the exposition text format, and a local append-only TSDB with a few hours to a couple weeks of retention. It is entirely self-sufficient for scraping and short-term querying.

Rule evaluator

Runs inside the same server process. On its own interval it re-executes each configured PromQL alerting/recording rule against local data and tracks per-rule state (inactive → pending → firing).

Alertmanager

A deliberately separate service. Receives firing alerts from one or many Prometheus servers, groups related alerts, deduplicates and silences, then routes to the correct notification channel based on labels.

remote_write & Grafana

remote_write streams samples out to a long-term-storage tier for multi-month/year retention and cross-server queries; Grafana (or the native UI) is the read path that issues PromQL against either the local server or the global store.

4. Deep Architecture & Scale-Out

A single Prometheus server is intentionally simple — single binary, local disk. Scale comes from running many of them (sharded by team/cluster) plus a federation layer that stitches them together for global queries and long retention.

Decision	Choice	Why
Metric transport model	Pull (server scrapes target)	Server controls its own load, sees target liveness for free via scrape failure/up, and doesn't need every app to know where to ship data.
High availability	Replicated Prometheus pairs + dedup at the global layer	A single instance is a SPOF; running two identical scrapers and deduping downstream is simpler than making one instance itself distributed.
Long-term storage	remote_write into object storage, compacted/downsampled by Thanos/Cortex/Mimir	Local disk on each server stays small and fast; cold data moves to cheap, durable, infinitely scalable object storage.
Cardinality control	Label allow-lists, relabeling at scrape time, per-tenant series limits	Unbounded label values (user IDs, request IDs) can blow up series count 100× overnight; the guard has to sit at ingestion, not query time.

5. Sequence Flows

Two flows carry the whole design: how a sample gets from a running process into storage, and how a stored sample turns into a page to an on-call engineer.

Flow 1 — Pull-based scrape cycle

up{job,instance}=1 for this target 7. on timeout/error instead: up=0, scrape recorded as failed

The scrape itself is the health check. There's no separate heartbeat mechanism: a target that stops responding to /metrics automatically flips its own up series to 0, which downstream alert rules can key off directly (e.g. up == 0 for 5 minutes → page).

Flow 2 — Alert rule evaluation and notification

The "for" duration is the design's built-in debounce: a threshold crossed for one evaluation doesn't fire immediately, which absorbs transient blips. Alertmanager then applies a second layer of debounce — grouping and silencing — so ten pods breaching the same rule become one notification, not ten pages.

6. Data Model

This isn't a user-facing app schema — it's the monitoring system's own control-plane and storage state: what series exist, what rules are defined, and what targets should be scraped.

Key modeling decisions

Chunked append-only series, not row-per-sampleA "series" is a labeled key plus a compressed, append-only stream of (timestamp, value) chunks — a relational row-per-sample table would be crushed by 1.3M inserts/sec.

Rule state stored separately from rule definitionpending_since and current state are mutable per-evaluation-cycle data; keeping them off the immutable rule definition avoids rewriting config on every tick.

scrape_targets is dynamic, not a static listTargets come from service discovery (Kubernetes, Consul, cloud APIs) and churn constantly with deploys/autoscaling — the table is a live materialized view, not hand-maintained config.

Labels are the cardinality risk surfaceEvery unique label combination is a new series; an unbounded-cardinality label (user_id, request_id) silently multiplies storage and query cost by orders of magnitude.

Approach	Fit for this workload
Purpose-built TSDB (chunked, columnar, compressed)	Optimized for high-throughput sequential appends and time-range scans; achieves ≈1–2 bytes/sample via delta-of-delta and XOR encoding.
General relational database	Row-per-sample storage and B-tree indexes can't sustain millions of inserts/sec at this compression ratio; would need constant vertical scaling and still cost far more per byte stored.

7. Deep Dives & Interview Probes

Why does a label like user_id cause a cardinality explosion?

Every distinct combination of metric name + label values is a separate time series. A counter with 5 stable labels might have a few thousand combinations; add a label with a million possible values (user IDs, request IDs, raw URLs) and the series count multiplies by that factor. Each series carries its own index entry and chunk overhead, so ingestion, memory, and query fan-out all degrade roughly linearly with series count — and cardinality can grow far faster than traffic does.

• Guard at the instrumentation layer: never label with unbounded identifiers, bucket them instead (e.g. a histogram, or a status/class dimension).
• Guard at ingestion: per-tenant series limits and relabeling rules that drop or hash risky labels before they're stored.
• Guard operationally: track scrape_series_added and cardinality-by-job as first-class metrics about the metrics system itself.

Pull vs push — when do you actually need a gateway?

Pull wins for long-lived services: the server controls scrape rate, gets liveness for free, and a misbehaving target can't flood the collector. It breaks down for short-lived batch/cron jobs that may finish before any scrape interval elapses — there's nothing running to be pulled from. The standard fix is a push gateway: the job pushes its final metrics to a small always-on intermediary, and Prometheus pulls from that gateway instead of the job directly. It's used narrowly, only for this class of workload, precisely to keep the pull model as the default everywhere else.

How do you downsample for long retention without losing incident-investigation detail?

Keep raw 15s-resolution data for a short, expensive window (days to a couple weeks) where investigators actually need fine-grained detail. Beyond that, compact into progressively coarser resolutions — e.g. 5-minute and 1-hour rollups storing min/max/sum/count per bucket — for the multi-month/year tier. Storing min/max alongside averages matters: it preserves spike visibility in the aggregate that a plain average would smooth away, so a downsampled graph still shows "there was a spike here" even if it can't show the exact millisecond.

How is alert flapping and notification-storm risk controlled?

Two layers work together. The for duration on a rule means a value has to stay past threshold for a sustained window before it ever becomes "firing" — this alone kills most single-sample noise. Downstream, Alertmanager groups alerts sharing labels into one notification, deduplicates identical alerts arriving from HA replica pairs, and honors silences during known maintenance — so a cluster-wide event that trips the same rule on 200 pods produces one grouped page, not 200.

- alert: HighErrorRate
  expr: |
    sum(rate(http_requests_total{status=~"5.."}[5m]))
    /
    sum(rate(http_requests_total[5m])) > 0.05
  for: 10m
  labels:
    severity: page
  annotations:
    summary: "Error rate above 5% for 10m on {{ $labels.job }}"

Who monitors the monitor? (avoiding a single point of observability failure)

The monitoring system must never rely on the infrastructure it watches to report its own health — if it did, an outage could silently blind you at the exact moment you need visibility most. The mitigations: run monitoring infrastructure on isolated compute/network paths from production; run HA replica pairs of each Prometheus server so one instance dying doesn't blank a shard; run a small separate "meta-monitoring" instance that scrapes the monitoring stack itself (its own up, scrape failures, rule evaluation latency); and page on dead-man's-switch style alerts — a rule that's expected to always fire on a heartbeat, where silence itself is the signal something upstream broke.

8. Summary

A metrics monitoring system is really three cooperating concerns: a pull-based collection loop that doubles as a liveness check, a storage engine built around cardinality and compression rather than row counts, and an alerting pipeline that debounces at two independent layers before anything reaches a human. The scale-out story — sharded servers, HA pairs, a global downsampling tier — exists to keep every one of those guarantees true under load, not just at demo scale.

Prometheus PromQL Alertmanager Thanos / Cortex / Mimir Cardinality management Time-series storage

Interview tip The strongest signal you can give in this design is raising cardinality control as a first-class constraint up front — in the requirements, in the data model, in the architecture decisions — rather than mentioning it as an afterthought if the interviewer probes. Cardinality explosion is the specific, well-known way real metrics systems fall over in production, and treating it that way from the start reads as hands-on experience rather than textbook recall.