Prometheus for Developers: Metric Types, PromQL, and Instrumentation

Prometheus is the de-facto open-source standard for collecting metrics — the cheap, aggregatable numbers that tell you that something is wrong and how much. But most teams adopt it, scrape a few endpoints, and never learn the one thing that decides whether their dashboards are useful: the Prometheus metric types. Pick the wrong type and your latency graph lies to you, your alerts flap, and your storage bill balloons. This guide explains the four metric types, how to instrument a service so it exposes them, and how to query them with PromQL — the pieces you need to turn raw counters into answers.

This is a supporting page in the Devgains observability cluster. Start with the observability pillar — metrics, logs, and traces for how metrics fit alongside the other two signals, then come back here for the metrics deep-dive.

Quick answer: what are the Prometheus metric types?

Prometheus has four core metric types, and each answers a different question:

• Counter — a value that only goes up (resets to zero on restart). How many times has this happened? Requests served, errors, bytes sent.
• Gauge — a value that can go up and down. What is this right now? Memory in use, queue depth, in-flight requests, temperature.
• Histogram — samples observations into configurable buckets and counts them. What is the distribution? Request latency, response size — the source of percentiles.
• Summary — like a histogram but calculates quantiles (e.g. the p99) on the client side over a sliding time window.

The one-line rule: use a counter for things you count, a gauge for things you measure, and a histogram for anything where you care about percentiles. Reach for a summary only when you cannot pre-define buckets and don't need to aggregate across instances.

Why the metric type matters

The type is not a label — it changes what math is legal. A counter is meaningless as a raw value (it might be 4,812,904 because the process has been up for 30 days); you almost always wrap it in rate() to get per-second change. A gauge is meaningful as-is and you take its current value, average, or max. A histogram is the only type you can aggregate into an accurate percentile across many instances, because the buckets add up. A summary's quantiles cannot be averaged across instances — averaging a p99 from three pods gives you a number that means nothing.

Get this wrong and the failure is silent. The graph renders. The alert fires. They're just measuring something other than what you think. That is the most expensive kind of observability bug, and it is exactly the trap the observability pillar warns about.

Architecture: how Prometheus collects metrics

Prometheus uses a pull model. Your service exposes a plain-text /metrics endpoint; the Prometheus server scrapes it on an interval (typically every 15s) and stores each sample in its time-series database, keyed by metric name and labels.

your-service:8080/metrics   ← exposition endpoint (text format)
        ▲  scrape every 15s
        │
  Prometheus server  ──►  TSDB (local disk)  ──►  PromQL query engine
        │                                             ▲
        │                                             │
   service discovery                             Grafana / alerts
   (k8s, static, DNS)

Two consequences fall out of the pull model. First, Prometheus needs to discover your targets — in Kubernetes it does this automatically via the API server, which is one more reason the Kubernetes architecture matters for operability. Second, a target that is down simply isn't scraped, so Prometheus itself becomes a coarse liveness signal — complementary to the app-level liveness and readiness probes your platform already runs.

A minimal scrape configuration looks like this:

# prometheus.yml
global:
  scrape_interval: 15s          # how often to pull /metrics
  evaluation_interval: 15s      # how often to evaluate alert rules
 
scrape_configs:
  - job_name: "checkout-api"
    static_configs:
      - targets: ["checkout-api:8080"]   # host:port exposing /metrics

The job_name becomes a job label on every series from those targets, and targets is the list of host:port pairs Prometheus will scrape. In production you replace static_configs with kubernetes_sd_configs so targets are discovered dynamically as pods come and go.

Step-by-step: instrument a service

Here is a small Python service using the official prometheus_client library. It exposes one metric of each of the three types you will use most.

from prometheus_client import Counter, Gauge, Histogram, start_http_server
import time, random
 
# Counter: total requests, partitioned by route and status code.
REQUESTS = Counter(
    "http_requests_total", "Total HTTP requests",
    ["route", "status"],
)
 
# Gauge: how many requests are being handled right now.
IN_FLIGHT = Gauge("http_requests_in_flight", "In-flight requests")
 
# Histogram: request duration, with explicit latency buckets in seconds.
LATENCY = Histogram(
    "http_request_duration_seconds", "Request duration",
    buckets=[0.05, 0.1, 0.25, 0.5, 1, 2.5, 5],
)
 
def handle_request(route: str):
    IN_FLIGHT.inc()
    start = time.perf_counter()
    try:
        time.sleep(random.random())          # simulate work
        REQUESTS.labels(route=route, status="200").inc()
    finally:
        LATENCY.observe(time.perf_counter() - start)
        IN_FLIGHT.dec()
 
if __name__ == "__main__":
    start_http_server(8080)                  # serves /metrics on :8080
    while True:
        handle_request("/checkout")

What each line does: Counter.inc() increments the count for a specific route/status combination — every distinct combination is its own time series, which is why labels must have bounded cardinality. Gauge.inc()/dec() track a live value that rises and falls. Histogram.observe() drops each duration into the first bucket it fits, so Prometheus can later reconstruct percentiles from the bucket counts. start_http_server(8080) exposes the standard /metrics text endpoint the scrape config above points at.

Query it with PromQL

PromQL is where metrics become answers. A few queries you will reach for constantly:

# Requests per second over the last 5 minutes, per route.
sum(rate(http_requests_total[5m])) by (route)
 
# Error ratio: 5xx responses as a fraction of all responses.
sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m]))
 
# p95 latency, aggregated correctly across all instances.
histogram_quantile(
  0.95,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
)

The pattern to internalize: counters go through rate() to become per-second rates before you sum them, and histograms are turned into percentiles with histogram_quantile() over the _bucket series grouped by the le ("less than or equal") label. The by (route) clause keeps the per-route breakdown; drop it to get a single global number.

Prometheus metric types compared

The practical takeaway: prefer histograms over summaries for latency. Histograms let you compute any percentile at query time and aggregate accurately across pods; summaries lock you into quantiles chosen at instrumentation time and cannot be combined across instances.

Best practices

• Name metrics with a unit suffix. _seconds, _bytes, _total for counters. Consistent naming makes dashboards self-documenting and PromQL predictable.
• Keep label cardinality bounded. Never put a user ID, request ID, or raw URL in a label — each unique value is a new time series and cardinality explosions are the number-one way to melt a Prometheus server.
• Use the four golden signals — latency, traffic, errors, saturation — as the backbone of every service dashboard.
• Alert on symptoms, not causes. Alert on error rate and latency (what users feel), not on CPU being high (which may be perfectly fine).
• Expose /metrics behind the same probes your platform already scrapes, and treat a missing scrape as a signal in its own right.

Common mistakes

• Using a gauge for something that only increases. You lose rate() semantics and restart detection. If it only goes up, it's a counter.
• Averaging percentiles. avg of a p99 across three instances is not the p99 of the fleet. Use histograms and histogram_quantile over summed buckets instead.
• Reading a counter's raw value. A counter of 4.8M tells you nothing without rate() or increase(). It's the change that's meaningful.
• Unbounded labels. A path label containing /user/12345/orders/98765 creates a series per request. Normalize it to /user/:id/orders/:id first.
• Scraping too aggressively. A 1s scrape interval multiplies storage and load for data you rarely query at that resolution. 15s is a sane default.

Key takeaways

• Prometheus has four metric types: counter (up only), gauge (up/down), histogram (bucketed distributions), and summary (client-side quantiles).
• Counters need rate(); gauges are read as-is; histograms produce aggregatable percentiles via histogram_quantile().
• Instrument by exposing a /metrics endpoint and let Prometheus pull it on an interval.
• The biggest operational risks are cardinality explosions and averaging percentiles — both silent, both avoidable.
• Prefer histograms over summaries for latency so you can aggregate across instances.

FAQ

What are the four Prometheus metric types? Counter, gauge, histogram, and summary. A counter only increases, a gauge moves up and down, a histogram buckets observations to compute distributions, and a summary calculates quantiles on the client side.

What is the difference between a counter and a gauge in Prometheus? A counter can only go up (and reset to zero on process restart), so you always wrap it in rate() to get a per-second rate. A gauge can go up or down and is meaningful as its current value — in-flight requests, memory usage, queue depth.

Should I use a histogram or a summary in Prometheus? Prefer a histogram. Histograms let you compute any percentile at query time with histogram_quantile() and aggregate accurately across many instances. Summaries compute quantiles per-instance and cannot be averaged across pods, which makes fleet-wide percentiles impossible.

How does Prometheus collect metrics? It uses a pull model: your service exposes a text /metrics endpoint and the Prometheus server scrapes it on an interval (commonly every 15 seconds), storing each sample in its local time-series database keyed by metric name and labels.

What is PromQL used for? PromQL is Prometheus's query language. You use it to turn raw metrics into answers — request rates with rate(), error ratios, and latency percentiles with histogram_quantile() — for dashboards in Grafana and for alerting rules.

What causes high cardinality in Prometheus? Putting high-variety values (user IDs, request IDs, raw URLs, emails) into labels. Each unique label combination is a separate time series, so unbounded labels can create millions of series and exhaust memory. Keep labels to bounded, low-variety dimensions like route and status code.

Conclusion

Metrics are the cheapest, most scalable of the three observability signals, and Prometheus is how most teams collect them — but only if you pick the right metric type for each measurement. Count with counters, measure with gauges, and reach for histograms whenever percentiles matter. Instrument a /metrics endpoint, let Prometheus pull it, and query it with PromQL, and you'll have dashboards that answer real questions instead of rendering plausible-looking lies.

From here, wire these metrics into your deployment pipeline — see deploying Docker containers to production and the broader observability and DevOps clusters — and pair them with the database-level signals from connection pooling, where a saturated pool is one of the first things a good gauge will catch.

References: Prometheus documentation — metric types, Prometheus — querying basics (PromQL), OpenTelemetry metrics data model, Google SRE book — monitoring distributed systems, and the CNCF project landscape, of which Prometheus is a graduated project.