SyntaxPilot
Courses/Go Programming/Lesson 70
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Go Programming · Lesson 70 of 89

Observability: logs, metrics and traces

three pillars of observabilitystructured request logging with slogcontext-aware loggingmetrics types (counter, gauge, histogram, summary)Prometheus client_golang/metrics endpointRED and USE methodsOpenTelemetrytraces and spanstrace context propagation (W3C traceparent)health checks (liveness, readiness)expvar

These are the notes for Go episode 70. Raju's order service now runs in production, and at 2 A.M. customers say "the app is slow, some orders fail". The servers are running, but nobody can say which requests fail, how many, or which of the three services is slow. It's like driving a car with no dashboard. In this episode we invent the dashboard, piece by piece: logs that machines can search, with a request id in every line; metrics (counters, gauges, histograms) that Prometheus collects from /metrics; the RED and USE methods that say what to measure; traces made of spans that follow one request across services with the traceparent header; and health checks that tell "alive" apart from "ready". Every program below ran for real.

All outputs are from real runs (Go 1.24.7, linux/amd64) with github.com/prometheus/client_golang v1.23.2 and go.opentelemetry.io/otel/sdk v1.39.0 (which brings go.opentelemetry.io/otel and go.opentelemetry.io/otel/trace v1.39.0). In this sandbox the modules come from a local module proxy. Trace ids, span ids, timestamps and timings change on every run, so your ids and milliseconds will differ from the ones printed here.

Facts and sources

From the episode's research, the Go documentation of the installed toolchain (go doc, Go 1.24.7), and the source and READMEs of the exact module versions used (read from the module cache in this run):

  • $(go env GOROOT)/api/go1.21.txt lists pkg log/slog: slog arrived in Go 1.21. expvar is listed in api/go1.txt (it has been in the standard library since Go 1).
  • go doc log/slog: "Package slog provides structured logging, in which log records include a message, a severity level, and various other attributes expressed as key-value pairs." and "In addition to Logger.Info, there are methods for Debug, Warn and Error levels."
  • go doc log/slog.Logger.With: "With returns a Logger that includes the given attributes in each output operation."
  • go doc expvar: "Package expvar provides a standardized interface to public variables, such as operation counters in servers. It exposes these variables via HTTP at /debug/vars in JSON format. As of Go 1.22, the /debug/vars request must use GET." It also registers cmdline (os.Args) and memstats (runtime.Memstats).
  • client_golang README: "the Go client library for Prometheus. It has two separate parts, one for instrumenting application code, and one for creating clients that talk to the Prometheus HTTP API".
  • client_golang v1.23.2 source comments: a Counter "represents a single numerical value that only ever goes up"; a Gauge "represents a single numerical value that can arbitrarily go up and down"; a Histogram "counts individual observations ... in configurable static buckets" and "quantiles can be calculated from a Histogram using the histogram_quantile PromQL function"; for a Summary, "the rank estimations cannot be aggregated in a meaningful way with the Prometheus query language (i.e. you cannot average or add them). If you need aggregatable quantiles (e.g. you want the 99th percentile latency of all queries served across all instances of a service), consider the Histogram metric type." Default histogram buckets: DefBuckets = []float64{.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10} (seconds).
  • opentelemetry-go README: "the Go implementation of OpenTelemetry. It provides a set of APIs to directly measure performance and behavior of your software and send this data to observability platforms". The v1.39.0 README's status table lists Traces: Stable, Metrics: Stable, Logs: Beta, and its exporters table lists OTLP (logs, metrics, traces), Prometheus (metrics), stdout (all three) and Zipkin (traces).
  • otel propagation/trace_context.go: "TraceContext is a propagator that supports the W3C Trace Context format" and it propagates "the traceparent and tracestate headers".
  • otel SDK trace/sampling.go: "TraceIDRatioBased samples a given fraction of traces."
  • W3C Trace Context: traceparent = version-trace-id-parent-id-trace-flags; trace-id is 16 bytes (32 hex characters), parent-id is 8 bytes (16 hex characters), version 00, flag 01 = sampled.
  • Everything else is from the real runs below. Where the notes mention a tool by name that the video names only as an example (Jaeger, Grafana Tempo, Kubernetes, the rate() query), it is marked "(named in the video)".

The problem: 2 A.M. and nobody can see inside

Real-life example: a car with no dashboard. No speedometer, no fuel gauge, no warning lights. You only find out something is wrong when the car stops.

Raju's team at 2 A.M.:

  • customers: "the app is slow, orders are failing";
  • the servers are up, CPU looks normal from outside;
  • but which requests fail? How many? Which of the three services (order, database, kitchen) is slow? Nobody can tell.

Observability is the dashboard: being able to understand what is happening inside a running system from the signals it gives out, without stopping it or adding a fmt.Println and redeploying. (Monitoring is watching those signals and raising an alarm when one crosses a limit.)

The three pillars of observability

There are three main kinds of signal, called the three pillars:

Pillar Question it answers Example from this episode
Logs what happened, to which request? (one event) payment failed for req-102: upi timeout
Metrics how many, how fast? (numbers over time) 1 of 20 requests failed; 2 took over 100 ms
Traces where did the time go? (one request across services) 120 of 135 ms were spent in the kitchen service

They work together: metrics raise the alarm ("errors went up"), a trace finds the slow or failing part ("the kitchen"), and logs explain why ("upi timeout after 2s").

Logs: a sentence vs key=value fields

Real-life example: a diary. "Today asha's order 81 failed after a long wait" is easy for a person to read, but try finding every failure for asha longer than 2 seconds in a million diary pages.

  • Unstructured logging writes sentences for humans (log.Printf("order %d failed for user %s ...")). Searching means grep and hoping nobody ever changed the wording.
  • Structured logging writes each event as named fields (key and value), often one JSON object per line. Log tools can then search user = "asha" AND ms > 2000 like a database.

Go has structured logging built in since Go 1.21: the log/slog package ("S-log"). A Logger has methods Debug, Info, Warn, Error; each takes a message and then key, value, key, value, ... pairs. A Handler decides the output format: TextHandler writes key=value, JSONHandler writes JSON.

Every program in these notes uses one of these go.mod files:

module shop

go 1.24

(standard library only), or the same with require github.com/prometheus/client_golang v1.23.2 (Prometheus programs), or require go.opentelemetry.io/otel/sdk v1.39.0 (tracing programs). The mini project needs both.

Printf vs slog, side by side

main.go (standard library only):

package main

import (
    "log"
    "log/slog"
    "os"
)

func main() {
    id, user, ms := 81, "asha", 2100

    // 1. unstructured: a sentence for humans
    log.SetOutput(os.Stdout)
    log.Printf("order %d failed for user %s after %dms", id, user, ms)

    // 2. slog's default logger: same sentence style, but with key=value pairs
    slog.Error("order failed", "order_id", id, "user", user, "ms", ms)

    // 3. a TextHandler: key=value fields
    text := slog.New(slog.NewTextHandler(os.Stdout, nil))
    text.Error("order failed", "order_id", id, "user", user, "ms", ms)

    // 4. a JSONHandler: one JSON object per line
    js := slog.New(slog.NewJSONHandler(os.Stdout, nil))
    js.Error("order failed", "order_id", id, "user", user, "ms", ms)
}
2026/11/13 23:58:10 order 81 failed for user asha after 2100ms
2026/11/13 23:58:10 ERROR order failed order_id=81 user=asha ms=2100
time=2026-11-13T23:58:10.922Z level=ERROR msg="order failed" order_id=81 user=asha ms=2100
{"time":"2026-11-13T23:58:10.922372583Z","level":"ERROR","msg":"order failed","order_id":81,"user":"asha","ms":2100}

(Timestamps are from this run; yours will show the current time.)

  1. log.Printf: a sentence. The numbers are inside the text.
  2. slog.Error on the default logger: goes through the log package, so it looks like a log line, but the fields are already key=value.
  3. TextHandler: time=... level=ERROR msg="order failed" order_id=81 ..., every value has a name.
  4. JSONHandler: one JSON object per line, the easiest for machines (log collectors, search tools).

Levels and With

A level says how important a line is: Debug (details while developing) < Info (normal events) < Warn (something odd) < Error (something failed). A handler's Level option is the minimum it prints; production usually shows Info and above.

main.go (standard library only):

package main

import (
    "log/slog"
    "os"
)

func noTime(_ []string, a slog.Attr) slog.Attr {
    if a.Key == slog.TimeKey {
        return slog.Attr{} // drop the time so the output is the same every run
    }
    return a
}

func main() {
    for _, min := range []slog.Level{slog.LevelInfo, slog.LevelDebug} {
        l := slog.New(slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{Level: min, ReplaceAttr: noTime}))
        l.Info("--- handler level " + min.String() + " ---")
        l.Debug("cache lookup", "key", "menu:chai")
        l.Info("order placed", "order_id", 81)
        l.Warn("stock low", "item", "samosa", "left", 3)
        l.Error("payment failed", "gateway", "upi")
    }
    // With: a child logger that adds fields to every line
    reqLog := slog.New(slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{ReplaceAttr: noTime})).
        With("request_id", "req-7")
    reqLog.Info("start")
    reqLog.Info("done", "status", 201)
    // a missing value: slog warns you with !BADKEY
    reqLog.Info("oops", "status")
}
level=INFO msg="--- handler level INFO ---"
level=INFO msg="order placed" order_id=81
level=WARN msg="stock low" item=samosa left=3
level=ERROR msg="payment failed" gateway=upi
level=INFO msg="--- handler level DEBUG ---"
level=DEBUG msg="cache lookup" key=menu:chai
level=INFO msg="order placed" order_id=81
level=WARN msg="stock low" item=samosa left=3
level=ERROR msg="payment failed" gateway=upi
level=INFO msg=start request_id=req-7
level=INFO msg=done request_id=req-7 status=201
level=INFO msg=oops request_id=req-7 !BADKEY=status
  • With Level: slog.LevelInfo (the default) the Debug line was dropped; with slog.LevelDebug it appeared.
  • ReplaceAttr lets you change or remove any field; returning an empty slog.Attr{} removes it. We remove time only so outputs repeat exactly (keep it in real apps).
  • With("request_id", "req-7") returns a child logger that adds that field to every line it writes.
  • "status" with no value printed !BADKEY=status: arguments must come in pairs.

Structured request logging with a request id

The problem: 100 requests run at the same time, and their log lines are mixed together. Which "payment failed" line belongs to which request?

The fix: give every request a request id (a short unique name such as req-102), put it in every line that request writes, and then searching for req-102 gives the whole story of that one request.

How the id reaches code deep inside the handler: the context. Every *http.Request carries a context.Context (r.Context()), a bag of request-scoped values that travels with the request (episode on context). A middleware (a function that wraps a handler and runs code before and after it) creates a logger with the id, puts it into the context, and the handler takes it back out. This is context-aware logging: the request's details travel in its context, so any function that gets the context can log with them.

The episode's program (LOGS)

main.go (standard library only):

package main

import (
    "context"
    "fmt"
    "log/slog"
    "net/http"
    "net/http/httptest"
    "os"
    "sync/atomic"
    "time"
)

type ctxKey struct{}

// every log line in this request carries its request_id
func logger(ctx context.Context) *slog.Logger {
    if l, ok := ctx.Value(ctxKey{}).(*slog.Logger); ok {
        return l
    }
    return slog.Default()
}

type recorder struct {
    http.ResponseWriter
    status int
}

func (r *recorder) WriteHeader(code int) { r.status = code; r.ResponseWriter.WriteHeader(code) }

var nextID atomic.Int64

// middleware: one structured line per request
func withLogging(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        l := slog.Default().With("request_id", fmt.Sprint("req-", 100+nextID.Add(1)))
        ctx := context.WithValue(r.Context(), ctxKey{}, l)
        rec := &recorder{w, 200}
        start := time.Now()
        next.ServeHTTP(rec, r.WithContext(ctx))
        l.Info("request", "method", r.Method, "path", r.URL.Path, "status", rec.status,
            "ms", time.Since(start).Milliseconds())
    })
}

func placeOrder(w http.ResponseWriter, r *http.Request) {
    if r.URL.Query().Get("pay") == "fail" {
        logger(r.Context()).Error("payment failed", "gateway", "upi", "err", "timeout after 2s")
        http.Error(w, "payment failed", http.StatusBadGateway)
        return
    }
    w.WriteHeader(http.StatusCreated)
}

func main() {
    // JSON lines: easy for machines to search ("time" removed here to fit the screen)
    slog.SetDefault(slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
        ReplaceAttr: func(_ []string, a slog.Attr) slog.Attr {
            if a.Key == slog.TimeKey {
                return slog.Attr{}
            }
            return a
        },
    })))
    srv := httptest.NewServer(withLogging(http.HandlerFunc(placeOrder)))
    defer srv.Close()
    http.Post(srv.URL+"/orders", "", nil)
    http.Post(srv.URL+"/orders?pay=fail", "", nil)
}
{"level":"INFO","msg":"request","request_id":"req-101","method":"POST","path":"/orders","status":201,"ms":0}
{"level":"ERROR","msg":"payment failed","request_id":"req-102","gateway":"upi","err":"timeout after 2s"}
{"level":"INFO","msg":"request","request_id":"req-102","method":"POST","path":"/orders","status":502,"ms":0}
  • Line 1: request req-101, POST /orders, status 201 (Created).
  • Line 2: an ERROR line, payment failed, gateway: upi, written deep inside placeOrder, and it carries request_id: req-102 even though placeOrder never saw the id: it came from the context.
  • Line 3: the request line for req-102, status 502 (Bad Gateway).
  • ms is 0 because these test requests are very fast; it's the time in milliseconds.

How the pieces work:

  • type ctxKey struct{}: a private type used as the context key, so no other package can clash with it.
  • logger(ctx) returns the logger stored in the context, or slog.Default() if there is none.
  • recorder wraps http.ResponseWriter and remembers the status code passed to WriteHeader (the default is 200, because a handler that never calls WriteHeader sends 200).
  • nextID is an atomic.Int64: handlers run on many goroutines at once, so the counter must be safe for concurrent use (episode on sync/atomic).
  • slog.SetDefault makes the JSON logger the default one.

Logging habits

Habit Why
one line per request + one line per real problem enough to debug, not so much that it costs a fortune
stable key names: request_id, user_id, ms, status searches and dashboards keep working
never log passwords, tokens, card numbers, OTPs logs are read by many people and kept for a long time
log an error once, where it's handled (episode 25) the same error logged at every level makes 5 lines for 1 problem
send the request id back in a response header (X-Request-ID) a customer or another service can quote it to support

Metrics: the car's dashboard

Real-life example: the dashboard shows numbers, not stories.

In a car Metric type Meaning Example
odometer: total km, only goes up Counter a number that only ever goes up (resets to 0 when the program restarts) http_requests_total
fuel gauge: up and down Gauge a value right now, can go up and down requests in flight, queue length
trips sorted by length: under 5 km, under 20 km... Histogram counts values into ranges called buckets request durations
(no car part) Summary computes percentiles inside the app p50, p99 of payment time

A metric is a named number measured over time. A time series (or series) is one metric with one exact set of labels (key=value tags such as code="500") and its values over time.

Prometheus is a popular system that collects metrics: every few seconds it scrapes each server, which means it sends GET /metrics, reads the plain-text numbers, and stores them over time. client_golang is its Go library: the part we use here instruments application code and serves /metrics.

The episode's program (METRICS)

main.go (require github.com/prometheus/client_golang v1.23.2):

package main

import (
    "bufio"
    "fmt"
    "net/http"
    "net/http/httptest"
    "strconv"
    "strings"
    "time"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
    requests = prometheus.NewCounterVec( // only goes up
        prometheus.CounterOpts{Name: "http_requests_total", Help: "Requests by path and status code."},
        []string{"path", "code"})
    inFlight = prometheus.NewGauge( // goes up and down
        prometheus.GaugeOpts{Name: "http_in_flight_requests", Help: "Requests being served right now."})
    duration = prometheus.NewHistogram( // counts values into buckets
        prometheus.HistogramOpts{Name: "http_request_duration_seconds", Help: "How long requests took.",
            Buckets: []float64{0.01, 0.05, 0.1, 0.5}})
)

type statusRecorder struct {
    http.ResponseWriter
    code int
}

func (s *statusRecorder) WriteHeader(c int) { s.code = c; s.ResponseWriter.WriteHeader(c) }

func instrument(path string, next http.HandlerFunc) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        inFlight.Inc()
        defer inFlight.Dec()
        start := time.Now()
        rec := &statusRecorder{w, 200}
        next(rec, r)
        duration.Observe(time.Since(start).Seconds())
        requests.WithLabelValues(path, strconv.Itoa(rec.code)).Inc()
    }
}

func orders(w http.ResponseWriter, r *http.Request) {
    if r.URL.Query().Get("slow") == "1" {
        time.Sleep(120 * time.Millisecond) // a slow database call
    }
    if r.URL.Query().Get("fail") == "1" {
        w.WriteHeader(http.StatusInternalServerError)
    }
}

func main() {
    reg := prometheus.NewRegistry()
    reg.MustRegister(requests, inFlight, duration)
    mux := http.NewServeMux()
    mux.HandleFunc("/orders", instrument("/orders", orders))
    mux.Handle("/metrics", promhttp.HandlerFor(reg, promhttp.HandlerOpts{})) // Prometheus scrapes this
    srv := httptest.NewServer(mux)
    defer srv.Close()

    for i := 0; i < 20; i++ { // 20 requests: 2 slow, 1 failing
        q := ""
        switch i {
        case 5, 12:
            q = "?slow=1"
        case 9:
            q = "?fail=1"
        }
        http.Get(srv.URL + "/orders" + q)
    }

    resp, err := http.Get(srv.URL + "/metrics")
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()
    sc := bufio.NewScanner(resp.Body)
    for sc.Scan() {
        if line := sc.Text(); strings.HasPrefix(line, "http_") {
            fmt.Println(line)
        }
    }
}
http_in_flight_requests 0
http_request_duration_seconds_bucket{le="0.01"} 18
http_request_duration_seconds_bucket{le="0.05"} 18
http_request_duration_seconds_bucket{le="0.1"} 18
http_request_duration_seconds_bucket{le="0.5"} 20
http_request_duration_seconds_bucket{le="+Inf"} 20
http_request_duration_seconds_sum 0.24065418900000002
http_request_duration_seconds_count 20
http_requests_total{code="200",path="/orders"} 19
http_requests_total{code="500",path="/orders"} 1

(The _sum value is the total seconds of all 20 requests; it changes slightly on every run.)

  • prometheus.NewCounterVec: a counter with labels (path, code). WithLabelValues("/orders", "200") picks one series; .Inc() adds 1. Result: 19 with code 200, 1 with code 500.
  • prometheus.NewGauge: inFlight.Inc() when a request starts, defer inFlight.Dec() when it ends. It is 0 now because nothing is running.
  • prometheus.NewHistogram with Buckets: []float64{0.01, 0.05, 0.1, 0.5} (seconds: 10, 50, 100, 500 ms). Observe(seconds) records one duration.
  • Histogram buckets are cumulative: le means "less than or equal". le="0.01"} 18 = 18 requests took at most 10 ms; still 18 at 100 ms; 20 at 500 ms, so 2 were slow (the two ?slow=1 requests with a 120 ms sleep). le="+Inf" is "everything" and always equals _count.
  • _sum / _count = the average duration (here about 0.24 s / 20 = 12 ms).
  • prometheus.NewRegistry() + MustRegister: a registry is the list of metrics to expose. promhttp.HandlerFor(reg, ...) is the HTTP handler that writes them: the /metrics endpoint.
  • instrument is a middleware: it updates all three metrics around every request, so handlers stay clean.

All four types, and what /metrics really looks like

The episode program printed only the http_ lines. Here is a full /metrics page, including a summary:

main.go (require github.com/prometheus/client_golang v1.23.2):

package main

import (
    "fmt"
    "net/http/httptest"
    "io"
    "strings"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

func main() {
    placed := prometheus.NewCounter(prometheus.CounterOpts{Name: "orders_placed_total", Help: "Orders placed since start."})
    waiting := prometheus.NewGauge(prometheus.GaugeOpts{Name: "customers_waiting", Help: "Customers in the queue now."})
    payHist := prometheus.NewHistogram(prometheus.HistogramOpts{Name: "payment_seconds", Help: "Payment time.",
        Buckets: []float64{0.1, 0.5, 1}})
    paySum := prometheus.NewSummary(prometheus.SummaryOpts{Name: "payment_summary_seconds", Help: "Payment time.",
        Objectives: map[float64]float64{0.5: 0.05, 0.99: 0.001}}) // percentile: allowed error

    reg := prometheus.NewRegistry()
    reg.MustRegister(placed, waiting, payHist, paySum)

    placed.Add(3)
    waiting.Set(5)
    waiting.Dec() // one customer served
    // 100 payments: 98 fast (0.05s), 2 slow (0.9s)
    for i := 0; i < 100; i++ {
        d := 0.05
        if i%50 == 49 {
            d = 0.9
        }
        payHist.Observe(d)
        paySum.Observe(d)
    }
    // placed.Add(-1) would panic: "counter cannot decrease in value"

    rec := httptest.NewRecorder()
    promhttp.HandlerFor(reg, promhttp.HandlerOpts{}).ServeHTTP(rec, httptest.NewRequest("GET", "/metrics", nil))
    body, _ := io.ReadAll(rec.Body)
    fmt.Print(strings.TrimSpace(string(body)), "\n")
}
# HELP customers_waiting Customers in the queue now.
# TYPE customers_waiting gauge
customers_waiting 4
# HELP orders_placed_total Orders placed since start.
# TYPE orders_placed_total counter
orders_placed_total 3
# HELP payment_seconds Payment time.
# TYPE payment_seconds histogram
payment_seconds_bucket{le="0.1"} 98
payment_seconds_bucket{le="0.5"} 98
payment_seconds_bucket{le="1"} 100
payment_seconds_bucket{le="+Inf"} 100
payment_seconds_sum 6.699999999999991
payment_seconds_count 100
# HELP payment_summary_seconds Payment time.
# TYPE payment_summary_seconds summary
payment_summary_seconds{quantile="0.5"} 0.05
payment_summary_seconds{quantile="0.99"} 0.9
payment_summary_seconds_sum 6.699999999999991
payment_summary_seconds_count 100
  • # HELP is the description you passed as Help; # TYPE says counter, gauge, histogram or summary. This text format is what Prometheus scrapes.
  • Counter: Add(3) → 3. Gauge: Set(5) then Dec() → 4.
  • The histogram says 98 payments took ≤ 0.1 s, all 100 took ≤ 1 s. It does not store each value, only the bucket counts.
  • The summary computed the percentiles itself: quantile="0.5" (the median) is 0.05, quantile="0.99" is 0.9. Objectives maps each percentile to the allowed error.
  • Prefer histograms: a summary's percentiles are computed inside one process and "cannot be aggregated" (you can't add or average p99s from 10 servers), while histogram buckets from all servers can simply be added, and the server computes percentiles from them (histogram_quantile in PromQL, Prometheus's query language).

A counter really can't go down:

main.go (require github.com/prometheus/client_golang v1.23.2):

package main

import "github.com/prometheus/client_golang/prometheus"

func main() {
    placed := prometheus.NewCounter(prometheus.CounterOpts{Name: "orders_placed_total", Help: "Orders placed."})
    placed.Add(3)
    placed.Add(-1) // a refund? counters only go up
}
panic: counter cannot decrease in value

(followed by the goroutine stack trace). Use a gauge for values that go down, or a second counter (refunds_total).

p99: why averages lie

A percentile answers "how slow were the slowest X%?". p50 (the median): half the requests were faster. p99: 99% of requests were faster than this; it shows the slow tail that an average hides. 1% sounds small, but at 1,000 requests a minute that's 10 unhappy customers every minute.

main.go (standard library only):

package main

import (
    "fmt"
    "sort"
    "time"
)

func main() {
    // 1000 requests: 985 take 10ms, 15 get stuck for 2s
    var d []time.Duration
    for i := 0; i < 1000; i++ {
        d = append(d, 10*time.Millisecond)
    }
    for i := 0; i < 15; i++ {
        d[i*60] = 2 * time.Second
    }
    var sum time.Duration
    for _, x := range d {
        sum += x
    }
    sort.Slice(d, func(i, j int) bool { return d[i] < d[j] })
    pct := func(p float64) time.Duration { return d[int(p*float64(len(d)))-1] } // the value p of requests are at or below
    fmt.Println("average:", sum/time.Duration(len(d)))
    fmt.Println("p50:    ", pct(0.50))
    fmt.Println("p90:    ", pct(0.90))
    fmt.Println("p99:    ", pct(0.99))
    fmt.Println("max:    ", d[len(d)-1])
}
average: 39.85ms
p50:     10ms
p90:     10ms
p99:     2s
max:     2s

The average says "40 ms, fine". p50 and p90 say "10 ms". Only p99 = 2s shows that 15 customers waited 2 seconds. That's why we measure durations with histograms and look at p99.

Labels and cardinality

A label splits one metric into several series: http_requests_total{code="200"} and {code="500"}. Cardinality is the number of different series a metric has = the number of different label-value combinations. Every series costs memory in your app and in Prometheus.

main.go (require github.com/prometheus/client_golang v1.23.2):

package main

import (
    "fmt"
    "io"
    "net/http/httptest"
    "strings"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

func series(reg *prometheus.Registry, name string) int {
    rec := httptest.NewRecorder()
    promhttp.HandlerFor(reg, promhttp.HandlerOpts{}).ServeHTTP(rec, httptest.NewRequest("GET", "/metrics", nil))
    b, _ := io.ReadAll(rec.Body)
    n := 0
    for _, line := range strings.Split(string(b), "\n") {
        if strings.HasPrefix(line, name+"{") {
            n++
        }
    }
    return n
}

func main() {
    good := prometheus.NewCounterVec(prometheus.CounterOpts{Name: "orders_total", Help: "Orders."},
        []string{"status"}) // few, fixed values
    bad := prometheus.NewCounterVec(prometheus.CounterOpts{Name: "orders_by_user_total", Help: "Orders."},
        []string{"user_id"}) // one value per customer!
    reg := prometheus.NewRegistry()
    reg.MustRegister(good, bad)

    for user := 1; user <= 5000; user++ { // 5000 customers order once each
        status := "ok"
        if user%100 == 0 {
            status = "failed"
        }
        good.WithLabelValues(status).Inc()
        bad.WithLabelValues(fmt.Sprint("u", user)).Inc()
    }
    fmt.Println("orders_total series:        ", series(reg, "orders_total"))
    fmt.Println("orders_by_user_total series:", series(reg, "orders_by_user_total"))

    // wrong number of label values: a panic
    defer func() { fmt.Println("panic:", recover()) }()
    good.WithLabelValues("ok", "upi").Inc()
}
orders_total series:         2
orders_by_user_total series: 5000
panic: inconsistent label cardinality: expected 1 label values but got 2 in []string{"ok", "upi"}
  • status has 2 values → 2 series. user_id has one value per customer → 5000 series, and it grows with every new customer (millions later). That's high cardinality: Prometheus slows down and memory explodes.
  • Rule: labels only with few, fixed values (route, method, status code). Never user ids, order ids, emails or full URLs with ids in them. Per-user detail belongs in logs and traces.
  • The wrong number of label values is a panic: inconsistent label cardinality: expected 1 label values but got 2 in []string{"ok", "upi"}. (Here we caught it with recover to print it.)

How Prometheus uses these numbers

  • It scrapes GET /metrics on every server every few seconds and stores each series over time.
  • Queries turn counters into rates. The video's example (named in the video): rate(http_requests_total{code="500"}[5m]) = errors per second over the last 5 minutes.
  • An alert fires when a query crosses a limit ("more than 1% errors for 5 minutes", "p99 above 500 ms").
  • Keep /metrics (and /debug/vars) off the public internet: they show internal details.

What to measure: RED and USE

Two checklists so you don't measure everything and still miss what matters.

RED, for every service that answers requests:

Letter Means Our metric
Rate requests per second http_requests_total (all codes)
Errors failed requests per second http_requests_total{code="500"}
Duration how long they take (p50, p99) http_request_duration_seconds histogram

Our three metrics in the METRICS program are exactly RED (plus the in-flight gauge).

USE, for every resource (CPU, memory, disk, a database connection pool, a queue):

Letter Means Example
Utilization how busy it is CPU 80%; 9 of 10 DB connections in use
Saturation how much work is waiting queue length; the pool's WaitCount (episode 67)
Errors failures of the resource connection errors, disk errors

RED tells you users are hurting; USE tells you which resource is the cause.

A USE metric for a pool, using a GaugeFunc (a gauge whose value is read by calling your function at scrape time):

main.go (require github.com/prometheus/client_golang v1.23.2):

package main

import (
    "fmt"
    "io"
    "net/http/httptest"
    "strings"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

func main() {
    pool := make(chan struct{}, 10) // a pool of 10 connections (a slot taken = in use)
    waiting := 0                    // requests waiting for a free slot

    reg := prometheus.NewRegistry()
    reg.MustRegister(
        prometheus.NewGaugeFunc(prometheus.GaugeOpts{Name: "db_pool_in_use", Help: "Utilization: connections in use."},
            func() float64 { return float64(len(pool)) }), // read at scrape time
        prometheus.NewGaugeFunc(prometheus.GaugeOpts{Name: "db_pool_waiting", Help: "Saturation: requests waiting."},
            func() float64 { return float64(waiting) }),
    )
    for i := 0; i < 9; i++ {
        pool <- struct{}{} // 9 of 10 busy
    }
    waiting = 4

    rec := httptest.NewRecorder()
    promhttp.HandlerFor(reg, promhttp.HandlerOpts{}).ServeHTTP(rec, httptest.NewRequest("GET", "/metrics", nil))
    b, _ := io.ReadAll(rec.Body)
    for _, l := range strings.Split(string(b), "\n") {
        if strings.HasPrefix(l, "db_") {
            fmt.Println(l)
        }
    }
}
db_pool_in_use 9
db_pool_waiting 4

Utilization 9 of 10, and 4 requests waiting: the pool is saturated. With database/sql you would return db.Stats().InUse and db.Stats().WaitCount (episode 67).

Traces: tracking a courier parcel

Logs and metrics say "orders are slow", but one order passes through the order service, the database and the kitchen service. Which one is slow?

Real-life example: a courier parcel.

Courier Distributed tracing
one tracking number for the whole journey trace id: one id for one request, across every service
each hub scans it in and out (in 9:00, out 11:00) span: one step of work with a name, a start, an end and a parent span
the label travels with the parcel the traceparent HTTP header carries the trace id to the next service (propagation)
  • A trace is the whole journey of one request. It is a tree of spans: the first span (the root span) has no parent; every other span has a parent, so you see what happened inside what.
  • Distributed tracing means the trace continues across many services (programs on different machines).
  • OpenTelemetry (OTel) is the open standard and set of libraries for traces, metrics and logs. In Go: the API (go.opentelemetry.io/otel, what your code calls), the SDK (go.opentelemetry.io/otel/sdk, which processes and samples spans) and an exporter (which sends finished spans somewhere).

The episode's program (TRACE)

main.go (require go.opentelemetry.io/otel/sdk v1.39.0):

package main

import (
    "context"
    "fmt"
    "net/http"
    "net/http/httptest"
    "time"

    "go.opentelemetry.io/otel/propagation"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    "go.opentelemetry.io/otel/sdk/trace/tracetest"
)

var (
    recorder = tracetest.NewSpanRecorder() // keeps finished spans so we can print them
    provider = sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(recorder))
    tracer   = provider.Tracer("shop")
    prop     = propagation.TraceContext{} // the W3C traceparent header
)

// the kitchen service: continues the trace it receives in the header
func kitchen(w http.ResponseWriter, r *http.Request) {
    ctx := prop.Extract(r.Context(), propagation.HeaderCarrier(r.Header))
    fmt.Println("  kitchen got traceparent:", r.Header.Get("traceparent"))
    _, span := tracer.Start(ctx, "kitchen.cook")
    time.Sleep(120 * time.Millisecond) // the slow part
    span.End()
}

func main() {
    kitchenSrv := httptest.NewServer(http.HandlerFunc(kitchen))
    defer kitchenSrv.Close()

    // the order service handles one customer request
    ctx, root := tracer.Start(context.Background(), "POST /orders")
    _, db := tracer.Start(ctx, "db.insert_order")
    time.Sleep(15 * time.Millisecond)
    db.End()

    req, _ := http.NewRequestWithContext(ctx, "POST", kitchenSrv.URL, nil)
    prop.Inject(ctx, propagation.HeaderCarrier(req.Header)) // trace id travels in a header
    http.DefaultClient.Do(req)
    root.End()

    for _, s := range recorder.Ended() {
        fmt.Printf("trace %s… span %s parent %-16s %-16s %v\n", s.SpanContext().TraceID().String()[:8],
            s.SpanContext().SpanID(), s.Parent().SpanID(), s.Name(), s.EndTime().Sub(s.StartTime()).Round(5*time.Millisecond))
    }
}
  kitchen got traceparent: 00-8c5d6237cdf64640fdc4338e1ebe0281-991a961f6cc4091d-01
trace 8c5d6237… span 42dc2abf36469ad6 parent 991a961f6cc4091d db.insert_order  15ms
trace 8c5d6237… span 91d81c7e12947fed parent 991a961f6cc4091d kitchen.cook     120ms
trace 8c5d6237… span 991a961f6cc4091d parent 0000000000000000 POST /orders     135ms

(Ids and the root's duration change every run: the video's run and this one both showed 135 ms; an earlier run in this session showed 140 ms.)

  • sdktrace.NewTracerProvider creates tracers; WithSpanProcessor(recorder) sends every finished span to a tracetest.SpanRecorder, which simply keeps them in memory so we can print them (real apps use an exporter instead).
  • provider.Tracer("shop") returns a tracer, the object that starts spans. "shop" names the code doing the tracing.
  • tracer.Start(ctx, "POST /orders") starts a span and returns a new context that contains it. Spans started from that context become its children. span.End() records the end time.
  • propagation.TraceContext{} implements the W3C Trace Context format. prop.Inject(ctx, HeaderCarrier(req.Header)) writes the traceparent header into the outgoing request; prop.Extract(r.Context(), HeaderCarrier(r.Header)) in the kitchen reads it back, so the kitchen's span joins the same trace.
  • Reading the output: all three spans have the same trace id. db.insert_order (15 ms) and kitchen.cook (120 ms) both have parent = the span id of POST /orders (135 ms). The traceparent the kitchen received ends with that same span id. So 120 of 135 ms were in the kitchen, found in one look.
  • The root is printed last because it finished last (recorder.Ended() is in end order). A parent id of all zeros means "no parent": the root.

Forgetting to propagate

The same program with the prop.Inject line commented out:

    // prop.Inject(ctx, propagation.HeaderCarrier(req.Header)) // forgotten!
  kitchen got traceparent: 
trace 13dc4df1… span d719d4e01d428831 parent ee032cf000613dc7 db.insert_order  15ms
trace 88335983… span 2ccc509ac6bb5feb parent 0000000000000000 kitchen.cook     120ms
trace 13dc4df1… span ee032cf000613dc7 parent 0000000000000000 POST /orders     135ms

The kitchen got an empty traceparent, so it started a new trace (88335983…, parent all zeros). Now there are two separate traces and nothing connects the slow kitchen to the customer's request. In real apps, instrumentation libraries do the Inject/Extract for you (see below), but only if every hop uses them.

Reading a traceparent header

traceparent: 00-<trace-id: 32 hex>-<parent span id: 16 hex>-01
Part Example Meaning
version 00 the format version
trace-id 11b5c5e1aa9367dc12e09556aadc27ca 16 bytes = 32 hex characters, the same for the whole trace
parent-id 8d5791a2829506ed 8 bytes = 16 hex characters, the id of the span that made the call
trace-flags 01 01 = sampled (this trace is being recorded), 00 = not sampled

(Hex = hexadecimal, digits 0-9 and a-f; two hex characters are one byte.)

main.go (require go.opentelemetry.io/otel/sdk v1.39.0):

package main

import (
    "context"
    "fmt"
    "net/http"
    "strings"

    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/otel/trace"
)

func main() {
    tp := "00-11b5c5e1aa9367dc12e09556aadc27ca-8d5791a2829506ed-01"
    p := strings.Split(tp, "-")
    fmt.Printf("version %s | trace id %s (%d hex) | parent id %s (%d hex) | flags %s\n",
        p[0], p[1], len(p[1]), p[2], len(p[2]), p[3])

    // what Extract makes of it (and of broken headers)
    prop := propagation.TraceContext{}
    for _, h := range []string{tp,
        "00-11b5c5e1aa9367dc12e09556aadc27ca-8d5791a2829506ed-00", // not sampled
        "00-11b5c5e1aa9367dc-8d5791a2829506ed-01",                 // trace id too short
    } {
        hdr := http.Header{}
        hdr.Set("traceparent", h)
        sc := trace.SpanContextFromContext(prop.Extract(context.Background(), propagation.HeaderCarrier(hdr)))
        fmt.Printf("valid=%-5v remote=%-5v sampled=%-5v trace=%s\n", sc.IsValid(), sc.IsRemote(), sc.IsSampled(), sc.TraceID())
    }
}
version 00 | trace id 11b5c5e1aa9367dc12e09556aadc27ca (32 hex) | parent id 8d5791a2829506ed (16 hex) | flags 01
valid=true  remote=true  sampled=true  trace=11b5c5e1aa9367dc12e09556aadc27ca
valid=true  remote=true  sampled=false trace=11b5c5e1aa9367dc12e09556aadc27ca
valid=false remote=false sampled=false trace=00000000000000000000000000000000
  • A good header gives a valid, remote (it came from another service) span context with the same trace id.
  • Flags 00: valid, but not sampled.
  • A trace id that's too short is invalid: Extract ignores it and the trace id is all zeros, so this service would start a new trace.

Sampling

Recording every span of every request costs network, storage and money. Sampling records only some traces. The SDK's TraceIDRatioBased(0.1) keeps about 1 trace in 10, decided from the trace id:

main.go (require go.opentelemetry.io/otel/sdk v1.39.0):

package main

import (
    "context"
    "fmt"

    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    "go.opentelemetry.io/otel/sdk/trace/tracetest"
)

func main() {
    rec := tracetest.NewSpanRecorder()
    provider := sdktrace.NewTracerProvider(
        sdktrace.WithSampler(sdktrace.TraceIDRatioBased(0.1)), // keep about 1 trace in 10
        sdktrace.WithSpanProcessor(rec))
    tracer := provider.Tracer("shop")
    for i := 0; i < 1000; i++ {
        _, span := tracer.Start(context.Background(), "POST /orders")
        span.End()
    }
    fmt.Println("requests: 1000, traces recorded:", len(rec.Ended()))
}
requests: 1000, traces recorded: 93

About 100 of 1000; the exact number changes every run because trace ids are random (other runs here printed 99 and 104). The decision is written into the traceparent flags (01 / 00) so the next service follows it.

OpenTelemetry in real apps

  • Your code calls the API, the SDK processes spans, and an exporter sends them out. The opentelemetry-go README lists the OTLP exporter (OpenTelemetry's own protocol, for logs, metrics and traces), plus Prometheus, stdout and Zipkin exporters.
  • Spans are sent to a tracing backend that stores them and draws the timeline (the video names Jaeger and Grafana Tempo as examples).
  • You rarely start HTTP spans by hand: instrumentation libraries (named in the video) wrap net/http, gRPC and database/sql and create spans and do the Inject/Extract for you. You add spans for your own important steps (db.insert_order, kitchen.cook).
  • Put the trace id in your log lines too (the mini project does): from a log line you can jump to its trace, and back.

Health checks: liveness and readiness

Real-life example: a clinic asks two different questions about a doctor.

At the clinic Health check Endpoint If it fails
Is the doctor alive and awake? liveness: is the process alive, or stuck? /healthz restart the instance
Can the doctor see patients right now? (not while setting up or leaving) readiness: should this instance get traffic now? /readyz stop sending it requests (don't restart)

A health check is an HTTP endpoint that a load balancer (the thing that spreads requests over instances) or an orchestrator (named in the video: Kubernetes) calls every few seconds. An instance (or pod) is one running copy of your service.

The episode's program (HEALTH)

main.go (standard library only):

package main

import (
    "expvar"
    "fmt"
    "io"
    "net/http"
    "net/http/httptest"
    "strings"
    "sync/atomic"
)

var (
    dbReady  atomic.Bool // set once the database answers
    draining atomic.Bool // set when SIGTERM arrives during a deploy
    orders   = expvar.NewInt("orders_placed") // shows up at /debug/vars
)

func main() {
    mux := http.NewServeMux()
    // liveness: "is the process alive?" Keep it dumb: if this fails, restart me.
    mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprint(w, "ok")
    })
    // readiness: "should I get traffic right now?"
    mux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
        if !dbReady.Load() || draining.Load() {
            http.Error(w, "not ready", http.StatusServiceUnavailable)
            return
        }
        fmt.Fprint(w, "ready")
    })
    mux.HandleFunc("/orders", func(w http.ResponseWriter, r *http.Request) { orders.Add(1) })
    mux.Handle("/debug/vars", expvar.Handler())
    srv := httptest.NewServer(mux)
    defer srv.Close()

    get := func(path string) string {
        resp, err := http.Get(srv.URL + path)
        if err != nil {
            return err.Error()
        }
        defer resp.Body.Close()
        b, _ := io.ReadAll(resp.Body)
        return fmt.Sprintf("%d %s", resp.StatusCode, strings.TrimSpace(string(b)))
    }
    fmt.Println("starting up:  healthz", get("/healthz"), "| readyz", get("/readyz"))
    dbReady.Store(true)
    fmt.Println("db connected: healthz", get("/healthz"), "| readyz", get("/readyz"))
    get("/orders")
    get("/orders")
    draining.Store(true)
    fmt.Println("SIGTERM:      healthz", get("/healthz"), "| readyz", get("/readyz"))

    vars := get("/debug/vars")
    i := strings.Index(vars, `"orders_placed"`)
    fmt.Println("expvar:", vars[i:i+len(`"orders_placed": 2`)], "(plus cmdline and memstats)")
}
starting up:  healthz 200 ok | readyz 503 not ready
db connected: healthz 200 ok | readyz 200 ready
SIGTERM:      healthz 200 ok | readyz 503 not ready
expvar: "orders_placed": 2 (plus cmdline and memstats)
  • Starting up: alive (200 ok) but not ready (503 not ready): the database isn't connected yet, so no traffic.
  • DB connected: 200 and 200.
  • SIGTERM (the "please stop" signal a deploy sends): still alive, but readiness says 503, so the load balancer stops sending new requests while the old ones finish. This is called draining.
  • 503 is Service Unavailable; http.Error writes the status and the text.
  • atomic.Bool flags are safe to read from handler goroutines while main changes them.

expvar

expvar is a standard-library package that publishes variables as JSON at /debug/vars. expvar.NewInt("orders_placed") creates a counter you .Add(1) to; expvar.Handler() serves the page (or importing expvar registers it on http.DefaultServeMux automatically). It also publishes cmdline (the program's arguments) and memstats (memory statistics). Above it showed "orders_placed": 2. Quick, no setup, no labels, no histograms: fine for a small tool, while Prometheus metrics are the usual choice for services.

Health check rules

  • Liveness: cheap, no database check. If it checked the database, a short database blip would make every instance fail liveness and get restarted at once, making the outage worse.
  • Readiness: dependencies OK? still warming up? draining?
  • On SIGTERM: fail readiness, finish the requests in progress, then exit (graceful shutdown, episode 74).
  • Keep /debug/vars and /metrics off the public internet.

Observability in one page

Signal Job Go tool
Metrics raise the alarm: RED for services, USE for resources client_golang: counter ↑, gauge ↕, histogram buckets → p99, few label values
Traces find where the time went OpenTelemetry: trace id + spans + traceparent propagation
Logs explain why log/slog JSON, request_id via context, levels, no secrets
Health alive vs ready /healthz, /readyz, expvar at /debug/vars

Mini project: observable order API

Make Raju's order API observable:

  1. a slog JSON middleware with a request_id (keep the caller's X-Request-ID header if it sent one, and send it back);
  2. RED metrics per route: a counter (route, code) and a histogram (route); /metrics;
  3. spans for the handler, the DB call and the kitchen call (with traceparent propagation to the kitchen service);
  4. /healthz and /readyz with a ready flag and a draining flag;
  5. make one route slow on purpose and find it three ways: in the metrics, in a trace, and in the logs.
Full solution

main.go:

package main

import (
    "bufio"
    "context"
    "fmt"
    "log/slog"
    "net/http"
    "net/http/httptest"
    "os"
    "strconv"
    "strings"
    "sync/atomic"
    "time"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
    "go.opentelemetry.io/otel/propagation"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    "go.opentelemetry.io/otel/sdk/trace/tracetest"
)

// ---------- tracing ----------
var (
    spans    = tracetest.NewSpanRecorder()
    provider = sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(spans))
    tracer   = provider.Tracer("order-api")
    prop     = propagation.TraceContext{}
)

// ---------- metrics (RED) ----------
var (
    reqTotal = prometheus.NewCounterVec(prometheus.CounterOpts{Name: "http_requests_total",
        Help: "Requests by route and status code."}, []string{"route", "code"}) // Rate + Errors
    reqDur = prometheus.NewHistogramVec(prometheus.HistogramOpts{Name: "http_request_duration_seconds",
        Help: "Request duration by route.", Buckets: []float64{0.05, 0.25}}, []string{"route"}) // Duration
)

// ---------- logging ----------
type ctxKey struct{}

func logger(ctx context.Context) *slog.Logger {
    if l, ok := ctx.Value(ctxKey{}).(*slog.Logger); ok {
        return l
    }
    return slog.Default()
}

type statusRecorder struct {
    http.ResponseWriter
    code int
}

func (s *statusRecorder) WriteHeader(c int) { s.code = c; s.ResponseWriter.WriteHeader(c) }

var nextID atomic.Int64

// observe wraps one route with a span, a request id, a log line and RED metrics.
func observe(route string, next http.HandlerFunc) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        id := r.Header.Get("X-Request-ID") // keep the caller's id if it sent one
        if id == "" {
            id = fmt.Sprint("req-", 100+nextID.Add(1))
        }
        w.Header().Set("X-Request-ID", id)

        ctx := prop.Extract(r.Context(), propagation.HeaderCarrier(r.Header))
        ctx, span := tracer.Start(ctx, r.Method+" "+route)
        defer span.End()

        l := slog.Default().With("request_id", id, "trace_id", span.SpanContext().TraceID().String()[:8])
        ctx = context.WithValue(ctx, ctxKey{}, l)

        rec := &statusRecorder{w, http.StatusOK}
        start := time.Now()
        next(rec, r.WithContext(ctx))
        took := time.Since(start)

        reqTotal.WithLabelValues(route, strconv.Itoa(rec.code)).Inc()
        reqDur.WithLabelValues(route).Observe(took.Seconds())
        l.Info("request", "method", r.Method, "route", route, "status", rec.code, "ms", took.Milliseconds())
    })
}

// ---------- the app ----------
var kitchenURL string

func menu(w http.ResponseWriter, r *http.Request) { fmt.Fprint(w, "chai, samosa") }

func placeOrder(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    if r.URL.Query().Get("qty") == "0" {
        logger(ctx).Warn("bad order", "reason", "qty must be 1-20")
        http.Error(w, "bad qty", http.StatusBadRequest)
        return
    }
    _, db := tracer.Start(ctx, "db.insert_order")
    time.Sleep(10 * time.Millisecond)
    db.End()

    ctx, call := tracer.Start(ctx, "call kitchen")
    req, _ := http.NewRequestWithContext(ctx, "POST", kitchenURL, nil)
    prop.Inject(ctx, propagation.HeaderCarrier(req.Header)) // the trace continues in the kitchen
    resp, err := http.DefaultClient.Do(req)
    call.End()
    if err != nil {
        logger(ctx).Error("kitchen call failed", "err", err)
        http.Error(w, "kitchen down", http.StatusBadGateway)
        return
    }
    resp.Body.Close()
    w.WriteHeader(http.StatusCreated)
}

func kitchen(w http.ResponseWriter, r *http.Request) {
    ctx := prop.Extract(r.Context(), propagation.HeaderCarrier(r.Header))
    _, span := tracer.Start(ctx, "kitchen.cook")
    time.Sleep(120 * time.Millisecond) // the slow part, on purpose
    span.End()
}

func noTime(_ []string, a slog.Attr) slog.Attr {
    if a.Key == slog.TimeKey {
        return slog.Attr{}
    }
    return a
}

func main() {
    slog.SetDefault(slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{ReplaceAttr: noTime})))
    reg := prometheus.NewRegistry()
    reg.MustRegister(reqTotal, reqDur)

    kitchenSrv := httptest.NewServer(http.HandlerFunc(kitchen))
    defer kitchenSrv.Close()
    kitchenURL = kitchenSrv.URL

    var ready, draining atomic.Bool
    mux := http.NewServeMux()
    mux.Handle("GET /menu", observe("/menu", menu))
    mux.Handle("POST /orders", observe("/orders", placeOrder))
    mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) { fmt.Fprint(w, "ok") })
    mux.HandleFunc("GET /readyz", func(w http.ResponseWriter, r *http.Request) {
        if !ready.Load() || draining.Load() {
            http.Error(w, "not ready", http.StatusServiceUnavailable)
            return
        }
        fmt.Fprint(w, "ready")
    })
    mux.Handle("GET /metrics", promhttp.HandlerFor(reg, promhttp.HandlerOpts{}))
    srv := httptest.NewServer(mux)
    defer srv.Close()

    status := func(method, path string, hdr ...string) int {
        req, _ := http.NewRequest(method, srv.URL+path, nil)
        if len(hdr) == 2 {
            req.Header.Set(hdr[0], hdr[1])
        }
        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            return 0
        }
        resp.Body.Close()
        return resp.StatusCode
    }

    fmt.Println("== health")
    fmt.Println("startup  healthz", status("GET", "/healthz"), "readyz", status("GET", "/readyz"))
    ready.Store(true)
    fmt.Println("ready    healthz", status("GET", "/healthz"), "readyz", status("GET", "/readyz"))

    fmt.Println("== logs")
    status("GET", "/menu")
    status("GET", "/menu")
    status("POST", "/orders")
    status("POST", "/orders?qty=0", "X-Request-ID", "mobile-42")

    fmt.Println("== metrics (way 1: which route is slow?)")
    resp, err := http.Get(srv.URL + "/metrics")
    if err != nil {
        panic(err)
    }
    sc := bufio.NewScanner(resp.Body)
    for sc.Scan() {
        if line := sc.Text(); strings.HasPrefix(line, "http_") && !strings.Contains(line, "_sum") {
            fmt.Println(line)
        }
    }
    resp.Body.Close()

    fmt.Println("== traces (way 2: where inside the route?)")
    for _, s := range spans.Ended() {
        fmt.Printf("trace %s… span %s parent %-16s %-16s %v\n", s.SpanContext().TraceID().String()[:8],
            s.SpanContext().SpanID(), s.Parent().SpanID(), s.Name(), s.EndTime().Sub(s.StartTime()).Round(5*time.Millisecond))
    }

    draining.Store(true) // SIGTERM arrives
    fmt.Println("== SIGTERM: healthz", status("GET", "/healthz"), "readyz", status("GET", "/readyz"))
}

go.mod:

module shop

go 1.24

require (
    github.com/prometheus/client_golang v1.23.2
    go.opentelemetry.io/otel/sdk v1.39.0
)
== health
startup  healthz 200 readyz 503
ready    healthz 200 readyz 200
== logs
{"level":"INFO","msg":"request","request_id":"req-101","trace_id":"c7391cec","method":"GET","route":"/menu","status":200,"ms":0}
{"level":"INFO","msg":"request","request_id":"req-102","trace_id":"dfd97f3c","method":"GET","route":"/menu","status":200,"ms":0}
{"level":"INFO","msg":"request","request_id":"req-103","trace_id":"ec21a898","method":"POST","route":"/orders","status":201,"ms":131}
{"level":"WARN","msg":"bad order","request_id":"mobile-42","trace_id":"32e61c33","reason":"qty must be 1-20"}
{"level":"INFO","msg":"request","request_id":"mobile-42","trace_id":"32e61c33","method":"POST","route":"/orders","status":400,"ms":0}
== metrics (way 1: which route is slow?)
http_request_duration_seconds_bucket{route="/menu",le="0.05"} 2
http_request_duration_seconds_bucket{route="/menu",le="0.25"} 2
http_request_duration_seconds_bucket{route="/menu",le="+Inf"} 2
http_request_duration_seconds_count{route="/menu"} 2
http_request_duration_seconds_bucket{route="/orders",le="0.05"} 1
http_request_duration_seconds_bucket{route="/orders",le="0.25"} 2
http_request_duration_seconds_bucket{route="/orders",le="+Inf"} 2
http_request_duration_seconds_count{route="/orders"} 2
http_requests_total{code="200",route="/menu"} 2
http_requests_total{code="201",route="/orders"} 1
http_requests_total{code="400",route="/orders"} 1
== traces (way 2: where inside the route?)
trace c7391cec… span 20e1cce57f6ba4cd parent 0000000000000000 GET /menu        0s
trace dfd97f3c… span 3148281a422cb468 parent 0000000000000000 GET /menu        0s
trace ec21a898… span f028648e1917fd9b parent 7b636908742f775b db.insert_order  10ms
trace ec21a898… span e2a1bb697f70e773 parent c45f98b40c707945 kitchen.cook     120ms
trace ec21a898… span c45f98b40c707945 parent 7b636908742f775b call kitchen     120ms
trace ec21a898… span 7b636908742f775b parent 0000000000000000 POST /orders     130ms
trace 32e61c33… span f99a0af2d536ac96 parent 0000000000000000 POST /orders     0s
== SIGTERM: healthz 200 readyz 503

(Trace ids, span ids and the milliseconds change every run.)

Finding the slow route three ways:

  1. Metrics: /menu has both requests in the le="0.05" bucket; /orders has only 1 of 2 under 50 ms (the other, the 400, returned early) and needed the 0.25 bucket. /orders is slow.
  2. Trace: in trace ec21a898…, POST /orders took 130 ms, of which call kitchen took 120 ms and its child kitchen.cook (in the kitchen service, same trace id thanks to traceparent) took 120 ms. The kitchen is the slow part.
  3. Logs: req-103 shows "route":"/orders","status":201,"ms":131, and its trace_id leads straight to the trace. The mobile-42 lines show the caller's own X-Request-ID was kept, and the WARN line and the request line share it.

Design notes: validation runs first, so a bad order (400) costs nothing; the route label is the pattern (/orders), not the full URL, so cardinality stays low; the logger carries both request_id and trace_id; /healthz, /readyz and /metrics are not wrapped in observe, so health checks don't flood the logs and metrics.

Common mistakes (and the real messages)

Mistake What you see Why Fix
log.Printf sentences order 81 failed for user asha after 2100ms (can't query user=asha AND ms>2000) no fields slog with key-value pairs
no request id lines from 100 requests mixed up nothing ties them together logger.With("request_id", id) in the context
odd number of slog arguments !BADKEY=status keys and values must come in pairs "status", 201
a counter that goes down panic: counter cannot decrease in value counters only go up a gauge, or a second counter
user id as a metric label orders_by_user_total series: 5000 (and growing) high cardinality few, fixed label values; ids go in logs and traces
wrong number of label values panic: inconsistent label cardinality: expected 1 label values but got 2 in []string{"ok", "upi"} one value per label name match the label list
registering a metric twice panic: duplicate metrics collector registration attempted MustRegister in a function that runs more than once register once, at startup
watching only averages average: 39.85ms while p99: 2s averages hide the tail histograms → p99
not propagating traceparent kitchen got traceparent: (empty), two different trace ids the next service starts a new trace Inject / Extract, or instrumentation libraries
a malformed traceparent valid=false ... trace=00000000000000000000000000000000 32 hex trace id, 16 hex parent id required let the propagator write it
liveness checks the database every instance restarted in a DB blip wrong question DB checks go in readiness
logging secrets passwords and tokens in log storage logs are widely read and kept never log them

The duplicate-registration program:

package main

import "github.com/prometheus/client_golang/prometheus"

func main() {
    c := prometheus.NewCounter(prometheus.CounterOpts{Name: "orders_total", Help: "Orders."})
    reg := prometheus.NewRegistry()
    reg.MustRegister(c)
    reg.MustRegister(c) // registered twice, e.g. in a handler that runs per request
}
panic: duplicate metrics collector registration attempted

Practice

  1. Change the LOGS program so the middleware also sends the request id back to the client in an X-Request-ID response header. Where must w.Header().Set go, before or after next.ServeHTTP?
  2. In the METRICS run, how many requests took between 100 ms and 500 ms? Which two lines tell you?
  3. Your API has routes /menu, /orders, /orders/{id}. A teammate labels http_requests_total with r.URL.Path. What goes wrong, and what should the label be?
  4. Which RED letter and which USE letter would show "the DB pool has 9 of 10 connections busy and 4 requests waiting"?
  5. A service receives traceparent: 00-11b5c5e1aa9367dc12e09556aadc27ca-8d5791a2829506ed-00. Is this request's trace recorded? Is it the same trace as one with flags 01 and the same trace id?
  6. Your /healthz pings the database. The database restarts for 20 seconds. What happens to your 30 instances? Fix it.
Solutions

1. Before. Once the handler writes the status or the body, headers are already sent. Put w.Header().Set("X-Request-ID", id) before next.ServeHTTP, exactly as the mini project's observe does (its run shows request_id: mobile-42 kept from the caller).

2. 2 requests. le="0.1"} 18 and le="0.5"} 20: buckets are cumulative, so 20 − 18 = 2.

3. /orders/1, /orders/2, ... each become a new series: unbounded cardinality, like the user_id run (5000 series for 5000 users). Use the route pattern (/orders/{id}) as the label, as the mini project does with observe("/orders", ...).

4. It's a resource, so USE: Utilization (9 of 10 in use) and Saturation (4 waiting). The GaugeFunc run above printed db_pool_in_use 9 and db_pool_waiting 4. RED's Duration would show the effect on users (slower requests).

5. Not recorded: flags 00 = not sampled (the parse run printed sampled=false). Yes, same trace id, so it's the same trace; the flag only says whether it is being recorded.

6. All 30 fail liveness and get restarted together, in the middle of the outage, and restarting instances can't serve anything. Keep /healthz returning ok without dependencies, and check the database in /readyz: instances stop getting traffic for 20 seconds, then come back without restarting.

Quick quiz

1. Orders placed since the server started; customers waiting in the queue right now; how long each payment took. Which metric types? (From the video.)

A. gauge, counter, histogram · B. counter, gauge, histogram · C. counter, counter, gauge · D. histogram, gauge, counter

Answer

B. Orders placed only goes up: a counter. Customers waiting goes up and down: a gauge. Payment durations are many values to spread over ranges: a histogram, which gives percentiles like p99.

2. A histogram prints _bucket{le="0.1"} 98 and _bucket{le="1"} 100. How many observations were above 0.1 s?

Answer

2. Buckets count values at or below le, so 100 − 98 = 2 (the two 0.9 s payments in the types run).

3. During a deploy, an instance gets SIGTERM. What should /healthz and /readyz return?

Answer

/healthz 200 (still alive, don't restart it) and /readyz 503 (stop sending new traffic while it finishes), exactly as the HEALTH run printed.

4. In the TRACE output, why is POST /orders printed after its children?

Answer

recorder.Ended() lists spans in the order they ended, and a parent ends after its children. The tree comes from the parent ids, not from the print order.

Interview questions

  1. What is observability, and what are its three pillars? Understanding a running system's inside from its outputs: logs (events), metrics (numbers over time), traces (one request across services). Metrics alert, traces locate, logs explain.
  2. Why structured logging? Fields can be searched, filtered, counted and alerted on; sentences need fragile grep. In Go: log/slog (Go 1.21+), JSONHandler in production.
  3. How do you get a request id into every log line? A middleware creates slog.Default().With("request_id", id), stores it in the request context, and handlers fetch it from r.Context(). Return the id in a response header.
  4. Counter vs gauge vs histogram vs summary? Only-up total; up-and-down current value; bucket counts (+ sum, count) from which percentiles are computed on the server and which can be aggregated; in-process percentiles that can't be aggregated across instances.
  5. What is label cardinality and why does it matter? The number of series = label-value combinations; unbounded values (user ids, raw paths) explode memory in the app and in Prometheus.
  6. Why p99 and not the average? Averages hide the slow tail; p99 shows what the slowest 1% experience.
  7. Explain RED and USE. RED for services: Rate, Errors, Duration. USE for resources: Utilization, Saturation, Errors.
  8. What is a span, and how does a trace cross services? A named, timed unit of work with a parent; the trace context is injected into an outgoing request's traceparent header and extracted by the receiver, so its spans join the same trace.
  9. What is in a traceparent header? version-traceid(32 hex)-parentid(16 hex)-flags, e.g. 00-...-...-01, where 01 = sampled.
  10. Why sample traces? Cost: recording everything is expensive; a ratio sampler keeps a fraction, and the decision travels in the flags.
  11. Liveness vs readiness? Liveness: alive or restart me (keep it cheap, no dependencies). Readiness: send me traffic now (dependencies, warm-up, draining).
  12. What does expvar give you? JSON variables at /debug/vars from the standard library, plus cmdline and memstats, with no setup.

Glossary

Term Meaning Example
Observability being able to understand what happens inside a running system from the signals it gives out the 2 A.M. dashboard
Monitoring watching signals and raising alarms when they cross a limit "errors above 1%" alert
Signal / telemetry data a program gives out about itself logs, metrics, traces
Three pillars the three main kinds of signal logs, metrics, traces
Production the real system real customers use Raju's live order API
Log a record of one event payment failed
Unstructured logging log lines as free sentences log.Printf("order %d failed...")
Structured logging log lines as named key-value fields {"msg":"order failed","user":"asha"}
Key-value pair a name and its value "ms", 2100 → ms=2100
JSON a text format of {"key": value} objects {"level":"ERROR"}
log/slog Go's structured logging package (Go 1.21+) slog.Info("hello", "count", 3)
Logger the object you call to write log lines *slog.Logger
Handler (slog) decides the format and destination of log lines slog.NewJSONHandler(os.Stdout, nil)
TextHandler writes key=value lines level=INFO msg=start
JSONHandler writes one JSON object per line {"level":"INFO",...}
Default logger the logger used by slog.Info etc. slog.SetDefault(...)
Level importance of a log line Debug < Info < Warn < Error
HandlerOptions.Level the minimum level a handler prints slog.LevelDebug
ReplaceAttr a function that changes or removes fields drop time
Attr one key-value field in slog slog.Attr{Key: "time"}
With returns a child logger that adds fields to every line .With("request_id", "req-7")
!BADKEY slog's marker for a value with no key !BADKEY=status
Request id a unique name for one request, in every line it writes req-102
X-Request-ID a common HTTP header that carries a request id mobile-42
Middleware a function wrapping a handler to run code before and after it withLogging, instrument
Context (context.Context) request-scoped values and deadlines that travel with a request r.Context()
Context-aware logging taking the logger (with request fields) from the context logger(r.Context()).Error(...)
Context key a private type used to store a value in a context type ctxKey struct{}
Status code the HTTP result number 200 OK, 201 Created, 400, 500, 502, 503
Response writer wrapper a type that wraps http.ResponseWriter to remember the status recorder, statusRecorder
atomic.Int64 / atomic.Bool values safe to change from many goroutines nextID.Add(1), draining.Store(true)
httptest.NewServer a real HTTP server on a local port, for tests and demos srv.URL
Metric a named number measured over time http_requests_total
Time series (series) one metric with one exact set of label values, over time http_requests_total{code="500"}
Counter a metric that only goes up orders_placed_total 3
Gauge a metric that goes up and down customers_waiting 4
GaugeFunc a gauge whose value comes from a function at scrape time db_pool_in_use 9
Histogram counts observations into buckets, plus sum and count payment_seconds_bucket{le="0.1"} 98
Bucket a range "at or below X" in a histogram le="0.5"
le "less than or equal": the bucket's upper limit le="+Inf" = everything
Cumulative each bucket includes all smaller ones 18, 18, 18, 20, 20
_sum / _count total of all observed values / number of observations sum ÷ count = average
Summary computes percentiles inside the app; can't be aggregated quantile="0.99"} 0.9
Objectives a summary's percentiles and allowed errors {0.5: 0.05, 0.99: 0.001}
Observe record one value in a histogram or summary duration.Observe(0.12)
Inc / Dec / Add / Set change a counter or gauge inFlight.Inc()
Label a key=value tag that splits a metric into series code="500"
CounterVec / HistogramVec a counter / histogram with labels WithLabelValues("/orders", "200")
Cardinality number of different series (label-value combinations) 2 for status, 5000 for user_id
High cardinality a label with too many (or unbounded) values user ids, order ids, raw URLs
Prometheus a popular system that scrapes and stores metrics scrapes /metrics
client_golang Prometheus's Go library github.com/prometheus/client_golang
Registry the set of metrics a handler exposes prometheus.NewRegistry()
MustRegister add metrics to a registry; panics on problems reg.MustRegister(requests)
promhttp.HandlerFor the HTTP handler that writes a registry's metrics mux.Handle("/metrics", ...)
/metrics endpoint the URL where a service shows its metrics as text # TYPE ... counter
# HELP / # TYPE description and type lines in the metrics text # TYPE payment_seconds histogram
Scrape Prometheus fetching /metrics every few seconds GET /metrics
PromQL Prometheus's query language rate(...), histogram_quantile
Rate change per second of a counter errors per second
Alert a notification when a query crosses a limit p99 > 500 ms
Latency / duration how long a request takes 120 ms
Percentile the value below which X% of observations fall p99
p50 (median) half of the requests were faster p50: 10ms
p99 99% of requests were faster than this p99: 2s
Tail latency the slow few requests that averages hide the 15 two-second requests
Average (mean) sum ÷ count average: 39.85ms
RED method Rate, Errors, Duration: what to measure for services http_requests_total + histogram
USE method Utilization, Saturation, Errors: what to measure for resources pool in use / waiting
Utilization how busy a resource is 9 of 10 connections
Saturation how much work is waiting for a resource 4 requests waiting
Connection pool a set of reusable database connections *sql.DB (episode 67)
Trace the whole journey of one request across services trace 8c5d6237…
Distributed tracing tracing a request across many services order → kitchen
Span one named, timed step of work, with a parent kitchen.cook 120ms
Root span the first span of a trace; it has no parent POST /orders (parent all zeros)
Parent / child span a span started inside another span db.insert_order is a child of POST /orders
Trace id 32-hex id shared by every span of a trace 8c5d6237cdf6...
Span id 16-hex id of one span 991a961f6cc4091d
OpenTelemetry (OTel) open standard and libraries for traces, metrics, logs go.opentelemetry.io/otel
API (OTel) what your code calls to make spans tracer.Start
SDK (OTel) processes, samples and exports spans go.opentelemetry.io/otel/sdk
TracerProvider creates tracers and holds span processing settings sdktrace.NewTracerProvider(...)
Tracer the object that starts spans provider.Tracer("shop")
Span processor receives spans as they start and end WithSpanProcessor(recorder)
tracetest.SpanRecorder a processor that keeps finished spans in memory (for tests) recorder.Ended()
Exporter sends finished spans/metrics/logs to a backend OTLP, Zipkin, stdout
OTLP OpenTelemetry's own protocol for sending signals the OTLP exporter
Tracing backend stores spans and draws timelines Jaeger, Grafana Tempo (named in the video)
Instrumentation library wraps a library (HTTP, gRPC, SQL) to create spans for you an HTTP wrapper
Propagation passing the trace context to the next service prop.Inject / prop.Extract
Propagator the code that writes and reads the trace context headers propagation.TraceContext{}
HeaderCarrier adapts http.Header for a propagator propagation.HeaderCarrier(req.Header)
Inject write the trace context into outgoing headers before http.DefaultClient.Do(req)
Extract read the trace context from incoming headers first line of kitchen
W3C Trace Context the web standard for trace headers traceparent, tracestate
traceparent header version-traceid-parentid-flags 00-8c5d...-991a...-01
Hex (hexadecimal) digits 0-9 and a-f; 2 hex characters = 1 byte 8d
Trace flags 01 sampled, 00 not sampled -01
Sampling recording only some traces to save cost TraceIDRatioBased(0.1)
Remote span context a trace context that came from another service remote=true
Health check an endpoint that reports whether an instance is OK /healthz
Liveness "is the process alive?"; failing means restart /healthz → 200 ok
Readiness "should I get traffic now?"; failing means no traffic /readyz → 503 not ready
Instance / pod one running copy of a service one of 30 copies
Load balancer spreads requests over instances skips not-ready instances
Orchestrator runs and restarts instances Kubernetes (named in the video)
SIGTERM the "please stop" signal sent during a deploy draining.Store(true)
Draining finishing current requests while refusing new ones readyz 503 after SIGTERM
Graceful shutdown drain, finish, then exit (episode 74) fail readiness first
503 Service Unavailable "can't serve right now" http.StatusServiceUnavailable
expvar standard package publishing variables as JSON at /debug/vars expvar.NewInt("orders_placed")
/debug/vars expvar's endpoint; also shows cmdline and memstats "orders_placed": 2
Panic a crash that stops the program unless recovered counter cannot decrease in value
recover catches a panic in a deferred function the cardinality demo

Summary

  • Observability = seeing inside a running system from its signals: metrics raise the alarm, traces find where, logs explain why.
  • Logs: log/slog (Go 1.21+) with JSONHandler, levels, stable keys, no secrets; a middleware puts With("request_id", id) into the context, so every line of a request shares its id.
  • Metrics with client_golang: counter (only up), gauge (up and down), histogram (cumulative buckets → p99, aggregatable), summary (in-app percentiles, not aggregatable); serve them at /metrics for Prometheus to scrape; keep label cardinality low.
  • Measure RED (Rate, Errors, Duration) for services and USE (Utilization, Saturation, Errors) for resources.
  • Traces with OpenTelemetry: one trace id, a tree of spans, carried across services by the W3C traceparent header (Inject / Extract); sample to save cost.
  • Health checks: cheap liveness (/healthz, restart me) vs readiness (/readyz, send me traffic; fails while starting and draining); expvar gives quick JSON at /debug/vars.

Next episode: Configuration and dependency injection: twelve-factor config, flags, environment and files, config validation, functional options, constructor injection, wiring in main.