These are the notes for Go episode 74. Last episode Raju packed his shop service into a 7.87 MB container image. Now it runs in the cloud, on Kubernetes, with a CPU limit and a memory limit, next to other people's services, and it gets stopped and replaced every time there is a release. Each of those facts breaks a naive Go program in a different way. We let each one fail for real, and invent the fix: GOMAXPROCS that matches the CPU limit, GOMEMLIMIT below the memory limit, probe endpoints, graceful shutdown on SIGTERM, state kept outside the replicas, config that fails fast, a Lambda handler, and finally the control loop that Kubernetes itself is built on.
All Go outputs below are from real runs (Go 1.24.7, linux/amd64) through the same runner the video's compiler uses (error streams are printed after normal output). The Docker runs are real too (Docker Engine 29.9.3, cgroup v1, a machine with 2 CPUs, 14 November 2026). There is no Kubernetes cluster in this sandbox: the Kubernetes YAML is shown and explained, never claimed to have run. Instead, a small Go helper plays the part of the kubelet (it starts the shop, probes it and sends SIGTERM, exactly as Kubernetes would), and another plays AWS Lambda's Runtime API. github.com/aws/aws-lambda-go v1.54.0 was cloned from GitHub into a local module proxy because proxy.golang.org is blocked here.
Facts and sources
- Go 1.25 release notes (tip.golang.org/doc/go1.25, "Container-aware GOMAXPROCS"): "In prior versions of Go, GOMAXPROCS defaults to the number of logical CPUs available at startup." "On Linux, the runtime considers the CPU bandwidth limit of the cgroup containing the process, if any." "In container runtime systems like Kubernetes, cgroup CPU bandwidth limits generally correspond to the 'CPU limit' option. The Go runtime does not consider the 'CPU requests' option." "On all OSes, the runtime periodically updates GOMAXPROCS if the number of logical CPUs available or the cgroup CPU bandwidth limit change." "Both of these behaviors are automatically disabled if GOMAXPROCS is set manually via the GOMAXPROCS environment variable or a call to runtime.GOMAXPROCS. They can also be disabled explicitly with the GODEBUG settings containermaxprocs=0 and updatemaxprocs=0, respectively." (This episode runs Go 1.24.7, so it shows the old behaviour.)
go doc runtime/debug SetMemoryLimit(Go 1.24.7): a "soft memory limit"; the runtime adjusts "the frequency of garbage collections" and returns "memory to the underlying system more aggressively"; it does not count "space used by the Go binary and memory external to Go", such as "memory allocated by C code"; "A zero limit or a limit that's lower than the amount of memory used by the Go runtime may cause the garbage collector to run nearly continuously"; the initial setting ismath.MaxInt64unlessGOMEMLIMITis set; suffixes B, KiB, MiB, GiB, TiB are powers of two.- Kubernetes docs (kubernetes/website on GitHub,
content/en/docs): "cpulimits are enforced by CPU throttling"; "memorylimits are enforced by the kernel with out of memory (OOM) kills"; "TheOOMKilledreason shows that the container tried to use more memory than its limit";0.5CPU can be written500m("five hundred millicpu"). Liveness probes "determine when to restart a container", for example to catch a deadlock; readiness probes "determine when a container is ready to accept traffic", and when one fails the Pod's IP is removed from the EndpointSlices of its Services; a startup probe holds back the other two until it succeeds. Pod termination: the Pod is taken out of the Service's endpoints, thepreStophook runs, "the container runtime ... send[s] a TERM signal to process 1 inside each container", the defaultterminationGracePeriodSecondsis 30 seconds, then the KILL signal. "ConfigMap does not provide secrecy or encryption." "Kubernetes Secrets are, by default, stored unencrypted in the API server's underlying data store." The client libraries page listsgithub.com/kubernetes/client-gofor Go. - Kubernetes operator pattern (kubernetes.io/docs/concepts/extend-kubernetes/operator): "Operators are software extensions to Kubernetes that make use of custom resources to manage applications and their components. Operators follow Kubernetes principles, notably the control loop." A controller is "a control loop that watches the shared state of the cluster through the apiserver and makes changes attempting to move the current state towards the desired state." Tools listed include kubebuilder and the Operator Framework.
- Google Cloud blog, "Kubernetes best practices: terminating with grace": the five steps of termination and "your application needs to handle the SIGTERM message and begin shutting down when it receives it".
- AWS Lambda docs ("Building Lambda functions with Go"): "The Go 1.x managed runtime for Lambda is deprecated"; use
provided.al2023(Amazon Linux 2023) orprovided.al2;github.com/aws/aws-lambda-go/lambdais "The implementation of the Lambda programming model for Go". aws-lambda-go README: withprovided,provided.al2orprovided.al2023"the executable within the .zip file should be namedbootstrap"; the default architecture is x86_64 (GOARCH=amd64), ARM needsGOARCH=arm64.lambda/rpc_function.go: "To drop the rpc dependencies, compile with-tags lambda.norpc". Lambda runtime API docs: theAWS_LAMBDA_RUNTIME_APIvariable,/2018-06-01/runtime/invocation/next, the headersLambda-Runtime-Aws-Request-IdandLambda-Runtime-Deadline-Ms, andPOST /runtime/invocation/{AwsRequestId}/response.
The problem: a shop in someone else's building
The cloud means renting someone else's computers by the hour. It's like renting desks in a co-working space instead of buying a building. Kubernetes is the space's manager: you tell it "keep three copies of my shop running", it places your containers on a group of machines (a cluster; each machine is a node) and restarts them when they crash.
| Kubernetes word | Plain meaning | Co-working example |
|---|---|---|
| Cluster | the group of machines Kubernetes manages | the building |
| Node | one machine in the cluster | one floor |
| Pod | the smallest thing Kubernetes runs: one or more containers that are scheduled together and share a network address | one team at one desk |
| Deployment | the instruction "run N Pods from this image, and replace them on each release" | "keep three desks filled with our team" |
| Service | a stable address that spreads traffic over the ready Pods | the reception desk that sends visitors to a free team |
| kubelet | the Kubernetes agent on each node that starts containers, runs probes and stops them | the floor manager |
| Request | how much CPU/memory the scheduler reserves for a container | "we usually need half a desk" |
| Limit | the most a container may use, enforced by the Linux kernel | "you get 1 desk, no more" |
Limits are enforced by cgroups (control groups), a Linux kernel feature that caps how much CPU and memory a group of processes can use. Docker uses the same mechanism, so docker run --cpus=1 --memory=64m reproduces a Pod's limits on a laptop.
CPU limits and GOMAXPROCS
GOMAXPROCS is how many operating system threads may run Go code at the same moment (the number of "P"s in Go's scheduler, episode 44). Up to Go 1.24 it defaults to the number of logical CPUs of the machine, whatever the container's limit. runtime.GOMAXPROCS(0) reads the current value without changing it.
CPU throttling: with a CPU limit of 1, the kernel gives the container one CPU's worth of time per scheduling period. Two busy threads use that up in half the period, and then the whole container is frozen until the next period starts. Every frozen moment is a request waiting.
The episode's program (CPU)
package main
import (
"crypto/sha256"
"fmt"
"os"
"runtime"
"sync"
"time"
)
func main() {
n := runtime.GOMAXPROCS(0) // 0 = just read the current value
fmt.Println("NumCPU:", runtime.NumCPU(), " GOMAXPROCS:", n)
start := time.Now()
var wg sync.WaitGroup
for range n { // one busy worker per P: hashing orders
wg.Add(1)
go func() {
defer wg.Done()
b := []byte("order")
for range 6_000_000 / n {
s := sha256.Sum256(b)
b = s[:]
}
}()
}
wg.Wait()
stat, _ := os.ReadFile("/sys/fs/cgroup/cpu/cpu.stat") // the kernel's CPU-limit counters
var periods, throttled int
fmt.Sscanf(string(stat), "nr_periods %d\nnr_throttled %d", &periods, &throttled)
fmt.Printf("work took %v, throttled in %d of %d periods\n", time.Since(start).Round(time.Millisecond), throttled, periods)
}
On the laptop (no limit):
NumCPU: 2 GOMAXPROCS: 2
work took 728ms, throttled in 0 of 0 periods
(/sys/fs/cgroup/cpu/cpu.stat is the cgroup v1 path used on this machine; on cgroup v2 the file is /sys/fs/cgroup/cpu.stat, and the throttled time is throttled_usec.)
In a container with one CPU (real)
$ docker run --rm --cpus=1 shop:cpu
NumCPU: 2 GOMAXPROCS: 2
work took 749ms, throttled in 7 of 8 periods
$ docker run --rm --cpus=1 -e GOMAXPROCS=1 shop:cpu
NumCPU: 2 GOMAXPROCS: 1
work took 730ms, throttled in 3 of 8 periods
With the default GOMAXPROCS of 2 the kernel throttled the container in almost every period. With GOMAXPROCS=1, matching the limit, far fewer periods were throttled. Not zero: one fully busy thread sits right at the edge of a one-CPU limit, so small extras still tip it over now and then. Four repeat runs of each gave 6-8 of 8-9 periods with the default and 1-2 of 8-10 with GOMAXPROCS=1; with a limit of 1.5 CPUs, GOMAXPROCS=1 was never throttled (0 of 8-9) while the default was throttled in 4-6 of 6-7. The total time is similar (one CPU's worth either way), but far less stop-and-go, which is what keeps request latency even. Leave some headroom in the CPU limit for the runtime's own work.
The rules
- Up to Go 1.24: set the
GOMAXPROCSenvironment variable to your CPU limit (rounded, at least 1). - Go 1.25 and later read the cgroup CPU limit themselves on Linux and update GOMAXPROCS if it changes. CPU requests are not considered. Setting GOMAXPROCS yourself turns the automatic behaviour off;
GODEBUG=containermaxprocs=0,updatemaxprocs=0turns it off explicitly. - A big node can have dozens of CPUs; a container limited to 1 or 2 CPUs with GOMAXPROCS equal to the node's count is throttled badly.
Memory limits and GOMEMLIMIT
Go's garbage collector (GC) frees memory the program no longer uses. GOGC (default 100, episode 56) lets the heap grow to about twice the live data before the next collection. That's fine on a laptop, but in a container with a hard memory limit, "twice the live data" can be more than the limit. Then the kernel's OOM killer (out-of-memory killer) stops the process with SIGKILL: no warning, no cleanup, no chance to finish orders. Docker and Kubernetes report this as OOMKilled, exit code 137 (128 + 9, and 9 is SIGKILL).
GOMEMLIMIT is a soft memory limit for the Go runtime: as memory approaches it, the GC runs more often and returns memory to the operating system sooner.
The episode's program (MEM)
package main
import (
"fmt"
"runtime"
"runtime/debug"
)
var cache [][]byte // live data the service keeps: 35 MB of menu and prices
func touch(b []byte) []byte { // write to every 4 KB page so it really uses RAM
for i := 0; i < len(b); i += 4096 {
b[i] = 1
}
return b
}
func main() {
fmt.Println("GOMEMLIMIT:", debug.SetMemoryLimit(-1)>>20, "MiB") // -1 = just read it
for range 35 {
cache = append(cache, touch(make([]byte, 1<<20)))
}
var keep []byte
for i := range 2000 { // handling orders: 1 MB of temporary garbage each
keep = touch(make([]byte, 1<<20))
keep[0] = byte(i)
}
var m runtime.MemStats
runtime.ReadMemStats(&m)
fmt.Printf("done: %d GC cycles, peak memory from the OS %d MiB\n", m.NumGC, m.Sys>>20)
}
On the laptop:
GOMEMLIMIT: 8796093022207 MiB
done: 58 GC cycles, peak memory from the OS 78 MiB
8796093022207 MiB is math.MaxInt64 shifted right by 20: "no limit". The GC count changes a little from run to run (58 or 59 here). Why touch? A fresh allocation from the operating system isn't really in RAM until it's written to; touching every page makes the program use memory the way real data does.
In a 64 MiB container (real)
$ docker run --name shop --memory=64m --memory-swap=64m shop:mem
GOMEMLIMIT: 8796093022207 MiB
$ docker inspect shop --format 'OOMKilled={{.State.OOMKilled}} ExitCode={{.State.ExitCode}}'
OOMKilled=true ExitCode=137
$ docker run --rm --memory=64m --memory-swap=64m -e GOMEMLIMIT=48MiB shop:mem
GOMEMLIMIT: 48 MiB
done: 256 GC cycles, peak memory from the OS 50 MiB
--memory-swap equal to --memory means no swap space, like a Kubernetes container. Without GOMEMLIMIT the program printed its first line and was killed: the "done" line never came. With GOMEMLIMIT=48MiB it finished, with many more GC cycles and only 50 MiB from the OS. More CPU spent collecting garbage, but alive.
GOMEMLIMIT, the rules
- Set it with the environment variable (
GOMEMLIMIT=48MiB) or in code (debug.SetMemoryLimit(48 << 20)). - Set it a little below the container limit: it doesn't count the binary itself, memory allocated by C code, or kernel memory held for the process.
- Units are powers of two: KiB = 1024 bytes, MiB = 1024 KiB.
- It can't create memory: if the live data alone is bigger than the limit, the GC may run nearly continuously. Then raise the limit or keep less live data.
- It works together with GOGC; some services use
GOGC=offplus GOMEMLIMIT so the GC only runs near the limit. Measure before you do this.
Liveness, readiness and startup probes
A probe is a check the kubelet runs on your container again and again.
| Probe | Question | When it fails | Sweet-shop example |
|---|---|---|---|
| Liveness | Is the process working, or stuck? | the container is restarted | the shopkeeper is asleep at the counter: replace him |
| Readiness | Can it take traffic right now? | the Pod is removed from the Service's endpoints (no restart) | the shutter is still down: send customers to another branch |
| Startup | Has it finished starting? | the container is killed and restarted; until it passes, the other probes don't run | the oven is still heating in the morning |
In Go each probe is an HTTP handler. Keep liveness cheap and local (don't check the database in it, or a database outage restarts every Pod). Readiness may check what the Pod needs to serve: caches loaded, not shutting down.
The obvious server and a helper that plays the kubelet (NAIVE)
// $ go build -o shop .
// $ go run ./kubelet
// ---- main.go ----
package main
import (
"fmt"
"net/http"
"sync/atomic"
"time"
)
var ready atomic.Bool // false until the menu cache is loaded
func healthz(w http.ResponseWriter, r *http.Request) { fmt.Fprint(w, "alive") } // liveness
func readyz(w http.ResponseWriter, r *http.Request) { // readiness
if !ready.Load() {
w.WriteHeader(http.StatusServiceUnavailable) // 503: no customers yet
}
fmt.Fprint(w, "ready=", ready.Load())
}
func order(w http.ResponseWriter, r *http.Request) {
time.Sleep(2 * time.Second) // packing an order takes 2 s
fmt.Fprint(w, "order #42 packed")
}
func main() {
http.HandleFunc("/healthz", healthz)
http.HandleFunc("/readyz", readyz)
http.HandleFunc("/order", order)
go func() { time.Sleep(time.Second); ready.Store(true) }() // load the cache, then ready
http.ListenAndServe(":8080", nil)
}
// ---- kubelet/main.go ----
package main // a tiny helper that plays Kubernetes: start the pod, probe it, SIGTERM it
import (
"fmt"
"io"
"net/http"
"os"
"os/exec"
"syscall"
"time"
)
func call(method, path string) string {
req, _ := http.NewRequest(method, "http://localhost:8080"+path, nil)
resp, err := http.DefaultClient.Do(req)
if err != nil {
return err.Error()
}
b, _ := io.ReadAll(resp.Body)
return fmt.Sprint(resp.StatusCode, " ", string(b))
}
func main() {
shop := exec.Command("./shop")
shop.Stdout = os.Stdout
shop.Start()
for r := ""; r != "200 ready=true"; {
time.Sleep(400 * time.Millisecond)
r = call("GET", "/readyz")
fmt.Println("probes: liveness", call("GET", "/healthz"), "| readiness", r)
}
result := make(chan string)
go func() { result <- call("POST", "/order") }() // a customer's 2-second order
time.Sleep(500 * time.Millisecond)
fmt.Println("kubelet: replacing the pod, sending SIGTERM")
shop.Process.Signal(syscall.SIGTERM)
fmt.Println("customer got:", <-result)
fmt.Println("shop exited:", shop.Wait())
}
Output:
$ go build -o shop .
$ go run ./kubelet
probes: liveness 200 alive | readiness 503 ready=false
probes: liveness 200 alive | readiness 503 ready=false
probes: liveness 200 alive | readiness 200 ready=true
kubelet: replacing the pod, sending SIGTERM
customer got: Post "http://localhost:8080/order": EOF
shop exited: signal: terminated
The probes work, but SIGTERM in the middle of an order kills the process at once (that is Go's default action for SIGTERM), so the customer gets EOF, "end of file": the connection closed before any answer. The order is lost. Why POST? Go's HTTP client may quietly retry an idempotent GET on a new connection when a reused connection closes; a POST is not retried, so the real failure is visible.
SIGTERM and graceful shutdown
How Kubernetes stops a Pod (from the Pod lifecycle docs and the Google Cloud blog):
- The Pod is marked Terminating and removed from the endpoints of its Services: new traffic stops coming.
- The
preStophook runs, if you configured one. - The TERM signal (SIGTERM, "please stop") is sent to process 1 in each container.
- Kubernetes waits for
terminationGracePeriodSeconds, 30 seconds by default (steps 2-4 overlap). - Anything still running gets SIGKILL, which cannot be caught.
Pods are stopped like this on every release, every scale-down and every node drain, so this is not a rare event.
Graceful shutdown means: on SIGTERM, stop taking new work, finish the work in progress, release resources, and exit by yourself before the grace period ends.
The fix (GRACEFUL)
Same handlers and the same helper; only main changes:
// $ go build -o shop .
// $ go run ./kubelet
// ---- main.go ----
package main
import (
"context"
"fmt"
"net/http"
"os/signal"
"sync/atomic"
"syscall"
"time"
)
// healthz, readyz, order and the ready flag: exactly as above
func main() {
http.HandleFunc("/healthz", healthz)
http.HandleFunc("/readyz", readyz)
http.HandleFunc("/order", order)
go func() { time.Sleep(time.Second); ready.Store(true) }()
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM) // SIGTERM cancels ctx
defer stop()
srv := &http.Server{Addr: ":8080"}
go srv.ListenAndServe()
<-ctx.Done() // wait here until Kubernetes says "stop"
ready.Store(false) // 1. fail readiness: no new customers
fmt.Println("shop: got SIGTERM, finishing open orders")
timeout, cancel := context.WithTimeout(context.Background(), 20*time.Second) // < grace period
defer cancel()
err := srv.Shutdown(timeout) // 2. stop listening, wait for open requests to finish
fmt.Println("shop: all orders done, exiting. Shutdown error:", err)
}
// kubelet/main.go: exactly as above
Output:
$ go build -o shop .
$ go run ./kubelet
probes: liveness 200 alive | readiness 503 ready=false
probes: liveness 200 alive | readiness 503 ready=false
probes: liveness 200 alive | readiness 200 ready=true
kubelet: replacing the pod, sending SIGTERM
shop: got SIGTERM, finishing open orders
customer got: 200 order #42 packed
shop: all orders done, exiting. Shutdown error: <nil>
shop exited: <nil>
signal.NotifyContext(parent, signals...)returns a context that is cancelled when one of the signals arrives, and astopfunction that restores the default behaviour.http.Server.Shutdown(ctx)closes the listeners (no new connections), closes idle connections, and waits for active requests to finish, or forctxto expire (then it returns the context's error).- The Shutdown timeout must be shorter than the grace period, leaving time for the rest of the cleanup; otherwise SIGKILL arrives first.
The checklist
- Catch SIGTERM (and
os.Interruptfor Ctrl+C on a laptop). - Fail readiness first, so the endpoint list stops sending traffic.
Shutdownwith a timeout belowterminationGracePeriodSeconds.- Then close database pools, flush logs and metrics, stop background workers (cancel their context, wait for them).
- Make sure your binary is process 1 in the container (exec form
ENTRYPOINT ["/shop"], episode 73). A shell started assh -c ./shopmay receive the signal and not pass it on. docker stopsends a stop signal and waits before killing (-t, --timeout"Seconds to wait before killing the container"), so you can test all of this locally.
Stateless services and horizontal scaling
Horizontal scaling means running more copies (replicas) of the same service behind a load balancer, instead of buying one bigger machine (vertical scaling). A load balancer spreads requests over the replicas; round-robin sends them to replica 0, 1, 0, 1 in turn. A stateless service keeps nothing important in its own memory between requests, so any replica can serve any request and Kubernetes can add, remove or replace replicas freely.
The episode's program (STATE)
package main
import "fmt"
type Store struct { // where a replica keeps the carts
carts map[string][]string
}
func NewStore() *Store { return &Store{carts: map[string][]string{}} }
func shop(replicas []*Store) {
next := 0
lb := func() (int, *Store) { // round-robin load balancer: replica 0, 1, 0, 1...
i := next % len(replicas)
next++
return i, replicas[i]
}
for _, item := range []string{"chai", "samosa", "jalebi"} {
i, r := lb()
r.carts["asha"] = append(r.carts["asha"], item)
fmt.Printf(" add %-7s -> replica %d\n", item, i)
}
i, r := lb()
fmt.Printf(" show cart -> replica %d: %v\n", i, r.carts["asha"])
}
func main() {
fmt.Println("2 replicas, each with its own memory:")
shop([]*Store{NewStore(), NewStore()})
fmt.Println("2 replicas, one shared store (think Redis or a database):")
shared := NewStore()
shop([]*Store{shared, shared})
}
Output:
2 replicas, each with its own memory:
add chai -> replica 0
add samosa -> replica 1
add jalebi -> replica 0
show cart -> replica 1: [samosa]
2 replicas, one shared store (think Redis or a database):
add chai -> replica 0
add samosa -> replica 1
add jalebi -> replica 0
show cart -> replica 1: [chai samosa jalebi]
The program simulates the replicas and the load balancer in one process (no mutex is needed because only one goroutine touches the stores; real replicas serve requests concurrently, so a real store must be safe for concurrent use) so you can see both stores side by side; with real Pods the effect is the same. Things that are commonly (and wrongly) kept in a replica's memory: sessions and carts, rate-limit counters, uploaded files on local disk, "run this job once" flags. Move them to a database, a cache server, object storage, or a queue. In-memory caches of data that lives elsewhere are fine: losing them only costs a reload.
ConfigMaps, Secrets and the Deployment
A ConfigMap holds configuration outside the code (the database host, feature flags). A Secret holds small sensitive data (passwords, API keys, certificates). Both reach the container as environment variables or as files in a mounted volume.
- A ConfigMap "does not provide secrecy or encryption": never put passwords in one.
- Secrets "are, by default, stored unencrypted" in the cluster's data store: the cluster admin should turn on encryption at rest and restrict who can read Secrets (RBAC).
- Prefer Secrets as files: environment variables are easy to leak (crash dumps,
/proc, child processes, debug endpoints). Mounted Secret files are also updated in place when the Secret changes; environment variables never are.
The Deployment (shown, not run: no cluster here)
apiVersion: apps/v1
kind: Deployment
metadata: { name: shop }
spec:
replicas: 3
template:
spec:
terminationGracePeriodSeconds: 30
containers:
- name: shop
image: shop:1.4.2
resources:
requests: { cpu: 500m, memory: 64Mi }
limits: { cpu: "1", memory: 64Mi }
env:
- { name: GOMAXPROCS, value: "1" }
- { name: GOMEMLIMIT, value: 48MiB }
envFrom: [ { configMapRef: { name: shop-config } } ]
livenessProbe: { httpGet: { path: /healthz, port: 8080 } }
readinessProbe: { httpGet: { path: /readyz, port: 8080 } }
volumeMounts: [ { name: secrets, mountPath: /etc/shop } ]
volumes: [ { name: secrets, secret: { secretName: shop-db } } ]
(For brevity the video's version leaves out the selector and Pod labels that a real Deployment also needs, e.g. selector: { matchLabels: { app: shop } } and metadata: { labels: { app: shop } } under template.)
Reading config in Go (CONFIG)
// $ go run .
// $ DB_HOST=postgres.shop.svc PAYMENT_MODE=live SECRETS_DIR=secrets go run .
// ---- main.go ----
package main
import (
"fmt"
"os"
"path/filepath"
"strings"
)
type Config struct{ DBHost, PaymentMode, DBPassword string }
func load() (Config, error) {
c := Config{DBHost: os.Getenv("DB_HOST"), PaymentMode: os.Getenv("PAYMENT_MODE")} // from a ConfigMap
if c.DBHost == "" {
return c, fmt.Errorf("config: DB_HOST is not set")
}
pw, err := os.ReadFile(filepath.Join(os.Getenv("SECRETS_DIR"), "db-password")) // a Secret, mounted as a file
if err != nil {
return c, fmt.Errorf("config: %w", err)
}
c.DBPassword = strings.TrimSpace(string(pw))
return c, nil
}
func main() {
c, err := load()
if err != nil {
fmt.Println(err) // fail fast at start-up, not at the first order
os.Exit(1)
}
fmt.Printf("db=%s payments=%s password=%d chars (never log the secret itself)\n", c.DBHost, c.PaymentMode, len(c.DBPassword))
}
// ---- secrets/db-password ----
s3cr3t-chai-42
Output:
$ go run .
config: DB_HOST is not set
$ DB_HOST=postgres.shop.svc PAYMENT_MODE=live SECRETS_DIR=secrets go run .
db=postgres.shop.svc payments=live password=14 chars (never log the secret itself)
exit status 1
(exit status 1 belongs to the first command; the runner prints error streams last.) Fail fast: check config at start-up so a missing value shows up at once in the Pod's status and logs (the Pod never becomes ready, the rollout stops), instead of at the first customer's order.
Serverless Go on AWS Lambda
Serverless means you hand the cloud a function and it runs it only when an event arrives, billing per use; you don't keep a server running. Like booking a cab per ride instead of owning a car. AWS Lambda is Amazon's serverless service. An event is what triggers a run (an HTTP call through an API gateway, a file upload, a queue message); the handler is your function that handles one event.
The episode's program (LAMBDA)
// $ GOOS=linux GOARCH=arm64 CGO_ENABLED=0 go build -tags lambda.norpc -o bootstrap-arm64 .
// $ go build -tags lambda.norpc -o bootstrap .
// $ ./bootstrap
// $ go run ./fakelambda
// ---- main.go ----
package main
import (
"context"
"fmt"
"github.com/aws/aws-lambda-go/lambda"
)
type Order struct {
Item string `json:"item"`
Qty int `json:"qty"`
}
func handle(ctx context.Context, o Order) (string, error) { // runs once per event
return fmt.Sprintf("billed %d x %s = Rs %d", o.Qty, o.Item, o.Qty*15), nil
}
func main() {
lambda.Start(handle) // ask the Runtime API for events, forever
}
// ---- go.mod ----
module shop
go 1.24
require github.com/aws/aws-lambda-go v1.54.0
// ---- fakelambda/main.go ----
package main // a tiny stand-in for AWS's Runtime API, to watch the protocol locally
import (
"fmt"
"io"
"net/http"
"os"
"os/exec"
)
func main() {
http.HandleFunc("/2018-06-01/runtime/invocation/next", func(w http.ResponseWriter, r *http.Request) {
fmt.Println("runtime asked: GET /runtime/invocation/next")
w.Header().Set("Lambda-Runtime-Aws-Request-Id", "req-1")
w.Header().Set("Lambda-Runtime-Deadline-Ms", "4102444800000")
fmt.Fprint(w, `{"item":"chai","qty":3}`)
})
http.HandleFunc("/2018-06-01/runtime/invocation/req-1/response", func(w http.ResponseWriter, r *http.Request) {
b, _ := io.ReadAll(r.Body)
fmt.Println("runtime answered: POST .../req-1/response", string(b))
os.Exit(0)
})
go http.ListenAndServe("127.0.0.1:9001", nil)
fn := exec.Command("./bootstrap")
fn.Env = append(os.Environ(), "AWS_LAMBDA_RUNTIME_API=127.0.0.1:9001")
fn.Run()
}
Output:
$ GOOS=linux GOARCH=arm64 CGO_ENABLED=0 go build -tags lambda.norpc -o bootstrap-arm64 .
$ go build -tags lambda.norpc -o bootstrap .
$ ./bootstrap
$ go run ./fakelambda
runtime asked: GET /runtime/invocation/next
runtime answered: POST .../req-1/response "billed 3 x chai = Rs 45"
2026/11/14 11:45:18 expected AWS Lambda environment variables [AWS_LAMBDA_RUNTIME_API] are not defined
(The last line is the error stream of ./bootstrap, printed last by the runner.) A Lambda function doesn't listen on a port. Lambda starts bootstrap with AWS_LAMBDA_RUNTIME_API set; the library calls GET /2018-06-01/runtime/invocation/next, gets the event and a request ID, runs your handler, posts the result to /runtime/invocation/{request id}/response, and asks for the next event. The stand-in only answers one event and exits, which is enough to watch the protocol.
Shipping it
GOOS=linux GOARCH=arm64 CGO_ENABLED=0 go build -tags lambda.norpc -o bootstrap .
zip lambda-handler.zip bootstrap
- Runtime:
provided.al2023(Amazon Linux 2023) orprovided.al2. The oldgo1.xmanaged runtime is deprecated. - The executable in the .zip must be named
bootstrap. - Default architecture x86_64 →
GOARCH=amd64; for ARM functionsGOARCH=arm64. CGO_ENABLED=0gives a static binary, so it doesn't depend on the glibc version of the Lambda image (the README warns aboutGLIBC_X.YZ' not founderrors otherwise).-tags lambda.norpcdrops the RPC code only the oldgo1.xruntime used.- Do expensive setup (database clients, config) once, outside the handler (in
mainor package variables), so it is reused across events in the same running instance.
Operators and client-go
Kubernetes is built from control loops. A controller watches the desired state (what you asked for, the spec) and the actual state (what is really running), and acts to make actual match desired; this is called reconciling. Like an air conditioner: you set 24 °C, the room is 29 °C, it cools and checks again, forever.
A custom resource is your own object type added to the Kubernetes API (for example kind: ShopDB). An operator is a controller for a custom resource: it encodes what a human operator knows (deploying, backups, upgrades, failover) as code.
A tiny control loop (LOOP)
package main
import (
"fmt"
"slices"
)
type ShopSpec struct{ Replicas int } // desired state: what the user asked for (a custom resource)
var pods []string // actual state: what is really running
var nextID = 1
func reconcile(want ShopSpec) { // one turn of the control loop
for len(pods) < want.Replicas {
pods = append(pods, fmt.Sprintf("shop-%d", nextID))
nextID++
fmt.Println(" created", pods[len(pods)-1])
}
for len(pods) > want.Replicas {
fmt.Println(" deleted", pods[len(pods)-1])
pods = pods[:len(pods)-1]
}
}
func main() {
want := ShopSpec{Replicas: 3}
for tick := 1; tick <= 4; tick++ {
switch tick {
case 2:
pods = slices.Delete(pods, 0, 1) // a node fails: shop-1 is gone
fmt.Println(" (shop-1 crashed)")
case 3:
want.Replicas = 2 // Diwali sale is over: the user edits the spec
fmt.Println(" (spec changed to 2 replicas)")
}
fmt.Printf("tick %d: want %d, have %v\n", tick, want.Replicas, pods)
reconcile(want)
}
fmt.Println("final:", pods)
}
Output:
tick 1: want 3, have []
created shop-1
created shop-2
created shop-3
(shop-1 crashed)
tick 2: want 3, have [shop-2 shop-3]
created shop-4
(spec changed to 2 replicas)
tick 3: want 2, have [shop-2 shop-3 shop-4]
deleted shop-4
tick 4: want 2, have [shop-2 shop-3]
final: [shop-2 shop-3]
The loop never asks "what happened?", only "what is different now?". That is why it heals crashes and handles spec changes with the same code, and why it's safe to run it again and again (idempotent).
In a real operator
- client-go is the official Go client library for the Kubernetes API (
github.com/kubernetes/client-go). It provides typed clients, and informers: a local cache of objects kept fresh by a watch, so controllers don't hammer the API server. - kubebuilder and the Operator Framework generate the project, the custom resource definition (CRD) from Go structs, and the controller skeleton; you write the
Reconcilefunction. - A reconcile function reads the custom resource, compares it with the objects it owns (Deployments, Services, Jobs), creates or updates them, writes a
status, and returns, often asking to be called again later.
Go in the cloud in one page
| Problem | What you saw | Fix |
|---|---|---|
| CPU limit < machine CPUs | throttled in almost every period | GOMAXPROCS = limit (Go ≤ 1.24); automatic in Go 1.25+ |
| Memory limit | OOMKilled=true ExitCode=137 |
GOMEMLIMIT a little below the limit |
| Is it alive? Ready? | probes | /healthz (cheap), /readyz (503 until ready, and while draining) |
| Pod replaced | customer got EOF | signal.NotifyContext + readiness false + Server.Shutdown(timeout < grace) |
| More replicas | cart shows [samosa] |
stateless replicas + shared store |
| Config and passwords | fails at the first order | ConfigMap → env, Secret → file, validate at start-up |
| Event jobs | — | Lambda: lambda.Start, bootstrap, provided.al2023 |
| Automation | — | control loops; operators with client-go / kubebuilder |
Mini project: a cloud-ready shop
Make Raju's shop cloud-ready:
/healthzand/readyz; readiness false until the cache loads.- SIGTERM → not ready →
Shutdown(20 s)→ close the store. - Carts behind a
Storeinterface, not a package-level map. - Config from environment variables plus a secret file; fail fast; log only the secret's length.
- Test it in Docker with
--cpus=1,--memory=64m, anddocker stop.
Full solution (main.go)
// Cloud-ready shop: probes, graceful shutdown, a Store interface, config that fails fast.
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"log"
"net/http"
"os"
"os/signal"
"path/filepath"
"strings"
"sync"
"sync/atomic"
"syscall"
"time"
)
type Config struct {
Addr, DBHost, DBPassword string
}
func loadConfig() (Config, error) {
c := Config{Addr: os.Getenv("ADDR"), DBHost: os.Getenv("DB_HOST")}
if c.Addr == "" {
c.Addr = ":8080"
}
if c.DBHost == "" {
return c, errors.New("config: DB_HOST is not set")
}
dir := os.Getenv("SECRETS_DIR")
if dir == "" {
dir = "/etc/shop"
}
pw, err := os.ReadFile(filepath.Join(dir, "db-password"))
if err != nil {
return c, fmt.Errorf("config: %w", err)
}
c.DBPassword = strings.TrimSpace(string(pw))
return c, nil
}
// Store is where carts live. In production: a database or Redis, shared by all replicas.
type Store interface {
Add(user, item string) error
Cart(user string) ([]string, error)
Close() error
}
type MemStore struct {
mu sync.Mutex
carts map[string][]string
}
func NewMemStore() *MemStore { return &MemStore{carts: map[string][]string{}} }
func (s *MemStore) Add(u, item string) error {
s.mu.Lock()
defer s.mu.Unlock()
s.carts[u] = append(s.carts[u], item)
return nil
}
func (s *MemStore) Cart(u string) ([]string, error) {
s.mu.Lock()
defer s.mu.Unlock()
return append([]string(nil), s.carts[u]...), nil
}
func (s *MemStore) Close() error { return nil }
type Server struct {
store Store
ready atomic.Bool
}
func (s *Server) routes() *http.ServeMux {
mux := http.NewServeMux()
mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) { fmt.Fprint(w, "alive") })
mux.HandleFunc("GET /readyz", func(w http.ResponseWriter, r *http.Request) {
if !s.ready.Load() {
http.Error(w, "not ready", http.StatusServiceUnavailable)
return
}
fmt.Fprint(w, "ready")
})
mux.HandleFunc("POST /cart/{user}/{item}", func(w http.ResponseWriter, r *http.Request) {
time.Sleep(500 * time.Millisecond) // pretend work
if err := s.store.Add(r.PathValue("user"), r.PathValue("item")); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
fmt.Fprintln(w, "added", r.PathValue("item"))
})
mux.HandleFunc("GET /cart/{user}", func(w http.ResponseWriter, r *http.Request) {
items, err := s.store.Cart(r.PathValue("user"))
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
json.NewEncoder(w).Encode(items)
})
return mux
}
func main() {
cfg, err := loadConfig()
if err != nil {
log.Fatal(err) // fail fast: Kubernetes shows this in the Pod's logs at once
}
log.Printf("config: db=%s password=%d chars", cfg.DBHost, len(cfg.DBPassword))
s := &Server{store: NewMemStore()}
srv := &http.Server{Addr: cfg.Addr, Handler: s.routes(), ReadHeaderTimeout: 5 * time.Second}
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, os.Interrupt)
defer stop()
go func() {
if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
log.Fatal(err)
}
}()
time.Sleep(time.Second) // warm caches here
s.ready.Store(true)
log.Println("ready on", cfg.Addr)
<-ctx.Done()
s.ready.Store(false)
log.Println("SIGTERM: draining")
shutdownCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
if err := srv.Shutdown(shutdownCtx); err != nil {
log.Println("shutdown:", err)
}
if err := s.store.Close(); err != nil {
log.Println("store close:", err)
}
log.Println("bye")
}
A real run (Go 1.24.7): first without config, then with config, a cart request in flight while SIGTERM arrives:
$ ./shop
2026/11/14 11:46:25 config: DB_HOST is not set
$ echo $?
1
$ DB_HOST=db SECRETS_DIR=secrets ADDR=:18090 ./shop &
2026/11/14 11:46:25 config: db=db password=14 chars
2026/11/14 11:46:26 ready on :18090
$ curl -s localhost:18090/readyz
ready
$ curl -s -X POST localhost:18090/cart/asha/chai & sleep 0.1; kill -TERM %1
2026/11/14 11:46:26 SIGTERM: draining
added chai
2026/11/14 11:46:27 bye
The cart request that was in flight when SIGTERM arrived still got added chai. For the Docker test, build it with CGO_ENABLED=0, put it in a scratch image (episode 73), mount a secrets folder, and run docker run --cpus=1 --memory=64m -e GOMAXPROCS=1 -e GOMEMLIMIT=48MiB -e DB_HOST=db -e SECRETS_DIR=/etc/shop -v $PWD/secrets:/etc/shop:ro ..., then docker stop it while a request is open.
Common mistakes (and the real messages)
| Mistake | What you see | Why | Fix |
|---|---|---|---|
| GOMAXPROCS = node CPUs (Go ≤ 1.24) | throttled in 7 of 8 periods |
more busy threads than the CPU limit | GOMAXPROCS = limit, or Go 1.25+ |
| No GOMEMLIMIT in a small container | OOMKilled=true ExitCode=137 |
GOGC lets the heap reach ~2× live data | GOMEMLIMIT a little below the limit |
| GOMEMLIMIT below the live data | GC runs nearly non-stop, high CPU | the limit is soft; live data can't be freed | raise the limit or keep less live data |
| No SIGTERM handling | customer got: Post "http://localhost:8080/order": EOF |
Go's default action for SIGTERM is to exit | signal.NotifyContext + Server.Shutdown |
| Shutdown timeout ≥ grace period | killed mid-drain (exit 137) | SIGKILL comes at the end of the grace period | timeout < terminationGracePeriodSeconds |
| App started by a shell | SIGTERM never reaches the app | the shell is process 1 | exec-form ENTRYPOINT ["/shop"] |
| Database check in the liveness probe | every Pod restarts when the DB blips | liveness failure = restart | keep liveness local; use readiness for dependencies |
| State in a replica's memory | show cart -> replica 1: [samosa] |
requests land on other replicas | a shared store |
| Password in a ConfigMap | anyone who can read config sees it | ConfigMaps have no secrecy | a Secret (with encryption at rest) |
| Logging secrets | the password in your log system | logs are copied everywhere | log only that it's set, or its length |
Lambda binary named main |
Lambda can't start the function | provided runtimes run bootstrap |
go build -o bootstrap |
| Running a Lambda binary locally | expected AWS Lambda environment variables [AWS_LAMBDA_RUNTIME_API] are not defined |
it needs the Runtime API | test the handler function directly, or use an emulator |
Practice
- Run the CPU program in Docker with
--cpus=0.5. What GOMAXPROCS do you get, and what happens to the throttled periods withGOMAXPROCS=1? - Change the memory program's cache to 60 MB with
--memory=64mandGOMEMLIMIT=48MiB. What happens, and why? - In the graceful server, change the Shutdown timeout to 1 second (the order takes 2). What do the customer and
Shutdownreport? - Add a startup probe endpoint
/startupzthat returns 200 once, after the cache loads, and write the YAML for it. - Write a
reconcilethat also handles a "version" field: Pods running an old version are replaced one at a time.
Solutions
- GOMAXPROCS is still 2 on Go 1.24 (the machine has 2 CPUs). With half a CPU, even one busy thread needs more than the quota, so the container is throttled in almost every period either way. A real run:
$ docker run --rm --cpus=0.5 shop:cpu
NumCPU: 2 GOMAXPROCS: 2
work took 1.501s, throttled in 15 of 16 periods
$ docker run --rm --cpus=0.5 -e GOMAXPROCS=1 shop:cpu
NumCPU: 2 GOMAXPROCS: 1
work took 1.524s, throttled in 16 of 16 periods
Under full load, a limit below one CPU means throttling no matter what; GOMAXPROCS can only keep it from getting worse with more threads. The real fixes are a bigger limit or less work per request. 2. The live data (60 MB) is above GOMEMLIMIT (48 MiB), so the GC can't get under the limit; the process crosses 64 MiB and is OOM killed anyway. A real run:
$ docker run --name p2 --memory=64m --memory-swap=64m -e GOMEMLIMIT=48MiB shop:mem60
GOMEMLIMIT: 48 MiB
$ docker inspect p2 --format 'OOMKilled={{.State.OOMKilled}} ExitCode={{.State.ExitCode}}'
OOMKilled=true ExitCode=137
GOMEMLIMIT can't free live data: raise the limit or shrink the cache.
3. Shutdown returns context deadline exceeded after 1 second, and main returns, so the process exits while the order is still open: the customer gets EOF again. A real run:
shop: got SIGTERM, finishing open orders
shop: all orders done, exiting. Shutdown error: context deadline exceeded
customer got: Post "http://localhost:8080/order": EOF
shop exited: <nil>
The timeout must cover your slowest normal request (and the message should not claim "all orders done" when Shutdown returned an error).
4. Handler: http.HandleFunc("/startupz", func(w http.ResponseWriter, r *http.Request) { if !started.Load() { w.WriteHeader(503) }; fmt.Fprint(w, "started=", started.Load()) }). YAML: startupProbe: { httpGet: { path: /startupz, port: 8080 }, failureThreshold: 30, periodSeconds: 2 } (up to 60 s to start).
5. One way:
type Pod struct{ Name, Version string }
type ShopSpec struct {
Replicas int
Version string
}
func reconcile(want ShopSpec) {
for i, p := range pods { // replace at most one old Pod per turn: a rolling update
if p.Version != want.Version {
pods[i] = Pod{fmt.Sprintf("shop-%d", nextID), want.Version}
nextID++
fmt.Println(" replaced", p.Name, "->", pods[i].Name, want.Version)
return
}
}
for len(pods) < want.Replicas {
pods = append(pods, Pod{fmt.Sprintf("shop-%d", nextID), want.Version})
nextID++
}
for len(pods) > want.Replicas {
pods = pods[:len(pods)-1]
}
}
Quick quiz
// pod: limits memory 64Mi, terminationGracePeriodSeconds: 30
func main() {
http.HandleFunc("/order", order) // takes 2 s
log.Fatal(http.ListenAndServe(":8080", nil))
}
// Kubernetes sends SIGTERM during an order
What does the customer get?
A. The order finishes, then it exits · B. Kubernetes waits 30 s, then the order finishes · C. The connection drops: EOF · D. SIGTERM is ignored
Answer
C. Without signal handling, Go's default action for SIGTERM is to exit at once, so the open connection closes with no response (we saw customer got: ... EOF). The grace period only helps a program that catches SIGTERM and drains: signal.NotifyContext + http.Server.Shutdown.
Interview questions
- Why can a Go service be slow in a container with a CPU limit, and how do you fix it? Up to Go 1.24, GOMAXPROCS defaults to the machine's CPU count, so more threads run than the CPU limit allows and the kernel throttles the cgroup, freezing it until the next period. Set GOMAXPROCS to the limit; Go 1.25+ does this automatically on Linux from the cgroup CPU limit (not the request).
- What is GOMEMLIMIT and when would you set it? A soft memory limit for the Go runtime. Near it the GC runs more often and returns memory sooner. Set it a little below a container's memory limit so the heap's normal growth under GOGC doesn't cross the limit and trigger an OOM kill.
- What does exit code 137 mean? 128 + 9: the process was killed by SIGKILL, in containers usually by the OOM killer (
OOMKilled) or at the end of the termination grace period. - Liveness vs readiness vs startup probe? Liveness failure restarts the container; readiness failure removes the Pod from Service endpoints without restarting it; startup holds back the other probes until the app has started.
- How do you shut down an HTTP server gracefully in Go?
signal.NotifyContextfor SIGTERM, fail readiness,srv.Shutdown(ctx)with a timeout shorter than the grace period, then close other resources. - What does
http.Server.Shutdowndo? Closes listeners, closes idle connections, waits for active connections to become idle, or until the context expires. After it is called,ListenAndServereturnshttp.ErrServerClosed. - What makes a service stateless, and why does Kubernetes care? It keeps no important data in its own memory or disk between requests, so any replica can serve any request and replicas can be added, removed or replaced at any time.
- ConfigMap vs Secret? Both carry configuration as env vars or files; Secrets are for sensitive data and can be encrypted at rest and access-restricted, ConfigMaps have no secrecy. Neither is encrypted by default.
- How is a Go program deployed to AWS Lambda today? Built for Linux (amd64 or arm64), usually with
CGO_ENABLED=0and-tags lambda.norpc, namedbootstrap, zipped, on theprovided.al2023runtime; the handler is registered withlambda.Startfromaws-lambda-go. - What is a Kubernetes operator? A controller for a custom resource: a control loop that reconciles the actual state of an application with the desired state in its custom resource, encoding operational knowledge such as backups and upgrades. In Go it's built with client-go, usually through kubebuilder or the Operator Framework.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Cloud | renting someone else's computers, storage and services, paying for what you use | running the shop on rented servers instead of your own |
| Container | a packed file system with a program and everything it needs, run isolated from others | the 7.87 MB shop:1.4.2 image from episode 73 |
| Kubernetes | a system that runs containers on a cluster of machines and keeps the number you ask for running | "keep 3 shop Pods running" |
| Cluster | the group of machines Kubernetes manages | 10 servers in a data centre |
| Node | one machine in the cluster | a server with 64 CPUs |
| Pod | the smallest thing Kubernetes runs: one or more containers scheduled together | one shop container |
| Deployment | Kubernetes object that runs N identical Pods and replaces them on updates | replicas: 3 |
| Service | a stable name and address that spreads traffic over ready Pods | shop.default.svc |
| EndpointSlice | the list of Pod addresses behind a Service | ready Pods only |
| kubelet | the Kubernetes agent on each node: starts containers, runs probes, sends stop signals | the helper in the video plays it |
| cgroups | Linux control groups: a kernel feature that limits CPU and memory for a group of processes | docker run --cpus=1 |
| Request (resources) | the CPU/memory reserved for a container when scheduling | cpu: 500m = half a CPU |
| Limit (resources) | the most CPU/memory a container may use, enforced by the kernel | memory: 64Mi |
| millicpu (m) | a thousandth of a CPU | 500m = 0.5 CPU |
| CPU throttling | the kernel pausing a cgroup that used its CPU quota until the next period | throttled in 6 of 8 periods |
| GOMAXPROCS | how many OS threads may run Go code at the same time | GOMAXPROCS=1 |
| Logical CPU | a CPU the operating system can schedule on (a core or a hardware thread) | runtime.NumCPU() = 2 |
| Garbage collector (GC) | the part of the Go runtime that frees memory no longer used | 256 GC cycles |
| GOGC | how much the heap may grow before the next GC, in percent of live data (default 100) | heap ≈ 2× live data |
| Live data | memory still reachable and in use | the 35 MB menu cache |
| Heap | memory for values whose lifetime the compiler can't bound; managed by the GC | make([]byte, 1<<20) |
| GOMEMLIMIT | a soft memory limit for the Go runtime; the GC works harder near it | GOMEMLIMIT=48MiB |
| Soft limit | a target the runtime tries to respect, not a hard wall | Go may still exceed it with too much live data |
| MiB, KiB | powers-of-two units: KiB = 1024 bytes, MiB = 1024 KiB | 64 MiB |
| OOM killer | the kernel feature that kills a process when a memory limit is exceeded | OOMKilled=true |
| Exit code 137 | 128 + 9: killed by SIGKILL | ExitCode=137 |
| Crash loop | a container that keeps crashing and being restarted (CrashLoopBackOff in Kubernetes) |
OOM killed on every start |
| Probe | a periodic check the kubelet runs on a container | httpGet /readyz |
| Liveness probe | "is it stuck?"; failure restarts the container | /healthz |
| Readiness probe | "can it take traffic now?"; failure removes it from the Service | /readyz returning 503 |
| Startup probe | "has it finished starting?"; other probes wait for it | a slow cache load |
| 503 Service Unavailable | HTTP status meaning "can't serve right now" | readiness while warming up |
| atomic.Bool | a boolean that many goroutines can read and write safely | the ready flag |
| Signal | a notification the OS sends to a process | SIGTERM, SIGKILL |
| SIGTERM | "please stop": can be caught and handled | sent when a Pod is replaced |
| SIGKILL | "stop now": cannot be caught | after the grace period |
| Process 1 (PID 1) | the first process in a container; it receives the stop signal | ENTRYPOINT ["/shop"] |
| preStop hook | a command or HTTP call Kubernetes runs before sending SIGTERM | sleep 5 to let endpoints update |
| Grace period | time between SIGTERM and SIGKILL (terminationGracePeriodSeconds, default 30 s) |
terminationGracePeriodSeconds: 30 |
| Graceful shutdown | stop taking work, finish open work, clean up, exit by yourself | srv.Shutdown(ctx) |
| signal.NotifyContext | returns a context cancelled when a signal arrives | signal.NotifyContext(ctx, syscall.SIGTERM) |
| http.Server.Shutdown | stops listening and waits for open requests | Shutdown error: <nil> |
| EOF | "end of file": a connection closed before the expected data arrived | customer got: ... EOF |
| Horizontal scaling | adding more copies of a service | 2 → 5 replicas on Diwali |
| Vertical scaling | giving one copy a bigger machine | 2 → 8 CPUs |
| Replica | one copy of a service | shop-2 |
| Load balancer | spreads requests over replicas | the Service, a cloud load balancer |
| Round-robin | send requests to replicas in turn | 0, 1, 0, 1 |
| Stateless | keeps nothing important in its own memory between requests | carts in a shared store |
| Shared store | storage all replicas use | a database, Redis |
| ConfigMap | Kubernetes object holding non-secret configuration | DB_HOST=postgres.shop.svc |
| Secret | Kubernetes object holding small sensitive data | db-password |
| Encryption at rest | encrypting stored data | turning it on for Secrets |
| Environment variable | a named value the OS passes to a process | os.Getenv("DB_HOST") |
| Fail fast | stop at start-up on a missing setting, with a clear message | config: DB_HOST is not set |
| Serverless | the cloud runs your function per event; no server to keep running | AWS Lambda |
| AWS Lambda | Amazon's serverless service | the billing function |
| Event | the thing that triggers a function run | {"item":"chai","qty":3} |
| Handler | the function that processes one event | handle(ctx, o Order) |
| aws-lambda-go | AWS's Go library for Lambda | lambda.Start(handle) |
| provided.al2023 | Lambda's OS-only runtime on Amazon Linux 2023, used for Go | runtime setting |
| bootstrap | the required name of the executable in a provided runtime's .zip | go build -o bootstrap |
| Runtime API | the HTTP API inside Lambda that hands events to the function | GET /2018-06-01/runtime/invocation/next |
| lambda.norpc | build tag that drops the old go1.x runtime's RPC code | -tags lambda.norpc |
| Control loop | compare desired and actual state, act, repeat | the AC thermostat |
| Desired state | what you asked for (the spec) | replicas: 3 |
| Actual state | what is really running | [shop-2 shop-3] |
| Reconcile | make actual state match desired state | create shop-4 |
| Controller | a program running a control loop in Kubernetes | the Deployment controller |
| Custom resource | your own object type in the Kubernetes API | kind: ShopDB |
| Operator | a controller for a custom resource that automates operating an app | a database operator doing backups |
| client-go | the official Go client library for the Kubernetes API | informers, typed clients |
| Informer | client-go's watch-backed local cache of objects | controllers read from it |
| kubebuilder / Operator Framework | tools that generate operator projects | Reconcile skeleton |
| Idempotent | safe to run again with the same result | running reconcile twice |
Summary
- In a container, CPU limits throttle and memory limits kill. Go ≤ 1.24 doesn't see the CPU limit: set
GOMAXPROCS(Go 1.25+ reads it on Linux). SetGOMEMLIMITa little below the memory limit to avoid OOM kills (exit 137). - Probes are just handlers: cheap
/healthzfor liveness,/readyzthat says 503 until ready and while draining. - Kubernetes stops Pods with SIGTERM, waits 30 s by default, then SIGKILL. Catch it with
signal.NotifyContext, fail readiness,http.Server.Shutdownwith a shorter timeout, then clean up. No moreEOFfor customers. - Keep services stateless so replicas are interchangeable; state goes to a shared store.
- ConfigMaps for settings, Secrets (as files, encrypted at rest) for passwords; validate at start-up and never log secrets.
- AWS Lambda:
lambda.Start, a staticbootstrapbinary forprovided.al2023, amd64 or arm64. - Kubernetes is control loops; operators add your own, written in Go with client-go and kubebuilder.
- Next episode: cgo and unsafe.