Linux 4: Processes, Signals and Jobs (ps, top, htop, kill, nohup, nice, zombies, load average)

These are the notes for Cloud and DevOps episode 5. In episode 4 you decided who can do what. Now DabbaGo's server program has to run all day, taking orders. You'll learn what a process is, run programs in the background, pause and resume them, find the one eating the CPU, stop programs with signals (politely and by force), keep them alive after you log out, and understand zombies, orphans, /proc and the load average.

Every command was run for real on the course machine (Ubuntu 24.04.5 LTS, bash 5.2.21, 2 CPUs) as user student in /srv/dabbago. The session ran in its own fresh process space (a PID namespace) with a tiny init program, dumb-init, as PID 1, so ps lists only this session's processes and the PIDs are small. On a real server you'll see many more processes and much bigger PIDs. Outputs are copied exactly; yours will differ.

The eight problems

Problem 1: program vs process

• The CPU is the chip that does the computing; memory (RAM) is where running programs keep their data.
• One program can run as many processes at once (two cooks, one recipe).
• Every command you type becomes a process while it runs.

student@vm:/srv/dabbago$ ps
    PID TTY          TIME CMD
      3 pts/0    00:00:00 bash
     13 pts/0    00:00:00 ps

ps (process status) lists the processes in your terminal: PID, TTY (the terminal it's attached to; pts/0 is a pseudo-terminal, ? means none), TIME (CPU time used so far) and CMD (the command).

Problem 2: my terminal is stuck (foreground, background, jobs)

A small demo program, kitchen.py:

import time
n = 1000
while True:
    n += 1
    print(f"cooking order #{n}...", flush=True)
    time.sleep(1)

student@vm:/srv/dabbago$ python3 kitchen.py
cooking order #1001...
cooking order #1002...
cooking order #1003...
cooking order #1004...
^CTraceback (most recent call last):
  File "/srv/dabbago/kitchen.py", line 6, in <module>
    time.sleep(1)
    ~~~~~~~~~~^^^
KeyboardInterrupt

A program in the foreground owns the terminal: you can't type other commands until it ends. Ctrl+C interrupts it (Python reports KeyboardInterrupt).

The real server, server.py, writes to a log file forever, and cleans up when it receives SIGTERM:

import signal, sys, time, datetime

def log(msg):
    with open("server.log", "a") as f:
        f.write(datetime.datetime.now().strftime("%H:%M:%S") + " " + msg + "\n")

def stop(signum, frame):
    log("got SIGTERM: finishing the current order, then exiting")
    sys.exit(0)

signal.signal(signal.SIGTERM, stop)
log("DabbaGo server started")
n = 1000
while True:
    n += 1
    log(f"served order #{n}")
    time.sleep(2)

The fix: & starts a command in the background, so you get your prompt back at once.

student@vm:/srv/dabbago$ python3 server.py &
[1] 15
student@vm:/srv/dabbago$ jobs
[1]+  Running                 python3 server.py &
student@vm:/srv/dabbago$ ps -f
UID          PID    PPID  C STIME TTY          TIME CMD
student        3       2  0 10:32 pts/0    00:00:00 -bash
student       15       3 10 10:32 pts/0    00:00:00 python3 server.py
student       16       3  0 10:32 pts/0    00:00:00 ps -f
student@vm:/srv/dabbago$ pstree -p student
bash(3)─┬─pstree(17)
        └─python3(15)
student@vm:/srv/dabbago$ tail -n 3 server.log
10:32:58 served order #1001
10:33:00 served order #1002
10:33:02 served order #1003

• [1] 15 = job number 1, PID 15. A job is the shell's own name for a process it started from this terminal; refer to it as %1.
• jobs lists this shell's jobs. + marks the current job, - the previous one.
• ps -f (full format) adds PPID, the parent process ID: every process is started by another, its parent. The server's parent is bash (3). C is recent CPU use, STIME the start time.
• pstree draws the family tree (-p shows PIDs).

Problem 3: pause and resume (Ctrl+Z, fg, bg, states)

student@vm:/srv/dabbago$ fg
python3 server.py
^Z
[1]+  Stopped                 python3 server.py
student@vm:/srv/dabbago$ jobs
[1]+  Stopped                 python3 server.py
student@vm:/srv/dabbago$ ps -o pid,stat,cmd
    PID STAT CMD
      3 S    -bash
     15 T    python3 server.py
     19 R+   ps -o pid,stat,cmd
student@vm:/srv/dabbago$ bg
[1]+ python3 server.py &
student@vm:/srv/dabbago$ jobs
[1]+  Running                 python3 server.py &

Process states (the STAT column)

Extra letters: + in the foreground of its terminal, s session leader, l multi-threaded, < high priority, N low priority (niced).

Problem 4: the server feels slow (ps aux, top, htop, nice)

student@vm:/srv/dabbago$ ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0   2588  1492 pts/0    S    10:32   0:00 dumb-init --single-child -- su - student
root           2  0.0  0.0   6292  4068 pts/0    S    10:32   0:00 su - student
student        3  0.0  0.0   5028  4316 pts/0    S    10:32   0:00 -bash
student       15  0.1  0.1  36720 12076 pts/0    S    10:32   0:00 python3 server.py
student       20  0.0  0.0   8348  4392 pts/0    R+   10:33   0:00 ps aux

ps aux: a = all users, u = user-oriented columns, x = include processes without a terminal. VSZ is the virtual memory size and RSS (resident set size) the real memory in use, both in KiB.

PID 1 is the first process, the ancestor of all others. Here it's dumb-init. On a normal server it's systemd: "When run as first process on boot (as PID 1), it acts as init system that brings up and maintains userspace services" (systemd(1) man page). Episode 8 covers systemd.

A CPU hog for the demo, busy.py:

n = 0
while True:      # pure CPU work, never sleeps
    n += 1

student@vm:/srv/dabbago$ python3 busy.py &
[2] 21
student@vm:/srv/dabbago$ top -b -n 1
top - 10:33:25 up 11 min,  0 user,  load average: 0.39, 0.51, 0.33
Tasks:   6 total,   2 running,   4 sleeping,   0 stopped,   0 zombie
%Cpu(s): 50.0 us,  0.0 sy,  0.0 ni, 50.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
MiB Mem :   8027.6 total,   2813.0 free,   1265.9 used,   4262.3 buff/cache     
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   6761.7 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
     21 student   20   0   35556  10916   7908 R  90.9   0.1   0:20.01 python3
      1 root      20   0    2588   1492   1376 S   0.0   0.0   0:00.00 dumb-init
      2 root      20   0    6292   4068   3624 S   0.0   0.0   0:00.00 su
      3 student   20   0    5028   4316   3632 S   0.0   0.1   0:00.01 bash
     15 student   20   0   36720  12076   8208 S   0.0   0.1   0:00.02 python3
     22 student   20   0    9388   5512   3388 R   0.0   0.1   0:00.00 top

• top is a live view sorted by CPU; it refreshes every few seconds. Inside it: q quit, P sort by CPU, M by memory, k kill, r renice. -b -n 1 prints one snapshot (useful in scripts and here).
• Header: uptime, load average (Problem 8), task counts by state, CPU split (us user programs, sy kernel, ni niced, id idle, wa waiting for disk, st stolen by the hypervisor), memory and swap.
• htop is a friendlier, colourful top with CPU bars, scrolling, search (F3), tree view (F5), nice (F7/F8) and kill (F9). Install it with the package manager if missing.

htop with three busy.py running on 2 CPUs:

    0[||||||||||||||||||||||||||||100.0%] Tasks: 7, 0 thr, 0 kthr; 2 running
    1[||||||||||||||||||||||||||||100.0%] Load average: 2.31 1.07 0.55
  Mem[||||||||||||||||||||||||965M/7.84G] Uptime: 00:13:00
  Swp[                             0K/0K]

  [Main] [I/O]
    PID USER       PRI  NI  VIRT   RES   SHR S  CPU%▽MEM%   TIME+  Command
     14 student     20   0 35556 10916  7904 R  68.0  0.1  0:52.69 python3 busy.py
     15 student     20   0 35556 10948  7936 R  64.7  0.1  0:53.77 python3 busy.py
     16 student     20   0 35556 10904  7896 R  64.7  0.1  0:54.38 python3 busy.py
      1 root        20   0  2588  1492  1376 S   0.0  0.0  0:00.00 dumb-init --singl
      2 root        20   0  6292  4036  3592 S   0.0  0.0  0:00.00 su - student
      3 student     20   0  5028  4312  3636 S   0.0  0.1  0:00.00 -bash
     20 student     20   0  5456  4228  3536 R   0.0  0.1  0:00.00 htop

nice and renice: CPU priority

The nice value goes from -20 (greediest, highest priority) to 19 (most polite). The default is 0.

student@vm:/srv/dabbago$ kill 21
student@vm:/srv/dabbago$ jobs
[1]-  Running                 python3 server.py &
[2]+  Terminated              python3 busy.py
student@vm:/srv/dabbago$ nice -n 10 python3 busy.py &
[2] 24
student@vm:/srv/dabbago$ ps -o pid,ni,%cpu,cmd
    PID  NI %CPU CMD
      3   0  0.0 -bash
     15   0  0.0 python3 server.py
     24  10 98.0 python3 busy.py
     25   0  0.0 ps -o pid,ni,%cpu,cmd
student@vm:/srv/dabbago$ renice -n 15 -p 24
24 (process ID) old priority 10, new priority 15
student@vm:/srv/dabbago$ renice -n -5 -p 24
renice: failed to set priority for 24 (process ID): Permission denied

• nice -n N command starts a command with nice value N.
• renice -n N -p PID changes a running process.
• A normal user can only make a process nicer (higher number). Only root can lower it (sudo renice -n -5 -p 24).
• A niced process still uses an idle CPU fully (98% here); it just yields when others need the CPU.

Problem 5: stop it, nicely or not (signals)

A signal is a small numbered message the kernel delivers to a process. A program can catch most signals (run its own handler), ignore them, or take the default action (usually to end).

kill sends SIGTERM

student@vm:/srv/dabbago$ kill 24
[2]+  Terminated              nice -n 10 python3 busy.py
student@vm:/srv/dabbago$ kill 15
student@vm:/srv/dabbago$ jobs
[1]+  Done                    python3 server.py
student@vm:/srv/dabbago$ tail -n 2 server.log
10:33:26 served order #1015
10:33:28 got SIGTERM: finishing the current order, then exiting

busy.py has no handler, so SIGTERM ended it (Terminated). The server caught SIGTERM, logged a goodbye and exited by itself with status 0, so bash says Done. That's a graceful shutdown, which real servers do to finish work and save data.

A program that ignores SIGTERM

import signal, time
signal.signal(signal.SIGTERM, signal.SIG_IGN)   # ignore polite requests
while True:
    time.sleep(1)

student@vm:/srv/dabbago$ python3 stuck.py &
[1] 29
student@vm:/srv/dabbago$ kill 29
student@vm:/srv/dabbago$ ps -o pid,stat,cmd -p 29
    PID STAT CMD
     29 S    python3 stuck.py
student@vm:/srv/dabbago$ kill -9 29
student@vm:/srv/dabbago$ jobs
[1]+  Killed                  python3 stuck.py

SIGKILL (9) is carried out by the kernel; the program never runs any code, so it can't be caught or ignored. Always try plain kill first and wait a few seconds: kill -9 skips cleanup (half-written files, leftover lock files, unsaved data).

student@vm:/srv/dabbago$ kill -l
 1) SIGHUP   2) SIGINT   3) SIGQUIT  4) SIGILL   5) SIGTRAP
 6) SIGABRT  7) SIGBUS   8) SIGFPE   9) SIGKILL 10) SIGUSR1
11) SIGSEGV 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM
16) SIGSTKFLT   17) SIGCHLD 18) SIGCONT 19) SIGSTOP 20) SIGTSTP
...
63) SIGRTMAX-1  64) SIGRTMAX    

kill, kill -15, kill -TERM and kill -SIGTERM are all the same. kill %1 targets a job number.

Problem 6: survive logging out (SIGHUP, nohup, pkill, killall)

When your terminal closes or your SSH connection drops, the shell sends SIGHUP ("hang up") to its jobs, and by default they end.

student@vm:/srv/dabbago$ python3 server.py &
[1] 31
student@vm:/srv/dabbago$ nohup python3 server.py &
[2] 32
student@vm:/srv/dabbago$ nohup: ignoring input and appending output to 'nohup.out'
student@vm:/srv/dabbago$ jobs
[1]-  Running                 python3 server.py &
[2]+  Running                 nohup python3 server.py &
student@vm:/srv/dabbago$ kill -HUP 31 32
student@vm:/srv/dabbago$ jobs
[1]-  Hangup                  python3 server.py
[2]+  Running                 nohup python3 server.py &

nohup ("no hang up") starts a command that ignores SIGHUP, and sends any terminal output to nohup.out. (Its message appeared just after the next prompt, because the background job printed it.) It's the quick fix; for real services use systemd (episode 8), and for interactive work that must survive disconnects, tmux or screen.

Find and kill by name

student@vm:/srv/dabbago$ python3 busy.py &
[3] 34
student@vm:/srv/dabbago$ pgrep -a python3
32 python3 server.py
34 python3 busy.py
student@vm:/srv/dabbago$ pkill -f server.py
student@vm:/srv/dabbago$ pgrep -a python3
34 python3 busy.py
[2]-  Done                    nohup python3 server.py
student@vm:/srv/dabbago$ killall python3
student@vm:/srv/dabbago$ jobs
[3]+  Terminated              python3 busy.py
student@vm:/srv/dabbago$ killall python3
python3: no process found

Be careful: killall python3 stops every python3 you're allowed to signal. (A normal user can only signal their own processes; root can signal anyone.)

Problem 7: the process that won't die (zombies, orphans)

fork creates a copy of the running process, the child. When a child ends, it leaves an exit status (its exit code) that the parent must collect with wait.

# zombie.py
import os, time
pid = os.fork()
if pid == 0:            # the child
    os._exit(0)         # finishes at once
time.sleep(300)         # the parent never calls wait()

# orphan.py
import os, time
pid = os.fork()
if pid > 0:             # the parent
    os._exit(0)         # leaves at once
time.sleep(300)         # the child keeps running

Zombie

student@vm:/srv/dabbago$ python3 zombie.py &
[1] 40
student@vm:/srv/dabbago$ ps -o pid,ppid,stat,cmd
    PID    PPID STAT CMD
      3       2 S    -bash
     40       3 S    python3 zombie.py
     41      40 Z    [python3] <defunct>
     42       3 R+   ps -o pid,ppid,stat,cmd
student@vm:/srv/dabbago$ kill -9 41
student@vm:/srv/dabbago$ ps -o pid,ppid,stat,cmd
    PID    PPID STAT CMD
      3       2 S    -bash
     40       3 S    python3 zombie.py
     41      40 Z    [python3] <defunct>
     43       3 R+   ps -o pid,ppid,stat,cmd
student@vm:/srv/dabbago$ kill 40
student@vm:/srv/dabbago$ ps -o pid,ppid,stat,cmd
    PID    PPID STAT CMD
      3       2 S    -bash
     44       3 R+   ps -o pid,ppid,stat,cmd
[1]+  Terminated              python3 zombie.py

A zombie (Z, <defunct>) has finished, but its parent hasn't collected its exit status, so the kernel keeps a small entry for it. It uses no CPU and almost no memory, but takes a slot in the process table, and thousands can exhaust it. You can't kill a zombie (it's already dead). Fix or restart the parent: when the parent dies, PID 1 adopts the zombie and collects it.

Orphan

student@vm:/srv/dabbago$ python3 orphan.py
student@vm:/srv/dabbago$ pgrep -af orphan
46 python3 orphan.py
student@vm:/srv/dabbago$ ps -o pid,ppid,stat,cmd -p 46
    PID    PPID STAT CMD
     46       1 S    python3 orphan.py
student@vm:/srv/dabbago$ kill 46

An orphan is a process whose parent has exited. It keeps running and is adopted by PID 1 (PPID becomes 1), which will collect it when it ends. Orphans are normal (that's how daemons used to detach); piles of zombies are a bug.

Problem 8: is the machine overloaded? (/proc, load average)

/proc is a virtual filesystem: not on disk, but created live by the kernel, with one folder per process, named by its PID. ps, top and friends read it.

student@vm:/srv/dabbago$ python3 server.py &
[1] 22
student@vm:/srv/dabbago$ ls /proc/22
arch_status      environ            maps           pagemap       stat
attr             exe                mem            personality   statm
autogroup        fd                 mountinfo      projid_map    status
auxv             fdinfo             mounts         root          syscall
cgroup           gid_map            mountstats     sched         task
clear_refs       io                 net            schedstat     timens_offsets
cmdline          ksm_merging_pages  ns             sessionid     timers
comm             ksm_stat           numa_maps      setgroups     timerslack_ns
coredump_filter  limits             oom_adj        smaps         uid_map
cpuset           loginuid           oom_score      smaps_rollup  wchan
cwd              map_files          oom_score_adj  stack

From the earlier session (where the server had PID 49):

student@vm:/srv/dabbago$ head -n 9 /proc/49/status
Name:   python3
Umask:  0002
State:  S (sleeping)
Tgid:   49
Ngid:   0
Pid:    49
PPid:   3
TracerPid:  0
Uid:    30033   30033   30033   30033
student@vm:/srv/dabbago$ ls -l /proc/49/cwd
lrwxrwxrwx 1 student student 0 Oct 28 10:33 /proc/49/cwd -> /srv/dabbago

Useful entries: status (name, state, PIDs, user), cmdline (the command, with NUL bytes between words), cwd (working folder), exe (the program file), fd/ (open files, episode 6), environ (environment variables, owner only). System-wide: /proc/loadavg, /proc/cpuinfo, /proc/meminfo.

Load average

student@vm:/srv/dabbago$ uptime
 10:34:02 up 11 min,  0 user,  load average: 0.22, 0.45, 0.32
student@vm:/srv/dabbago$ python3 busy.py &
[1] 14
student@vm:/srv/dabbago$ python3 busy.py &
[2] 15
student@vm:/srv/dabbago$ python3 busy.py &
[3] 16
student@vm:/srv/dabbago$ uptime
 10:35:22 up 12 min,  0 user,  load average: 2.31, 1.07, 0.55
student@vm:/srv/dabbago$ nproc
2

The load average is the average number of processes that are running or waiting to run (on Linux, also those in D state waiting for the disk), over the last 1, 5 and 15 minutes. Compare it with the number of CPUs (nproc):

Three hogs on 2 CPUs pushed the 1-minute load to 2.31 within 80 seconds (and still rising toward 3). The 15-minute value moves slowly, so the three numbers show the trend.

Common mistakes (real messages)

Mini project: run DabbaGo's server like an admin

cd /srv/dabbago
nohup python3 server.py &          # survives logout
echo $! > server.pid               # $! = PID of the last background job
cat server.pid
pgrep -af server.py                # confirm it runs
tail -f server.log                 # watch it (Ctrl+C stops tail, not the server)
kill "$(cat server.pid)"           # polite stop: SIGTERM
sleep 3; tail -n 1 server.log      # "got SIGTERM: ..."
pgrep -af server.py || echo "server stopped"

$! holds the PID of the most recent background job; saving it in a PID file is how many services remember what to stop.

Practice

1. Start sleep 300 & twice. List the jobs, bring the second to the foreground, pause it, and resume it in the background.
2. Find your shell's PID (echo $$) and its parent's PID from /proc/$$/status.
3. Start a busy.py. Use top to find its PID, then renice it to 19.
4. A program ignores Ctrl+C and kill. What's left? What's the risk?
5. ps shows 300 <defunct> processes with the same PPID. What do you do?
6. On an 8-CPU server, uptime shows load average: 12.40, 11.90, 6.10. Is it overloaded? Getting better or worse?

sleep 300 &
sleep 300 &
jobs
fg %2        # then Ctrl+Z
bg %2
jobs

Quick quiz

A program has handlers for every signal it can handle. Which signal stops it for sure?

• A. SIGTERM
• B. SIGINT
• C. SIGKILL
• D. SIGHUP

Interview questions

• What's the difference between a program and a process? What are PID and PPID?
• How do you run a command in the background, and bring it back?
• What do the states R, S, D, T and Z mean?
• SIGTERM vs SIGKILL: what's the difference, and which do you send first?
• What does Ctrl+C send? Ctrl+Z?
• How do you keep a process running after logout?
• What is a zombie process? Why can't you kill it? How do you get rid of it?
• What is an orphan process, and who adopts it?
• What is /proc?
• Explain the three load average numbers. When is a load of 4 a problem?
• What is the nice value, and who can lower it?

Glossary

Summary

• A process is a running program with a PID and a parent (PPID); PID 1 is the ancestor (systemd on real servers).
• See processes with ps, ps -f, ps aux, pstree, top, htop; read states R, S, D, T, Z.
• & runs in the background; jobs, fg, bg, Ctrl+Z (pause) and Ctrl+C (interrupt) move jobs around.
• nice/renice set CPU politeness; only root can raise priority.
• Signals: kill sends SIGTERM (catchable, graceful); kill -9 sends SIGKILL (instant, no cleanup) as a last resort.
• Logout sends SIGHUP; nohup (or systemd) keeps servers alive. pgrep, pkill -f, killall work by name.
• Zombies can't be killed: fix the parent. Orphans are adopted by PID 1.
• /proc holds live process data; load average ÷ CPUs above 1 means queueing.

Next episode: Linux 5: redirection, pipes and filters. Save output to files, hide errors, and chain small commands into one powerful line.