SyntaxPilot
Courses/C Programming/Lesson 21

C Programming · Lesson 21 of 30 · 12 min

Debugging with gdb

queued

These are the notes for C episode 21. So far, when a program went wrong, we stared at the code and guessed. This episode starts with two real bugs: a report card that prints a wrong average, and a canteen program that crashes. First you try the obvious tool, printf debugging, and watch it work for one bug and fail completely for the other, because a crash throws away buffered output. Then you invent the fix: a debugger that watches the program from the outside. With gdb you find the crash in four commands, and the wrong answer without editing a single line. Programs were compiled with gcc 13 (the SyntaxPilot compiler's command is gcc -O2 -Wno-unused-result main.c -o main -lm); the gdb sessions are real runs of GNU gdb 15 on Ubuntu 24.04. The two [Thread debugging using libthread_db enabled] lines gdb prints after Starting program, and its "Quit anyway?" prompt, are left out to save space.

What a bug is

A bug is a mistake that makes a program do what you wrote instead of what you meant. Debugging is finding the place where those two split apart. The compiler catches mistakes in the form of your code (syntax, types); bugs are mistakes in its logic, and they compile perfectly.

Bug 1: a report card that lies

#include <stdio.h>

int total_marks(int m[], int n) {
    int sum = 0;
    for (int i = 0; i < n; i++) {
        sum = m[i];
    }
    return sum;
}

int main() {
    int marks[5] = {78, 92, 65, 88, 71};
    int total = total_marks(marks, 5);
    printf("Total: %d\n", total);
    printf("Average: %.1f\n", total / 5.0);
    return 0;
}

Output:

Total: 71
Average: 14.2

Marks in the seventies and nineties, average 14.2. No warning, no crash, just wrong.

The obvious attempt: printf debugging

Real-life example: when a shopkeeper's total is wrong, he adds again and writes down the running total after every item, then looks for the step where it goes wrong. In C, writing down each step means a printf inside the loop:

    for (int i = 0; i < n; i++) {
        sum = m[i];
        printf("i = %d, sum = %d\n", i, sum);
    }
i = 0, sum = 78
i = 1, sum = 92
i = 2, sum = 65
i = 3, sum = 88
i = 4, sum = 71
Total: 71
Average: 14.2

sum should grow (78, 170, 235, ...), but it jumps to each mark: it is replaced, not added to. Line 6 should be sum += m[i];. With the fix (and the printf removed):

Total: 394
Average: 78.8

printf debugging works for small bugs, but it costs: edit, rebuild, read, edit again, and remember to remove every printf afterwards.

Bug 2: the crash that eats your printfs

#include <stdio.h>
#include <string.h>

struct item { char name[20]; int price; };

struct item menu[] = {{"Chai", 20}, {"Samosa", 15}, {"Lassi", 40}};

struct item *find(const char *name) {
    for (int i = 0; i < 3; i++)
        if (strcmp(menu[i].name, name) == 0)
            return &menu[i];
    return NULL;
}

int price_of(const char *name) {
    struct item *it = find(name);
    return it->price;
}

int main() {
    char *order[] = {"Chai", "Samosa", "Coffee", "Lassi"};
    int total = 0;
    for (int i = 0; i < 4; i++) {
        printf("adding %s\n", order[i]);
        total += price_of(order[i]);
    }
    printf("Total: Rs %d\n", total);
    return 0;
}

In the SyntaxPilot compiler the only output is:

Segmentation fault

Not one adding ... line, although printf ran at least twice before the crash. In a terminal you see the difference:

$ gcc menu.c -o app
$ ./app
adding Chai
adding Samosa
adding Coffee
Segmentation fault
$ ./app > log.txt
Segmentation fault
$ cat log.txt
$

Why: output buffering

printf doesn't write straight to the screen. It drops the text into a buffer (a waiting bucket in memory), and the bucket is emptied later, all at once, because that is much faster than writing every character.

Where stdout goes When the buffer is emptied (flushed)
A terminal screen at every \n (line buffered)
A file, a pipe, an online compiler only when the buffer is full, or when the program ends normally (fully buffered)
stderr (any destination) immediately (unbuffered)

A crash is not a normal ending: the operating system kills the program and the bucket with your adding lines is thrown away. Workarounds: fflush(stdout); after the printf, or print debugging lines to stderr with fprintf(stderr, ...). The real lesson: asking a crashing program to talk is unreliable. We need to watch it from the outside.

The idea: a debugger

Real-life example: the third umpire. The on-field umpire decides in real time. The third umpire freezes the replay at the exact moment, goes frame by frame, zooms in on the bat, and uses ball tracking to see the path that led there.

Third umpire gdb What it does
Freeze the replay break, run stop the program at a line
Frame by frame next, step run one line at a time
Zoom in on the bat print sum look inside any variable
Ball tracking bt the chain of function calls that led here

A debugger runs your program under its control so you can stop it, step through it and inspect its memory. gdb, the GNU debugger, comes with gcc on Linux (on macOS the equivalent is lldb).

Step 1: compile with -g

gcc -g menu.c -o app
gdb -q ./app
  • -g stores debug information in the program: a map from machine code back to your file names, line numbers and variable names. Without it gdb only sees addresses.
  • Don't use -O2 while debugging: optimisation moves, merges and removes code, so the program no longer matches your lines.
  • -q (quiet) skips gdb's welcome banner.

Finding the crash: run, bt, print, up

$ gcc -g menu.c -o app
$ gdb -q ./app
Reading symbols from ./app...
(gdb) run
Starting program: /tmp/w/e21/app

Program received signal SIGSEGV, Segmentation fault.
0x0000555555555226 in price_of (name=0x555555556010 "Coffee") at menu.c:17
17      return it->price;

The program is frozen at the exact line that crashed, and gdb even shows the argument: name is "Coffee". SIGSEGV is the signal the operating system sends for a segmentation fault.

(gdb) bt
#0  0x0000555555555226 in price_of (name=0x555555556010 "Coffee") at menu.c:17
#1  0x00005555555552b5 in main () at menu.c:25
(gdb) print it
$1 = (struct item *) 0x0
(gdb) up
#1  0x00005555555552b5 in main () at menu.c:25
25          total += price_of(order[i]);
(gdb) print i
$2 = 2
(gdb) print order[i]
$3 = 0x555555556010 "Coffee"
  • bt (backtrace): the call stack, newest call on top. Read it bottom up: main (line 25) called price_of, which crashed at line 17. Each line is a frame.
  • print it: the pointer is 0x0, which is NULL. find returned NULL and the code did NULL->price.
  • up: move to the caller's frame (down goes back). Now print i and print order[i] show the local variables of main.

Diagnosis: "Coffee" is not on the menu, find returned NULL, and nobody checked. Fix it where the pointer is used:

int price_of(const char *name) {
    struct item *it = find(name);
    if (it == NULL) {
        printf("%s: not on the menu\n", name);
        return 0;
    }
    return it->price;
}
Coffee: not on the menu
Total: Rs 75

Finding the wrong answer: break, next, print, watch

A crash freezes the program by itself; a wrong answer doesn't. So you choose where to freeze: a breakpoint.

Breakpoint form Stops at
break total_marks the first line of the function
break 6 line 6 of the current file
break marks.c:6 line 6 of marks.c
break fact if n == 1 a conditional breakpoint: only when the condition is true
$ gcc -g marks.c -o report
$ gdb -q ./report
(gdb) break total_marks
Breakpoint 1 at 0x1178: file marks.c, line 4.
(gdb) run
Breakpoint 1, total_marks (m=0x7fffffff73e0, n=5) at marks.c:4
4       int sum = 0;
(gdb) next
5       for (int i = 0; i < n; i++) {
(gdb) next
6           sum = m[i];
(gdb) next
5       for (int i = 0; i < n; i++) {
(gdb) print sum
$1 = 78

gdb stops before the shown line runs, and next runs one line. After one pass sum is 78, which is still correct. Pressing next a hundred times is slow, so use a watchpoint: stop whenever a value changes.

(gdb) watch sum
Hardware watchpoint 2: sum
(gdb) continue
Continuing.
Hardware watchpoint 2: sum
Old value = 78
New value = 92
total_marks (m=0x7fffffff73e0, n=5) at marks.c:5
5       for (int i = 0; i < n; i++) {
(gdb) continue
Continuing.
Hardware watchpoint 2: sum
Old value = 92
New value = 65
total_marks (m=0x7fffffff73e0, n=5) at marks.c:5
5       for (int i = 0; i < n; i++) {
(gdb) info locals
i = 2
sum = 65

78 then 92 should give 170, not 92: sum is being replaced. The line that changes it, sum = m[i];, is the bug. Found without editing the program.

step vs next, and finish

(gdb) break 13
Breakpoint 1 at 0x11f0: file marks.c, line 13.
(gdb) run
Breakpoint 1, main () at marks.c:13
13      int total = total_marks(marks, 5);
(gdb) step
total_marks (m=0x7fffffff73e0, n=5) at marks.c:4
4       int sum = 0;
(gdb) finish
Run till exit from #0  total_marks (m=0x7fffffff73e0, n=5) at marks.c:4
0x0000555555555201 in main () at marks.c:13
13      int total = total_marks(marks, 5);
Value returned is $1 = 71
  • next runs the whole call on the current line as one step (steps over functions).
  • step goes into the function being called.
  • finish runs until the current function returns and shows the return value.

gdb cheat sheet

Command Short What it does
run r start the program (add arguments after it: run 250 4)
break X b X breakpoint at a function, a line, or file:line
info breakpoints i b list breakpoints and watchpoints
delete N d N remove breakpoint N (delete alone removes all)
next n run one line, stepping over calls
step s run one line, stepping into calls
continue c run until the next breakpoint, watchpoint or crash
finish run until the current function returns, show the value
print X p X show a variable or any C expression (p m[1], p *it)
display X print X automatically after every step
watch X stop whenever X changes (Old value / New value)
info locals show every local variable of the current function
bt backtrace: the chain of calls that led here
up / down move to the caller's frame / back
list l show the source around the current line
kill stop the running program
quit q leave gdb

Pressing Enter on an empty line repeats the last command, which makes next + Enter + Enter very quick.

Common mistakes (and the real messages)

$ gcc marks.c -o nog                  # forgot -g
$ gdb -q ./nog
Reading symbols from ./nog...
(No debugging symbols found in ./nog)
(gdb) print sum
No symbol "sum" in current context.
(gdb) next
The program is not being run.
$ gcc -g -O2 marks.c -o opt           # optimised
(gdb) break total_marks
Breakpoint 1 at 0x11a0: file marks.c, line 3.
(gdb) run
Total: 71
Average: 14.2
[Inferior 1 (process 1718) exited normally]
Mistake What you see Why Fix
printf debugging before a crash Only Segmentation fault, no printf lines The stdout buffer is thrown away gdb, or fflush(stdout) / fprintf(stderr, ...)
Compiled without -g (No debugging symbols found in ./app) No map from code to names and lines gcc -g ...
print sum in the wrong function (or before run) No symbol "sum" in current context. sum lives in total_marks break inside it, or up/down to its frame
next before run The program is not being run. Nothing is running yet run first
Debugging an -O2 build Breakpoint never hit, or <optimized out> The optimiser inlined or removed code Debug builds: -g, no -O2 (or -Og)
Forgetting that gdb stops before the line print shows the old value The shown line hasn't run yet next, then print

Mini project: debug a broken program yourself

Copy the canteen program (bug 2) and the report card (bug 1) into files, compile them with -g, and reproduce both sessions above without looking. Then make a new bug on purpose (for example change i < 3 in find to i <= 3) and use print i and print menu[i] at a breakpoint in find to see it read past the end of the menu.

Practice

  1. Make printf debugging survive the crash in bug 2 with one extra line.
  2. Use a watchpoint to follow stars in this loop and predict each Old/New value first: c int stars = 0; for (int row = 1; row <= 3; row++) stars += row;
  3. Step through a recursive fact(4) and use bt to see every call waiting on the stack.
  4. Which command would you use to (a) leave a long function you entered by mistake, (b) stop only on the 1,000th loop pass, (c) see all local variables at once?
Solutions

1. Add fflush(stdout); right after printf("adding %s\n", order[i]);. Verified with the output sent to a file: the file now contains adding Chai, adding Samosa, adding Coffee, and then the program crashes. fprintf(stderr, "adding %s\n", order[i]); works too, because stderr is not buffered.

2. Real session (after break 5, run, watch stars, then continue each time):

Old value = 0
New value = 1
...
Old value = 1
New value = 3
...
Old value = 3
New value = 6

The fourth continue prints Watchpoint 2 deleted because the program has left the block in which its expression is valid.: stars stops existing when main ends.

3. With int fact(int n) { if (n == 0) return 1; return n * fact(n - 1); } and a conditional breakpoint:

(gdb) break fact if n == 1
Breakpoint 1 at 0x1158: file fact.c, line 4.
(gdb) run
Breakpoint 1, fact (n=1) at fact.c:4
4       if (n == 0)
(gdb) bt
#0  fact (n=1) at fact.c:4
#1  0x0000555555555172 in fact (n=2) at fact.c:6
#2  0x0000555555555172 in fact (n=3) at fact.c:6
#3  0x0000555555555172 in fact (n=4) at fact.c:6
#4  0x000055555555518a in main () at fact.c:10
(gdb) up
#1  0x0000555555555172 in fact (n=2) at fact.c:6
6       return n * fact(n - 1);
(gdb) print n
$2 = 2
(gdb) continue
Continuing.
24

Every recursive call has its own frame with its own n.

4. (a) finish; (b) a conditional breakpoint such as break 6 if i == 999; (c) info locals.

Quick quiz

You are stopped in main on the line int total = total_marks(marks, 5); and you want to go inside total_marks. Which command?

  • A. next
  • B. step
  • C. continue
  • D. finish
Answer

B: step. next runs the whole call as one line, continue runs to the next stop, and finish leaves the current function. The real session above shows step landing on line 4 of total_marks.

Interview questions

  • What is the difference between a compile error, a runtime error (crash) and a logic bug?
  • Why might printf output not appear before a segmentation fault? What is buffering, and how do fflush and stderr help?
  • What does the -g flag do? Why is debugging an optimised (-O2) build confusing?
  • What is a breakpoint? A conditional breakpoint? A watchpoint?
  • What is the difference between next and step? What does finish do?
  • What is a backtrace (call stack), and how do you read it? What do up and down do?
  • How would you find the cause of a segmentation fault with gdb?
  • What is a core dump, and how can gdb use one? (A file with the crashed program's memory; gdb ./app core opens it at the moment of the crash.)
  • What other debugging tools exist for C? (Sanitizers such as -fsanitize=address, valgrind, IDE debuggers built on gdb or lldb; episode 26.)

Glossary

Term Meaning Example
Bug A mistake in a program's logic: it does what you wrote, not what you meant sum = m[i] instead of sum += m[i]
Debugging Finding and fixing bugs this episode
printf debugging Adding printf lines to see values while the program runs printf("i = %d, sum = %d\n", i, sum);
Buffer A block of memory where output waits before being written the stdout buffer
Flush Write out everything waiting in a buffer fflush(stdout);
Line buffered Flushed at every newline (stdout to a terminal) ./app in a terminal
Fully buffered Flushed only when full or at a normal exit (stdout to a file or pipe) ./app > log.txt
stderr The error output stream; not buffered fprintf(stderr, "...")
Debugger A program that runs another program under control: stop, step, inspect gdb, lldb
gdb The GNU debugger, for C, C++ and more gdb -q ./app
-g gcc option that adds debug information (lines and names) gcc -g menu.c -o app
Debug information (symbols) The map from machine code back to source lines and variable names missing without -g
Signal A message the operating system sends to a program SIGSEGV for a segmentation fault
Segmentation fault The OS stopping a program that touched memory it may not use NULL->price
Breakpoint A place where gdb stops the program before running that line break total_marks
Conditional breakpoint A breakpoint that stops only when a condition is true break fact if n == 1
Watchpoint Stops the program whenever a variable's value changes watch sum
Step over / step into Run a call as one line (next) / go inside it (step) next, step
Call stack The list of function calls that are still running, newest on top main → price_of
Frame One entry on the call stack: a call with its own local variables #1 main () at menu.c:25
Backtrace gdb's printout of the call stack bt
Local variable A variable that belongs to one function call i, sum
Inline (inlining) The optimiser copying a function's body into its caller, so the call disappears why -O2 skipped the breakpoint
Core dump A file with a crashed program's memory, which gdb can open later gdb ./app core

Summary

  • A bug compiles fine; it's a mistake in logic. printf debugging works for small bugs but means editing, rebuilding and cleaning up.
  • stdout is buffered; to a file or the online compiler it is flushed only when full or at a normal exit, so a crash throws your printf lines away.
  • A debugger watches the program from outside. Compile with gcc -g and no -O2, then gdb -q ./app.
  • For a crash: run stops at the exact line, bt shows how you got there, print and up let you look inside. Always check pointers for NULL.
  • For a wrong answer: break, then next / step / finish, print, info locals, and watch to stop when a value changes.

Next episode: Linked lists: nodes, pointers and malloc, a list that grows one item at a time, and why inserting at the front of an array is so slow.