SyntaxPilot
Courses/Python/Lesson 20

Python · Lesson 20 of 40 · 13 min

File handling, CSV and JSON

queued

These are the notes for Python episode 20. Every program we've written so far has the same weakness: the moment it ends, everything it knew is gone. In this episode a chai shop's order app forgets yesterday's orders, and we invent a way to save data to a file on the disk. You will learn why variables vanish, open(), write(), read() and close(), why a forgotten close() silently loses data and how with fixes it, the file modes "r", "w" and "a", encoding="utf-8", reading line by line with strip(), why files hold only text, handling a missing file, why splitting CSV lines by hand breaks, the csv module, and JSON for dictionaries and lists. Every example below runs as-is in the SyntaxPilot compiler, with its real output from Python 3.13.

About the online compiler: every run gets a fresh, empty folder. So whenever an example needs a saved file, it writes the file and reads it back in the same run. On your own computer, the file stays in the same folder as your .py file, and is still there tomorrow.

The problem: data that vanishes

A chai shop writes the day's orders on a whiteboard. Fast and easy. At night the board is wiped, and next morning nobody knows who ordered what, or who still owes money.

Variables work exactly like that whiteboard:

orders = []
orders.append(input("Order: "))
print("All orders so far:", orders)

First run, input chai:

Order: All orders so far: ['chai']

Second run (the next day), input samosa:

Order: All orders so far: ['samosa']

Where's the chai? Gone. Every run starts with an empty list.

Why: variables live in the computer's memory, called RAM. RAM is fast, but it is cleared when the program stops. To remember anything, the data has to go somewhere that survives: a file on the disk (the SSD or hard drive). A file is a notebook instead of a whiteboard.

Whiteboard (RAM) Notebook (file on disk)
Very fast Slower
Wiped when the program ends Stays until you delete it
Variables, lists, dicts orders.txt, bills.csv, asha.json

open, write, read, close

Using a file is like using a notebook:

The notebook In Python
Open the notebook, to write or to read f = open("orders.txt", "w") gives you a file object
Write a line f.write("chai\n") adds text
Read it back the next morning f.read() returns the whole file as one str
Close it and put it on the shelf f.close() finishes the job
f = open("orders.txt", "w")
f.write("chai\n")
f.write("samosa\n")
f.close()

f = open("orders.txt", "r")
print(f.read())
f.close()

Output:

chai
samosa

  • open(name, mode) takes the file name (a path) and a mode: "w" to write, "r" to read.
  • write() does not add a newline for you. \n is the newline character, so each order goes on its own line.
  • read() returns everything in the file as one string, including the newlines. (The extra blank line in the output is the last \n plus the one print adds.)

The forgotten close()

Same program, but we forget to close the file after writing, then open it again to read. repr() shows exactly what's inside, quotes and all:

f = open("orders.txt", "w")
f.write("chai\n")
f.write("samosa\n")

g = open("orders.txt")
print(repr(g.read()))

Output:

''

The file is empty, even though we wrote two orders and got no error.

Why: writing to the disk is slow, so Python collects your writes in a waiting area in memory, called a buffer, and saves them in one go when the file is closed (or the buffer fills up). No close(), no save yet. If the program crashes before close(), the data can be lost.

The fix: with

Remembering close() every time, even when an error happens in between, is hard. A with block closes the file for you when the block ends, even if an error happens inside it:

with open("orders.txt", "w") as f:
    f.write("chai\n")
    f.write("samosa\n")

with open("orders.txt") as f:
    print(repr(f.read()))

Output:

'chai\nsamosa\n'

Everything is saved. as f gives the file object a name, and the indented block is where you use it. Always open files with with. (How with works inside, the context manager, is episode 26.)

Using a file after its with block has ended is an error:

with open("orders.txt") as f:
    pass
f.read()
ValueError: I/O operation on closed file.

Modes: "r", "w", "a"

Day one writes chai and samosa. Day two opens the file again in "w" mode and writes vada pav:

with open("orders.txt", "w") as f:   # day 1
    f.write("chai\nsamosa\n")

with open("orders.txt", "w") as f:   # day 2
    f.write("vada pav\n")

with open("orders.txt") as f:
    print(f.read())

Output:

vada pav

Day one's orders are gone. "w" means start a fresh, empty file: if the file exists, it is wiped first, with no warning.

The fix for day two is "a", append: open the file and add to the end.

with open("orders.txt", "w") as f:   # day 1
    f.write("chai\nsamosa\n")

with open("orders.txt", "a") as f:   # day 2
    f.write("vada pav\n")

with open("orders.txt") as f:
    print(f.read())

Output:

chai
samosa
vada pav

Mode Meaning File missing? File exists?
"r" read (the default) FileNotFoundError read from the start
"w" write creates it wipes it first
"a" append creates it adds at the end

Opening in one mode and doing the other thing fails: reading a file opened with "w" gives io.UnsupportedOperation: not readable, and writing to a file opened with "r" gives io.UnsupportedOperation: not writable. (There are also "r+", "x" and binary modes like "rb", which you'll rarely need as a beginner.)

encoding="utf-8"

A file stores bytes, and an encoding is the rule that turns text into bytes and back. If your text has Hindi letters, emoji or the rupee sign, say which encoding to use, so the file reads the same on Windows, Mac and Linux:

with open("menu.txt", "w", encoding="utf-8") as f:
    f.write("चाय ₹15\n")
    f.write("समोसा ₹20\n")

with open("menu.txt", encoding="utf-8") as f:
    for line in f:
        print(line.strip())

Output:

चाय ₹15
समोसा ₹20

UTF-8 is the encoding almost the whole web uses. Without encoding=, Python uses the computer's default, which on some Windows machines is not UTF-8 and gives a UnicodeDecodeError or garbled text.

Reading line by line, and strip()

Big files are read one line at a time: a file object can be used in a for loop.

with open("orders.txt", "w") as f:
    f.write("chai\nsamosa\nvada pav\n")

with open("orders.txt") as f:
    for line in f:
        print(line)

Output:

chai

samosa

vada pav

An empty line after every order! Each line from the file still ends with its own \n, and print adds one more: two newlines, one blank line. The fix: line.strip(), which removes spaces and newlines from both ends.

with open("orders.txt", "w") as f:
    f.write("chai\nsamosa\nvada pav\n")

with open("orders.txt") as f:
    for line in f:
        print(line.strip())

Output:

chai
samosa
vada pav

Ways to read:

Code Gives you
f.read() the whole file as one str
for line in f: one line at a time (best for big files)
f.readlines() a list of lines, each ending in \n
f.readline() just the next line

Files hold text

with open("sales.txt", "w") as f:
    f.write(15)
Traceback (most recent call last):
  File "main.py", line 2, in <module>
    f.write(15)
    ~~~~~~~^^^^
TypeError: write() argument must be str, not int

A text file holds text. Convert first with str() or an f-string, and when you read it back it's text again, so convert with int() or float():

with open("sales.txt", "w") as f:
    f.write(str(15) + "\n")
    f.write(f"{20}\n")

total = 0
with open("sales.txt") as f:
    for line in f:
        total += int(line.strip())
print("Total sales:", total)

Output:

Total sales: 35

A missing file: FileNotFoundError

On the very first day there is no saved file yet. Reading it raises FileNotFoundError. This is where episode 18 pays off:

try:
    with open("yesterday.txt") as f:
        print(f.read())
except FileNotFoundError:
    print("No orders saved yet. Starting fresh!")

Output:

No orders saved yet. Starting fresh!

Without the try, the program would crash with:

FileNotFoundError: [Errno 2] No such file or directory: 'yesterday.txt'

Every program that loads a file should be ready for the file to be missing. Also check the path: a relative name like "orders.txt" is looked up in the current working folder (where you ran the program from).

CSV files

The shop wants to save bills: item, price, quantity. One bill per line, values separated by commas. That's CSV: comma separated values. Reading it back looks easy: split(",") each line into three pieces. Until the menu gets an item called Bun, butter:

lines = ["chai,15,2", "Bun, butter,30,1"]

for line in lines:
    item, price, qty = line.split(",")
    print(item, "->", int(price) * int(qty))

Output:

chai -> 30
Traceback (most recent call last):
  File "main.py", line 4, in <module>
    item, price, qty = line.split(",")
    ^^^^^^^^^^^^^^^^
ValueError: too many values to unpack (expected 3)

Bun, butter was cut into two pieces, so the line had four.

Why: split can't tell a comma that separates values from a comma inside a value. Real CSV files solve this with quoting: a value that contains a comma is wrapped in double quotes. The csv module (standard library) knows all these rules, so don't split CSV by hand.

Writing with csv.writer

import csv

bills = [["item", "price", "qty"],
         ["chai", 15, 2],
         ["Bun, butter", 30, 1]]

with open("bills.csv", "w", newline="") as f:
    csv.writer(f).writerows(bills)

with open("bills.csv") as f:
    print(f.read())

Output:

item,price,qty
chai,15,2
"Bun, butter",30,1

  • The first row is the header row: the column names.
  • The csv module quoted "Bun, butter" automatically.
  • writerow(row) writes one row; writerows(rows) writes many.
  • Open CSV files with newline="". The csv module writes its own line endings; without it, you get an extra blank line between rows on Windows.
  • Numbers are written as text automatically.

Reading with csv.DictReader

csv.DictReader turns each row into a dictionary, with the header row as its keys, so you can use columns by name:

import csv

with open("bills.csv", "w", newline="") as f:
    w = csv.writer(f)
    w.writerows([["item", "price", "qty"], ["chai", 15, 2], ["Bun, butter", 30, 1]])

with open("bills.csv") as f:
    for row in csv.DictReader(f):
        print(row["item"], "->", int(row["price"]) * int(row["qty"]))

Output:

chai -> 30
Bun, butter -> 30

Values come back as strings, so convert numbers with int() or float(). (csv.reader(f) is the simpler version: each row is a list, like ['chai', '15', '2'].) The same file opens in Excel or Google Sheets as a neat table.

Writing dictionaries with csv.DictWriter

import csv
rows = [{"item": "chai", "price": 15, "qty": 2}, {"item": "Bun, butter", "price": 30, "qty": 1}]
with open("bills.csv", "w", newline="") as f:
    w = csv.DictWriter(f, fieldnames=["item", "price", "qty"])
    w.writeheader()
    w.writerows(rows)
with open("bills.csv") as f:
    print(f.read())

Output:

item,price,qty
chai,15,2
"Bun, butter",30,1

fieldnames sets the column order, and writeheader() writes the header row.

JSON files

CSV is great for tables. But a customer profile isn't a table: a name, a credit amount, and a list of orders inside it. The obvious way to save it is str():

customer = {"name": "Asha", "credit": 120, "orders": ["chai", "bun"]}

with open("asha.txt", "w") as f:
    f.write(str(customer))

with open("asha.txt") as f:
    data = f.read()

print(data)
print(data["name"])

Output:

{'name': 'Asha', 'credit': 120, 'orders': ['chai', 'bun']}
Traceback (most recent call last):
  File "main.py", line 10, in <module>
    print(data["name"])
          ~~~~^^^^^^^^
TypeError: string indices must be integers, not 'str'

The print looks perfect, but data is a string, not a dictionary. It's like a photo of the page: it looks right, but you can't look things up in it.

We need a text format that can be turned back into real dictionaries and lists. That's JSON, short for JavaScript Object Notation. Nearly every app, website and API uses it to save and send data.

import json

customer = {"name": "Asha", "credit": 120, "orders": ["chai", "bun"]}

with open("asha.json", "w") as f:
    json.dump(customer, f, indent=2)

with open("asha.json") as f:
    data = json.load(f)

print(data["name"], "owes", data["credit"])
print(type(data))

Output:

Asha owes 120
<class 'dict'>

A real dictionary again. Turning Python data into JSON text is called serializing; turning it back is deserializing (or parsing).

What JSON looks like

json.dumps (with an s for string) gives the JSON text without a file:

import json

customer = {"name": "Asha", "credit": 120, "orders": ["chai", "bun"], "regular": True}
print(json.dumps(customer, indent=2))

Output:

{
  "name": "Asha",
  "credit": 120,
  "orders": [
    "chai",
    "bun"
  ],
  "regular": true
}
Function From To
json.dump(obj, f) Python object JSON in a file
json.load(f) JSON in a file Python object
json.dumps(obj) Python object JSON string
json.loads(text) JSON string Python object

Memory trick: the s means string.

JSON vs Python

JSON looks almost like Python, with small differences:

Python JSON
dict object { }
list (and tuple) array [ ]
"text" or 'text' only "text" (double quotes)
True / False true / false
None null
int, float number
import json
text = json.dumps({"name": "Asha", "phone": None, "regular": True, "rating": 4.5})
print(text)
back = json.loads(text)
print(back)
print(json.dumps({"stock": (1, 2)}))

Output:

{"name": "Asha", "phone": null, "regular": true, "rating": 4.5}
{'name': 'Asha', 'phone': None, 'regular': True, 'rating': 4.5}
{"stock": [1, 2]}

Notes:

  • A tuple comes back as a list. Sets can't be saved: json.dumps({1, 2}) gives TypeError: Object of type set is not JSON serializable.
  • JSON keys are always strings: a dict with key 1 comes back with key "1".
  • Hindi text is escaped by default ("चाय"). Add ensure_ascii=False to keep it readable: json.dumps({"item": "चाय"}, ensure_ascii=False) gives {"item": "चाय"}.
  • Single quotes are not JSON: json.loads("{'name': 'Asha'}") gives json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1).

CSV or JSON?

CSV JSON
Tables: rows and columns of the same shape Nested data: dicts and lists inside each other
Bills, marks, sales Profiles, settings, API responses
Opens in Excel / Google Sheets Used by apps, websites and APIs
csv module json module

Common mistakes

Mistake What you see Why Fix
f.write(...) with no close() No error: the file is empty ('') writes wait in a buffer with open(...) as f:
open("orders.txt", "w") on day 2 No error: yesterday's data is wiped "w" empties the file first open(..., "a")
print(line) in a file loop No error: blank lines between rows each line keeps its \n line.strip()
f.write(15) TypeError: write() argument must be str, not int files hold text f.write(str(15))
open("yesterday.txt") FileNotFoundError: [Errno 2] No such file or directory: 'yesterday.txt' the file doesn't exist try / except FileNotFoundError
line.split(",") with "Bun, butter" ValueError: too many values to unpack (expected 3) a comma inside a value the csv module
f.write(str(customer)), then data["name"] TypeError: string indices must be integers, not 'str' you get text back, not a dict json.dump / json.load
f.read() after the with block ValueError: I/O operation on closed file. with already closed it read inside the block
f.read() on a file opened with "w" io.UnsupportedOperation: not readable wrong mode open with "r"
JSON with a trailing comma {"a": 1,} json.decoder.JSONDecodeError: Illegal trailing comma before end of object: line 1 column 8 (char 7) not valid JSON let json.dump write the file
CSV opened without newline="" blank rows between lines on Windows double line endings open(..., newline="")

Mini project: a khata that survives restarts

A khata is the credit book every Indian shop keeps: who owes how much. It must survive restarts.

  • load_book() reads the JSON file, or returns an empty dict {} on the first day (no file yet).
  • add(name, amount) loads the book, adds the amount with book.get(name, 0) + amount (episode 12), and saves it again.
  • Because every call loads from the file, each call is like a fresh start of the program on a new day.
# Mini project: a khata (credit book) that survives restarts
import json

def load_book():
    try:
        with open("khata.json") as f:
            return json.load(f)
    except FileNotFoundError:
        return {}

def add(name, amount):
    book = load_book()
    book[name] = book.get(name, 0) + amount
    with open("khata.json", "w") as f:
        json.dump(book, f)
    print(name, "now owes", book[name])

add("Asha", 120)   # Monday
add("Ravi", 50)    # Tuesday
add("Asha", 30)    # Wednesday
print(load_book())

Output:

Asha now owes 120
Ravi now owes 50
Asha now owes 150
{'Asha': 150, 'Ravi': 50}

On Wednesday, Asha owes 150: the file remembered Monday. Note that saving uses "w" on purpose: we load the whole book, change it, and write the whole book back.

Extension: a pay() function.

import json

def load_book():
    try:
        with open("khata.json") as f:
            return json.load(f)
    except FileNotFoundError:
        return {}

def save(book):
    with open("khata.json", "w") as f:
        json.dump(book, f)

def add(name, amount):
    book = load_book()
    book[name] = book.get(name, 0) + amount
    save(book)
    print(name, "now owes", book[name])

def pay(name, amount):
    book = load_book()
    book[name] = book.get(name, 0) - amount
    save(book)
    print(name, "paid", amount, "and now owes", book[name])

add("Asha", 120)
add("Ravi", 50)
pay("Asha", 70)
print(load_book())

Output:

Asha now owes 120
Ravi now owes 50
Asha paid 70 and now owes 50
{'Asha': 50, 'Ravi': 50}

Practice

1. Save 5 friends' names to a file, one per line, then read them back and number them.

Solution
friends = ["Asha", "Ravi", "Meena", "Arjun", "Kabir"]
with open("friends.txt", "w") as f:
    for name in friends:
        f.write(name + "\n")

with open("friends.txt") as f:
    for i, line in enumerate(f, start=1):
        print(i, line.strip())

Output:

1 Asha
2 Ravi
3 Meena
4 Arjun
5 Kabir

2. A to-do list that appends one task per run. (Here the loop stands in for three runs.)

Solution
for day in range(3):
    task = ["buy milk", "call Ravi", "pay rent"][day]
    with open("todo.txt", "a") as f:
        f.write(task + "\n")

with open("todo.txt") as f:
    print(f.read().strip())

Output:

buy milk
call Ravi
pay rent

On your computer, use task = input("New task: ") and run the program again each day: "a" keeps the old tasks.

3. Write marks.csv and print the class average.

Solution
import csv
with open("marks.csv", "w", newline="") as f:
    w = csv.writer(f)
    w.writerow(["name", "marks"])
    w.writerows([["Asha", 82], ["Ravi", 67], ["Meena", 91]])

total = count = 0
with open("marks.csv") as f:
    for row in csv.DictReader(f):
        total += int(row["marks"])
        count += 1
print("Class average:", round(total / count, 2))

Output:

Class average: 80.0

4. Save your settings dict to JSON and load it back.

Solution
import json
settings = {"theme": "dark", "font_size": 16, "languages": ["en", "hi"]}
with open("settings.json", "w") as f:
    json.dump(settings, f, indent=2)
with open("settings.json") as f:
    loaded = json.load(f)
print(loaded["languages"][1], loaded == settings)

Output:

hi True

Quick quiz

1. The video's quiz:

with open("a.txt", "w") as f:
    f.write("1\n")
with open("a.txt", "a") as f:
    f.write("2\n")
with open("a.txt", "w") as f:
    f.write("3\n")
with open("a.txt") as f:
    print(f.read().strip())
Answer

Just 3. Append added 2 after 1, but the final "w" wiped the file before writing 3.

2. Why did the file come back empty when we forgot close()?

Answer

Writes wait in a buffer in memory and are saved to disk when the file is closed (or the buffer fills). Without close(), nothing had been saved yet when we read it. with closes the file automatically.

3. What type is row["price"] when reading with csv.DictReader?

Answer

str. CSV files hold text; convert with int() or float().

4. What does json.dumps({"ok": True, "x": None}) return?

Answer

The string '{"ok": true, "x": null}': True becomes true and None becomes null.

Interview questions

  1. Why do we need files? Variables live in RAM, which is cleared when the program ends. Files on disk persist.
  2. Why use with open(...)? It closes the file automatically when the block ends, even if an exception happens, so buffered data is saved and the file is released.
  3. Difference between "w" and "a"? "w" empties the file (or creates it) before writing; "a" adds to the end (or creates it).
  4. What is buffering? Collecting writes in memory and writing them to disk in larger chunks, because disk access is slow. close() or flush() pushes the buffer to disk.
  5. read() vs readlines() vs iterating the file? read() returns one string; readlines() returns a list of lines; for line in f reads lazily, one line at a time, which is best for big files.
  6. Why use the csv module instead of split(",")? It handles quoting (commas, quotes and newlines inside values) and line endings correctly.
  7. Why newline="" when opening CSV files? The csv module writes its own line endings; without it, Windows gets an extra blank line per row.
  8. json.dump vs json.dumps? dump writes to a file; dumps returns a string. Likewise load / loads.
  9. Which Python types can be saved as JSON? dict, list, tuple (as a list), str, int, float, bool and None. Sets and custom objects can't be saved directly.
  10. What does encoding="utf-8" do? It fixes how text is turned into bytes, so non-English text reads the same on every computer.

Glossary

Term Meaning Example
RAM (memory) Fast storage that is cleared when the program ends; where variables live orders = []
Disk Storage that keeps data after the program ends an SSD or hard drive
File Named data saved on disk orders.txt
Path Where a file is: its name, plus folders if needed "orders.txt", "data/bills.csv"
Persist To keep data after the program ends saving the khata to JSON
open() Opens a file and returns a file object open("orders.txt", "w")
File object What open() returns; you read or write through it f
Mode What you want to do with the file "r", "w", "a"
"r" Read mode, the default open("a.txt")
"w" Write mode: creates or wipes the file open("a.txt", "w")
"a" Append mode: adds at the end open("a.txt", "a")
write() Writes a string to the file (no automatic newline) f.write("chai\n")
read() Reads the whole file as one string f.read()
readlines() Reads all lines into a list ['chai\n', 'samosa\n']
close() Saves buffered data and releases the file f.close()
Buffer A waiting area in memory for writes, saved to disk in one go the empty-file bug
with A block that closes the file automatically when it ends with open(...) as f:
Context manager An object that sets up and cleans up for a with block (episode 26) a file object
Newline \n The character that ends a line "chai\n"
strip() Removes spaces and newlines from both ends of a string "chai\n".strip() → "chai"
repr() Shows a value exactly, with quotes and escape codes 'chai\nsamosa\n'
Encoding The rule for turning text into bytes and back encoding="utf-8"
UTF-8 The standard encoding for all languages and emoji Hindi text, ₹
FileNotFoundError Raised when opening a missing file for reading open("yesterday.txt")
CSV Comma separated values: a table as text, one row per line chai,15,2
Header row The first CSV row, holding the column names item,price,qty
Quoting Wrapping a value in " so commas inside it are safe "Bun, butter"
csv.writer Writes lists as CSV rows csv.writer(f).writerows(rows)
csv.reader Reads CSV rows as lists ['chai', '15', '2']
csv.DictReader Reads CSV rows as dicts keyed by the header row["price"]
csv.DictWriter Writes dicts as CSV rows w.writeheader()
newline="" Lets the csv module control line endings open("b.csv", "w", newline="")
JSON JavaScript Object Notation: text for dicts and lists {"name": "Asha"}
json.dump / json.load Python object to/from a JSON file json.dump(customer, f)
json.dumps / json.loads Python object to/from a JSON string json.dumps(customer)
indent Pretty-prints JSON with line breaks and spaces indent=2
null JSON's version of None "phone": null
Serialize Turn data into text (or bytes) to save or send json.dumps
JSONDecodeError Raised when text isn't valid JSON single quotes, trailing comma
Khata An Indian shop's credit book: who owes how much the mini project

Summary

  • Variables live in RAM and vanish when the program ends; files on disk stay.
  • open(name, mode) gives a file object; write() writes text, read() reads it back.
  • Writes wait in a buffer until close(). Always use with open(...) as f:, which closes for you.
  • "r" reads (default), "w" wipes and writes, "a" appends. Add encoding="utf-8" for non-English text.
  • Read big files line by line with for line in f, and strip() each line.
  • Files hold text: str() to write numbers, int() to read them back.
  • Be ready for a missing file: try / except FileNotFoundError.
  • Don't split CSV by hand: use csv.writer and csv.DictReader, with newline="".
  • Use JSON (json.dump / json.load) for dictionaries and lists. s = string versions.

Next episode: object oriented programming, part 1. Our khata stores customers as dictionaries. Next time, you'll invent your own types: classes and objects.