SyntaxPilot
Courses/Python Coding Questions/Lesson 28
Video coming soonThis lesson’s video is being made

Start with the notes below; the video appears here as soon as it is on YouTube.

Python Coding Questions · Lesson 28 of 30

File and data questions

count words in a file (with)top-k frequent words (heapq.nlargest)parse a CSV and total a column (csv module, bad rows)

These are the notes for Python Coding Questions, episode 28: file and data questions. Real programs read real files, and real files are messy. Three interview questions: count the words in a file, find the top-k frequent words, and total a column of a CSV file that has bad rows.

Every program was run with Python 3.13 (python3 main.py), the SyntaxPilot compiler's Python command. Every output below is real. Timings change from machine to machine and run to run; what matters is how they compare.

Words for today:

  • File path: the name (and folder) of a file, like review.txt. A bare name means "in the program's current folder".
  • Mode: how a file is opened: "r" read (the default), "w" write (creates or empties it), "a" append.
  • Context manager: an object used with with that sets something up and always cleans up; open() closes the file at the end of the with block, even after an error.
  • Whitespace: spaces, tabs and new lines.
  • Regular expression (regex): a pattern for text; re.findall(pattern, text) returns every match.
  • Top-k: the k biggest items by some value.
  • Heap: a structure that always gives the smallest item fast; heapq.nlargest(k, ...) uses one to keep a shortlist of k.
  • CSV: comma separated values, a spreadsheet saved as plain text. Values that contain commas are wrapped in quotes.

Q1: Count the words in a file

Real-life example: a café saves each customer review in a text file. The owner wants the number of words, and how often each word appears.

Attempt 1: open the file and split on spaces

f = open("review.txt")
text = f.read()
print(len(text.split(" ")))

Output:

Traceback (most recent call last):
  File "main.py", line 1, in <module>
    f = open("review.txt")
FileNotFoundError: [Errno 2] No such file or directory: 'review.txt'

FileNotFoundError: [Errno 2] No such file or directory: 'review.txt'. The file is looked up in the program's current folder, and it isn't there. Check the name and the folder, or (as here) create it first.

Writing and reading safely with with

with open("review.txt", "w") as f:
    f.write(review)

with closes the file at the end of the block, even if an error happens inside. Without it you must call f.close() yourself; forget it and written data may still sit in a memory buffer, not on the disk.

review = """Great chai.  Great samosa!
The chai was hot,
the samosa was crisp.
"""
with open("review.txt", "w") as f:      # closed for us at the end
    f.write(review)
print("closed?", f.closed)

with open("review.txt") as f:
    text = f.read()

print(len(text.split(" ")))             # split on single spaces
print(text.split(" ")[:5])
print(len(text.split()))                # split on any whitespace

Output:

closed? True
11
['Great', 'chai.', '', 'Great', 'samosa!\nThe']
12

f.closed is True after the block. But split(" ") gives 11, while split() gives 12: split(" ") cuts only at single spaces, so the double space makes an empty string '', and the new line glues 'samosa!\nThe' into one piece. split() with no argument cuts at any run of whitespace and drops empty pieces.

Counting each word

Plain split() still treats "Great" and "great", "chai." and "chai" as different words. Clean each line first: line.lower(), then re.findall(r"[a-z']+", ...) keeps only runs of letters (and apostrophes, for words like don't). Count with a dictionary, and read the file line by line (for line in f) so even a huge file never has to fit in memory.

import re

with open("review.txt", "w") as f:
    f.write("Great chai.  Great samosa!\nThe chai was hot,\n"
            "the samosa was crisp.\n")

def count_words(path):
    counts = {}
    with open(path) as f:
        for line in f:                  # one line at a time
            for word in re.findall(r"[a-z']+", line.lower()):
                counts[word] = counts.get(word, 0) + 1
    return counts

counts = count_words("review.txt")
print(sum(counts.values()), "words,", len(counts), "different")
print(counts)

Output:

12 words, 7 different
{'great': 2, 'chai': 2, 'samosa': 2, 'the': 2, 'was': 2, 'hot': 1, 'crisp': 1}

O(n) time for n characters; memory is one line plus one entry per different word.

Way to split Result on "Great chai. Great samosa!\nThe ..."
text.split(" ") 'chai.', '', 'samosa!\nThe': wrong count
text.split() 'chai.', 'samosa!', 'The': right count, punctuation kept
re.findall(r"[a-z']+", text.lower()) 'chai', 'samosa', 'the': clean words

Q2: Top-k frequent words

Real-life example: a college canteen writes down every order. The manager wants the 3 most ordered snacks for the menu board.

Attempt 1: sorted(counts)[:3]

words = ("chai samosa chai vada chai samosa dosa "
         "vada chai idli samosa dosa chai").split()

counts = {}
for w in words:
    counts[w] = counts.get(w, 0) + 1
print(counts)

print(sorted(counts)[:3])               # top 3?

Output:

{'chai': 5, 'samosa': 3, 'vada': 2, 'dosa': 2, 'idli': 1}
['chai', 'dosa', 'idli']

The counts are right, but the "top 3" is chai, dosa, idli, and idli was ordered once. Sorting a dictionary sorts its keys (the words, alphabetically); it never looks at the counts.

The fix: sort by the count, or keep a shortlist

  • counts.items() gives (word, count) pairs.
  • sorted(pairs, key=lambda kv: kv[1], reverse=True)[:k] sorts by the count: O(n log n).
  • heapq.nlargest(k, pairs, key=...) keeps only a heap of the k best so far: O(n log k).
  • collections.Counter(words) is a dictionary that counts for you; .most_common(k) gives the top k.
  • Ties keep the order the words were first seen in (all three are stable).
import heapq, random, time
from collections import Counter

words = ("chai samosa chai vada chai samosa dosa "
         "vada chai idli samosa dosa chai").split()
counts = Counter(words)                 # a dict that counts

pairs = counts.items()
by_count = sorted(pairs, key=lambda kv: kv[1], reverse=True)
print(by_count[:3])
print(heapq.nlargest(3, pairs, key=lambda kv: kv[1]))
print(counts.most_common(3))
print(heapq.nlargest(10, Counter(["a"]).items()))

big = Counter({f"w{i}": random.randint(1, 10**6)
               for i in range(10**6)})     # a million words
start = time.perf_counter()
sorted(big.items(), key=lambda kv: kv[1], reverse=True)[:5]
print("sorted  ", f"{time.perf_counter() - start:.2f} s")
start = time.perf_counter()
heapq.nlargest(5, big.items(), key=lambda kv: kv[1])
print("nlargest", f"{time.perf_counter() - start:.2f} s")

Output:

[('chai', 5), ('samosa', 3), ('vada', 2)]
[('chai', 5), ('samosa', 3), ('vada', 2)]
[('chai', 5), ('samosa', 3), ('vada', 2)]
[('a', 1)]
sorted   1.84 s
nlargest 0.12 s

All three give chai 5, samosa 3, vada 2 (vada and dosa both have 2; vada was seen first). Asking for more than there are just returns what exists. On a million words, nlargest is more than ten times faster than sorting everything.

Way Time When
sorted(..., key=...)[:k] O(n log n) you also want the full ranking
heapq.nlargest(k, ..., key=...) O(n log k) k is much smaller than n
Counter(...).most_common(k) uses a heap for k you are counting anyway

Q3: Parse a CSV file and total a column

Real-life example: a canteen's sales sheet with item, qty, price. The day's total is qty × price summed over every row. The sheet was typed by hand: "Samosa, large" has a comma inside quotes, vada pav's quantity is N/A, and the dosa row has no price.

Attempt 1: split each line on commas

data = """item,qty,price
Chai,2,15
"Samosa, large",3,20
Vada pav,N/A,25
Dosa,1
Idli,4,10
"""
with open("sales.csv", "w") as f:
    f.write(data)

total = 0
with open("sales.csv") as f:
    for line in f:
        item, qty, price = line.strip().split(",")
        total += int(qty) * float(price)
print("total:", total)

Output:

Traceback (most recent call last):
  File "main.py", line 15, in <module>
    total += int(qty) * float(price)
             ~~~^^^^^
ValueError: invalid literal for int() with base 10: 'qty'

ValueError: invalid literal for int() with base 10: 'qty': the first line is the header (the column names). Skip it with next(f), which reads one line and throws it away:

data = """item,qty,price
Chai,2,15
"Samosa, large",3,20
Vada pav,N/A,25
Dosa,1
Idli,4,10
"""
with open("sales.csv", "w") as f:
    f.write(data)

total = 0
with open("sales.csv") as f:
    next(f)                     # skip the header line
    for line in f:
        item, qty, price = line.strip().split(",")
        total += int(qty) * float(price)
print("total:", total)

Output:

Traceback (most recent call last):
  File "main.py", line 15, in <module>
    item, qty, price = line.strip().split(",")
    ^^^^^^^^^^^^^^^^
ValueError: too many values to unpack (expected 3)

ValueError: too many values to unpack (expected 3): "Samosa, large" was cut into 4 pieces. CSV has rules (quotes protect commas, and can even hold new lines) and split(",") ignores them.

The csv module

Open CSV files with newline="" (the csv documentation asks for it, so the module handles line endings itself). csv.reader turns each line into a list of strings, respecting quotes:

import csv

data = """item,qty,price
Chai,2,15
"Samosa, large",3,20
Vada pav,N/A,25
Dosa,1
Idli,4,10
"""
with open("sales.csv", "w", newline="") as f:
    f.write(data)

with open("sales.csv", newline="") as f:
    for row in csv.reader(f):
        print(row)

Output:

['item', 'qty', 'price']
['Chai', '2', '15']
['Samosa, large', '3', '20']
['Vada pav', 'N/A', '25']
['Dosa', '1']
['Idli', '4', '10']

The fix: csv.DictReader, named exceptions, and a report

  • csv.DictReader uses the header as keys: row["qty"] instead of row[1], and it keeps working if the columns move. A short row gets None for the missing values.
  • int("N/A") raises ValueError; float(None) raises TypeError.
  • Catch exactly those errors, note the line number and item, and carry on. Line numbers start at 2 because line 1 is the header.
import csv

with open("sales.csv", "w", newline="") as f:
    f.write('item,qty,price\nChai,2,15\n"Samosa, large",3,20\n'
            'Vada pav,N/A,25\nDosa,1\nIdli,4,10\n')

def total_sales(path):
    total, bad = 0, []
    with open(path, newline="") as f:
        for n, row in enumerate(csv.DictReader(f), start=2):
            try:
                total += int(row["qty"]) * float(row["price"])
            except (ValueError, TypeError) as e:
                bad.append((n, row["item"], type(e).__name__))
    return total, bad

total, bad = total_sales("sales.csv")
print("total:", total)
for line in bad:
    print("skipped line", *line)

Output:

total: 130.0
skipped line 4 Vada pav ValueError
skipped line 5 Dosa TypeError

Total 130.0 (chai 30 + samosa 60 + idli 40), and the two bad lines are reported, not silently lost. O(rows) time.

Common mistakes (and what you really see)

Mistake What you see Why Fix
open("review.txt") on a missing file FileNotFoundError: [Errno 2] No such file or directory: 'review.txt' wrong folder or name check the path; create it first
f = open(...) and no close() data may never reach the disk the file is never closed with open(...) as f:
text.split(" ") 11 words instead of 12 '' pieces and words glued by \n text.split()
counting raw words Great and great counted apart case and punctuation lower() + re.findall
sorted(counts)[:3] ['chai', 'dosa', 'idli'] sorts the keys key=lambda kv: kv[1] or key=counts.get
line.split(",") on a CSV ValueError: too many values to unpack (expected 3) quotes protect commas csv.reader / csv.DictReader
forgetting the header ValueError: invalid literal for int() with base 10: 'qty' line 1 is column names next(f) or DictReader
except: pass bad rows (and your bugs) vanish silently catches everything except (ValueError, TypeError) and report

Sorting a dict sorts its keys:

counts = {"chai": 5, "samosa": 3, "vada": 2, "dosa": 2, "idli": 1}
print(sorted(counts))
print(sorted(counts, key=counts.get, reverse=True)[:3])

Output:

['chai', 'dosa', 'idli', 'samosa', 'vada']
['chai', 'samosa', 'vada']

A bare except: pass even hides a typo (prce), so the total is silently 0:

rows = [["Chai", "2", "15"], ["Vada pav", "N/A", "25"]]
total = 0
for item, qty, price in rows:
    try:
        total += int(qty) * float(prce)        # a typo: prce
    except:
        pass                                   # hides EVERYTHING
print("total:", total)

Output:

total: 0

Your turn: the best seller (solution)

Question: from the same sales.csv, return the item that earned the most money (qty × price), skipping bad rows.

Idea: DictReader, a dictionary of totals per item, then max(earned, key=earned.get): the key whose value is biggest.

import csv

with open("sales.csv", "w", newline="") as f:
    f.write('item,qty,price\nChai,2,15\n"Samosa, large",3,20\n'
            'Vada pav,N/A,25\nDosa,1\nIdli,4,10\n')

def best_item(path):
    earned = {}
    with open(path, newline="") as f:
        for row in csv.DictReader(f):
            try:
                money = int(row["qty"]) * float(row["price"])
            except (ValueError, TypeError):
                continue                        # skip bad rows
            earned[row["item"]] = earned.get(row["item"], 0) + money
    if not earned:
        return None
    best = max(earned, key=earned.get)          # the key with the biggest value
    return best, earned[best]

print(best_item("sales.csv"))

Output:

('Samosa, large', 60.0)

Mini project: a review report

All three ideas together: read a CSV of reviews, skip rows with bad star ratings, average the stars, and find the top words without common "stop words".

import csv, heapq, re
from collections import Counter

with open("reviews.csv", "w", newline="") as f:
    f.write('id,stars,text\n'
            '1,5,"Great chai, great samosa!"\n'
            '2,4,The chai was hot and the vada was crisp\n'
            '3,five,"Samosa cold, chai great"\n'
            '4,2,"Too sweet. The chai, not the samosa."\n')

STOP = {"the", "a", "and", "was", "not", "too"}

def review_report(path, k=3):
    words, stars, bad = Counter(), [], []
    with open(path, newline="") as f:
        for n, row in enumerate(csv.DictReader(f), start=2):
            try:
                stars.append(int(row["stars"]))
            except ValueError:
                bad.append(n)
                continue
            words.update(w for w in re.findall(r"[a-z']+", row["text"].lower())
                         if w not in STOP)
    avg = sum(stars) / len(stars) if stars else None
    return avg, heapq.nlargest(k, words.items(), key=lambda kv: kv[1]), bad

avg, top, bad = review_report("reviews.csv")
print(f"average stars: {avg:.2f}")
print("top words:", top)
print("bad lines:", bad)

Output:

average stars: 3.67
top words: [('chai', 3), ('great', 2), ('samosa', 2)]
bad lines: [4]

Practice questions (with solutions)

1. Your own wc: count the lines, words and characters of a file.

Solution
def wc(path):
    lines = words = chars = 0
    with open(path) as f:
        for line in f:
            lines += 1
            words += len(line.split())
            chars += len(line)
    return lines, words, chars

with open("poem.txt", "w") as f:
    f.write("Chai in the morning,\nchai at night.\n\nChai makes  everything right.\n")
print(wc("poem.txt"))

Output:

(4, 11, 67)

4 lines (the empty line counts), 11 words (split() ignores the double space), 67 characters including the new lines.

2. Merge two CSV files by item: prices from one file, stock from another.

Solution

Load the smaller file into a dictionary, then look each row of the other one up: O(n + m). A missing price comes back as None from dict.get.

import csv

with open("prices.csv", "w", newline="") as f:
    f.write("item,price\nChai,15\nSamosa,20\nIdli,10\n")
with open("stock.csv", "w", newline="") as f:
    f.write("item,stock\nChai,40\nIdli,25\nDosa,12\n")

def merge(prices_path, stock_path):
    with open(prices_path, newline="") as f:
        prices = {r["item"]: float(r["price"]) for r in csv.DictReader(f)}
    merged = []
    with open(stock_path, newline="") as f:
        for r in csv.DictReader(f):
            merged.append((r["item"], prices.get(r["item"]), int(r["stock"])))
    return merged

for row in merge("prices.csv", "stock.csv"):
    print(row)

Output:

('Chai', 15.0, 40)
('Idli', 10.0, 25)
('Dosa', None, 12)

3. Top-k words without stop words like "the", "was", "and".

Solution
import re
from collections import Counter

STOP = {"the", "a", "an", "and", "was", "is", "of"}

def top_words(text, k):
    words = re.findall(r"[a-z']+", text.lower())
    return Counter(w for w in words if w not in STOP).most_common(k)

text = "The chai was hot and the samosa was crisp. The chai is the best chai of the city."
print(top_words(text, 3))

Output:

[('chai', 3), ('hot', 1), ('samosa', 1)]

Quick quiz

What does this print?

text = "tea  tea\ncoffee"
print(len(text.split(" ")), len(text.split()))

(A) 3 3 (B) 4 3 (C) 3 2 (D) 4 2

Answer

(A). Same count, different pieces: only split() gives the real words.

text = "tea  tea\ncoffee"
print(len(text.split(" ")), len(text.split()))
print(text.split(" "), text.split())

Output:

3 3
['tea', '', 'tea\ncoffee'] ['tea', 'tea', 'coffee']

Interview questions

  1. Why use with open(...)? It closes the file at the end of the block even if an exception is raised, so data is flushed and the file handle isn't leaked.
  2. Difference between "w" and "a"? "w" creates or empties the file; "a" appends to the end.
  3. How do you read a 10 GB file? Line by line (for line in f) or in fixed-size chunks (f.read(65536)), never f.read() all at once.
  4. split() vs split(" ")? split() splits on any run of whitespace and drops empty strings; split(" ") splits on every single space.
  5. How do you sort a dict by value? sorted(d.items(), key=lambda kv: kv[1], reverse=True) or sorted(d, key=d.get, reverse=True).
  6. heapq.nlargest vs sorting? O(n log k) vs O(n log n); nlargest wins when k is small.
  7. What does Counter.most_common(k) return? A list of (item, count) pairs, biggest counts first.
  8. Why not split CSV lines on commas? Quoted values may contain commas (and new lines); the csv module follows the format's rules.
  9. Why newline="" when opening CSV files? So the csv module sees the raw line endings and handles quoted new lines correctly.
  10. How should a data job handle bad rows? Catch the specific errors, skip the row, and report it (line number and reason), so the result can be trusted and the data fixed.

Glossary

Term Meaning Example
File path Name and folder of a file review.txt
FileNotFoundError Raised when the file doesn't exist [Errno 2] No such file or directory
Mode How a file is opened "r", "w", "a"
with / context manager Sets up and always cleans up the file is closed after the block
f.closed True once the file is closed after with
Whitespace Spaces, tabs, new lines "\n"
split() Split on any whitespace "a b".split() → ['a', 'b']
Regex A text pattern [a-z']+
re.findall Every match of a pattern the words of a line
lower() Lower-case copy of a string "Great" → "great"
dict.get(k, 0) The value, or 0 if missing counting
Counter A dict that counts Counter(words)
most_common(k) Top k (item, count) pairs [('chai', 5), ...]
Top-k The k biggest by some value 3 snacks
Heap / heapq Fast access to the smallest item heapq.nlargest
key= What to sort or compare by key=lambda kv: kv[1]
lambda A small one-line function lambda kv: kv[1]
Stable sort Equal items keep their order vada before dosa
CSV Comma separated values item,qty,price
Header The first row: column names item,qty,price
next(f) Read (and skip) one line skip the header
csv.reader Rows as lists, quotes respected ['Samosa, large', '3', '20']
csv.DictReader Rows as dicts keyed by the header row["qty"]
ValueError Right type, bad value int("N/A")
TypeError Wrong type float(None)
Bare except Catches every error hides bugs
Stop words Very common words left out of counts the, was, and

Summary

  • Files: with open(path, mode) as f: always closes the file; FileNotFoundError means a wrong name or folder; read big files line by line.
  • Words: split(), not split(" "); clean with lower() and re.findall(r"[a-z']+", ...); count with a dict or Counter.
  • Top-k: sorted(dict) sorts keys; sort the pairs by count, or use heapq.nlargest (O(n log k)) or Counter.most_common.
  • CSV: use the csv module (newline="", DictReader), catch the errors you expect (ValueError, TypeError), and report skipped rows.
  • Next episode: the machine coding round: a key-value store with TTL and an event emitter, tested with unittest.