These are the notes for Low-Level Design (LLD) episode 71: designing a task scheduler, the part of a system that runs the right job at the right time: an OTP now, a reminder in ten minutes, a report every morning, a retry after a failure. We start with a first-in-first-out queue that runs tasks late and fix one problem at a time: a PriorityQueue by next run time, one-time, delayed and recurring tasks, task cancellation, a DelayQueue with worker threads, Java's ScheduledExecutorService and missed executions, retries with backoff, and cron expressions.
How the code was run: every program was saved as Main.java in a fresh folder and run with javac Main.java && java -cp . Main on OpenJDK 21. Every output below is real. Programs that use the real clock print slightly different milliseconds on every run; the fake-clock programs print the same output every time.
1. The problem: Dadi's phone reminders
| On Dadi's phone | In a scheduler |
|---|---|
| "Wake up at 6 AM tomorrow": rings once at a fixed time | A one-time task |
| "Medicine every 8 hours": rings again and again | A recurring task |
| "Remind me in 10 minutes": a snooze, counted from now | A delayed task |
| The battery died overnight: which alarms should ring now? | Missed executions and a policy for them |
2. The words
- Task: a piece of work (e.g.
send-otp) plus when to run it. - Task scheduler: keeps all the tasks and runs each one on time.
- One-time task: runs once, at a fixed time (6 AM).
- Delayed task: runs once, after a delay counted from now (in 10 minutes).
- Recurring task: runs again every period (every 8 hours).
- Next run time: when a task should run next. For a recurring task it moves forward after every run.
3. Attempt 1: first in, first out
A FIFO queue (first in, first out, like a ticket counter line) that sleeps until each task is due:
import java.util.*;
record Task(String name, long runAt) {} // runAt: milliseconds after the start
public class Main { public static void main(String[] args) throws Exception {
long start = System.currentTimeMillis();
Queue<Task> tasks = new ArrayDeque<>(); // attempt: first in, first out
tasks.add(new Task("daily-report", 600));
tasks.add(new Task("send-otp", 100)); // added later, but due FIRST
tasks.add(new Task("reminder", 300));
while (!tasks.isEmpty()) {
Task t = tasks.poll();
long wait = t.runAt() - (System.currentTimeMillis() - start);
if (wait > 0) Thread.sleep(wait); // sleep until this task is due
long now = System.currentTimeMillis() - start;
System.out.printf("%-12s wanted %3d ms, ran at %3d ms, late by %3d ms%n",
t.name(), t.runAt(), now, now - t.runAt());
} } }
Output:
daily-report wanted 600 ms, ran at 601 ms, late by 1 ms
send-otp wanted 100 ms, ran at 644 ms, late by 544 ms
reminder wanted 300 ms, ran at 646 ms, late by 346 ms
Why it failed
FIFO runs tasks in the order they were added, not the order they are due. send-otp waited behind the daily report and ran about half a second late. Sorting the whole list on every add would fix the order but costs O(n log n) each time.
4. The fix: a PriorityQueue ordered by next run time
A PriorityQueue always gives you its smallest item first; with Comparator.comparingLong(t -> t.nextRun) the smallest is the earliest task. Inside it is a binary heap, a tree kept partly sorted, so add and poll cost O(log n) and peek is O(1).
- Recurring: after a run,
nextRun += period, then add the task back. - Cancellation: don't search the heap; just mark the task
cancelledand skip it when it comes out (this is called lazy deletion). - A fake clock: instead of sleeping, jump the clock straight to the next task's time. The output is exact and the test is instant.
import java.util.*;
class Task {
final String name; long nextRun; final long period; boolean cancelled; // period 0 = once
Task(String n, long next, long period) { name = n; nextRun = next; this.period = period; }
}
class Scheduler { // a fake clock: we jump straight to the next task
long now = 0;
final PriorityQueue<Task> queue = new PriorityQueue<>(Comparator.comparingLong(t -> t.nextRun));
Task at(String name, long time) { return add(new Task(name, time, 0)); } // one-time
Task after(String name, long delay) { return add(new Task(name, now + delay, 0)); } // delayed
Task every(String name, long period) { return add(new Task(name, now + period, period)); } // recurring
Task add(Task t) { queue.add(t); return t; }
void runUntil(long end) {
while (!queue.isEmpty() && queue.peek().nextRun <= end) {
Task t = queue.poll(); // the EARLIEST task, whatever the order it was added
now = t.nextRun;
if (t.cancelled) { System.out.printf("%4d ms (skip %s: cancelled)%n", now, t.name); continue; }
System.out.printf("%4d ms %s%n", now, t.name);
if (t.period > 0) { t.nextRun += t.period; queue.add(t); } } } // recurring: put it back
}
public class Main { public static void main(String[] args) {
Scheduler s = new Scheduler();
s.at("daily-report", 600); s.after("send-otp", 100);
Task beat = s.every("heartbeat", 250);
s.runUntil(700);
beat.cancelled = true; // cancellation: just mark it
System.out.println("-- heartbeat cancelled at 700 ms --");
s.runUntil(1200);
System.out.println("tasks left in the queue: " + s.queue.size()); } }
Output:
100 ms send-otp
250 ms heartbeat
500 ms heartbeat
600 ms daily-report
-- heartbeat cancelled at 700 ms --
750 ms (skip heartbeat: cancelled)
tasks left in the queue: 0
Time order, whatever the order of adding; the cancelled heartbeat comes out at 750 ms, is skipped and not put back.
5. Real time: DelayQueue and worker threads
With a real clock, a loop must sleep until the earliest task. Two new problems:
- A new task that is due earlier can arrive while the loop sleeps; nobody wakes the loop, so it runs late.
- If the loop runs tasks itself, one slow task blocks every task behind it.
Fixes:
- DelayQueue (
java.util.concurrent): a thread-safe priority queue ofDelayeditems. Each item answersgetDelay("how long until I'm due?").take()blocks (waits) until the head is due, and wakes up early if an earlier item is added. - Worker threads: the scheduler thread only takes due tasks and hands them to a thread pool (
Executors.newFixedThreadPool(2)), a fixed set of threads that run submitted tasks. The clock never waits for the work.
| At the railway station | In the scheduler |
|---|---|
| A timetable sorted by departure time | The priority queue by next run time |
| The station master waits; a special early train? he looks again | DelayQueue.take() blocks and wakes early |
| Porters load the trains | Worker threads |
| "Train cancelled" on the board | Mark the task cancelled; skip it |
import java.util.concurrent.*;
class Job implements Delayed { // DelayQueue asks each job: how long until you're due?
final String name; final long runAt;
Job(String name, long delayMs) { this.name = name; runAt = System.nanoTime() + delayMs * 1_000_000; }
public long getDelay(TimeUnit unit) { return unit.convert(runAt - System.nanoTime(), TimeUnit.NANOSECONDS); }
public int compareTo(Delayed o) { return Long.compare(runAt, ((Job) o).runAt); }
}
public class Main { public static void main(String[] args) throws Exception {
long start = System.nanoTime();
DelayQueue<Job> queue = new DelayQueue<>();
ExecutorService workers = Executors.newFixedThreadPool(2); // worker threads do the real work
Thread scheduler = new Thread(() -> {
try { while (true) {
Job job = queue.take(); // BLOCKS until the earliest job is due
if (job.name.equals("STOP")) break;
workers.submit(() -> System.out.printf("%-12s ran at %3d ms on %s%n", job.name,
(System.nanoTime() - start) / 1_000_000, Thread.currentThread().getName()));
} } catch (InterruptedException e) { }
});
scheduler.start();
queue.put(new Job("daily-report", 600)); // the scheduler now waits for 600 ms...
Thread.sleep(50);
queue.put(new Job("send-otp", 100)); // ...but this one, added at 50 ms, is due at 150
queue.put(new Job("reminder", 300));
queue.put(new Job("STOP", 800));
scheduler.join(); workers.shutdown(); workers.awaitTermination(1, TimeUnit.SECONDS);
} }
Output (one run):
send-otp ran at 168 ms on pool-1-thread-1
reminder ran at 362 ms on pool-1-thread-2
daily-report ran at 612 ms on pool-1-thread-1
send-otp was added at 50 ms while the scheduler was waiting for the 600 ms report, and still ran first, at about 150 ms.
6. Java's ready-made scheduler: ScheduledExecutorService
Executors.newScheduledThreadPool(n) gives a ScheduledExecutorService: a delay queue plus n worker threads.
| Method | Meaning |
|---|---|
schedule(task, delay, unit) |
A delayed, one-time task |
scheduleAtFixedRate(task, initialDelay, period, unit) |
Start a run every period, by the clock (0, 200, 400, ...) |
scheduleWithFixedDelay(task, initialDelay, delay, unit) |
Wait delay after each run ends, then run again |
ScheduledFuture.cancel(false) |
Stop future runs (false: don't interrupt a run in progress) |
Missed execution: a run is due but every thread is busy, so it starts late.
import java.util.concurrent.*;
public class Main {
static long start;
static void log(String what) {
System.out.printf(" %4d ms %s%n", (System.nanoTime() - start) / 1_000_000 / 10 * 10, what); }
static void demo(int threads) throws Exception {
System.out.println("pool of " + threads + " thread(s):");
ScheduledExecutorService ses = Executors.newScheduledThreadPool(threads);
start = System.nanoTime();
ses.schedule(() -> log("send-otp (one-time)"), 100, TimeUnit.MILLISECONDS);
ScheduledFuture<?> beat = ses.scheduleAtFixedRate(() -> log("heartbeat"), 0, 200, TimeUnit.MILLISECONDS);
ses.schedule(() -> { log("backup starts, takes 500 ms");
try { Thread.sleep(500); } catch (InterruptedException e) { }
log("backup ends"); }, 250, TimeUnit.MILLISECONDS);
ses.schedule(() -> { beat.cancel(false); log("heartbeat cancelled"); }, 1050, TimeUnit.MILLISECONDS);
Thread.sleep(1200); ses.shutdown();
}
public static void main(String[] args) throws Exception { demo(1); demo(2); }
}
Output (one run; times rounded down to 10 ms):
pool of 1 thread(s):
10 ms heartbeat
100 ms send-otp (one-time)
200 ms heartbeat
250 ms backup starts, takes 500 ms
750 ms backup ends
750 ms heartbeat
750 ms heartbeat
800 ms heartbeat
1000 ms heartbeat
1060 ms heartbeat cancelled
pool of 2 thread(s):
0 ms heartbeat
100 ms send-otp (one-time)
200 ms heartbeat
250 ms backup starts, takes 500 ms
400 ms heartbeat
600 ms heartbeat
750 ms backup ends
800 ms heartbeat
1000 ms heartbeat
1050 ms heartbeat cancelled
With one thread, the 500 ms backup holds it from 250 to 750 ms. The heartbeats due at 400 and 600 are missed, then both fire late, back to back. With two threads every heartbeat is on time.
What the Javadoc says about scheduleAtFixedRate: "If any execution of this task takes longer than its period, then subsequent executions may start late, but will not concurrently execute." And if an execution throws an exception, "Subsequent executions are suppressed": the task silently stops. Real run:
import java.util.concurrent.*;
public class Main { public static void main(String[] a) throws Exception {
ScheduledExecutorService ses = Executors.newScheduledThreadPool(1);
int[] n = {0};
ScheduledFuture<?> f = ses.scheduleAtFixedRate(() -> { n[0]++; System.out.println("run " + n[0]);
if (n[0] == 2) throw new IllegalStateException("disk full"); }, 0, 50, TimeUnit.MILLISECONDS);
Thread.sleep(300);
System.out.println("isDone=" + f.isDone() + " runs=" + n[0]);
try { f.get(); } catch (ExecutionException e) { System.out.println("cause: " + e.getCause()); }
ses.shutdown(); } }
run 1
run 2
isDone=true runs=2
cause: java.lang.IllegalStateException: disk full
So always catch exceptions inside a recurring task and log them.
7. Retries with exponential backoff
- Retry: run a failed task again.
- Retrying at once usually fails again (the service is still down) and hammers it.
- Backoff: wait longer before each try. Exponential backoff doubles the wait: base, 2×base, 4×base, 8×base...
- Max attempts, then a dead-letter list: failed tasks parked for a human to look at.
- Jitter: a small random extra wait, so a thousand clients that failed together don't retry at the same instant.
import java.util.*;
public class Main {
static boolean gatewayUp(long now) { return now >= 250; } // the payment gateway is down until 250 ms
static void attempt(String how, long[] times) {
for (int a = 0; a < times.length; a++) {
boolean ok = gatewayUp(times[a]);
System.out.printf(" %-8s try %d at %4d ms: %s%n", how, a + 1, times[a], ok ? "charged" : "failed");
if (ok) return;
}
System.out.println(" " + how + ": gave up after " + times.length + " tries -> dead-letter list");
}
static long[] backoff(int maxTries, long base) { // wait base, 2*base, 4*base, ...
long[] t = new long[maxTries]; long wait = base;
for (int a = 1; a < maxTries; a++) { t[a] = t[a - 1] + wait; wait *= 2; }
return t;
}
public static void main(String[] args) {
System.out.println("retry at once, 4 tries:");
attempt("instant", new long[]{0, 0, 0, 0}); // attempt: hammer it straight away
System.out.println("exponential backoff, base 50 ms, 4 tries:");
attempt("backoff", backoff(4, 50));
System.out.println("backoff times for 6 tries: " + Arrays.toString(backoff(6, 50)));
}
}
Output:
retry at once, 4 tries:
instant try 1 at 0 ms: failed
instant try 2 at 0 ms: failed
instant try 3 at 0 ms: failed
instant try 4 at 0 ms: failed
instant: gave up after 4 tries -> dead-letter list
exponential backoff, base 50 ms, 4 tries:
backoff try 1 at 0 ms: failed
backoff try 2 at 50 ms: failed
backoff try 3 at 150 ms: failed
backoff try 4 at 350 ms: charged
backoff times for 6 tries: [0, 50, 150, 350, 750, 1550]
The try times are the running sum of the waits: 0, 0+50, 50+100, 150+200, ...
8. Cron expressions
A fixed period can't say "every 15 minutes, 9 to 5, Monday to Friday". A cron expression (from the Unix tool cron) has five fields: minute, hour, day of month, month, day of week.
| Syntax | Meaning | Example |
|---|---|---|
* |
every value | * in the hour field: every hour |
a-b |
a range, both ends included | 9-17: 9 AM to 5 PM |
*/n |
every n-th value | */15 in minutes: 0, 15, 30, 45 |
a,b |
a list | 1,15: the 1st and the 15th |
From the crontab(5) manual: minute 0-59, hour 0-23, day of month 1-31, month 1-12, day of week 0-7 where 0 or 7 is Sunday (names like mon also work); ranges are inclusive, and /<number> "specifies skips of the number's value through the range". Our small parser uses 1 = Monday to 7 = Sunday (Java's DayOfWeek), so 1-5 means Monday to Friday in both.
To find the next run: start one minute after now and step forward a minute at a time until all five fields match (fine for an interview; real libraries jump field by field).
Missed executions after downtime: if the scheduler was down, some cron runs were missed. Three common policies:
- FIRE_ONCE: run once now, then continue normally (good for "send the report").
- RUN_ALL: run every missed run (good when each run matters, e.g. billing per hour).
- SKIP: ignore them and wait for the next one (good for a heartbeat).
import java.time.*; import java.util.*;
class Cron { // 5 fields: minute hour day-of-month month day-of-week (1 = Monday)
static final int[][] RANGE = {{0, 59}, {0, 23}, {1, 31}, {1, 12}, {1, 7}};
final List<Set<Integer>> fields = new ArrayList<>();
Cron(String expr) { String[] f = expr.trim().split("\\s+");
for (int i = 0; i < 5; i++) fields.add(parse(f[i], RANGE[i][0], RANGE[i][1])); }
static Set<Integer> parse(String field, int lo, int hi) { // * */15 9-17 1,15
Set<Integer> out = new TreeSet<>();
for (String part : field.split(",")) {
String[] s = part.split("/"); int step = s.length > 1 ? Integer.parseInt(s[1]) : 1;
int a = lo, b = hi; String[] r = s[0].split("-");
if (r.length == 2) { a = Integer.parseInt(r[0]); b = Integer.parseInt(r[1]); }
else if (!s[0].equals("*")) a = b = Integer.parseInt(s[0]);
for (int v = a; v <= b; v += step) out.add(v); } return out; }
boolean matches(LocalDateTime t) {
int[] v = {t.getMinute(), t.getHour(), t.getDayOfMonth(), t.getMonthValue(), t.getDayOfWeek().getValue()};
for (int i = 0; i < 5; i++) if (!fields.get(i).contains(v[i])) return false; return true; }
LocalDateTime next(LocalDateTime after) { // step minute by minute
LocalDateTime t = after.withSecond(0).withNano(0).plusMinutes(1);
while (!matches(t)) t = t.plusMinutes(1); return t; } }
public class Main { public static void main(String[] args) {
Cron c = new Cron("*/15 9-17 * * 1-5"); // every 15 min, 9 to 5, Monday to Friday
LocalDateTime t = LocalDateTime.of(2026, 11, 13, 17, 20); // Friday 17:20
for (int i = 0; i < 3; i++) { t = c.next(t); System.out.println("next run: " + t + " " + t.getDayOfWeek()); }
LocalDateTime lastRun = LocalDateTime.of(2026, 11, 16, 9, 45), back = LocalDateTime.of(2026, 11, 16, 10, 50);
List<LocalDateTime> missed = new ArrayList<>(); // the scheduler was down 9:45 -> 10:50
for (LocalDateTime m = c.next(lastRun); !m.isAfter(back); m = c.next(m)) missed.add(m);
System.out.println("missed while down: " + missed.size() + " " + missed);
System.out.println("policy FIRE_ONCE: run 1 now, RUN_ALL: run " + missed.size() + ", SKIP: run 0"); } }
Output:
next run: 2026-11-13T17:30 FRIDAY
next run: 2026-11-13T17:45 FRIDAY
next run: 2026-11-16T09:00 MONDAY
missed while down: 4 [2026-11-16T10:00, 2026-11-16T10:15, 2026-11-16T10:30, 2026-11-16T10:45]
policy FIRE_ONCE: run 1 now, RUN_ALL: run 4, SKIP: run 0
The weekend is skipped (Friday 17:45 → Monday 09:00); 4 runs were missed between 9:45 and 10:50.
9. The class diagram
Client ─ at(time) · after(delay) · every(period) · cron("*/15 9-17 * * 1-5") ─▶ Scheduler
Task: name, nextRun, period | cron, attempts, cancelled
Scheduler: DelayQueue<Task> (a priority queue by nextRun that blocks until due)
scheduler thread: take() ─▶ hand the task to a worker pool (worker threads)
recurring: nextRun += period (or cron.next) and put it back
failure: retry at now + base × 2^attempt, up to N tries, then a dead-letter list
missed runs after downtime: FIRE_ONCE · RUN_ALL · SKIP cancel: mark, skip on take
10. Interview follow-ups
| Question | Answer |
|---|---|
| Many machines run the scheduler | Take a lock (e.g. a database row lock) so only one machine runs each task |
| Survive a restart | Store tasks, next run, last run and attempts in a database; the last run tells you what was missed |
| Millions of tasks | Shard tasks by id across scheduler machines; or a timing wheel (buckets by time) |
| A task runs too long | A timeout per task; cancel it with Future.cancel(true) |
| Two runs of one task overlap | Fixed-rate in Java never overlaps; across machines, use the lock |
| Priority between tasks due at the same time | Compare by (nextRun, priority) |
11. Common mistakes
| Mistake | What you see (real message) | Why | Fix |
|---|---|---|---|
| A FIFO queue | send-otp wanted 100 ms, ran at ~600 ms |
Added order ≠ due order | PriorityQueue by next run time |
A PriorityQueue of objects with no comparator |
java.lang.ClassCastException: class Task cannot be cast to class java.lang.Comparable |
It doesn't know how to order them | new PriorityQueue<>(Comparator.comparingLong(t -> t.at)) |
Thread.sleep without handling the exception |
error: unreported exception InterruptedException; must be caught or declared to be thrown |
A checked exception | throws Exception or try/catch |
| Sleeping until the earliest task | A new earlier task runs late | Nobody wakes the loop | DelayQueue.take() |
| One thread for slow and fast tasks | Two late heartbeats, back to back | Missed executions | More worker threads |
| An exception in a fixed-rate task | isDone=true runs=2 and nothing more |
"Subsequent executions are suppressed" | Catch inside the task |
| Retrying at once | gave up after 4 tries |
The service is still down | Exponential backoff + jitter |
Forgetting shutdown() |
The program never exits | Pool threads keep the JVM alive | shutdown() + awaitTermination |
Mini project: a real-time scheduler
A DelayQueue, a scheduler thread and a worker pool; delayed and recurring tasks; cancellation; retries with exponential backoff and seeded jitter; a dead-letter list; an exception never kills a recurring task.
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicInteger;
// Mini project: a real-time task scheduler. One-time, delayed and recurring tasks on a DelayQueue,
// a scheduler thread + a worker pool, cancellation, retries with exponential backoff and jitter,
// a dead-letter list, and exceptions that never kill a recurring task.
interface Job { void run() throws Exception; }
class ScheduledTask implements Delayed {
final String name; final Job job; final long period; final int maxTries;
volatile long runAt; volatile boolean cancelled; int attempt = 1;
ScheduledTask(String name, Job job, long runAt, long period, int maxTries) {
this.name = name; this.job = job; this.runAt = runAt; this.period = period; this.maxTries = maxTries;
}
public long getDelay(TimeUnit u) { return u.convert(runAt - System.nanoTime(), TimeUnit.NANOSECONDS); }
public int compareTo(Delayed o) { return Long.compare(runAt, ((ScheduledTask) o).runAt); }
void cancel() { cancelled = true; }
}
class TaskScheduler {
private final DelayQueue<ScheduledTask> queue = new DelayQueue<>();
private final ExecutorService workers;
private final Thread loop;
private final Random random = new Random(42); // seeded, so the jitter is the same every run
final List<String> deadLetters = new CopyOnWriteArrayList<>();
private final long start = System.nanoTime();
TaskScheduler(int workerCount) {
workers = Executors.newFixedThreadPool(workerCount);
loop = new Thread(() -> {
try {
while (true) {
ScheduledTask t = queue.take(); // blocks until the earliest task is due
if (t.name.equals("__stop")) return;
if (t.cancelled) continue; // cancellation: skip it
workers.submit(() -> execute(t));
}
} catch (InterruptedException e) { Thread.currentThread().interrupt(); }
}, "scheduler");
loop.start();
}
long ms() { return (System.nanoTime() - start) / 1_000_000; }
private static long nanos(long ms) { return ms * 1_000_000; }
ScheduledTask after(String name, long delayMs, Job job) { return add(new ScheduledTask(name, job, System.nanoTime() + nanos(delayMs), 0, 3)); }
ScheduledTask every(String name, long periodMs, Job job) { return add(new ScheduledTask(name, job, System.nanoTime() + nanos(periodMs), periodMs, 1)); }
private ScheduledTask add(ScheduledTask t) { queue.put(t); return t; }
private void execute(ScheduledTask t) {
try {
t.job.run();
System.out.printf("%4d ms %-10s ok (try %d)%n", ms(), t.name, t.attempt);
t.attempt = 1;
} catch (Exception e) { // never let an exception kill the schedule
System.out.printf("%4d ms %-10s failed (try %d): %s%n", ms(), t.name, t.attempt, e.getMessage());
if (t.period == 0 && t.attempt < t.maxTries) { // retry with exponential backoff + jitter
long backoff = 50L << (t.attempt - 1); // 50, 100, 200, ...
long jitter = random.nextInt((int) (backoff / 5) + 1); // up to 20% extra
t.attempt++;
t.runAt = System.nanoTime() + nanos(backoff + jitter);
queue.put(t);
return;
}
if (t.period == 0) { deadLetters.add(t.name); return; }
}
if (t.period > 0 && !t.cancelled) { t.runAt += nanos(t.period); queue.put(t); } // recurring
}
void shutdown(long afterMs) throws InterruptedException {
queue.put(new ScheduledTask("__stop", () -> {}, System.nanoTime() + nanos(afterMs), 0, 1));
loop.join();
workers.shutdown();
workers.awaitTermination(1, TimeUnit.SECONDS);
}
}
public class Main {
public static void main(String[] args) throws Exception {
TaskScheduler s = new TaskScheduler(2);
long gatewayUpAt = 180;
AtomicInteger beats = new AtomicInteger();
s.after("send-otp", 30, () -> {});
s.after("charge", 20, () -> { if (s.ms() < gatewayUpAt) throw new Exception("gateway down"); });
s.after("refund", 40, () -> { throw new Exception("bank says no"); });
ScheduledTask beat = s.every("heartbeat", 100, () -> {
if (beats.incrementAndGet() == 2) throw new Exception("one bad beat");
});
s.after("cancel-hb", 450, beat::cancel);
s.shutdown(700);
System.out.println("heartbeats run: " + beats.get());
System.out.println("dead letters: " + s.deadLetters);
}
}
Output (one run):
35 ms charge failed (try 1): gateway down
41 ms send-otp ok (try 1)
56 ms refund failed (try 1): bank says no
113 ms refund failed (try 2): bank says no
113 ms charge failed (try 2): gateway down
115 ms heartbeat ok (try 1)
215 ms heartbeat failed (try 1): one bad beat
220 ms refund failed (try 3): bank says no
232 ms charge ok (try 3)
315 ms heartbeat ok (try 1)
415 ms heartbeat ok (try 1)
466 ms cancel-hb ok (try 1)
heartbeats run: 4
dead letters: [refund]
charge succeeds on try 3 once the gateway is up; refund fails 3 times and lands in the dead letters; the failing heartbeat run doesn't stop the heartbeat; the heartbeat is cancelled at about 450 ms.
Practice
1. Add jitter: a random 0 to 20% extra wait.
Answer
long jitter = random.nextInt((int) (backoff / 5) + 1); then wait backoff + jitter. Seed the Random in tests (new Random(42)) so runs repeat. See TaskScheduler.execute in the mini project.
2. Use cron.next() for recurring tasks.
Answer
Store a Cron in the task instead of a period. After each run: task.nextRun = cron.next(task.nextRun), then put it back in the queue. With the episode's Cron, */15 9-17 * * 1-5 after Friday 17:45 gives Monday 09:00.
3. Implement FIRE_ONCE for missed cron runs.
Answer
On start, compute missed from the stored last run up to now (as in the episode). If it isn't empty, run the task once now, store lastRun = now, and set nextRun = cron.next(now).
4. Wrap every recurring task so an exception can't stop it.
Answer
Runnable safe = () -> { try { task.run(); } catch (Exception e) { log(e); } }; and schedule safe. Without it, the Javadoc says subsequent executions are suppressed (see the disk full run).
5. What is the difference between scheduleAtFixedRate and scheduleWithFixedDelay for a task that takes 300 ms with a period/delay of 1000 ms?
Answer
Fixed rate starts at 0, 1000, 2000 ms (by the clock). Fixed delay starts at 0, 1300, 2600 ms: 1000 ms after each run ends.
Quick quiz
Exponential backoff, base wait 100 ms, the wait doubles, try 1 at 0 ms. When does try 5 start? (a) 400 ms (b) 500 ms (c) 1500 ms (d) 1600 ms
Answer
(c) 1500 ms. The waits are 100, 200, 400, 800, so the tries start at 0, 100, 300, 700 and 1500 ms. The episode's backoff(5, 100) gives [0, 100, 300, 700, 1500].
Interview questions
- Which data structure for a scheduler, and why? A priority queue (heap) by next run time: earliest first, O(log n) add and poll.
- How do you cancel a task? Mark it cancelled and skip it when it is taken (lazy deletion), instead of removing it from the heap.
- Why a DelayQueue instead of sleep?
take()blocks until the head is due and wakes up when an earlier task is added. - Why worker threads? So a slow task never delays the clock or the tasks behind it.
- Fixed rate vs fixed delay? Fixed rate keeps a clock rhythm (may run late, never overlaps); fixed delay waits after each run.
- What happens if a fixed-rate task throws? Later runs are suppressed; catch exceptions inside.
- What is exponential backoff and why add jitter? Doubling waits between retries; jitter spreads retries so clients don't retry together.
- What goes to a dead-letter list? Tasks that failed all their attempts, for a human or a later job.
- How do you compute the next cron run? Step forward from now until all five fields match (or jump field by field).
- How do you handle missed runs after downtime? A per-task policy: fire once, run all, or skip.
Glossary
| Term | Meaning, with an example |
|---|---|
| Task | Work plus when to run it: send-otp at 100 ms |
| Task scheduler | Keeps tasks and runs each on time |
| One-time task | Runs once at a fixed time: 6 AM tomorrow |
| Delayed task | Runs once after a delay from now: in 10 minutes |
| Recurring task | Runs every period: every 8 hours |
| Period | The gap between runs of a recurring task |
| Next run time | When a task should run next |
| FIFO | First in, first out: ArrayDeque used as a queue |
| PriorityQueue | A queue that returns the smallest item first |
| Comparator | An object that says how to order two items: Comparator.comparingLong(t -> t.nextRun) |
| Binary heap | A tree kept partly sorted; the smallest is at the top |
| O(log n) | Work that grows very slowly: 1,000,000 items → about 20 steps |
| Cancellation | Stopping a task before it runs |
| Lazy deletion | Mark as deleted; skip it when it comes out |
| Fake clock | A clock moved by hand, for exact tests |
| Thread | One line of work running inside a program |
| Blocking | A call that waits: take() waits until a task is due |
| DelayQueue | A thread-safe priority queue whose take() waits until the head is due |
| Delayed / getDelay | The interface DelayQueue items implement: "how long until I'm due?" |
| Worker threads | Threads that only run tasks handed to them |
| Thread pool | A fixed set of reusable threads: newFixedThreadPool(2) |
| ScheduledExecutorService | Java's scheduler: a delay queue + worker threads |
| scheduleAtFixedRate | Run every period by the clock |
| scheduleWithFixedDelay | Run, wait a delay after it ends, run again |
| ScheduledFuture | The handle a scheduled task returns; cancel(false) stops it |
| Missed execution | A run that couldn't start on time |
| Retry | Running a failed task again |
| Backoff | Waiting longer before each retry |
| Exponential backoff | The wait doubles: 50, 100, 200, 400 ms |
| Jitter | A small random extra wait |
| Dead-letter list | Where tasks that failed every try are parked |
| Cron | The Unix tool that runs commands on a schedule |
| Cron expression | Five fields: minute hour day month weekday: */15 9-17 * * 1-5 |
| Range / step / list | 9-17 / */15 / 1,15 in a cron field |
| FIRE_ONCE / RUN_ALL / SKIP | Policies for runs missed during downtime |
| Timing wheel | Buckets of tasks by time slot, for huge numbers of timers |
Summary
- A FIFO queue ran
send-otpabout half a second late: tasks must run by time, not by the order added. - A PriorityQueue by next run time gives the earliest task first; one-time, delayed and recurring tasks; cancel by marking.
- DelayQueue.take() waits until a task is due and wakes for earlier ones; worker threads do the work.
- ScheduledExecutorService packages this; a busy single thread causes missed executions; an exception in a fixed-rate task suppresses later runs.
- Retries with exponential backoff (and jitter), then a dead-letter list.
- Cron expressions say exactly when; a policy decides what to do with missed runs.
- Next episode: design a notification service.