Skip to main content
YC.Lin
← Back to blog
7 min readSystems & Infra

My Backup Reported Success Every Hour. It Had Never Succeeded Once.

My backup status file looked like this:

{
  "last_success": "2026-08-20T23:07:09",
  "consecutive_failures": 0,
  "alert": false
}

Updated every hour, for days. That backup had never once succeeded.

Not "failed occasionally" — it had never produced a single commit since the day I set it up. And every indicator I had, the status file, the task scheduler, the alerting logic I wrote myself, said everything was fine.

This isn't a post about the bugs I fixed. It's about what I noticed that night: whatever declares "success" and whatever actually does the work are separated by a layer of assumption that nobody ever verified. That layer almost certainly exists in your system too.

First, take "success" apart

Before anything else you need a ruler. I eventually realised that "the backup succeeded" was quietly conflating three different claims:

  1. The command succeeded — the process I invoked returned 0
  2. The job succeeded — everything this run was supposed to do got done
  3. The data is off-site and restorable

I thought I was monitoring layer 3. I was monitoring layer 1, and I was watching the wrong command at that.

The four things below all happened in one night. I've ordered them by how far each one sits from layer 3, not by when they happened.

1. That exit code belongs to a different step

Four lines from the backup script:

_, out = git("commit", "-m", msg)          # exit code discarded
code, out = git("push", "origin", "main")
if code != 0:
    raise RuntimeError(f"push failed: {out}")

Git identity was never configured on that machine. Every commit returned exit code 128, Author identity unknown.

But the push on the next line returned 0, correctly, because there was nothing to push.

I only checked the second exit code. So every hourly backup "succeeded" while files piled up in the staging area, uncommitted.

This is the simplest of the four, and the only place I'll allow myself a self-deprecating aside: that discarded _ is the whole article in one character.

The real fix isn't "remember to check return values" — that's too cheap. The fix is to verify the end state, not the exit code of the last command:

local  = git("rev-parse", "HEAD")
remote = git("rev-parse", "origin/main")
if local != remote:
    raise RuntimeError("push returned 0, but the remote HEAD didn't move")

A push returning 0 means git didn't error. It does not mean the remote has your data. And having the data on the remote is the entire point of a backup.

2. The one time the system told the truth, something else erased it

The second one starts with an honest failure.

The first cloud upload was 14.5 GB. I wrapped rclone like this:

subprocess.run([...], timeout=21600)   # 6 hours

Six hours in, 8.86 GB transferred, TimeoutExpired. The status file honestly recorded a failure.

The problem is that a subprocess timeout is a kill, not a wind-down. It signals the process regardless of where rclone is in the current file. Six hours of transfer ended with no chance to finish cleanly.

The fix isn't a longer timeout — that just defers the problem until the dataset grows. The fix is to let the tool watch its own clock:

rclone("copy", src, dst, "--max-duration", "10h", ...)

rclone finishes the file it's holding, then exits with code 10. The next run picks up where it left off. And the outcome has to go from two states to three:

complete  /  failed  /  partial

"Partial" is neither a failure nor a success. Without a name for it, "runs daily, never finishes" continues silently forever.

So far, so good — the system told the truth. The problem came next.

That failure was the run that started at 00:05 and got killed at 06:06. The schedule fires daily at 03:00, with MultipleInstances=IgnoreNew — if an instance is already running, skip the new trigger. (Kubernetes calls this concurrencyPolicy: Forbid.)

The 03:00 trigger was skipped. And a skipped trigger is not a failure as far as Task Scheduler is concerned — it leaves a 0 in LastTaskResult.

So the next morning I saw: task state Ready, last result 0.

The one time the system told the truth, it was overwritten by a 0 that meant "did nothing".

I believed the backup had succeeded.

I did this debugging alongside an AI assistant, and it reached that wrong conclusion first — it read LastTaskResult = 0 off the task scheduler, reported "the upload finished", and I took it. Only on opening the status file did I see last_success was null and consecutive_failures was 1.

I'm leaving that in because it demonstrates what this trap actually catches. It doesn't catch careless people. It catches the act of stopping at the first authoritative-looking signal. Humans do that. So do machines.

The fix is to be explicit about authority: for any job that maintains its own status file, the status file decides success or failure. LastTaskResult is only good for "was it triggered". The success your OS reports is success at its own layer — it has no idea whether your job finished.

3. My verifier and my backup used the same glob

The first two were, at bottom, me failing to propagate errors. This one is different in kind.

My knowledge base indexes live in two possible layouts. The actual precedence in the code is:

db = new_path if new_path.exists() else legacy_path

Use the new location if it exists, otherwise fall back to the legacy one. Which means the legacy location is still live, readable and writable.

My backup script picked files like this:

databases = glob("data/indexes/*/knowledge.db")

New layout only. Anything in the legacy location was outside the backup entirely — with no symptom whatsoever. It still showed up in the namespace list. It was still searchable.

That's not the bad part. The bad part is the restore verification script I wrote afterwards, which opens with:

databases = glob("data/indexes/*/knowledge.db")

The same line.

So all that verification could ever prove was "the things the glob finds are restorable". It was structurally incapable of finding the gap, because it shared an assumption with the thing it was verifying.

Verifying a backup using the backup's own assumptions proves the two agree. It doesn't prove either is right.

What was actually on disk that night: 10 databases in the legacy location, 9 of which also existed in the new location with more rows (pre-migration snapshots), and the one that existed only in the legacy location happened to hold zero entries.

So nothing was lost. But that was luck, not design.

The fix is to give "which data needs backing up" an authoritative source independent of the backup script, have both sides ask it, and make the verifier explicitly diff the two — what does production have that the backup doesn't?

An aside: a read-only query that left something on disk

One more thing from the same night, and it's the mirror image of the above.

I typo'd a namespace — 個人-ai-cli-mcp instead of ai-cli-mcp — and ran a query. A read-only operation.

A zero-row database appeared on disk.

The root cause isn't the typo. It's that _db_path() mixed two responsibilities: computing a path, and creating it if absent. And sqlite3.connect() creates the file by itself — so this has to be blocked before connecting. Cleaning up afterwards is too late.

The consequence of that empty database: the next day's cloud backup would fail its integrity check on a zero-row snapshot and abort the entire off-site backup. One typo, discovered two days later.

There's an honest irony here: that integrity check did work. It just reported the wrong reason.

I'm not ending this with "and then I fixed everything"

All four are fixed, with tests. But ending on "everything is fine now" would commit exactly the error this whole post is about.

The question worth asking is the other one: which of the green lights I'm still looking at are produced by the same mechanism?

Four questions I now keep:

  1. Does every step on this path that can fail have its exit code checked? Or only the last one?
  2. Whose success am I looking at — the OS's, the tool's, or my job's?
  3. When a long-running job is interrupted, is the work lost or resumable? Can it say how much is left?
  4. Could my verifier be making the same mistake as the thing it verifies?

And one more thing, honestly.

Not one of these four was caught by alerting I had designed. Every one of them I found because I happened to be looking, and happened to read one more line of the status file.

The entire reason a backup exists is that nobody is looking.