That Time a Backup Script Ran for Six Months Without Actually Backing Up

True story. A client had a nightly database backup set up as a cron job. It ran every night at 2 AM. The logs said "completed." Everyone slept well.

Then they needed to restore from backup after a bad migration. Turns out the script had been silently failing since February. The exit code was 0 because the wrapper script caught the error and moved on. The "completed" log was from the wrapper, not the actual backup.

Six months of backups. All empty files.

Nobody checks cron output

That's kind of the whole problem with scheduled tasks. They run in the background. You set them up, they work for a while, and then something changes — a path moves, a disk fills up, permissions shift after an update — and the job starts failing. Silently.

Most teams don't have monitoring on their cron jobs. They monitor their web app, their API, maybe their database. But the nightly report generator? The cache cleanup? The SSL cert renewal check? Those just... run. Hopefully.

What actually works

The pattern I like is dead simple: your cron job pings an external endpoint when it finishes successfully. If the ping doesn't arrive within the expected window, you get an alert.

It's called heartbeat monitoring (or dead man's switch — same idea). The job has to actively prove it ran. No ping = something broke.

`# at the end of your backup script
curl -s https://monitor.example.com/ping/your-job-id`

One line. That's it. If that curl never fires, you know within minutes instead of finding out six months later when you actually need the backup.

What to watch for

A few gotchas from experience:

• Put the ping after the actual work, not before. I've seen scripts that ping first and then fail — defeats the purpose.

• Check exit codes. mysqldump && curl only pings if the dump succeeds.

• Set a grace period that makes sense. A job that usually takes 5 minutes might occasionally take 20. Don't alert on that.

The backup script story still bugs me. Not because it failed — stuff fails. But because nobody knew for half a year. That part is fixable.