It's 11pm. An engineer at GitLab — a code-hosting SaaS that had raised around $25 million at the time — is fighting a database problem that won't die. A copy of the main database has fallen behind and needs to be wiped and rebuilt. He types the delete command.
He runs it on the wrong server.
The command hits the live production database — the one holding real customer data. He realizes within about two seconds and kills it. Too late. Roughly 300GB of live customer data is gone.
Painful — but survivable, right? This is exactly what backups are for. GitLab had five separate ways to get that data back.
Then came the worst hour of the night.
━━━━━━━━━━━━━━━
🧯 Five backups. Five failures.
One by one, the team reached for each safety net. One by one, each gave way:
❌ The scheduled database export had been silently failing for weeks. The backup software version didn't match the database version, so the export produced nothing — every single day.
❌ The failure alerts never arrived. The emails warning that backups were failing were silently rejected by an email security setting. Nobody ever saw one.
❌ The cloud storage backup bucket was empty. The team's own live notes that night recorded that the S3 backups apparently didn't work either.
❌ Disk snapshots weren't turned on for the database servers. They assumed the other backup methods were enough, so they never enabled them.
❌ The standby copy of the database was empty too — wiping it was part of the repair attempt that started the whole night.
In their own incident notes, GitLab admitted that none of the five backup and replication methods they had deployed were working reliably — some had never been set up properly in the first place.
What finally saved them was luck: that same engineer had manually taken a copy six hours earlier for an unrelated task. They rebuilt from that.
The bill for the night: six hours of customer data lost forever — GitLab's own estimate is that it hit roughly 5,000 projects, 5,000 comments and 700 new user accounts — plus around 18 hours of downtime, with the recovery livestreamed to thousands of viewers.
💬 A company with 100 employees and five backup systems lost customer data — not because it had no backups, but because nobody had ever proved a single one of them could be restored.
━━━━━━━━━━━━━━━
😰 Why this should worry a solo founder more than it worried GitLab
GitLab survived because it had funding, a team, goodwill — and one lucky manual copy. Your solo SaaS has none of those cushions. If your data is gone, your product is gone, and so is every customer's trust on the same day.
And the odds are worse than most founders think. Veeam — one of the biggest backup software companies — surveyed over 3,000 enterprise IT decision makers and found that 58% of backups fail. (Scope: that's 58% of backup operations across the organizations surveyed — failing either because the backup job ends with errors, or because the restore doesn't work when attempted.) Those are companies with dedicated IT staff. A solo founder with a checkbox ticked in a dashboard is not beating those odds.
Here's the uncomfortable truth the GitLab night teaches: backups fail silently. Restores are where you find out. And if the first restore you ever attempt is during a real disaster, you've chosen the worst possible moment to run the test.
📦 What's a restore drill? It's a fire drill for your data. You take a real backup file and actually rebuild a working copy of your app's data from it — on a throwaway copy, never on your live system — then check the data is really all there. The whole point is to find the broken step on a calm Tuesday instead of during a disaster.
━━━━━━━━━━━━━━━
🔍 Five questions to ask your backup setup tonight
Each of these maps to one of GitLab's five failures. Answer honestly — every "I think so" counts as a no.
🔍 1️⃣ EXISTS — does a backup actually exist?
Not "should exist" — exists. Go look at the actual file or snapshot list right now. This bites hardest on modern indie stacks: Supabase's free plan takes zero automated backups — daily backups start on the paid Pro plan ($25/month per project), and Supabase itself recommends free-plan projects regularly export their own copies. If you're on a free database tier and never set up your own export, your answer is already no.
🕐 2️⃣ FRESH — when did the last one actually succeed?
GitLab's export had been failing every day for weeks, and the alert emails were eaten silently. So don't check whether the job is scheduled — check the newest file's date and size. A backup file dated three weeks ago, or one that's suspiciously tiny (2 KB when your database is 200 MB), is a failed backup that looks successful.
🧭 3️⃣ ELSEWHERE — does a copy live somewhere else?
If your only backup sits in the same account as your live database, one deleted account — or one wrong click — takes both. Keep at least one copy somewhere your production login can't touch: a different cloud storage account, or even a folder on your own machine that syncs to a personal drive.
🧩 4️⃣ COMPLETE — what does your backup NOT cover?
This is the trap almost nobody checks. On Supabase, a database restore brings back your database only — uploaded files (user avatars, attachments) live in Storage and do not come back. After a restore, your app can happily point at images that no longer exist. The same idea applies on any stack: environment secrets, uploaded files, and third-party data (like your payment provider's records) usually sit outside the database backup. Write down what yours misses.
⚡ 5️⃣ RESTORED — have you ever actually restored one?
If the answer is no, then everything above is theory. An untested backup is a hope, not a plan. Which brings us to the drill.
━━━━━━━━━━━━━━━
🕐 The 30-minute restore drill — worked example on Supabase
My own product, IVP, runs on Supabase — so I'll use that stack for the walkthrough. The drill is five moves — dump, inspect, rebuild elsewhere, load, compare — and each step below is named after its move, so the same drill translates to any Postgres, MySQL, or MongoDB setup.
👉 Step 1 — DUMP: make tonight's backup (5 min). Grab your database connection string from your dashboard (in Supabase: Settings → Database). Then run the export from your terminal: Supabase's CLI wraps this into one command —
supabase db dump -f backup.sql --db-url "YOUR_CONNECTION_STRING". You get a plain text file of your entire database.👉 Step 2 — INSPECT: open the file and look (2 min). Is the file size sensible for your data? Open it in any text editor: do you see your real table names and real rows? A 2 KB file means your backup is a checkmark, not a backup — you just caught GitLab's weeks-of-silent-failure problem in two minutes.
👉 Step 3 — REBUILD ELSEWHERE: create a throwaway project (10 min). Spin up a second, free Supabase project — a blank one. This is your practice dummy. You never restore onto your live project during a drill.
👉 Step 4 — LOAD: fill the throwaway with your backup (8 min). Point the standard Postgres command at the throwaway project's connection string:
psql "THROWAWAY_CONNECTION_STRING" -f backup.sql. Watch it rebuild your tables from nothing.👉 Step 5 — COMPARE: verify like a skeptic (5 min). Pick your three most important tables (say: users, projects, payments). Count the rows in the throwaway and in production — the counts should match. Then open two or three real records and eyeball them. Done: you have now proved your backup restores.
⏰ Worked time budget: 5 + 2 + 10 + 8 + 5 = 30 minutes — and that's the slow, first-time run. The second time, steps 3 and 4 already exist as muscle memory and pasted commands, and the drill drops under 15 minutes.
Two follow-ups that turn one drill into a system:
✅ Put a repeating 30-minute event in your calendar — first Monday of every month, per product. GitLab's backups were fine once too. The drill only protects you while it keeps happening.
✅ Automate the export. A free scheduled job (a GitHub Actions workflow on a timer works) can run the dump command nightly and push the file to separate off-site storage — which also fixes Question 3.
━━━━━━━━━━━━━━━
🗂️ The index card
💾 EXISTS → FRESH → ELSEWHERE → COMPLETE → RESTORED
✅ EXISTS — a backup file is really there. You've seen it, not assumed it.
✅ FRESH — the newest one succeeded recently, and its size makes sense.
✅ ELSEWHERE — at least one copy lives outside your production account.
✅ COMPLETE — you've written down what it does NOT cover (uploaded files, secrets).
✅ RESTORED — you've actually rebuilt a working copy from it, at least once.
When your backup passes all five, it's a plan. Until then, it's a hope.
Don't mix the card up with the drill: the card is the health checklist for your backup setup, and the drill (dump, inspect, rebuild elsewhere, load, compare) is simply how you turn the last word — RESTORED — into a yes.
GitLab's night cost them six hours of customer data, 18 hours of downtime, and a public apology from the CEO — with five backup systems and 100 employees. The full protection for a solo SaaS costs one honest pass through five questions and one 30-minute drill.
Cheapest insurance you'll buy all year.
━━━━━━━━━━━━━━━
Over to you: when did you last restore a backup of your product — not take one, restore one? And what did the drill catch? Hit reply and tell me your stack and your answer — I read every reply, and the failure stories teach all of us more than the success ones.
— Bibhash
━━━━━━━━━━━━━━━