The chat ping came in at 4:47 PM. “Hey, is the app down? All my stuff is gone.”

I pulled up the site. It loaded fine. Login worked. Dashboard rendered. And it was empty. No records. No users. No history. A perfect, pristine, brand-new install — on a server that had been running for fourteen months.

Nothing was broken. That’s what made my stomach drop. A crash leaves a corpse. This was worse. The app was alive and humming, cheerfully serving a database that had been born thirty seconds ago.

The investigation

First instinct: bad migration. Nope — no migrations had run. Second instinct: someone fat-fingered a DELETE. But the Postgres-shaped panic faded fast, because this app didn’t use Postgres. It used a SQLite file. A single file, living in the app directory.

That detail mattered more than I realized.

I checked the logs. The last thing that happened before the data vanished was a deploy. Routine. Boring. A dev had pushed a tiny CSS fix and run the deploy script, same as always.

So I opened the deploy script. And there it was, line nine, smiling at me like a loaded gun:

scp -r ./app/* deploy@198.51.100.10:/srv/app/

./app/*. The whole directory. Including app/data/app.db.

The dev’s local app.db was empty — fresh clone, never seeded. Every deploy had been copying that empty file up and stomping the live one. The only reason nobody noticed sooner was that this dev happened to have a blank local copy. Previous deploys from other machines had local data that masked the bug.

Here’s the shape of the disaster:

   DEV MACHINE                        PRODUCTION
 ┌──────────────┐                  ┌──────────────┐
 │ app/         │                  │ app/         │
 │  ├ main.py   │ ── scp -r ──►    │  ├ main.py   │
 │  ├ static/   │   "deploy"       │  ├ static/   │
 │  └ data/     │                  │  └ data/     │
 │     app.db   │ ════ EMPTY ════► │     app.db   │  ◄ 14 months
 │   (0 rows)   │    OVERWRITES    │   (was full) │     GONE
 └──────────────┘                  └──────────────┘

The fix wasn’t on the live disk. But the box ran nightly off-site snapshots, and the most recent one was eleven hours old. We restored that, replayed the day’s activity from request logs by hand, and lost almost nothing. Lucky. Stupid lucky.

app codemain.pystatic/prod dataapp.dbuploads/rsync--exclude data/code >> data & never the reverse
The deploy should push code over the wall and leave production's data untouched.

The fix

The old script blanket-copied everything. The new one does three things it refused to do before: backs up prod data first, uses rsync with explicit excludes, and never touches the data paths at all.

#!/usr/bin/env bash
set -euo pipefail

HOST="deploy@198.51.100.10"
APP_DIR="/srv/app"
STAMP="$(date +%Y%m%d-%H%M%S)"

# 1. Back up prod data BEFORE we touch anything.
ssh "$HOST" "tar czf /srv/backups/data-${STAMP}.tar.gz \
    -C ${APP_DIR} data uploads"

# 2. Push CODE only. Exclude every path that holds state.
rsync -avz --delete \
    --exclude='data/' \
    --exclude='uploads/' \
    --exclude='*.db' \
    --exclude='*.sqlite' \
    --exclude='.env' \
    ./app/ "${HOST}:${APP_DIR}/"

echo "Deployed. Pre-deploy snapshot: data-${STAMP}.tar.gz"

Note --delete is now safe because the data dirs are excluded — rsync won’t reach in and prune data/ if it never syncs data/ in the first place. And the backup runs before the push, so even a botched exclude has a parachute.

Why it happened

scp -r ./app/* isn’t a bug. It does exactly what it says. The bug was a design decision nobody made on purpose: letting state live inside the deploy artifact. The database sat in the same folder as the code, so “deploy the code” silently meant “deploy the database too.”

The destructive path wasn’t unlikely. It ran on every single deploy. It just stayed invisible until a dev with an empty local copy pulled the trigger.

Takeaways

  • Deploy code, not data. Application state — databases, uploads, user files — must live outside anything a deploy can overwrite.
  • Any deploy that can touch the database is a loaded gun. Add explicit --exclude rules for every data and upload path, and verify them in a dry run (rsync -n).
  • Back up before you deploy, not after. A snapshot taken the instant before the push is the cheapest insurance you’ll ever buy.
  • Make the destructive path impossible, not just unlikely. “Be careful” is not a control. Excludes and pre-deploy backups are.
  • Boring near-misses are gifts. We got lucky with an eleven-hour-old snapshot. Don’t design around luck — design so the gun can’t fire.