How My Own Backup Deleted My Files

The ticket was boring. Cleanup. A client wanted some stale files gone from their primary share. I deleted them. Coffee. Done.

Twenty minutes later the same nine files vanished from the backup node too.

That’s the part that made my stomach drop. The backup node existed precisely so that “oops, deleted it on the primary” was a non-event. It was supposed to be the safety net. Instead it watched me cut the trapeze and then helpfully cut its own rope to match.

The investigation

First instinct: ransomware, a rogue cron, a fat-fingered teammate. All wrong. The truth was dumber and worse.

I’d inherited this setup and never read the sync config closely. I assumed “backup node” meant one-way. It didn’t.

grep -i 'mode\|direction\|propagat' /etc/syncthing/config.xml

<folder id="primary-share" type="sendreceive" ...>

sendreceive. Bidirectional. The backup node wasn’t a backup at all — it was a mirror. And a mirror does exactly one job: make the other side look like this side. Including the absences.

I deleted nine files on the primary. The sync engine saw nine deletions, decided they were authoritative changes, and faithfully replicated them downstream. No malice. No bug. It did exactly what I configured it to do.

┌──────────────┐   delete 9 files    ┌──────────────┐
│   PRIMARY    │ ──────────────────► │  "BACKUP"    │
│  (source)    │   sendreceive       │  node        │
└──────────────┘   propagates ✗      └──────────────┘
       │                                    │
       ▼                                    ▼
   files gone                          files gone too
                                       ▓ safety net: GONE

The “aha” wasn’t a clever diagnostic. It was reading one attribute and feeling like an idiot. Sync is not backup. A sync engine has no concept of “retain this.” Its entire worldview is converge. Deletion is just another change to converge on.

sync mirrors deletions · backup keeps themPRIMARYsendreceive ✗pull-only ✓delete →← delete propagatesreceive only, never push
A bidirectional mirror converges on deletions. A pull-only backup receives changes but never propagates them back.

The fix

Recover first, redesign second. The deleted files came back from a daily snapshot that — by luck, not design — lived off the sync path. Then I changed the node from a mirror into an actual backup.

# 1) flip the backup node to PULL-ONLY: it receives, never pushes
#    Syncthing: set this folder to "Receive Only"
#    config.xml -> type="receiveonly"

# 2) route deletions to a versioned trash instead of hard-deleting
#    <versioning type="trashcan"> with a retention window
#    (deletes land in .stversions/, not the void)

# 3) shorten the sync interval to bound the blast radius
#    rescanIntervalS: 3600 -> 300   # 5 min, not 1 hour

# 4) confirm the daily snapshot job runs OUTSIDE the synced tree
systemctl list-timers | grep snapshot
ls -la /var/snapshots/   # snapshots here, NOT under the synced folder

After the flip, the verification was the satisfying part. Delete a test file on the primary, watch the backup node not care:

# on primary
touch /srv/share/CANARY.txt
rm  /srv/share/CANARY.txt

# on backup node, a few minutes later
ls /srv/backup/share/CANARY.txt
# CANARY.txt is still there ✓  (delete did not propagate)

Receive-only means upstream deletions are treated as conflicts to ignore, not commands to obey. The node now keeps things the primary throws away. That’s the whole point of a backup.

Why it happened

Because “sync” and “backup” use the same verbs — copy, mirror, replicate — people treat them as the same noun. They aren’t.

Sync optimizes for agreement: every node looks identical, fast. Backup optimizes for survival: keep a copy even when the source loses it. Those goals conflict the instant you delete something. A mirror resolves the conflict by deleting too. A backup resolves it by holding on.

I’d labeled a mirror “backup” and trusted the label. The config said sendreceive the whole time. I just never looked.

Takeaways

  • Sync is not backup. Bidirectional sync replicates deletions perfectly — that’s a feature, and it will eat your data.
  • A backup node must be one-way. Receive-only / pull-only: it accepts changes but never pushes or propagates deletes upstream-to-down.
  • Route deletions to trash, not the void. A versioned trashcan with retention turns “gone forever” into “gone for now.”
  • Keep snapshots off the sync path. If your point-in-time recovery lives inside the synced tree, the sync can delete your recovery too.
  • Read the config before you trust the label. grep the one attribute that defines behavior — sendreceive vs receiveonly is the whole story.