Batch jobs have a talent for turning one weird row into a full-day emergency.

The source sends a date that does not exist. A business name contains a character nobody tested. One detail page returns half an HTML document. A record claims it was updated in the year 2099. The importer reaches that row, throws an exception, and leaves the whole dataset stuck at yesterday.

The usual reaction is to make the parser more forgiving. That can be worse. A parser that accepts everything eventually turns corruption into official data.

The better answer is quarantine.

A bad record is not a bad run

A robust importer treats records as individual claims.

Each claim passes through structural, semantic, and consistency checks. Good records continue. Suspicious records move to a separate holding area with the reason they were rejected.

record 4172
  identifier: present
  date: 2099-31-44       X invalid
  required fields: pass
  action: quarantine
  reason: impossible date

The run can finish with 7,999 accepted records and one quarantined record. That is not a hidden partial success. It is an explicit result with a number, a reason, and something a human can inspect.

Quarantine is not the trash

Rejected records must remain recoverable. The point is to prevent them from contaminating the public dataset while preserving enough evidence to repair the rule or correct the source later.

A useful quarantine entry includes:

  • source identifier and source URL when safe;
  • retrieval time;
  • validation rule that failed;
  • bounded copy or hash of the raw input;
  • parser version;
  • whether the record existed in the previous dataset;
  • review status and resolution.

Do not dump an entire credential-bearing response into a log. Quarantine should preserve evidence, not create a second security incident.

The totals must add up

The fastest way to lose trust in a pipeline is to make records disappear between counters.

Every run should satisfy a reconciliation equation:

retrieved = accepted + unchanged + quarantined + explicitly skipped

If it does not, the run is incomplete.

That equation sounds obvious. It is also where many pipelines fail. They log “processed 8,000” at the start, “saved 7,997” at the end, and never explain the missing three.

Thin logs turn those three records into ghosts. A complete ledger turns them into work.

Tomorrow should not inherit today’s wedge

Quarantine also protects future runs.

If one malformed row stays in the upstream source for a week, a crash-on-first-error collector will fail seven days in a row. A quarantine-aware collector will flag the same record, deduplicate the alert, and continue collecting everything else. When the source fixes it, the row can re-enter through the normal validation path.

The operational difference is enormous. One is a brittle daily ceremony. The other is a system that can live in the real world.

Bad rows are inevitable. Silent rows are dangerous. Quarantined rows are manageable.