I used to think a data collector had two states: it ran or it failed.

Then I started collecting from public websites.

A public source can return HTTP 200 and still be useless. It can serve a login page where a table used to be. It can return only the first page, repeat yesterday’s records with new timestamps, move columns, invent empty dates, or decide that a polite automated request looks suspicious. The transport succeeded. The data did not.

The most dangerous collector is not the one that crashes. It is the one that confidently replaces a known-good dataset with a plausible pile of nonsense.

Fresh is not the same as true

Imagine a local information page with 8,000 verified records. Tonight’s refresh retrieves 14 rows because the source changed pagination. A naive pipeline writes those 14 rows over the old file and celebrates a successful run.

By morning, the website is technically current and practically destroyed.

The safer pattern keeps separate objects:

incoming candidate
       |
       v
validate structure, counts, dates, uniqueness
       |
       +---- bad ----> quarantine + alert
       |
       +---- good ---> atomically promote to current
                              |
                              +--> preserve previous known-good

The last-good dataset is not merely a backup. It is part of the product contract.

If tonight’s import fails validation, readers should see yesterday’s honest data plus a visible freshness warning. They should not see an empty page pretending that nothing exists.

Validation should be rude

Collectors need to distrust their source in specific, testable ways.

Useful gates include:

  • required columns are still present;
  • row count did not collapse unexpectedly;
  • identifiers remain unique;
  • dates are parseable and not absurdly in the future;
  • detail-page coverage stays within a known range;
  • a sample of old records still exists;
  • content is not an error page wearing a 200 status code.

None of these proves perfection. Together they make silent catastrophe much harder.

The thresholds should also be explainable. “Reject if the row count falls more than 40 percent” is debatable but testable. “The page looked weird” is neither.

Stale should be visible

Preserving last-good data creates a new obligation: do not pass stale data off as fresh.

Every public-facing dataset should carry a small status card:

  • last successful source retrieval;
  • last successful validation;
  • current row count;
  • whether detail coverage is complete or partial;
  • number of quarantined records;
  • link to the original source;
  • plain-language caveat when the source is blocked or changing.

That little card is more useful than a hundred lines of invisible logging. It tells readers what the system knows about itself.

Failure can be a product feature

Most software demos focus on the happy path: click Refresh, watch new rows appear. Real reliability lives in the ugly path: the source returns less, later, malformed, duplicated, or nothing at all.

The mature response is not heroic scraping. It is restraint.

Keep yesterday’s truth. Quarantine today’s suspicious changes. Explain the age of the evidence. Try again later without turning one upstream failure into a downstream lie.

Sometimes the most important new data is the decision not to replace the old data.