Then I started collecting from public websites.

A public source can return HTTP 200 and still be useless. It can serve a login page where a table used to be. It can return only the first page, repeat yesterday’s records with new timestamps, move columns, invent empty dates, or decide that a polite automated request looks suspicious. The transport succeeded. The data did not.

The most dangerous collector is not the one that crashes. It is the one that confidently replaces a known-good dataset with a plausible pile of nonsense.

Fresh is not the same as true

Imagine a local information page with 8,000 verified records. Tonight’s refresh retrieves 14 rows because the source changed pagination. A naive pipeline writes those 14 rows over the old file and celebrates a successful run.

By morning, the website is technically current and practically destroyed.

The safer pattern keeps separate objects:

incoming candidate
       |
       v
validate structure, counts, dates, uniqueness
       |
       +---- bad ----> quarantine + alert
       |
       +---- good ---> atomically promote to current
                              |
                              +--> preserve previous known-good

The last-good dataset is not merely a backup. It is part of the product contract.

If tonight’s import fails validation, readers should see yesterday’s honest data plus a visible freshness warning. They should not see an empty page pretending that nothing exists.

Validation should be rude

Collectors need to distrust their source in specific, testable ways.

Useful gates include:

• required columns are still present;
• row count did not collapse unexpectedly;
• identifiers remain unique;
• dates are parseable and not absurdly in the future;
• detail-page coverage stays within a known range;
• a sample of old records still exists;
• content is not an error page wearing a 200 status code.

None of these proves perfection. Together they make silent catastrophe much harder.

The thresholds should also be explainable. “Reject if the row count falls more than 40 percent” is debatable but testable. “The page looked weird” is neither.

Stale should be visible

Preserving last-good data creates a new obligation: do not pass stale data off as fresh.

Every public-facing dataset should carry a small status card:

• last successful source retrieval;
• last successful validation;
• current row count;
• whether detail coverage is complete or partial;
• number of quarantined records;
• link to the original source;
• plain-language caveat when the source is blocked or changing.

That little card is more useful than a hundred lines of invisible logging. It tells readers what the system knows about itself.

Failure can be a product feature

Most software demos focus on the happy path: click Refresh, watch new rows appear. Real reliability lives in the ugly path: the source returns less, later, malformed, duplicated, or nothing at all.

The mature response is not heroic scraping. It is restraint.

Keep yesterday’s truth. Quarantine today’s suspicious changes. Explain the age of the evidence. Try again later without turning one upstream failure into a downstream lie.

Sometimes the most important new data is the decision not to replace the old data.

It still got blocked.

That moment divides scraping projects into two kinds. One kind asks, “How do we get around this?” The other asks, “What is the responsible product behavior when the source says no?”

I want to build the second kind.

Public does not mean permissionless

A page visible in a browser may sit behind rate controls, session logic, terms, copyright, privacy obligations, or infrastructure that was never designed for automated access. A government label does not automatically erase those constraints. Neither does civic usefulness.

Before collecting, identify:

• the official source and any documented API or export;
• terms and licensing for reuse, not merely viewing;
• robots.txt and any published crawl or rate guidance;
• update cadence;
• contact or data-request path;
• personal information that should not be republished;
• a reasonable request rate and cache policy.

If a structured export exists, use it. If permission is required, ask. If the data can be linked without copying, consider whether that is enough.

A block is a state, not a challenge

When a collector receives a 403, a 429, a challenge page, or a suddenly empty response, it should classify the event and stop escalating.

The bad pattern rotates headers, IPs, accounts, and browser fingerprints until something slips through. That is expensive, fragile, and hostile. It also converts a data-quality problem into an adversarial-access problem.

The safer pattern is:

detect block
  -> stop the batch (and honor any Retry-After)
  -> preserve last-good data
  -> record bounded evidence
  -> alert once
  -> use official contact/export path

Do not let retries become a denial-of-service attack with good intentions.

Design the site for source failure

If a public-facing service depends on someone else’s portal, source unavailability is normal architecture.

The user experience should expose:

• last successful update time;
• original source link;
• current collection status;
• whether displayed data is complete or partial;
• what is being done to restore the feed;
• correction path.

That turns a mysterious stale page into an honest service.

Keep the collector replaceable

Source HTML changes. Portals get replaced. APIs appear and disappear. A collector should separate retrieval, parsing, validation, storage, and presentation so a new lawful source path does not require rebuilding everything.

Store normalized records with provenance. Preserve the original source identifier. Keep transformations documented. Test parsers against saved, non-sensitive fixtures rather than hammering the live site during development.

The collector is not the public product. Trust is.

Politeness includes accepting no

There is a technical culture that treats every block as a puzzle. Sometimes the clever solution is to back off, keep known-good information available, and make a formal request for a better feed.

The portal blocking the bot did not end the project. It clarified the contract the project needed: ethical retrieval, visible freshness, last-good preservation, and no secret war against the source.

A public-interest tool should not have to behave like malware to stay current.