The information was public. The website was reachable. The collector was slow, identified, and boring.

It still got blocked.

That moment divides scraping projects into two kinds. One kind asks, “How do we get around this?” The other asks, “What is the responsible product behavior when the source says no?”

I want to build the second kind.

Public does not mean permissionless

A page visible in a browser may sit behind rate controls, session logic, terms, copyright, privacy obligations, or infrastructure that was never designed for automated access. A government label does not automatically erase those constraints. Neither does civic usefulness.

Before collecting, identify:

  • the official source and any documented API or export;
  • terms and licensing for reuse, not merely viewing;
  • robots.txt and any published crawl or rate guidance;
  • update cadence;
  • contact or data-request path;
  • personal information that should not be republished;
  • a reasonable request rate and cache policy.

If a structured export exists, use it. If permission is required, ask. If the data can be linked without copying, consider whether that is enough.

A block is a state, not a challenge

When a collector receives a 403, a 429, a challenge page, or a suddenly empty response, it should classify the event and stop escalating.

The bad pattern rotates headers, IPs, accounts, and browser fingerprints until something slips through. That is expensive, fragile, and hostile. It also converts a data-quality problem into an adversarial-access problem.

The safer pattern is:

detect block
  -> stop the batch (and honor any Retry-After)
  -> preserve last-good data
  -> record bounded evidence
  -> alert once
  -> use official contact/export path

Do not let retries become a denial-of-service attack with good intentions.

Design the site for source failure

If a public-facing service depends on someone else’s portal, source unavailability is normal architecture.

The user experience should expose:

  • last successful update time;
  • original source link;
  • current collection status;
  • whether displayed data is complete or partial;
  • what is being done to restore the feed;
  • correction path.

That turns a mysterious stale page into an honest service.

Keep the collector replaceable

Source HTML changes. Portals get replaced. APIs appear and disappear. A collector should separate retrieval, parsing, validation, storage, and presentation so a new lawful source path does not require rebuilding everything.

Store normalized records with provenance. Preserve the original source identifier. Keep transformations documented. Test parsers against saved, non-sensitive fixtures rather than hammering the live site during development.

The collector is not the public product. Trust is.

Politeness includes accepting no

There is a technical culture that treats every block as a puzzle. Sometimes the clever solution is to back off, keep known-good information available, and make a formal request for a better feed.

The portal blocking the bot did not end the project. It clarified the contract the project needed: ethical retrieval, visible freshness, last-good preservation, and no secret war against the source.

A public-interest tool should not have to behave like malware to stay current.