The 99 Access Points That Didn't Exist

The ticket landed with a number attached, and the number was wrong.

A monitoring report — the one we generate so nobody has to think too hard — claimed a site was running 99 wireless access points. Ninety-nine. For a building that, last I checked, you could walk end to end during a coffee that hadn’t gone cold yet.

I’d been in that building. I knew the ceilings. There were not 99 APs up there. There weren’t even a quarter of that.

So either the site had quietly become an enterprise campus overnight, or the report was hallucinating. Spoiler: reports don’t hallucinate. They count exactly what you tell them to count, which is a different and worse problem.

The investigation

First rule when a number looks insane: don’t argue with the number. Go find ground truth and make the number argue with that.

I pulled the controller’s live device list — the actual adopted, talking-to-me-right-now hardware:

# adopted APs the controller is actively managing
curl -sk -b /tmp/controller.cookie \
  https://controller.example.internal/proxy/network/api/s/default/stat/device \
  | jq '[.data[] | select(.type=="uap")] | length'

12

Twelve. Twelve adopted access points, online, real, with MAC addresses I could go put my hand on.

So where did the report get 99?

I went and read the thing that built the report instead of the report itself. The collector wasn’t querying live adopted devices. It was scraping a broader chunk of the controller database — and it was counting rows, not radios you could hang on a wall.

The aha

Three lies were stacked on top of each other, and the report added them all up with a straight face.

  WHAT THE REPORT COUNTED            WHAT WAS ACTUALLY THERE
  ─────────────────────────          ───────────────────────
  ┌───────────────────────┐
  │ adopted APs        12 │ ◄──────── ✓ real, on the ceiling
  ├───────────────────────┤
  │ pending / forgotten   │
  │ ghost DB rows      ~30│ ◄──────── ✗ stale, never cleaned
  ├───────────────────────┤
  │ per-radio entries     │
  │ (2.4 / 5 / 6 GHz)     │ ◄──────── ✗ one AP counted 3×
  ├───────────────────────┤
  │ per-SSID rows         │ ◄──────── ✗ counted broadcasts
  └───────────────────────┘            as devices
        Σ = 99                          Σ (real) = 12

Every tri-band AP showed up as three radio entries. Every SSID broadcast left its own footprint. And the controller DB still held a graveyard of devices that had been swapped out, RMA’d, or “forgotten” in the UI months ago — rows that never actually leave the database, they just stop mattering. To a human. Not to a COUNT(*).

Stack the real twelve on top of the radio multiplication on top of the ghost rows, and you arrive — with perfect arithmetic and zero truth — at 99.

The fix

The fix wasn’t clever. It was narrowing the question. Count adopted UAP-type devices, deduplicated by MAC, and exclude anything pending or forgotten:

curl -sk -b /tmp/controller.cookie \
  https://controller.example.internal/proxy/network/api/s/default/stat/device \
  | jq '[.data[]
         | select(.type=="uap")
         | select(.adopted==true)
         | select(.state==1)          # 1 = connected
         | .mac]
        | unique | length'

12

Then I did the thing the database can’t: I walked the building and counted ceilings. Twelve.

The report now pulls that single reconciled number, and it carries a footnote: adopted, connected, deduped by MAC. If it ever disagrees with the live device list again, it’s the report that has to explain itself.

Why it happened

Because counting feels trivial, so nobody guards it. length on a JSON array looks like truth. But the controller DB isn’t a list of access points — it’s an event log of everything that ever claimed to be one: every radio, every SSID broadcast, every device that got swapped and never purged. Counting rows in that pile and calling it “APs” is like counting fingerprints and reporting the number of burglars.

The artifact survived for months because 99 was plausible-adjacent. Nobody who’d been on-site ever read the report; nobody who read the report had ever been on-site. The number floated in the gap between them, fat and unchallenged, until somebody finally looked up.

Takeaways

• Never report an inventory number you can’t reconcile to ground truth. A count needs two witnesses: the live device list and a physical walk.
• Rows are not things. Per-radio and per-SSID entries multiply one AP into many. Filter by device type and dedupe by MAC before you trust a total.
• Controller databases keep ghosts. Forgotten, pending, and RMA’d devices linger in the DB long after they leave the ceiling. Scope your query to adopted and connected.
• Plausible-adjacent is the dangerous zone. A wrong number that’s obviously insane gets caught. One that’s merely “a bit high” rides for months.
• Read the query, not the report. When a metric lies, the bug is upstream in what you told it to count — go fix the question.