The TLS Cert Only an API Could Renew

A client’s user control panel popped a browser security wall. Expired certificate. Classic.

The panel runs on a FreePBX appliance, on a high port, separate from the main admin UI. Users hit it for voicemail and call history. Now they hit a full-page TLS warning instead.

No big deal, I figured. Renew the cert, reload, done before the coffee gets cold.

The coffee got cold.

The investigation

First I confirmed it wasn’t the browser lying to me:

openssl s_client -connect 192.0.2.10:8003 </dev/null 2>/dev/null \
  | openssl x509 -noout -dates -subject
# notBefore=Apr  1 00:00:00 2024 GMT
# notAfter=Apr  1 00:00:00 2025 GMT

Dead for over a year. The panel had been quietly serving a corpse the whole time, and nobody noticed until a browser update got strict about it.

So I did the obvious things. The UI has a big friendly Apply Config button. Clicked it. Green checkmark. “Reload completed successfully.”

Re-ran my openssl check. Same expired dates.

Fine. Bigger hammer. From the CLI:

fwconsole reload
fwconsole restart
fwconsole ma updateall

Three clean exits. Three “success” lines. Three pieces of nothing — the panel kept serving the exact same expired cert, byte for byte.

That’s the moment the job stops being annoying and gets interesting. When every “fix” reports success and the resource never changes, you’re not fixing the resource. You’re poking something that doesn’t own it.

The “aha”

I drew the actual ownership chain — the thing nobody draws until they’re an hour in:

┌──────────────────────────────────────────────────────┐
│  reload / apply-config / updateall                    │
│      │                                                │
│      ▼  "rebuild my running config from what exists"  │
│  ┌───────────────────┐                                │
│  │  User Panel :8003 │  ── binds ──►  cert file ✗     │
│  └───────────────────┘               (expired, stale) │
│                                            ▲           │
│                                            │ OWNS      │
│                                  ┌───────────────────┐ │
│                                  │ Certificate Mgr   │ │
│                                  │  (lifecycle here) │ │
│                                  └───────────────────┘ │
└──────────────────────────────────────────────────────┘

The reload path rebuilds config from whatever certs already exist on disk. It never regenerates one. The thing that actually mints and renews the cert is the Certificate Manager module. Reload was faithfully re-binding the panel to a file that was still expired — and honestly reporting success, because re-binding did succeed. The cert was just garbage.

Nobody was renewing it because nobody told the component that owns renewals to do its job.

The fix

Stop driving the reload path. Drive the API of the thing that owns the cert.

The Certificate Manager exposes a regeneration call. Invoke it directly, let it reissue, then point the panel at the fresh cert and reload once to bind:

# Ask the module that OWNS the cert to reissue it
fwconsole certificates --regen-self-signed

# (or, when an ACME/managed cert is in play)
fwconsole certificates --update

# Now bind the panel to the freshly minted cert, then reload to apply
fwconsole certificates --default=<new_cert_id>
fwconsole reload

Then verify against the live socket — not the UI’s word, the actual TLS handshake:

openssl s_client -connect 192.0.2.10:8003 </dev/null 2>/dev/null \
  | openssl x509 -noout -dates
# notBefore=Jun  9 00:00:00 2026 GMT
# notAfter=Jun  9 00:00:00 2028 GMT   ✓

Fresh dates. Panel loaded clean. Coffee remade.

Why it happened

“Reload” and “apply config” are rebuild operations. They take the current declared state and re-render it. If the declared state references a cert file, they’ll happily wire it up — expired or not. Validity isn’t their job.

Renewal lives in exactly one place: the module that issues certs. Every other button just consumes the output. So every reload was a no-op on the resource, dressed up as a success because the binding step really did succeed.

The trap is that “success” was true at the wrong altitude. The command did what it was scoped to do. It just wasn’t scoped to the thing I needed.

Takeaways

• When every fix reports success but the resource never changes, you’re poking the wrong subsystem. Find the component that actually owns the resource’s lifecycle and drive its API.
• “reload” / “apply config” usually means re-render, not regenerate. They consume artifacts; they rarely create them.
• Trust the wire, not the UI. openssl s_client ... | openssl x509 -noout -dates against the live port is ground truth. A green checkmark is a claim.
• Map ownership before you start hammering buttons. One quick “who actually mints this?” diagram beats an hour of confident no-ops.
• Expired certs hide for months. Monitor notAfter on every TLS port — including the weird high-port sub-services nobody remembers exist.