[glm-grade=B] fix(monitoring): remap loopback upstream probes to host gateway (DC-053)
The caddy-upstream-watcher runs inside the dashcaddy-api container but
probes upstreams declared for Caddy, which runs on the HOST. Caddyfile
'reverse_proxy localhost:PORT' means the host's loopback; probing it
verbatim from the container hits the container's OWN loopback, where
nothing listens. Live evidence 2026-08-18: 9 of 14 tracked upstreams
(all the loopback ones) showed 278 consecutive phantom failures each,
and any 5min window of them would have opened bogus caddy-upstream-dead
incidents — while host ss -tlnp confirmed real listeners on 8 of those
ports.
Fix:
- Probe loopback targets via host.docker.internal instead, pinned to the
host bridge IP by start.sh (--add-host=host.docker.internal:host-gateway,
Docker >= 20.10). Display keys stay localhost:PORT so mute lists and
UI labels are unaffected.
- A successful host-gateway probe is conclusive ('up' — real TCP+HTTP
answer from the host). A FAILED probe is epistemically inconclusive
(127.0.0.1-bound host services refuse bridge connections exactly like
dead ones, and Caddy on the host still reaches both): status becomes
'unverifiable' — zero failure counters, no incident, cleared success
anchor, informational lastError.
- IN_CONTAINER=false disables the remap (bare-metal deployments).
- Snapshot sort extended: dead > down > muted > unverifiable > up > unknown.
Tests: 5 new (23/23 in suite) covering remap targeting (localhost,
127.0.0.1, 127.x), non-loopback pass-through, unverifiable semantics,
and sort order. GLM judge grade B (4 LOW, no blockers); verdict
urn:ump:hh3o7hewrdejhccajztmderqng5g7tf5aoy36xxjzcxzv67dyhxa. Regrade
with Codex when quota resets 2026-08-24.
This commit is contained in:
@@ -88,6 +88,14 @@ run_image_layer_migration
|
||||
# git.dashcaddy.net → DashCaddy upstream git
|
||||
# ca.sami → local CA (DN2 + DN3 both have their own)
|
||||
ADD_HOST_FLAGS=(
|
||||
# host.docker.internal → host bridge IP (Docker host-gateway). The caddy
|
||||
# upstream watcher probes Caddy site upstreams from INSIDE this container;
|
||||
# `reverse_proxy localhost:PORT` in a site file means the HOST's loopback,
|
||||
# so the watcher remaps loopback probe targets to this name (see
|
||||
# dashcaddy-api/src/monitoring/caddy-upstream-watcher.js). Without this
|
||||
# entry the probes would hit the container's own loopback and report every
|
||||
# host-side upstream as dead.
|
||||
--add-host=host.docker.internal:host-gateway
|
||||
--add-host=dns3.sami:100.81.59.99
|
||||
--add-host=gitea:100.81.59.99
|
||||
--add-host=dns3-wan.sami:74.208.167.19
|
||||
|
||||
Reference in New Issue
Block a user