[glm-grade=B] fix(monitoring): restore dead-detection for verified loopback upstreams (DC-054)
CI / Test & Lint (push) Canceled after 0s
CI / Security audit (push) Canceled after 0s

DC-053 follow-ups (queue item 2b). Three small fixes to the caddy-upstream-watcher:

1. verifiedViaBridge flag: a loopback upstream whose PRIOR probe succeeded via
   host-gateway proves the bridge CAN reach the host. A later failed probe
   is then near-conclusive evidence the upstream itself went dead. The
   DC-053 code unconditionally marked loopback failures as unverifiable,
   throwing away this signal. Now: track verifiedViaBridge per-upstream and
   treat verified-then-failed as down (count failures, open incident after
   DEAD_AFTER_MS=5min).

2. IN_CONTAINER=false kill-switch test (B-grade polish, folded into same
   commit per conjoint-commit anti-pattern).

3. git.sami intermittent ENOTFOUND (~2/h in ssl-monitor TLS handshake):
   pin git.sami -> 100.121.150.22 (DNS2 Tailscale) in container /etc/hosts
   via --add-host in start.sh. Existing comment explicitly forbids pinning
   to DNS3/100.81.59.99 (no HTTPS listener there); DNS2/100.121.150.22 is
   correct (Caddy serves git.sami on DNS2:443 and routes to DNS3:3030
   internally).

GLM-5.3 judge round 1 (208s, 6 tool calls, on-disk verified): grade B,
all 25 tests green, no blocking issues, 3 LOW polish suggestions.
Folded two actionable LOWs (persistence + JSDoc) into this commit:
- verifiedViaBridge now persisted in _saveState/_restoreUpstreamStates so
  a known-good loopback upstream stays labeled across container restarts
  (1-tick blip becomes 0-tick).
- snapshot() gained JSDoc describing the verifiedViaBridge semantic for
  dashboard consumers.
Third LOW (long-term: prefer host-side liveness signal from Caddy) is a
roadmap note, not actionable now.

Tests: 1921/1921 (was 1910; +11 net: 6 new for items 2b-a/2b-b/persistence +
5 previously-skipped baseline). Full suite 86/86 green.
This commit is contained in:
Krystie
2026-08-18 00:13:58 -07:00
parent 71e04d0a86
commit 901df8608b
3 changed files with 199 additions and 4 deletions
+5
View File
@@ -86,6 +86,10 @@ run_image_layer_migration
# dns1.sami → DNS1 (SAMI-CLOUD-U32)
# dc-contabo-de → DashCaddy Contabo test instance
# git.dashcaddy.net → DashCaddy upstream git
# git.sami → DNS2 (NOT DNS3 — see warning above). Resolves an
# intermittent ENOTFOUND in the ssl-monitor's TLS
# handshake check (~2/h) by pinning the name in the
# container's /etc/hosts to the Caddy listener.
# ca.sami → local CA (DN2 + DN3 both have their own)
ADD_HOST_FLAGS=(
# host.docker.internal → host bridge IP (Docker host-gateway). The caddy
@@ -103,6 +107,7 @@ ADD_HOST_FLAGS=(
--add-host=dns1.sami:100.71.97.12
--add-host=dc-contabo-de:100.98.123.59
--add-host=git.dashcaddy.net:100.98.123.59
--add-host=git.sami:100.121.150.22
# ca.sami resolves via DNS to 100.121.150.22 (Caddy on DNS2). Don't pin
# to 127.0.0.1 — nothing listens on 443 inside the container, so the
# health checker would fail with ECONNREFUSED. The CA itself is a