933606ce3f3a88380df7e06e688eaa92ccf7b194
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Krystie
|
901df8608b |
[glm-grade=B] fix(monitoring): restore dead-detection for verified loopback upstreams (DC-054)
DC-053 follow-ups (queue item 2b). Three small fixes to the caddy-upstream-watcher: 1. verifiedViaBridge flag: a loopback upstream whose PRIOR probe succeeded via host-gateway proves the bridge CAN reach the host. A later failed probe is then near-conclusive evidence the upstream itself went dead. The DC-053 code unconditionally marked loopback failures as unverifiable, throwing away this signal. Now: track verifiedViaBridge per-upstream and treat verified-then-failed as down (count failures, open incident after DEAD_AFTER_MS=5min). 2. IN_CONTAINER=false kill-switch test (B-grade polish, folded into same commit per conjoint-commit anti-pattern). 3. git.sami intermittent ENOTFOUND (~2/h in ssl-monitor TLS handshake): pin git.sami -> 100.121.150.22 (DNS2 Tailscale) in container /etc/hosts via --add-host in start.sh. Existing comment explicitly forbids pinning to DNS3/100.81.59.99 (no HTTPS listener there); DNS2/100.121.150.22 is correct (Caddy serves git.sami on DNS2:443 and routes to DNS3:3030 internally). GLM-5.3 judge round 1 (208s, 6 tool calls, on-disk verified): grade B, all 25 tests green, no blocking issues, 3 LOW polish suggestions. Folded two actionable LOWs (persistence + JSDoc) into this commit: - verifiedViaBridge now persisted in _saveState/_restoreUpstreamStates so a known-good loopback upstream stays labeled across container restarts (1-tick blip becomes 0-tick). - snapshot() gained JSDoc describing the verifiedViaBridge semantic for dashboard consumers. Third LOW (long-term: prefer host-side liveness signal from Caddy) is a roadmap note, not actionable now. Tests: 1921/1921 (was 1910; +11 net: 6 new for items 2b-a/2b-b/persistence + 5 previously-skipped baseline). Full suite 86/86 green. |
||
|
Hermes
|
71e04d0a86 |
[glm-grade=B] fix(monitoring): remap loopback upstream probes to host gateway (DC-053)
The caddy-upstream-watcher runs inside the dashcaddy-api container but
probes upstreams declared for Caddy, which runs on the HOST. Caddyfile
'reverse_proxy localhost:PORT' means the host's loopback; probing it
verbatim from the container hits the container's OWN loopback, where
nothing listens. Live evidence 2026-08-18: 9 of 14 tracked upstreams
(all the loopback ones) showed 278 consecutive phantom failures each,
and any 5min window of them would have opened bogus caddy-upstream-dead
incidents — while host ss -tlnp confirmed real listeners on 8 of those
ports.
Fix:
- Probe loopback targets via host.docker.internal instead, pinned to the
host bridge IP by start.sh (--add-host=host.docker.internal:host-gateway,
Docker >= 20.10). Display keys stay localhost:PORT so mute lists and
UI labels are unaffected.
- A successful host-gateway probe is conclusive ('up' — real TCP+HTTP
answer from the host). A FAILED probe is epistemically inconclusive
(127.0.0.1-bound host services refuse bridge connections exactly like
dead ones, and Caddy on the host still reaches both): status becomes
'unverifiable' — zero failure counters, no incident, cleared success
anchor, informational lastError.
- IN_CONTAINER=false disables the remap (bare-metal deployments).
- Snapshot sort extended: dead > down > muted > unverifiable > up > unknown.
Tests: 5 new (23/23 in suite) covering remap targeting (localhost,
127.0.0.1, 127.x), non-loopback pass-through, unverifiable semantics,
and sort order. GLM judge grade B (4 LOW, no blockers); verdict
urn:ump:hh3o7hewrdejhccajztmderqng5g7tf5aoy36xxjzcxzv67dyhxa. Regrade
with Codex when quota resets 2026-08-24.
|
||
|
Sami Ahmed
|
45cfa83bad |
feat(monitoring): dead-upstream surfacing + mute toggle (DC-049) [mm-grade=B+]
Caddy's own reverse_proxy health_checker logs every 10s about unreachable
tenant upstreams (see recurring 100.120.159.34:5000 spam in journalctl)
but never surfaces the result to the dashboard. Adds:
- caddy-upstream-watcher.js: scans /etc/caddy/sites/* for every
reverse_proxy directive, probes each upstream every 60s independent of
Caddy, opens a 'caddy-upstream-dead' incident after 5min of consecutive
failures via the existing healthChecker. Mute list persisted to
data/caddy-upstreams.json. Probes stamp X-DashCaddy-HealthCheck: 1 so
forward_auth doesn't 401 them.
- routes/caddy-upstreams.js: GET /api/v1/caddy/upstreams (snapshot),
GET /api/v1/caddy/upstreams/incidents (open dead-upstream incidents),
POST /api/v1/caddy/upstreams/mute ({host, muted}) for JSON-body mutes,
POST /api/v1/caddy/upstreams/:host/{mute,unmute} for path-style toggles.
All mounted under the auth-gated apiRouter in app.js.
- 16 unit tests + 2 route smoke tests, all passing.
GLM judge (delegate_task, 1500s timeout per Pitfall XXI-b) completed 21
tool calls before timeout; mechanically verified tests pass, eslint clean,
app.js module load OK, RST-mid-body ECONNRESET is caught by req.on('error').
Found two MEDIUM defects which are now fixed in this commit:
1. (MEDIUM) scanSites file-extension filter `/\.(sami|caddy|conf)$/i`
silently skipped real prod filenames like zap.sami-ahmed.net,
samitest.space, blocks.cryptographic-triangles.org where the file
extension is .net/.space/.org. Replaced with positive filter that
excludes README/.bak/.swp/Caddyfile + content pre-check
(must contain 'reverse_proxy'). Added test covering the prod filenames.
2. (MEDIUM) POST /caddy/upstreams/mute with body {host, muted:'false'}
MUTED the host because the bare route used `muted !== false` which is
true for the string 'false'. Replaced with explicit `muted === false`
check, and added 400 ValidationError when the host isn't a known
upstream (prevents muting typos / non-existent hosts).
Self-grade: B+ (after applying GLM partial review). Re-grade with Codex
when its quota resets 2026-08-24.
|