Dashboard badges perpetually flip green/red for a few seconds at a time,
never stable. Root cause: health-checker emitted 'status-check' on every
probe (every 30s) and dashboard-ws forwarded every one as 'status-change'
to the browser with no diff; live-events.js then unconditionally called
setBadge(). A single transient 5xx (Caddy reload, container restart, TLS
handshake blip) flipped the badge and the next green probe flipped back.
Fix: _computeDisplayedStatus applies asymmetric hysteresis — DOWN_THRESHOLD
(default 2, env-tunable HEALTH_DOWN_THRESHOLD) consecutive probes that
disagree with the displayed 'up' state flip to red; UP_THRESHOLD (default
1, HEALTH_UP_THRESHOLD) flips back to green. History and consecutiveFailures
still record every raw probe so postmortem analysis is unchanged. Only the
SSE broadcast is filtered. getCurrentStatus now returns the displayed
status so a page reload shows the same badge as the live stream.
10 new tests cover first-emit, same-status-dedup, the actual flicker bug
(one-down-then-up keeps green), two-down flips red, one-up recovers fast,
long-steady-green produces exactly one emit, and env-var tuning. All 63
existing health-checker tests still pass. Full suite: 2484/2484.
Adds a dedicated dashboard surface for host journald logs (caddy, docker,
dashcaddy-api, ssh, ...) via a read-only bind-mount of /var/log/journal +
journalctl. Closes queue item #2: the only way to see the recurring
'100.120.159.34:5000 i/o timeout' spam in Caddy's health_checker logs was
SSH into DNS2.
Backend (dashcaddy-api/):
- src/monitoring/journald-reader.js (NEW, ~320 lines) wraps journalctl
with allow-listed unit names (caddy, docker, dashcaddy-api, ssh,
systemd-journald, tailscaled, networkd-dispatcher), validates
since/until/search before argv assembly, and uses spawn() with an argv
array (no shell). Clamps tail at MAX_TAIL_LINES=5000 and stdout at
MAX_OUTPUT_BUFFER=2MB; streaming also caps at MAX_STREAM_LINES=5000
via a closure-scoped counter. Maps ENOENT cleanly to 'journalctl
unavailable'.
- routes/logs.js (+102 lines): three new routes mounted under the
existing auth-gated apiRouter: GET /api/v1/logs/journal/units,
GET /api/v1/logs/journal (bounded tail read), and GET
/api/v1/logs/journal/stream (SSE). Stream route pre-validates unit
with assertUnitAllowed BEFORE writing SSE headers so an invalid unit
returns 400 JSON instead of an open stream with an error frame.
- 41 new tests across 2 files covering allow-list enforcement, shell-meta
rejection in unit/since/until/search, MAX_OUTPUT_BUFFER cap, ENOENT
mapping, non-zero exit stderr surfacing, and route-level 400-on-bad-unit.
Full local suite 1831/1831 (+41 net).
Container plumbing (start.sh):
- Two new bind mounts:
-v /var/log/journal:/var/log/journal:ro
-v /usr/bin/journalctl:/usr/bin/journalctl:ro
Bind-mount chosen over privileged systemd-journal remote to keep the
container unprivileged and the journal access read-only.
Frontend (status/js/):
- journald.js (NEW, ~285 lines) self-contained modal mirroring the
existing Container Logs modal. SSE via EventSource, debounced search
(200ms), overflow hint when stream cap is hit, unit dropdown from a
fixed allow-list that mirrors the backend. Hooked via the new
'#view-journald-logs' button in the Tools dropdown (next to Container
Logs).
- build.js (+4 lines) adds journald.js to the features bundle. Bundle
rebuild succeeded (features.js 27 files, 466 KB raw / 1229 KB min).
CSP hash unchanged (no inline script changes).
GLM judge (round 1, 178s, 14 tool calls, cold diff + 8 file reads):
GRADE=B. Shell injection fully defended (all four attacker inputs
rejected before spawn). Route-level allow-list holds (streamEntries not
called for bad unit). SSE cleanup correct. Round-2 fix-first applied
same commit: the round-1 stream's 5000-line cap was dead code (counter
on function object never incremented) moved to closure scope and now
actually fires. Also dropped deprecated req.on('aborted') listener
(Node 18+ fires 'close' for both clean and abort).
Container live HEAD 901df86 [glm-grade=B]; deploy via start.sh atomic
swap. Live verify: status.sami=200, container Up + healthy, the new
bundle and index.html served.
DC-053 follow-ups (queue item 2b). Three small fixes to the caddy-upstream-watcher:
1. verifiedViaBridge flag: a loopback upstream whose PRIOR probe succeeded via
host-gateway proves the bridge CAN reach the host. A later failed probe
is then near-conclusive evidence the upstream itself went dead. The
DC-053 code unconditionally marked loopback failures as unverifiable,
throwing away this signal. Now: track verifiedViaBridge per-upstream and
treat verified-then-failed as down (count failures, open incident after
DEAD_AFTER_MS=5min).
2. IN_CONTAINER=false kill-switch test (B-grade polish, folded into same
commit per conjoint-commit anti-pattern).
3. git.sami intermittent ENOTFOUND (~2/h in ssl-monitor TLS handshake):
pin git.sami -> 100.121.150.22 (DNS2 Tailscale) in container /etc/hosts
via --add-host in start.sh. Existing comment explicitly forbids pinning
to DNS3/100.81.59.99 (no HTTPS listener there); DNS2/100.121.150.22 is
correct (Caddy serves git.sami on DNS2:443 and routes to DNS3:3030
internally).
GLM-5.3 judge round 1 (208s, 6 tool calls, on-disk verified): grade B,
all 25 tests green, no blocking issues, 3 LOW polish suggestions.
Folded two actionable LOWs (persistence + JSDoc) into this commit:
- verifiedViaBridge now persisted in _saveState/_restoreUpstreamStates so
a known-good loopback upstream stays labeled across container restarts
(1-tick blip becomes 0-tick).
- snapshot() gained JSDoc describing the verifiedViaBridge semantic for
dashboard consumers.
Third LOW (long-term: prefer host-side liveness signal from Caddy) is a
roadmap note, not actionable now.
Tests: 1921/1921 (was 1910; +11 net: 6 new for items 2b-a/2b-b/persistence +
5 previously-skipped baseline). Full suite 86/86 green.
The caddy-upstream-watcher runs inside the dashcaddy-api container but
probes upstreams declared for Caddy, which runs on the HOST. Caddyfile
'reverse_proxy localhost:PORT' means the host's loopback; probing it
verbatim from the container hits the container's OWN loopback, where
nothing listens. Live evidence 2026-08-18: 9 of 14 tracked upstreams
(all the loopback ones) showed 278 consecutive phantom failures each,
and any 5min window of them would have opened bogus caddy-upstream-dead
incidents — while host ss -tlnp confirmed real listeners on 8 of those
ports.
Fix:
- Probe loopback targets via host.docker.internal instead, pinned to the
host bridge IP by start.sh (--add-host=host.docker.internal:host-gateway,
Docker >= 20.10). Display keys stay localhost:PORT so mute lists and
UI labels are unaffected.
- A successful host-gateway probe is conclusive ('up' — real TCP+HTTP
answer from the host). A FAILED probe is epistemically inconclusive
(127.0.0.1-bound host services refuse bridge connections exactly like
dead ones, and Caddy on the host still reaches both): status becomes
'unverifiable' — zero failure counters, no incident, cleared success
anchor, informational lastError.
- IN_CONTAINER=false disables the remap (bare-metal deployments).
- Snapshot sort extended: dead > down > muted > unverifiable > up > unknown.
Tests: 5 new (23/23 in suite) covering remap targeting (localhost,
127.0.0.1, 127.x), non-loopback pass-through, unverifiable semantics,
and sort order. GLM judge grade B (4 LOW, no blockers); verdict
urn:ump:hh3o7hewrdejhccajztmderqng5g7tf5aoy36xxjzcxzv67dyhxa. Regrade
with Codex when quota resets 2026-08-24.
Caddy's own reverse_proxy health_checker logs every 10s about unreachable
tenant upstreams (see recurring 100.120.159.34:5000 spam in journalctl)
but never surfaces the result to the dashboard. Adds:
- caddy-upstream-watcher.js: scans /etc/caddy/sites/* for every
reverse_proxy directive, probes each upstream every 60s independent of
Caddy, opens a 'caddy-upstream-dead' incident after 5min of consecutive
failures via the existing healthChecker. Mute list persisted to
data/caddy-upstreams.json. Probes stamp X-DashCaddy-HealthCheck: 1 so
forward_auth doesn't 401 them.
- routes/caddy-upstreams.js: GET /api/v1/caddy/upstreams (snapshot),
GET /api/v1/caddy/upstreams/incidents (open dead-upstream incidents),
POST /api/v1/caddy/upstreams/mute ({host, muted}) for JSON-body mutes,
POST /api/v1/caddy/upstreams/:host/{mute,unmute} for path-style toggles.
All mounted under the auth-gated apiRouter in app.js.
- 16 unit tests + 2 route smoke tests, all passing.
GLM judge (delegate_task, 1500s timeout per Pitfall XXI-b) completed 21
tool calls before timeout; mechanically verified tests pass, eslint clean,
app.js module load OK, RST-mid-body ECONNRESET is caught by req.on('error').
Found two MEDIUM defects which are now fixed in this commit:
1. (MEDIUM) scanSites file-extension filter `/\.(sami|caddy|conf)$/i`
silently skipped real prod filenames like zap.sami-ahmed.net,
samitest.space, blocks.cryptographic-triangles.org where the file
extension is .net/.space/.org. Replaced with positive filter that
excludes README/.bak/.swp/Caddyfile + content pre-check
(must contain 'reverse_proxy'). Added test covering the prod filenames.
2. (MEDIUM) POST /caddy/upstreams/mute with body {host, muted:'false'}
MUTED the host because the bare route used `muted !== false` which is
true for the string 'false'. Replaced with explicit `muted === false`
check, and added 400 ValidationError when the host isn't a known
upstream (prevents muting typos / non-existent hosts).
Self-grade: B+ (after applying GLM partial review). Re-grade with Codex
when its quota resets 2026-08-24.
Three coordinated changes to stop every gated *.sami service flipping
red after ~20 probes:
1. health-checker.js _doRequest() now sends X-DashCaddy-HealthCheck: 1
on every outgoing probe. Caddy uses this header (combined with a
trusted source IP via the new @healthcheckProbe matcher in the
dashcaddy_auth snippet) to bypass forward_auth for local container
probes. Without the bypass, forward_auth 401's every probe, and the
authLimiter (20 req / 15 min, DC-027) caps us out within minutes.
2. evaluateHealth() default expectedStatusCodes now includes 401, 403,
and 429. Defense in depth — if a future Caddy reload drops the
bypass, 429 from the rate-limited gate no longer marks the service
as down (it just means the gate answered, which proves the service
is reachable through Caddy).
3. (start.sh — already shipped on the running container, will land
with the next release build) ca.sami now maps to 100.121.150.22
(DNS2) instead of 127.0.0.1, which is the container's own loopback
where nothing serves :443. The CA web UI lives on DNS2's Caddy.
Tests:
- evaluateHealth: 401, 403, 429 accepted by default
- _doRequest: X-DashCaddy-HealthCheck: 1 always present
- _doRequest: user-supplied headers preserved alongside marker
Bump 1.14.7 → 1.14.8.
After DC-005 refactor moved health-checker.js into src/monitoring/, the
require path was never updated. Tests in __tests__/health-checker.test.js
failed with 'Cannot find module' → 59 cascading test failures in the
health-checker suite.
Path: src/monitoring/health-checker.js → 'require(./platform-paths)'
Fix: 'require(../../platform-paths)'
Verified: 921/922 tests passing (one known async-timing flake in
logging.test.js 'writes entry to ERROR_LOG_FILE with context').
After the DC-005 module reorganization (41 files moved into src/ subdirs),
138 test suites failed because the refactor script's path-rewrite logic
missed three categories:
1. Files inside src/ doing 'require("./src/...")' — should be 'require("../...")'
2. Files in src/X/Y/ doing 'require("../../../src/...")' — should be 'require("../../...")'
3. Test files in __tests__/ with leftover 'require("../../../src/...")' paths
Root cause: the original refactor script ran before all files were moved,
so it computed relative paths against stale filesystem state.
Result:
- 30/30 test suites pass
- 879/879 tests pass (was: 18/30 suites, 614/687 tests)
Also fixed:
- routes/apps/restore.js: wrong responses import path
- routes/*/*.js: '../../src/utilities/X' → '../src/utilities/X' (depth 2 routes)
Hermes