The caddy-upstream-watcher runs inside the dashcaddy-api container but
probes upstreams declared for Caddy, which runs on the HOST. Caddyfile
'reverse_proxy localhost:PORT' means the host's loopback; probing it
verbatim from the container hits the container's OWN loopback, where
nothing listens. Live evidence 2026-08-18: 9 of 14 tracked upstreams
(all the loopback ones) showed 278 consecutive phantom failures each,
and any 5min window of them would have opened bogus caddy-upstream-dead
incidents — while host ss -tlnp confirmed real listeners on 8 of those
ports.
Fix:
- Probe loopback targets via host.docker.internal instead, pinned to the
host bridge IP by start.sh (--add-host=host.docker.internal:host-gateway,
Docker >= 20.10). Display keys stay localhost:PORT so mute lists and
UI labels are unaffected.
- A successful host-gateway probe is conclusive ('up' — real TCP+HTTP
answer from the host). A FAILED probe is epistemically inconclusive
(127.0.0.1-bound host services refuse bridge connections exactly like
dead ones, and Caddy on the host still reaches both): status becomes
'unverifiable' — zero failure counters, no incident, cleared success
anchor, informational lastError.
- IN_CONTAINER=false disables the remap (bare-metal deployments).
- Snapshot sort extended: dead > down > muted > unverifiable > up > unknown.
Tests: 5 new (23/23 in suite) covering remap targeting (localhost,
127.0.0.1, 127.x), non-loopback pass-through, unverifiable semantics,
and sort order. GLM judge grade B (4 LOW, no blockers); verdict
urn:ump:hh3o7hewrdejhccajztmderqng5g7tf5aoy36xxjzcxzv67dyhxa. Regrade
with Codex when quota resets 2026-08-24.
The DC-049 dead-upstream watcher reads reverse_proxy host directives from
/etc/caddy/sites/*. Bind-mount the directory into the container so the
in-container watcher can see what the host's Caddy is configured to proxy.
Without this mount the watcher would see zero sites and silently no-op.
After every rebuild the freshly-baked dashboard bundle lives in
/opt/dashcaddy/status/dist/ + sw.js. DNS2 also serves files from
/var/www/dashcaddy-status/dist/ + sw.js (the original Windows-installer
mirror path). Without an explicit copy step between build and start.sh,
the served bundle stays on whatever hash was there before, while the API
responds with new code. That mismatch is what shows up in the dashboard
as "version unavailable" + "no data" widgets — saw it in the deploy
that followed DC-046/047 (fixed by a manual cp this time, never again).
Sync block runs before docker run:
cp /opt/dashcaddy/status/dist/*.js /var/www/dashcaddy-status/dist/
cp /opt/dashcaddy/status/sw.js /var/www/dashcaddy-status/
cp /opt/dashcaddy/status/index.html /var/www/dashcaddy-status/
All guarded so set -e doesn't kill the container start on a
single per-file failure (e.g. read-only mount, missing dir). Missing
source dir is a WARN + no-op rather than a fatal — fresh installs
without status/dist/ don't get a stale-bundle problem, just a log line.
Test: scripts/test-start-sh-sync.sh — 7 assertions across 4 cases
(fresh-copy, missing-source, idempotent-re-sync, set-e-survives-permission-
denied). All pass.
Three-part fix for the silent data-loss failure mode that survives DC-039:
If SERVICES_FILE env was unset, platformPaths.dataDir resolved to /etc/dashcaddy
(image-layer path), and audit/license/error logs would silently land there and
vanish on every container recreate.
1. platform-paths.assertSafe({mode:'production'}) — throws FATAL on forbidden
zones (/app/src,routes,scripts,utils,managers,security + /etc/* + /usr + /var).
Bypassed with SKIP_DATA_DIR_GUARD=1.
2. server.js calls assertSafe() before any runtime work.
3. start.sh one-time migration: scans 6 known image-layer zombie paths,
copies non-empty content to bind mount with 'migrated-' prefix,
gated by sentinel file. Survives set -e per-file failures.
19/19 platform-paths tests + 5/5 shell migration tests.
Suite: 1066/1067 (1 pre-existing public-routes-drift failure from in-flight
auth refactor, untouched by this commit).
Verified live on DNS2: live audit log at /app/data/audit-log.json (315KB,
active) is unaffected; vestigial 2-byte /app/src/security/audit-log.json +
140KB /app/src/utils/error.log (pre-DC-039 era) will be recovered on next
container recreate.
Two related fixes from the dashboard 11/15 false-negatives:
1. The Sami Home Network CA cert (/etc/ssl/sami-ca/root.crt) was not
mounted into the container, so the health-checker's HTTPS probe to
*.sami hosts failed with "certificate verify failed". Added a bind
mount + CA_CERT_PATH env var so the app's httpsAgent picks it up
(verified at startup: "HTTPS agent configured with CA certificate").
2. The --add-host=ca.sami:127.0.0.1 line pinned ca.sami to the
container's loopback, but nothing listens on 443 inside the
container. Probe failed with ECONNREFUSED 127.0.0.1:443. Removed
the override so ca.sami resolves via DNS to 100.121.150.22 (Caddy
on DNS2) and the probe reaches the real service.
After both fixes: 15/15 services healthy, 0 429s on the health checker,
caddy.ok=true on /health/ready.
The previous getTailscaleStatus() in src/app.js was a hard-coded
`return null` stub with a TODO saying it would be populated by context.
The context had a tailscale.* namespace declared with null function
stubs (routes/context.js:71), but nothing ever set them to real
functions. routes/tailscale.js has been calling ctx.tailscale.getStatus()
/ getLocalIP() / isTailscaleIP() and getting undefined back, silently
returning empty device lists. The tailscaleAuthMiddleware's allowedTailnet
check (DC-121, device-not-in-tailnet 403) was dead code for the same reason.
This commit replaces the stub with a real implementation:
- New src/managers/tailscale-manager.js shells out to the host's
`tailscale status --json` (cached 5 minutes), parses the result, and
exposes getStatus / getLocalIP / getSummary / getDevices / isTailscaleIP /
invalidateCache / getAccessToken (stub) / startSyncTimer / stopSyncTimer
/ syncAPI (stub). All failure modes (CLI missing, tailscaled down,
malformed JSON, EACCES) are handled gracefully — return null with no
cache poisoning.
- src/context/index.js now wires the manager into ctx.tailscale.* so
routes/tailscale.js and middleware.js's allowedTailnet gate get the
real functions.
- src/app.js:189 getTailscaleStatus() now delegates to the manager
instead of returning null.
- The duplicate isTailscaleIP() in src/app.js:179 (no malformed-input
guards) is removed in favor of the canonical version in
src/utilities/network-detector.js (DC-031) which the manager also uses.
- start.sh now bind-mounts /usr/bin/tailscale (statically linked Go binary
— works under Alpine libc) and /var/run/tailscale/ into the container,
read-only. Lets the container invoke the CLI without needing its own
tailscale install.
- 41 new unit tests in __tests__/tailscale-manager.test.js cover: CLI
success/missing/daemon-down/malformed-JSON paths, 5-min cache hit/miss,
1-hour installed-cache hit/miss, getLocalIP IPv4/IPv6/missing-choices,
getSummary shape, getDevices shape with full + minimal peer fields,
startSyncTimer/stopSyncTimer interval + idempotency, TAILSCALE_BIN env
override.
Total: 1138 tests pass (was 1097, +41 new), 0 new ESLint warnings.
What this unlocks:
- /api/v1/tailscale/status → real installed/connected/hostname/ip/
peerCount/onlinePeerCount summary instead of empty
- /api/v1/tailscale/devices → real device list (was returning [])
- /api/v1/tailscale/check-connection → works (uses real isTailscaleIP)
- tailscaleAuthMiddleware allowedTailnet check (DC-121) is no longer
dead code — a request from a Tailscale IP not in the allowed tailnet
now actually gets 403 instead of being silently allowed.
The container's health-checker runs against Caddy via /etc/hosts resolution.
The node:20-alpine base image has no entries for *.sami, so without explicit
--add-host flags every *.sami probe resolves via the configured DNS server
(100.121.150.22 Technitium or 8.8.8.8) — both of which DO resolve *.sami but
return the WAN/Tailscale IP. That works for most services because Caddy on
DNS2:443 handles them.
BUT: a previous container run passed --add-host=git.sami:100.81.59.99
(DNS3's Tailscale IP). DNS3 does NOT serve HTTPS on 443 — Gitea listens on
:3030 only. So git.sami health checks inside the container hit DNS3:443,
get ECONNREFUSED, and the dashboard shows git.sami as down even though Caddy
on DNS2:443 correctly routes git.sami → 100.81.59.99:3030.
Fix: inject the correct --add-host flags from start.sh (the source of truth
for container setup) so future recreates get consistent resolution. git.sami
is intentionally left OUT — Caddy on DNS2:443 is the only correct ingress
for git.sami traffic.
Also documents the rationale so the next person doesn't reintroduce the
git.sami override by accident.
Live verified:
- container /etc/hosts has all needed entries except git.sami
- curl https://git.sami/ from inside container → 200 (via Caddy on :443)
- curl https://sync.sami/ from inside container → 302 (upstream redirect)
- curl https://router.sami/ from inside container → 302 (upstream redirect)
[DC-026] routes/auth/sso-gate.js — fix sessionDuration='never' bypass
Both /auth/gate/:serviceId and /auth/app-token/:serviceId had a session
check gated on `sessionDuration !== 'never'`. An admin setting TOTP to
never-expire accidentally created an authentication-free path to credential
injection (Basic Auth, X-Api-Key, Plex/Prowlarr tokens). Patched: session
required whenever TOTP is enabled, period. Added 8 regression tests.
[DC-027] src/utilities/middleware.js — rate limit /auth/*
New authLimiter (20 req / 15 min) on /auth/keys, /auth/jwt, /auth/gate,
/auth/app-token. These endpoints expose credentials and were unmetered.
Without this, an attacker with a guessed session cookie could burn through
every credential-touching endpoint. Added 5 tests.
[DC-028] src/security/audit-logger.js — log credential exposures
/auth/gate and /auth/app-token were in SKIP_PATHS, silently dropping
every credential-exposure event from the audit log. Combined with the
GET-skip rule, NONE of these events were being recorded. Now logged
with named actions: auth.credential-injection, auth.app-token-issue,
auth.api-key-generate, auth.api-key-revoke, auth.jwt-mint. Added 9 tests.
[start.sh] Disable in-container self-updater
DASHCADDY_UPDATE_ENABLED=false. Without this, the container kept writing
trigger.json every 30 min and clobbered my in-progress host edits. The
path unit on the host is still active for manual triggers, but the
container won't auto-update itself — only when an admin clicks the
update button or a new release is manually published.
[package.json] Bump to 1.14.7
Test results: 1066/1066 passing across 39 suites (added 22 new tests).
- VERSION: bump from 1.14.4 to 1.14.6 to match package.json (HEAD had stale value)
- middleware.js: apply existing totpLimiter (10/15min) to /totp/setup endpoint
(was previously unmetered, allowing secret enumeration)
- dashcaddy-update.sh: hook post-deploy-patches.sh into the update flow
so the container can survive transitions between broken → fixed tarballs
- start.sh: add --add-host flags for get.dashcaddy.net and get2.dashcaddy.net
so the container can resolve the release server (was failing with ENOTFOUND)
Hermes