DC-053 follow-ups (queue item 2b). Three small fixes to the caddy-upstream-watcher: 1. verifiedViaBridge flag: a loopback upstream whose PRIOR probe succeeded via host-gateway proves the bridge CAN reach the host. A later failed probe is then near-conclusive evidence the upstream itself went dead. The DC-053 code unconditionally marked loopback failures as unverifiable, throwing away this signal. Now: track verifiedViaBridge per-upstream and treat verified-then-failed as down (count failures, open incident after DEAD_AFTER_MS=5min). 2. IN_CONTAINER=false kill-switch test (B-grade polish, folded into same commit per conjoint-commit anti-pattern). 3. git.sami intermittent ENOTFOUND (~2/h in ssl-monitor TLS handshake): pin git.sami -> 100.121.150.22 (DNS2 Tailscale) in container /etc/hosts via --add-host in start.sh. Existing comment explicitly forbids pinning to DNS3/100.81.59.99 (no HTTPS listener there); DNS2/100.121.150.22 is correct (Caddy serves git.sami on DNS2:443 and routes to DNS3:3030 internally). GLM-5.3 judge round 1 (208s, 6 tool calls, on-disk verified): grade B, all 25 tests green, no blocking issues, 3 LOW polish suggestions. Folded two actionable LOWs (persistence + JSDoc) into this commit: - verifiedViaBridge now persisted in _saveState/_restoreUpstreamStates so a known-good loopback upstream stays labeled across container restarts (1-tick blip becomes 0-tick). - snapshot() gained JSDoc describing the verifiedViaBridge semantic for dashboard consumers. Third LOW (long-term: prefer host-side liveness signal from Caddy) is a roadmap note, not actionable now. Tests: 1921/1921 (was 1910; +11 net: 6 new for items 2b-a/2b-b/persistence + 5 previously-skipped baseline). Full suite 86/86 green.
185 lines
9.0 KiB
Bash
Executable File
185 lines
9.0 KiB
Bash
Executable File
#!/bin/bash
|
|
set -e
|
|
CONTAINER_NAME="dashcaddy-api"
|
|
IMAGE="dashcaddy-dashcaddy-api:latest"
|
|
DATA_DIR="/opt/dashcaddy/dashcaddy-api/data"
|
|
CADDYFILE="/etc/caddy/Caddyfile"
|
|
ASSETS_DIR="/var/www/dashcaddy-status/assets"
|
|
UPDATES_DIR="/opt/dashcaddy/updates"
|
|
BACKUPS_DIR="/opt/dashcaddy/backups"
|
|
HOST_IP="172.17.0.1"
|
|
# Local Technitium (binds 0.0.0.0:53) resolves *.sami + recurses for docker subnet
|
|
# external fallback. Without this the container only has 8.8.8.8 and every
|
|
# *.sami health-check probe fails with ENOTFOUND (uptime bars stay empty).
|
|
DNS_PRIMARY="100.121.150.22" # Technitium (Tailscale IP) — resolves *.sami
|
|
DNS_FALLBACK="8.8.8.8"
|
|
|
|
# --- One-time migration from Docker image layer to bind mount --------------
|
|
# DC-039 follow-up. Before v1.14.10, certain modules (audit-logger, license-
|
|
# keygen, credential-manager) defaulted their files to /app/src/* via
|
|
# path.join(__dirname, 'foo.json'). Those writes landed in the Docker image
|
|
# layer and VANISHED on every container recreate. This step scans for any
|
|
# non-empty zombie files left over from a previous image (where /opt/dashcaddy/
|
|
# previously used /opt/dashcaddy/dashcaddy-api/src/... as the path root) and
|
|
# copies their contents into the bind-mounted data dir ONCE.
|
|
#
|
|
# Idempotent: bails out if the migration sentinel file already exists.
|
|
# Designed to be a no-op on every fresh install.
|
|
MIGRATION_SENTINEL="${DATA_DIR}/.migrated-from-image-layer"
|
|
IMAGE_LAYER_ZOMBIES=(
|
|
"/opt/dashcaddy/dashcaddy-api/src/security/audit-log.json"
|
|
"/opt/dashcaddy/dashcaddy-api/src/security/.encryption-key"
|
|
"/opt/dashcaddy/dashcaddy-api/src/security/.encryption-key.bak"
|
|
"/opt/dashcaddy/dashcaddy-api/src/utils/error.log"
|
|
"/opt/dashcaddy/dashcaddy-api/src/managers/.license-secret"
|
|
"/opt/dashcaddy/dashcaddy-api/src/managers/.license-counter"
|
|
)
|
|
# Note: set -e is active at top of script. Each per-file step uses an
|
|
# explicit `|| true` (or guarded `if`) so a single unreadable zombie file
|
|
# can't take down the whole container. The sentinel write at the end is
|
|
# outside any conditional so it always runs once.
|
|
run_image_layer_migration() {
|
|
if [ -f "${MIGRATION_SENTINEL}" ]; then
|
|
return 0
|
|
fi
|
|
mkdir -p "${DATA_DIR}" || { echo "[start.sh] [migration] mkdir failed: ${DATA_DIR}"; return 0; }
|
|
local migrated=0
|
|
for src in "${IMAGE_LAYER_ZOMBIES[@]}"; do
|
|
if [ -f "${src}" ] && [ -s "${src}" ]; then
|
|
local dest_name dest
|
|
dest_name="$(basename "${src}")"
|
|
dest="${DATA_DIR}/migrated-${dest_name}"
|
|
if [ ! -f "${dest}" ]; then
|
|
echo "[start.sh] [migration] Recovering image-layer file: ${src} -> ${dest}"
|
|
if cp -a "${src}" "${dest}" 2>/dev/null; then
|
|
migrated=$((migrated + 1))
|
|
else
|
|
echo "[start.sh] [migration] WARN: failed to copy ${src} (continuing)"
|
|
fi
|
|
fi
|
|
fi
|
|
done
|
|
if [ "$migrated" -gt 0 ]; then
|
|
echo "[start.sh] [migration] Recovered ${migrated} file(s) from image layer."
|
|
echo "[start.sh] [migration] Review files prefixed 'migrated-' in ${DATA_DIR} and merge or delete."
|
|
fi
|
|
# Sentinel write MUST run regardless of any per-file failure above.
|
|
date -u +%Y-%m-%dT%H:%M:%SZ > "${MIGRATION_SENTINEL}" 2>/dev/null || echo "1" > "${MIGRATION_SENTINEL}"
|
|
}
|
|
run_image_layer_migration
|
|
|
|
# --- /etc/hosts overrides for the container ---------------------------------
|
|
# The base image (node:20-alpine) has no entries for *.sami. We must inject
|
|
# them via --add-host so health checks inside the container can resolve LAN
|
|
# and Tailscale IPs to the right destinations.
|
|
#
|
|
# IMPORTANT: Do NOT add `git.sami:100.81.59.99` (DNS3). DNS3 does not serve
|
|
# HTTPS on 443 — it only serves Gitea on :3030. Setting git.sami → DNS3 in
|
|
# the container would make health checks bypass Caddy and hit a closed port.
|
|
# Let Caddy (on DNS2:443) handle git.sami and route to DNS3:3030 internally.
|
|
#
|
|
# Layout:
|
|
# dns3.sami / gitea → DNS3 (Tailscale IP, used by tools inside the container
|
|
# that need to talk to Gitea directly, e.g. backups)
|
|
# dns3-wan.sami → DNS3 Contabo WAN fallback
|
|
# dns2.sami → DNS2 (this host) — for cross-service references
|
|
# dns1.sami → DNS1 (SAMI-CLOUD-U32)
|
|
# dc-contabo-de → DashCaddy Contabo test instance
|
|
# git.dashcaddy.net → DashCaddy upstream git
|
|
# git.sami → DNS2 (NOT DNS3 — see warning above). Resolves an
|
|
# intermittent ENOTFOUND in the ssl-monitor's TLS
|
|
# handshake check (~2/h) by pinning the name in the
|
|
# container's /etc/hosts to the Caddy listener.
|
|
# ca.sami → local CA (DN2 + DN3 both have their own)
|
|
ADD_HOST_FLAGS=(
|
|
# host.docker.internal → host bridge IP (Docker host-gateway). The caddy
|
|
# upstream watcher probes Caddy site upstreams from INSIDE this container;
|
|
# `reverse_proxy localhost:PORT` in a site file means the HOST's loopback,
|
|
# so the watcher remaps loopback probe targets to this name (see
|
|
# dashcaddy-api/src/monitoring/caddy-upstream-watcher.js). Without this
|
|
# entry the probes would hit the container's own loopback and report every
|
|
# host-side upstream as dead.
|
|
--add-host=host.docker.internal:host-gateway
|
|
--add-host=dns3.sami:100.81.59.99
|
|
--add-host=gitea:100.81.59.99
|
|
--add-host=dns3-wan.sami:74.208.167.19
|
|
--add-host=dns2.sami:100.121.150.22
|
|
--add-host=dns1.sami:100.71.97.12
|
|
--add-host=dc-contabo-de:100.98.123.59
|
|
--add-host=git.dashcaddy.net:100.98.123.59
|
|
--add-host=git.sami:100.121.150.22
|
|
# ca.sami resolves via DNS to 100.121.150.22 (Caddy on DNS2). Don't pin
|
|
# to 127.0.0.1 — nothing listens on 443 inside the container, so the
|
|
# health checker would fail with ECONNREFUSED. The CA itself is a
|
|
# public-facing service that goes through Caddy just like every other *.sami.
|
|
)
|
|
|
|
# Always recreate to ensure env vars are correct (CONFIG_FILE defaults to /etc/dashcaddy/ which doesn't exist)
|
|
if docker ps -a --format "{{.Names}}" | grep -q "^${CONTAINER_NAME}$"; then
|
|
echo "[start.sh] Recreating container to apply correct env vars..."
|
|
docker rm -f ${CONTAINER_NAME}
|
|
fi
|
|
|
|
# Tailscale CLI + control socket — lets the container invoke
|
|
# `tailscale status --json` to populate /api/v1/tailscale/status etc.
|
|
# The binary is statically linked (Go), so the bind-mount works under
|
|
# the container's Alpine libc without any library forwarding.
|
|
# Both mounts are read-only: `tailscale status --json` is a read query
|
|
# that the local tailscaled handles; we never need to mutate state
|
|
# from inside the container.
|
|
echo "[start.sh] Creating container with full config..."
|
|
|
|
# Sync the freshly-built dashboard bundle into the static directory Caddy
|
|
# serves. The Docker image bakes dist/ from the source tree at build time,
|
|
# but DNS2 also serves /var/www/dashcaddy-status/dist/ (the original
|
|
# Windows installer mirror path). If we don't sync after every build, the
|
|
# served bundle keeps the OLD hash while the API responds with new code,
|
|
# which shows up in the dashboard as "version unavailable" + "no data"
|
|
# widgets because the new API surface doesn't match the old widget code.
|
|
# This step is idempotent and ~50ms — always safe to run.
|
|
echo "[start.sh] Syncing dashboard bundle into static dir..."
|
|
mkdir -p /var/www/dashcaddy-status/dist
|
|
if [ -d /opt/dashcaddy/status/dist ]; then
|
|
cp /opt/dashcaddy/status/dist/*.js /var/www/dashcaddy-status/dist/ 2>/dev/null || true
|
|
cp /opt/dashcaddy/status/sw.js /var/www/dashcaddy-status/ 2>/dev/null || true
|
|
cp /opt/dashcaddy/status/index.html /var/www/dashcaddy-status/ 2>/dev/null || true
|
|
echo "[start.sh] Bundle synced ($(ls /opt/dashcaddy/status/dist/*.js 2>/dev/null | wc -l) bundle files + sw.js + index.html)."
|
|
else
|
|
echo "[start.sh] WARN: /opt/dashcaddy/status/dist missing — skipping sync (frontend will be stale)."
|
|
fi
|
|
|
|
docker run -d --restart unless-stopped --name ${CONTAINER_NAME} \
|
|
--memory=1g --memory-swap=2g --cpus=2 \
|
|
--add-host=get.dashcaddy.net:194.233.88.206 \
|
|
--add-host=get2.dashcaddy.net:194.233.88.206 \
|
|
--dns ${DNS_PRIMARY} \
|
|
--dns ${DNS_FALLBACK} \
|
|
"${ADD_HOST_FLAGS[@]}" \
|
|
-p 127.0.0.1:3001:3001 \
|
|
-v ${DATA_DIR}:/app/data \
|
|
-v ${BACKUPS_DIR}:/app/backups \
|
|
-v ${CADDYFILE}:/caddyfile \
|
|
-v /etc/caddy/sites:/etc/caddy/sites:ro \
|
|
-v /var/run/docker.sock:/var/run/docker.sock \
|
|
-v ${ASSETS_DIR}:/app/assets \
|
|
-v ${UPDATES_DIR}:/app/updates \
|
|
-v /opt/sami-files/logs:/opt/sami-files/logs:ro \
|
|
-v /usr/bin/tailscale:/usr/bin/tailscale:ro \
|
|
-v /var/run/tailscale:/var/run/tailscale:ro \
|
|
-v /etc/ssl/sami-ca:/etc/ssl/sami-ca:ro \
|
|
-e NODE_ENV=production \
|
|
-e SERVICES_FILE=/app/data/services.json \
|
|
-e CONFIG_FILE=/app/data/config.json \
|
|
-e BACKUP_DIR=/app/backups \
|
|
-e DNS_CREDENTIALS_FILE=/app/data/dns-credentials.json \
|
|
-e CREDENTIALS_FILE=/app/data/credentials.json \
|
|
-e ENCRYPTION_KEY_FILE=/app/data/.encryption-key \
|
|
-e HEALTH_HISTORY_FILE=/app/data/health-history.json \
|
|
-e HEALTH_CONFIG_FILE=/app/data/health-config.json \
|
|
-e CADDYFILE_PATH=/caddyfile \
|
|
-e CADDY_ADMIN_URL=http://${HOST_IP}:2019 \
|
|
-e ASSETS_DIR=/app/assets \
|
|
-e DASHCADDY_API_SOURCE_DIR=/opt/dashcaddy/dashcaddy-api \
|
|
-e DASHCADDY_UPDATE_ENABLED=false \
|
|
-e CA_CERT_PATH=/etc/ssl/sami-ca/root.crt \
|
|
${IMAGE} |