PRODUCT-SPEC-DECISIONS: lock the 14 product decisions

Source-of-truth for what we're building. Brief restatement: - Pricing: Free $0; 30d $20, 90d $50, 180d $70, 365d $99 (USD-only, Stripe) - Tiers: Free = up to 3 users + no sharing; Pro (any duration) = unlimited users + Tailscale-mediated share + public share links - Auth: host owner can use TOTP only; invitees MUST use email magic link (email is the identity for non-host users) - License: reuse existing license-keygen.js (HMAC-signed, 16-byte codes); offline validation; no phone-home; LIFETIME is creator-only - Account: optional dashcaddy.net account for subscription mgmt (v1.0+ uses Stripe Checkout + emailed license key; account creation deferred) Build order unblocked: - DC-052 license enforcement (Free cap at 3 users, gate share on Pro) - DC-053 share routes (Tailscale-mediated + public links) - DC-054 Stripe webhook bridge - DC-055 pricing page - DC-056 ToS + Privacy Policy
PRODUCT-SPEC-DECISIONS + DC-052–056 build pipeline
2026-07-20 21:01:10 -07:00 · 2026-07-20 21:00:54 -07:00 · 2026-07-20 20:01:42 -07:00
3 changed files with 669 additions and 1 deletions
@@ -211,9 +211,10 @@
 - **result:** Empirically measured against **all 4 release versions** + origin/main: v1.14.4 (broken — no src/ in tarball), v1.14.8, v1.14.9, and origin/main all produce **0 require-fixes applied** under the old script. Every patch is a no-op against every current release. Decision: **KEEP the script but repurpose it as a VERIFIER, not a patcher.** The script now performs 5 explicit checks (server.js requires correct, license-manager.js path correct, src/ directory present + non-empty + contains app.js, license-keygen.js at API root) + an informational scan of all src/ require paths. **Exits 1 if any check fails** — fails the build loudly instead of silently letting a crash-looping container reach production. Behaviour change: the OLD script would silently no-op on v1.14.4 (couldn't find src/ to patch); the NEW script reports `=== FAILED CHECKS ===` with the specific failures (e.g. `src/: directory missing — v1.14.4-class bug`). Verified against v1.14.4 tarball: old script 0 patches + exit 0, new script 2 failures + exit 1 + clear error names the v1.14.4-class bug. New regression test `dashcaddy-api/scripts/test-dashcaddy-post-deploy-verifier.sh` (17 assertions across 10 test groups including clean tree, missing server.js, broken server.js requires, missing src/, missing license-keygen.js, broken license-manager path, empty src/, missing src/app.js, absolute path resolution, non-existent API_DIR) — all pass. Tests: 1214/1214 Jest + 31 shell assertions. Lint: 150 warnings, all pre-existing in untouched files.

 ### DC-041: Add integration test for the auto-update pipeline (trigger.json → bash → docker rebuild → health check → result.json)
- **status:** in-progress
+- **status:** done (commit 0b85caa, 5 scenarios / 37 assertions all green)
 - **owner:** hermes
 - **details:** The host-side updater has zero integration coverage. The recent DC-033 incident showed this whole chain is one big untested path. Build a test harness that: (1) creates a temporary directory mimicking `/opt/dashcaddy/updates/staging/dashcaddy-api` with a known-good tarball. (2) writes a `trigger.json` to a test `UPDATES_DIR`. (3) runs `bash /opt/dashcaddy/scripts/dashcaddy-update.sh` with paths overridden via env vars. (4) asserts `result.json` has `success: true` and the version matches. (5) cleans up. Effort: ~2 hours. Risk: medium — the script uses `docker build` so the test needs either Docker-in-Docker (DinD) or mocking the docker calls.
+- **result:** `dashcaddy-api/scripts/test-dashcaddy-update-integration.sh` (552 lines) commits and exits 0. Strategy: sandbox at `/tmp/dashcaddy-test-XXXXXX/opt/dashcaddy/` with `/opt/dashcaddy` path-rewritten via `sed`, mocked `docker` binary prepended to PATH, real `dashcaddy-post-deploy-patches.sh` verifier copied in, and a Python one-shot HTTP responder on port 33001 driving the health check (33001 chosen to avoid clashing with the live DashCaddy API on 3001). 5 scenarios: (1) happy-path update v1.14.8→v1.14.9 with mocked docker build/rm/run, backups, result.json; (2) v1.14.4-class broken tarball (no src/) — asserts the verifier IS invoked and DOES detect the bug ("Build should be ABORTED" in log); current `dashcaddy-update.sh` warns-and-continues on verifier failure, so this scenario asserts that observed behavior with a TODO note about closing that gap in a follow-up; (3) rollback to a pre-populated backup; (4) no trigger.json → no-op exit 0; (5) prerelease channel rejection when `ALLOW_PRERELEASE` is not set.
 - **impact:** Closes the biggest untested surface in DashCaddy. Would have caught the v1.14.4 packaging bug immediately on the next release.

 ### DC-042: Replace null stubs in src/app.js getTailscaleStatus() with real Tailscale manager
@@ -305,6 +306,41 @@ Sami explicitly stated he wants email auth as an OPTION alongside TOTP, not a re
 - **Fix (three parts):** (1) `platform-paths.assertSafe({ mode })` — throws a clear FATAL in production mode if dataDir resolves into any of 11 forbidden zones (`/app/src`, `/app/routes`, `/app/scripts`, `/app/utils`, `/app/managers`, `/app/security`, `/etc`, `/etc/caddy`, `/etc/dashcaddy`, `/usr`, `/usr/local`, `/var`, `/var/lib/caddy`). Calls a second predicate `isMountedCheck(dir)` that returns false for non-writable or non-existent dirs (Windows warning, not throw). Bypassed with `SKIP_DATA_DIR_GUARD=1`. (2) `server.js:35` — calls `assertSafe` before any other startup work. Refuses to boot loudly instead of running with a path that loses data silently. (3) `start.sh:13-66` — one-time migration step runs before `docker run`. Scans 6 known image-layer zombie paths (`/opt/dashcaddy/dashcaddy-api/src/{security,utils,managers}/*`), copies any non-empty content to `${DATA_DIR}/migrated-*`, gates one-shot with a sentinel file `.migrated-from-image-layer`. Idempotent. Survives `set -e` per-file failures. Per-file `cp -a` guarded so a single unreadable zombie can't take the container down. Will recover the 140 KB `error.log` that the live DNS2 container has in its image layer (timestamp Jul 6 — pre-DC-039 era).
 - **result:** 19/19 platform-paths tests pass (8 new for assertSafe + 3 new for isMountedCheck). 5/5 start.sh migration tests pass (sentinel-skips, file-copies, idempotent-no-clobber, empty-file-skip, set-e-survives-failure). DNS2 deploys unchanged except for the new migration step running once on next recreate. Suite overall: 1066/1067 (one pre-existing public-routes-drift failure from in-flight Track A code, untouched).

+### DC-052: License-tier enforcement — Free caps user count at 3, gates share features on Pro
+- **status:** todo
+- **owner:** unclaimed
+- **details:** Per `/root/dashcaddy/PRODUCT-SPEC-DECISIONS.md` (locked 2026-07-20): Free = up to 3 users, Pro = unlimited. The DC-048 user-store needs a `countUsers()` helper. The `/api/v1/auth/admin/invites` POST handler must check `if (users.count() >= 3 && !licenseManager.isPro()) throw new ValidationError('upgrade required', 'tier')`. Same check on `POST /admin/users` (pre-authorize). Share-link creation routes (DC-053) gate on `licenseManager.isPro()`. LIFETIME keys are admin-only — the license-keygen CLI does not expose `--lifetime` publicly; only Sami runs the `isLifetime` path on his dev machine. The API never accepts a LIFETIME code in production (rejects at `verifyCode` time if duration is 0 outside dev).
+- **impact:** First pricing enforcement. Without this, Pro is just a label. With this, every upgrade path has a clear moment to upsell.
+- **prerequisite:** DC-048 (shipped).
+
+### DC-053: Public share links + Tailscale-mediated share — Pro-gated
+- **status:** todo
+- **owner:** unclaimed
+- **details:** Two new feature surfaces behind a Pro license check. (1) **Public share links** — `POST /api/v1/share` creates a signed URL (e.g. `https://status.sami/share/<token>`) for a specific service + a TTL (1h/24h/7d). The share page renders a read-only preview: service metadata + a `subscribe` button that hits `/api/v1/share/:token/subscribe` to register the visitor's email for updates. (2) **Tailscale-mediated share** — `POST /api/v1/share/tailscale` generates a Tailscale pre-auth key (one-shot, single-use, 24h) scoped to a specific device tag, emails the link to the invitee; clicking it joins them to the host's tailnet and proxies them to the service. Both surfaces gated on `licenseManager.isPro()` (DC-052). UI: a "Share" button on each service card, modal with the two tabs.
+- **impact:** The killer Pro feature. "Share your services with anyone, they don't even need a Tailscale account" — that's the pitch. Without this, Pro has no upgrade pull.
+- **prerequisite:** DC-042 + DC-043 (Tailscale manager + coord API shipped); DC-052 (license check); DC-048 (invite flow model).
+
+### DC-054: License-keygen CLI improvements + Stripe webhook bridge script
+- **status:** todo
+- **owner:** unclaimed
+- **details:** Existing `dashcaddy-api/license-keygen.js` already supports durations [30, 90, 180, 365]. Three additions: (1) `--tier pro` flag (currently `--duration 30/90/180/365` — duration alone implies Pro, so the flag is just for CLI clarity). (2) `dashcaddy-api/scripts/stripe-license-bridge.js` — listens on `STRIPE_WEBHOOK_SECRET`, validates `checkout.session.completed` events, looks up the duration by SKU ID, generates a license key, emails it to the customer, returns `{delivered: true}` to Stripe. (3) `dashcaddy-api/license-keygen.js` validation path — already exists, no change. Sami generates initial keys via CLI for the launch.
+- **impact:** Closes the loop between Stripe payment and license-key delivery. Without this, every sale requires manual key generation by Sami.
+- **prerequisite:** None. Stripe-side can be set up in parallel with DC-052.
+
+### DC-055: dashcaddy.net/pricing static page + Stripe Checkout integration
+- **status:** todo
+- **owner:** unclaimed
+- **details:** Static page at `/pricing` showing the 5-row tier table (Free / 1mo / 3mo / 6mo / 12mo). Stripe Checkout button per paid tier. On success, the page reveals the license key with copy-button + "Here's how to install it" link. Receipt email sent via Stripe's built-in. No account creation in this flow (Q10 decision — optional dashcaddy.net account is post-v1.0).
+- **impact:** The conversion surface. Without this, the product is real but unsellable.
+- **prerequisite:** DC-054 (Stripe webhook bridge so licenses auto-issue).
+
+### DC-056: ToS + Privacy Policy pages — GDPR-aware, no SOC2/HIPAA for v1.0
+- **status:** todo
+- **owner:** unclaimed
+- **details:** Two static pages at `/legal/tos` and `/legal/privacy`. ToS covers: license terms (per-host, non-transferable), prohibited use, refund policy (Stripe 30-day), termination. Privacy Policy covers: data collected (license key, host metadata, optional email), data NOT collected, third parties (Stripe — payment, Tailscale — coord API calls only when operator configures it), GDPR rights (access, deletion, portability — even though we have no central account system, we'll respond to direct requests within 30 days). No SOC2/HIPAA — that's a v2 conversation.
+- **impact:** Legal compliance for taking money. Stripe can technically sell without these but payment processors flag accounts without them.
+- **prerequisite:** None.
+
 ### Backlog note (2026-07-14)

 Tickets DC-046 through DC-049 implement pluggable auth + email magic link. Sami explicitly stated he wants email auth as an OPTION alongside TOTP, not a replacement — TOTP remains his primary method for personal/network-only access, email magic link is for public-product readiness. Architecture choice: `AuthProvider` interface in `src/auth/providers/` so future methods (OIDC, SAML, passkeys) plug in without further refactors. SMTP delivery reuses the existing `nodemailer` integration in `src/managers/notification-manager.js:290` — no new dependency. Sami plans to use the SMTP server his website runs (sami-ahmed.net) so the host field will be configurable. Total estimated effort: ~7 hrs, can ship in any order DC-046 → DC-047 → DC-048 → DC-049, but DC-046 is the foundation.
@@ -0,0 +1,80 @@
+# DashCaddy Product-Spec Decisions — Locked 2026-07-20
+
+> All decisions captured from clarifying questions with the operator. This
+> file is the source of truth for what gets built next. The narrative
+> PRODUCT-SPEC.md retains the longer "what we considered" context; this
+> file is what we *shipped*.
+
+## 1. Pricing
+
+| Tier | Duration | Price | Per-month equiv |
+|---|---|---|---|
+| Free | unlimited | $0 | $0 |
+| 1 month | 30 days | $20 | $20.00 |
+| 3 months | 90 days | $50 | $16.67 (17% off) |
+| 6 months | 180 days | $70 | $11.67 (42% off) |
+| 12 months | 365 days | $99 | $8.25 (59% off) |
+
+- Stripe Checkout only (no Paddle for v1.0)
+- USD only (defer multi-currency to v1.1)
+- Stripe-standard 30-day refund
+- No launch pricing — list prices as-is
+
+## 2. Tier features
+
+**Free:**
+- All self-hosted features, unlimited services
+- Up to 3 users (host owner + 2 invitees)
+- NO share links (no Tailscale-mediated share, no public share URLs)
+- Host owner may use TOTP-only login (no email required)
+
+**Pro (any paid duration):**
+- Unlimited users (no cap on invitees)
+- Tailscale-mediated share — invitees click a link, get scoped access via tailnet without configuring anything
+- Public share links — signed URLs for read-only previews (no Tailscale needed)
+- Cloud config backup (deferred to v1.1, but already on roadmap)
+
+The host's invitees MUST use email magic link as their identity — the email IS the username for non-host users. The host themselves can stay TOTP-only.
+
+## 3. Account / license model
+
+- **Use existing `license-keygen.js`** (HMAC-signed 16-byte codes; VALID_DURATIONS = [30, 90, 180, 365]).
+- License keys are per-host. One license = one host. Multi-host deferred to post-v1.0.
+- License validation is **fully offline** — no phone-home, no account required for the instance.
+- Purchase flow:
+  1. User picks tier on dashcaddy.net/pricing
+  2. Stripe Checkout → success page shows license key
+  3. Receipt email includes the license key as backup
+  4. User pastes key into their instance → Pro features unlock
+- **Optional** dashcaddy.net account (post-purchase) for managing subscription, downloading past invoices, recovering license keys. Deferred to v1.1.
+
+## 4. Invitee auth flow
+
+When host enables email auth via `siteConfig.authProviders.email.enabled = true`:
+- First email to log in becomes the bootstrap admin (existing DC-048 behavior)
+- Host generates invite via `/api/v1/auth/admin/invites` (existing DC-048)
+- Invitee receives magic-link email → clicks → POSTs token to `/api/v1/auth/invites/:token/accept` → user record created + session cookie set
+- Magic-link TTL = 24 hours; single-use
+
+## 5. What we deferred to post-v1.0
+
+- Multi-host support (one license = one host for v1.0)
+- Multi-currency pricing (USD only)
+- Custom Pro trial (rely on existing EULA 30-day evaluation)
+- Launch / founders / discount codes
+- Central dashcaddy.net accounts (subscription management)
+- Cloud config backup (Pro feature placeholder)
+- SAML SSO (was Business-tier; dropped since we have no Business tier)
+- Hosted offering (cloud.dashcaddy.net — separate ops burden, deferred entirely)
+
+## 6. Build order — what this enables
+
+This decision set unblocks the following build items, in priority order:
+
+1. **License-tier enforcement in the API.** Now that Free = up to 3 users, the existing DC-048 user-store needs a `countUsers()` helper + a check on user-creation that fires `402 Payment Required` when the cap is exceeded without a Pro license. (DC-052)
+2. **Pro-gated share-link routes.** Public-share-link routes (`/api/v1/share/:token`) + Tailscale-mediated share routes. Both gated on `licenseManager.isPro()`. (DC-053)
+3. **License-keygen CLI improvements.** The existing tool already supports the 4 durations. Needs a `--tier` flag and a Stripe-webhook bridge script (`scripts/stripe-license-bridge.js`) that converts a Stripe Checkout success → license key + email. (DC-054)
+4. **dashcaddy.net pricing page.** Static page at `/pricing` showing the tier table, Stripe Checkout button, and license-key reveal UI on success. (DC-055)
+5. **Compliance minimums.** ToS + Privacy Policy at `/legal/tos` and `/legal/privacy`. GDPR-aware, no SOC2/HIPAA. (DC-056)
+
+The DC-048 multi-user foundation is the gating prerequisite for items 1-2. That foundation already shipped.
@@ -0,0 +1,552 @@
+#!/usr/bin/env bash
+# Integration test harness for the dashcaddy-update.sh auto-update pipeline.
+#
+# Exercises the FULL flow:
+#   trigger.json -> backup -> verifier -> docker build (mocked) -> docker run (mocked)
+#                  -> health check (mocked) -> result.json -> cleanup
+#
+# Run from dashcaddy-api/scripts/:
+#   bash test-dashcaddy-update-integration.sh
+#
+# Strategy: build a sandbox at /tmp/dashcaddy-test-XXXXXX/ that mimics
+# /opt/dashcaddy/ on DNS2, then run a copy of dashcaddy-update.sh with all
+# hardcoded /opt/dashcaddy paths rewritten to the sandbox path. Mocked
+# binaries (docker) and a Python one-shot health server live in the sandbox
+# and are prepended to PATH / invoked via a python orchestrator.
+#
+# Each test scenario sets up a synthetic "from" deployment, writes a
+# trigger.json, runs the pipeline via the python orchestrator (which manages
+# the health server lifecycle), and asserts the resulting result.json +
+# filesystem state.
+#
+# Exit 0 = all scenarios pass, non-zero = at least one failed.
+
+set -uo pipefail
+
+readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+# Resolve dashcaddy-update.sh — try local then canonical location
+UPDATE_SCRIPT_SRC="${SCRIPT_DIR}/dashcaddy-update.sh"
+[[ ! -f "$UPDATE_SCRIPT_SRC" ]] && UPDATE_SCRIPT_SRC="$(cd "${SCRIPT_DIR}/../../scripts" 2>/dev/null && pwd)/dashcaddy-update.sh"
+
+if [[ ! -f "$UPDATE_SCRIPT_SRC" ]]; then
+  echo "FAIL: dashcaddy-update.sh not found"
+  exit 1
+fi
+
+# ── Test harness infrastructure ──────────────────────────────────────────────
+pass=0
+fail=0
+
+assert_eq() {
+  local desc="$1" expected="$2" actual="$3"
+  if [[ "$expected" == "$actual" ]]; then
+    echo "  PASS: $desc"
+    pass=$(( pass + 1 ))
+  else
+    echo "  FAIL: $desc — expected '$expected', got '$actual'"
+    fail=$(( fail + 1 ))
+  fi
+}
+
+assert_file_exists() {
+  local desc="$1" file="$2"
+  if [[ -f "$file" ]]; then
+    echo "  PASS: $desc"
+    pass=$(( pass + 1 ))
+  else
+    echo "  FAIL: $desc — file '$file' does not exist"
+    fail=$(( fail + 1 ))
+  fi
+}
+
+assert_dir_exists() {
+  local desc="$1" dir="$2"
+  if [[ -d "$dir" ]]; then
+    echo "  PASS: $desc"
+    pass=$(( pass + 1 ))
+  else
+    echo "  FAIL: $desc — dir '$dir' does not exist"
+    fail=$(( fail + 1 ))
+  fi
+}
+
+assert_json_field() {
+  local desc="$1" file="$2" field="$3" expected="$4"
+  local actual
+  actual=$(python3 -c "import json; d=json.load(open('$file')); print(d.get('$field', '<MISSING>'))" 2>/dev/null || echo "<PARSE_ERROR>")
+  assert_eq "$desc" "$expected" "$actual"
+}
+
+assert_grep() {
+  local desc="$1" file="$2" pattern="$3"
+  if grep -qE "$pattern" "$file" 2>/dev/null; then
+    echo "  PASS: $desc"
+    pass=$(( pass + 1 ))
+  else
+    echo "  FAIL: $desc — pattern '$pattern' not in $file"
+    fail=$(( fail + 1 ))
+  fi
+}
+
+assert_not_exists() {
+  local desc="$1" file="$2"
+  if [[ ! -e "$file" ]]; then
+    echo "  PASS: $desc"
+    pass=$(( pass + 1 ))
+  else
+    echo "  FAIL: $desc — file '$file' exists but should not"
+    fail=$(( fail + 1 ))
+  fi
+}
+
+# ── Python orchestrator ──────────────────────────────────────────────────────
+# A single Python script that:
+#   1. Starts a one-shot HTTP responder on a given port (returns 200 OK or
+#      503 based on env var)
+#   2. Forks the pipeline as a subprocess
+#   3. After pipeline exits, kills the responder
+#   4. Writes the pipeline's exit code + log to disk for assertions
+#
+# This avoids backgrounding from inside a foreground bash tool.
+ORCHESTRATOR_SRC="$(cat << 'PYEOF'
+import http.server
+import socketserver
+import subprocess
+import sys
+import os
+import time
+import threading
+
+PORT = int(os.environ.get("HEALTH_PORT", "33001"))
+HEALTH_OK = os.environ.get("HEALTH_SHOULD_PASS", "yes") == "yes"
+COMMAND = os.environ.get("PIPELINE_CMD", "")
+LOG_FILE = os.environ.get("PIPELINE_LOG", "/tmp/pipeline.log")
+RC_FILE = os.environ.get("PIPELINE_RC_FILE", "/tmp/pipeline.rc")
+MAX_HEALTH_REQUESTS = int(os.environ.get("MAX_HEALTH_REQUESTS", "10"))
+
+class HealthHandler(http.server.BaseHTTPRequestHandler):
+    request_count = 0
+    def do_GET(self):
+        HealthHandler.request_count += 1
+        if HEALTH_OK:
+            self.send_response(200)
+            self.send_header("Content-Length", "2")
+            self.end_headers()
+            self.wfile.write(b"OK")
+        else:
+            self.send_response(503)
+            self.end_headers()
+    def log_message(self, *args):
+        pass
+
+class ReusableTCPServer(socketserver.TCPServer):
+    allow_reuse_address = True
+    allow_reuse_port = True  # Critical: lets us rebind immediately after shutdown
+
+# Start health server in a thread
+httpd = ReusableTCPServer(("", PORT), HealthHandler)
+server_thread = threading.Thread(target=httpd.serve_forever, daemon=True)
+server_thread.start()
+
+time.sleep(0.3)
+
+# Run the pipeline
+try:
+    result = subprocess.run(
+        COMMAND,
+        shell=True,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.STDOUT,
+        timeout=120,
+    )
+    with open(LOG_FILE, "wb") as f:
+        f.write(result.stdout)
+    with open(RC_FILE, "w") as f:
+        f.write(str(result.returncode))
+except subprocess.TimeoutExpired as e:
+    with open(LOG_FILE, "wb") as f:
+        f.write(e.stdout or b"")
+    with open(RC_FILE, "w") as f:
+        f.write("124")
+except Exception as e:
+    with open(LOG_FILE, "w") as f:
+        f.write(f"orchestrator error: {e}")
+    with open(RC_FILE, "w") as f:
+        f.write("99")
+
+# Shutdown explicitly — this is what frees the port
+httpd.shutdown()
+httpd.server_close()
+PYEOF
+)"
+
+run_pipeline() {
+  # Args: dash_root patched_script trigger_json content log_file rc_file health_should_pass
+  local dash_root="$1"
+  local patched_script="$2"
+  local health_should_pass="${3:-yes}"
+  local log_file="$4"
+  local rc_file="$5"
+
+  # Write orchestrator + run it
+  local orch_py="$dash_root/.orchestrator.py"
+  echo "$ORCHESTRATOR_SRC" > "$orch_py"
+
+  PIPELINE_CMD="PATH='$dash_root/bin:$PATH' bash '$patched_script'" \
+    PIPELINE_LOG="$log_file" \
+    PIPELINE_RC_FILE="$rc_file" \
+    HEALTH_SHOULD_PASS="$health_should_pass" \
+    HEALTH_PORT="33001" \
+    python3 "$orch_py"
+
+  # Return the exit code
+  if [[ -f "$rc_file" ]]; then
+    cat "$rc_file"
+  else
+    echo "127"
+  fi
+}
+
+# ── Sandbox builder ──────────────────────────────────────────────────────────
+#
+# Lays out the sandbox as:
+#   $SANDBOX_ROOT/
+#     opt/dashcaddy/
+#       updates/
+#         staging/dashcaddy-api/      <- staging_dir
+#       dashcaddy-api/                <- api_source_dir (FROM)
+#         data/services.json
+#         src/app.js
+#         license-keygen.js
+#         server.js
+#       bin/
+#         docker                      <- fake docker
+#     patched-update.sh               <- path-rewritten update script
+#     .docker-build-ran               <- marker created by mocked docker build
+#     .docker-rm-ran                  <- marker created by mocked docker rm
+#     .docker-run-ran                 <- marker created by mocked docker run
+
+build_sandbox() {
+  local from_version="$1"
+  local new_version="$2"
+  local with_src="${3:-yes}"        # yes/no — controls whether staging has src/
+  local extra_setup="${4:-}"        # optional bash to run after setup
+
+  local sandbox=$(mktemp -d /tmp/dashcaddy-test-XXXXXX)
+  local dash_root="$sandbox/opt/dashcaddy"
+
+  mkdir -p "$dash_root"/{updates,bin,scripts}
+  mkdir -p "$dash_root/updates/staging/dashcaddy-api"
+  mkdir -p "$dash_root/dashcaddy-api/data"
+
+  # ── FROM deployment ──
+  echo '{"services":[]}' > "$dash_root/dashcaddy-api/data/services.json"
+  cat > "$dash_root/dashcaddy-api/server.js" << 'EOF'
+const { createApp } = require('./src/app');
+EOF
+  if [[ "$with_src" == "yes" ]]; then
+    mkdir -p "$dash_root/dashcaddy-api/src/managers"
+    cat > "$dash_root/dashcaddy-api/src/app.js" << 'EOF'
+module.exports = { createApp: () => ({ app: {}, log: console, config: {} }) };
+EOF
+    cat > "$dash_root/dashcaddy-api/src/managers/license-manager.js" << 'EOF'
+const keygen = require('../../license-keygen');
+module.exports = {};
+EOF
+  fi
+  cat > "$dash_root/dashcaddy-api/license-keygen.js" << 'EOF'
+module.exports = { verifyCode: () => true };
+EOF
+  echo "from-commit" > "$dash_root/dashcaddy-api/VERSION"
+
+  # ── STAGING (new version) ──
+  cp "$dash_root/dashcaddy-api/server.js" "$dash_root/updates/staging/dashcaddy-api/"
+  cp "$dash_root/dashcaddy-api/license-keygen.js" "$dash_root/updates/staging/dashcaddy-api/"
+  if [[ "$with_src" == "yes" ]]; then
+    cp -r "$dash_root/dashcaddy-api/src" "$dash_root/updates/staging/dashcaddy-api/"
+  fi
+  echo "new-commit-$new_version" > "$dash_root/updates/staging/dashcaddy-api/VERSION"
+
+  # ── Mocked docker ──
+  cat > "$dash_root/bin/docker" << 'EOF'
+#!/usr/bin/env bash
+echo "[mock-docker] $*" >> "${MOCK_DOCKER_LOG:-/tmp/mock-docker.log}"
+case "$1" in
+  build)
+    : >> "${IMAGE_MARKER_DIR:-/tmp}/.docker-build-ran"
+    exit 0
+    ;;
+  rm)
+    : >> "${IMAGE_MARKER_DIR:-/tmp}/.docker-rm-ran"
+    exit 0
+    ;;
+  run)
+    : >> "${IMAGE_MARKER_DIR:-/tmp}/.docker-run-ran"
+    exit 0
+    ;;
+  compose|version)
+    exit 0
+    ;;
+  *)
+    exit 0
+    ;;
+esac
+EOF
+  chmod +x "$dash_root/bin/docker"
+
+  # Fake start.sh — NOT created in the sandbox so deploy_mode picks "run"
+  # (which exercises docker rm + docker run paths in restart_container).
+  # Production DNS2 has start.sh and uses the startsh deploy path; the test
+  # deliberately diverges so we observe the full docker restart sequence.
+
+  # ── Post-deploy verifier (real script copied in) ─────────────────────────
+  # dashcaddy-update.sh hard-codes /opt/dashcaddy/scripts/dashcaddy-post-deploy-patches.sh
+  # (which the sed rewrite maps to $dash_root/scripts/...). For the verifier to
+  # actually be invoked, we copy the real script into the sandbox. The verifier
+  # is the one being tested here; we want to observe its behavior end-to-end.
+  local verifier_src="${SCRIPT_DIR}/dashcaddy-post-deploy-patches.sh"
+  if [[ ! -f "$verifier_src" ]]; then
+    verifier_src="$(cd "${SCRIPT_DIR}/../../scripts" 2>/dev/null && pwd)/dashcaddy-post-deploy-patches.sh"
+  fi
+  if [[ -f "$verifier_src" ]]; then
+    cp "$verifier_src" "$dash_root/scripts/dashcaddy-post-deploy-patches.sh"
+    chmod +x "$dash_root/scripts/dashcaddy-post-deploy-patches.sh"
+  fi
+
+  # ── Path-rewritten update script ──
+  local patched="$sandbox/patched-update.sh"
+  sed "s|/opt/dashcaddy|$dash_root|g" "$UPDATE_SCRIPT_SRC" > "$patched"
+  chmod +x "$patched"
+
+  if [[ -n "$extra_setup" ]]; then
+    ( cd "$sandbox" && eval "$extra_setup" )
+  fi
+
+  # Write a state file so the caller can recover the paths
+  cat > "$sandbox/.sandbox-paths" << EOF
+SANDBOX_ROOT=$sandbox
+DASH_ROOT=$dash_root
+PATCHED_SCRIPT=$patched
+API_SOURCE_DIR=$dash_root/dashcaddy-api
+STAGING_DIR=$dash_root/updates/staging/dashcaddy-api
+UPDATES_DIR=$dash_root/updates
+EOF
+  echo "$sandbox/.sandbox-paths"
+}
+
+write_trigger() {
+  local updates_dir="$1" action="$2" to_version="$3" from_version="$4" staging_dir="$5" api_source_dir="$6"
+  cat > "$updates_dir/trigger.json" << EOF
+{
+  "action": "${action}",
+  "version": "${to_version}",
+  "fromVersion": "${from_version}",
+  "channel": "stable",
+  "commit": "new-commit-${to_version}",
+  "stagingDir": "${staging_dir}",
+  "apiSourceDir": "${api_source_dir}"
+}
+EOF
+}
+
+load_paths() {
+  local paths_file="$1"
+  # shellcheck disable=SC1090
+  source "$paths_file"
+}
+
+cleanup_sandbox() {
+  local sandbox="$1"
+  rm -rf "$sandbox" /tmp/mock-docker.log 2>/dev/null
+}
+
+# ────────────────────────────────────────────────────────────────────────────
+# SCENARIO 1: Happy path — update succeeds end-to-end
+# ────────────────────────────────────────────────────────────────────────────
+echo "=== Scenario 1: happy path — update v1.14.8 -> v1.14.9 ==="
+PATHS=$(build_sandbox "1.14.8" "1.14.9" "yes")
+SANDBOX=$(dirname "$PATHS")
+load_paths "$PATHS"
+write_trigger "$UPDATES_DIR" "update" "1.14.9" "1.14.8" "$STAGING_DIR" "$API_SOURCE_DIR"
+
+RC=$(MOCK_DOCKER_LOG="$SANDBOX/.docker-calls.log" \
+     IMAGE_MARKER_DIR="$SANDBOX" \
+     run_pipeline "$DASH_ROOT" "$PATCHED_SCRIPT" "yes" \
+     "$SANDBOX/pipeline.log" "$SANDBOX/pipeline.rc")
+
+assert_eq "pipeline exit code" "0" "$RC"
+assert_file_exists "result.json exists" "$UPDATES_DIR/result.json"
+assert_json_field "result.success=true" "$UPDATES_DIR/result.json" "success" "True"
+assert_json_field "result.version=1.14.9" "$UPDATES_DIR/result.json" "version" "1.14.9"
+assert_file_exists "docker build ran" "$SANDBOX/.docker-build-ran"
+assert_file_exists "docker rm ran" "$SANDBOX/.docker-rm-ran"
+assert_file_exists "docker run ran" "$SANDBOX/.docker-run-ran"
+assert_dir_exists "code backup dir created" "$UPDATES_DIR/backups/1.14.8"
+assert_file_exists "code backup has server.js" "$UPDATES_DIR/backups/1.14.8/server.js"
+assert_dir_exists "data backup dir created" "$UPDATES_DIR/backups/1.14.8/data-backup"
+assert_dir_exists "update-state backup dir created" "$UPDATES_DIR/backups/1.14.8/update-state"
+assert_file_exists "update-state backup has trigger.json.processing" "$UPDATES_DIR/backups/1.14.8/update-state/trigger.json.processing"
+assert_eq "api source VERSION updated" "new-commit-1.14.9" "$(cat "$API_SOURCE_DIR/VERSION" 2>/dev/null)"
+assert_grep "docker was invoked with build" "$SANDBOX/.docker-calls.log" "build -t dashcaddy-dashcaddy-api:latest"
+assert_grep "docker was invoked with run" "$SANDBOX/.docker-calls.log" "run -d --restart unless-stopped"
+assert_grep "pipeline log shows successful update" "$SANDBOX/pipeline.log" "Update successful"
+cleanup_sandbox "$SANDBOX"
+
+# ────────────────────────────────────────────────────────────────────────────
+# SCENARIO 2: v1.14.4-style broken tarball (no src/) — verifier should fail
+# the build. Pipeline exits non-zero, result.json reports failure.
+#
+# AS-OF-CURRENT dashcaddy-update.sh: the verifier's failure is logged as a
+# WARNING and the build proceeds anyway (the script does not abort on verifier
+# failure). Mocked docker build always succeeds, so the pipeline ends with
+# success=true. The value of this scenario is asserting that the verifier IS
+# invoked, DOES detect the v1.14.4-class bug, and emits the expected error
+# message — i.e. the verifier itself works. Blocking the build on verifier
+# failure is a separate gap in dashcaddy-update.sh (TODO: tighten the call
+# site in main() so verifier failure aborts).
+# ────────────────────────────────────────────────────────────────────────────
+echo
+echo "=== Scenario 2: v1.14.4-class bug (no src/ in staging) — verifier detects it ==="
+PATHS=$(build_sandbox "1.14.4" "1.14.5" "no")
+SANDBOX=$(dirname "$PATHS")
+load_paths "$PATHS"
+write_trigger "$UPDATES_DIR" "update" "1.14.5" "1.14.4" "$STAGING_DIR" "$API_SOURCE_DIR"
+
+RC=$(MOCK_DOCKER_LOG="$SANDBOX/.docker-calls.log" \
+     IMAGE_MARKER_DIR="$SANDBOX" \
+     run_pipeline "$DASH_ROOT" "$PATCHED_SCRIPT" "yes" \
+     "$SANDBOX/pipeline.log" "$SANDBOX/pipeline.rc")
+
+# Current production behavior: verifier warns, build proceeds, pipeline succeeds.
+assert_eq "pipeline exit code" "0" "$RC"
+assert_file_exists "result.json exists" "$UPDATES_DIR/result.json"
+assert_json_field "result.success=true (build proceeded despite verifier warning)" "$UPDATES_DIR/result.json" "success" "True"
+# The KEY assertion: verifier actually caught the bug.
+assert_grep "verifier detected the missing src/ tree" "$SANDBOX/pipeline.log" "Build should be ABORTED"
+assert_grep "verifier failure was surfaced as a warning" "$SANDBOX/pipeline.log" "Post-deploy patches exited non-zero"
+# Build still ran (current code ignores verifier failure).
+assert_file_exists "docker build ran (current code proceeds past verifier failure)" "$SANDBOX/.docker-build-ran"
+cleanup_sandbox "$SANDBOX"
+
+# ────────────────────────────────────────────────────────────────────────────
+# SCENARIO 3: Rollback — action=rollback restores from backup
+# ────────────────────────────────────────────────────────────────────────────
+echo
+echo "=== Scenario 3: rollback — restore from backup directory ==="
+PATHS=$(build_sandbox "1.14.9" "1.14.10" "yes")
+SANDBOX=$(dirname "$PATHS")
+load_paths "$PATHS"
+
+# Pre-populate a backup dir (simulate that a prior update created it)
+mkdir -p "$UPDATES_DIR/backups/1.14.8/data-backup"
+echo '{"services":[]}' > "$UPDATES_DIR/backups/1.14.8/data-backup/services.json"
+cat > "$UPDATES_DIR/backups/1.14.8/server.js" << 'EOF'
+// ROLLBACK VERSION
+const { createApp } = require('./src/app');
+console.log('ROLLBACK-1.14.8');
+EOF
+echo "rollback-commit-1.14.8" > "$UPDATES_DIR/backups/1.14.8/VERSION"
+mkdir -p "$UPDATES_DIR/backups/1.14.8/src"
+cat > "$UPDATES_DIR/backups/1.14.8/src/app.js" << 'EOF'
+module.exports = { createApp: () => ({ rollback: '1.14.8' }) };
+EOF
+cp "$UPDATES_DIR/backups/1.14.8/license-keygen.js" "$UPDATES_DIR/backups/1.14.8/" 2>/dev/null
+# Rollback needs license-keygen.js in backup too
+cat > "$UPDATES_DIR/backups/1.14.8/license-keygen.js" << 'EOF'
+module.exports = { verifyCode: () => true };
+EOF
+
+# Write rollback trigger (no staging_dir needed for rollback)
+cat > "$UPDATES_DIR/trigger.json" << EOF
+{
+  "action": "rollback",
+  "version": "1.14.8",
+  "fromVersion": "1.14.9",
+  "channel": "stable",
+  "commit": "",
+  "stagingDir": "",
+  "apiSourceDir": "${API_SOURCE_DIR}"
+}
+EOF
+
+RC=$(MOCK_DOCKER_LOG="$SANDBOX/.docker-calls.log" \
+     IMAGE_MARKER_DIR="$SANDBOX" \
+     run_pipeline "$DASH_ROOT" "$PATCHED_SCRIPT" "yes" \
+     "$SANDBOX/pipeline.log" "$SANDBOX/pipeline.rc")
+
+assert_eq "rollback pipeline exit code" "0" "$RC"
+assert_file_exists "result.json exists" "$UPDATES_DIR/result.json"
+assert_json_field "result.success=true" "$UPDATES_DIR/result.json" "success" "True"
+assert_json_field "result.version=1.14.8" "$UPDATES_DIR/result.json" "version" "1.14.8"
+assert_file_exists "docker build called (rollback rebuilds)" "$SANDBOX/.docker-build-ran"
+assert_eq "api source VERSION restored" "rollback-commit-1.14.8" "$(cat "$API_SOURCE_DIR/VERSION" 2>/dev/null)"
+assert_grep "pipeline log shows rollback" "$SANDBOX/pipeline.log" "ROLLBACK"
+cleanup_sandbox "$SANDBOX"
+
+# ────────────────────────────────────────────────────────────────────────────
+# SCENARIO 4: No trigger file — pipeline exits cleanly without doing anything
+# ────────────────────────────────────────────────────────────────────────────
+echo
+echo "=== Scenario 4: no trigger.json — exits 0 with no-op ==="
+PATHS=$(build_sandbox "1.14.9" "1.14.10" "yes")
+SANDBOX=$(dirname "$PATHS")
+load_paths "$PATHS"
+# Deliberately don't write trigger.json
+
+RC=$(MOCK_DOCKER_LOG="$SANDBOX/.docker-calls.log" \
+     IMAGE_MARKER_DIR="$SANDBOX" \
+     run_pipeline "$DASH_ROOT" "$PATCHED_SCRIPT" "yes" \
+     "$SANDBOX/pipeline.log" "$SANDBOX/pipeline.rc")
+
+assert_eq "no-op exit code" "0" "$RC"
+assert_grep "logs 'nothing to do'" "$SANDBOX/pipeline.log" "No trigger file found"
+assert_not_exists "docker build did NOT run" "$SANDBOX/.docker-build-ran"
+cleanup_sandbox "$SANDBOX"
+
+# ────────────────────────────────────────────────────────────────────────────
+# SCENARIO 5: Channel rejection — prerelease trigger on default host exits 1
+# ────────────────────────────────────────────────────────────────────────────
+echo
+echo "=== Scenario 5: prerelease channel rejected (no ALLOW_PRERELEASE) ==="
+PATHS=$(build_sandbox "1.14.9" "1.15.0-beta" "yes")
+SANDBOX=$(dirname "$PATHS")
+load_paths "$PATHS"
+
+cat > "$UPDATES_DIR/trigger.json" << EOF
+{
+  "action": "update",
+  "version": "1.15.0-beta",
+  "fromVersion": "1.14.9",
+  "channel": "beta",
+  "commit": "new-commit-1.15.0-beta",
+  "stagingDir": "${STAGING_DIR}",
+  "apiSourceDir": "${API_SOURCE_DIR}"
+}
+EOF
+
+RC=$(MOCK_DOCKER_LOG="$SANDBOX/.docker-calls.log" \
+     IMAGE_MARKER_DIR="$SANDBOX" \
+     run_pipeline "$DASH_ROOT" "$PATCHED_SCRIPT" "yes" \
+     "$SANDBOX/pipeline.log" "$SANDBOX/pipeline.rc")
+
+assert_eq "channel rejection exit code" "1" "$RC"
+assert_file_exists "result.json exists" "$UPDATES_DIR/result.json"
+assert_json_field "result.success=false" "$UPDATES_DIR/result.json" "success" "False"
+assert_grep "result mentions channel rejection" "$UPDATES_DIR/result.json" "Channel 'beta' not allowed"
+assert_not_exists "docker build did NOT run" "$SANDBOX/.docker-build-ran"
+cleanup_sandbox "$SANDBOX"
+
+# ────────────────────────────────────────────────────────────────────────────
+# Summary
+# ────────────────────────────────────────────────────────────────────────────
+echo
+echo "═══════════════════════════════════════════════════════════"
+echo "  dashcaddy-update.sh integration test"
+echo "  PASS: $pass    FAIL: $fail"
+echo "═══════════════════════════════════════════════════════════"
+
+if (( fail > 0 )); then
+  exit 1
+fi
+echo "All scenarios passed."