dashcaddy.net/src/app/docs/troubleshooting/page.tsx

import Navbar from '@/components/Navbar';
import Footer from '@/components/Footer';
import DocsLayout from '@/components/docs/DocsLayout';

export default function DocsTroubleshootingPage() {
  return (
    <div className="flex min-h-screen flex-col bg-surface-950 text-surface-50">
      <Navbar />
      <DocsLayout
        title="Troubleshooting"
        intro="Because DashCaddy sits across runtime, DNS, reverse proxy, certificates, and dashboard state, the fastest way to debug it is layer by layer instead of guessing. This guide walks each layer with the common failures and fixes."
      >
        <p>
          DashCaddy orchestrates several independent layers — a container runtime, a DNS server, a reverse proxy, a
          certificate authority, and its own API and dashboard. When a service is unreachable, the failure is almost
          always in exactly one of these layers while the others are healthy. This guide gives you a structured,
          layer-by-layer diagnostic procedure with the exact commands to run and the fixes to apply.
        </p>
        <p>
          The single most important habit: <strong>localize before you fix</strong>. Resist the urge to restart
          everything. Use the health endpoints to narrow down which layer is broken, then dig into that layer with the
          commands below. You will solve problems far faster than by reloading the whole stack.
        </p>

        <h2>Health check endpoints — start here</h2>
        <p>
          Every investigation begins with the built-in probes. They tell you whether the DashCaddy process itself is
          healthy and whether its dependencies are wired up, in two seconds:
        </p>
        <ul>
          <li><code>/healthz</code> — <strong>liveness</strong>. Returns 200 if the DashCaddy process is up.</li>
          <li><code>/readyz</code> — <strong>readiness</strong>. Returns 200 only when DashCaddy can serve traffic, including connectivity to Docker, Caddy, and DNS where configured.</li>
        </ul>
        <pre className="mt-4 overflow-x-auto rounded-lg border border-surface-700/50 bg-surface-950/80 p-4 text-sm"><code>{`# Print just the HTTP status codes
curl -s -o /dev/null -w "healthz: %{http_code}\\n" https://dashcaddy-host/healthz
curl -s -o /dev/null -w "readyz:  %{http_code}\\n" https://dashcaddy-host/readyz`}</code></pre>
        <p>
          Interpret the result:
        </p>
        <ul>
          <li><strong>Both 200</strong> — DashCaddy and its dependencies are up. The problem is downstream of the platform (the service itself, DNS, cert trust, or the client).</li>
          <li><strong><code>/healthz</code> 200, <code>/readyz</code> fails</strong> — the process is up but a dependency is unreachable: Docker socket, Caddy Admin API, or Technitium DNS. Read the <code>/readyz</code> body for which dependency failed.</li>
          <li><strong><code>/healthz</code> fails</strong> — the DashCaddy process itself is down. Check <code>docker ps</code> and <code>docker logs dashcaddy</code>.</li>
        </ul>

        <h2>The debug order — work bottom-up</h2>
        <p>
          When a specific service is unreachable, walk the stack from the container outward to the client. Each step
          depends on the one before it, so the first failing step is your root cause:
        </p>
        <ol>
          <li><strong>Backend container</strong> — is it running and healthy? (<code>docker ps</code>, <code>docker logs</code>)</li>
          <li><strong>Backend port</strong> — is the service listening and reachable on the host? (<code>curl localhost:port</code>)</li>
          <li><strong>Reverse proxy route</strong> — did Caddy apply the route correctly? (Caddyfile-as-Code view, Admin API)</li>
          <li><strong>DNS</strong> — does the hostname resolve to the right host? (<code>dig</code>, <code>nslookup</code>)</li>
          <li><strong>Certificate trust</strong> — does the client trust the CA? (<code>openssl s_client</code>, browser cert store)</li>
          <li><strong>Dashboard / API state</strong> — does DashCaddy reflect reality? (compare UI vs. actual container state)</li>
        </ol>
        <p>
          The sections below cover each layer in detail with the commands and fixes for the most common failures.
        </p>

        <h2>DNS issues</h2>
        <p>
          DNS problems show up as &ldquo;hostname does not resolve&rdquo; or &ldquo;resolves to the wrong address.&rdquo;
          Because DashCaddy uses Technitium for internal zones, the most common cause is a client using a public resolver
          that does not know about your private zones.
        </p>
        <ul>
          <li><strong>Check</strong>: is the client using Technitium as its resolver? Public resolvers (8.8.8.8, 1.1.1.1) will not resolve internal <code>.lab</code> zones. Point the client&apos;s DNS at Technitium, or use Tailscale MagicDNS / split-DNS for remote clients.</li>
          <li><strong>Check</strong>: is the record present in the correct zone? DNS automation fails silently when the zone name is wrong — a record in <code>lab</code> vs <code>lab.</code> is a different zone.</li>
          <li><strong>Check</strong>: is the Technitium API token valid and scoped for writes? An expired or read-only token will let records appear to &ldquo;work&rdquo; in the UI but fail to actually create.</li>
          <li><strong>Fix</strong>: re-run the DNS step from the service&apos;s action menu, or recreate the record manually in Technitium and let DashCaddy reconcile.</li>
        </ul>
        <pre className="mt-4 overflow-x-auto rounded-lg border border-surface-700/50 bg-surface-950/80 p-4 text-sm"><code>{`# Query Technitium directly (bypass the client's resolver)
dig @technitium-host media.lab +short
nslookup media.lab technitium-host

# Check what the client's resolver returns (may differ)
dig media.lab +short

# Trace the full resolution path
dig media.lab +trace`}</code></pre>
        <p>
          If <code>dig @technitium-host</code> returns the right IP but <code>dig media.lab</code> does not, the client
          is not using Technitium. If Technitium itself returns nothing, the record was never created — check the token
          and zone, then recreate it.
        </p>

        <h2>TLS / certificate problems</h2>
        <p>
          Certificate problems show up as browser warnings (<code>NET::ERR_CERT_AUTHORITY_INVALID</code>) or TLS
          handshake failures. There are two distinct causes, and the fix is different for each.
        </p>
        <h3>Cause 1: client does not trust the internal CA</h3>
        <p>
          For internal (<code>.lab</code>) services, Caddy uses its internal CA and DashCA distributes the root
          certificate. The root cert must be installed as a trusted CA on <strong>each client device</strong> — not just
          the server. Download it from the <strong>DashCA</strong> page and follow the per-platform instructions (macOS
          Keychain, Windows certmgr, Linux <code>update-ca-certificates</code>, mobile profiles).
        </p>
        <h3>Cause 2: certificate issuance failed</h3>
        <p>
          If Caddy could not reach its CA at deploy time (internal CA down, or ACME unreachable for public domains), no
          certificate is issued and the TLS handshake fails outright. Confirm the Caddy Admin API is reachable, then
          redeploy or re-trigger TLS for the service.
        </p>
        <pre className="mt-4 overflow-x-auto rounded-lg border border-surface-700/50 bg-surface-950/80 p-4 text-sm"><code>{`# Inspect the certificate a server presents
echo | openssl s_client -connect media.lab:443 -servername media.lab 2>/dev/null \\
  | openssl x509 -noout -issuer -subject -dates

# Verify the chain against a specific CA bundle
openssl s_client -connect media.lab:443 -CAfile /path/to/dashca-root.crt </dev/null`}</code></pre>
        <p>
          If <code>openssl s_client</code> shows the issuer is Caddy&apos;s internal CA and your browser still warns,
          the root cert is not installed on that client. If <code>s_client</code> shows no certificate at all, issuance
          failed — check Caddy.
        </p>
        <blockquote className="border-l-4 border-brand-500/50 bg-brand-500/5 p-4 rounded-r-lg">
          <p className="text-surface-300">
            <strong className="text-brand-400">Tip:</strong> After installing the root CA, restart the browser. Chrome
            and Firefox maintain separate trust stores on some platforms — Firefox may need the import done from its own
            settings rather than the OS store.
          </p>
        </blockquote>

        <h2>Reverse proxy debugging (Caddy)</h2>
        <p>
          If the service is up, the port is reachable, and DNS resolves, but the URL returns 502, 504, or does not route,
          the problem is in the Caddy layer. DashCaddy drives Caddy through its Admin API, so two things can go wrong:
          the Admin API is unreachable, or the generated config is wrong.
        </p>
        <ul>
          <li><strong>Check</strong>: is the Caddy Admin API reachable from the DashCaddy API server? (<code>curl localhost:2019/config/</code> on the host)</li>
          <li><strong>Check</strong>: does the Caddy route point at the correct upstream host:port? Use the <strong>Caddyfile-as-Code view</strong> to inspect the generated config.</li>
          <li><strong>Check</strong>: Caddy logs — <code>docker logs caddy</code> or your Caddy service logs — for upstream connection errors and reload failures.</li>
          <li><strong>Fix</strong>: re-apply the route from the service&apos;s action menu; DashCaddy reconciles the Caddy configuration atomically. If the config is invalid, DashCaddy rejects it before Caddy ever sees it.</li>
        </ul>
        <pre className="mt-4 overflow-x-auto rounded-lg border border-surface-700/50 bg-surface-950/80 p-4 text-sm"><code>{`# Query the live Caddy config via the Admin API
curl -s localhost:2019/config/ | jq

# Find the route for a specific hostname
curl -s localhost:2019/config/ | jq '.. | .match? // empty | select(.host[]? | contains("media.lab"))'

# Tail Caddy logs for upstream errors
docker logs caddy --tail 50 -f`}</code></pre>

        <h2>Container health</h2>
        <p>
          If a service shows <em>Unhealthy</em> or <em>Down</em> on the dashboard, the problem is the container itself.
          Go straight to Docker.
        </p>
        <ul>
          <li><strong>Check</strong>: <code>docker ps -a</code> — is the container running, restarting, or exited?</li>
          <li><strong>Check</strong>: <code>docker logs &lt;container&gt;</code> — look for crash loops, missing files, bad config, or auth failures.</li>
          <li><strong>Check</strong>: the container&apos;s healthcheck (if defined). DashCaddy surfaces container healthchecks in the UI; a failing healthcheck means the app is up but not ready (e.g. still migrating a database).</li>
          <li><strong>Check</strong>: are volumes mounted and environment variables correct? Bad secrets (wrong DB password, missing API key) are the most common cause of immediate exits.</li>
        </ul>
        <pre className="mt-4 overflow-x-auto rounded-lg border border-surface-700/50 bg-surface-950/80 p-4 text-sm"><code>{`# List all containers including stopped ones
docker ps -a --filter "name=media"

# Tail recent logs
docker logs media --tail 100

# Inspect the healthcheck status and exit codes
docker inspect media --format '{{.State.Health.Status}} {{.State.ExitCode}}'

# Check resource usage if the container is OOM-killing
docker stats --no-stream media`}</code></pre>

        <h2>Performance issues</h2>
        <p>
          If DashCaddy itself is slow or unresponsive, the cause is usually resource pressure on the host or an
          overloaded dependency.
        </p>
        <ul>
          <li><strong>Host resources</strong>: check CPU, memory, and disk with <code>htop</code>, <code>free -h</code>, and <code>df -h</code>. DashCaddy is lightweight, but a host running dozens of containers can starve it.</li>
          <li><strong>Disk I/O</strong>: slow disks make Docker operations (deploy, inspect, logs) sluggish. Check <code>iostat -x 1</code> for high <code>%util</code>.</li>
          <li><strong>Docker daemon load</strong>: a wedged Docker daemon slows every operation. <code>docker info</code> and <code>systemctl status docker</code> reveal daemon-level issues.</li>
          <li><strong>DNS latency</strong>: if Technitium is overloaded or remote, every DNS operation in DashCaddy slows down. Check Technitium&apos;s own health and resource usage.</li>
          <li><strong>Polling overhead</strong>: if you have many scripts polling the REST API, switch them to the WebSocket channel or Prometheus endpoint to reduce load.</li>
        </ul>
        <pre className="mt-4 overflow-x-auto rounded-lg border border-surface-700/50 bg-surface-950/80 p-4 text-sm"><code>{`# Quick host health snapshot
free -h && df -h | grep -E "^/dev|Filesystem"
docker stats --no-stream
uptime`}</code></pre>

        <h2>Common error messages</h2>
        <p>
          The table maps the most frequently seen errors to their likely cause and fix. For the full catalog of
          structured error codes across all modules, see the <a href="/docs/api">API guide</a>.
        </p>
        <table>
          <thead>
            <tr>
              <th>Error</th>
              <th>Likely cause</th>
              <th>Fix</th>
            </tr>
          </thead>
          <tbody>
            <tr>
              <td><code>NET::ERR_CERT_AUTHORITY_INVALID</code></td>
              <td>Client does not trust the DashCA root certificate</td>
              <td>Install the root CA from the DashCA page on the client device</td>
            </tr>
            <tr>
              <td><code>502 Bad Gateway</code></td>
              <td>Caddy route points at a wrong/unreachable upstream port</td>
              <td>Check the Caddyfile-as-Code view; fix the upstream host:port; re-apply</td>
            </tr>
            <tr>
              <td><code>504 Gateway Timeout</code></td>
              <td>Upstream is up but too slow to respond within the proxy timeout</td>
              <td>Inspect container logs; increase Caddy proxy timeout if the app legitimately needs more time</td>
            </tr>
            <tr>
              <td>Hostname does not resolve</td>
              <td>Client is not using Technitium as its resolver, or the record was not created</td>
              <td>Point client DNS at Technitium; verify the record exists; re-run DNS step</td>
            </tr>
            <tr>
              <td><code>DNS_TOKEN_INVALID</code></td>
              <td>Technitium API token expired or revoked</td>
              <td>Regenerate the token in Technitium; update it under Settings → DNS</td>
            </tr>
            <tr>
              <td><code>PROXY_CADDY_UNREACHABLE</code></td>
              <td>Caddy Admin API (localhost:2019) is down or firewalled</td>
              <td>Restart Caddy; confirm the Admin API port is open to DashCaddy</td>
            </tr>
            <tr>
              <td><code>DEPLOY_PORT_CONFLICT</code></td>
              <td>Another container already holds the requested host port</td>
              <td>Stop the conflicting container or choose a different port</td>
            </tr>
            <tr>
              <td><code>LICENSE_EXPIRED</code></td>
              <td>Premium license expired past the 7-day grace period</td>
              <td>Renew from Settings → Licensing; free-tier features remain available</td>
            </tr>
            <tr>
              <td><code>LICENSE_MACHINE_LIMIT</code></td>
              <td>License already bound to another machine</td>
              <td>Deactivate on the old host before activating on the new one</td>
            </tr>
            <tr>
              <td><code>AUTH_PERMISSION_DENIED</code></td>
              <td>User/API key lacks the RBAC role for the action</td>
              <td>Assign the needed role in Settings → Users</td>
            </tr>
            <tr>
              <td>WebSocket updates stall</td>
              <td>A reverse proxy or firewall is dropping the WS upgrade</td>
              <td>Allow WebSocket upgrades on the DashCaddy route in Caddy/firewall</td>
            </tr>
            <tr>
              <td><code>429 Too Many Requests</code></td>
              <td>API client exceeded the per-token rate limit</td>
              <td>Back off and retry after <code>Retry-After</code>; switch polling to WS/Prometheus</td>
            </tr>
          </tbody>
        </table>

        <h2>Debug mode</h2>
        <p>
          When the standard checks do not reveal the problem, enable debug logging for verbose output from every layer.
          Set the <code>LOG_LEVEL</code> environment variable to <code>debug</code> and restart DashCaddy:
        </p>
        <pre className="mt-4 overflow-x-auto rounded-lg border border-surface-700/50 bg-surface-950/80 p-4 text-sm"><code>{`# Enable debug logging (docker run)
docker run -d \\
  -e LOG_LEVEL=debug \\
  -v /var/run/docker.sock:/var/run/docker.sock \\
  -p 3000:3000 \\
  ghcr.io/dashcaddy/dashcaddy:latest

# Or in docker-compose.yml
services:
  dashcaddy:
    environment:
      - LOG_LEVEL=debug

# Then tail the logs
docker logs dashcaddy -f --tail 200`}</code></pre>
        <p>
          Debug mode emits detailed logs for Docker operations, Caddy Admin API calls, DNS requests, certificate
          workflows, and the AI/MCP layer. Reproduce the problem while debug logging is on, then grep the logs for the
          relevant module. Disable debug mode when done — it is verbose and not recommended for long-term production use.
        </p>

        <h2>Support resources</h2>
        <p>
          If you have worked through the layers above and are still stuck, the following resources can help:
        </p>
        <ul>
          <li><strong>Integrations guide</strong> — <a href="/docs/integrations">Infrastructure Integrations</a> explains what each layer expects and how to configure it.</li>
          <li><strong>API error reference</strong> — the <a href="/docs/api">API and Automation</a> guide lists all 80 structured error codes across 12 modules.</li>
          <li><strong>Installation</strong> — <a href="/docs/installation">Installation Guide</a> covers first-run setup and the Smart Defaults Wizard.</li>
          <li><strong>Premium / licensing</strong> — <a href="/docs/premium">Premium Features</a> covers license validation, grace periods, and machine binding.</li>
          <li><strong>Priority support</strong> — Premium license holders get priority support. Open a ticket from Settings → Support in the dashboard.</li>
        </ul>

        <h2>Mindset</h2>
        <p>
          Most DashCaddy problems are really one dependency layer failing while the others are healthy. Use the health
          endpoints to localize, then walk the debug order from the container outward. Fixing the right layer first is
          always faster than reloading the whole stack. When in doubt, enable debug mode, reproduce the issue, and read
          the logs for the failing module — the answer is almost always there.
        </p>
      </DocsLayout>
      <Footer />
    </div>
  );
}