dashcaddy/SIMPLIFY.md

# DashCaddy — Code Simplification & Maintainability

## Goal

Keep all existing functionality while making the codebase:
- **Easier to read** (fewer files, clearer structure)
- **Easier to modify** (focused modules, fewer edge cases)
- **Easier to debug** (deterministic flows, focused logging)
- **Easier to test** (focused unit tests, reliable mocks)

---

## 1. Monolithic → Modular Consolidation

### What was fragmented
- **Configuration** spread across `services.json`, `config.json`, `dns-credentials.json`, `credentials.json.enc`
- **API surface** split across multiple `routes/*` modules without a clear hierarchy
- **Build** custom `esbuild` + `package.json` shenanigans
- **Security** scattered across `middleware.js`, `input-validator.js`, `csrf-protection.js`

### Consolidation strategy

#### A. Single Config (`config.yaml`)

```yaml
# Replace all JSON configs with this single source of truth
# Loaded once at startup, with env overrides

# Services (previously services.json)
services:
  - id: plex
    type: "media-server"
    port: 32400
    host: "192.168.1.50"
    auth:
      enabled: true
      username: "admin"
      password_encrypted: "..."

# Core config (previously config.json)
core:
  domain: "example.com"
  timezone: "America/Los_Angeles"
  log_path: "/opt/dashcaddy/data/logs"
  backup_retention: 30

# DNS config (previously dns-credentials.json)
dns:
  provider: "coredns"
  # provider-specific config
  servers: ["10.0.0.1", "10.0.1.1"]

# Encryption key (previously credentials.json.enc)
encryption_key_encrypted: "..."
```

#### B. Unified API Router

**Previous pattern:**
- `routes/health.js`, `routes/auth.js`, `routes/dns.js`, `routes/services.js`
- Each exports its own middleware chain, scattered imports

**New pattern:**
- **Single `routes/index.js`** — entry point that declares routes once, with schema validation
- **Per-feature submodules** under `routes/core/`, `routes/admin/`, `routes/integrations/` (but importable directly)
- **Centralized rate limiting, validation, auth** middleware stack

```javascript
// routes/index.js (single file, but organized with requires)
const express = require('express');
const router = express.Router();

// Core system routes
router.use('/health', require('./core/health'));
router.use('/api/v1', require('./core/api'));

// Admin routes
router.use('/api/v1/admin', require('./admin/users'));
router.use('/api/v1/admin/services', require('./admin/services'));

// Service integrations
router.use('/api/v1/integrations/plex', require('./integrations/plex'));

module.exports = router;
```

#### C. Consolidated Security Middleware

**Previous:**
- `middleware.js` (generic)
- `input-validator.js` (Joi)
- `csrf-protection.js` (express-csrf)
- `auth-manager.js` (session + TOTP)

**Unified:**
- **`security.js`** — exports `authenticate`, `validate`, `csrfProtect`, `rateLimit` etc.
- **Single initialization** in `server.js`
- **Clear order**: CORS → Helmet → CSRF → Auth → Rate Limit → Validation

#### D. Simplified Build

**Previous:**
- `status/build.js` with complex esbuild config
- Separate build for `frontend`, `backend`
- Hard to run locally

**Unified:**
- **`scripts/build.js`** — runnable from repo root
- **Vite frontend** (optional) OR **esbuild** (default)
- **Docker-first**: Build inside container, serve via Caddy

---

## 2. Layered Architecture (Presentation → Core → Infrastructure)

```
┌───────────────────────────────────────────────────────────────┐
│                       Presentation                            │
│                     (status/ folder)                          │
│  ├─ index.html      ← Static HTML template                    │
│  ├─ dist/           ← Bundled JavaScript                      │
│  ├─ assets/         ← Images, CSS, static assets              │
│  └─ sw.js          ← Service worker                           │
├───────────────────────────────────────────────────────────────┤
│                       Business Logic                           │
│                    (dashcaddy-api/src/)                       │
│  ├─ app/               ← Express app factory                  │
│  ├─ services/          ← Service CRUD, discovery, auth       │
│  ├─ security/           ← Unified auth + validation          │
│  ├─ dns/                ← DNS provider abstraction           │
│  ├─ backups/            ← Backup/restore operations           │
│  └─ license/            ← License management                 │
├───────────────────────────────────────────────────────────────┤
│                      Infrastructure                           │
│                 (node_modules, external)                     │
│  ├─ dockerode            ← Docker operations                  │
│  ├─ ssh2-sftp-client     ← File transfers                     │
│  ├─ webdav               ← WebDAV integration                │
│  └─ tls-certificate     ← Let's Encrypt automation         │
└───────────────────────────────────────────────────────────────┘
```

### Benefits

| Aspect | Before | After |
|--------|--------|-------|
| **Finding a route** | `grep -r "app.get" routes/` | `grep -r "router.use" routes/index.js` |
| **Adding a new service type** | Add `routes/service-type.js`, wire in `server.js` | Add to `src/services/` → auto-discovery via `services/discovery.js` |
| **Security patch** | Edit multiple files | Edit single `security.js` |
| **Running tests** | `npm run test:unit && npm run test:routes && npm run test:security` | `npm test` (single entry point) |

---

## 3. Deterministic File Layout

### Problem
Paths varied across platforms, making CI/CD and local dev confusing.

### Solution
**Zero-config, platform-agnostic layout:**

```
repo/
├─ README.md              ← Always present (quick install)
├─ INSTALL.md             ← Detailed setup (platform-specific)
├─ .env.example           ← Env variable documentation
├─ docker-compose.yml     ← Single-compose, multi-profile
├─ dashcaddy-api/         ← API source (Node.js)
├─ status/                ← Dashboard frontend source
├─ dashcaddy-installer/   ← Cross-platform installers
├─ scripts/               ← Helper scripts (daily-update, adversarial-find-errors, etc.)
├─ skills/                ← Hermes skills (orchestration)
└─ docs/                  ← Architecture, API, CONTRIBUTING
```

**Rules:**
- **No nested repo root changes** (no `src/` inside `dashcaddy-api/`, no `lib/` inside `status/`)
- **`data/` lives outside the repo** (`/opt/dashcaddy/data` on Linux, `E:/dockerdata/dashcaddy` on Windows)
- **Static assets** (`status/dist/`, `status/assets/`) are built and deployed, not source
- **`platform-paths.js`** resolves everything at runtime — no hardcoded platform checks in application code

---

## 4. Simplified Testing Strategy

### Test Pyramid

1. **Unit Tests** (`__tests__/core/*.test.js`)
   - Test individual functions (no external calls)
   - Mock `fs`, `dockerode`, external HTTP

2. **Integration Tests** (`__tests__/routes/`, `__tests__/admin/`)
   - Test route chains end-to-end with mocked external deps
   - Fast, deterministic, no real Docker/containers

3. **Adversarial Tests** (`adversarial-find-errors.py`)
   - Live contract checks against running instance
   - Same test as CI/CD, runs locally via `npm run adversarial`

4. **E2E/Contract Tests** (`__tests__/integration/`, `docker-compose -f docker-compose.test.yml`)
   - Real Docker container stack (for UI flows, real DNS, etc.)

### Simplified Test Runner

**Previous:**
```bash
# Complex
npm run test:ci
# or
npm run test:unit && npm run test:routes && npm run test:security
```

**Unified:**
```javascript
// package.json scripts
"scripts": {
  "test": "jest",
  "test:ci": "jest --ci --coverage --maxWorkers=2",
  "test:integration": "jest --testPathPattern=__tests__/integration",
  "adversarial": "python3 scripts/adversarial-find-errors.py"
}
```

**Single command for CI:** `npm run test:ci`

---

## 5. Simplified Logging & Monitoring

### Problem
Multiple log files, unclear severity levels, no structured output.

### Solution
**Unified logging system:**

1. **`src/logging/`** — single module
   - Levels: `INFO`, `WARN`, `ERROR`, `DEBUG`
   - Structured output: `{ timestamp, level, area, message, context }`
   - Console + file (JSON lines) + optional syslog

2. **Consistent area names:**
   - `auth`, `dns`, `services`, `security`, `backups`, `license`, `integrations/plex`

3. **Single audit-log:**
   - All state changes go to `/opt/dashcaddy/data/audit-log.jsonl`
   - One-liner entry: `{ "ts": "2026-08-21T02:40:16Z", "area": "services", "event": "create", "payload": {"id": "plex"} }`

### Example logging call

```javascript
// In src/services/index.js
const logger = require('../logging');

logger.log('INFO', 'services', 'Service created', { id: serviceId, type: 'plex' });
logger.error('DNS', 'Failed to provision DNS record', { record: 'plex.example.com', error: err.message });
```

---

## 6. Simplified Deployment Pipeline

### Before: Complex Docker orchestration
```bash
# Build
./dashcaddy-installer/install.sh
# Deploy
ssh root@dns2 /opt/dashcaddy/start.sh
# Update
git checkout new-feature && ./dashcaddy-installer/install.sh
```

### Unified: Docker Compose + Profiles

```yaml
# docker-compose.yml (single file)
services:
  dashcaddy-api:
    build: .
    profiles: [prod, windows]
    volumes:
      - ./dashcaddy-api:/app/src
      - ./status:/app/dashboard
      - ./data:/opt/dashcaddy/data
    environment:
      - NODE_ENV=production
    depends_on:
      - caddy
  
  caddy:
    image: caddy:2.10-alpine
    profiles: [prod]
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./caddy/Caddyfile:/etc/caddy/Caddyfile
      - ./caddy/data:/data
```

**Profiles:**
- `prod` — Production stack (Caddy + API + DNS)
- `dev` — API only (local development)
- `windows` — Windows container variant

**Commands:**
```bash
# Start production
docker compose --profile prod up -d

# Local dev (no Caddy, no DNS)
docker compose --profile dev up -d

# Windows native (if using Windows containers)
docker compose --profile windows up -d
```

---

## 7. Simplified Installer Scripts

### Unified `install.sh` / `install.ps1`

**Single command installs:**
- Docker (if not present)
- Caddy (via package manager)
- DashCaddy repo (auto-pull latest)
- Environment variables (`.env`)
- Optional Tailscale setup
- Start services

**No manual steps needed:**
- No `apt install`, `systemctl enable`, etc.
- All platform detection inside script
- Rollback on failure

### Example usage

```bash
# Linux/macOS/WSL
curl -fsSL https://dashcaddy.net/install.sh | bash

# Windows
irm https://dashcaddy.net/install.ps1 | iex
```

---

## 8. Simplified Documentation

### Docs structure

```
/docs/
├─ ARCHITECTURE.md               # System overview, layering, platform paths
├─ CONTRIBUTING.md               # Code style, testing, PR process
├─ API-REFERENCE.md              # All API endpoints, parameters, responses
├─ DNS_PROVIDERS.md              # How to add new DNS provider
├─ SECURITY.md                   # Threat model, best practices
└─ TROUBLESHOOTING.md            # Common issues + solutions
```

**Single source of truth** — CLI docs, README, and web docs generated from these.

---

## 9. Simplified CI/CD Pipeline

### One CI job for all platforms

```yaml
# .github/workflows/ci.yml
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '20' }
      - run: npm ci
      - run: npm run lint
      - run: npm run test:ci

  build:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: docker/setup-buildx-action@v3
      - uses: docker/build-push-action@v5
        with:
          platforms: linux/amd64,linux/arm64,windows/amd64
          push: ${{ github.event_name == 'push' }}
          tags: dashcaddy/dashcaddy-api:${{ github.sha }}

  windows:
    needs: test
    runs-on: windows-latest
    steps:
      - uses: docker/setup-buildx-action@v3
      - uses: docker/build-push-action@v5
        with:
          platforms: windows/amd64
          push: ${{ github.event_name == 'push' }}
          tags: dashcaddy/dashcaddy-api:${{ github.sha }}-windows
```

**Benefits:**
- Deterministic builds across platforms
- Same test suite runs everywhere
- Single PR triggers all platform builds

---

## 10. Simplified Upgrade Path

### Versioning policy
- **Semantic Versioning** (MAJOR.MINOR.PATCH)
- **One minor version** = new feature, no breaking changes
- **Patch** = bug fixes only
- **Major** = breaking changes (rare, documented 6 months ahead)

### Upgrade commands

```bash
# Upgrade to latest stable
curl -fsSL https://dashcaddy.net/install.sh | bash

# Or via existing Docker compose
docker compose pull && docker compose --profile prod up -d
```

### Migration guides
- Each major version includes a `/docs/MIGRATION-vX.Y.md`
- Auto-generated release notes

---

## 11. Simplified Monitoring & Health Checks

### Health check endpoints

```bash
# System health
curl http://localhost:3001/api/v1/health
# Dashboard health
curl http://localhost:3001/api/v1/health/dashboard
# DNS health
curl http://localhost:3001/api/v1/health/dns
```

### Unified status reporting
- Every 5 minutes: `cron/sweep.sh` collects logs, generates `/tmp/dashcaddy-errors/adversarial-report.md`
- Daily: `cron/dc-daily-update.py` posts summary to Telegram topic
- Alerts: Slack/Email webhook if errors > threshold

### Structured metrics
- All metrics go to `data/metrics.jsonl` (one JSON object per line)
- Prometheus exporter (optional) for integration with monitoring stack

---

## 12. Simplified Training & Onboarding

### README-first approach
- `README.md` includes **one-line install** + **basic usage**
- Clickable links to `INSTALL.md` (platform-specific) + `ARCHITECTURE.md`

### Code comments
- **Clear purpose**: `/** * Describe what this function does * */`
- **Usage examples**: `// Example: router.get('/', homeHandler)`
- **Side effects**: Document async operations, external calls

### Pull request template
- **Required checklist:**
  - [ ] Tests pass (`npm run test:ci`)
  - [ ] Lint clean (`npm run lint`)
  - [ ] No new files outside allowed directories
  - [ ] Updated `CHANGELOG.md` with concise description
  - [ ] Added `docs/` if new feature/feature change

---

## Summary of Simplification

| Area | Before | After |
|------|--------|-------|
| **Config** | 3+ JSON files scattered | 1 `config.yaml` with env overrides |
| **API routes** | 20+ files, scattered imports | 1 `routes/index.js`, organized submodules |
| **Security** | 4+ middleware files | 1 `security.js` with clear order |
| **Build** | Custom esbuild + manual steps | Single `scripts/build.js` |
| **Testing** | 3+ npm scripts, different scopes | 1 `npm test` + optional `adversarial` |
| **Logging** | Mixed console.log, error.log | Structured JSON lines in `audit-log.jsonl` |
| **Deployment** | Manual docker + custom scripts | Docker Compose + Profiles |
| **Installer** | Separate scripts per platform | Unified `install.sh`/`install.ps1` |
| **Docs** | Wikipedia-sized README | Split into focused markdown files |
| **CI/CD** | Platform-specific pipelines | Single matrix build with multi-arch |

**Result:** Much easier to understand, modify, and extend while preserving 100% of existing functionality.

---

## Next Steps

1. **Run the simplified tests**: `npm run test:ci`
2. **Review the new config**: Edit `config.yaml` and run `./scripts/validate-config.js`
3. **Test the installer**: `curl -fsSL https://dashcaddy.net/install.sh | bash` (in VM)
4. **Check the new logs**: `cat /opt/dashcaddy/data/audit-log.jsonl`
5. **Upgrade existing deployment**: `docker compose --profile prod up -d`

All changes are **backward compatible** — no breaking changes, no data loss, no API changes.

---

*DashCaddy v2.0 — Simpler by design, stronger by execution.*