Compare commits
14
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a21e06bf5b | ||
|
|
95d4b3f4bc | ||
|
|
acc2e1939e | ||
|
|
f3934fd257 | ||
|
|
27beae22a8 | ||
|
|
30acd6a237 | ||
|
|
dad6af4003 | ||
|
|
84374aab38 | ||
|
|
3be4cda695 | ||
|
|
6891b51a1e | ||
|
|
f6feb0184d | ||
|
|
92482980dd | ||
|
|
a1d7208686 | ||
|
|
cdf9e8d3ef |
@@ -0,0 +1,36 @@
|
||||
version: 2
|
||||
updates:
|
||||
- package-ecosystem: "npm"
|
||||
directory: "/dashcaddy-api"
|
||||
schedule:
|
||||
interval: "weekly"
|
||||
open-pull-requests-limit: 5
|
||||
labels:
|
||||
- "dependencies"
|
||||
- "automated"
|
||||
groups:
|
||||
dev-dependencies:
|
||||
patterns:
|
||||
- "jest"
|
||||
- "eslint"
|
||||
- "supertest"
|
||||
update-types:
|
||||
- "minor"
|
||||
- "patch"
|
||||
production-dependencies:
|
||||
patterns:
|
||||
- "*"
|
||||
exclude-patterns:
|
||||
- "jest"
|
||||
- "eslint"
|
||||
- "supertest"
|
||||
update-types:
|
||||
- "patch"
|
||||
|
||||
- package-ecosystem: "github-actions"
|
||||
directory: "/"
|
||||
schedule:
|
||||
interval: "weekly"
|
||||
labels:
|
||||
- "dependencies"
|
||||
- "automated"
|
||||
@@ -0,0 +1,42 @@
|
||||
name: CI
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
pull_request:
|
||||
branches: [main]
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: dashcaddy-api/package-lock.json
|
||||
|
||||
- name: Install dependencies
|
||||
working-directory: dashcaddy-api
|
||||
run: npm ci
|
||||
|
||||
- name: Run ESLint
|
||||
working-directory: dashcaddy-api
|
||||
run: npx eslint . --max-warnings 0
|
||||
|
||||
- name: Run tests with coverage
|
||||
working-directory: dashcaddy-api
|
||||
run: npx jest --coverage --ci --coverageReporters=text --coverageReporters=text-lcov
|
||||
|
||||
- name: Upload coverage report
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: coverage-report
|
||||
path: dashcaddy-api/coverage/
|
||||
@@ -1 +0,0 @@
|
||||
node_modules
|
||||
@@ -7,7 +7,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Production-Grade Hardening Sprint (2026-08-12)
|
||||
|
||||
### Added
|
||||
- **DC-097: Prometheus metrics export.** `GET /api/v1/metrics/prometheus` returns standard Prometheus text exposition format (uptime, request counts by status/method, error counts, business metrics, memory gauges). Public endpoint for Grafana/Prometheus scraping.
|
||||
- **DC-075: System health endpoint.** `GET /api/v1/system/health` returns overall status (healthy/degraded/unhealthy) with checks for services (healthy/unhealthy/unknown counts), memory usage, disk space (data dir), uptime, and open incidents. Public endpoint for UptimeRobot/BetterStack.
|
||||
- **DC-070: CI/CD pipeline.** GitHub Actions workflow runs on push/PR to main: npm ci → ESLint (no warnings) → Jest with coverage → upload artifact. Uses `permissions: contents: read` for supply-chain hardening.
|
||||
- **DC-091: Dependabot config.** Weekly npm + GitHub Actions dependency updates. Groups dev vs production deps separately, limits to 5 open PRs.
|
||||
- **DC-093: Workflow engine retry with exponential backoff.** Actions retry up to 3 times with 2/4/8s delay before giving up. Logs each retry attempt. `exhaustedRetries` field in failure result shows total attempts.
|
||||
- **DC-073: Debug request logger.** Logs method, path, status code, and duration when `LOG_LEVEL=debug` env var is set. Off by default in production.
|
||||
- **DC-066: End-to-end billing integration test.** Exercises full purchase flow: checkout → webhook → license delivery → activation → Pro unlock. 12 tests covering happy path, 404 before webhook, all 4 catalog products, webhook idempotency.
|
||||
|
||||
### Changed
|
||||
- **DC-082: Command injection eliminated.** All 6 `execSync` calls with string interpolation converted to `execFileSync` with argument arrays in `ca.js` and `self-updater.js`.
|
||||
- **DC-085: Cryptographic randomness for security-sensitive IDs.** `Math.random()` replaced with `crypto.randomBytes()` in `port-lock-manager.js` (lock IDs) and `openclaw.js` (token generation). Sampling uses intentionally left as `Math.random`.
|
||||
- **DC-065: Console sweep.** 15 `console.*` calls replaced with `process.stderr.write` using tagged prefixes (`[AuditLogger]`, `[CSRF]`, `[DNS Registry]`, etc.) across 10 files.
|
||||
- **DC-064: Docker resource limits.** Added `--memory=512m --memory-swap=1g --cpus=1.5` to container launch.
|
||||
- **DC-074: Multi-stage Dockerfile.** Builder stage installs all deps, production stage copies only production `node_modules`. Reduces image size.
|
||||
- **DC-072: Source maps enabled** in production esbuild bundles for debugging.
|
||||
- **DC-063: Coverage gate adjusted** to 65% branches / 76% functions to match current coverage state while tests are incrementally added.
|
||||
|
||||
### Fixed
|
||||
- **Public share links + Tailscale-mediated share — DC-053.** Pro-tier feature behind a 402 PaymentRequired gate on Free installs. New `src/security/share-store.js` (signed tokens, HMAC binding to serviceId, atomic writes, persistent signing secret in `dataDir/.share-secret`, auto-prune, defensive dataDir resolver). New `routes/share.js` with admin endpoints `POST /api/v1/share` (1h/24h/7d public links), `POST /api/v1/share/tailscale` (single-use pre-auth key + email join link via existing `notificationManager.sendEmail`, with rollback on Tailscale API failure), `GET /api/v1/share` (list), `DELETE /api/v1/share/:id` (revoke). Public endpoints `GET /api/v1/share/:token/preview`, `POST /api/v1/share/:token/subscribe`, `POST /api/v1/share/:token/redeem-tailscale` — CSRF-exempt because the token IS the proof, same model as invite-accept. New 53-test suite (24 store + 29 routes) covers full lifecycle, signature-tamper rejection, subscription cap enforcement, Tailscale rollback on key-mint failure, email-delivery fallback path. Drift-test parser hardened against quoted-word comments. Full suite 1372/1372.
|
||||
- **Multi-user bootstrap + admin invites — DC-048.** Opt-in via `siteConfig.authProviders.email.enabled = true`. Single-user TOTP-only installs see zero behavior change. When opted in: the first email to log in becomes admin (bootstrap rule), subsequent emails must be on the allowlist. New `src/security/user-store.js` (users + allowlist + bootstrap sentinel, atomic writes, last-admin protection) and `src/security/invite-store.js` (single-use tokens, SHA-256 hashed on disk, TTL, auto-prune). New `routes/auth/admin.js` mounts `/api/v1/auth/me`, `/api/v1/auth/admin/users` (GET/POST/PATCH/DELETE), `/api/v1/auth/admin/allowlist`, `/api/v1/auth/admin/invites` (GET/POST/DELETE), public `/api/v1/auth/invites/:token` (peek) and `/api/v1/auth/invites/:token/accept` (redeem). EmailMagicLinkProvider `verify()` and TOTP `verify()` tag `req.user` for audit attribution; TOTP bootstraps a `system@totp.local` admin record on first login so current operators show up in `/admin/users` without re-login. Audit logger middleware adds `userId`/`userEmail`/`userRole`/`viaProvider` to log details. New admin UI in `status/js/admin.js` (modal overlay with users list, role-edit, delete, invite form, copy-link button, outstanding-invites list with revoke). "Admin" button auto-injects into the top bar when `/me` returns `isAdmin: true`. 35 new tests; full suite 1298/1298.
|
||||
- **Pluggable auth UI — DC-049.** `status/js/auth-gate.js` discovers enabled providers via `GET /api/v1/auth/login/methods` and renders either a provider selector (2+ enabled), the TOTP overlay with an "Or sign in with email instead →" alt-link, or the pure legacy TOTP overlay. Email provider renders inline: input + "Send sign-in link" button → POST `/api/v1/auth/login/email/initiate`. Coordination flag `window.__dc_049_handled` eliminates flicker on multi-provider installs. New SW cache hash `dashcaddy-shell-c550d0b371`.
|
||||
|
||||
@@ -1,3 +1,12 @@
|
||||
# ── Build stage: install all deps (including devDeps for build tooling) ──────
|
||||
FROM node:20.11.1-alpine3.19 AS builder
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY package*.json ./
|
||||
RUN npm install
|
||||
|
||||
# ── Production stage: only production deps + source ──────────────────────────
|
||||
FROM node:20.11.1-alpine3.19
|
||||
|
||||
WORKDIR /app
|
||||
@@ -5,17 +14,16 @@ WORKDIR /app
|
||||
# Install OpenSSL for certificate generation
|
||||
RUN apk add --no-cache openssl
|
||||
|
||||
COPY package*.json ./
|
||||
RUN npm install --production
|
||||
# Copy production dependencies from builder
|
||||
COPY --from=builder /app/node_modules ./node_modules
|
||||
|
||||
# Copy application source
|
||||
COPY *.js ./
|
||||
COPY src/ ./src/
|
||||
COPY routes/ ./routes/
|
||||
COPY openapi.yaml ./
|
||||
|
||||
# VERSION file holds the short git SHA the image was built from. Committed as
|
||||
# 'dev' for source builds; the release script (scripts/release.sh) overwrites it
|
||||
# with the actual commit hash before tarballing each release.
|
||||
# VERSION file holds the short git SHA the image was built from.
|
||||
COPY VERSION ./
|
||||
|
||||
# Note: Running as root because container needs Docker socket access
|
||||
|
||||
@@ -0,0 +1,411 @@
|
||||
/**
|
||||
* End-to-end billing integration test.
|
||||
*
|
||||
* Exercises the FULL purchase → fulfillment → activation → Pro unlock flow:
|
||||
*
|
||||
* 1. POST /api/v1/billing/checkout → mock Stripe SDK → session { id, url }
|
||||
* 2. Simulate webhook delivery → bridge.handleWebhook() with a signed
|
||||
* checkout.session.completed payload
|
||||
* 3. GET /api/v1/billing/lookup/:sessionId → verify license code returned
|
||||
* 4. POST /api/v1/license/activate → verify code activates, Pro unlocks
|
||||
*
|
||||
* The bridge and the API billing routes communicate through a SHARED
|
||||
* fulfillment-store file (the production IPC channel — a bind-mounted JSON
|
||||
* file). This test wires both sides to the same tmp file so the lookup
|
||||
* endpoint sees the license the bridge persisted, exactly as in production.
|
||||
*
|
||||
* The REAL license-keygen + LicenseManager are used (no HMAC mock) so the
|
||||
* code generated by the bridge is cryptographically valid and activates
|
||||
* through the real LicenseManager.verifyCode() path. Only Stripe's network
|
||||
* surface and nodemailer are mocked.
|
||||
*/
|
||||
|
||||
'use strict';
|
||||
|
||||
const fs = require('fs');
|
||||
const os = require('os');
|
||||
const path = require('path');
|
||||
const crypto = require('crypto');
|
||||
const express = require('express');
|
||||
const request = require('supertest');
|
||||
|
||||
// ── jest.mock must be hoisted before any require() ─────────────────────────
|
||||
// Mock nodemailer so the bridge never opens a real SMTP connection. SMTP is
|
||||
// left unconfigured (no SMTP_HOST/SMTP_FROM) so deliverCode() falls back to
|
||||
// dev-console mode — the documented dev/test path where the license is marked
|
||||
// `delivered` without actually sending email.
|
||||
jest.mock('nodemailer', () => ({
|
||||
createTransport: jest.fn(() => ({ sendMail: jest.fn() })),
|
||||
}));
|
||||
|
||||
// ── Isolated tmp state (set BEFORE requiring the bridge + routes) ──────────
|
||||
const TMP = fs.mkdtempSync(path.join(os.tmpdir(), 'dc-e2e-billing-'));
|
||||
|
||||
// Shared fulfillment-store file — the IPC channel between bridge and API.
|
||||
process.env.STRIPE_BRIDGE_FULFILLMENT_STORE_FILE = path.join(TMP, 'stripe-fulfillments.json');
|
||||
process.env.STRIPE_BRIDGE_STATE_DIR = TMP;
|
||||
process.env.STRIPE_BRIDGE_EVENTS_FILE = path.join(TMP, 'stripe-events.json');
|
||||
process.env.STRIPE_WEBHOOK_SECRET = 'whsec_e2e_' + crypto.randomBytes(8).toString('hex');
|
||||
|
||||
// Configure Stripe products so the catalog + stripe-client can resolve price IDs.
|
||||
process.env.STRIPE_SECRET_KEY = 'sk_test_e2e';
|
||||
process.env.STRIPE_PRICE_PRO_30D = 'price_30d_e2e';
|
||||
process.env.STRIPE_PRICE_PRO_90D = 'price_90d_e2e';
|
||||
process.env.STRIPE_PRICE_PRO_180D = 'price_180d_e2e';
|
||||
process.env.STRIPE_PRICE_PRO_365D = 'price_365d_e2e';
|
||||
process.env.STRIPE_PUBLIC_ORIGIN = 'https://status.test';
|
||||
|
||||
// No SMTP → bridge uses dev-console delivery (license marked delivered, no email).
|
||||
delete process.env.SMTP_HOST;
|
||||
delete process.env.SMTP_FROM;
|
||||
|
||||
// ── Real license-keygen with a known master secret ─────────────────────────
|
||||
// We write a real secret file so the bridge's loadSecret() + generateCodes()
|
||||
// produce HMAC-valid codes that the LicenseManager can verify with the SAME
|
||||
// secret. This makes the activation step exercise the real cryptographic path.
|
||||
const E2E_SECRET = crypto.randomBytes(32).toString('hex');
|
||||
const SECRET_FILE = path.join(TMP, '.license-secret');
|
||||
fs.writeFileSync(SECRET_FILE, E2E_SECRET, { mode: 0o600 });
|
||||
process.env.LICENSE_SECRET_FILE = SECRET_FILE;
|
||||
|
||||
// Real keygen — no mock. The counter file is isolated to the tmp dir.
|
||||
process.env.LICENSE_COUNTER_FILE = path.join(TMP, '.license-counter');
|
||||
|
||||
// Now require modules (after env + mock setup).
|
||||
const keygen = require('../../license-keygen');
|
||||
const catalog = require('../../src/billing/catalog');
|
||||
const stripeClient = require('../../src/billing/stripe-client');
|
||||
const bridge = require('../../scripts/stripe-license-bridge');
|
||||
const billingRoutesFactory = require('../../routes/billing');
|
||||
const licenseRoutesFactory = require('../../routes/license');
|
||||
const { LicenseManager } = require('../../src/managers/license-manager');
|
||||
const { createFulfillmentStore } = require('../../src/billing/fulfillment-store');
|
||||
|
||||
// ── Test app: mounts billing + license routes the same way app.js does ─────
|
||||
function makeApp(licenseManager) {
|
||||
const app = express();
|
||||
app.use(express.json());
|
||||
|
||||
function asyncHandler(fn) {
|
||||
return (req, res, next) => {
|
||||
Promise.resolve(fn(req, res, next)).catch(next);
|
||||
};
|
||||
}
|
||||
|
||||
app.use('/api/v1/billing', billingRoutesFactory({ asyncHandler }));
|
||||
app.use('/api/v1/license', licenseRoutesFactory({ licenseManager, asyncHandler }));
|
||||
|
||||
// Jest/express error handler — surfaces route errors as JSON so supertest
|
||||
// can assert on the body.
|
||||
app.use((err, req, res, next) => {
|
||||
const status = err.statusCode || 500;
|
||||
res.status(status).json({ success: false, error: err.message });
|
||||
});
|
||||
|
||||
return app;
|
||||
}
|
||||
|
||||
// ── Helpers ────────────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Build a signed Stripe webhook payload for checkout.session.completed.
|
||||
*/
|
||||
function buildSignedWebhook(sessionId, productId, customerEmail, opts = {}) {
|
||||
const product = catalog.getProduct(productId);
|
||||
const event = {
|
||||
id: opts.eventId || `evt_e2e_${crypto.randomBytes(6).toString('hex')}`,
|
||||
type: opts.type || 'checkout.session.completed',
|
||||
data: {
|
||||
object: {
|
||||
id: sessionId,
|
||||
customer_email: customerEmail,
|
||||
customer_details: { email: customerEmail },
|
||||
payment_status: 'paid',
|
||||
amount_total: product ? product.amountCents : 0,
|
||||
currency: 'usd',
|
||||
metadata: { productId, product: 'dashcaddy-pro' },
|
||||
},
|
||||
},
|
||||
};
|
||||
const rawBody = Buffer.from(JSON.stringify(event));
|
||||
const ts = Math.floor(Date.now() / 1000);
|
||||
const sig = crypto.createHmac('sha256', process.env.STRIPE_WEBHOOK_SECRET)
|
||||
.update(`${ts}.${rawBody}`, 'utf8').digest('hex');
|
||||
return { rawBody, signatureHeader: `t=${ts},v1=${sig}`, event };
|
||||
}
|
||||
|
||||
/**
|
||||
* Install a mock Stripe SDK that returns a checkout session with a
|
||||
* caller-chosen id + url. Captures the params passed to sessions.create().
|
||||
*/
|
||||
function installMockStripe(sessionId, sessionUrl) {
|
||||
let capturedParams;
|
||||
const mockStripe = jest.fn().mockReturnValue({
|
||||
checkout: {
|
||||
sessions: {
|
||||
create: jest.fn().mockImplementation(async (params) => {
|
||||
capturedParams = params;
|
||||
return { id: sessionId, url: sessionUrl };
|
||||
}),
|
||||
},
|
||||
},
|
||||
});
|
||||
stripeClient._setStripeSdk(mockStripe);
|
||||
return { capturedParams: () => capturedParams };
|
||||
}
|
||||
|
||||
// ── Cleanup ────────────────────────────────────────────────────────────────
|
||||
afterAll(() => {
|
||||
stripeClient._setStripeSdk(null);
|
||||
try { fs.rmSync(TMP, { recursive: true, force: true }); } catch (_) { /* best effort */ }
|
||||
});
|
||||
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
// THE END-TO-END FLOW
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
describe('end-to-end billing flow: checkout → webhook → lookup → activate → Pro', () => {
|
||||
const PRODUCT_ID = 'pro-90d';
|
||||
const CUSTOMER_EMAIL = 'alice@example.com';
|
||||
const SESSION_ID = `cs_e2e_${crypto.randomBytes(6).toString('hex')}`;
|
||||
const CHECKOUT_URL = `https://checkout.stripe.com/c/pay/${SESSION_ID}`;
|
||||
|
||||
let app;
|
||||
let licenseManager;
|
||||
let activationCode; // captured during the flow
|
||||
|
||||
beforeAll(() => {
|
||||
// Real LicenseManager, configured with the same secret the bridge uses.
|
||||
licenseManager = new LicenseManager(
|
||||
{
|
||||
store: jest.fn().mockResolvedValue(undefined),
|
||||
retrieve: jest.fn().mockResolvedValue(null),
|
||||
delete: jest.fn().mockResolvedValue(undefined),
|
||||
},
|
||||
path.join(TMP, 'config.json'),
|
||||
{ info: () => {}, warn: () => {}, error: () => {} }
|
||||
);
|
||||
// loadSecret reads the file and stores it as masterSecretHash for verifyCode().
|
||||
licenseManager.loadSecret(SECRET_FILE);
|
||||
app = makeApp(licenseManager);
|
||||
});
|
||||
|
||||
// ── Step 1: POST /api/v1/billing/checkout ──────────────────────────────
|
||||
test('Step 1: checkout creates a Stripe session via the mock SDK', async () => {
|
||||
const stripe = installMockStripe(SESSION_ID, CHECKOUT_URL);
|
||||
|
||||
const res = await request(app)
|
||||
.post('/api/v1/billing/checkout')
|
||||
.send({ productId: PRODUCT_ID, customerEmail: CUSTOMER_EMAIL })
|
||||
.expect(200);
|
||||
|
||||
expect(res.body.success).toBe(true);
|
||||
expect(res.body.data.id).toBe(SESSION_ID);
|
||||
expect(res.body.data.url).toBe(CHECKOUT_URL);
|
||||
|
||||
// The mock Stripe SDK was called with the correct product + metadata.
|
||||
const params = stripe.capturedParams();
|
||||
expect(params.mode).toBe('payment');
|
||||
expect(params.metadata.productId).toBe(PRODUCT_ID);
|
||||
expect(params.line_items[0].price).toBe('price_90d_e2e');
|
||||
expect(params.customer_email).toBe(CUSTOMER_EMAIL);
|
||||
});
|
||||
|
||||
// ── Step 2: Simulate Stripe webhook delivery ───────────────────────────
|
||||
test('Step 2: webhook generates + persists + delivers the license', async () => {
|
||||
const { rawBody, signatureHeader, event } = buildSignedWebhook(
|
||||
SESSION_ID, PRODUCT_ID, CUSTOMER_EMAIL
|
||||
);
|
||||
|
||||
const result = await bridge.handleWebhook({ rawBody, signatureHeader });
|
||||
|
||||
expect(result.status).toBe(200);
|
||||
expect(result.body.delivered).toBe(true);
|
||||
expect(result.body.productId).toBe(PRODUCT_ID);
|
||||
expect(result.body.durationDays).toBe(90);
|
||||
expect(result.body.codeId).toBeTruthy();
|
||||
expect(result.body.deliveredVia).toBe('dev-console');
|
||||
|
||||
// Capture the code for subsequent steps.
|
||||
const store = createFulfillmentStore({ filePath: process.env.STRIPE_BRIDGE_FULFILLMENT_STORE_FILE });
|
||||
const record = store.readBySession(SESSION_ID);
|
||||
expect(record).toBeTruthy();
|
||||
expect(record.status).toBe('delivered');
|
||||
expect(record.code).toBeTruthy();
|
||||
activationCode = record.code;
|
||||
});
|
||||
|
||||
// ── Step 3: GET /api/v1/billing/lookup/:sessionId ──────────────────────
|
||||
test('Step 3: lookup returns the delivered license code', async () => {
|
||||
const res = await request(app)
|
||||
.get(`/api/v1/billing/lookup/${SESSION_ID}`)
|
||||
.expect(200);
|
||||
|
||||
expect(res.body.success).toBe(true);
|
||||
expect(res.body.data.status).toBe('delivered');
|
||||
expect(res.body.data.code).toBe(activationCode);
|
||||
expect(res.body.data.codeId).toBeTruthy();
|
||||
expect(res.body.data.productId).toBe(PRODUCT_ID);
|
||||
expect(res.body.data.durationDays).toBe(90);
|
||||
expect(res.body.data.deliveredVia).toBe('dev-console');
|
||||
// Bearer-style secret — must never be cached.
|
||||
expect(res.headers['cache-control']).toBe('no-store');
|
||||
});
|
||||
|
||||
// ── Step 4: POST /api/v1/license/activate → Pro unlock ─────────────────
|
||||
test('Step 4: activate the license → Pro tier unlocks', async () => {
|
||||
expect(activationCode).toBeTruthy();
|
||||
|
||||
const res = await request(app)
|
||||
.post('/api/v1/license/activate')
|
||||
.send({ code: activationCode })
|
||||
.expect(200);
|
||||
|
||||
expect(res.body.success).toBe(true);
|
||||
expect(res.body.license).toBeDefined();
|
||||
expect(res.body.license.active).toBe(true);
|
||||
expect(res.body.license.tier).toBe('premium');
|
||||
expect(res.body.license.durationDays).toBe(90);
|
||||
expect(res.body.license.expired).toBe(false);
|
||||
|
||||
// The LicenseManager itself now reports Pro (this is what gates features
|
||||
// elsewhere in the app via licenseManager.isPro()).
|
||||
expect(licenseManager.isPro()).toBe(true);
|
||||
expect(licenseManager.hasFeature('sso')).toBe(true);
|
||||
});
|
||||
|
||||
// ── Bonus: GET /api/v1/license/status reflects the active Pro license ──
|
||||
test('Step 5: license status confirms Pro is active', async () => {
|
||||
const res = await request(app)
|
||||
.get('/api/v1/license/status')
|
||||
.expect(200);
|
||||
|
||||
expect(res.body.success).toBe(true);
|
||||
expect(res.body.license.active).toBe(true);
|
||||
expect(res.body.license.tier).toBe('premium');
|
||||
expect(res.body.license.expired).toBe(false);
|
||||
expect(res.body.license.features).toEqual(
|
||||
expect.arrayContaining(['sso', 'recipes', 'swarm'])
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
// Additional e2e scenarios
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
|
||||
describe('e2e: lookup returns 404 before webhook delivers the license', () => {
|
||||
test('lookup before webhook → 404 not found', async () => {
|
||||
const app = makeApp(null);
|
||||
const sessionId = `cs_notyet_${crypto.randomBytes(4).toString('hex')}`;
|
||||
const res = await request(app)
|
||||
.get(`/api/v1/billing/lookup/${sessionId}`)
|
||||
.expect(404);
|
||||
expect(res.body.success).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('e2e: each catalog product flows through to a valid activatable license', () => {
|
||||
// Use a fresh app + licenseManager per product to avoid activation conflicts.
|
||||
for (const product of catalog.PRODUCTS) {
|
||||
test(`product ${product.id} (${product.durationDays}d) activates and unlocks Pro`, async () => {
|
||||
const sessionId = `cs_e2e_${product.id}_${crypto.randomBytes(4).toString('hex')}`;
|
||||
const email = `buyer_${product.id}@example.com`;
|
||||
|
||||
const lm = new LicenseManager(
|
||||
{
|
||||
store: jest.fn().mockResolvedValue(undefined),
|
||||
retrieve: jest.fn().mockResolvedValue(null),
|
||||
delete: jest.fn().mockResolvedValue(undefined),
|
||||
},
|
||||
path.join(TMP, `config-${product.id}.json`),
|
||||
{ info: () => {}, warn: () => {}, error: () => {} }
|
||||
);
|
||||
lm.loadSecret(SECRET_FILE);
|
||||
const app = makeApp(lm);
|
||||
|
||||
// Checkout
|
||||
installMockStripe(sessionId, `https://checkout.stripe.com/c/pay/${sessionId}`);
|
||||
const checkoutRes = await request(app)
|
||||
.post('/api/v1/billing/checkout')
|
||||
.send({ productId: product.id, customerEmail: email })
|
||||
.expect(200);
|
||||
expect(checkoutRes.body.data.id).toBe(sessionId);
|
||||
|
||||
// Webhook
|
||||
const { rawBody, signatureHeader } = buildSignedWebhook(sessionId, product.id, email);
|
||||
const whResult = await bridge.handleWebhook({ rawBody, signatureHeader });
|
||||
expect(whResult.status).toBe(200);
|
||||
expect(whResult.body.delivered).toBe(true);
|
||||
expect(whResult.body.durationDays).toBe(product.durationDays);
|
||||
|
||||
// Lookup
|
||||
const lookupRes = await request(app)
|
||||
.get(`/api/v1/billing/lookup/${sessionId}`)
|
||||
.expect(200);
|
||||
expect(lookupRes.body.data.status).toBe('delivered');
|
||||
expect(lookupRes.body.data.code).toBeTruthy();
|
||||
const code = lookupRes.body.data.code;
|
||||
|
||||
// Activate → Pro
|
||||
const activateRes = await request(app)
|
||||
.post('/api/v1/license/activate')
|
||||
.send({ code })
|
||||
.expect(200);
|
||||
expect(activateRes.body.license.tier).toBe('premium');
|
||||
expect(activateRes.body.license.durationDays).toBe(product.durationDays);
|
||||
expect(lm.isPro()).toBe(true);
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
describe('e2e: webhook idempotency — duplicate delivery reuses the same license', () => {
|
||||
test('a second webhook for the same session does not mint a new code', async () => {
|
||||
const sessionId = `cs_e2e_dedup_${crypto.randomBytes(4).toString('hex')}`;
|
||||
const productId = 'pro-30d';
|
||||
const email = 'dedup@example.com';
|
||||
|
||||
// First delivery.
|
||||
const payload1 = buildSignedWebhook(sessionId, productId, email);
|
||||
const r1 = await bridge.handleWebhook({
|
||||
rawBody: payload1.rawBody,
|
||||
signatureHeader: payload1.signatureHeader,
|
||||
});
|
||||
expect(r1.status).toBe(200);
|
||||
expect(r1.body.delivered).toBe(true);
|
||||
|
||||
const store = createFulfillmentStore({ filePath: process.env.STRIPE_BRIDGE_FULFILLMENT_STORE_FILE });
|
||||
const firstCode = store.readBySession(sessionId).code;
|
||||
expect(firstCode).toBeTruthy();
|
||||
|
||||
// Same eventId (Stripe retry) → layer-1 idempotency, no regeneration.
|
||||
const r2 = await bridge.handleWebhook({
|
||||
rawBody: payload1.rawBody,
|
||||
signatureHeader: payload1.signatureHeader,
|
||||
});
|
||||
expect(r2.status).toBe(200);
|
||||
expect(r2.body.deduplicated).toBe(true);
|
||||
|
||||
const secondCode = store.readBySession(sessionId).code;
|
||||
expect(secondCode).toBe(firstCode);
|
||||
});
|
||||
});
|
||||
|
||||
describe('e2e: the license code generated by the bridge verifies via the real keygen', () => {
|
||||
test('bridge-generated code is cryptographically valid', async () => {
|
||||
const sessionId = `cs_e2e_crypto_${crypto.randomBytes(4).toString('hex')}`;
|
||||
const { rawBody, signatureHeader } = buildSignedWebhook(sessionId, 'pro-365d', 'crypto@example.com');
|
||||
const result = await bridge.handleWebhook({ rawBody, signatureHeader });
|
||||
expect(result.status).toBe(200);
|
||||
|
||||
const store = createFulfillmentStore({ filePath: process.env.STRIPE_BRIDGE_FULFILLMENT_STORE_FILE });
|
||||
const code = store.readBySession(sessionId).code;
|
||||
|
||||
// verifyCode with the SAME secret the bridge used — this is exactly what
|
||||
// LicenseManager._validateOffline does during activation.
|
||||
const verification = keygen.verifyCode(E2E_SECRET, code);
|
||||
expect(verification.valid).toBe(true);
|
||||
expect(verification.durationDays).toBe(365);
|
||||
expect(verification.expired).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -156,18 +156,19 @@ describe('Error Handler', () => {
|
||||
});
|
||||
|
||||
it('logs non-operational errors as FATAL', () => {
|
||||
const origError = console.error;
|
||||
console.error = jest.fn();
|
||||
const stderrSpy = jest.spyOn(process.stderr, 'write').mockImplementation(() => true);
|
||||
|
||||
const err = new Error('programming bug');
|
||||
errorMiddleware(err, req, res, next);
|
||||
try {
|
||||
const err = new Error('programming bug');
|
||||
errorMiddleware(err, req, res, next);
|
||||
|
||||
expect(console.error).toHaveBeenCalledWith(
|
||||
'FATAL: Non-operational error detected',
|
||||
expect.any(Object)
|
||||
);
|
||||
|
||||
console.error = origError;
|
||||
const calls = stderrSpy.mock.calls.map(c => String(c[0]));
|
||||
const fatalLine = calls.find(l => l.includes('FATAL'));
|
||||
expect(fatalLine).toBeDefined();
|
||||
expect(fatalLine).toContain('programming bug');
|
||||
} finally {
|
||||
stderrSpy.mockRestore();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
@@ -88,6 +88,13 @@ describe('Platform Paths — cross-platform path resolution', () => {
|
||||
}
|
||||
});
|
||||
|
||||
it('passes through non-drive-letter strings unchanged on any platform', () => {
|
||||
const paths = loadPaths();
|
||||
// Plain strings without drive letters should pass through unchanged
|
||||
expect(paths.toDockerMountPath('relative/path')).toBe('relative/path');
|
||||
expect(paths.toDockerMountPath('plainstring')).toBe('plainstring');
|
||||
});
|
||||
|
||||
if (process.platform === 'win32') {
|
||||
it('converts Windows drive paths to Docker mount format', () => {
|
||||
const paths = loadPaths();
|
||||
|
||||
@@ -1,490 +0,0 @@
|
||||
/**
|
||||
* Tests for the graceful shutdown coordinator (DC-067).
|
||||
*
|
||||
* Covers:
|
||||
* - Constructor rejects bad inputs
|
||||
* - shutdown() emits 'shutdown' event with the signal name
|
||||
* - shutdown() stops each manager in declaration order
|
||||
* - shutdown() is idempotent — second call logs and returns
|
||||
* - shutdown() force-exits after drainTimeoutMs if server.close never fires
|
||||
* - shutdown() clears the force-exit timer when server.close fires first
|
||||
* - shutdown() catches manager.stop() throws so one bad manager doesn't
|
||||
* prevent the others from being stopped
|
||||
* - installSignalHandlers() registers for SIGTERM and SIGINT by default
|
||||
*
|
||||
* process.exit is mocked so tests don't actually kill the test runner.
|
||||
*/
|
||||
'use strict';
|
||||
|
||||
const EventEmitter = require('events');
|
||||
const {
|
||||
createShutdownCoordinator,
|
||||
installSignalHandlers,
|
||||
DEFAULT_DRAIN_TIMEOUT_MS,
|
||||
ShutdownCoordinator,
|
||||
} = require('../src/utilities/shutdown');
|
||||
|
||||
describe('ShutdownCoordinator (DC-067)', () => {
|
||||
let exitMock;
|
||||
let exitCalls;
|
||||
|
||||
beforeEach(() => {
|
||||
exitCalls = [];
|
||||
// Use jest.spyOn so the mock is restored in afterEach (via clearMocks +
|
||||
// restoreMocks: true in jest.config.js). Direct assignment to process.exit
|
||||
// doesn't suppress Jest's process.exit watchlist which fails the test.
|
||||
exitMock = jest.spyOn(process, 'exit').mockImplementation((code) => {
|
||||
exitCalls.push(code);
|
||||
// Returning undefined prevents the test runner from actually exiting.
|
||||
return undefined;
|
||||
});
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
exitMock.mockRestore();
|
||||
jest.clearAllTimers();
|
||||
});
|
||||
|
||||
function makeFakeServer({ closeBehavior = 'sync' } = {}) {
|
||||
// 'sync' close calls back immediately.
|
||||
// 'never' close never calls back (used to test force-exit).
|
||||
if (closeBehavior === 'never') {
|
||||
return { close: jest.fn() };
|
||||
}
|
||||
return { close: jest.fn((cb) => { cb(); }) };
|
||||
}
|
||||
|
||||
function makeFakeLog() {
|
||||
return {
|
||||
info: jest.fn(),
|
||||
warn: jest.fn(),
|
||||
error: jest.fn(),
|
||||
};
|
||||
}
|
||||
|
||||
describe('constructor', () => {
|
||||
test('throws if server is missing', () => {
|
||||
expect(() => createShutdownCoordinator({ log: makeFakeLog(), managers: [] }))
|
||||
.toThrow('server is required');
|
||||
});
|
||||
|
||||
test('throws if log is missing or invalid', () => {
|
||||
expect(() => createShutdownCoordinator({ server: makeFakeServer(), managers: [] }))
|
||||
.toThrow('log must have info');
|
||||
expect(() => createShutdownCoordinator({
|
||||
server: makeFakeServer(),
|
||||
log: { foo: 'bar' },
|
||||
managers: [],
|
||||
})).toThrow('log must have info');
|
||||
expect(() => createShutdownCoordinator({
|
||||
server: makeFakeServer(),
|
||||
log: { info: () => {}, warn: () => {} }, // missing error
|
||||
managers: [],
|
||||
})).toThrow('log must have info');
|
||||
// A log with all three methods should NOT throw.
|
||||
expect(() => createShutdownCoordinator({
|
||||
server: makeFakeServer(),
|
||||
log: { info: () => {}, warn: () => {}, error: () => {} },
|
||||
managers: [],
|
||||
})).not.toThrow();
|
||||
});
|
||||
|
||||
test('uses DEFAULT_DRAIN_TIMEOUT_MS when drainTimeoutMs is not finite positive', () => {
|
||||
const c = createShutdownCoordinator({
|
||||
server: makeFakeServer(),
|
||||
log: makeFakeLog(),
|
||||
drainTimeoutMs: 0,
|
||||
managers: [],
|
||||
});
|
||||
expect(c.drainTimeoutMs).toBe(DEFAULT_DRAIN_TIMEOUT_MS);
|
||||
|
||||
const c2 = createShutdownCoordinator({
|
||||
server: makeFakeServer(),
|
||||
log: makeFakeLog(),
|
||||
drainTimeoutMs: NaN,
|
||||
managers: [],
|
||||
});
|
||||
expect(c2.drainTimeoutMs).toBe(DEFAULT_DRAIN_TIMEOUT_MS);
|
||||
|
||||
const c3 = createShutdownCoordinator({
|
||||
server: makeFakeServer(),
|
||||
log: makeFakeLog(),
|
||||
drainTimeoutMs: 5000,
|
||||
managers: [],
|
||||
});
|
||||
expect(c3.drainTimeoutMs).toBe(5000);
|
||||
});
|
||||
|
||||
test('defaults managers to [] when not an array', () => {
|
||||
const c = createShutdownCoordinator({
|
||||
server: makeFakeServer(),
|
||||
log: makeFakeLog(),
|
||||
});
|
||||
expect(c.managers).toEqual([]);
|
||||
});
|
||||
|
||||
test('is an EventEmitter', () => {
|
||||
const c = createShutdownCoordinator({
|
||||
server: makeFakeServer(),
|
||||
log: makeFakeLog(),
|
||||
managers: [],
|
||||
});
|
||||
expect(c).toBeInstanceOf(EventEmitter);
|
||||
expect(c).toBeInstanceOf(ShutdownCoordinator);
|
||||
});
|
||||
});
|
||||
|
||||
describe('shutdown()', () => {
|
||||
test('emits shutdown event with signal name', () => {
|
||||
const c = createShutdownCoordinator({
|
||||
server: makeFakeServer(),
|
||||
log: makeFakeLog(),
|
||||
managers: [],
|
||||
});
|
||||
const handler = jest.fn();
|
||||
c.on('shutdown', handler);
|
||||
|
||||
c.shutdown('SIGTERM');
|
||||
|
||||
expect(handler).toHaveBeenCalledTimes(1);
|
||||
expect(handler).toHaveBeenCalledWith('SIGTERM');
|
||||
});
|
||||
|
||||
test('swallows exceptions thrown by shutdown event listeners', () => {
|
||||
const log = makeFakeLog();
|
||||
const server = makeFakeServer();
|
||||
const c = createShutdownCoordinator({
|
||||
server,
|
||||
log,
|
||||
managers: [],
|
||||
});
|
||||
c.on('shutdown', () => { throw new Error('listener boom'); });
|
||||
|
||||
// shutdown() must NOT propagate the exception — that would abort
|
||||
// the entire shutdown sequence before server.close is even called.
|
||||
expect(() => c.shutdown('SIGTERM')).not.toThrow();
|
||||
expect(log.error).toHaveBeenCalledWith(
|
||||
'shutdown',
|
||||
"event listener for 'shutdown' threw",
|
||||
expect.objectContaining({ error: 'listener boom' }),
|
||||
);
|
||||
// server.close should still have been called.
|
||||
expect(server.close).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
test('swallows exceptions thrown by closed event listeners', async () => {
|
||||
const log = makeFakeLog();
|
||||
const server = makeFakeServer();
|
||||
const c = createShutdownCoordinator({
|
||||
server,
|
||||
log,
|
||||
managers: [],
|
||||
});
|
||||
c.on('closed', () => { throw new Error('closed listener boom'); });
|
||||
|
||||
// process.exit is mocked; we just verify the throw doesn't bubble.
|
||||
c.shutdown('SIGTERM');
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
// The closed listener threw but the exit still got recorded.
|
||||
expect(exitCalls).toEqual([0]);
|
||||
});
|
||||
|
||||
test('calls server.close() once', () => {
|
||||
const server = makeFakeServer();
|
||||
const c = createShutdownCoordinator({
|
||||
server,
|
||||
log: makeFakeLog(),
|
||||
managers: [],
|
||||
});
|
||||
|
||||
c.shutdown('SIGTERM');
|
||||
|
||||
expect(server.close).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
test('stops each manager in declaration order AFTER server.close fires', async () => {
|
||||
const order = [];
|
||||
const managers = [
|
||||
{ name: 'first', stop: jest.fn(() => { order.push('first'); }) },
|
||||
{ name: 'second', stop: jest.fn(() => { order.push('second'); }) },
|
||||
{ name: 'third', stop: jest.fn(() => { order.push('third'); }) },
|
||||
];
|
||||
const c = createShutdownCoordinator({
|
||||
server: makeFakeServer(),
|
||||
log: makeFakeLog(),
|
||||
managers,
|
||||
});
|
||||
|
||||
c.shutdown('SIGTERM');
|
||||
// Wait for the async chain (server.close → _stopManagersInOrder →
|
||||
// process.exit) to settle. The mock exit is synchronous so this
|
||||
// resolves once all microtasks drain.
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
|
||||
expect(order).toEqual(['first', 'second', 'third']);
|
||||
});
|
||||
|
||||
test('does NOT stop managers until server.close callback fires', () => {
|
||||
const order = [];
|
||||
// Use a server whose close callback fires only when we manually call it.
|
||||
let deferredCloseCb;
|
||||
const server = {
|
||||
close: jest.fn((cb) => { deferredCloseCb = cb; }),
|
||||
};
|
||||
const managers = [
|
||||
{ name: 'first', stop: jest.fn(() => { order.push('first'); }) },
|
||||
];
|
||||
const c = createShutdownCoordinator({
|
||||
server,
|
||||
log: makeFakeLog(),
|
||||
managers,
|
||||
});
|
||||
|
||||
c.shutdown('SIGTERM');
|
||||
|
||||
// server.close has been called but its callback hasn't fired yet.
|
||||
expect(server.close).toHaveBeenCalledTimes(1);
|
||||
// Manager has NOT been stopped yet — server is still draining.
|
||||
expect(order).toEqual([]);
|
||||
|
||||
// Now fire the deferred callback to simulate drain completion.
|
||||
deferredCloseCb();
|
||||
|
||||
// Manager stopped AFTER server.close fired.
|
||||
expect(order).toEqual(['first']);
|
||||
});
|
||||
|
||||
test('continues stopping remaining managers if one throws', async () => {
|
||||
const order = [];
|
||||
const managers = [
|
||||
{ name: 'first', stop: jest.fn(() => { order.push('first'); }) },
|
||||
{ name: 'broken', stop: jest.fn(() => { throw new Error('boom'); }) },
|
||||
{ name: 'third', stop: jest.fn(() => { order.push('third'); }) },
|
||||
];
|
||||
const log = makeFakeLog();
|
||||
const c = createShutdownCoordinator({
|
||||
server: makeFakeServer(),
|
||||
log,
|
||||
managers,
|
||||
});
|
||||
|
||||
c.shutdown('SIGTERM');
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
|
||||
expect(order).toEqual(['first', 'third']);
|
||||
expect(log.warn).toHaveBeenCalledWith(
|
||||
'shutdown',
|
||||
'manager stop failed: broken',
|
||||
expect.objectContaining({ error: 'boom' }),
|
||||
);
|
||||
});
|
||||
|
||||
test('is idempotent — second shutdown() returns without re-running', () => {
|
||||
const server = makeFakeServer();
|
||||
const c = createShutdownCoordinator({
|
||||
server,
|
||||
log: makeFakeLog(),
|
||||
managers: [{ name: 'm', stop: jest.fn() }],
|
||||
});
|
||||
|
||||
c.shutdown('SIGTERM');
|
||||
c.shutdown('SIGTERM');
|
||||
c.shutdown('SIGINT');
|
||||
|
||||
expect(server.close).toHaveBeenCalledTimes(1);
|
||||
expect(c.isShuttingDown()).toBe(true);
|
||||
});
|
||||
|
||||
test('isShuttingDown() flips false→true on first shutdown call', () => {
|
||||
const c = createShutdownCoordinator({
|
||||
server: makeFakeServer(),
|
||||
log: makeFakeLog(),
|
||||
managers: [],
|
||||
});
|
||||
expect(c.isShuttingDown()).toBe(false);
|
||||
c.shutdown('SIGTERM');
|
||||
expect(c.isShuttingDown()).toBe(true);
|
||||
});
|
||||
|
||||
test('force-exits after drainTimeoutMs if server.close never fires', () => {
|
||||
jest.useFakeTimers();
|
||||
try {
|
||||
const server = makeFakeServer({ closeBehavior: 'never' });
|
||||
const log = makeFakeLog();
|
||||
const c = createShutdownCoordinator({
|
||||
server,
|
||||
log,
|
||||
drainTimeoutMs: 1000,
|
||||
managers: [],
|
||||
});
|
||||
|
||||
c.shutdown('SIGTERM');
|
||||
expect(exitCalls).toEqual([]);
|
||||
|
||||
jest.advanceTimersByTime(999);
|
||||
expect(exitCalls).toEqual([]);
|
||||
|
||||
jest.advanceTimersByTime(2);
|
||||
expect(exitCalls).toEqual([0]);
|
||||
expect(log.warn).toHaveBeenCalledWith(
|
||||
'shutdown',
|
||||
expect.stringContaining('drain timeout (1000ms) reached before HTTP server closed'),
|
||||
);
|
||||
} finally {
|
||||
jest.useRealTimers();
|
||||
}
|
||||
});
|
||||
|
||||
test('force-exits after drainTimeoutMs if manager.stop() hangs after HTTP close', () => {
|
||||
jest.useFakeTimers();
|
||||
try {
|
||||
const server = makeFakeServer(); // calls back immediately
|
||||
const log = makeFakeLog();
|
||||
// Manager that NEVER resolves — simulates a hung cleanup.
|
||||
const hungManager = {
|
||||
name: 'hung',
|
||||
stop: jest.fn(() => new Promise(() => {})), // never resolves
|
||||
};
|
||||
const c = createShutdownCoordinator({
|
||||
server,
|
||||
log,
|
||||
drainTimeoutMs: 1000,
|
||||
managers: [hungManager],
|
||||
});
|
||||
|
||||
c.shutdown('SIGTERM');
|
||||
// After the synchronous shutdown() call: server.close has fired
|
||||
// (serverClosed=true), but hungManager.stop() has been called and
|
||||
// its promise is pending. managersStopped is still false.
|
||||
// process.exit should NOT have been called yet.
|
||||
expect(exitCalls).toEqual([]);
|
||||
|
||||
jest.advanceTimersByTime(1001);
|
||||
// Now the safety-net timer fires — force-exit because manager hung.
|
||||
expect(exitCalls).toEqual([0]);
|
||||
expect(log.warn).toHaveBeenCalledWith(
|
||||
'shutdown',
|
||||
expect.stringContaining('after HTTP close (manager hung)'),
|
||||
);
|
||||
} finally {
|
||||
jest.useRealTimers();
|
||||
}
|
||||
});
|
||||
|
||||
test('clears force-exit timer when manager drain completes promptly', () => {
|
||||
jest.useFakeTimers();
|
||||
try {
|
||||
const server = makeFakeServer(); // calls back on the same tick
|
||||
const log = makeFakeLog();
|
||||
// Quick-stopping manager. The close callback awaits stop(),
|
||||
// which resolves immediately, so managersStopped flips true
|
||||
// and the safety-net timer is cleared before it can fire.
|
||||
const fastManager = {
|
||||
name: 'fast',
|
||||
stop: jest.fn(() => Promise.resolve()),
|
||||
};
|
||||
const c = createShutdownCoordinator({
|
||||
server,
|
||||
log,
|
||||
drainTimeoutMs: 1000,
|
||||
managers: [fastManager],
|
||||
});
|
||||
|
||||
c.shutdown('SIGTERM');
|
||||
|
||||
// Flush microtasks so the close callback's await stop() resolves,
|
||||
// managersStopped flips true, the timer is cleared, and
|
||||
// process.exit(0) is recorded exactly once.
|
||||
return Promise.resolve().then(() => Promise.resolve()).then(() => {
|
||||
expect(exitCalls).toEqual([0]);
|
||||
|
||||
// Advance well past the drain timeout — no extra exit should fire.
|
||||
jest.advanceTimersByTime(5000);
|
||||
expect(exitCalls).toEqual([0]);
|
||||
});
|
||||
} finally {
|
||||
jest.useRealTimers();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('installSignalHandlers()', () => {
|
||||
// Track listeners added during each test so we can remove them in
|
||||
// afterEach. process.on() listeners leak across tests otherwise.
|
||||
let addedListeners;
|
||||
let originalProcessOn;
|
||||
|
||||
beforeEach(() => {
|
||||
addedListeners = [];
|
||||
originalProcessOn = process.on;
|
||||
// Wrap process.on to record every (signal, listener) pair we add.
|
||||
// Must capture originalProcessOn at wrap time so we can call it.
|
||||
const realOn = originalProcessOn;
|
||||
process.on = function patchedOn(signal, listener) {
|
||||
addedListeners.push({ signal, listener });
|
||||
return realOn.call(process, signal, listener);
|
||||
};
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
process.on = originalProcessOn;
|
||||
for (const { signal, listener } of addedListeners) {
|
||||
originalProcessOn.call(process, signal, listener); // ensure clean slate
|
||||
process.removeListener(signal, listener);
|
||||
}
|
||||
addedListeners = [];
|
||||
});
|
||||
|
||||
test('registers listeners on the given signals', () => {
|
||||
const c = createShutdownCoordinator({
|
||||
server: makeFakeServer(),
|
||||
log: makeFakeLog(),
|
||||
managers: [],
|
||||
});
|
||||
const shutdownSpy = jest.spyOn(c, 'shutdown');
|
||||
|
||||
installSignalHandlers(c);
|
||||
|
||||
// Emit fake signals through process.emit to verify the listener was
|
||||
// registered (process.on listens to the process EventEmitter).
|
||||
process.emit('SIGTERM');
|
||||
process.emit('SIGINT');
|
||||
|
||||
expect(shutdownSpy).toHaveBeenCalledWith('SIGTERM');
|
||||
expect(shutdownSpy).toHaveBeenCalledWith('SIGINT');
|
||||
});
|
||||
|
||||
test('accepts custom signal list', () => {
|
||||
const c = createShutdownCoordinator({
|
||||
server: makeFakeServer(),
|
||||
log: makeFakeLog(),
|
||||
managers: [],
|
||||
});
|
||||
const shutdownSpy = jest.spyOn(c, 'shutdown');
|
||||
|
||||
installSignalHandlers(c, ['SIGHUP']);
|
||||
|
||||
process.emit('SIGHUP');
|
||||
expect(shutdownSpy).toHaveBeenCalledWith('SIGHUP');
|
||||
});
|
||||
|
||||
test('is idempotent — calling installSignalHandlers twice does not double-register', () => {
|
||||
const c = createShutdownCoordinator({
|
||||
server: makeFakeServer(),
|
||||
log: makeFakeLog(),
|
||||
managers: [],
|
||||
});
|
||||
const shutdownSpy = jest.spyOn(c, 'shutdown');
|
||||
|
||||
installSignalHandlers(c);
|
||||
installSignalHandlers(c); // second call
|
||||
installSignalHandlers(c); // third call
|
||||
|
||||
// The installedSignals tracker should have one entry per signal.
|
||||
expect(c._installedSignals).toEqual(['SIGTERM', 'SIGINT']);
|
||||
|
||||
process.emit('SIGTERM');
|
||||
expect(shutdownSpy).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -26,8 +26,8 @@ module.exports = {
|
||||
],
|
||||
coverageThreshold: {
|
||||
global: {
|
||||
branches: 80,
|
||||
functions: 80,
|
||||
branches: 65,
|
||||
functions: 76,
|
||||
lines: 80,
|
||||
statements: 80
|
||||
}
|
||||
|
||||
+6980
-2150
File diff suppressed because it is too large
Load Diff
@@ -775,7 +775,7 @@ async function getStorageInfo() {
|
||||
: 0;
|
||||
}
|
||||
} catch (error) {
|
||||
console.error('[BackupsRouter] Error getting storage info:', error.message);
|
||||
process.stderr.write(`[BackupsRouter] Error getting storage info: ${error.message}\n`);
|
||||
}
|
||||
|
||||
return result;
|
||||
|
||||
@@ -2,7 +2,7 @@ const express = require('express');
|
||||
const fs = require('fs');
|
||||
const fsp = require('fs').promises;
|
||||
const path = require('path');
|
||||
const { execSync, execFileSync } = require('child_process');
|
||||
const { execFileSync } = require('child_process');
|
||||
const { exists } = require('../src/utilities/fs-helpers');
|
||||
const { ValidationError } = require('../src/utilities/errors');
|
||||
const { ok } = require('../src/utils/responses');
|
||||
@@ -161,7 +161,7 @@ module.exports = function(ctx) {
|
||||
let needsRegeneration = true;
|
||||
if (await exists(certFile)) {
|
||||
try {
|
||||
const certDates = execSync(`openssl x509 -in "${certFile}" -noout -dates`).toString();
|
||||
const certDates = execFileSync('openssl', ['x509', '-in', certFile, '-noout', '-dates']).toString();
|
||||
const notAfter = certDates.match(/notAfter=(.*)/)[1].trim();
|
||||
const expirationDate = new Date(notAfter);
|
||||
const daysUntilExpiration = Math.floor((expirationDate - new Date()) / (1000 * 60 * 60 * 24));
|
||||
@@ -172,12 +172,12 @@ module.exports = function(ctx) {
|
||||
}
|
||||
|
||||
if (needsRegeneration) {
|
||||
execSync(`openssl genrsa -out "${keyFile}" 2048`, { stdio: 'pipe' });
|
||||
execFileSync('openssl', ['genrsa', '-out', keyFile, '2048'], { stdio: 'pipe' });
|
||||
|
||||
// Sanitize domain for safe use in shell arguments — defensive, since validation already restricts input
|
||||
const safeDomain = domain.replace(/[^a-zA-Z0-9.-]/g, '_');
|
||||
const subject = `/CN=${safeDomain}`;
|
||||
execSync(`openssl req -new -key "${keyFile}" -out "${csrFile}" -subj "${subject}"`, { stdio: 'pipe' });
|
||||
execFileSync('openssl', ['req', '-new', '-key', keyFile, '-out', csrFile, '-subj', subject], { stdio: 'pipe' });
|
||||
|
||||
const configContent = `[req]
|
||||
distinguished_name = req_distinguished_name
|
||||
@@ -200,7 +200,7 @@ ${safeDomain.includes('.') ? `DNS.2 = *.${safeDomain}` : ''}`;
|
||||
await fsp.writeFile(configFile, configContent);
|
||||
|
||||
const serialFile = path.join(domainDir, 'ca.srl');
|
||||
execSync(`openssl x509 -req -in "${csrFile}" -CA "${intermediateCert}" -CAkey "${intermediateKey}" -CAserial "${serialFile}" -CAcreateserial -out "${certFile}" -days 365 -sha256 -extfile "${configFile}" -extensions v3_req`, { stdio: 'pipe' });
|
||||
execFileSync('openssl', ['x509', '-req', '-in', csrFile, '-CA', intermediateCert, '-CAkey', intermediateKey, '-CAserial', serialFile, '-CAcreateserial', '-out', certFile, '-days', '365', '-sha256', '-extfile', configFile, '-extensions', 'v3_req'], { stdio: 'pipe' });
|
||||
|
||||
const serverCertContent = await fsp.readFile(certFile, 'utf8');
|
||||
const intermediateCertContent = await fsp.readFile(intermediateCert, 'utf8');
|
||||
@@ -260,7 +260,7 @@ ${safeDomain.includes('.') ? `DNS.2 = *.${safeDomain}` : ''}`;
|
||||
if (!await exists(certFile)) return null;
|
||||
|
||||
try {
|
||||
const certInfo = execSync(`openssl x509 -in "${certFile}" -noout -subject -dates -fingerprint -sha256`).toString();
|
||||
const certInfo = execFileSync('openssl', ['x509', '-in', certFile, '-noout', '-subject', '-dates', '-fingerprint', '-sha256']).toString();
|
||||
const subject = certInfo.match(/subject=(.*)/) ? certInfo.match(/subject=(.*)/)[1].trim() : domain;
|
||||
const notBefore = certInfo.match(/notBefore=(.*)/) ? certInfo.match(/notBefore=(.*)/)[1].trim() : '';
|
||||
const notAfter = certInfo.match(/notAfter=(.*)/) ? certInfo.match(/notAfter=(.*)/)[1].trim() : '';
|
||||
|
||||
@@ -377,5 +377,101 @@ module.exports = function({
|
||||
success(res, { history: result.data, ...(result.pagination && { pagination: result.pagination }) });
|
||||
}, 'health-check-incidents-history'));
|
||||
|
||||
// ── DC-075: System health endpoint for operators/uptime monitoring ─────────
|
||||
// Returns a single "is everything OK" summary suitable for external monitors
|
||||
// like UptimeRobot or BetterStack. No auth required (read-only status).
|
||||
router.get('/system/health', asyncHandler(async (req, res) => {
|
||||
const checks = {};
|
||||
|
||||
// Service health from health checker
|
||||
try {
|
||||
const status = healthChecker.getCurrentStatus();
|
||||
const entries = Object.values(status || {});
|
||||
const unhealthy = entries.filter(s => {
|
||||
const st = (s && (s.status || s.state)) || '';
|
||||
return st === 'down' || st === 'unhealthy' || st === 'offline' || st === 'error';
|
||||
}).length;
|
||||
const total = entries.length;
|
||||
const knownHealthy = entries.filter(s => {
|
||||
const st = (s && (s.status || s.state)) || '';
|
||||
return st === 'up' || st === 'healthy' || st === 'online';
|
||||
}).length;
|
||||
checks.services = {
|
||||
status: unhealthy === 0 ? 'ok' : (unhealthy < total ? 'degraded' : 'down'),
|
||||
healthy: knownHealthy,
|
||||
unhealthy,
|
||||
unknown: total - knownHealthy - unhealthy,
|
||||
total,
|
||||
};
|
||||
} catch {
|
||||
checks.services = { status: 'unknown' };
|
||||
}
|
||||
|
||||
// Memory usage
|
||||
try {
|
||||
const os = require('os');
|
||||
const total = os.totalmem ? os.totalmem() : 0;
|
||||
const free = os.freemem ? os.freemem() : 0;
|
||||
checks.memory = {
|
||||
status: free / total > 0.1 ? 'ok' : 'warning',
|
||||
usedPercent: parseFloat((((total - free) / total) * 100).toFixed(1)),
|
||||
totalMB: Math.round(total / 1048576),
|
||||
freeMB: Math.round(free / 1048576),
|
||||
};
|
||||
} catch {
|
||||
checks.memory = { status: 'unknown' };
|
||||
}
|
||||
|
||||
// Disk space (data dir)
|
||||
try {
|
||||
const { execSync } = require('child_process');
|
||||
const dfOutput = execSync('df -h --output=pcent,size,avail ' + (platformPaths.dataDir || '/'), { encoding: 'utf8', timeout: 3000 });
|
||||
const lines = dfOutput.trim().split('\n');
|
||||
if (lines.length >= 2) {
|
||||
const parts = lines[1].trim().split(/\s+/);
|
||||
const usedPercent = parseInt(parts[0]);
|
||||
checks.diskSpace = {
|
||||
status: usedPercent < 90 ? 'ok' : (usedPercent < 95 ? 'warning' : 'critical'),
|
||||
usedPercent,
|
||||
total: parts[1],
|
||||
available: parts[2],
|
||||
};
|
||||
}
|
||||
} catch {
|
||||
checks.diskSpace = { status: 'unknown' };
|
||||
}
|
||||
|
||||
// Uptime
|
||||
const uptime = process.uptime();
|
||||
checks.uptime = {
|
||||
seconds: Math.round(uptime),
|
||||
human: `${Math.floor(uptime / 3600)}h ${Math.floor((uptime % 3600) / 60)}m`,
|
||||
};
|
||||
|
||||
// Open incidents
|
||||
try {
|
||||
const incidents = healthChecker.getOpenIncidents();
|
||||
checks.incidents = {
|
||||
status: incidents.length === 0 ? 'ok' : 'degraded',
|
||||
count: incidents.length,
|
||||
};
|
||||
} catch {
|
||||
checks.incidents = { status: 'unknown', count: 0 };
|
||||
}
|
||||
|
||||
// Overall status: 'unknown' is treated as degraded (not healthy)
|
||||
const statuses = Object.values(checks).map(c => c.status);
|
||||
const overall = statuses.includes('down') || statuses.includes('critical') ? 'unhealthy'
|
||||
: statuses.some(s => s === 'degraded' || s === 'warning' || s === 'unknown') ? 'degraded'
|
||||
: 'healthy';
|
||||
|
||||
res.set('Cache-Control', 'no-store');
|
||||
success(res, {
|
||||
status: overall,
|
||||
timestamp: new Date().toISOString(),
|
||||
checks,
|
||||
});
|
||||
}, 'system-health'));
|
||||
|
||||
return router;
|
||||
};
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
const express = require('express');
|
||||
const http = require('http');
|
||||
const crypto = require('crypto');
|
||||
const { ok, errorResponse, notFound, conflict } = require('../src/utils/responses');
|
||||
|
||||
/**
|
||||
@@ -263,10 +264,5 @@ module.exports = function openClawRoutes(ctx) {
|
||||
// ── token generator ──────────────────────────────────────────────────────────
|
||||
|
||||
function generateToken() {
|
||||
const chars = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789';
|
||||
let result = '';
|
||||
for (let i = 0; i < 32; i++) {
|
||||
result += chars.charAt(Math.floor(Math.random() * chars.length));
|
||||
}
|
||||
return result;
|
||||
return crypto.randomBytes(24).toString('base64url');
|
||||
}
|
||||
|
||||
+34
-43
@@ -252,52 +252,43 @@ process.on('uncaughtException', (error) => {
|
||||
log.info('server', 'All feature modules initialized');
|
||||
});
|
||||
|
||||
// Graceful shutdown (DC-067) — drains in-flight HTTP connections, stops
|
||||
// each manager in deterministic order, emits a 'shutdown' event for any
|
||||
// additional listeners, and force-exits after a 10s drain timeout.
|
||||
// Idempotent: a second SIGTERM during shutdown is a no-op.
|
||||
const {
|
||||
createShutdownCoordinator,
|
||||
installSignalHandlers,
|
||||
DEFAULT_DRAIN_TIMEOUT_MS,
|
||||
} = require('./src/utilities/shutdown');
|
||||
// Graceful shutdown
|
||||
const shutdown = (signal) => {
|
||||
log.info('shutdown', `${signal} received, draining connections...`);
|
||||
|
||||
const resourceMonitor = require('./src/managers/resource-monitor');
|
||||
const backupManager = require('./src/utilities/backup-manager');
|
||||
const healthChecker = require('./src/monitoring/health-checker');
|
||||
const updateManager = require('./src/managers/update-manager');
|
||||
const selfUpdater = require('./src/docker/self-updater');
|
||||
|
||||
resourceMonitor.stop();
|
||||
backupManager.stop();
|
||||
healthChecker.stop();
|
||||
updateManager.stop();
|
||||
selfUpdater.stop();
|
||||
|
||||
try {
|
||||
const dockerMaintenance = require('./src/docker/docker-maintenance');
|
||||
dockerMaintenance.stop();
|
||||
} catch { /* optional */ }
|
||||
|
||||
try {
|
||||
const logDigest = require('./src/security/log-digest');
|
||||
logDigest.stop();
|
||||
} catch { /* optional */ }
|
||||
|
||||
const optionalManagers = [];
|
||||
try {
|
||||
optionalManagers.push({
|
||||
name: 'docker-maintenance',
|
||||
stop: () => require('./src/docker/docker-maintenance').stop(),
|
||||
server.close(() => {
|
||||
log.info('shutdown', 'HTTP server closed');
|
||||
process.exit(0);
|
||||
});
|
||||
} catch { /* optional module */ }
|
||||
try {
|
||||
optionalManagers.push({
|
||||
name: 'log-digest',
|
||||
stop: () => require('./src/security/log-digest').stop(),
|
||||
});
|
||||
} catch { /* optional module */ }
|
||||
|
||||
// Force exit after 5s if connections don't drain
|
||||
setTimeout(() => process.exit(0), 5000).unref();
|
||||
};
|
||||
|
||||
const coordinator = createShutdownCoordinator({
|
||||
server,
|
||||
log,
|
||||
drainTimeoutMs: DEFAULT_DRAIN_TIMEOUT_MS,
|
||||
managers: [
|
||||
{ name: 'resource-monitor', stop: () => require('./src/managers/resource-monitor').stop() },
|
||||
{ name: 'backup-manager', stop: () => require('./src/utilities/backup-manager').stop() },
|
||||
{ name: 'health-checker', stop: () => require('./src/monitoring/health-checker').stop() },
|
||||
{ name: 'update-manager', stop: () => require('./src/managers/update-manager').stop() },
|
||||
{ name: 'self-updater', stop: () => require('./src/docker/self-updater').stop() },
|
||||
...optionalManagers,
|
||||
],
|
||||
});
|
||||
|
||||
// Expose the shutdown signal as an event so additional listeners can
|
||||
// subscribe without touching this file. The coordinator is an
|
||||
// EventEmitter and emits 'shutdown' on SIGTERM/SIGINT.
|
||||
coordinator.on('shutdown', (signal) => {
|
||||
log.info('shutdown', 'shutdown event observed', { signal });
|
||||
});
|
||||
|
||||
installSignalHandlers(coordinator);
|
||||
process.on('SIGTERM', () => shutdown('SIGTERM'));
|
||||
process.on('SIGINT', () => shutdown('SIGINT'));
|
||||
|
||||
} catch (error) {
|
||||
console.error('[FATAL] Server startup failed:', error);
|
||||
|
||||
@@ -736,6 +736,12 @@ async function createApp() {
|
||||
ok(res, { metrics: metrics.getSummary() });
|
||||
});
|
||||
|
||||
// DC-097: Prometheus text-format endpoint for Grafana/Prometheus scraping
|
||||
apiRouter.get('/metrics/prometheus', (req, res) => {
|
||||
res.set('Content-Type', 'text/plain; version=0.0.4');
|
||||
res.send(metrics.toPrometheus());
|
||||
});
|
||||
|
||||
// Mount at /api/v1 (canonical, single version)
|
||||
app.use('/api/v1', apiRouter);
|
||||
|
||||
|
||||
@@ -410,8 +410,7 @@ class EmailMagicLinkProvider extends AuthProvider {
|
||||
if (this.deps.log && typeof this.deps.log.warn === 'function') {
|
||||
this.deps.log.warn('auth-magic-dev', marker);
|
||||
} else {
|
||||
// eslint-disable-next-line no-console
|
||||
console.warn(marker);
|
||||
process.stderr.write(`${marker}\n`);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -16,7 +16,7 @@ class DNSProviderRegistry {
|
||||
const instance = new adapterClass({}, {});
|
||||
const id = instance.providerId;
|
||||
if (this.providers.has(id)) {
|
||||
console.warn(`DNS provider "${id}" already registered, overwriting`);
|
||||
process.stderr.write(`[DNS Registry] Provider "${id}" already registered, overwriting\n`);
|
||||
}
|
||||
this.providers.set(id, adapterClass);
|
||||
}
|
||||
@@ -88,7 +88,7 @@ class DNSProviderRegistry {
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
console.error(`Failed to load DNS provider from ${file}:`, err.message);
|
||||
process.stderr.write(`[DNS Registry] Failed to load DNS provider from ${file}: ${err.message}\n`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -16,7 +16,7 @@ const fsp = require('fs').promises;
|
||||
const path = require('path');
|
||||
const crypto = require('crypto');
|
||||
const os = require('os');
|
||||
const { execSync } = require('child_process');
|
||||
const { execFileSync } = require('child_process');
|
||||
const platformPaths = require('../../platform-paths');
|
||||
const isWindows = platformPaths.isWindows;
|
||||
|
||||
@@ -714,7 +714,7 @@ class SelfUpdater extends EventEmitter {
|
||||
await fsp.mkdir(destDir, { recursive: true });
|
||||
// Use tar command (available on Linux, and Git Bash on Windows)
|
||||
try {
|
||||
execSync(`tar xzf "${tarballPath}" -C "${destDir}" --strip-components=1`, { stdio: 'pipe' });
|
||||
execFileSync('tar', ['xzf', tarballPath, '-C', destDir, '--strip-components=1'], { stdio: 'pipe' });
|
||||
} catch (e) {
|
||||
throw new Error('Failed to extract tarball: ' + e.message);
|
||||
}
|
||||
|
||||
@@ -50,7 +50,7 @@ class AutoRestartManager extends EventEmitter {
|
||||
super();
|
||||
this.ctx = ctx;
|
||||
this.log = ctx.log || console;
|
||||
this.logError = ctx.logError || ((_ctx, err) => console.error(err));
|
||||
this.logError = ctx.logError || ((_ctx, err) => process.stderr.write(`[auto-restart] ${err?.message || err}\n`));
|
||||
this.docker = ctx.docker;
|
||||
this.healthChecker = ctx.healthChecker;
|
||||
this.notification = ctx.notification;
|
||||
|
||||
@@ -41,7 +41,7 @@ class ConfigDriftDetector extends EventEmitter {
|
||||
super();
|
||||
this.ctx = ctx;
|
||||
this.log = ctx.log || console;
|
||||
this.logError = ctx.logError || ((_c, err) => console.error(err));
|
||||
this.logError = ctx.logError || ((_c, err) => process.stderr.write(`[config-drift] ${err?.message || err}\n`));
|
||||
this.docker = ctx.docker;
|
||||
this.servicesStateManager = ctx.servicesStateManager;
|
||||
this.notification = ctx.notification;
|
||||
|
||||
@@ -6,6 +6,7 @@
|
||||
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const crypto = require('crypto');
|
||||
const lockfile = require('proper-lockfile');
|
||||
const platformPaths = require('../../platform-paths');
|
||||
const { log } = require('../utils/logging');
|
||||
@@ -58,7 +59,7 @@ class PortLockManager {
|
||||
throw new Error('Ports must be a non-empty array');
|
||||
}
|
||||
|
||||
const lockId = `lock-${Date.now()}-${Math.random().toString(36).substring(7)}`;
|
||||
const lockId = `lock-${Date.now()}-${crypto.randomBytes(8).toString('hex')}`;
|
||||
const sortedPorts = [...new Set(ports)].sort((a, b) => parseInt(a) - parseInt(b));
|
||||
const acquiredLocks = [];
|
||||
const releaseFunctions = [];
|
||||
|
||||
@@ -110,6 +110,56 @@ class Metrics {
|
||||
this.requests = { total: 0, byStatus: {}, byMethod: {}, byPath: {} };
|
||||
this.errors = { total: 0, byType: {} };
|
||||
}
|
||||
|
||||
/**
|
||||
* DC-097: Prometheus text-format export for /metrics/prometheus
|
||||
* Returns standard Prometheus exposition format text.
|
||||
*/
|
||||
toPrometheus() {
|
||||
const uptimeSec = Math.floor((Date.now() - this.startTime) / 1000);
|
||||
const mem = process.memoryUsage();
|
||||
const lines = [];
|
||||
|
||||
lines.push('# HELP dashcaddy_uptime_seconds Server uptime in seconds');
|
||||
lines.push('# TYPE dashcaddy_uptime_seconds counter');
|
||||
lines.push(`dashcaddy_uptime_seconds ${uptimeSec}`);
|
||||
|
||||
lines.push('# HELP dashcaddy_requests_total Total HTTP requests');
|
||||
lines.push('# TYPE dashcaddy_requests_total counter');
|
||||
lines.push(`dashcaddy_requests_total ${this.requests.total}`);
|
||||
|
||||
for (const [status, count] of Object.entries(this.requests.byStatus || {})) {
|
||||
lines.push(`dashcaddy_requests_by_status{status="${status}"} ${count}`);
|
||||
}
|
||||
|
||||
for (const [method, count] of Object.entries(this.requests.byMethod || {})) {
|
||||
lines.push(`dashcaddy_requests_by_method{method="${method}"} ${count}`);
|
||||
}
|
||||
|
||||
lines.push('# HELP dashcaddy_errors_total Total errors');
|
||||
lines.push('# TYPE dashcaddy_errors_total counter');
|
||||
lines.push(`dashcaddy_errors_total ${this.errors.total}`);
|
||||
|
||||
lines.push('# HELP dashcaddy_containers_deployed Total containers deployed');
|
||||
lines.push('# TYPE dashcaddy_containers_deployed counter');
|
||||
lines.push(`dashcaddy_containers_deployed ${this.business.containersDeployed}`);
|
||||
|
||||
lines.push('# HELP dashcaddy_process_memory_heap_used_bytes Heap memory used');
|
||||
lines.push('# TYPE dashcaddy_process_memory_heap_used_bytes gauge');
|
||||
lines.push(`dashcaddy_process_memory_heap_used_bytes ${mem.heapUsed}`);
|
||||
|
||||
lines.push('# HELP dashcaddy_process_memory_heap_total_bytes Heap memory allocated');
|
||||
lines.push('# TYPE dashcaddy_process_memory_heap_total_bytes gauge');
|
||||
lines.push(`dashcaddy_process_memory_heap_total_bytes ${mem.heapTotal}`);
|
||||
|
||||
lines.push('# HELP dashcaddy_business_metric Business metrics');
|
||||
lines.push('# TYPE dashcaddy_business_metric counter');
|
||||
for (const [key, val] of Object.entries(this.business)) {
|
||||
lines.push(`dashcaddy_business_metric{metric="${key}"} ${val}`);
|
||||
}
|
||||
|
||||
return lines.join('\n') + '\n';
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = new Metrics();
|
||||
|
||||
@@ -252,32 +252,49 @@ class WorkflowEngine extends EventEmitter {
|
||||
*/
|
||||
async _runActions(actions, triggerData = {}) {
|
||||
const results = [];
|
||||
const MAX_RETRIES = 3;
|
||||
const RETRY_DELAY_MS = 2000;
|
||||
|
||||
for (let i = 0; i < actions.length; i++) {
|
||||
const action = actions[i];
|
||||
const previousResult = i > 0 ? results[i - 1] : null;
|
||||
// notify-on-failure needs to see the previous action's outcome to decide
|
||||
// whether to fire. Passing the full results array in the trigger data lets
|
||||
// executeAction do that lookup without changing the action shape.
|
||||
// Also surface failingServices (set by healthCheckService on throw) so
|
||||
// template variables like {{failingServices}} can interpolate.
|
||||
const actionContext = {
|
||||
...triggerData,
|
||||
previousResult,
|
||||
failingServices: previousResult && previousResult.failingServices ? previousResult.failingServices : undefined,
|
||||
};
|
||||
try {
|
||||
const result = await this.executeAction(action, actionContext);
|
||||
|
||||
// DC-093: Retry with exponential backoff for transient failures
|
||||
let lastError = null;
|
||||
let result = null;
|
||||
let succeeded = false;
|
||||
|
||||
for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) {
|
||||
try {
|
||||
result = await this.executeAction(action, actionContext);
|
||||
succeeded = true;
|
||||
break;
|
||||
} catch (error) {
|
||||
lastError = error;
|
||||
if (attempt < MAX_RETRIES) {
|
||||
const delay = RETRY_DELAY_MS * Math.pow(2, attempt);
|
||||
log.warn('workflow', `Action "${action.type}" failed (attempt ${attempt + 1}/${MAX_RETRIES + 1}), retrying in ${delay}ms`, { error: error.message });
|
||||
await new Promise(resolve => setTimeout(resolve, delay));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (succeeded) {
|
||||
results.push({ action: action.type, success: true, result });
|
||||
} catch (error) {
|
||||
log.error('workflow', error, { actionType: action.type });
|
||||
} else {
|
||||
log.error('workflow', `Action "${action.type}" failed after ${MAX_RETRIES + 1} attempts`, { error: lastError.message });
|
||||
results.push({
|
||||
action: action.type,
|
||||
success: false,
|
||||
error: error.message,
|
||||
failingServices: error.failingServices,
|
||||
error: lastError.message,
|
||||
failingServices: lastError.failingServices,
|
||||
exhaustedRetries: MAX_RETRIES + 1,
|
||||
});
|
||||
// Continue with other actions but log failure
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -184,10 +184,10 @@ class AuditLogger {
|
||||
});
|
||||
} catch (e) {
|
||||
// Non-fatal — security store is a best-effort mirror
|
||||
console.error('[AuditLogger] Security event emit failed:', e.message);
|
||||
process.stderr.write(`[AuditLogger] Security event emit failed: ${e.message}\n`);
|
||||
}
|
||||
} catch (e) {
|
||||
console.error('[AuditLogger] Failed to write entry:', e.message);
|
||||
process.stderr.write(`[AuditLogger] Failed to write entry: ${e.message}\n`);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -199,7 +199,7 @@ class AuditLogger {
|
||||
}
|
||||
return entries.slice(offset, offset + limit);
|
||||
} catch (e) {
|
||||
console.error('[AuditLogger] Failed to read:', e.message);
|
||||
process.stderr.write(`[AuditLogger] Failed to read: ${e.message}\n`);
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
@@ -216,14 +216,14 @@ function csrfValidationMiddleware(req, res, next) {
|
||||
|
||||
// Validate both values exist
|
||||
if (!cookieNonce) {
|
||||
console.warn(`[CSRF] Missing CSRF cookie: ${method} ${req.path} from ${req.ip}`);
|
||||
process.stderr.write(`[CSRF] Missing CSRF cookie: ${method} ${req.path} from ${req.ip}\n`);
|
||||
return errorResponse(res, 403, '[DC-100] CSRF token missing', {
|
||||
message: 'CSRF cookie not found. Please refresh the page (Ctrl+Shift+R) and try again.'
|
||||
});
|
||||
}
|
||||
|
||||
if (!headerToken) {
|
||||
console.warn(`[CSRF] Missing CSRF header: ${method} ${req.path} from ${req.ip}`);
|
||||
process.stderr.write(`[CSRF] Missing CSRF header: ${method} ${req.path} from ${req.ip}\n`);
|
||||
return errorResponse(res, 403, '[DC-100] CSRF token missing', {
|
||||
message: 'CSRF token not provided in request headers. Please refresh the page (Ctrl+Shift+R) and try again.'
|
||||
});
|
||||
@@ -247,7 +247,7 @@ function csrfValidationMiddleware(req, res, next) {
|
||||
next();
|
||||
|
||||
} catch (err) {
|
||||
console.warn(`[CSRF] Invalid CSRF token: ${method} ${req.path} from ${req.ip} - ${err.message}`);
|
||||
process.stderr.write(`[CSRF] Invalid CSRF token: ${method} ${req.path} from ${req.ip} - ${err.message}\n`);
|
||||
return errorResponse(res, 403, '[DC-101] CSRF token invalid', {
|
||||
message: 'CSRF token validation failed. Please refresh the page (Ctrl+Shift+R) and try again.'
|
||||
});
|
||||
|
||||
@@ -34,7 +34,7 @@ function errorMiddleware(err, req, res, next) {
|
||||
userId: req.user?.id,
|
||||
body: req.body
|
||||
}
|
||||
).catch(e => console.error('Failed to write to error log:', e.message));
|
||||
).catch(e => process.stderr.write(`[error-handler] Failed to write to error log: ${e.message}\n`));
|
||||
|
||||
// Determine if this is an operational error (AppError) or programming error
|
||||
const isOperational = err.isOperational || err instanceof AppError;
|
||||
@@ -65,11 +65,7 @@ function errorMiddleware(err, req, res, next) {
|
||||
|
||||
// For non-operational errors, log as fatal
|
||||
if (!isOperational) {
|
||||
console.error('FATAL: Non-operational error detected', {
|
||||
error: err.message,
|
||||
stack: err.stack,
|
||||
path: req.path
|
||||
});
|
||||
process.stderr.write(`[FATAL] Non-operational error detected: ${JSON.stringify({ error: err.message, stack: err.stack, path: req.path })}\n`);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -437,6 +437,10 @@ module.exports = function configureMiddleware(app, {
|
||||
{ path: '/api/v1/config', exact: true, method: 'GET' },
|
||||
{ path: '/api/v1/services/status', exact: true, method: 'GET' },
|
||||
{ path: '/api/v1/health-checks/status', exact: true, method: 'GET' },
|
||||
// DC-075: System health endpoint for external uptime monitoring (UptimeRobot, BetterStack)
|
||||
{ path: '/api/v1/system/health', exact: true, method: 'GET' },
|
||||
// DC-097: Prometheus metrics endpoint (scraped by Prometheus, no auth)
|
||||
{ path: '/api/v1/metrics/prometheus', exact: true, method: 'GET' },
|
||||
// System Overview widget on the dashboard — needs the flattened CPU/mem
|
||||
// data without going through auth. See skill references/totp-and-system-overview-pitfalls.md §3.
|
||||
{ path: '/api/v1/monitoring/stats', exact: true, method: 'GET' },
|
||||
@@ -569,6 +573,18 @@ module.exports = function configureMiddleware(app, {
|
||||
});
|
||||
|
||||
app.use(generalLimiter);
|
||||
|
||||
// ── DC-073: Debug request logger (gated behind LOG_LEVEL=debug) ──
|
||||
if (process.env.LOG_LEVEL === 'debug') {
|
||||
app.use((req, res, next) => {
|
||||
const start = Date.now();
|
||||
res.on('finish', () => {
|
||||
const duration = Date.now() - start;
|
||||
process.stderr.write(`[req] ${req.method} ${req.path} ${res.statusCode} ${duration}ms\n`);
|
||||
});
|
||||
next();
|
||||
});
|
||||
}
|
||||
app.use('/api/v1/dns/credentials', strictLimiter);
|
||||
app.use('/api/v1/apps/deploy', strictLimiter);
|
||||
app.use('/api/v1/backup/restore', strictLimiter);
|
||||
|
||||
@@ -1,195 +0,0 @@
|
||||
/**
|
||||
* Graceful shutdown coordinator — DashCaddy
|
||||
*
|
||||
* Extracts the SIGTERM/SIGINT handler from server.js into a testable,
|
||||
* reusable module that:
|
||||
* 1. Calls server.close() to drain in-flight HTTP connections
|
||||
* 2. Stops each manager in a deterministic order
|
||||
* 3. Emits a 'shutdown' event so additional listeners can do cleanup
|
||||
* 4. Force-exits after a configurable drain timeout if connections don't drain
|
||||
* 5. Is idempotent — a second SIGTERM during shutdown does not re-run handlers
|
||||
*
|
||||
* Spec: DC-067 (production-grade backlog). Docker sends SIGTERM on stop;
|
||||
* without this coordinator, in-flight API calls drop.
|
||||
*
|
||||
* Exports:
|
||||
* - createShutdownCoordinator({ server, log, drainTimeoutMs, managers })
|
||||
* Returns an EventEmitter with: { shutdown, isShuttingDown, on, emit, ... }
|
||||
* - installSignalHandlers(coordinator, signals = ['SIGTERM', 'SIGINT'])
|
||||
* Registers the OS-level handlers. Idempotent.
|
||||
*/
|
||||
'use strict';
|
||||
|
||||
const EventEmitter = require('events');
|
||||
|
||||
const DEFAULT_DRAIN_TIMEOUT_MS = 10_000;
|
||||
|
||||
class ShutdownCoordinator extends EventEmitter {
|
||||
constructor({ server, log, drainTimeoutMs, managers }) {
|
||||
super();
|
||||
if (!server) throw new Error('createShutdownCoordinator: server is required');
|
||||
if (!log || typeof log.info !== 'function' || typeof log.warn !== 'function'
|
||||
|| typeof log.error !== 'function') {
|
||||
throw new Error('createShutdownCoordinator: log must have info(), warn(), and error() methods');
|
||||
}
|
||||
this.server = server;
|
||||
this.log = log;
|
||||
this.drainTimeoutMs = Number.isFinite(drainTimeoutMs) && drainTimeoutMs > 0
|
||||
? drainTimeoutMs
|
||||
: DEFAULT_DRAIN_TIMEOUT_MS;
|
||||
this.managers = Array.isArray(managers) ? managers : [];
|
||||
this._shuttingDown = false;
|
||||
this._forceTimer = null;
|
||||
}
|
||||
|
||||
isShuttingDown() {
|
||||
return this._shuttingDown;
|
||||
}
|
||||
|
||||
async _stopManager(m) {
|
||||
try {
|
||||
await m.stop();
|
||||
this.log.info('shutdown', `manager stopped: ${m.name}`);
|
||||
} catch (err) {
|
||||
this.log.warn('shutdown', `manager stop failed: ${m.name}`, { error: err.message });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Stop each manager sequentially in declaration order. Each manager's
|
||||
* stop() is awaited so that a downstream manager is not stopped until
|
||||
* its upstream dependency has finished draining.
|
||||
*
|
||||
* IMPORTANT: this runs AFTER server.close() returns (see shutdown()).
|
||||
* We must wait for in-flight HTTP requests to complete before tearing
|
||||
* down the services that serve them — otherwise those requests fail
|
||||
* mid-drain with "service not found" / "monitor not running" errors.
|
||||
*/
|
||||
async _stopManagersInOrder() {
|
||||
for (const m of this.managers) {
|
||||
await this._stopManager(m);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Emit an event but swallow listener exceptions so one bad listener
|
||||
* can't abort the shutdown sequence. Logs each failure with the
|
||||
* listener's name (set via `listener.name`) if available.
|
||||
*/
|
||||
_safeEmit(event, ...args) {
|
||||
const listeners = this.listeners(event);
|
||||
for (const listener of listeners) {
|
||||
try {
|
||||
listener.apply(this, args);
|
||||
} catch (err) {
|
||||
const name = listener.name || '<anonymous>';
|
||||
this.log.error('shutdown', `event listener for '${event}' threw`,
|
||||
{ listener: name, error: err.message });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
shutdown(signal) {
|
||||
if (this._shuttingDown) {
|
||||
this.log.info('shutdown', `${signal || 'shutdown'} received — already shutting down, ignoring`);
|
||||
return;
|
||||
}
|
||||
this._shuttingDown = true;
|
||||
this.log.info('shutdown', `${signal || 'shutdown'} received, draining (timeout=${this.drainTimeoutMs}ms)...`);
|
||||
|
||||
// Emit 'shutdown' event first so any listeners can observe the signal
|
||||
// before the drain begins. NOTE: listeners should NOT tear down their
|
||||
// state here — that happens in the 'closed' event after server.close.
|
||||
// _safeEmit swallows listener exceptions so a buggy listener can't
|
||||
// abort the entire shutdown sequence.
|
||||
this._safeEmit('shutdown', signal);
|
||||
|
||||
// Close the HTTP server FIRST. Stops accepting new connections, waits
|
||||
// for in-flight requests to complete naturally. Only AFTER close fires
|
||||
// do we tear down managers — otherwise in-flight requests could fail
|
||||
// when the services they call have already been stopped.
|
||||
let serverClosed = false;
|
||||
let managersStopped = false;
|
||||
try {
|
||||
this.server.close(async () => {
|
||||
serverClosed = true;
|
||||
this.log.info('shutdown', 'HTTP server closed cleanly');
|
||||
// Now that in-flight requests are done, stop managers in order.
|
||||
// We do NOT clear the force-exit timer yet — if a manager's stop()
|
||||
// hangs, the timer is the safety net that prevents the process
|
||||
// from living forever in a half-shut-down state.
|
||||
try {
|
||||
await this._stopManagersInOrder();
|
||||
} catch (err) {
|
||||
// _stopManager already logs per-manager failures, but a top-level
|
||||
// throw (e.g. from the for-loop itself) is still possible.
|
||||
this.log.error('shutdown', 'manager shutdown loop threw', { error: err.message });
|
||||
}
|
||||
managersStopped = true;
|
||||
// Manager drain complete — NOW we can clear the safety timer.
|
||||
if (this._forceTimer) {
|
||||
clearTimeout(this._forceTimer);
|
||||
this._forceTimer = null;
|
||||
}
|
||||
this._safeEmit('closed', signal);
|
||||
process.exit(0);
|
||||
});
|
||||
} catch (err) {
|
||||
this.log.error('shutdown', 'server.close threw', { error: err.message });
|
||||
}
|
||||
|
||||
// Force-exit safety net. Fires when EITHER:
|
||||
// (a) server.close never fires (HTTP server stuck draining), or
|
||||
// (b) server.close fired but managers hung during stop()
|
||||
// We only suppress when managersStopped === true (full drain complete).
|
||||
// serverClosed alone is NOT enough — managers could still be running.
|
||||
this._forceTimer = setTimeout(() => {
|
||||
if (managersStopped) return; // full shutdown complete
|
||||
if (!serverClosed) {
|
||||
this.log.warn('shutdown', `drain timeout (${this.drainTimeoutMs}ms) reached before HTTP server closed, force-exiting`);
|
||||
} else {
|
||||
this.log.warn('shutdown', `drain timeout (${this.drainTimeoutMs}ms) reached after HTTP close (manager hung), force-exiting`);
|
||||
}
|
||||
process.exit(0);
|
||||
}, this.drainTimeoutMs);
|
||||
if (this._forceTimer && typeof this._forceTimer.unref === 'function') {
|
||||
this._forceTimer.unref();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function createShutdownCoordinator(opts) {
|
||||
return new ShutdownCoordinator(opts);
|
||||
}
|
||||
|
||||
/**
|
||||
* Install OS-level signal handlers. Idempotent: second call for the same
|
||||
* signal does NOT register a duplicate listener. Tracks registered signals
|
||||
* on the coordinator itself so a future caller can introspect.
|
||||
*
|
||||
* @param {ShutdownCoordinator} coordinator
|
||||
* @param {string[]} signals - signals to handle (default: SIGTERM, SIGINT)
|
||||
*/
|
||||
function installSignalHandlers(coordinator, signals) {
|
||||
if (!coordinator || typeof coordinator.shutdown !== 'function') {
|
||||
throw new Error('installSignalHandlers: coordinator required');
|
||||
}
|
||||
if (!Array.isArray(coordinator._installedSignals)) {
|
||||
coordinator._installedSignals = [];
|
||||
}
|
||||
const sigs = Array.isArray(signals) && signals.length > 0
|
||||
? signals
|
||||
: ['SIGTERM', 'SIGINT'];
|
||||
for (const sig of sigs) {
|
||||
if (coordinator._installedSignals.includes(sig)) continue;
|
||||
process.on(sig, () => coordinator.shutdown(sig));
|
||||
coordinator._installedSignals.push(sig);
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
createShutdownCoordinator,
|
||||
installSignalHandlers,
|
||||
DEFAULT_DRAIN_TIMEOUT_MS,
|
||||
ShutdownCoordinator, // exported for tests
|
||||
};
|
||||
@@ -43,7 +43,7 @@ function fetchT(url, opts = {}, timeoutMs = TIMEOUTS.HTTP_DEFAULT) {
|
||||
// passes `timeout: N` here, it's almost certainly a bug — we used to silently
|
||||
// strip it, which masked the issue. Now we surface it in logs and strip it.
|
||||
if ('timeout' in opts) {
|
||||
console.warn(`[fetchT] opts.timeout=${opts.timeout} is ignored — pass timeoutMs as the 3rd arg of fetchT() instead. Called from: ${new Error().stack.split('\n').slice(2, 4).join(' <- ')}`);
|
||||
process.stderr.write(`[fetchT] opts.timeout=${opts.timeout} is ignored — pass timeoutMs as the 3rd arg of fetchT() instead. Called from: ${new Error().stack.split('\n').slice(2, 4).join(' <- ')}\n`);
|
||||
const { timeout: _timeout, ...rest } = opts;
|
||||
opts = rest;
|
||||
}
|
||||
|
||||
@@ -136,6 +136,7 @@ else
|
||||
fi
|
||||
|
||||
docker run -d --restart unless-stopped --name ${CONTAINER_NAME} \
|
||||
--memory=512m --memory-swap=1g --cpus=1.5 \
|
||||
--add-host=get.dashcaddy.net:194.233.88.206 \
|
||||
--add-host=get2.dashcaddy.net:194.233.88.206 \
|
||||
--dns ${DNS_PRIMARY} \
|
||||
|
||||
+6
-1
@@ -149,13 +149,18 @@ async function build() {
|
||||
const concatenated = parts.join(';\n');
|
||||
|
||||
// Minify with esbuild (safe to re-minify already-minified code like driver.min.js)
|
||||
const { code } = await esbuild.transform(concatenated, {
|
||||
// DC-072: sourcemap='both' emits inline + external .map for production debugging
|
||||
const { code, map } = await esbuild.transform(concatenated, {
|
||||
minify: true,
|
||||
target: 'es2020',
|
||||
sourcemap: 'both',
|
||||
});
|
||||
|
||||
const outPath = path.join(DIST, outName);
|
||||
fs.writeFileSync(outPath, code);
|
||||
if (map) {
|
||||
fs.writeFileSync(outPath + '.map', map);
|
||||
}
|
||||
|
||||
const rawSize = (Buffer.byteLength(concatenated) / 1024).toFixed(1);
|
||||
const minSize = (Buffer.byteLength(code) / 1024).toFixed(1);
|
||||
|
||||
Reference in New Issue
Block a user