Compare commits
15
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
88f1d4a414 | ||
|
|
dd1110ef52 | ||
|
|
6732a1e1df | ||
|
|
ea96abe95a | ||
|
|
3ccf66754a | ||
|
|
c71b794ccc | ||
|
|
f2285a2550 | ||
|
|
54a1df5ac4 | ||
|
|
eb546bf468 | ||
|
|
84e051d975 | ||
|
|
628bbe32f6 | ||
|
|
f8b99f9b5a | ||
|
|
eab2b00b13 | ||
|
|
be021588c7 | ||
|
|
d8459a4a87 |
+16
@@ -401,6 +401,22 @@ Tickets DC-046 through DC-049 implement pluggable auth + email magic link. Sami
|
||||
- **result:** Shipped codex-graded A. All 49 `console.*` sites in `src/managers/update-manager.js` now route through `log.info/log.warn/log.error` from `src/utils/logging` (tag = `'update'`). Mixed-content strings extracted into structured meta payloads (`containerName`, `schedule`, `imageName`, `error.message`, `digestPrefix`, `oldImageIdPrefix`, `httpStatus`, `maxAttempts`, `attempt`, `durationMs`, `scheduledTime`, etc.) so fields are queryable instead of inlined into the message. Errors now go through `log.error(ctx, errObj)` so they land in error.log with full stack trace + context, not just stderr. 1539/1539 Jest tests pass (78/78 update-manager tests still pass). ESLint: 14 pre-existing warnings in this file unchanged, zero new warnings introduced (verified with `git stash` baseline check).
|
||||
|
||||
|
||||
### DC-086: Service-status flicker fix — asymmetric hysteresis on the badge
|
||||
- **status:** in-progress
|
||||
- **owner:** hermes
|
||||
- **details:** Dashboard service badges perpetually flip between green and red for "a few seconds at a time, never stable" (Sami's report, 2026-08-20). Root cause: `src/monitoring/health-checker.js` `recordStatus()` emits `'status-check'` on EVERY probe (every 30s), and `src/websocket/dashboard-ws.js` forwards every probe as `'status-change'` to the browser with no diff. The frontend `live-events.js` then unconditionally calls `setBadge()` — which resets the icon + pill text on every event. A single transient 5xx (Caddy reload, container CPU steal, mid-flight TLS handshake, container restart during probe) flips the badge red and the next green probe flips it back. Fix: add asymmetric hysteresis in `_computeDisplayedStatus(serviceId, rawStatus)` — going DOWN requires 2 consecutive "down" probes (default `HEALTH_DOWN_THRESHOLD=2`), going UP requires only 1 (default `HEALTH_UP_THRESHOLD=1`). History + `consecutiveFailures` still record raw probe results (operators want full fidelity for postmortems); only the dashboard broadcast is filtered. `getCurrentStatus()` now returns the displayed status so a page reload shows the same badge as the live SSE stream. Both thresholds are env-var configurable so operators can tune. New tests in `__tests__/health-checker-hysteresis.test.js` cover: first probe emits; second probe same-status does NOT re-emit; one-down-then-up keeps green; two-down flips to red; one-up after down flips back to green; `getCurrentStatus` returns displayed not raw. Effort: ~30 min. Risk: low — pure behavior filter, no schema breaks, all 63 existing health-checker tests must stay green.
|
||||
- **impact:** Operators stop seeing perpetual red/green flicker on healthy services. Real outages still get flagged (2 consecutive 30s probes = ~60s before badge flips red, which is still faster than a human notices). Background probe history is unchanged so postmortem analysis still works.
|
||||
- **prerequisite:** None.
|
||||
- **result:** _pending — ship + codex round_
|
||||
|
||||
### DC-085: Link-first invite — Discord-style "share it however you want"
|
||||
- **status:** in-progress
|
||||
- **owner:** hermes
|
||||
- **details:** Today `POST /api/v1/auth/admin/invites` defaults to sending the invite link via SMTP; if SMTP is not configured it spams the server console with `[DC-048-DEV-INVITE-LINK]` log lines. Sami wants Discord-style: the link is always returned in the response, and email is an opt-in checkbox. Operators should be free to copy the link and share it via iMessage / SMS / WhatsApp / Telegram / Signal / Discord / paste-in-email — whatever fits. (1) Flip default `sendEmail !== false` to `sendEmail === true` in `routes/auth/admin.js` so omitting the field means "no email, just hand me the link." (2) Stop logging the raw invite URL to error.log when SMTP is unconfigured — that path was only useful when there was no UI way to grab the link; now there is. (3) Add a `shareText` field to the response: `"Join my DashCaddy as <role> — <acceptUrl> — expires in Nh."` for one-tap paste into any messenger. (4) Frontend: `status/js/admin.js` `_renderInviteForm` flips the "Send email" checkbox default to **unchecked**, updates `_renderIssuedInviteBanner` to show both the raw link AND the shareText (with its own copy button + `navigator.share()` native share-sheet button where available). (5) New tests in `__tests__/admin-invites.test.js` covering: default sendEmail=false (no SMTP send attempted, no console log); `sendEmail: true` triggers SMTP send; `shareText` is present and well-formed; `acceptUrl` is always returned; expired sendEmail path doesn't leak token to logs. Effort: ~1 hr. Risk: low — pure behavior flip + UI additive change.
|
||||
- **impact:** Closes the friction between "host wants to add a friend" and "host has to configure SMTP first." Mirrors Discord/Slack/Linear invite flows where the link IS the deliverable. No new tier changes, no schema breaks.
|
||||
- **prerequisite:** DC-048 (invite store + admin route), DC-052 (Pro gate stays).
|
||||
- **result:** _pending — ship + codex round_
|
||||
|
||||
### DC-084: Remove redundant active Caddy health check from `arch.sami` site — eliminate 6 syslog spam lines/min
|
||||
- **status:** done
|
||||
- **owner:** hermes
|
||||
|
||||
@@ -0,0 +1,285 @@
|
||||
/**
|
||||
* Tests for DC-085: link-first invite (Discord-style "share it however you want").
|
||||
*
|
||||
* - default sendEmail omission = no email sent, link returned, no token in logs
|
||||
* - sendEmail:true triggers SMTP send when configured
|
||||
* - sendEmail:true + SMTP unconfigured = deliveredVia:'failed', no token leaked
|
||||
* - shareText field present and well-formed in every response
|
||||
* - acceptUrl always present (regardless of sendEmail)
|
||||
* - role + ttl validation unchanged from DC-048
|
||||
*
|
||||
* Strategy: drive the route handler directly with mock req/res, mount the admin
|
||||
* router against an isolated userStore + inviteStore + email-sender stub.
|
||||
*/
|
||||
|
||||
'use strict';
|
||||
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const os = require('os');
|
||||
const express = require('express');
|
||||
|
||||
function _tmpDir() {
|
||||
return fs.mkdtempSync(path.join(os.tmpdir(), 'dashcaddy-admin-invites-test-'));
|
||||
}
|
||||
function _cleanup(dir) {
|
||||
try { fs.rmSync(dir, { recursive: true, force: true }); } catch {}
|
||||
}
|
||||
|
||||
// Stub email-sender so we can assert "was it called?" without an SMTP server.
|
||||
// NOTE: the variable name MUST start with `mock` so Jest's hoisted `jest.mock()`
|
||||
// call is allowed to reference it (Babel guard against out-of-scope access).
|
||||
const mockEmailSender = {
|
||||
isConfigured: jest.fn(() => false),
|
||||
sendEmail: jest.fn(async () => undefined),
|
||||
};
|
||||
jest.mock('../src/auth/providers/email-sender', () => mockEmailSender);
|
||||
|
||||
describe('DC-085: link-first admin invites', () => {
|
||||
let dir, app, request;
|
||||
let logCalls; // captured { level, msg, meta } from our fake log
|
||||
|
||||
beforeEach(async () => {
|
||||
jest.clearAllMocks();
|
||||
dir = _tmpDir();
|
||||
logCalls = [];
|
||||
|
||||
// Set up email auth enable flag so userStore mounts.
|
||||
process.env.NODE_ENV = 'test';
|
||||
|
||||
const { createUserStore } = require('../src/security/user-store');
|
||||
const userStore = createUserStore({ dataDir: dir });
|
||||
|
||||
// Bootstrap the admin so we have a session-attributable user.
|
||||
await userStore.login({ email: 'admin@sami-host.me' });
|
||||
|
||||
// Build a tiny Express app with the admin router mounted, but skip the
|
||||
// global auth gate (we inject req.user directly).
|
||||
const adminRouter = require('../routes/auth/admin')({
|
||||
asyncHandler: (fn) => (req, res, next) => Promise.resolve(fn(req, res, next)).catch(next),
|
||||
errorResponse: (_res, code, msg) => ({ status: code, msg }),
|
||||
log: {
|
||||
info: (topic, msg, meta) => logCalls.push({ level: 'info', topic, msg, meta }),
|
||||
warn: (topic, msg, meta) => logCalls.push({ level: 'warn', topic, msg, meta }),
|
||||
error: (topic, msg, meta) => logCalls.push({ level: 'error', topic, msg, meta }),
|
||||
},
|
||||
session: { isSessionValid: () => true, create: () => {}, setCookie: () => {} },
|
||||
dataDir: dir,
|
||||
});
|
||||
|
||||
app = express();
|
||||
app.use(express.json());
|
||||
// Inject req.user = admin so /admin/* passes the role gate.
|
||||
app.use((req, _res, next) => {
|
||||
req.user = { id: 'admin-id', email: 'admin@sami-host.me', role: 'admin' };
|
||||
req.app.locals = req.app.locals || {};
|
||||
req.app.locals.siteConfig = {}; // no publicBaseUrl — route uses req.headers
|
||||
req.app.locals.emailConfig = null; // SMTP not configured by default
|
||||
next();
|
||||
});
|
||||
app.use('/api/v1/auth', adminRouter);
|
||||
// Error handler — last in chain.
|
||||
app.use((err, _req, res, _next) => {
|
||||
const code = (err && err.statusCode) || 500;
|
||||
res.status(code).json({
|
||||
success: false,
|
||||
error: err && err.message,
|
||||
code: err && err.code,
|
||||
});
|
||||
});
|
||||
|
||||
request = require('supertest');
|
||||
});
|
||||
|
||||
afterEach(() => _cleanup(dir));
|
||||
|
||||
test('default sendEmail (omitted) returns link and does NOT send email', async () => {
|
||||
const res = await request(app)
|
||||
.post('/api/v1/auth/admin/invites')
|
||||
.send({ email: 'friend@example.com', role: 'operator' });
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body.success).toBe(true);
|
||||
expect(mockEmailSender.sendEmail).not.toHaveBeenCalled();
|
||||
expect(res.body.acceptUrl).toMatch(/\/api\/v1\/auth\/invites\/[^/]+\/accept$/);
|
||||
expect(res.body.deliveredVia).toBe('manual');
|
||||
});
|
||||
|
||||
test('default sendEmail does NOT log raw token to server log', async () => {
|
||||
const res = await request(app)
|
||||
.post('/api/v1/auth/admin/invites')
|
||||
.send({ email: 'friend@example.com', role: 'operator' });
|
||||
|
||||
const acceptUrl = res.body.acceptUrl;
|
||||
// Extract the token from the URL and verify it does NOT appear in any log call.
|
||||
const token = acceptUrl.match(/invites\/([^/]+)\/accept/)[1];
|
||||
const tokenLeaked = logCalls.some(c =>
|
||||
typeof c.msg === 'string' && c.msg.includes(token)
|
||||
);
|
||||
expect(tokenLeaked).toBe(false);
|
||||
|
||||
// Also assert no log entry mentions the URL verbatim (the old
|
||||
// `[DC-048-DEV-INVITE-LINK] url=...` spam).
|
||||
const oldSpam = logCalls.find(c =>
|
||||
typeof c.msg === 'string' && c.msg.includes('[DC-048-DEV-INVITE-LINK]')
|
||||
);
|
||||
expect(oldSpam).toBeUndefined();
|
||||
});
|
||||
|
||||
test('shareText is present and well-formed in every response', async () => {
|
||||
const res = await request(app)
|
||||
.post('/api/v1/auth/admin/invites')
|
||||
.send({ email: 'friend@example.com', role: 'operator', ttlHours: 24 });
|
||||
|
||||
expect(res.body.shareText).toBeDefined();
|
||||
expect(res.body.shareText).toContain('Join my DashCaddy');
|
||||
expect(res.body.shareText).toContain('operator');
|
||||
expect(res.body.shareText).toContain(res.body.acceptUrl);
|
||||
expect(res.body.shareText).toContain('expires in 24h');
|
||||
});
|
||||
|
||||
test('acceptUrl is always returned regardless of sendEmail', async () => {
|
||||
const r1 = await request(app)
|
||||
.post('/api/v1/auth/admin/invites')
|
||||
.send({ email: 'a@x.com', sendEmail: false });
|
||||
const r2 = await request(app)
|
||||
.post('/api/v1/auth/admin/invites')
|
||||
.send({ email: 'b@x.com' });
|
||||
expect(r1.body.acceptUrl).toBeTruthy();
|
||||
expect(r2.body.acceptUrl).toBeTruthy();
|
||||
});
|
||||
|
||||
test('sendEmail: true triggers SMTP send when configured', async () => {
|
||||
// Build a SECOND app instance where emailConfig is a real-looking object,
|
||||
// so isConfigured() returns true. The first app uses emailConfig=null.
|
||||
mockEmailSender.isConfigured.mockReturnValueOnce(true);
|
||||
mockEmailSender.sendEmail.mockResolvedValueOnce(undefined);
|
||||
const app2 = express();
|
||||
app2.use(express.json());
|
||||
app2.use((req, _res, next) => {
|
||||
req.user = { id: 'admin-id', email: 'admin@sami-host.me', role: 'admin' };
|
||||
req.app.locals = req.app.locals || {};
|
||||
req.app.locals.siteConfig = {};
|
||||
req.app.locals.emailConfig = { host: 'smtp.test', from: 'noreply@test' };
|
||||
next();
|
||||
});
|
||||
const { createUserStore } = require('../src/security/user-store');
|
||||
const userStore2 = createUserStore({ dataDir: dir });
|
||||
await userStore2.login({ email: 'admin@sami-host.me' });
|
||||
const router2 = require('../routes/auth/admin')({
|
||||
asyncHandler: (fn) => (req, res, next) => Promise.resolve(fn(req, res, next)).catch(next),
|
||||
errorResponse: (_res, code, msg) => ({ status: code, msg }),
|
||||
log: { info() {}, warn: (t, m, meta) => logCalls.push({ level: 'warn', topic: t, msg: m, meta }), error() {} },
|
||||
session: { isSessionValid: () => true, create: () => {}, setCookie: () => {} },
|
||||
dataDir: dir,
|
||||
});
|
||||
app2.use('/api/v1/auth', router2);
|
||||
|
||||
const res = await request(app2)
|
||||
.post('/api/v1/auth/admin/invites')
|
||||
.send({ email: 'friend@example.com', role: 'viewer', sendEmail: true });
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
expect(mockEmailSender.sendEmail).toHaveBeenCalledTimes(1);
|
||||
const [_cfg, to, subject, text, html] = mockEmailSender.sendEmail.mock.calls[0];
|
||||
expect(to).toBe('friend@example.com');
|
||||
expect(subject).toMatch(/invited/i);
|
||||
expect(text).toContain(res.body.acceptUrl);
|
||||
expect(html).toContain(res.body.acceptUrl);
|
||||
expect(res.body.deliveredVia).toBe('email');
|
||||
});
|
||||
|
||||
test('sendEmail: true + SMTP unconfigured returns deliveredVia:failed and does NOT leak token', async () => {
|
||||
mockEmailSender.isConfigured.mockReturnValueOnce(false);
|
||||
|
||||
const res = await request(app)
|
||||
.post('/api/v1/auth/admin/invites')
|
||||
.send({ email: 'friend@example.com', role: 'operator', sendEmail: true });
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
expect(mockEmailSender.sendEmail).not.toHaveBeenCalled();
|
||||
expect(res.body.deliveredVia).toBe('failed');
|
||||
// acceptUrl + shareText still present so the operator can share manually.
|
||||
expect(res.body.acceptUrl).toBeTruthy();
|
||||
expect(res.body.shareText).toBeTruthy();
|
||||
// Token does NOT appear in any log call.
|
||||
const token = res.body.acceptUrl.match(/invites\/([^/]+)\/accept/)[1];
|
||||
const tokenLeaked = logCalls.some(c =>
|
||||
typeof c.msg === 'string' && c.msg.includes(token)
|
||||
);
|
||||
expect(tokenLeaked).toBe(false);
|
||||
});
|
||||
|
||||
test('invalid role silently defaults to operator (DC-048 behavior preserved)', async () => {
|
||||
// DC-048: the route's `(role && VALID_ROLES.has(role)) ? role : 'operator'`
|
||||
// silently substitutes default rather than throwing. This test pins that
|
||||
// behavior so a future "strict role validation" change is a deliberate
|
||||
// decision, not a silent regression.
|
||||
const res = await request(app)
|
||||
.post('/api/v1/auth/admin/invites')
|
||||
.send({ email: 'a@x.com', role: 'superuser' });
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body.role).toBe('operator');
|
||||
expect(mockEmailSender.sendEmail).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('email validation: missing email still rejected', async () => {
|
||||
const res = await request(app)
|
||||
.post('/api/v1/auth/admin/invites')
|
||||
.send({ role: 'operator' });
|
||||
expect(res.status).toBe(400);
|
||||
expect(mockEmailSender.sendEmail).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('ttlHours: 1 still produces shareText with correct expiry wording', async () => {
|
||||
const res = await request(app)
|
||||
.post('/api/v1/auth/admin/invites')
|
||||
.send({ email: 'a@x.com', ttlHours: 1 });
|
||||
expect(res.body.shareText).toContain('expires in 1h');
|
||||
});
|
||||
|
||||
test('DC-089: SMTP-unconfigured warn log masks the invite email (no raw PII)', async () => {
|
||||
mockEmailSender.isConfigured.mockReturnValueOnce(false);
|
||||
|
||||
const res = await request(app)
|
||||
.post('/api/v1/auth/admin/invites')
|
||||
.send({ email: 'friend@example.com', role: 'operator', sendEmail: true });
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body.deliveredVia).toBe('failed');
|
||||
const warn = logCalls.find(c =>
|
||||
c.level === 'warn' && c.topic === 'auth-invite-send'
|
||||
);
|
||||
expect(warn).toBeDefined();
|
||||
// The raw address must not appear; the masked form must.
|
||||
expect(JSON.stringify(warn.meta)).not.toContain('friend@example.com');
|
||||
expect(warn.meta.email).toBe('fr****@example.com');
|
||||
});
|
||||
|
||||
test('DC-089: invite-accepted info log masks the created user email (no raw PII)', async () => {
|
||||
// Pre-authorize the email (POST /admin/users) so userStore.login doesn't
|
||||
// reject with not_authorized — bootstrap already happened in beforeEach.
|
||||
const preauth = await request(app)
|
||||
.post('/api/v1/auth/admin/users')
|
||||
.send({ email: 'newfriend@example.com' });
|
||||
expect(preauth.status).toBe(200);
|
||||
|
||||
const issue = await request(app)
|
||||
.post('/api/v1/auth/admin/invites')
|
||||
.send({ email: 'newfriend@example.com', role: 'viewer' });
|
||||
expect(issue.status).toBe(200);
|
||||
const token = issue.body.acceptUrl.match(/invites\/([^/]+)\/accept/)[1];
|
||||
|
||||
const res = await request(app)
|
||||
.post(`/api/v1/auth/invites/${token}/accept`)
|
||||
.send({});
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
const info = logCalls.find(c =>
|
||||
c.level === 'info' && c.msg === 'invite accepted, user created'
|
||||
);
|
||||
expect(info).toBeDefined();
|
||||
expect(JSON.stringify(info.meta)).not.toContain('newfriend@example.com');
|
||||
expect(info.meta.email).toBe('ne****@example.com');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,297 @@
|
||||
/**
|
||||
* Tests for DC-086: asymmetric hysteresis on the dashboard service badge.
|
||||
*
|
||||
* - First probe always emits (no prior state).
|
||||
* - Same-status probe does NOT re-emit (dedup against repeated green).
|
||||
* - One "down" then back to "up" keeps the badge green (no flicker).
|
||||
* - Two consecutive "down" probes flip the badge to red.
|
||||
* - One "up" after a down streak flips back to green (fast recovery).
|
||||
* - History retains every raw probe even when no emit happens.
|
||||
* - getCurrentStatus returns displayed status, not raw.
|
||||
*/
|
||||
|
||||
'use strict';
|
||||
|
||||
const path = require('path');
|
||||
const fs = require('fs');
|
||||
const os = require('os');
|
||||
|
||||
// Use an isolated data dir so test history doesn't pollute the real one.
|
||||
const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'dashcaddy-hyst-'));
|
||||
process.env.HEALTH_DATA_DIR = tmpDir;
|
||||
process.env.HEALTH_CONFIG_FILE = path.join(tmpDir, 'health-config.json');
|
||||
process.env.HEALTH_HISTORY_FILE = path.join(tmpDir, 'health-history.json');
|
||||
|
||||
// Module exports a singleton instance, not a class — see module.exports in
|
||||
// src/monitoring/health-checker.js. The test creates fresh state by replacing
|
||||
// the relevant maps on the singleton in beforeEach.
|
||||
const healthCheckerSingleton = require('../src/monitoring/health-checker');
|
||||
const originalDownThreshold = process.env.HEALTH_DOWN_THRESHOLD;
|
||||
const originalUpThreshold = process.env.HEALTH_UP_THRESHOLD;
|
||||
|
||||
function restoreEnv(name, value) {
|
||||
if (value === undefined) delete process.env[name];
|
||||
else process.env[name] = value;
|
||||
}
|
||||
|
||||
function makeUp(serviceId = 'svc1') {
|
||||
return {
|
||||
serviceId,
|
||||
timestamp: new Date().toISOString(),
|
||||
status: 'up',
|
||||
responseTime: 50,
|
||||
statusCode: 200,
|
||||
message: 'Service is healthy',
|
||||
details: { headers: {}, bodyLength: 12 }
|
||||
};
|
||||
}
|
||||
|
||||
function makeDown(serviceId = 'svc1') {
|
||||
return {
|
||||
serviceId,
|
||||
timestamp: new Date().toISOString(),
|
||||
status: 'down',
|
||||
responseTime: 50,
|
||||
statusCode: 500,
|
||||
message: 'fail',
|
||||
details: { headers: {}, bodyLength: 0 }
|
||||
};
|
||||
}
|
||||
|
||||
describe('DC-086: hysteresis on the dashboard badge', () => {
|
||||
let hc;
|
||||
let emitSpy;
|
||||
|
||||
beforeEach(() => {
|
||||
// Reset the singleton's per-test state so each case starts clean.
|
||||
healthCheckerSingleton.displayedStatus = new Map();
|
||||
healthCheckerSingleton.consecutiveSinceChange = new Map();
|
||||
healthCheckerSingleton.currentStatus = new Map();
|
||||
healthCheckerSingleton.history = {};
|
||||
healthCheckerSingleton.removeAllListeners('status-check');
|
||||
emitSpy = jest.fn();
|
||||
healthCheckerSingleton.on('status-check', emitSpy);
|
||||
hc = healthCheckerSingleton;
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
restoreEnv('HEALTH_DOWN_THRESHOLD', originalDownThreshold);
|
||||
restoreEnv('HEALTH_UP_THRESHOLD', originalUpThreshold);
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
fs.rmSync(tmpDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
test('first probe (no prior state) emits', () => {
|
||||
hc.recordStatus('svc1', makeUp());
|
||||
expect(emitSpy).toHaveBeenCalledTimes(1);
|
||||
expect(emitSpy.mock.calls[0][0].status).toBe('up');
|
||||
});
|
||||
|
||||
test('second probe with same status does NOT re-emit', () => {
|
||||
hc.recordStatus('svc1', makeUp());
|
||||
hc.recordStatus('svc1', makeUp());
|
||||
expect(emitSpy).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
test('one "down" then "up" keeps the badge green (the flicker bug)', () => {
|
||||
hc.recordStatus('svc1', makeUp()); // baseline: green, emit 1
|
||||
hc.recordStatus('svc1', makeDown()); // one blip — keep green, no emit
|
||||
hc.recordStatus('svc1', makeUp()); // recovered — still green, no emit
|
||||
expect(emitSpy).toHaveBeenCalledTimes(1);
|
||||
expect(hc.displayedStatus.get('svc1').status).toBe('up');
|
||||
});
|
||||
|
||||
test('up, down, up, down, down resets the first streak before flipping', () => {
|
||||
hc.recordStatus('svc1', makeUp());
|
||||
hc.recordStatus('svc1', makeDown());
|
||||
hc.recordStatus('svc1', makeUp());
|
||||
hc.recordStatus('svc1', makeDown());
|
||||
expect(hc.displayedStatus.get('svc1').status).toBe('up');
|
||||
expect(emitSpy).toHaveBeenCalledTimes(1);
|
||||
|
||||
hc.recordStatus('svc1', makeDown());
|
||||
expect(hc.displayedStatus.get('svc1').status).toBe('down');
|
||||
expect(emitSpy).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
test('two consecutive "down" probes flip the badge to red', () => {
|
||||
hc.recordStatus('svc1', makeUp()); // baseline: green
|
||||
hc.recordStatus('svc1', makeDown()); // blip #1 — keep green (counter=1)
|
||||
hc.recordStatus('svc1', makeDown()); // blip #2 — flip red (counter=2 >= DOWN_THRESHOLD)
|
||||
expect(emitSpy).toHaveBeenCalledTimes(2);
|
||||
expect(emitSpy.mock.calls[1][0].status).toBe('down');
|
||||
expect(hc.displayedStatus.get('svc1').status).toBe('down');
|
||||
});
|
||||
|
||||
test('one "up" after a down streak flips back to green (fast recovery)', () => {
|
||||
hc.recordStatus('svc1', makeUp());
|
||||
hc.recordStatus('svc1', makeDown());
|
||||
hc.recordStatus('svc1', makeDown()); // now red
|
||||
expect(hc.displayedStatus.get('svc1').status).toBe('down');
|
||||
|
||||
hc.recordStatus('svc1', makeUp()); // first green — flip back
|
||||
expect(emitSpy).toHaveBeenCalledTimes(3);
|
||||
expect(emitSpy.mock.calls[2][0].status).toBe('up');
|
||||
expect(hc.displayedStatus.get('svc1').status).toBe('up');
|
||||
});
|
||||
|
||||
test('history retains every raw probe even when no emit happens', () => {
|
||||
hc.recordStatus('svc1', makeUp());
|
||||
hc.recordStatus('svc1', makeDown()); // blip, no emit
|
||||
hc.recordStatus('svc1', makeUp()); // recovery, no emit
|
||||
expect(hc.history['svc1'].length).toBe(3);
|
||||
expect(hc.history['svc1'][0].status).toBe('up');
|
||||
expect(hc.history['svc1'][1].status).toBe('down');
|
||||
expect(hc.history['svc1'][2].status).toBe('up');
|
||||
});
|
||||
|
||||
test('getCurrentStatus returns the displayed status, not the raw probe', () => {
|
||||
const displayedUp = makeUp();
|
||||
displayedUp.timestamp = '2026-08-22T09:59:00.000Z';
|
||||
displayedUp.statusCode = 200;
|
||||
displayedUp.message = 'healthy';
|
||||
displayedUp.details = { source: 'accepted-up' };
|
||||
hc.recordStatus('svc1', displayedUp);
|
||||
const latestRaw = makeDown();
|
||||
latestRaw.timestamp = '2026-08-22T10:00:00.000Z';
|
||||
latestRaw.responseTime = 987;
|
||||
latestRaw.statusCode = 500;
|
||||
latestRaw.message = 'failed probe';
|
||||
latestRaw.error = 'upstream failure';
|
||||
latestRaw.details = { source: 'suppressed-down' };
|
||||
hc.recordStatus('svc1', latestRaw); // raw=down, displayed=up
|
||||
const out = hc.getCurrentStatus();
|
||||
expect(out['svc1'].status).toBe('up'); // shown to API consumers
|
||||
expect(out['svc1'].timestamp).toBe(displayedUp.timestamp);
|
||||
expect(out['svc1'].statusCode).toBe(200);
|
||||
expect(out['svc1'].message).toBe('healthy');
|
||||
expect(out['svc1'].error).toBeUndefined();
|
||||
expect(out['svc1'].details).toEqual({ source: 'accepted-up' });
|
||||
expect(hc.currentStatus.get('svc1')).toBe(latestRaw);
|
||||
});
|
||||
|
||||
test('a long steady-green run produces exactly ONE emit (no per-probe spam)', () => {
|
||||
for (let i = 0; i < 50; i++) hc.recordStatus('svc1', makeUp());
|
||||
expect(emitSpy).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
test('a long steady-green-then-steady-red transition: 1 emit (up), 1 emit (red)', () => {
|
||||
for (let i = 0; i < 10; i++) hc.recordStatus('svc1', makeUp());
|
||||
expect(emitSpy).toHaveBeenCalledTimes(1);
|
||||
hc.recordStatus('svc1', makeDown());
|
||||
hc.recordStatus('svc1', makeDown()); // flips to red
|
||||
expect(emitSpy).toHaveBeenCalledTimes(2);
|
||||
for (let i = 0; i < 10; i++) hc.recordStatus('svc1', makeDown());
|
||||
expect(emitSpy).toHaveBeenCalledTimes(2); // no further broadcasts
|
||||
});
|
||||
|
||||
test('DOWN_THRESHOLD env var is honored', () => {
|
||||
process.env.HEALTH_DOWN_THRESHOLD = '3';
|
||||
jest.resetModules();
|
||||
const HC2Module = require('../src/monitoring/health-checker');
|
||||
// Module is a singleton with DOWN_THRESHOLD captured at module load —
|
||||
// resetModules gives us a fresh module-level instance with the new env.
|
||||
const hc2 = HC2Module;
|
||||
hc2.displayedStatus = new Map();
|
||||
hc2.consecutiveSinceChange = new Map();
|
||||
hc2.currentStatus = new Map();
|
||||
hc2.history = {};
|
||||
hc2.removeAllListeners('status-check');
|
||||
const spy = jest.fn();
|
||||
hc2.on('status-check', spy);
|
||||
hc2.recordStatus('svc1', makeUp());
|
||||
hc2.recordStatus('svc1', makeDown()); // 1
|
||||
hc2.recordStatus('svc1', makeDown()); // 2 — still green (need 3)
|
||||
expect(spy).toHaveBeenCalledTimes(1);
|
||||
expect(hc2.displayedStatus.get('svc1').status).toBe('up');
|
||||
hc2.recordStatus('svc1', makeDown()); // 3 — flip
|
||||
expect(spy).toHaveBeenCalledTimes(2);
|
||||
expect(hc2.displayedStatus.get('svc1').status).toBe('down');
|
||||
});
|
||||
|
||||
test.each(['not-a-number', '0', '-2', '1.5'])('malformed DOWN_THRESHOLD %s falls back to 2', value => {
|
||||
process.env.HEALTH_DOWN_THRESHOLD = value;
|
||||
jest.resetModules();
|
||||
const hc2 = require('../src/monitoring/health-checker');
|
||||
hc2.displayedStatus = new Map();
|
||||
hc2.consecutiveSinceChange = new Map();
|
||||
hc2.currentStatus = new Map();
|
||||
hc2.history = {};
|
||||
hc2.removeAllListeners('status-check');
|
||||
const spy = jest.fn();
|
||||
hc2.on('status-check', spy);
|
||||
hc2.recordStatus('svc1', makeUp());
|
||||
hc2.recordStatus('svc1', makeDown());
|
||||
expect(spy).toHaveBeenCalledTimes(1);
|
||||
hc2.recordStatus('svc1', makeDown());
|
||||
expect(spy).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
test('UP_THRESHOLD env var greater than 1 is honored', () => {
|
||||
process.env.HEALTH_UP_THRESHOLD = '2';
|
||||
jest.resetModules();
|
||||
const hc2 = require('../src/monitoring/health-checker');
|
||||
hc2.displayedStatus = new Map();
|
||||
hc2.consecutiveSinceChange = new Map();
|
||||
hc2.currentStatus = new Map();
|
||||
hc2.history = {};
|
||||
hc2.removeAllListeners('status-check');
|
||||
const spy = jest.fn();
|
||||
hc2.on('status-check', spy);
|
||||
hc2.recordStatus('svc1', makeDown());
|
||||
hc2.recordStatus('svc1', makeUp());
|
||||
expect(spy).toHaveBeenCalledTimes(1);
|
||||
expect(hc2.displayedStatus.get('svc1').status).toBe('down');
|
||||
hc2.recordStatus('svc1', makeUp());
|
||||
expect(spy).toHaveBeenCalledTimes(2);
|
||||
expect(hc2.displayedStatus.get('svc1').status).toBe('up');
|
||||
});
|
||||
|
||||
test.each(['not-a-number', '0', '-2', '1.5'])('malformed UP_THRESHOLD %s falls back to 1', value => {
|
||||
process.env.HEALTH_UP_THRESHOLD = value;
|
||||
jest.resetModules();
|
||||
const hc2 = require('../src/monitoring/health-checker');
|
||||
hc2.displayedStatus = new Map();
|
||||
hc2.consecutiveSinceChange = new Map();
|
||||
hc2.currentStatus = new Map();
|
||||
hc2.history = {};
|
||||
hc2.removeAllListeners('status-check');
|
||||
const spy = jest.fn();
|
||||
hc2.on('status-check', spy);
|
||||
hc2.recordStatus('svc1', makeDown());
|
||||
hc2.recordStatus('svc1', makeUp());
|
||||
expect(spy).toHaveBeenCalledTimes(2);
|
||||
expect(hc2.displayedStatus.get('svc1').status).toBe('up');
|
||||
});
|
||||
|
||||
test('removeService clears hysteresis state before the same ID is re-added', () => {
|
||||
hc.config.services.svc1 = { name: 'Service 1' };
|
||||
hc.recordStatus('svc1', makeUp());
|
||||
hc.recordStatus('svc1', makeDown());
|
||||
expect(hc.displayedStatus.has('svc1')).toBe(true);
|
||||
expect(hc.consecutiveSinceChange.get('svc1')).toBe(1);
|
||||
hc.consecutiveFailures.set('svc1', 3);
|
||||
const timer = setTimeout(() => {}, 60_000);
|
||||
hc.serviceTimers.set('svc1', timer);
|
||||
|
||||
hc.saveConfig = jest.fn();
|
||||
hc.removeService('svc1');
|
||||
|
||||
expect(hc.displayedStatus.has('svc1')).toBe(false);
|
||||
expect(hc.consecutiveSinceChange.has('svc1')).toBe(false);
|
||||
expect(hc.currentStatus.has('svc1')).toBe(false);
|
||||
expect(hc.consecutiveFailures.has('svc1')).toBe(false);
|
||||
expect(hc.serviceTimers.has('svc1')).toBe(false);
|
||||
|
||||
hc.config.services.svc1 = { name: 'Service 1 re-added' };
|
||||
const emitSpyAfterReAdd = jest.fn();
|
||||
hc.on('status-check', emitSpyAfterReAdd);
|
||||
hc.recordStatus('svc1', makeDown());
|
||||
|
||||
expect(emitSpyAfterReAdd).toHaveBeenCalledTimes(1);
|
||||
expect(hc.displayedStatus.get('svc1').status).toBe('down');
|
||||
expect(hc.consecutiveSinceChange.has('svc1')).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,186 @@
|
||||
/**
|
||||
* Tests for DC-090: outage incidents follow the DISPLAYED (post-hysteresis)
|
||||
* status — the same signal that flips the dashboard badge.
|
||||
*
|
||||
* - A single raw "down" blip that hysteresis suppresses opens NO outage
|
||||
* incident (the DC-089-noted raw-transition bug).
|
||||
* - A suppressed blip does not resolve a real open outage (UP_THRESHOLD=2).
|
||||
* - DOWN_THRESHOLD consecutive downs open exactly ONE outage incident.
|
||||
* - The incident payload carries the displayed snapshot, not the raw probe.
|
||||
* - Direct callers without hysteresis state keep legacy raw semantics.
|
||||
*
|
||||
* The probe() helper replicates checkService's exact call order: capture the
|
||||
* pre-probe raw + displayed state, recordStatus (updates both maps), then
|
||||
* checkForIncidents with both previous states.
|
||||
*/
|
||||
|
||||
'use strict';
|
||||
|
||||
const path = require('path');
|
||||
const fs = require('fs');
|
||||
const os = require('os');
|
||||
|
||||
// Use an isolated data dir so test history doesn't pollute the real one.
|
||||
const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'dashcaddy-incpar-'));
|
||||
process.env.HEALTH_DATA_DIR = tmpDir;
|
||||
process.env.HEALTH_CONFIG_FILE = path.join(tmpDir, 'health-config.json');
|
||||
process.env.HEALTH_HISTORY_FILE = path.join(tmpDir, 'health-history.json');
|
||||
|
||||
// Module exports a singleton instance, not a class. Reset per-test state by
|
||||
// replacing the relevant maps on the singleton in beforeEach.
|
||||
const healthCheckerSingleton = require('../src/monitoring/health-checker');
|
||||
const originalDownThreshold = process.env.HEALTH_DOWN_THRESHOLD;
|
||||
const originalUpThreshold = process.env.HEALTH_UP_THRESHOLD;
|
||||
|
||||
function restoreEnv(name, value) {
|
||||
if (value === undefined) delete process.env[name];
|
||||
else process.env[name] = value;
|
||||
}
|
||||
|
||||
function makeUp(serviceId = 'svc1') {
|
||||
return {
|
||||
serviceId,
|
||||
timestamp: new Date().toISOString(),
|
||||
status: 'up',
|
||||
responseTime: 50,
|
||||
statusCode: 200,
|
||||
message: 'Service is healthy',
|
||||
details: { headers: {}, bodyLength: 12 }
|
||||
};
|
||||
}
|
||||
|
||||
function makeDown(serviceId = 'svc1') {
|
||||
return {
|
||||
serviceId,
|
||||
timestamp: new Date().toISOString(),
|
||||
status: 'down',
|
||||
responseTime: 50,
|
||||
statusCode: 500,
|
||||
message: 'fail',
|
||||
details: { headers: {}, bodyLength: 0 }
|
||||
};
|
||||
}
|
||||
|
||||
describe('DC-090: outage incidents follow the displayed (hysteresis) status', () => {
|
||||
let hc;
|
||||
let incidentCreatedSpy;
|
||||
let incidentResolvedSpy;
|
||||
|
||||
beforeEach(() => {
|
||||
healthCheckerSingleton.displayedStatus = new Map();
|
||||
healthCheckerSingleton.consecutiveSinceChange = new Map();
|
||||
healthCheckerSingleton.currentStatus = new Map();
|
||||
healthCheckerSingleton.history = {};
|
||||
healthCheckerSingleton.incidents = [];
|
||||
healthCheckerSingleton.removeAllListeners('incident-created');
|
||||
healthCheckerSingleton.removeAllListeners('incident-resolved');
|
||||
incidentCreatedSpy = jest.fn();
|
||||
incidentResolvedSpy = jest.fn();
|
||||
healthCheckerSingleton.on('incident-created', incidentCreatedSpy);
|
||||
healthCheckerSingleton.on('incident-resolved', incidentResolvedSpy);
|
||||
hc = healthCheckerSingleton;
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
restoreEnv('HEALTH_DOWN_THRESHOLD', originalDownThreshold);
|
||||
restoreEnv('HEALTH_UP_THRESHOLD', originalUpThreshold);
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
fs.rmSync(tmpDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
// Replicates checkService's record+incident sequence for one raw probe.
|
||||
function probe(status, config = {}) {
|
||||
const previousStatus = hc.currentStatus.get(status.serviceId);
|
||||
const previousDisplayed = hc.displayedStatus.get(status.serviceId) || null;
|
||||
hc.recordStatus(status.serviceId, status);
|
||||
hc.checkForIncidents(status.serviceId, status, config, previousStatus, previousDisplayed);
|
||||
}
|
||||
|
||||
test('a single down blip between two ups opens NO outage incident', () => {
|
||||
probe(makeUp()); // baseline: displayed up
|
||||
probe(makeDown()); // blip — hysteresis keeps displayed up
|
||||
probe(makeUp()); // recovered
|
||||
expect(hc.incidents).toHaveLength(0);
|
||||
expect(incidentCreatedSpy).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test('DOWN_THRESHOLD consecutive downs open exactly one outage incident (critical)', () => {
|
||||
probe(makeUp());
|
||||
probe(makeDown()); // counter=1, displayed still up
|
||||
probe(makeDown()); // counter=2 → displayed flips down → incident
|
||||
expect(hc.incidents).toHaveLength(1);
|
||||
const incident = hc.incidents[0];
|
||||
expect(incident.type).toBe('outage');
|
||||
expect(incident.severity).toBe('critical');
|
||||
expect(incident.status).toBe('open');
|
||||
expect(incidentCreatedSpy).toHaveBeenCalledTimes(1);
|
||||
|
||||
probe(makeDown()); // still down — no new transition, no second incident
|
||||
expect(hc.incidents).toHaveLength(1);
|
||||
expect(incident.occurrences).toBe(1); // occurrences count displayed flips, not raw probes
|
||||
expect(incidentCreatedSpy).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
test('the outage incident payload carries the displayed snapshot, not the raw blip', () => {
|
||||
probe(makeUp());
|
||||
const blip = makeDown();
|
||||
blip.statusCode = 599;
|
||||
probe(blip); // suppressed blip — must not appear in any incident
|
||||
probe(makeDown()); // flip
|
||||
expect(hc.incidents).toHaveLength(1);
|
||||
// The incident's details snapshot is the probe that FLIPPED the displayed
|
||||
// state (the second down), not the earlier suppressed blip.
|
||||
expect(hc.incidents[0].details.statusCode).not.toBe(599);
|
||||
});
|
||||
|
||||
test('a suppressed up blip does not resolve a real open outage (UP_THRESHOLD=2)', () => {
|
||||
process.env.HEALTH_UP_THRESHOLD = '2';
|
||||
jest.resetModules();
|
||||
const hc2 = require('../src/monitoring/health-checker');
|
||||
hc2.displayedStatus = new Map();
|
||||
hc2.consecutiveSinceChange = new Map();
|
||||
hc2.currentStatus = new Map();
|
||||
hc2.history = {};
|
||||
hc2.incidents = [];
|
||||
hc2.removeAllListeners('incident-created');
|
||||
hc2.removeAllListeners('incident-resolved');
|
||||
|
||||
const p2 = (status) => {
|
||||
const prevRaw = hc2.currentStatus.get(status.serviceId);
|
||||
const prevDisp = hc2.displayedStatus.get(status.serviceId) || null;
|
||||
hc2.recordStatus(status.serviceId, status);
|
||||
hc2.checkForIncidents(status.serviceId, status, {}, prevRaw, prevDisp);
|
||||
};
|
||||
|
||||
p2(makeUp());
|
||||
p2(makeDown());
|
||||
p2(makeDown()); // displayed down → outage opens
|
||||
expect(hc2.incidents).toHaveLength(1);
|
||||
expect(hc2.incidents[0].status).toBe('open');
|
||||
|
||||
p2(makeUp()); // counter=1 < UP_THRESHOLD=2 → displayed still down
|
||||
expect(hc2.displayedStatus.get('svc1').status).toBe('down');
|
||||
expect(hc2.incidents[0].status).toBe('open'); // NOT resolved by the blip
|
||||
|
||||
p2(makeUp()); // counter=2 → displayed up → incident resolves
|
||||
expect(hc2.displayedStatus.get('svc1').status).toBe('up');
|
||||
expect(hc2.incidents[0].status).toBe('resolved');
|
||||
});
|
||||
|
||||
test('legacy direct callers (no displayed state) keep raw transition semantics', () => {
|
||||
hc.currentStatus.set('svc1', { status: 'up' });
|
||||
const status = { status: 'down', timestamp: new Date().toISOString(), responseTime: 100 };
|
||||
hc.checkForIncidents('svc1', status, {}); // 4-arg call, no previousDisplayed
|
||||
expect(hc.incidents).toHaveLength(1);
|
||||
expect(hc.incidents[0].type).toBe('outage');
|
||||
});
|
||||
|
||||
test('slow-response detection still fires per-probe regardless of hysteresis', () => {
|
||||
const slowUp = makeUp();
|
||||
slowUp.responseTime = 6000;
|
||||
probe(slowUp, { slowResponseThreshold: 5000 });
|
||||
expect(hc.incidents.some(i => i.type === 'slow-response')).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -203,6 +203,55 @@ describe('HealthChecker', () => {
|
||||
expect(result.error).toBe('ECONNREFUSED');
|
||||
});
|
||||
|
||||
it('opens and resolves an outage incident across real checkService transitions', async () => {
|
||||
// DC-090: incidents follow the DISPLAYED (post-hysteresis) status.
|
||||
// DOWN_THRESHOLD defaults to 2, so it takes two consecutive failed
|
||||
// probes to flip displayed down and open the outage; one up probe
|
||||
// (UP_THRESHOLD=1) resolves it.
|
||||
healthChecker._doRequest = jest.fn()
|
||||
.mockResolvedValueOnce({ healthy: true, statusCode: 200, message: 'ok', details: {} })
|
||||
.mockResolvedValueOnce({ healthy: false, statusCode: 500, message: 'down', details: {} })
|
||||
.mockResolvedValueOnce({ healthy: false, statusCode: 500, message: 'down', details: {} })
|
||||
.mockResolvedValueOnce({ healthy: true, statusCode: 200, message: 'ok', details: {} });
|
||||
|
||||
const config = { url: 'http://test.local' };
|
||||
await healthChecker.checkService('svc1', config);
|
||||
await healthChecker.checkService('svc1', config);
|
||||
expect(healthChecker.incidents).toHaveLength(0); // one down alone: suppressed blip
|
||||
|
||||
await healthChecker.checkService('svc1', config); // second down flips displayed → open
|
||||
expect(healthChecker.incidents).toHaveLength(1);
|
||||
expect(healthChecker.incidents[0]).toMatchObject({
|
||||
serviceId: 'svc1',
|
||||
type: 'outage',
|
||||
status: 'open'
|
||||
});
|
||||
|
||||
await healthChecker.checkService('svc1', config); // up resolves
|
||||
expect(healthChecker.incidents[0].status).toBe('resolved');
|
||||
expect(healthChecker.incidents[0].resolvedAt).toBeDefined();
|
||||
});
|
||||
|
||||
it('does not resurrect state when an in-flight probe resolves after removal', async () => {
|
||||
let resolveProbe;
|
||||
healthChecker.config.services.svc1 = { url: 'http://test.local' };
|
||||
healthChecker._doRequest = jest.fn(() => new Promise(resolve => {
|
||||
resolveProbe = resolve;
|
||||
}));
|
||||
|
||||
const pending = healthChecker.checkService('svc1', healthChecker.config.services.svc1);
|
||||
healthChecker.saveConfig = jest.fn();
|
||||
healthChecker.removeService('svc1');
|
||||
resolveProbe({ healthy: true, statusCode: 200, message: 'late', details: {} });
|
||||
await pending;
|
||||
|
||||
expect(healthChecker.currentStatus.has('svc1')).toBe(false);
|
||||
expect(healthChecker.displayedStatus.has('svc1')).toBe(false);
|
||||
expect(healthChecker.consecutiveFailures.has('svc1')).toBe(false);
|
||||
expect(healthChecker.history.svc1).toBeUndefined();
|
||||
expect(healthChecker.incidents).toEqual([]);
|
||||
});
|
||||
|
||||
it('increments consecutive failures on error', async () => {
|
||||
healthChecker._doRequest = jest.fn().mockRejectedValue(new Error('fail'));
|
||||
|
||||
@@ -549,6 +598,113 @@ describe('HealthChecker', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('DC-088: removeService generation tombstones + incident closure', () => {
|
||||
it('does not leak a serviceGenerations entry and records a tombstone', () => {
|
||||
healthChecker.configureService('svc1', { url: 'http://test.local' });
|
||||
expect(healthChecker.serviceGenerations.has('svc1')).toBe(true);
|
||||
|
||||
healthChecker.removeService('svc1');
|
||||
|
||||
expect(healthChecker.serviceGenerations.has('svc1')).toBe(false);
|
||||
const tomb = healthChecker.removedGenerations.get('svc1');
|
||||
expect(tomb).toBeDefined();
|
||||
expect(tomb.generation).toBeGreaterThan(0);
|
||||
expect(tomb.removedAt).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it('re-added service gets a strictly higher generation (no ABA)', () => {
|
||||
healthChecker.configureService('svc1', { url: 'http://test.local' });
|
||||
const gen1 = healthChecker.serviceGenerations.get('svc1');
|
||||
|
||||
healthChecker.removeService('svc1');
|
||||
healthChecker.configureService('svc1', { url: 'http://test.local/v2' });
|
||||
|
||||
const gen2 = healthChecker.serviceGenerations.get('svc1');
|
||||
expect(gen2).toBeGreaterThan(gen1);
|
||||
expect(healthChecker.removedGenerations.has('svc1')).toBe(false);
|
||||
});
|
||||
|
||||
it('closes open incidents for the removed service as resolved', () => {
|
||||
healthChecker.saveConfig = jest.fn();
|
||||
healthChecker.incidents.push({
|
||||
id: 'incident-test-1',
|
||||
serviceId: 'svc1',
|
||||
type: 'outage',
|
||||
status: 'open',
|
||||
createdAt: new Date(Date.now() - 60_000).toISOString()
|
||||
});
|
||||
healthChecker.incidents.push({
|
||||
id: 'incident-other',
|
||||
serviceId: 'svc2',
|
||||
type: 'outage',
|
||||
status: 'open',
|
||||
createdAt: new Date(Date.now() - 60_000).toISOString()
|
||||
});
|
||||
const resolvedSpy = jest.fn();
|
||||
healthChecker.on('incident-resolved', resolvedSpy);
|
||||
|
||||
healthChecker.removeService('svc1');
|
||||
|
||||
const closed = healthChecker.incidents.find(i => i.id === 'incident-test-1');
|
||||
expect(closed.status).toBe('resolved');
|
||||
expect(closed.resolvedBy).toBe('service-removed');
|
||||
expect(closed.resolvedAt).toBeDefined();
|
||||
expect(closed.duration).toBeGreaterThan(0);
|
||||
expect(healthChecker.incidents.find(i => i.id === 'incident-other').status).toBe('open');
|
||||
expect(resolvedSpy).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('in-flight probe captured before removal is discarded via tombstone', async () => {
|
||||
let resolveProbe;
|
||||
healthChecker.config.services.svc1 = { url: 'http://test.local' };
|
||||
healthChecker._doRequest = jest.fn(() => new Promise(resolve => {
|
||||
resolveProbe = resolve;
|
||||
}));
|
||||
|
||||
const pending = healthChecker.checkService('svc1', healthChecker.config.services.svc1);
|
||||
healthChecker.saveConfig = jest.fn();
|
||||
healthChecker.removeService('svc1');
|
||||
resolveProbe({ healthy: true, statusCode: 200, message: 'late', details: {} });
|
||||
await pending;
|
||||
|
||||
expect(healthChecker.currentStatus.has('svc1')).toBe(false);
|
||||
expect(healthChecker.consecutiveFailures.has('svc1')).toBe(false);
|
||||
});
|
||||
|
||||
it('a rejected in-flight probe after removal does not re-create failure state', async () => {
|
||||
let rejectProbe;
|
||||
healthChecker.config.services.svc1 = { url: 'http://test.local' };
|
||||
healthChecker._doRequest = jest.fn(() => new Promise((resolve, reject) => {
|
||||
rejectProbe = reject;
|
||||
}));
|
||||
|
||||
const pending = healthChecker.checkService('svc1', healthChecker.config.services.svc1);
|
||||
healthChecker.saveConfig = jest.fn();
|
||||
healthChecker.removeService('svc1');
|
||||
rejectProbe(new Error('late failure'));
|
||||
await pending;
|
||||
|
||||
expect(healthChecker.consecutiveFailures.has('svc1')).toBe(false);
|
||||
expect(healthChecker.currentStatus.has('svc1')).toBe(false);
|
||||
});
|
||||
|
||||
it('sweeps expired tombstones in cleanupHistory', () => {
|
||||
healthChecker.removedGenerations.set('svc1', {
|
||||
generation: 1,
|
||||
removedAt: Date.now() - 60 * 60 * 1000 // 1h ago, TTL default 10m
|
||||
});
|
||||
healthChecker.removedGenerations.set('svc2', {
|
||||
generation: 2,
|
||||
removedAt: Date.now() // fresh
|
||||
});
|
||||
|
||||
healthChecker.cleanupHistory();
|
||||
|
||||
expect(healthChecker.removedGenerations.has('svc1')).toBe(false);
|
||||
expect(healthChecker.removedGenerations.has('svc2')).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('cleanupHistory', () => {
|
||||
it('removes entries older than retention period', () => {
|
||||
const old = new Date(Date.now() - 35 * 24 * 60 * 60 * 1000).toISOString(); // 35 days ago
|
||||
|
||||
@@ -26,6 +26,19 @@ jest.mock('dockerode', () => {
|
||||
function buildApp({ configOk = true, servicesOk = true, dockerOk = true, caddyOk = true } = {}) {
|
||||
process.env.MOCK_DOCKER_DOWN = dockerOk ? '0' : '1';
|
||||
|
||||
// DC-087 — mirror src/app.js faithfully: the caddy check goes through
|
||||
// fetchT (which injects the Origin header Caddy's enforce_origin allowlist
|
||||
// requires), and is MOCKED so the suite is hermetic — no live request to a
|
||||
// real Caddy admin on :2019. The previous raw-`fetch` mirror sent an
|
||||
// Origin-less probe to the LIVE admin whenever the full suite ran on the
|
||||
// prod host (adversarial cron every 30 min): 12 journal 403 lines per run,
|
||||
// ~700/day of `client is not allowed to access from origin ''` noise,
|
||||
// plus a false checks.caddy.ok=false in the mirrored readiness payload.
|
||||
const fetchT = jest.spyOn(require('../src/utils/http'), 'fetchT')
|
||||
.mockImplementation(async () => (caddyOk
|
||||
? { ok: true, status: 200 }
|
||||
: { ok: false, status: 403 }));
|
||||
|
||||
const app = express();
|
||||
const config = {
|
||||
CONFIG_FILE: '/tmp/dc-test-config.json',
|
||||
@@ -103,9 +116,13 @@ function buildApp({ configOk = true, servicesOk = true, dockerOk = true, caddyOk
|
||||
allOk = false;
|
||||
}
|
||||
|
||||
// DC-087 — mirror src/app.js exactly (fetchT, not raw fetch). fetchT is
|
||||
// mocked at buildApp() scope, so this stays hermetic: no live probe to a
|
||||
// real Caddy admin (the old raw-fetch mirror 403-spammed the prod journal
|
||||
// every time the adversarial cron ran the full suite on this host).
|
||||
try {
|
||||
const caddyUrl = config.CADDY_ADMIN_URL || 'http://localhost:2019';
|
||||
const response = await fetch(`${caddyUrl}/config/apps/http/servers/srv0/listen`, { signal: AbortSignal.timeout(10000) });
|
||||
const response = await fetchT(`${caddyUrl}/config/apps/http/servers/srv0/listen`, {}, 10000);
|
||||
checks.caddy = { ok: response.ok, status: response.status };
|
||||
if (!response.ok) allOk = false;
|
||||
} catch (e) {
|
||||
|
||||
@@ -33,9 +33,18 @@ jest.mock('dockerode', () => {
|
||||
|
||||
// Mirror the canonical handler block from src/app.js — if this drifts from
|
||||
// the real handler, these tests will start failing and force a sync.
|
||||
function buildApp({ configOk = true, servicesOk = true, dockerOk = true } = {}) {
|
||||
function buildApp({ configOk = true, servicesOk = true, dockerOk = true, caddyOk = true } = {}) {
|
||||
process.env.MOCK_DOCKER_DOWN = dockerOk ? '0' : '1';
|
||||
|
||||
// DC-087 — mirror src/app.js: caddy check via fetchT (Origin-injecting),
|
||||
// mocked here so the suite is hermetic. The old raw-fetch mirror probed the
|
||||
// LIVE Caddy admin on :2019 whenever the full suite ran on the prod host
|
||||
// (adversarial cron): Origin-less → 403 → 12 journal error lines per run.
|
||||
const fetchT = jest.spyOn(require('../src/utils/http'), 'fetchT')
|
||||
.mockImplementation(async () => (caddyOk
|
||||
? { ok: true, status: 200 }
|
||||
: { ok: false, status: 403 }));
|
||||
|
||||
const app = express();
|
||||
const config = {
|
||||
CONFIG_FILE: '/tmp/dc-test-config.json',
|
||||
@@ -108,8 +117,10 @@ function buildApp({ configOk = true, servicesOk = true, dockerOk = true } = {})
|
||||
allOk = false;
|
||||
}
|
||||
try {
|
||||
// DC-087 — mirror src/app.js exactly: fetchT (mocked above), not raw
|
||||
// fetch. Hermetic: no live request to a real Caddy admin.
|
||||
const caddyUrl = config.CADDY_ADMIN_URL || 'http://localhost:2019';
|
||||
const response = await fetch(`${caddyUrl}/config/apps/http/servers/srv0/listen`, { signal: AbortSignal.timeout(10000) });
|
||||
const response = await fetchT(`${caddyUrl}/config/apps/http/servers/srv0/listen`, {}, 10000);
|
||||
checks.caddy = { ok: response.ok, status: response.status };
|
||||
if (!response.ok) allOk = false;
|
||||
} catch (e) {
|
||||
|
||||
@@ -118,6 +118,67 @@ describe('Caddyfile + utils/http.js — Origin header construction (DC-051)', ()
|
||||
expect(offenders).toEqual([]);
|
||||
});
|
||||
|
||||
test('all :2019 call sites in TESTS use fetchT or a mocked fetchT (not raw fetch)', () => {
|
||||
// DC-087 — the same rule, extended into __tests__. The api-code walk above
|
||||
// skips __tests__, which let two mirrored health-handler test files keep a
|
||||
// raw await-fetch caddy probe long after src/app.js moved to fetchT. On a
|
||||
// host where the suite runs alongside a live Caddy admin (the prod box
|
||||
// runs the full jest suite every 30 min via a cron adversarial check),
|
||||
// that Origin-less raw fetch 403-spammed the Caddy journal (~700
|
||||
// client-not-allowed error lines per day) while the tests still passed —
|
||||
// checks.caddy.ok=false was silently accepted as sandbox noise. Mirrors
|
||||
// MUST call fetchT (mocked at buildApp scope for hermeticity). A raw
|
||||
// await-fetch at a Caddy-admin-URL call site in a test is an offender.
|
||||
// NOTE: keep this comment free of backticks — stripComments pairs
|
||||
// backtick spans across lines, and a stray pair shields real code from
|
||||
// the comment stripper (this test self-flagged its first draft).
|
||||
//
|
||||
// Detection is deliberately FILE-LEVEL, not call-window: the historical
|
||||
// drift kept the fetch call itself token-free (the URL came from a
|
||||
// caddyUrl variable defined on a PREVIOUS line from CADDY_ADMIN_URL),
|
||||
// so a call-window regex never fired. Any raw await-fetch in a file
|
||||
// that also references the Caddy admin anywhere is an offender.
|
||||
// Escape hatch for future tests that intentionally assert Origin-less
|
||||
// 403 behavior against their own local listener: put the marker
|
||||
// DC-087-ALLOW-RAW-FETCH in the file and it is skipped.
|
||||
const testsRoot = path.join(__dirname);
|
||||
const offenders = [];
|
||||
const skipped = [];
|
||||
function walk(dir) {
|
||||
for (const entry of fs.readdirSync(dir, { withFileTypes: true })) {
|
||||
if (entry.name === 'node_modules') continue;
|
||||
const p = path.join(dir, entry.name);
|
||||
if (entry.isDirectory()) walk(p);
|
||||
else if (entry.name.endsWith('.js')) {
|
||||
const rawText = fs.readFileSync(p, 'utf8');
|
||||
// Escape hatch (checked on RAW text so a comment marker works —
|
||||
// comments are stripped below): a file carrying the
|
||||
// DC-087-ALLOW-RAW-FETCH marker declares it intentionally
|
||||
// raw-fetches the Caddy admin (e.g. asserting Origin-less 403
|
||||
// against its own local listener). The guard file itself is
|
||||
// always scanned (never skipped) so the hatch can't be used to
|
||||
// blind this very test.
|
||||
if (p !== __filename && /DC-087-ALLOW-RAW-FETCH/.test(rawText)) {
|
||||
skipped.push(p);
|
||||
continue;
|
||||
}
|
||||
const text = stripComments(rawText);
|
||||
const hasAdminToken = /:2019|adminUrl|admin_api_url|CADDY_ADMIN/.test(text);
|
||||
const hasRawAwaitFetch = /await\s+fetch\(/.test(text);
|
||||
if (hasAdminToken && hasRawAwaitFetch) {
|
||||
offenders.push(`${p}: raw await-fetch in a file referencing the Caddy admin (mock fetchT instead; documented escape-hatch marker available for intentional 403 tests)`);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
walk(testsRoot);
|
||||
if (skipped.length) {
|
||||
// Visibility for hatch use — shows up in jest output for reviewers.
|
||||
console.info('[DC-087 guard] escape-hatch skipped:', skipped.join(', '));
|
||||
}
|
||||
expect(offenders).toEqual([]);
|
||||
});
|
||||
|
||||
test('readiness handler in src/app.js probes the exact URL the watcher needs', () => {
|
||||
const raw = fs.readFileSync(
|
||||
path.join(__dirname, '../src/app.js'),
|
||||
@@ -127,9 +188,9 @@ describe('Caddyfile + utils/http.js — Origin header construction (DC-051)', ()
|
||||
expect(raw).toMatch(/\/config\/apps\/http\/servers\/srv0\/listen/);
|
||||
// Goes through fetchT, NOT bare fetch — that's how the Origin injection
|
||||
// takes effect. Look at the 800 chars BEFORE the probe URL on the same
|
||||
// line / call site — the call must be `fetchT(...)`, not `await fetch(...)`.
|
||||
// (We look backward because the URL sits inside the call's argument list,
|
||||
// so the call site comes before the URL token.)
|
||||
// line / call site — the call must be fetchT(...), never a raw await of
|
||||
// the global fetch. (We look backward because the URL sits inside the
|
||||
// call's argument list, so the call site comes before the URL token.)
|
||||
const idx = raw.indexOf('srv0/listen');
|
||||
const around = raw.substr(Math.max(0, idx - 400), 800);
|
||||
expect(around).toMatch(/fetchT\(/);
|
||||
|
||||
@@ -29,26 +29,10 @@ const platformPaths = require('../../platform-paths');
|
||||
const { createUserStore } = require('../../src/security/user-store');
|
||||
const { createInviteStore } = require('../../src/security/invite-store');
|
||||
const emailSender = require('../../src/auth/providers/email-sender');
|
||||
const AuthProvider = require('../../src/auth/providers/base');
|
||||
const { ValidationError, NotFoundError, ForbiddenError, PaymentRequiredError } = require('../../src/utilities/errors');
|
||||
const { ok, successMessage } = require('../../src/utils/responses');
|
||||
|
||||
/**
|
||||
* Build the URL an invitee should click. Mirrors EmailMagicLinkProvider's
|
||||
* _resolvePublicUrl logic — kept duplicated (not extracted) because the two
|
||||
* callers have slightly different link paths and the duplication is smaller
|
||||
* than the abstraction would be.
|
||||
*/
|
||||
function _buildInviteUrl(req, siteConfig, token) {
|
||||
if (siteConfig && siteConfig.publicBaseUrl) {
|
||||
return siteConfig.publicBaseUrl.replace(/\/+$/, '') +
|
||||
'/api/v1/auth/invites/' + encodeURIComponent(token) + '/accept';
|
||||
}
|
||||
const proto = (req.headers && req.headers['x-forwarded-proto']) || (req.protocol || 'https');
|
||||
const host = (req.headers && (req.headers['x-forwarded-host'] || req.headers.host))
|
||||
|| (siteConfig && siteConfig.dashboardHost) || 'localhost:3001';
|
||||
return `${proto}://${host}/api/v1/auth/invites/${encodeURIComponent(token)}/accept`;
|
||||
}
|
||||
|
||||
function _requireAdmin(req, _res, next) {
|
||||
if (!req.user || req.user.role !== 'admin') {
|
||||
return next(new ForbiddenError('Admin role required'));
|
||||
@@ -240,11 +224,21 @@ module.exports = function({ asyncHandler, errorResponse, log, session, dataDir }
|
||||
});
|
||||
if (!issued.ok) throw new ValidationError(issued.reason, 'email');
|
||||
|
||||
let deliveredVia = 'none';
|
||||
const maskedEmail = email.replace(/(^.).+(@.*$)/, '$1***$2');
|
||||
if (sendEmail !== false) {
|
||||
// Best-effort send. If SMTP isn't configured, log to error.log (dev path).
|
||||
const acceptUrl = _buildInviteUrl(req, /* siteConfig */ req.app.locals && req.app.locals.siteConfig, issued.token);
|
||||
// Build the accept URL once — used both for the response and for email delivery.
|
||||
const baseUrl = (req.app.locals && req.app.locals.siteConfig && req.app.locals.siteConfig.publicBaseUrl
|
||||
? req.app.locals.siteConfig.publicBaseUrl.replace(/\/+$/, '')
|
||||
: ((req.headers['x-forwarded-proto'] || req.protocol || 'https') + '://' +
|
||||
(req.headers['x-forwarded-host'] || req.headers.host || 'localhost:3001')));
|
||||
const acceptUrl = baseUrl + '/api/v1/auth/invites/' + encodeURIComponent(issued.token) + '/accept';
|
||||
|
||||
// DC-085: link-first delivery. Default = no email, just hand the link back.
|
||||
// Operators opt INTO email by sending { sendEmail: true } (or the admin UI
|
||||
// checks the "Send email" checkbox). When SMTP is unconfigured AND the
|
||||
// operator did opt in, we surface the failure as `deliveredVia: 'failed'`
|
||||
// but NEVER leak the raw token into the server log — the link is already
|
||||
// in the response, so the operator has a UI-side fallback.
|
||||
let deliveredVia = 'manual';
|
||||
if (sendEmail === true) {
|
||||
const ttlHoursOut = Math.round(issued.ttlMs / (60 * 60 * 1000));
|
||||
const text = _buildEmailText({ acceptUrl, ttlHours: ttlHoursOut, role: issued.role });
|
||||
const html = _buildEmailHtml({ acceptUrl, ttlHours: ttlHoursOut, role: issued.role });
|
||||
@@ -254,33 +248,37 @@ module.exports = function({ asyncHandler, errorResponse, log, session, dataDir }
|
||||
await emailSender.sendEmail(smtpConfig, issued.email, 'You\'re invited to DashCaddy', text, html);
|
||||
deliveredVia = 'email';
|
||||
} else {
|
||||
// Dev fallback — log the raw link so operators can grab it.
|
||||
log.warn && log.warn('auth-invite-dev',
|
||||
'[DC-048-DEV-INVITE-LINK] email=' + issued.email +
|
||||
' role=' + issued.role + ' url=' + acceptUrl);
|
||||
deliveredVia = 'dev-console';
|
||||
// Operator asked for email but SMTP isn't configured. Surface the
|
||||
// failure cleanly; the link is still in the response so the
|
||||
// operator can share it manually. Do NOT log the raw URL — it
|
||||
// would duplicate what's already in the response and pollute the
|
||||
// server log on every unconfigured-install invite.
|
||||
log.warn && log.warn('auth-invite-send',
|
||||
'invite send skipped: SMTP not configured (operator opted in)',
|
||||
{ inviteId: issued.id, email: AuthProvider.maskEmail(issued.email) || '[unmaskable-email]' });
|
||||
deliveredVia = 'failed';
|
||||
}
|
||||
} catch (sendErr) {
|
||||
log.warn && log.warn('auth-invite-send',
|
||||
'invite send failed: ' + (sendErr.message || String(sendErr)));
|
||||
'invite send failed: ' + (sendErr.message || String(sendErr)),
|
||||
{ inviteId: issued.id });
|
||||
deliveredVia = 'failed';
|
||||
}
|
||||
} else {
|
||||
deliveredVia = 'manual';
|
||||
}
|
||||
|
||||
const maskedEmail = email.replace(/(^.).+(@.*$)/, '$1***$2');
|
||||
const ttlHoursOut = Math.round(issued.ttlMs / (60 * 60 * 1000));
|
||||
const shareText =
|
||||
'Join my DashCaddy as ' + issued.role + ' — ' + acceptUrl +
|
||||
' — expires in ' + ttlHoursOut + 'h.';
|
||||
|
||||
return ok(res, {
|
||||
id: issued.id,
|
||||
email: issued.email,
|
||||
role: issued.role,
|
||||
expiresAt: issued.expiresAt,
|
||||
// The raw token is returned ONCE so the admin UI can show/copy the
|
||||
// link. It is also embedded in the email when sendEmail !== false.
|
||||
acceptUrl: (req.app.locals && req.app.locals.siteConfig && req.app.locals.siteConfig.publicBaseUrl
|
||||
? req.app.locals.siteConfig.publicBaseUrl.replace(/\/+$/, '')
|
||||
: ((req.headers['x-forwarded-proto'] || req.protocol || 'https') + '://' +
|
||||
(req.headers['x-forwarded-host'] || req.headers.host || 'localhost:3001'))) +
|
||||
'/api/v1/auth/invites/' + encodeURIComponent(issued.token) + '/accept',
|
||||
acceptUrl,
|
||||
shareText,
|
||||
deliveredVia,
|
||||
maskedEmail,
|
||||
});
|
||||
@@ -372,7 +370,7 @@ module.exports = function({ asyncHandler, errorResponse, log, session, dataDir }
|
||||
|
||||
log.info && log.info('auth', 'invite accepted, user created', {
|
||||
userId: userResult.user.id,
|
||||
email: userResult.user.email,
|
||||
email: AuthProvider.maskEmail(userResult.user.email) || '[unmaskable-email]',
|
||||
role: userResult.user.role,
|
||||
inviteId: invite.id,
|
||||
});
|
||||
|
||||
@@ -32,6 +32,30 @@ const CHECK_INTERVAL = parseInt(process.env.HEALTH_CHECK_INTERVAL || '30000', 10
|
||||
const MAX_CHECK_INTERVAL = parseInt(process.env.HEALTH_CHECK_MAX_INTERVAL || '300000', 10); // 5 minutes max backoff
|
||||
const MAX_ENTRIES_PER_SERVICE = parseInt(process.env.HEALTH_MAX_ENTRIES || '500', 10); // Cap to prevent disk explosion
|
||||
const HISTORY_RETENTION_DAYS = parseInt(process.env.HEALTH_HISTORY_RETENTION || '30', 10);
|
||||
// DC-088: how long a removal tombstone outlives the removal itself. Only needs
|
||||
// to cover the max in-flight probe lifetime (timeout + scheduling headroom);
|
||||
// swept by cleanupHistory so removed services cannot accumulate map entries.
|
||||
const REMOVED_GENERATION_TTL_MS = parseInt(process.env.HEALTH_REMOVED_GEN_TTL || '600000', 10);
|
||||
|
||||
// DC-086: hysteresis thresholds for badge display.
|
||||
// The raw probe result can flap on a single transient blip (Caddy reload,
|
||||
// container CPU steal, network hiccup, mid-flight TLS handshake). Showing
|
||||
// every probe result as-is to the dashboard creates the "perpetual flicker"
|
||||
// UX. Asymmetric thresholds: going red is slow (don't false-alarm), going
|
||||
// green is fast (don't keep showing red after recovery).
|
||||
// - DOWN_THRESHOLD = N consecutive "down" probes before the badge flips to red
|
||||
// - UP_THRESHOLD = N consecutive "up" probes before the badge flips back to green
|
||||
// Single probe flips to green on purpose — false-positive-green is much less
|
||||
// painful than perpetual-red (operators notice red, ignore green).
|
||||
function readPositiveIntEnv(name, fallback) {
|
||||
const raw = process.env[name];
|
||||
if (raw === undefined || raw === '') return fallback;
|
||||
const value = Number(raw);
|
||||
return Number.isSafeInteger(value) && value >= 1 ? value : fallback;
|
||||
}
|
||||
|
||||
const DOWN_THRESHOLD = readPositiveIntEnv('HEALTH_DOWN_THRESHOLD', 2);
|
||||
const UP_THRESHOLD = readPositiveIntEnv('HEALTH_UP_THRESHOLD', 1);
|
||||
|
||||
class HealthChecker extends EventEmitter {
|
||||
constructor() {
|
||||
@@ -39,11 +63,27 @@ class HealthChecker extends EventEmitter {
|
||||
this.config = this.loadConfig();
|
||||
this.history = this.loadHistory();
|
||||
this.currentStatus = new Map();
|
||||
// DC-086: the status the dashboard SHOULD display (post-hysteresis).
|
||||
// Distinct from currentStatus, which is the latest raw probe result.
|
||||
this.displayedStatus = new Map();
|
||||
// DC-086: counter of consecutive healthy/unhealthy probes since the
|
||||
// last displayed-status change. Reset to 0 whenever displayed status flips.
|
||||
this.consecutiveSinceChange = new Map();
|
||||
this.incidents = [];
|
||||
this.checking = false;
|
||||
this.checkInterval = null;
|
||||
this.consecutiveFailures = new Map(); // serviceId -> failure count
|
||||
this.serviceTimers = new Map(); // serviceId -> timer for per-service backoff
|
||||
// Invalidate probe completions that race with removal/reconfiguration.
|
||||
this.serviceGenerations = new Map(); // serviceId -> configuration generation
|
||||
// DC-088: monotonically increasing sequence so generation numbers can never
|
||||
// repeat across remove -> re-add cycles (prevents ABA on the stale check).
|
||||
this.generationSeq = 0;
|
||||
// DC-088: serviceId -> { generation, removedAt } tombstones. A live entry in
|
||||
// serviceGenerations means the service is (re)configured; a tombstone with a
|
||||
// HIGHER generation than the captured one marks the capture as stale. Entry
|
||||
// is deleted when the service is removed, so the live map cannot leak.
|
||||
this.removedGenerations = new Map();
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -111,11 +151,27 @@ class HealthChecker extends EventEmitter {
|
||||
this.cleanupHistory();
|
||||
}
|
||||
|
||||
/**
|
||||
* DC-088: true when a probe's captured generation no longer matches the
|
||||
* service's current configuration state. A live serviceGenerations entry
|
||||
* must match exactly. With no live entry the service was never configured
|
||||
* in this process (disk-loaded / direct callers) — stale only if a removal
|
||||
* tombstone with a HIGHER generation exists.
|
||||
*/
|
||||
_isStaleCapture(serviceId, generation) {
|
||||
if (this.serviceGenerations.has(serviceId)) {
|
||||
return this.serviceGenerations.get(serviceId) !== generation;
|
||||
}
|
||||
const tomb = this.removedGenerations.get(serviceId);
|
||||
return Boolean(tomb && tomb.generation > generation);
|
||||
}
|
||||
|
||||
/**
|
||||
* Check a single service
|
||||
*/
|
||||
async checkService(serviceId, config) {
|
||||
const startTime = Date.now();
|
||||
const generation = this.serviceGenerations.get(serviceId) || 0;
|
||||
|
||||
try {
|
||||
const result = await this.performHealthCheck(config);
|
||||
@@ -131,6 +187,10 @@ class HealthChecker extends EventEmitter {
|
||||
details: result.details
|
||||
};
|
||||
|
||||
if (this._isStaleCapture(serviceId, generation)) {
|
||||
return status;
|
||||
}
|
||||
|
||||
// Track consecutive failures for exponential backoff
|
||||
if (result.healthy) {
|
||||
this.consecutiveFailures.delete(serviceId);
|
||||
@@ -138,16 +198,15 @@ class HealthChecker extends EventEmitter {
|
||||
this.consecutiveFailures.set(serviceId, (this.consecutiveFailures.get(serviceId) || 0) + 1);
|
||||
}
|
||||
|
||||
const previousStatus = this.currentStatus.get(serviceId);
|
||||
const previousDisplayed = this.displayedStatus.get(serviceId);
|
||||
this.recordStatus(serviceId, status);
|
||||
this.checkForIncidents(serviceId, status, config);
|
||||
this.checkForIncidents(serviceId, status, config, previousStatus, previousDisplayed);
|
||||
|
||||
return status;
|
||||
} catch (error) {
|
||||
const responseTime = Date.now() - startTime;
|
||||
|
||||
// Increment failure count for backoff
|
||||
this.consecutiveFailures.set(serviceId, (this.consecutiveFailures.get(serviceId) || 0) + 1);
|
||||
|
||||
const status = {
|
||||
serviceId,
|
||||
timestamp: new Date().toISOString(),
|
||||
@@ -156,8 +215,18 @@ class HealthChecker extends EventEmitter {
|
||||
error: error.message
|
||||
};
|
||||
|
||||
if (this._isStaleCapture(serviceId, generation)) {
|
||||
return status;
|
||||
}
|
||||
|
||||
// Increment failure count for backoff — only after the result is known
|
||||
// to be non-stale, so a removed service cannot re-create map entries.
|
||||
this.consecutiveFailures.set(serviceId, (this.consecutiveFailures.get(serviceId) || 0) + 1);
|
||||
|
||||
const previousStatus = this.currentStatus.get(serviceId);
|
||||
const previousDisplayed = this.displayedStatus.get(serviceId);
|
||||
this.recordStatus(serviceId, status);
|
||||
this.checkForIncidents(serviceId, status, config);
|
||||
this.checkForIncidents(serviceId, status, config, previousStatus, previousDisplayed);
|
||||
|
||||
return status;
|
||||
}
|
||||
@@ -273,27 +342,109 @@ class HealthChecker extends EventEmitter {
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* Compute the displayed status for a service given the latest raw probe
|
||||
* result. Applies asymmetric hysteresis:
|
||||
* - Going DOWN: requires DOWN_THRESHOLD (default 2) consecutive "down"
|
||||
* probes since the last display-state change. A single blip keeps the
|
||||
* badge green.
|
||||
* - Going UP: requires UP_THRESHOLD (default 1) consecutive "up" probes.
|
||||
* Any single "up" after a down streak flips back to green so the badge
|
||||
* doesn't linger red after the service has recovered.
|
||||
*
|
||||
* Returns the displayed status object (same shape as the raw status) so
|
||||
* recordStatus can use it both for the displayed map and as the broadcast
|
||||
* payload when the displayed status actually changes.
|
||||
*/
|
||||
_computeDisplayedStatus(serviceId, rawStatus) {
|
||||
const currentDisplayed = this.displayedStatus.get(serviceId);
|
||||
const previousStatus = currentDisplayed ? currentDisplayed.status : null;
|
||||
|
||||
// If no prior state, accept the raw probe as-is (first-check bootstrap).
|
||||
if (!previousStatus) {
|
||||
return rawStatus;
|
||||
}
|
||||
|
||||
// Probe agrees with current displayed → no change, reset the counter so
|
||||
// a brief blip doesn't accumulate against the displayed state.
|
||||
if (rawStatus.status === previousStatus) {
|
||||
this.consecutiveSinceChange.set(serviceId, 0);
|
||||
return rawStatus;
|
||||
}
|
||||
|
||||
// Probe disagrees with displayed. Bump the streak counter — this counts
|
||||
// CONSECUTIVE probes that disagree with what's shown, regardless of
|
||||
// whether the raw value itself changed between probes. That's what
|
||||
// makes "down, down" flip after threshold but "down, up, down" not flip.
|
||||
const prev = this.consecutiveSinceChange.get(serviceId) || 0;
|
||||
const next = prev + 1;
|
||||
|
||||
if (rawStatus.status === 'down') {
|
||||
// Going DOWN: need DOWN_THRESHOLD consecutive probes that disagree
|
||||
// with the displayed "up" state.
|
||||
if (previousStatus === 'up' && next < DOWN_THRESHOLD) {
|
||||
this.consecutiveSinceChange.set(serviceId, next);
|
||||
// Keep the last internally-consistent displayed snapshot. Mixing the
|
||||
// raw failure metadata with status="up" would expose contradictory
|
||||
// API data (for example statusCode=500 on an "up" service).
|
||||
return currentDisplayed;
|
||||
}
|
||||
// Threshold met (or already down) — flip to red.
|
||||
this.consecutiveSinceChange.set(serviceId, 0);
|
||||
return rawStatus;
|
||||
}
|
||||
|
||||
// rawStatus.status === 'up' (must be — the equal-to-displayed case above
|
||||
// already returned). Going UP after a down streak: need UP_THRESHOLD.
|
||||
if (previousStatus === 'down' && next < UP_THRESHOLD) {
|
||||
this.consecutiveSinceChange.set(serviceId, next);
|
||||
return currentDisplayed;
|
||||
}
|
||||
this.consecutiveSinceChange.set(serviceId, 0);
|
||||
return rawStatus;
|
||||
}
|
||||
|
||||
/**
|
||||
* Record service status
|
||||
*
|
||||
* DC-086: history + consecutiveFailures are updated for EVERY probe
|
||||
* (operators want full probe history for postmortems). The dashboard's
|
||||
* `status-check` event is only emitted when the DISPLAYED status changes,
|
||||
* so the badge stops re-rendering on every probe.
|
||||
*/
|
||||
recordStatus(serviceId, status) {
|
||||
// Update current status
|
||||
// Update current (raw) status — used by checkForIncidents and history.
|
||||
this.currentStatus.set(serviceId, status);
|
||||
|
||||
// Add to history
|
||||
// Add raw probe to history (full fidelity — operators rely on this).
|
||||
if (!this.history[serviceId]) {
|
||||
this.history[serviceId] = [];
|
||||
}
|
||||
|
||||
this.history[serviceId].push(status);
|
||||
|
||||
|
||||
// Cap entries to prevent unbounded growth (disk explosion fix)
|
||||
if (this.history[serviceId].length > MAX_ENTRIES_PER_SERVICE) {
|
||||
this.history[serviceId] = this.history[serviceId].slice(-MAX_ENTRIES_PER_SERVICE);
|
||||
}
|
||||
|
||||
// Emit status event
|
||||
this.emit('status-check', status);
|
||||
// Compute the post-hysteresis displayed status; only emit when it changes.
|
||||
// _computeDisplayedStatus compares the raw probe against the DISPLAYED
|
||||
// status (not the previous raw status), so the "consecutive since
|
||||
// change" counter doesn't depend on the order of writes here.
|
||||
const displayed = this._computeDisplayedStatus(serviceId, status);
|
||||
const previousDisplayed = this.displayedStatus.get(serviceId);
|
||||
const displayChanged =
|
||||
!previousDisplayed || previousDisplayed.status !== displayed.status;
|
||||
|
||||
this.displayedStatus.set(serviceId, displayed);
|
||||
|
||||
if (displayChanged) {
|
||||
// Emit with the displayed status so the dashboard renders the same
|
||||
// state the hysteresis just decided. The raw probe result is still
|
||||
// in `history` and `currentStatus` for anyone who wants it.
|
||||
this.emit('status-check', displayed);
|
||||
}
|
||||
|
||||
// Save history periodically
|
||||
if (Math.random() < 0.05) { // 5% chance (every ~20 checks)
|
||||
@@ -304,11 +455,27 @@ class HealthChecker extends EventEmitter {
|
||||
/**
|
||||
* Check for incidents (downtime, slow response, etc.)
|
||||
*/
|
||||
checkForIncidents(serviceId, status, config) {
|
||||
const previous = this.currentStatus.get(serviceId);
|
||||
checkForIncidents(serviceId, status, config, previous = this.currentStatus.get(serviceId), previousDisplayed = null) {
|
||||
|
||||
// Check for status change (up -> down or down -> up)
|
||||
if (previous && previous.status !== status.status) {
|
||||
// DC-090: outage incidents follow the DISPLAYED (post-hysteresis) status —
|
||||
// the same signal that flips the dashboard badge. A single raw "down"
|
||||
// blip that hysteresis suppresses must not open a critical outage
|
||||
// incident (and a suppressed blip must not resolve a real one). When the
|
||||
// caller supplies the pre-probe displayed state (checkService always
|
||||
// does), transitions are evaluated displayed-vs-displayed using the
|
||||
// post-recordStatus state in this.displayedStatus. Direct callers with
|
||||
// no hysteresis state (previousDisplayed === null) keep the legacy
|
||||
// raw-probe transition semantics.
|
||||
if (previousDisplayed) {
|
||||
const displayed = this.displayedStatus.get(serviceId);
|
||||
if (displayed && displayed.status !== previousDisplayed.status) {
|
||||
if (displayed.status === 'down') {
|
||||
this.createIncident(serviceId, 'outage', 'Service is down', displayed);
|
||||
} else if (displayed.status === 'up') {
|
||||
this.resolveIncident(serviceId, 'outage', displayed);
|
||||
}
|
||||
}
|
||||
} else if (previous && previous.status !== status.status) {
|
||||
if (status.status === 'down') {
|
||||
this.createIncident(serviceId, 'outage', 'Service is down', status);
|
||||
} else if (status.status === 'up') {
|
||||
@@ -445,19 +612,29 @@ class HealthChecker extends EventEmitter {
|
||||
}
|
||||
|
||||
/**
|
||||
* Get current status for all services
|
||||
* Get current status for all services.
|
||||
*
|
||||
* DC-086: returns the DISPLAYED status (post-hysteresis), not the latest
|
||||
* raw probe. A page reload should show the same badge state the live
|
||||
* SSE stream is currently showing — otherwise an operator who reloads
|
||||
* the page after a single blip sees red even though the hysteresis kept
|
||||
* the badge green for them.
|
||||
*/
|
||||
getCurrentStatus() {
|
||||
const result = {};
|
||||
|
||||
for (const [serviceId, status] of this.currentStatus.entries()) {
|
||||
|
||||
for (const [serviceId, rawStatus] of this.currentStatus.entries()) {
|
||||
const config = this.config.services[serviceId];
|
||||
const uptime24h = this.calculateUptime(serviceId, 24);
|
||||
const uptime7d = this.calculateUptime(serviceId, 168);
|
||||
const avgResponseTime = this.calculateAverageResponseTime(serviceId, 24);
|
||||
|
||||
|
||||
// Prefer the displayed status if we've already computed one; fall back
|
||||
// to the raw probe on the very first call (before recordStatus has run).
|
||||
const displayed = this.displayedStatus.get(serviceId) || rawStatus;
|
||||
|
||||
result[serviceId] = {
|
||||
...status,
|
||||
...displayed,
|
||||
name: config?.name || serviceId,
|
||||
uptime: {
|
||||
'24h': uptime24h,
|
||||
@@ -467,7 +644,7 @@ class HealthChecker extends EventEmitter {
|
||||
sla: config?.sla
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
@@ -530,6 +707,13 @@ class HealthChecker extends EventEmitter {
|
||||
this.config.services = {};
|
||||
}
|
||||
|
||||
// DC-088: monotonic instance-wide sequence — a re-added service can never
|
||||
// recycle a previous generation number, and any older in-flight capture is
|
||||
// invalidated by definition.
|
||||
this.generationSeq += 1;
|
||||
this.serviceGenerations.set(serviceId, this.generationSeq);
|
||||
// Re-configuration supersedes any prior removal tombstone.
|
||||
this.removedGenerations.delete(serviceId);
|
||||
this.config.services[serviceId] = {
|
||||
enabled: config.enabled !== false,
|
||||
name: config.name || serviceId,
|
||||
@@ -552,12 +736,42 @@ class HealthChecker extends EventEmitter {
|
||||
* Remove service configuration
|
||||
*/
|
||||
removeService(serviceId) {
|
||||
// DC-088: tombstone the captured generation instead of leaking an entry.
|
||||
// The live map entry is deleted; an in-flight probe captured BEFORE this
|
||||
// point sees no live entry but a higher tombstone generation, so it is
|
||||
// discarded. configureService clears the tombstone on re-add.
|
||||
this.generationSeq += 1;
|
||||
this.serviceGenerations.delete(serviceId);
|
||||
this.removedGenerations.set(serviceId, {
|
||||
generation: this.generationSeq,
|
||||
removedAt: Date.now()
|
||||
});
|
||||
if (this.config.services) {
|
||||
delete this.config.services[serviceId];
|
||||
this.saveConfig();
|
||||
}
|
||||
|
||||
// DC-088: open incidents for a removed service must not linger forever.
|
||||
// Close them through the same resolve path a recovery would, annotated so
|
||||
// history shows why (dashboard renders resolved incidents green + duration).
|
||||
for (const incident of this.incidents) {
|
||||
if (incident.serviceId === serviceId && incident.status === 'open') {
|
||||
incident.status = 'resolved';
|
||||
incident.resolvedAt = new Date().toISOString();
|
||||
incident.duration = new Date(incident.resolvedAt) - new Date(incident.createdAt);
|
||||
incident.resolvedBy = 'service-removed';
|
||||
this.emit('incident-resolved', incident);
|
||||
this.emit('log', 'info', `Incident closed by service removal: ${incident.id}`);
|
||||
}
|
||||
}
|
||||
|
||||
this.currentStatus.delete(serviceId);
|
||||
this.displayedStatus.delete(serviceId);
|
||||
this.consecutiveSinceChange.delete(serviceId);
|
||||
this.consecutiveFailures.delete(serviceId);
|
||||
const timer = this.serviceTimers.get(serviceId);
|
||||
if (timer) clearTimeout(timer);
|
||||
this.serviceTimers.delete(serviceId);
|
||||
delete this.history[serviceId];
|
||||
}
|
||||
|
||||
@@ -576,6 +790,18 @@ class HealthChecker extends EventEmitter {
|
||||
this.history[serviceId] = this.history[serviceId].slice(-MAX_ENTRIES_PER_SERVICE);
|
||||
}
|
||||
}
|
||||
|
||||
// DC-088: sweep expired removal tombstones. After the TTL no probe that
|
||||
// captured a pre-removal generation can still be in flight (timeout is
|
||||
// bounded by performHealthCheck), so the tombstone has done its job.
|
||||
if (this.removedGenerations.size > 0) {
|
||||
const now = Date.now();
|
||||
for (const [serviceId, tomb] of this.removedGenerations) {
|
||||
if (now - tomb.removedAt > REMOVED_GENERATION_TTL_MS) {
|
||||
this.removedGenerations.delete(serviceId);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
Vendored
+71
-71
File diff suppressed because one or more lines are too long
+65
-4
@@ -216,8 +216,8 @@
|
||||
_el('input', { name: 'ttlHours', type: 'number', min: '1', max: '168', value: '24', style: 'padding:6px;width:80px' }),
|
||||
));
|
||||
form.appendChild(_el('label', { style: 'display:flex;gap:4px;align-items:center;font-size:0.85rem' },
|
||||
_el('input', { name: 'sendEmail', type: 'checkbox', checked: true }),
|
||||
_el('span', { text: 'Send email' }),
|
||||
_el('input', { name: 'sendEmail', type: 'checkbox', checked: false }),
|
||||
_el('span', { text: 'Also send via email (optional)' }),
|
||||
));
|
||||
form.appendChild(_el('button', { type: 'submit', class: 'btn-sm', style: 'padding:6px 12px', text: 'Issue invite' }));
|
||||
container.appendChild(form);
|
||||
@@ -297,16 +297,77 @@
|
||||
},
|
||||
});
|
||||
banner.appendChild(copyBtn);
|
||||
if (invite.deliveredVia === 'dev-console') {
|
||||
|
||||
// DC-085: pre-formatted message for one-tap paste into iMessage / WhatsApp /
|
||||
// Telegram / SMS / Signal / Discord / paste-into-email. The operator can
|
||||
// copy this as a sentence instead of dealing with the raw URL.
|
||||
if (invite.shareText) {
|
||||
const shareBlock = _el('div', { style: 'margin-top:12px' });
|
||||
shareBlock.appendChild(_el('div', {
|
||||
style: 'font-size:0.8rem;color:#86efac;margin-bottom:4px',
|
||||
text: 'Share this message:',
|
||||
}));
|
||||
shareBlock.appendChild(_el('div', {
|
||||
style: 'padding:8px;background:#000;border-radius:4px;color:#d1fae5;white-space:pre-wrap',
|
||||
text: invite.shareText,
|
||||
}));
|
||||
const shareActions = _el('div', { style: 'margin-top:6px;display:flex;gap:6px;flex-wrap:wrap' });
|
||||
const copyTextBtn = _el('button', {
|
||||
class: 'btn-sm', style: 'padding:4px 10px',
|
||||
text: 'Copy message',
|
||||
onclick: async () => {
|
||||
try {
|
||||
await navigator.clipboard.writeText(invite.shareText);
|
||||
copyTextBtn.textContent = 'Copied!';
|
||||
setTimeout(() => { copyTextBtn.textContent = 'Copy message'; }, 2000);
|
||||
} catch (e) {
|
||||
window.errorHandler && window.errorHandler.show('Clipboard blocked: select the text manually.');
|
||||
}
|
||||
},
|
||||
});
|
||||
shareActions.appendChild(copyTextBtn);
|
||||
// Native share sheet on mobile / supported browsers. Falls back silently
|
||||
// (the copy buttons cover the same intent).
|
||||
if (typeof navigator !== 'undefined' && typeof navigator.share === 'function') {
|
||||
const nativeShareBtn = _el('button', {
|
||||
class: 'btn-sm', style: 'padding:4px 10px',
|
||||
text: 'Share via…',
|
||||
onclick: async () => {
|
||||
try {
|
||||
await navigator.share({
|
||||
title: 'DashCaddy invite',
|
||||
text: invite.shareText,
|
||||
url: invite.acceptUrl,
|
||||
});
|
||||
} catch (e) {
|
||||
// User-cancelled throws AbortError — that's fine, just stay quiet.
|
||||
if (e && e.name && e.name !== 'AbortError') {
|
||||
window.errorHandler && window.errorHandler.show('Share failed: ' + e.message);
|
||||
}
|
||||
}
|
||||
},
|
||||
});
|
||||
shareActions.appendChild(nativeShareBtn);
|
||||
}
|
||||
shareBlock.appendChild(shareActions);
|
||||
banner.appendChild(shareBlock);
|
||||
}
|
||||
|
||||
if (invite.deliveredVia === 'failed') {
|
||||
banner.appendChild(_el('p', {
|
||||
style: 'margin-top:8px;color:#fbbf24;font-size:0.8rem',
|
||||
text: 'SMTP not configured — the invite was logged to the server console (search for [DC-048-DEV-INVITE-LINK]).',
|
||||
text: 'Email could not be sent (SMTP not configured). Share the link above instead — it works the same way.',
|
||||
}));
|
||||
} else if (invite.deliveredVia === 'email') {
|
||||
banner.appendChild(_el('p', {
|
||||
style: 'margin-top:8px;color:#86efac;font-size:0.8rem',
|
||||
text: 'Email sent to ' + invite.email + '.',
|
||||
}));
|
||||
} else if (invite.deliveredVia === 'manual') {
|
||||
banner.appendChild(_el('p', {
|
||||
style: 'margin-top:8px;color:#86efac;font-size:0.8rem',
|
||||
text: 'Share the link above via text, chat, or any messenger.',
|
||||
}));
|
||||
}
|
||||
parent.appendChild(banner);
|
||||
}
|
||||
|
||||
+1
-1
@@ -1,4 +1,4 @@
|
||||
const CACHE = 'dashcaddy-shell-c25cea8485';
|
||||
const CACHE = 'dashcaddy-shell-497e1f671c';
|
||||
const PRECACHE = [
|
||||
'/',
|
||||
'/index.html',
|
||||
|
||||
Reference in New Issue
Block a user