checkForIncidents compared raw probe transitions while the dashboard badge
(DC-086) follows post-hysteresis displayed status. A single raw down blip
between two ups opened AND resolved a critical outage incident; a suppressed
up blip during a real outage resolved it early. Incidents now open/resolve
on displayed-vs-displayed transitions; previousDisplayed=null keeps legacy
raw semantics for direct callers. 6 new parity tests + legacy checkService
test moved to a 4-probe chain. Suite 2616/2616 (110).
Verdict: urn:ump:yc5rdlnmnmhch5audc5fifgbt6d7moi2stqfs5vidsbh6x6zkvgq
- serviceGenerations no longer leaks entries: removeService deletes the live
entry and records a TTL'd (10min) tombstone swept by cleanupHistory
- monotonic instance-wide generationSeq prevents generation reuse across
remove->re-add cycles (ABA) and supersedes tombstones on re-configure
- _isStaleCapture(): presence-aware stale check — live entry must match
exactly; no entry is stale only under a higher-generation tombstone
(preserves correct behavior for disk-loaded never-configured services)
- catch path increments consecutiveFailures only after the stale check, so
a late-rejected probe cannot resurrect state for a removed service
- open incidents for a removed service close via the standard resolve path
(resolvedBy=service-removed, WS/SSE incident-resolved broadcast)
- 6 regression tests; full suite 2608/2608 green
Judge: GLM-5.3 cold read (Codex stand-in), verdict B/ship, zero blockers
Round-2 folds the judge-round fixes into DC-086:
- serviceGenerations map: checkService captures the config generation at
entry and re-validates it before ANY state write (success + error
paths). In-flight probes that resolve after removeService/updateService
are discarded — deleted services can no longer resurrect status entries,
fire incidents, or poke consecutiveFailures from beyond the grave.
- removeService now purges ALL per-service state: displayedStatus,
consecutiveSinceChange, consecutiveFailures, pending backoff timers,
and the serviceTimers entry (leaked a live setTimeout before).
- readPositiveIntEnv(): HEALTH_DOWN_THRESHOLD / HEALTH_UP_THRESHOLD
parsing hardened — empty, non-numeric, fractional, zero, and negative
values all fall back to defaults instead of Math.max(1, NaN)=NaN.
- previousStatus is captured BEFORE recordStatus() writes the new probe,
so checkForIncidents() compares against the true prior state instead
of the just-overwritten one (latent incident-suppression bug).
- Same-status hysteresis path returns the raw consistent snapshot
(not the stale displayed one) so timestamps stay current without
mixing contradictory fields.
- Tests: +14 (86 total across the two suites). New coverage: streak
reset on agreement, malformed env fallbacks (each.of not-a-number/0/
-2/1.5), in-flight probe after removeService does not resurrect state,
getCurrentStatus serves internally-consistent displayed snapshot while
raw currentStatus keeps the suppressed failure. Full suite 2601/2601.
Judge: GLM-5.3 cold-read via delegate_task (deleg_c9fd5900 task-0),
grade A round 1, zero blocking issues. Verdict URN:
urn:ump:quhs33ph2hhmsxjti63eg3ro4aiy34r6nws7z66ofk3bg3rcb3ca
(Codex primary quota-walled until 2026-08-29; GLM-4.6 direct 401;
stand-in chain per codex-as-judge SKILL.md, Sami 2026-08-17.)
Dashboard badges perpetually flip green/red for a few seconds at a time,
never stable. Root cause: health-checker emitted 'status-check' on every
probe (every 30s) and dashboard-ws forwarded every one as 'status-change'
to the browser with no diff; live-events.js then unconditionally called
setBadge(). A single transient 5xx (Caddy reload, container restart, TLS
handshake blip) flipped the badge and the next green probe flipped back.
Fix: _computeDisplayedStatus applies asymmetric hysteresis — DOWN_THRESHOLD
(default 2, env-tunable HEALTH_DOWN_THRESHOLD) consecutive probes that
disagree with the displayed 'up' state flip to red; UP_THRESHOLD (default
1, HEALTH_UP_THRESHOLD) flips back to green. History and consecutiveFailures
still record every raw probe so postmortem analysis is unchanged. Only the
SSE broadcast is filtered. getCurrentStatus now returns the displayed
status so a page reload shows the same badge as the live stream.
10 new tests cover first-emit, same-status-dedup, the actual flicker bug
(one-down-then-up keeps green), two-down flips red, one-up recovers fast,
long-steady-green produces exactly one emit, and env-var tuning. All 63
existing health-checker tests still pass. Full suite: 2484/2484.
Three coordinated changes to stop every gated *.sami service flipping
red after ~20 probes:
1. health-checker.js _doRequest() now sends X-DashCaddy-HealthCheck: 1
on every outgoing probe. Caddy uses this header (combined with a
trusted source IP via the new @healthcheckProbe matcher in the
dashcaddy_auth snippet) to bypass forward_auth for local container
probes. Without the bypass, forward_auth 401's every probe, and the
authLimiter (20 req / 15 min, DC-027) caps us out within minutes.
2. evaluateHealth() default expectedStatusCodes now includes 401, 403,
and 429. Defense in depth — if a future Caddy reload drops the
bypass, 429 from the rate-limited gate no longer marks the service
as down (it just means the gate answered, which proves the service
is reachable through Caddy).
3. (start.sh — already shipped on the running container, will land
with the next release build) ca.sami now maps to 100.121.150.22
(DNS2) instead of 127.0.0.1, which is the container's own loopback
where nothing serves :443. The CA web UI lives on DNS2's Caddy.
Tests:
- evaluateHealth: 401, 403, 429 accepted by default
- _doRequest: X-DashCaddy-HealthCheck: 1 always present
- _doRequest: user-supplied headers preserved alongside marker
Bump 1.14.7 → 1.14.8.
After DC-005 refactor moved health-checker.js into src/monitoring/, the
require path was never updated. Tests in __tests__/health-checker.test.js
failed with 'Cannot find module' → 59 cascading test failures in the
health-checker suite.
Path: src/monitoring/health-checker.js → 'require(./platform-paths)'
Fix: 'require(../../platform-paths)'
Verified: 921/922 tests passing (one known async-timing flake in
logging.test.js 'writes entry to ERROR_LOG_FILE with context').
After the DC-005 module reorganization (41 files moved into src/ subdirs),
138 test suites failed because the refactor script's path-rewrite logic
missed three categories:
1. Files inside src/ doing 'require("./src/...")' — should be 'require("../...")'
2. Files in src/X/Y/ doing 'require("../../../src/...")' — should be 'require("../../...")'
3. Test files in __tests__/ with leftover 'require("../../../src/...")' paths
Root cause: the original refactor script ran before all files were moved,
so it computed relative paths against stale filesystem state.
Result:
- 30/30 test suites pass
- 879/879 tests pass (was: 18/30 suites, 614/687 tests)
Also fixed:
- routes/apps/restore.js: wrong responses import path
- routes/*/*.js: '../../src/utilities/X' → '../src/utilities/X' (depth 2 routes)
Hermes