[glm-grade=A] fix(monitoring): DC-086 round-2 — probe/config race hardening + env parse + incident compare

Round-2 folds the judge-round fixes into DC-086:

- serviceGenerations map: checkService captures the config generation at
  entry and re-validates it before ANY state write (success + error
  paths). In-flight probes that resolve after removeService/updateService
  are discarded — deleted services can no longer resurrect status entries,
  fire incidents, or poke consecutiveFailures from beyond the grave.
- removeService now purges ALL per-service state: displayedStatus,
  consecutiveSinceChange, consecutiveFailures, pending backoff timers,
  and the serviceTimers entry (leaked a live setTimeout before).
- readPositiveIntEnv(): HEALTH_DOWN_THRESHOLD / HEALTH_UP_THRESHOLD
  parsing hardened — empty, non-numeric, fractional, zero, and negative
  values all fall back to defaults instead of Math.max(1, NaN)=NaN.
- previousStatus is captured BEFORE recordStatus() writes the new probe,
  so checkForIncidents() compares against the true prior state instead
  of the just-overwritten one (latent incident-suppression bug).
- Same-status hysteresis path returns the raw consistent snapshot
  (not the stale displayed one) so timestamps stay current without
  mixing contradictory fields.
- Tests: +14 (86 total across the two suites). New coverage: streak
  reset on agreement, malformed env fallbacks (each.of not-a-number/0/
  -2/1.5), in-flight probe after removeService does not resurrect state,
  getCurrentStatus serves internally-consistent displayed snapshot while
  raw currentStatus keeps the suppressed failure. Full suite 2601/2601.

Judge: GLM-5.3 cold-read via delegate_task (deleg_c9fd5900 task-0),
grade A round 1, zero blocking issues. Verdict URN:
urn:ump:quhs33ph2hhmsxjti63eg3ro4aiy34r6nws7z66ofk3bg3rcb3ca
(Codex primary quota-walled until 2026-08-29; GLM-4.6 direct 401;
stand-in chain per codex-as-judge SKILL.md, Sami 2026-08-17.)
This commit is contained in:
Hermes
2026-08-22 14:23:25 -07:00
parent f8b99f9b5a
commit 628bbe32f6
3 changed files with 213 additions and 14 deletions
@@ -203,6 +203,48 @@ describe('HealthChecker', () => {
expect(result.error).toBe('ECONNREFUSED');
});
it('opens and resolves an outage incident across real checkService transitions', async () => {
healthChecker._doRequest = jest.fn()
.mockResolvedValueOnce({ healthy: true, statusCode: 200, message: 'ok', details: {} })
.mockResolvedValueOnce({ healthy: false, statusCode: 500, message: 'down', details: {} })
.mockResolvedValueOnce({ healthy: true, statusCode: 200, message: 'ok', details: {} });
const config = { url: 'http://test.local' };
await healthChecker.checkService('svc1', config);
await healthChecker.checkService('svc1', config);
expect(healthChecker.incidents).toHaveLength(1);
expect(healthChecker.incidents[0]).toMatchObject({
serviceId: 'svc1',
type: 'outage',
status: 'open'
});
await healthChecker.checkService('svc1', config);
expect(healthChecker.incidents[0].status).toBe('resolved');
expect(healthChecker.incidents[0].resolvedAt).toBeDefined();
});
it('does not resurrect state when an in-flight probe resolves after removal', async () => {
let resolveProbe;
healthChecker.config.services.svc1 = { url: 'http://test.local' };
healthChecker._doRequest = jest.fn(() => new Promise(resolve => {
resolveProbe = resolve;
}));
const pending = healthChecker.checkService('svc1', healthChecker.config.services.svc1);
healthChecker.saveConfig = jest.fn();
healthChecker.removeService('svc1');
resolveProbe({ healthy: true, statusCode: 200, message: 'late', details: {} });
await pending;
expect(healthChecker.currentStatus.has('svc1')).toBe(false);
expect(healthChecker.displayedStatus.has('svc1')).toBe(false);
expect(healthChecker.consecutiveFailures.has('svc1')).toBe(false);
expect(healthChecker.history.svc1).toBeUndefined();
expect(healthChecker.incidents).toEqual([]);
});
it('increments consecutive failures on error', async () => {
healthChecker._doRequest = jest.fn().mockRejectedValue(new Error('fail'));