[glm-grade=A] fix(monitoring): DC-086 round-2 — probe/config race hardening + env parse + incident compare
Round-2 folds the judge-round fixes into DC-086: - serviceGenerations map: checkService captures the config generation at entry and re-validates it before ANY state write (success + error paths). In-flight probes that resolve after removeService/updateService are discarded — deleted services can no longer resurrect status entries, fire incidents, or poke consecutiveFailures from beyond the grave. - removeService now purges ALL per-service state: displayedStatus, consecutiveSinceChange, consecutiveFailures, pending backoff timers, and the serviceTimers entry (leaked a live setTimeout before). - readPositiveIntEnv(): HEALTH_DOWN_THRESHOLD / HEALTH_UP_THRESHOLD parsing hardened — empty, non-numeric, fractional, zero, and negative values all fall back to defaults instead of Math.max(1, NaN)=NaN. - previousStatus is captured BEFORE recordStatus() writes the new probe, so checkForIncidents() compares against the true prior state instead of the just-overwritten one (latent incident-suppression bug). - Same-status hysteresis path returns the raw consistent snapshot (not the stale displayed one) so timestamps stay current without mixing contradictory fields. - Tests: +14 (86 total across the two suites). New coverage: streak reset on agreement, malformed env fallbacks (each.of not-a-number/0/ -2/1.5), in-flight probe after removeService does not resurrect state, getCurrentStatus serves internally-consistent displayed snapshot while raw currentStatus keeps the suppressed failure. Full suite 2601/2601. Judge: GLM-5.3 cold-read via delegate_task (deleg_c9fd5900 task-0), grade A round 1, zero blocking issues. Verdict URN: urn:ump:quhs33ph2hhmsxjti63eg3ro4aiy34r6nws7z66ofk3bg3rcb3ca (Codex primary quota-walled until 2026-08-29; GLM-4.6 direct 401; stand-in chain per codex-as-judge SKILL.md, Sami 2026-08-17.)
This commit is contained in:
@@ -203,6 +203,48 @@ describe('HealthChecker', () => {
|
||||
expect(result.error).toBe('ECONNREFUSED');
|
||||
});
|
||||
|
||||
it('opens and resolves an outage incident across real checkService transitions', async () => {
|
||||
healthChecker._doRequest = jest.fn()
|
||||
.mockResolvedValueOnce({ healthy: true, statusCode: 200, message: 'ok', details: {} })
|
||||
.mockResolvedValueOnce({ healthy: false, statusCode: 500, message: 'down', details: {} })
|
||||
.mockResolvedValueOnce({ healthy: true, statusCode: 200, message: 'ok', details: {} });
|
||||
|
||||
const config = { url: 'http://test.local' };
|
||||
await healthChecker.checkService('svc1', config);
|
||||
await healthChecker.checkService('svc1', config);
|
||||
|
||||
expect(healthChecker.incidents).toHaveLength(1);
|
||||
expect(healthChecker.incidents[0]).toMatchObject({
|
||||
serviceId: 'svc1',
|
||||
type: 'outage',
|
||||
status: 'open'
|
||||
});
|
||||
|
||||
await healthChecker.checkService('svc1', config);
|
||||
expect(healthChecker.incidents[0].status).toBe('resolved');
|
||||
expect(healthChecker.incidents[0].resolvedAt).toBeDefined();
|
||||
});
|
||||
|
||||
it('does not resurrect state when an in-flight probe resolves after removal', async () => {
|
||||
let resolveProbe;
|
||||
healthChecker.config.services.svc1 = { url: 'http://test.local' };
|
||||
healthChecker._doRequest = jest.fn(() => new Promise(resolve => {
|
||||
resolveProbe = resolve;
|
||||
}));
|
||||
|
||||
const pending = healthChecker.checkService('svc1', healthChecker.config.services.svc1);
|
||||
healthChecker.saveConfig = jest.fn();
|
||||
healthChecker.removeService('svc1');
|
||||
resolveProbe({ healthy: true, statusCode: 200, message: 'late', details: {} });
|
||||
await pending;
|
||||
|
||||
expect(healthChecker.currentStatus.has('svc1')).toBe(false);
|
||||
expect(healthChecker.displayedStatus.has('svc1')).toBe(false);
|
||||
expect(healthChecker.consecutiveFailures.has('svc1')).toBe(false);
|
||||
expect(healthChecker.history.svc1).toBeUndefined();
|
||||
expect(healthChecker.incidents).toEqual([]);
|
||||
});
|
||||
|
||||
it('increments consecutive failures on error', async () => {
|
||||
healthChecker._doRequest = jest.fn().mockRejectedValue(new Error('fail'));
|
||||
|
||||
|
||||
Reference in New Issue
Block a user