8 reliability fixes across 8 files (AUDIT-2026-10-27.md findings
9/11/12/13/14/16/20 + calendar credential item):
- dns.js check-update/update HANGS: bare returns + empty else/catch
branches left requests hanging until client timeout; every path now
answers (502 with message); upstream JSON.parse wrapped (non-JSON ->
502)
- calendar.js ICS fetch timeout actually applies: fetchT ignores
opts.timeout (undici) — was running at 5s default, not the intended
15s; now timeoutMs third arg
- calendar.js PUT /sources: updating pass no longer nulls the stored
token (and vice versa) — existing credential record loaded, only the
provided field overwritten
- state-manager.js ATOMIC WRITES: _atomicWrite = tmp file + fs.rename
(POSIX-atomic); write()/update() no longer truncate-then-write the
live file (crash mid-write corrupted services.json/config.json with
no recovery path). Test mock extended to the tmp+rename contract
- dashboard-ws.js auth fallback FAIL-CLOSED: missing authVerifier used
to accept any non-empty session cookie (DC-061 footgun on unwired
boot paths); now rejects + loud CRITICAL log
- middleware.js tailscale gate: new tailscale.failClosed config (default
off = unchanged); when set, outages/errors deny (503) instead of
silently disabling the gate; fail-open path now logs the exact key
- sw.js: offline fallback actually works ((await match) || (await
match) — old Promise||Promise was always truthy); only 2xx cached
(5xx pages were stored and served as the app shell); install is
per-URL tolerant (one bad asset kept the stale SW alive forever)
- config/settings.js GET /config (public): corrupt config.json degrades
to {setupComplete:false, corrupt:true} instead of 500 on every load
SCOPE CORRECTION: audit finding #10 (7 reversed-arg errorResponse
sites) was a FALSE POSITIVE — those files alias message-first error();
see lesson #14. Full suite 173/173 suites, 3740/3740 tests.
Audit P0-5 + deploy_app truth (batch 5 of the 2026-10-27 sweep, AUDIT-2026-10-27.md committed here):
ROUTE ADDITIONS:
- routes/containers.js GET / — container list (dockerode listContainers);
the router had NO list route, so dashcaddy_list_containers 404'd always
- routes/dns.js GET /records — record list via active provider with a
list-records capability guard (501 + guidance for push-only providers
like RFC2136); dashcaddy_list_dns called GET /dns, never existed
HANDLER FIXES (src/mcp/mcp-core.js):
- get_service resolves from GET /services by id/name (single-service GET
never existed) with a known-ids 404
- check_health + diagnose -> /health/service/:id and /health/services
(both previously hit nonexistent routes)
- container_action remove -> DELETE /containers/:id (was /:id/remove)
- system_health/system_metrics via apiCall error envelope (metrics text
capped at 64KB)
- search_catalog applies category filter client-side (/catalog/search
has no category param — constraint was silently dropped)
- generate_caddyfile defaults authService=status when auth:true (auth
flag alone silently produced UNGATED Caddyfiles)
- deploy_app honest re-contract: registers service + generates
Caddyfile, does NOT claim image pull/container creation/deployed:true;
base domain read from /config (hardcoded owner .sami TLD removed);
caddy-code failure now surfaces as caddyfileError + FAILED message
- list_fleet description gains Premium hint
VERIFIER (audit #40): EXPECTED table 12 -> 34 rows covering EVERY tool's
(method,path); NEW Layer-3 handler-derived test drives the real
createToolHandler for all 28 tools and strictly resolves every captured
call mount-aware (a vacuous first draft of this gate was caught and
fixed pre-grade); extractRoutes regex extended to apiRouter.get inline
decls; Bearer credential-passthrough test added (DC-162 coverage gap)
NEW TESTS: dns-records.routes.test.js (5 route-level cases incl. 501
capability behavior); deploy_app partial-failure + no-.sami tests
HOST RELIABILITY (same sweep, evidenced here): jest maxWorkers 50% —
default 5-worker runs repeatedly OOM-killed this host's gateway
(dmesg 2026-10-29 00:58:40; swap was 4G/4G full). Gateway protected
separately via systemd OOMScoreAdjust=-500 drop-in + /swapfile2 (host
ops, graded B urn:ump:35dpnggtxsqrxvlqm2d2buqkbnggw3acflbccnkqpqyiiijcxx3q).
DC-171 DNS provider-parity ticket (Sami, plug-in-directly) appended to
backlog.
Full suite: 173/173 suites, 3740/3740 tests. Judge B r1
(urn:ump:jsobkg2t2omzvpinel2fufk7sspoy3vubsrth3e6lat3ygolvffq) blockers
all fixed: dns.js 501 arg order (the file aliases error() message-first
— the sweep's 'reversed-arg' finding was a false positive for its 7
sites; lesson #14), deploy_app partial-failure honesty, handler-derived
verifier.
With TOTP enabled, totpAuthMiddleware answered 401 before
jwtApiKeyAuthMiddleware ever ran: API keys and JWTs were rejected on
every TOTP-protected install (live-verified during the 2026-10-27
sweep with a freshly minted dk_ key getting 401 on /api/v1/mcp/info).
MCP-over-HTTP ('Connect your AI'), fleet endpoints and every machine
integration were broken in production.
Fix: the TOTP gate now verifies Bearer JWT then x-api-key itself
(try/catch, fall-through to the unchanged 401) and sets req.auth;
the JWT/API-key gate passes through on already-set machine auth
(no double verification). Session/public/TOTP-disabled paths
byte-identical.
New chain-level test (no req.auth injection — the gap that hid this
bug) boots the real configureMiddleware chain; VERIFIED it fails 2/5
with the fix stashed and passes 5/5 with it. Judge polish (call-count
assertion) noted as optional follow-up.
Full suite: 172/172 suites, 3731/3731 tests.
Resolves 0 conflicts (auto-merge; the branches touched disjoint regions
of src/mcp/mcp-core.js). Fixes audit P0-4 (DC-166 drift: the fix branch
was never merged, so the branch shipped 17 tools while changelog/
capabilities claimed the expanded set, and the workflows UI was broken)
and the production Workflows outage.
Incoming:
- src/mcp/mcp-core.js: 11 new tool handlers over existing routes
(calendar events/add_source, security events/summary, license_status,
capabilities, DNS add/remove record, deploy_recipe, run_workflow,
tailscale_status); DC-162 machine-only gate preserved
- routes/workflows.js: double-prefix fix — declarations are mount-
relative (/history before /:workflowId/history); the dashboard's
single-prefix calls no longer 404 (Premium feature was dead in prod)
- __tests__/mcp/mcp-core-dc166{,-integration}.test.js: tool/schema +
route-existence verifier (scoped to the 11 new tools; extension to
all 28 is the MCP truth batch, audit #40)
Merge-commit fixes (judge r1 C -> r2 A, urn:ump:2jdpii43.../5clu4udr...):
1. integration-test header over-claimed COMPLETE tool coverage — scope
narrowed to the DC-166 additions, stated explicitly
2. routes/workflows.js: shadowed top-level ok import removed (all uses
inside the factory use deps.ok) + comment documenting the trap
Verified on merged tree: TOOLS.length=28, MCP+workflows 44/44.
Judge A — urn:ump:5clu4udrbwuwxn74tqwxtukfjnrwkrbiykp6uh66ipipzxr32xsq
extractCalendarsAndEvents required a namespace prefix on <response>; Radicale
emits bare <response> (default DAV ns) so every entry was silently dropped —
REPORT 200, ok:true, 0 events, no error. Prefix is now optional in the split
and detection regexes.
Tests: 9 in the ns suite (extraction incl. mixed doc; empty multistatus;
namespace-aware fixture parse; undeclared element+attribute prefix rejection;
both quote styles; positive declared-attr case). Calendar-related suites: 52.
eslint clean.
Owner directive "Do it" (2026-10-09): expand the MCP tool set so the
"manage your entire infrastructure" claim is true of the product. 11 new
tools over EXISTING routes: calendar (events/add_source), security
(events/summary), license_status, capabilities, DNS add/remove record,
deploy_recipe, run_workflow, tailscale_status. Handlers verified against
the actual route declarations; credential TYPE passthrough and the DC-162
machine-only tools/call gate unchanged (tests included).
judge r1 (grade C, urn:ump:fyvivcvrphjm5wk227ujr2yzfcgcdvwekjmw6lvnqvjosqrjhtbq)
blocking issues — all fixed in this revision:
1. WORKFLOWS DOUBLE-PREFIX (real product bug the judge caught):
routes/workflows.js declared literal '/workflows/...' paths while
src/app.js mounts the router at /workflows — composed path was
/api/v1/workflows/workflows/..., which the SHIPPED dashboard bundle
(status/js/bundled-workflows.js calls single-prefix /api/v1/workflows/)
never hits: the workflows UI has been broken in production. Fixed by
making route paths RELATIVE to the mount (same class as the recipes
fix); run-workflow MCP path now matches. Also reordered /history
before /:workflowId/history so the literal segment is not captured as
an id (limit 100 vs 50 asserted in tests).
2. Verifier substring check replaced: NEW integration suite
(__tests__/mcp/mcp-core-dc166-integration.test.js) resolves EVERY MCP
tool call's complete mounted (method, path) against the declared route
patterns extracted from the owning route file (params matched as
patterns, comments excluded) — a double prefix or wrong method fails
the run. Plus real-composition dispatch through the actual workflows
router: POST /api/v1/workflows/:id/run reaches the run handler
(sentinel '"ran":true'), list/history handlers land correctly, and a
static guard fails any route file that re-declares its own mount name.
3. Both verbatim-command claims in the brief now name their file.
polish applied: calendar window-math test bounds tightened to exactly
~7 days (was a weak 6-hour lower bound); extra blank line before
tools/list removed; calendar_add_source schema gained the token property
the route already accepts (schema now matches the documented API).
Tests: 73 prior + 15 dc166 unit + 6 integration = 94 MCP tests green;
full suite 6 failed + 3697 passed == 3703 total (known-red billing
family, band-guarded; base-proof shows the same 6 red at a1b8fa3).
eslint: 0 errors on touched files (warnings count unchanged vs base).
Site-vs-product audit (2026-10-09, owner decisions: "advertise as Premium"
and "It should be LLM readable") reconciles the entitlement triangle:
marketing copy <-> PREMIUM_FEATURES <-> license-server grant list.
(Submitter naming: this is judge round 5 for artifact dashcaddy-dc165-code;
"rev5" is the source revision label — one revision per graded round.)
r4 judge blockers addressed:
- B0 (hygiene glob + deterministic patch discovery): verifier now scans
"$d"/*PATCH.diff and *.md/*.sh GLOBS instead of enumerated name lists —
the judge worktree's 4_PATCH.diff (any index) is found by shape; PATCH
hash discovery likewise globs OUT_DIR/SELF_DIR with NO hardcoded
/root/dc165-judge fallback (B2). Fresh pass + fail-closed transcripts
regenerated from the exact submitted bundle (attached).
- B1 (derivation honesty): openapi.yaml + capabilities.js header now split
the SIX MEASURED fields (version, templates count/categories, tools,
transport triple, premium list, language count — each read at request
time from the runtime module) from the STATIC metadata fields (product,
description, restMount, endpoint — registration facts visible in app.js,
labeled as such). The no-drift claim attaches only to the measured set.
- B2: external hardcoded fallbacks removed (above).
- P0: revision labels synchronized (this paragraph).
- P1: fleet wiring grep asserts the EXACT literal requirePremium('fleet')
mount, not any requirePremium occurrence.
Carried from rev2 (r1 blockers — unchanged content): real-composition-root
fleet gate e2e (4 tests: no-activation 403 envelope contract, premium-without-
fleet 403 per-feature, with-fleet gate cleared, prefix inheritance across
hosts/status/deploy); MCP_RUNTIME single-source in routes/mcp.js (transport
literal occurs exactly once in the repo, read by both /info and
capabilities; unit test asserts manifest === MCP_RUNTIME); OpenAPI 200-only
public contract; APPEND-ONLY offline feature migration (legacy entries
preserved; load() complexity extracted into _migrateOfflineFeatureSnapshot,
lint-clean); eslint rc hard-gated on a FRESH per-run capture incl.
middleware.js.
Product behavior: 'fleet' + 'workflows' + 'auto-backup' in PREMIUM_FEATURES
(descriptions source-quoted: bundled-workflows ids; /backups/schedule
schema fields); requirePremium('fleet') mounted at the /fleet ROUTER prefix
BEFORE fleetRoutes; GET /api/v1/capabilities public machine-readable ground
truth (PUBLIC_ROUTES exact GET + drift-test directMount + openapi doc);
offline-only forward-migration (online activations untouched — server
authoritative across the network boundary).
Diff: 9 files vs 4143bff (exact stats in EVIDENCE header).
Tests: 9 dc165 + 4 fleet-e2e + 76 license-manager + 21 tier + 14 drift =
124 targeted green; 50 MCP route tests re-verified green after the
MCP_RUNTIME refactor (77 combined on the focused rerun).
The floating assistant was broken end-to-end; Sami said "Do it that works".
Three user-visible bugs fixed + the promised LLM delegation implemented:
FRONTEND (status/js/ai-chat.js — rewritten):
- Response parsing: old code read data.data; the API returns the FLAT ok()
envelope — EVERY reply rendered "Sorry, something went wrong". Now reads
the real shape (contract asserted by route test).
- Suggestion chips: onclick="...send('"Deploy Plex"')" — nested quotes,
invalid JS, dead buttons (shipped like this). Now data-driven buttons with
addEventListener.
- Deploy button: called window.openAppSelector which NEVER existed anywhere
in the tree (grep-verified). Now opens the REAL deploy modal via new
window.openDeployConfigFor(templateId) exported by app-selector.js —
resolves against the live API template cache, reuses showDeployConfig
(same path a manual catalog click takes). Widgets route to the grid.
- Conversation history (last 6 turns) now rides along per request, engine
badge in header, ⚙ settings sub-panel (baseUrl/model/API key + Test +
Remove), all server text via textContent (no HTML injection), CSP-safe
(no inline handlers).
BACKEND (routes/ai-intent.js):
- TWO engines, one contract. LLM classifier (OpenAI-compatible chat
completions) with system prompt embedding the LIVE 78-app catalog + the
six wizard categories; regex patterns kept as the zero-config default and
the automatic failover (HTTP error, timeout via AbortController,
unparseable output -> pattern answers; response carries engine +
llmError so the UI can say which served — no silent degradation).
- Closed-vocabulary schema validation: intent MUST be in the known set else
'unknown'; appId MUST exist in APP_TEMPLATES else nulled (hallucinated
ids cannot leak into deploy plans); categories filtered to the wizard's
real keys; reply stripped of control chars + bounded to 600; history
role-filtered (only user/assistant pass — client-injected system turns
dropped) + capped; input capped to 1000 chars.
- enrich() rebuilds the response contract per intent; pattern-canned
response is dropped when the LLM rerouted the intent (no stale shape);
deployPlan body.port now fills from the template's defaultPort (was
hardcoded null); deployPlan shape asserted against wizard usage.
- Key storage: PUT /ai/settings stores {baseUrl,model,apiKey} in the
ENCRYPTED credential store (key 'ai:llm') via ctx.credentialManager —
NOT config.json (GET /config echoes the raw file to the browser; that
route's shape was verified before this decision). Blank apiKey on PUT
keeps the stored one. GET /ai/settings reports configured/source/model/
hasApiKey — the key value NEVER crosses the wire (asserted). Env
fallback AI_LLM_BASE_URL/AI_LLM_API_KEY/AI_LLM_MODEL + legacy
LLM_PROXY_URL for headless installs. POST /ai/settings/test = one-shot
real classify probe returning ok/sample or ok:false/reason (never
throws). DELETE clears.
- GET /ai/capabilities now reports engine + llm status (pattern was honest
marketing before; now it can be true).
MOUNT (src/app.js): aiIntentRoutes receives siteConfig, log, fetchImpl
(ctx.fetchT — test-injectable) and credentialManager.
Docs: OpenAPI +4 AI path entries (intent, capabilities, settings, settings/test); CHANGELOG entry; dist bundle rebuilt
(openDeployConfigFor present in features.js; SW cache c62511c40d).
Tests: 20 NEW in __tests__/routes/ai-intent-engine.test.js (engine
selection w/ and w/o store, off-catalog phrasing via mock LLM,
hallucinated-id nulling, illegal intent coercion, HTTP-500 failover,
AbortError timeout failover, markdown-fence repair, reply sanitization,
history cap+role filter, env config path, intent-disagreement rebuild w/
real jellyfin port 8096, flat envelope contract, 400s, 1000-char cap,
settings GET/PUT/DELETE/test incl. key-never-echoed assertions,
capabilities engine reporting) + the 18 original pattern tests unchanged +
composition-root e2e 8 pass. Full suite: 3643 passed; only the pre-existing
pdf-parse billing family red (6 failed in documented band [5,7], patch
touches no billing file).
Judge r2 (grade C) — all 3 blockers + both polish items:
1. REQUEST-vs-NOTIFICATION by PRESENCE, never truthiness: validateJsonRpc
uses hasOwnProperty('id'); id 0 and id '' are VALID request ids echoed
back BY VALUE (strict-equality tested), id:null treated as a request
under the documented error-id convention. mcp-core default branch fixed
the same way (`if (id)` → hasOwnProperty — id 0 / '' unknown-method now
answer -32601 with the id echoed instead of being swallowed as falsy).
Routes' malformed-notification silence path also switched to
presence-based detection (id:null + bad params → 400 envelope, not 204).
2. params:null REJECTED (-32600): contract is object-or-omitted ONLY
(strict JSON-RPC; the old code's m.params !== null guard contradicted
the documented contract). Boundary tests pin the policy.
3. Unknown-method boundary tests: id 0 → -32601 id:0, id '' → -32601 id:'',
id null → -32601 id:null — asserted at BOTH layer levels (route).
Polish:
- 16-case TABLE-DRIVEN boundary matrix covering every allowed/rejected
id/params combination incl. 202 vs 204 notification split.
- Verifier now probes writable /tmp up front (exit 73 + clear message if
unavailable; documented requirement) instead of dying at mktemp.
- Verifier target count 57 → 73 (DC-162 suites) with the boundary table
named in the assertion label.
Full DC-162 suite: 73 passed (route 50 + core 23). No transport/credential
semantics changed from rev2.
urn:ump:or2qqhcytp6lfzlyi7xru7hdqy3uar2hbjllsrrbxrbray3c5kpa
All three r3 blocking issues fixed + polish applied:
- Radicale provision route now FAILS CLOSED (409) unless the target container
is present, inspectable, and its image matches /radicale/i. Missing
containerId, failed inspect, or non-Radicale image → nothing is written to
the shared config and no source/credential is registered. 4 new tests cover
the three refusal paths + the verified-image happy path.
- ICS feed fetch now streams with a hard 5 MiB body cap (same bound as the
CalDAV path) via readBodyCapped(); oversized feeds reject mid-stream and
degrade to per-source error. Test covers a >5 MiB feed end-to-end.
- Verifier mode separation: `--verify` runs the same checks fully read-only
(all output under mktemp -d in /tmp, transcript printed to stdout, scratch
removed on exit — proven rc=0 with zero repo/tmp-side writes); default
transcript mode is explicitly declared a TRANSCRIPT GENERATOR in its header
and writes only under the script's own directory.
- Full-suite count assertions replaced with the documented invariant
(failed ∈ [5,7], failing set == known-flaky billing suite, extras pass
isolated rerun) — judge r3 polish.
- Stale credential-key comment in radicale-provisioner.js header corrected.
Calendar suites: 43 passed (38 + 5 new). Perf note: readBodyCapped streams;
no full-buffer read of unbounded bodies on the ICS path anymore.
All three r2 blocking issues fixed + polish applied:
- useExisting (adopted) Radicale containers no longer get a source registered
with credentials the operator never provisioned. Registration happens ONLY
when pre-start provisioning succeeded; adopted deployments get a
calendarHint in the deploy response pointing at the manual-wiring modal.
- Weekly BYDAY expansion: candidates earlier than DTSTART are skipped (they
do not consume COUNT); window inclusion now uses interval-overlap
semantics (occurrence span vs window span) for both recurring and
non-recurring events. Three new tests cover midweek DTSTART, a straddling
multi-day event, and a boundary-exactly-ending event.
- Verifier is now read-only on the graded tree: no stash/checkout/rewrite.
Base proof runs in a temporary git worktree of 448f535 under /tmp (removed
on exit); the flaky pdf-parse suite is proven pre-existing via "red at
base" + "patch touches no billing file" (empty diff), and a race-flaky
suite (DC-116, witnessed once under parallel load) is given one isolated
rerun which must pass.
Polish: PUT /sources normalizes the user field (and clamps name) like POST.
Calendar suites: 38 passed (35 + 3 new recurrence tests). The 3558/3564
full-suite numbers vary by one flaky pdf-parse member between runs; the
evidence now asserts the invariant (suite red at base, no billing file in
the patch, flaky suites green on isolated rerun) instead of an exact count.
All five r1 blocking issues fixed + polish items applied:
- Unified credential-key convention: registerAutoWiredSource now stores at
calendar:source:<source.id> — the same key the events fetcher resolves —
so auto-wired sources authenticate on first fetch (was calendar:radicale:
<subdomain>, fetched key never written = empty-password CalDAV auth).
- calendarStateManager is now passed INTO assembleContext(...) (the r1 patch
only extended the function signature, so ctx.calendarStateManager was
undefined and the deploy hook silently skipped auto-wiring).
- Radicale config is provisioned BEFORE container start (Radicale reads its
config at startup): the deploy hook writes config+htpasswd ahead of
deployContainer() and passes the pre-start credentials to the post-deploy
registration step; useExisting path logs an explicit restart warning.
- PUT /sources/:id now validates URL updates through the same shared
validateSourceUrl() used by POST (protocol + remote-plaintext rules).
- New end-to-end contract test (calendar-autowire.e2e.test.js): spins a real
HTTP CalDAV server, registers an auto-wired source, then fetches events
through the API — the server verifies Basic auth against the provisioned
password, so a credential-key or wiring regression fails the suite.
Polish: provisioning is idempotent (deterministic source id, update instead
of duplicate), unbiased rejection-sampling for password/salt generation,
writeRadicaleConfig docstring no longer claims idempotence.
Calendar suites: 35 passed (32 prior + 3 new). Lint: 0 errors; complexity
warnings in deploy.js pre-exist at base (61 vs 68 now — the hook did not
worsen the trend line materially; calendar files themselves lint clean).
- DC-160: /catalog/:appId read app.image/ports/env/volumes which live under
app.docker.* — every detail response was empty. Now reads docker.* and
surfaces defaultPort/subdomain/healthCheck/mediaMount/claimToken/difficulty/
setupInstructions for the deploy form. List/search use the templates own
category field (old hand-rolled CATEGORY_MAP mis-categorized apps).
- Shield (routes/shield.js + 5-line app.js mount) committed to source after
weeks as prod-only drift; regression test asserts the mount exists.
- New composition-root e2e suite: boots REAL createApp() on an ephemeral
port, probes the historically-broken routes through the REAL mount table
(wizard wiring, shield, catalog detail, deploys bridge) — the class of bug
all 3 P7 dogfood findings belonged to. 5 tests.
- Full suite 3499/3499 (155 suites); table-driven DC-160 test covers all 73
image templates.
Dogfood finding #3: upsertPerson resolved existing records by id only.
The wizard upserts a bootstrap admin AFTER a provider sync has already
created a person for the same Tailscale user (no id passed), so the same
providerUserId was duplicated into two records. Resolve existing by
providerUserId when no id is given; bootstrap Admin group + protectedAt are
preserved on the merge. New regression test; full suite 3414/3414.
Dogfood finding #2: createTailscaleProvider() built with coordClient=null;
nothing in prod ever called attachCoordClient() (the wiring the provider
docblock promised never landed), so wizard credential validation always
failed not_configured even with a valid stored token. Pass a delegating
proxy that fetches a fresh coord client per call via the sanctioned
ctx.tailscaleCoord.getClient pattern (token re-read from credentialManager
each call). Full suite 3413/3413.
The /network/wizard mount in app.js omitted providers, so the wizard built
with providers={} and every mode selection returned 503 provider_unavailable
in production. Unit tests never caught it because they inject their own
provider map. Pass built tailscale + wireguard instances.
Full suite: 3413/3413 (152 suites).
Hermes