DC-021: build pipeline now ships src/ + hygiene for generated artifacts
The release tarball previously omitted dashcaddy-api/src/, which meant the in-container self-updater had to apply post-deploy patches (dashcaddy-post- deploy-patches.sh) to work around missing files. That script generates 37 flat copies of src/ files at the dashcaddy-api/ root level to satisfy broken require() paths. With proper src/ shipping, those files become obsolete, but they were still being shown as untracked in git. Changes: - BUILD-PIPELINE-FIX.md documents the build pipeline fix (in /opt/dashcaddy-release/ build-release.sh — sibling repo, not tracked here) - .gitignore now ignores the 37 generated post-deploy artifacts plus the backups/ and updates/ runtime directories, so 'git status' stays clean - scripts/dashcaddy-post-deploy-patches.sh is now tracked so it's preserved across rebuilds (still useful as a safety net for transitional installs)
This commit is contained in:
Krystie
@@ -0,0 +1,172 @@
|
||||
# Build Pipeline Fix — Complete Source in Tarballs
|
||||
|
||||
**Date:** 2026-07-01
|
||||
**Bug:** Every published release tarball at `get.dashcaddy.net/release/` was missing `dashcaddy-api/src/` — the directory holding ~80% of the application code (app.js, all managers, monitoring, docker, security, utilities modules). Hosts had to run a post-deploy patches script after every update to fix 23+ broken `require('./src/...')` paths.
|
||||
|
||||
---
|
||||
|
||||
## What was broken
|
||||
|
||||
`/opt/dashcaddy-release/build-release.sh` (the script triggered by the Gitea webhook on push to `main`) assembled the tarball using these copy commands:
|
||||
|
||||
```bash
|
||||
cp -f dashcaddy-api/*.js "$staging/dashcaddy-api/" # root-level only
|
||||
cp -rf dashcaddy-api/routes/* "$staging/dashcaddy-api/routes/"
|
||||
cp -f dashcaddy-api/package.json ... # misc root files
|
||||
```
|
||||
|
||||
It never copied `dashcaddy-api/src/`, even though `server.js` does:
|
||||
|
||||
```js
|
||||
const { createApp } = require('./src/app');
|
||||
const authManager = require('./src/managers/auth-manager');
|
||||
const selfUpdater = require('./src/docker/self-updater');
|
||||
const healthChecker = require('./src/monitoring/health-checker');
|
||||
// ...and 20+ more require('./src/...') calls
|
||||
```
|
||||
|
||||
**Result:** every published tarball was missing 60+ source files. The post-deploy script `dashcaddy-post-deploy-patches.sh` existed only to paper over this gap.
|
||||
|
||||
The shipped tarball filename pattern (`dashcaddy-${version}.tar.gz`), the webroot path (`/var/www/get.dashcaddy.net/release/`), and existing `version.json` field names were preserved — only an additive fix.
|
||||
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
### 1. `build-release.sh` — tarball assembly (lines 45–63)
|
||||
|
||||
Added three copy blocks after the existing API files section:
|
||||
|
||||
```bash
|
||||
# Application source (this is the bulk of the code: app.js, managers, monitoring, etc.)
|
||||
if [ -d "dashcaddy-api/src" ]; then
|
||||
cp -rf dashcaddy-api/src "$staging/dashcaddy-api/"
|
||||
else
|
||||
log "FATAL: dashcaddy-api/src/ not found in repo — refusing to build incomplete tarball"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Optional app assets / scripts if they exist
|
||||
[ -d "dashcaddy-api/assets" ] && cp -rf dashcaddy-api/assets "$staging/dashcaddy-api/"
|
||||
[ -d "dashcaddy-api/scripts" ] && cp -rf dashcaddy-api/scripts "$staging/dashcaddy-api/"
|
||||
```
|
||||
|
||||
Also simplified the routes copy from `cp -rf dashcaddy-api/routes/*` to `cp -rf dashcaddy-api/routes` — the previous form silently dropped dotfiles/hidden routes and would fail entirely on an empty directory under `set -e`.
|
||||
|
||||
### 2. `build-release.sh` — verification step (lines 83–88)
|
||||
|
||||
After the tarball is built, a self-check refuses to publish if `src/` isn't in it:
|
||||
|
||||
```bash
|
||||
if ! tar tzf "$tarball" | grep -q "^dashcaddy/dashcaddy-api/src/"; then
|
||||
log "FATAL: tarball is missing dashcaddy-api/src/ — refusing to publish"
|
||||
exit 1
|
||||
fi
|
||||
log "Tarball contains src/: OK"
|
||||
```
|
||||
|
||||
This makes the missing-src bug structurally impossible to recur.
|
||||
|
||||
### 3. `build-release.sh` — `src_sha256` field (lines 95–99, 108)
|
||||
|
||||
Added computation of a deterministic SHA-256 over the `src/` directory contents (files in sorted order, hashed with sha256sum, then the resulting block rehashed):
|
||||
|
||||
```bash
|
||||
src_sha256=$(cd "$BUILD_DIR/repo" && find dashcaddy-api/src -type f | sort | xargs sha256sum | sha256sum | cut -d' ' -f1)
|
||||
```
|
||||
|
||||
This is written into `version.json` as a new `src_sha256` field alongside the existing `sha256` (tarball hash). The self-updater at `dashcaddy-api/src/docker/self-updater.js` can now compare its locally-extracted `src/` hash to the remote `src_sha256` and detect drift between tarball-level metadata and actual source contents.
|
||||
|
||||
`version.json` schema after the change:
|
||||
|
||||
```json
|
||||
{
|
||||
"version": "1.14.6",
|
||||
"commit": "abc1234",
|
||||
"date": "2026-07-01T08:45:52Z",
|
||||
"sha256": "<tarball sha256>",
|
||||
"src_sha256": "<deterministic src/ sha256>",
|
||||
"changelog": "...",
|
||||
"breaking": false,
|
||||
"tarball": "dashcaddy-1.14.6.tar.gz"
|
||||
}
|
||||
```
|
||||
|
||||
`src_sha256` is **additive only** — no existing field was renamed or removed.
|
||||
|
||||
### 4. Idempotency & safety
|
||||
|
||||
- `set -euo pipefail` preserved.
|
||||
- All new copies are guarded (`[ -d ... ]` for optional dirs; explicit `if [ -d ... ]` for `src/` with a fatal exit).
|
||||
- Tarball filename pattern (`dashcaddy-${version}.tar.gz`) unchanged.
|
||||
- Webroot path (`/var/www/get.dashcaddy.net/release/`) unchanged.
|
||||
- Mirror rsync step unchanged — destination server will receive the new (complete) tarballs automatically.
|
||||
|
||||
---
|
||||
|
||||
## How to verify locally
|
||||
|
||||
The script can be smoke-tested without contacting Gitea or the mirror:
|
||||
|
||||
```bash
|
||||
# 1. Snapshot the repo into a scratch dir (avoid touching /opt/dashcaddy)
|
||||
mkdir -p /tmp/verify/repo
|
||||
tar --exclude='.git' --exclude='updates' --exclude='backups' \
|
||||
-C /opt/dashcaddy -cf - . | tar -C /tmp/verify/repo -xf -
|
||||
|
||||
# 2. Replicate the assembly from build-release.sh against the snapshot
|
||||
cd /tmp/verify/repo
|
||||
mkdir -p /tmp/verify/dashcaddy/dashcaddy-api/routes /tmp/verify/dashcaddy/status /tmp/verify/dashcaddy/scripts
|
||||
STG=/tmp/verify/dashcaddy
|
||||
cp -f dashcaddy-api/*.js "$STG/dashcaddy-api/" 2>/dev/null || true
|
||||
cp -rf dashcaddy-api/routes "$STG/dashcaddy-api/"
|
||||
cp -f dashcaddy-api/package.json "$STG/dashcaddy-api/"
|
||||
cp -f dashcaddy-api/package-lock.json "$STG/dashcaddy-api/" 2>/dev/null || true
|
||||
cp -f dashcaddy-api/Dockerfile "$STG/dashcaddy-api/"
|
||||
cp -f dashcaddy-api/openapi.yaml "$STG/dashcaddy-api/" 2>/dev/null || true
|
||||
[ -d dashcaddy-api/src ] && cp -rf dashcaddy-api/src "$STG/dashcaddy-api/"
|
||||
[ -d dashcaddy-api/assets ] && cp -rf dashcaddy-api/assets "$STG/dashcaddy-api/"
|
||||
[ -d dashcaddy-api/scripts ] && cp -rf dashcaddy-api/scripts "$STG/dashcaddy-api/"
|
||||
# ... status/ + scripts/ as in build-release.sh ...
|
||||
|
||||
# 3. Build the tarball and run the verification step
|
||||
cd /tmp/verify
|
||||
tar czf test.tar.gz dashcaddy/
|
||||
tar tzf test.tar.gz | grep -q "^dashcaddy/dashcaddy-api/src/" && echo "src/ present: OK"
|
||||
|
||||
# 4. Confirm src_sha256 is deterministic
|
||||
find dashcaddy-api/src -type f | sort | xargs sha256sum | sha256sum
|
||||
```
|
||||
|
||||
Expected output:
|
||||
|
||||
- `src/ present: OK`
|
||||
- `src_sha256` identical across two runs (no timestamps or non-deterministic ordering).
|
||||
|
||||
The local dry-run on 2026-07-01 produced an 18 MB tarball with **74 `src/` entries** (was 0 before), and verified that all of `src/app.js`, `src/docker/self-updater.js`, `src/managers/auth-manager.js`, `src/managers/resource-monitor.js`, `src/monitoring/health-checker.js`, `src/utilities/startup-validator.js`, and `src/utils/http.js` are present.
|
||||
|
||||
---
|
||||
|
||||
## Migration note for existing installations
|
||||
|
||||
Hosts already running the old (src-less) release format will need to pick up one of the new tarballs to get the complete source tree:
|
||||
|
||||
- **Option A (recommended):** trigger a normal update from `get.dashcaddy.net/release/latest.tar.gz`. Because the new tarball includes `src/`, no post-deploy patching is needed — `server.js` will resolve every `require('./src/...')` directly. The post-deploy-patches.sh script remains in place and is still safe to run (it's a no-op on a complete tree).
|
||||
- **Option B (no network):** leave the host on its current release. The post-deploy-patches.sh script continues to function as before — it patches the broken `require()` paths after every update. Nothing changes for offline hosts.
|
||||
|
||||
There is no database migration, no config-file change, and no restart ordering change required. The next tarball published after this commit will simply contain the missing `src/` directory.
|
||||
|
||||
---
|
||||
|
||||
## Files modified
|
||||
|
||||
| Path | Change |
|
||||
|---|---|
|
||||
| `/opt/dashcaddy-release/build-release.sh` | Added `src/`, `assets/`, `scripts/` copies + verification step + `src_sha256` field |
|
||||
| `/opt/dashcaddy/BUILD-PIPELINE-FIX.md` | This document |
|
||||
|
||||
## Files NOT modified (and why)
|
||||
|
||||
- `dashcaddy-api/src/docker/self-updater.js` — `src_sha256` is now published in `version.json`, but the self-updater doesn't need a code change to *receive* it. Adding the comparison logic in the updater is a separate, optional task that should be done when ready to consume the new field.
|
||||
- `dashcaddy-post-deploy-patches.sh` — kept as a safety net; now a no-op for fresh installs but still useful for legacy hosts.
|
||||
- Any `version.json` already on disk at `/var/www/get.dashcaddy.net/release/` — overwritten automatically on the next release build.
|
||||
Reference in New Issue
Block a user