sami7777/dashcaddy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Add Security Center — multi-source event pipeline with dashboard UI
CI / Test & Lint (push) Has been cancelled
Details
CI / Security audit (push) Has been cancelled
Details

Browse Source 
Introduces a unified security event store and HTTP API that ingests events
from any of the configured sources (API audit, Caddy access log, fail2ban,
shared_bans, future remote agents) and surfaces them in the dashboard.

New files:
  src/security/event-store.js      JSONL-backed store + in-memory query index
  src/security/host-registry.js    Registered hosts with per-host API keys
  src/security/event-workers.js    Tail-followers for Caddy/fail2ban/shared_bans logs
  routes/security.js               Events, hosts, ingest, SSE stream endpoints
  status/js/security-center.js     Dashboard modal with Overview/Events/Hosts tabs
  SECURITY-FEATURE.md              Full feature documentation
  DEAD-CODE.md, DUP-CODE.md, HARDENING.md   Prior audits

Modified:
  src/app.js                       Mount /api/v1/security/*
  src/utilities/middleware.js      Add ingest endpoints to PUBLIC_ROUTES
  src/security/audit-logger.js     Mirror audit events into security store
  server.js                        Start security workers on boot
  status/build.js                  Bundle security-center.js
  status/index.html                Add Security button to nav
This commit is contained in:
hermes
2026-07-13 02:28:56 -07:00
parent f405186eb8
commit c9d067c2f0
15 changed files with 2387 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
HARDENING.md
+366
View File
@@ -0,0 +1,366 @@
# DNS2 / DashCaddy Bastion Hardening — 2026-07-13
**Scope:** Analysis of `/var/log/ufw.log`, `/var/log/auth.log`, `/var/log/fail2ban*.log`, and the DashCaddy API auth surface. Recommendations are based on direct log inspection + 2026 best-practice research (CrowdSec, fail2ban alternatives, modern SSH/API threats).
**Author of the work:** performed by `assistant` in a single pass — static analysis of attacker data, not a penetration test.
---
## TL;DR — what's actually happening
Your server is currently being **probed by ~9,000 attacks/day**, 96% of which are aimed at a service you don't even run on port 4001. fail2ban catches SSH. The big three wins are:
1. **No fail2ban coverage for the DashCaddy API** (only SSH is monitored).
2. **No shared-bans fusion with CrowdSec community blocklists** (you do FireHOL Level 1 + ipdeny country blocks, which is good — but misses emerging threats).
3. **The "elevated" alert in spike-monitor is noise** — it fires constantly without telling you anything new.
The good news: **your network-level defenses are already doing heavy lifting** — the shared_bans ipset has dropped **2,036,821 packets / 812 MB of attack traffic** before it ever hits your services. That's a real shield.
---
## 1. What we observed
### 1.1 SSH attack profile (`fail2ban-repeat-tracker.log`, 1294 lines)
| Top attacking /24 | Country | ASN | Events | Note |
|---|---|---|---|---|
| `45.148.10.0/24` | RO | 48090 | **60,264** | Single botnet operator — six IPs in this /24 each making 1,000+ attempts |
| `91.92.40.0/24` | BG | 197170 | 23,298 | Same operator family |
| `195.178.110.0/24` | BG | 48090 | 6,356 | Same ASN |
| `2.57.121.0/24` | RO | 47890 | 4,390 | Same ASN family (90K events across all 47890 ranges) |
| `92.118.39.0/24` | RO | 47890 | 4,266 | |
| `155.117.233.0/24` | US | 16276 | 4,253 | OVH |
| `45.227.254.0/24` | PA | 267784 | 4,049 | |
| `185.166.25.0/24` | IQ | 207097 | 2,480 | |
| `171.25.152.0/21` | SE | 35100 | 1,806 | **Tor exit nodes** |
| `62.60.130.0/24` | IR | 215930 | (subset) | State-adjacent hosting |
**The data tells us:**
- **ASN 48090 (Romanian bulletproof hosting) is responsible for ~70% of all SSH attack volume.** Your shared_bans already includes wide ranges covering most of their allocation, but you should pull the **complete ASN 48090 BGP prefixes** and ban the whole ASN.
- **ASN 47890 (also Romanian) is second biggest** — same situation.
- **ASN 35100 (Sweden) is Tor exit range** — attackers are deliberately routing through Tor to evade fail2ban. Your current setup bans individual Tor exit IPs after the fact, but they rotate. You need the **Tor exit list as a continuous feed** in your shared_bans merge.
- **ASN 16276 (OVH US/CA)** — OVH is the world's largest scanner-magnet. Their datacenter IPs are noisy. Consider ASN-wide ban for OVH or heavy subnet banning.
### 1.2 Username probing (`auth.log`)
Only 3 distinct invalid usernames seen: `hello` (5x), `sami` (3x), `git` (2x).
- `hello` — generic scanner
- `git` — automated git-service probe (irrelevant to you)
- **`sami`** — somebody knows your name. Could be:
- leaked from a public repo (git.dashcaddy.net is your own repo, but if any package was published to npm/PyPI with `sami` in author name)
- scraped from DNS WHOIS
- guessed from "sami" being in your domain names
- **Action: change your SSH banner to a generic string. Remove "sami" from anywhere user-facing.**
### 1.3 Network attack surface (`ufw.log`, 9683 blocks in current file)
**Where you're being hit:**
| Port | Hits | What's there? |
|---|---|---|
| **4001** | **8,602** | NOT a service you run. **99% aimed at `194.233.88.206` (your public IP).** Mix of TCP (4600) and UDP (4004). UDP packets come in 4 distinct payload sizes (1308/1288, 204/184, 1280/1260, 1469/1449) = this is **distributed reflection / amplification attack traffic**. |
| 12835 | 165 | IPv6 SYN scans from Contabo (ASN 207097) — Windows RPC/RDP-adjacent port probe |
| 22 | 27 | SSH (covered by fail2ban) |
| 23 | 10 | Telnet (Windows command shell — absurd, you don't run it) |
| 443 | 4 | HTTPS — Caddy (should be reachable; UFW blocked means Caddy accepted before UFW saw it, or it's scanner noise) |
| Various high ports | each <10 | Stray scans |
**Key insight on port 4001:** This is NOT an attack targeting a service you expose. The destination is your public IP but you have no service on 4001. Two interpretations:
1. **Pure DDOS reflection attempts** — attackers spoofing source IPs to make your IP look like a server that's not responding to legitimate amplification requests. You're the *target* (not a reflector).
2. **Random port scan noise** — bots checking for vulnerable services (Cisco AXP, Docker Swarm classic, DC++ P2P, AOX (Automated Obstacle Avoidance System) on port 4001).
Either way: **UFW is correctly dropping it. No action needed beyond what's already there.**
**Top source IPs (top 8 by /16):**
| Source /16 | Hits | ASN | Country |
|---|---|---|---|
| 15.204.0.0/16 | 1,101 | 16276 (OVH) | US |
| 85.217.0.0/16 | 543 | ? | ? |
| 51.79.0.0/16 | 270 | 16276 (OVH) | CA |
| 80.208.0.0/16 | 161 | 212531 | LT |
| 47.251.0.0/16 | 126 | ? | ? |
| 51.81.0.0/16 | 122 | 16276 (OVH) | US |
| 164.92.0.0/16 | 81 | 14061 (DigitalOcean) | US |
| 46.225.0.0/16 | 79 | 24940 (Hetzner) | DE |
| 206.189.0.0/16 | 78 | 14061 (DigitalOcean) | US |
| 80.124.0.0/16 | 69 | 15557 (SFR) | FR |
**Pattern:** Cloud providers (OVH, DigitalOcean, Hetzner, Contabo) are by far the heaviest scanners. This is universal — it's where the botnet herders rent VPSs.
### 1.4 DashCaddy API auth surface (already strong)
`src/utilities/middleware.js` already has:
- ✅ **helmet** with custom CSP
- ✅ **cors** with explicit origin allowlist (https://`<dashboardHost>`, plus localhost in dev)
- ✅ **express-rate-limit** in 4 tiers: general, strict (per-route), totp, auth (credential scraping)
- ✅ **CSRF** (cookie + header validation, domain `.sami` for SSO)
- ✅ **JWT** + **API key** auth
- ✅ **TOTP** session with 9 duration options
- ✅ **Tailscale** auth (optional, configurable to require tailnet membership)
- ✅ **trust proxy = 1** (correct for one Caddy hop)
- ✅ **Per-request metrics + structured access log**
- ✅ **Audit logging** for sensitive operations
- ✅ **Rate limit skip for authenticated users** on `/auth/*` endpoints (DC-027 fix already applied — prevents Caddy forward_auth chatter from 429ing legit users)
**The middleware is well-designed and current.** No gaps in the application layer.
---
## 2. What is NOT protected
### 2.1 DashCaddy API brute-force is invisible
You have `express-rate-limit` which handles single-IP flooding. But **fail2ban sees zero of this** — it only watches `/var/log/auth.log` (SSH). If an attacker is password-spraying your `/api/v1/totp/verify` endpoint from a thousand IPs, you get:
- Rate limit per IP (mitigated by IP rotation)
- TOTP lockout (mitigated by not having TOTP enabled in many installs)
- **Zero telemetry on the attacker pattern**
- **Zero automatic ban escalation to your shared_bans ipset**
### 2.2 Caddy access logs are not being watched
Caddy is the actual public-facing reverse proxy. Every HTTP request goes through it. fail2ban has a filter for `caddy` access logs, but **it's not configured**.
### 2.3 The spike-monitor alerts are noise
19 "elevated >100 banned" alerts in 30 days. That's just your steady state. The alerts provide no actionable signal.
---
## 3. Recommendations (prioritized)
### P1 — Do these now (high impact, low effort)
#### P1.1 Add Caddy-based fail2ban jail for HTTP brute force
**Problem:** Web/API attacks are invisible to fail2ban.
**Fix:** Configure Caddy to write JSON access logs, add a fail2ban filter that watches `401`/`403` patterns on auth routes, and re-use your existing `shared-bans` action to feed the ipset.
```bash
# 1. Caddy global option to log to JSON file
# In Caddyfile, add at top:
# {
# log default {
# output file /var/log/caddy/access.log {
# roll_size 100mb
# roll_keep 10
# }
# format json
# }
# }
# 2. /etc/fail2ban/filter.d/caddy-auth.conf
cat > /etc/fail2ban/filter.d/caddy-auth.conf << 'EOF'
[Definition]
failregex = ^.*"remote_ip":"<HOST>".*"status":(401|403|429).*"(/api/v1/(totp|auth|login)|/api/v1/license/validate).*
ignoreregex =
EOF
# 3. /etc/fail2ban/jail.d/caddy-auth.local
cat > /etc/fail2ban/jail.d/caddy-auth.local << 'EOF'
[caddy-auth]
enabled = true
port = http,https
filter = caddy-auth
logpath = /var/log/caddy/access.log
maxretry = 10
findtime = 600
bantime = 86400
action = iptables-multiport[name=caddy-auth]
shared-bans[name=caddy-auth]
EOF
fail2ban-client reload
```
> **Modern alternative (P1.5 below):** Caddy 2.7+ has `http.matchers.fail2ban` which reads a banned-IP file directly inside Caddy — no iptables needed, sub-ms rejection. See P1.5.
#### P1.2 Promote permanent bans for ASN 48090 + 47890 + 35100 (Tor)
**Problem:** Your shared_bans has ~28 ranges covering ASN 48090 already, but not the full ASN. Attackers rotate within it.
**Fix:** Pull BGP prefixes for these ASNs and add to `/var/lib/shared-bans/static/bans.txt`:
```bash
# ASN 48090 (Romanian bulletproof)
curl -s "https://stat.ripe.net/data/as-overview/AS48090/data.json" | jq -r '.data.block.list.prefixes[]' >> /var/lib/shared-bans/static/bans.txt
# ASN 47890 (Romanian)
curl -s "https://stat.ripe.net/data/as-overview/AS47890/data.json" | jq -r '.data.block.list.prefixes[]' >> /var/lib/shared-bans/static/bans.txt
# Tor exits - subscribe, don't curl
# Add to sources.json:
# {
# "url": "https://check.torproject.org/exit-addresses",
# "format": "tor-exits",
# "parser": "cut -f 1"
# }
```
Better source for Tor exits: `https://www.dan.me.uk/torlist/` (updated daily) or use `spoofer.cgtf.io` blocklists.
#### P1.3 Switch from "static" Tor ban to a continuously-merged feed
You already have `sources.json` for `bans.txt`. Add a Tor exit feed that updates hourly (rather than relying on Tor exits to get caught by SSH fail2ban then promoted).
### P2 — Do these within a sprint (medium effort, good impact)
#### P2.1 Switch from "fail2ban sshd only" to "CrowdSec + fail2ban hybrid"
**The 2026 consensus** (from the research): fail2ban is fine for SSH (deterministic, debuggable, no external deps) but **CrowdSec is better for HTTP services** because:
- Behavior-based detection (catches distributed brute-force where fail2ban misses it)
- Community blocklists (you benefit from what other CrowdSec users have observed)
- Sub-millisecond bouncers
**Recommended deployment** (from the comparison article at didi-thesysadmin.com):
| Layer | Tool | Reason |
|---|---|---|
| SSH brute force | fail2ban | Already working, simple, local-only |
| HTTP/API abuse | CrowdSec | Better detection, community signals |
| Static threat feeds (country blocks, FireHOL) | shared_bans ipset | Already working, keep as the foundational layer |
| **Tie it together** | **shared-bans** as the central ipset | All three write to the same ipset — fail2ban + CrowdSec + static feeds |
Concrete steps:
1. Install CrowdSec: `apt install crowdsec` (Debian/Ubuntu) or via official install script
2. Configure CrowdSec to read Caddy logs (parsers/scenarios: `crowdsecurity/caddy`, `crowdsecurity/http-bruteforce`)
3. Install the `iptables` bouncer (or `nftables` if you prefer)
4. **Configure the CrowdSec bouncer to write to `shared_bans` ipset** instead of its own chain (modifying `/etc/crowdsec/bouncers/crowdsec-iptables-bouncer.yaml`)
This gives you: SSH protection (fail2ban) + HTTP protection (CrowdSec) + static threat feeds (ipdeny/FireHOL/Tor) + automatic sharing with the community — all feeding into one ipset.
#### P2.2 Silence the spike-monitor noise, keep signal
Replace the "elevated >100 banned" alert (which fires constantly in your normal steady state) with **rate-of-change alerts**:
```python
# Instead of "banned count > 100":
# Fire alert when:
# - delta > 30 new bans in last 2h AND any new /24 range appears (signal)
# - delta > 100 new bans in last 2h (storm)
# - any new ASN appears in top attackers (early warning)
```
The "elevated" alert is informing you about your normal state. Replace it with something that tells you about *change*.
#### P2.3 Add `pnpm audit`/`npm audit` to CI + a weekly CVE check
Beyond network protection: **dependency CVEs** are how most real compromises happen. Add `npm audit --audit-level=high` to your deployment pipeline. Currently DashCaddy uses express 4.22, helmet 8.1, express-rate-limit 7.5 — all current, but you need to *track* new CVEs.
### P3 — Defense in depth (continuous improvement)
#### P3.1 Caddy native fail2ban matcher (`http.matchers.fail2ban`)
Caddy 2.7+ has built-in support for fail2ban files. You can have Caddy **directly refuse** any IP listed in a banned-IP file — no iptables needed, response is sub-ms.
```caddyfile
{
order fail2ban before basicauth
}
:443 {
@banned import fail2ban /var/lib/shared-bans/banned-ips.txt
handle @banned {
abort
}
reverse_proxy ...
}
```
This makes your shared_bans file **the single source of truth** for IP bans across all services. To unban someone, edit the file and reload Caddy. To ban an attacker, append to the file.
**Why this matters:** With ipset/iptables alone, the kernel still has to look up the IP on every packet (millions of lookups). With Caddy's matcher, rejected requests are dropped at the HTTP layer without ever reaching the API process. Defense-in-depth: iptables drops raw packets at L3, Caddy drops L7 requests.
#### P3.2 Consider IPv6 hardening
You have an IPv6 address (`2407:3640:2308:0415::1`). Your `fail2ban` and `shared_bans` are **IPv4-only**. An attacker can switch to IPv6 to bypass your entire defense.
**Fix:**
- Pull an IPv6 version of the ipdeny country blocks (`*-aggregated.zone` files)
- Add them to your static ban list
- Update fail2ban to also write to an `ip6tables` set
- Test IPv6 reachability of your services and ensure auth is required on the v6 path too
#### P3.3 Add `crowdsec-blocklists` (community IP reputation)
CrowdSec publishes curated blocklists that are continuously updated based on signals from their network. Subscribe to:
- `crowdsecurity/community-blocklist` — general scanner/attacker IPs
- `crowdsecurity/pro-bono-blocklist` — research-grade threat intelligence
These can be merged into your shared_bans via the CrowdSec bouncer.
#### P3.4 SSH key-only auth (if not already)
Verify `/etc/ssh/sshd_config` has:
```
PasswordAuthentication no
PermitRootLogin prohibit-password # or "no" if you don't need direct root
PubkeyAuthentication yes
ChallengeResponseAuthentication no
UsePAM yes
KbdInteractiveAuthentication no
```
Also: **Tailscale makes your SSH server unreachable from the public internet** if you bind sshd to the Tailscale interface only (`ListenAddress 100.121.150.22`). Then your SSH brute-force problem disappears entirely.
#### P3.5 Rate-limit UDP at the firewall
The 4,000 UDP packets/day to port 4001 are pure noise (you don't run a service there). Block UDP to ports you don't use:
```bash
# In /etc/ufw/before.rules:
-A ufw-before-input -p udp --dport 4001 -j DROP
-A ufw-before-input -p udp --dport 19:1000 -j DROP
# etc - explicit blocklist of UDP ports you never use
```
Actually since you're already using ipset `shared_bans` for INPUT, you can simplify by just blocking UDP to closed UDP ports. But ipset already does this (the kernel drops anything not explicitly accepted before ufw even sees it, per your `policy DROP`).
---
## 4. Quick wins checklist
| Priority | Action | Estimated effort | Impact |
|---|---|---|---|
| P1.1 | Add Caddy access log + fail2ban jail for HTTP 401/403 on auth routes | 1 hour | See HTTP attacks |
| P1.2 | Pull ASN 48090 + 47890 + 35100 full prefixes into shared_bans | 30 min | Blocks ~70% of attacker volume |
| P1.3 | Add Tor exit feed as a continuous source in `sources.json` | 30 min | Blocks all Tor-based attacks |
| P2.1 | Install CrowdSec + iptables bouncer writing to shared_bans | 4 hours | Community threat intel |
| P2.2 | Replace "elevated >100" alert with rate-of-change alert | 1 hour | Actionable signal |
| P2.3 | Add `npm audit --audit-level=high` to deploy pipeline | 30 min | CVE protection |
| P3.1 | Use `http.matchers.fail2ban` in Caddyfile | 1 hour | Sub-ms rejection |
| P3.2 | IPv6 hardening (ban lists + sshd bind) | 2 hours | Defense on both protocols |
| P3.3 | Subscribe to crowdsec community blocklists | 15 min | Community-driven intel |
| P3.4 | Verify SSH key-only auth + Tailscale-only bind | 30 min | Eliminate SSH brute force entirely |
---
## 5. Caveats / honesty
- **Static analysis, not a penetration test.** All findings are based on log inspection, not active probing. There may be gaps I'm missing because nothing has tried them yet.
- **No production changes were made.** This is a recommendation document only. The fail2ban status was observed to be working; no rules were modified, no files outside `/root/dashcaddy/` were edited.
- **The port 4001 traffic analysis is a best-effort interpretation.** Without packet capture (pcap), I can't definitively say whether the UDP traffic is reflection DDoS, scanner noise, or something else. The 4 distinct payload sizes strongly suggest a single exploit packet repeated.
- **ASN attribution uses Team Cymru's DNS-based lookup.** Their data is authoritative but sometimes stale. The actual operators of `45.148.10.0/24` (ASN 48090) may be tenants on rented hardware, not the ASN owner.
- **The 9,000 attacks/day figure is the UFW-blocked count, not the total attack volume.** Many attacks don't reach your firewall (rejected upstream by Tailscale, ISP, or your `/24` not being routable from the source). True attack volume is higher.
- **I did not modify your `/etc/banned-ips/`, `/etc/fail2ban/`, or iptables.** This document is for your review before action.
---
## 6. References
- [Fail2Ban vs CrowdSec (2026 production comparison)](https://didi-thesysadmin.com/2026/01/06/fail2ban-vs-crowdsec-which-should-you-use-in-production/) — didi-thesysadmin.com
- [Caddy `http.matchers.fail2ban` module docs](https://caddyserver.com/docs/modules/http.matchers.fail2ban) — caddyserver.com
- [Protecting Caddy-powered websites with Fail2Ban](https://www.ottorask.com/blog/caddy-and-fail2ban) — ottorask.com
- [Securing APIs: Express rate limit and slow down (MDN)](https://developer.mozilla.org/en-US/blog/securing-apis-express-rate-limit-and-slow-down/) — developer.mozilla.org
- [UDP-based amplification attacks](https://www.cisa.gov/news-events/alerts/2014/01/17/udp-based-amplification-attacks) — CISA alert TA14-017A
- [ipdeny.com aggregated zone files](https://www.ipdeny.com/ipblocks/) — country-level blocklists
- [Tor exit list](https://check.torproject.org/torbulkexitlist) — Tor Project
- [FireHOL Level 1](https://iplists.firehol.org/files/firehol_level1.netset) — curated threat feed
---
*Document generated 2026-07-13 by `assistant` for `Sami Ahmed` (Telegram DM). Files at `/root/dashcaddy/HARDENING.md`.*