Ali Husain Software engineer and product designer

03 Infrastructure/Homelab

Homelab

Platform · open source

Fifteen services on one small machine, run as code, with backups that are proven every week.

pi@raspberrypi:~/docker$ sudo _host/backup-restore-test.shPASS adguard: AdGuardHome.yaml present with http/dns sectionsPASS couchdb: 8 .couch files (live 8), docker.ini presentPASS kuma: integrity_check ok, monitor=13 (live now 13)PASS vaultwarden: integrity_check ok, ciphers=412 (live now 412)PASS dockge: integrity_check ok, user=1 (live now 1)PASS syncthing: key.pem + parseable cert.pem + config.xmlPASS homeassistant: tar opens, backup.json, link count 2PASS platform: bundle clones (146 files, 0 commits behind)PASS secrets: 6 files, all identical to live (contents not shown)restore test: 9 passed, 0 failedpi@raspberrypi:~/docker$ 
15
services, on one small machine
43
automated tests, run on every change
9
backup sets, restored every week to prove them
0
ports forwarded from the internet
Network
Remote access behind carrier-grade NAT with zero port forwards: a 7-node mesh VPN, split-horizon DNS and one audited public endpoint.
Backup and recovery
Every set written and verified nightly, copies on three machines, a restore drill every week and a written bare-metal rebuild plan.
Observability
77 alert conditions, 13 uptime probes every 60 seconds, self-healing watchdogs and an external dead-man switch.
As code
450 commits, 21 systemd timers and no cron, a tool that proves the live machine still matches git, and about 4,000 lines of runbooks.
Operations
An always-on AI operations agent is on call: it wakes on alerts, works inside a written charter, and logs every change.

Why it exists

A home depends on a few services: DNS, the password vault, notes, the smart home. When one fails, people see the symptom, and nobody is on call.

The usual protections fail quietly. A backup nobody restores is a guess. An engine that deletes old archives after writing a broken new one is worse than none.

What it does

I run fifteen services on one small home server the way an operations team would; this public copy shows nine of them. Every hand-written file, host configuration included, lives in git at its live path, and a tool reports where the machine differs.

Backups are verified every night and restored in a drill every week. A failure reaches one phone as one push.

How it is built

  • Verify before pruning

    The backup engine checks each new archive (SQLite's quick_check, a row count, the required files) before it deletes older ones. A set that fails keeps everything it had.

  • The repository is the machine

    Host files sit at their live paths with a manifest. The install tool refuses uncommitted changes, so what is installed is always a committed version, and the drift check reports any difference.

  • No port forwarded

    No port is forwarded. Remote devices use Tailscale, a private network built on WireGuard. A firewall rule stops ports that Docker publishes from bypassing the host firewall.

  • An operator under a charter

    An always-on AI session investigates alerts under a written charter. Every change it makes is a commit with a reason, and one command stops it.

How I know it works

43 checks
Tests of the notifier and the network probe behind stubs, on every change. A run must end with zero real calls.
9 sets
In this public copy, CI validates the backup list and all nine sets pass the restore drill in the sandboxed demo.
10 seconds
The weekly drill takes about 10 seconds: it unpacks every set's newest archive into RAM and compares it with live data.

What was hard

The drill failed a good backup

The weekly drill once failed the secrets set: 1 of 24 restored files differed from live. The backup was fine. Home Assistant runs its own nightly backup, sometimes after this one, and rewrites the history part of the file that holds its backup encryption key. The key was identical.

I fixed the test rather than run it again, because a drill that cries wolf teaches people to ignore it. It now compares that file by its key only. A changed key still fails, tested with a tampered copy.

Limits

  • The server is the network's only DNS resolver, on purpose: if it is down, lookups stop.
  • If the machine dies, the backup sets can lose up to about 26 hours.
  • CI does not run a backup or a restore; those run on the live machine and in a sandboxed demo.
  • The public copy carries 9 of the 15 containers, with placeholder values.

Also in Infrastructure

Open to remote work.

ali@gallowforge.dev