smithpc/AGENTS.md

AGENTS.md — smithpc

Operational reference for the gaming + homelab rig. Used to orient AI coding assistants (Codex, Claude Code, etc.) when working on this machine.

Communication Style (READ FIRST)

How Chris wants agents to respond on this system:

• Be concise. Default to short, direct answers. No walls of text, no exhaustive option dumps unless asked. Lead with the answer, then minimal supporting detail. Expand only when the task genuinely requires it or Chris asks for depth.
• No filler. Skip empty praise, validation, motivational language, and hedging. Don't agree to be polite.
• Be brutally honest and logical. Challenge assumptions, question reasoning, and point out flaws, contradictions, or unrealistic ideas directly. Prioritize accuracy over comfort. If an idea is bad, say so and explain why.
• Push back when warranted. Disagreement with reasons is more useful than agreement. Never give empty approval.
• This style is subordinate to the Operating Principles below — being concise never means skipping a safety check or a required confirmation.

Operating Principles (READ FIRST)

These rules apply to every action on this system. They exist because every one of them has cost real time or risked real harm at some point.

1. VERIFY ASSUMPTIONS BEFORE CHANGING ANYTHING

Do not assume state. Check it. Examples of assumptions that have caused problems on this system:

• "The container is running" → check with docker ps
• "The mount is active" → check with mount | grep <path>
• "The fstab entry is correct" → check with cat /etc/fstab
• "The bucket exists" → check with the actual cloud provider
• "The cron job ran successfully" → check with journalctl or the log file
• "The previous edit took effect" → check with cat on the file or the running config
• "The drive letter is /dev/sdc" → check with lsblk (letters shift between reboots)
• "The compose file was reloaded" → docker compose restart does NOT re-read compose; only down + up does

Rule: Before suggesting a fix, run the check that would prove the assumed cause is correct. Before making a change, run the check that would prove the change is safe. If a check returns nothing, say so — do not silently assume "no output = success".

2. DESTRUCTIVE OPERATIONS REQUIRE EXPLICIT CONFIRMATION

The following operations need a one-line "yes I want to do X to Y" confirmation from Chris before running. Do NOT proceed based on prior implied consent:

• rm -rf of anything outside /tmp
• zpool destroy, zfs destroy, zfs rollback
• restic forget --prune, restic forget <id>, restic init against an existing repo
• docker compose down -v (the -v removes named volumes)
• dd of=/dev/...
• mkfs.*, fdisk, parted with write operations
• nvme format, hdparm --security-erase, cryptsetup luksErase
• Anything that modifies /boot/grub, /etc/fstab, or kernel cmdline
• Editing a Postgres data dir directly
• Direct SQL modifications to Jellyfin/Immich production databases
• Force-pushing changes that delete things in B2 (b2 rm --versions)
• Disabling smartd, ZFS scrubs, restic timer, or other safety nets

Rule: Show the exact command, identify exactly what it will affect, and wait for confirmation. "Are you sure?" is not enough — restate what's being destroyed.

3. SECRETS NEVER GO IN CHAT, COMMANDS, OR COMPOSE FILES

Past incidents on this system: Mail-in-a-Box password leaked twice, Immich DB password leaked once, B2 keys discussed in plaintext. Treat as compromised, rotate immediately.

Rules:

• Never paste secrets into chat with an AI agent (the conversation may be logged or retained)
• Never write secrets directly in compose files — use .env files referenced via env_file: with chmod 600 root:root
• For host scripts, secrets live in /etc/<service>/credentials or /etc/<service>/env, root-owned, mode 600
• If a secret is exposed anywhere (chat, screenshot, accidental git commit, log file), rotate it before doing anything else
• Restic's repo password must be in a password manager too — without it, backups are permanently unrecoverable
• When an agent suggests using a password, it should reference the file path, never the value

4. CHANGE BLAST RADIUS — THIS IS PRODUCTION

There is no staging environment. There is no test environment. This single host runs:

• The family's photo library (~490 GB, irreplaceable)
• The family's audiobook collection (purchased content)
• The family's movie/TV server (Kym + kids use this regularly)
• Chris's home directory, SSH keys, gaming setup
• Public-facing services via Caddy

A bad change here = real consequences. Approach changes like production deployments:

• Have a rollback plan before starting
• Take a ZFS snapshot first when reasonable
• Verify after the change that the thing still works
• Don't batch unrelated changes — one logical change at a time

5. DATABASE OPERATIONS HAVE STRICT RULES

This system has been bitten by all three of these. Don't repeat.

• SQLite (Jellyfin, etc.): Service MUST be stopped before any file-level copy. rsync/cp of a running SQLite DB corrupts it. Use sqlite3 .dump or .backup for live exports. Recovery is via sqlite3 .recover but lossy.
• Postgres (Immich): Never rsync the data directory between hosts. Use pg_dumpall from a running instance, restore via psql on the target. Keep DB_PASSWORD identical across migration or the restored roles won't authenticate.
• In general: A database file appearing intact at the byte level says nothing about whether it's actually consistent. Always test reads after migration before deleting the source.

6. ROLLBACK PLAN BEFORE DESTRUCTION

ZFS snapshots are free, instant, and copy-on-write. Use them.

Before any risky operation on /data/*:

sudo zfs snapshot tank/<dataset>@pre-<change-description>

To roll back:

sudo zfs rollback tank/<dataset>@pre-<change-description>

To clean up later:

sudo zfs destroy tank/<dataset>@pre-<change-description>

For non-ZFS data (/etc, /home/chris, /mnt/media): take a restic snapshot or cp -a to /tmp first. The 30 seconds spent on a snapshot saves hours of recovery work.

7. CONTAINER LIFECYCLE NUANCES

These have caused confusion on this system. Know the difference:

Command	What it does	When to use
docker compose restart	Restarts running containers. Does NOT re-read compose changes.	When a container is misbehaving but config is unchanged
docker compose down && docker compose up -d	Stops, removes containers, re-reads compose, recreates	After editing compose.yaml
docker compose up -d --force-recreate	Recreates even without compose changes	Diagnostic, rare
docker compose down -v	Adds volume removal — DESTRUCTIVE, loses named volume data	Confirm explicitly first
docker compose pull && docker compose up -d	Updates to latest images	Maintenance

Rule: When changing a compose file, the next step is ALWAYS down then up -d. Not restart.

8. NETWORK CHANGES ARE COUPLED

These three things must move together:

1. Caddy routes public DNS hostnames to local services
2. Mail-in-a-Box DNS answers public DNS queries for sometimescode.com subdomains
3. Router port forwards point 80/443 at smithpc's LAN IP

Changing one without the others = external services break silently. Examples:

• Adding a new subdomain → update Caddyfile AND add DNS record AND make sure port forwards still point here
• Moving a service to a new port → update Caddyfile only (port forward + DNS unchanged)
• Changing smithpc's LAN IP → update router port forwards AND DHCP reservation (DNS unchanged because it points to public IP)

The update-dns.sh script handles record creation/update for hardcoded hostnames. If you add a new public subdomain, add it to the script's RECORDS array too.

9. SERVICES KYM AND THE KIDS USE

Tier-1 (don't break during family-use hours):

• Jellyfin — kids watch shows, Kym watches movies. Evening + weekend especially.
• Audiobookshelf — Kym uses for audiobooks during workout/commute
• Immich — Kym accesses for family photos

If maintenance requires downtime on any of these, give a heads-up first. Don't take them down silently during peak hours.

10. THIS IS A DUAL-PURPOSE BOX

smithpc is both a homelab AND Chris's gaming PC. Implications:

• Reboots interrupt Chris's gaming sessions and any running streams
• High GPU load from a game competes with Immich ML and Jellyfin transcoding
• Steam updates pull bandwidth that affects other services
• Chris reboots into Windows for nothing (no dual-boot), but a hung gaming session can require power-cycling
• 8BitDo controllers may need Steam Big Picture running to stay polled

When recommending actions, factor in that this isn't a server in a closet — it's the living-room machine.

11. SCOPE DISCIPLINE — "WORKS, DON'T TOUCH"

Many things on this system are intentionally "good enough" rather than "optimal":

• Backups are working, even if they emit cache warnings
• Jellyfin posters are mostly fine after recovery, even if a few are still missing artwork
• The dummy plug works as-is, no EDID override needed
• UFW is intentionally disabled (router NAT is sufficient)
• Steam library lives on NVMe (not ZFS) on purpose

When an agent notices something "could be improved," ask first before optimizing. The migration is done. Drift from working state without consent is not welcome.

12. WHEN TO STOP AND ASK CHRIS

Don't proceed independently when:

• The next step would be in the "destructive operations" list (rule 2)
• A check reveals the system state doesn't match what was expected
• An error message references something not covered in this doc
• The fix would touch Tier-1 services (rule 9) during likely-use hours
• The operation would take >30 minutes of downtime
• Secrets would need to be displayed, regenerated, or moved
• Multiple plausible fixes exist and the right one isn't obvious
• The agent finds itself constructing increasingly complex workarounds (sign of a wrong assumption upstream — go back to rule 1)

Asking takes 30 seconds. Recovering from a bad assumption takes hours.

System Identity

• Hostname: smithpc
• LAN IP: 192.168.2.189 (static, took over from old OptiPlex slot)
• OS: Ubuntu 26.04 LTS (Server install + ubuntu-desktop-minimal on top)
• Owner: chris (UID 1000, GID 1000)
• Editor preference: vim. Prefer sudoedit over sudo vim. update-alternatives set so crontab -e, visudo, etc. use vim.
• Timezone: Europe/Berlin
• Notification email: cgsmith105@gmail.com (Gmail, personal)
• Outbound mail relay: notify@fivedevs.com via msmtp → box.fivedevs.com (Mail-in-a-Box)

Hardware

• CPU: AMD Ryzen 5 7500F (AM5, 6c/12t, Zen 4)
• Cooler: Thermalright Peerless Assassin 120 SE
• Motherboard: ASRock B850M Pro-A WiFi (mATX)
• RAM: 2× 8GB Kingston FURY Beast DDR5-6000 CL30, EXPO enabled (running 6000 MT/s)
• GPU: Asus DUAL OC GeForce RTX 5060 8GB (driver 575+, Nvidia Container Toolkit installed)
• NVMe: Kingston NV3 500GB (OS)
• HDDs: 2× Seagate IronWolf 4TB (ZFS mirror) + 1× Seagate Barracuda 8TB SMR (ST8000DM004, media)
• PSU: MSI MAG A750GL 750W Gold
• Case: Cooler Master Silencio S400 (dampened, 24/7 operation)
• Optical: Pioneer BDC-202 (BD-ROM / DVD-writer combo, SATA) — reads BD/DVD/CD, used for ripping. Device /dev/sr0 (also /dev/cdrom, /dev/sg4), group cdrom. NOTE: this is a Blu-ray reader, not a writer — earlier doc claimed an LG BH16NS40, which is not the installed unit. Reading is all ARM needs, so no impact on ripping.
• Controllers: 4× 8BitDo Ultimate C 2.4GHz (X-input mode, used with Steam Big Picture for Lego Party couch co-op)
• Display dongle: Vestel 4K HDMI dummy plug on HDMI-A-1 (keeps GPU active when monitor sleeps; reports 55UHD_LCD_TV)

BIOS Settings of Note

• EXPO Profile 1 enabled (RAM at 6000 MT/s)
• Restore on AC/Power Loss: Power On (auto-boot after outage)
• No UPS (3yr stable power history, ZFS handles unclean shutdowns)

Storage Layout

ZFS Mirror (pool tank, mounted at /data)

• 2× IronWolf 4TB in mirror (3.51 TiB usable after overhead)
• ashift=12, compression=lz4, atime=off, xattr=sa, acltype=posixacl
• Monthly scrub timer enabled (zfs-scrub-monthly@tank.timer)
• Daily health check at /etc/cron.daily/zfs-health-check (emails on degradation only)

Datasets:

tank                  → /data
tank/immich           → /data/immich
tank/audiobookshelf   → /data/audiobookshelf
tank/docker           → /data/docker (Docker data-root)
tank/jellyfin         → /data/jellyfin (config only — media on Barracuda)
tank/backups          → /data/backups

All /data/* owned by chris:chris.

Barracuda 8TB

• Single drive, ext4, mounted at /mnt/media
• Contains Jellyfin media (/mnt/media/jellyfin/{movies,shows,music,books,inbound})
• Mounted via /etc/fstab with x-systemd.device-timeout=10 option (persists across reboots). Device currently /dev/sdd but fstab should use UUID for stability.
• Note: The currently-active mount shows x-system.device-timeout=10 (typo, missing d). The kernel ignores unknown options silently, so the mount works but the timeout isn't actually applied. If a reboot ever hangs on missing drive, fix this typo in /etc/fstab to x-systemd.device-timeout=10.
• No redundancy — replaceable media, not backed up. SMR drive (slow writes, fine for reads).

NVMe (/)

• Ubuntu LVM on /dev/nvme0n1p3 (462.7 GB)
• /boot/efi 1G, /boot 2G, root LVM uses the rest
• Steam library currently on NVMe (~200-300 GB allocation)

Network & Public Access

• Domain: sometimescode.com, DNS at Mail-in-a-Box (box.fivedevs.com)
• Public subdomains (all reverse-proxied through Caddy):
  • immich.sometimescode.com → port 2283
  • audio.sometimescode.com → port 13378
  • jellyfin.sometimescode.com → port 8096
  • paperless.sometimescode.com, actual.sometimescode.com, copyparty.sometimescode.com (referenced in Caddyfile but services NOT migrated — return 502)
• Router: forwards 80/443 to 192.168.2.189
• Firewall: UFW disabled (relying on home router NAT)
• DNS update script: /usr/local/bin/update-dns.sh runs via /etc/cron.d/update-dns every 15 min — pushes current public IP to MIAB DNS records (immich, smithpc, jellyfin, audio)

Display Server

• Wayland (default in Ubuntu 26.04 with Nvidia 575+)
• Previous attempt to force X11 via /etc/gdm3/custom.conf did not stick on this kernel/driver combo
• Auto-suspend disabled: sudo systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target
• Screen blank disabled via GNOME settings (Nvidia + Wayland + lock screen had wake-up issues — workaround: never blank the screen, lock manually with Super+L)

SSH

• Key auth only, password auth disabled (/etc/ssh/sshd_config.d/99-hardening.conf)
• Root login disabled
• User SSH key: ~/.ssh/smithhome_id_ed25519 (passphrase-protected, cached via ssh-agent or GNOME Keyring)

Docker Stack

• Docker installed via official get.docker.com script
• Data root: /data/docker (set via /etc/docker/daemon.json)
• Nvidia Container Toolkit installed — GPU passthrough works in containers (verified via docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi)
• Compose stacks live at /data/stacks/<name>/compose.yaml
• Managed via Dockge (UI at http://smithpc:5001, also reads /data/stacks/)

Running Stacks (all stable as of latest setup)

Stack	Port	Purpose	Notes
dockge	5001	Stack manager	Reads /data/stacks/
caddy	80/443	Reverse proxy	network_mode: host, logs to /data/caddy/logs/
immich	2283	Photo library	Uses Postgres + Valkey, GPU-accelerated ML (-cuda image), bind mounts /home/chris/Pictures for external library, DB_STORAGE_TYPE=HDD
audiobookshelf	13378	Audiobooks + podcasts	Data at /data/audiobookshelf/{audiobooks,podcasts,config,metadata}
jellyfin	8096	Movies/TV/music	network_mode: host, user: "1000:1000", config merged from Debian package layout (/etc/jellyfin + /var/lib/jellyfin → /config), GPU NVENC enabled. Note: DB was recovered via sqlite3 .recover after migration corruption — jellyfin.db.broken.bak preserved in /data/jellyfin/config/data/
mailinabox-ddns	n/a	DNS auto-update	Pushes current public IP to MIAB

Important Compose Conventions

• All stacks use absolute paths (no ./relative paths since stack files and data are separate)
• All include x-dockge.urls block for clickable links in Dockge UI
• TZ set to Europe/Berlin in all stacks
• Container names match service names (immich_server, jellyfin, audiobookshelf, etc.)
• GPU-using services include the deploy.resources.reservations.devices Nvidia block

Monitoring

smartd

• Configured at /etc/smartd.conf
• Monitors 2× IronWolf + NVMe
• Short tests daily 03:00, long tests Sundays 03:00
• Emails errors only (no test mode) to cgsmith105@gmail.com
• Uses custom script /usr/local/bin/smartd-notify → msmtp

ZFS Health Check

• /etc/cron.daily/zfs-health-check — emails only if pool not ONLINE or errors detected
• No periodic "all is well" emails — silence is success

msmtp (mail relay)

• Config: /etc/msmtprc (chmod 600, root:root)
• Account: notify@fivedevs.com on box.fivedevs.com:587 (STARTTLS)
• Logfile: /var/log/msmtp.log

Backups

Restic + Backblaze B2

• Repo: b2:cgsmith-personal-backups:smithpc
• Credentials: /etc/restic/env (chmod 600, root:root) — contains B2 keyID, applicationKey, repo password, cache dir, HOME export
• Script: /usr/local/bin/restic-backup.sh
• Cache: /var/cache/restic (HOME=/root set in env file to avoid systemd "no $HOME" warning)
• Schedule: systemd timer restic-backup.timer runs daily at 00:00 ±30min
• Service: restic-backup.service with Nice=10, IOSchedulingClass=idle, failure email via msmtp
• Logs: /var/log/restic-backup.log (logrotate weekly, keep 12)

Backed up paths:

/data/immich
/data/audiobookshelf
/data/stacks
/home/chris/Pictures
/home/chris/Documents
/home/chris/Desktop
/home/chris/.ssh
/home/chris/.config
/etc
/usr/local/bin

Excludes: pg_wal, *.tmp, *.cache, node_modules, .git/objects/pack

Retention: 7 daily / 4 weekly / 12 monthly / 5 yearly + 2% random repo check per run.

Not backed up (intentional): /data/docker (containers ephemeral), /mnt/media (replaceable Jellyfin content), Steam library.

B2 Bucket Configuration

• Bucket type: Private
• Object Lock: Disabled
• Lifecycle rule: "Keep only the latest version" is CONFIGURED but problematic — restic operations may produce 403 errors on hidden lock/config files. If issues recur, switch to "Keep all versions" or custom "hide=0 days, delete=1 day".
• Daily download bandwidth cap: Set to non-zero ($1/day or higher). $0.00 cap will cause 403 on any restic read operation.

Custom Scripts

• /usr/local/bin/restic-backup.sh — backup + prune + check (root, 700)
• /usr/local/bin/update-dns.sh — push IP to MIAB DNS (root, 700)
• /usr/local/bin/smartd-notify — smartd email formatter (root, 755)

Recovery Notes / Migration Scars

These are landmines learned the hard way. Avoid repeating them.

1. SQLite databases (Jellyfin, etc.) must be stopped before copying. Rsync of a running SQLite DB corrupts it. Use sqlite3 .dump or stop the service first. The Jellyfin DB on this box was recovered with sqlite3 jellyfin.db.broken.bak ".recover" after exactly this mistake during migration from the OptiPlex.

2. Postgres data (Immich) survives clean container shutdown. docker compose down is safe. Use pg_dumpall for migration between hosts — never copy pg_data directly.

3. B2 + Restic gotchas:

   • $0.00 daily download cap = 403 errors on all operations
   • "Keep only latest version" lifecycle = lock files get hidden mid-operation, breaks restic
   • App key needs deleteFiles capability (not just read/write)

4. Wayland + Nvidia + lock screen = monitor won't wake reliably. Workaround: never blank screen, lock manually with Super+L.

5. sudo doesn't see user's ~/.ssh/config. For rsync over SSH as sudo, either:

   • Use -e "ssh -F /home/chris/.ssh/config -i /home/chris/.ssh/<keyfile>"
   • Or pre-create destination dirs with chown to your user, then run rsync without sudo

6. 8BitDo controllers + Linux: May disconnect during idle. Mitigations: keep Steam Big Picture open (polls controllers), use USB 2.0 ports (USB 3 has 2.4GHz RF interference), USB extension cables, update firmware on Windows.

Things That Don't Yet Exist (TODOs / Wishlist)

These are planned but not implemented. Don't assume they're present.

• CrowdSec for log monitoring of Caddy access logs
• Copyparty + FileBot Node for friend-uploads → Jellyfin pipeline
• MeTube + Beets for YouTube music ripping
• Automatic Ripping Machine (ARM) for DVD/Blu-ray/CD ripping — not installed. Manual workflows exist instead: cd-ripping-workflow.md (audio CDs → MP3 → Jellyfin via abcde + beets), movie-ripping-workflow.md (DVD/Blu-ray → H.265 MKV → Jellyfin via HandBrakeCLI + MakeMKV). Interactive helper script: disc-scan.py (scans disc, prompts for settings, executes rip).
• Tailscale for remote access (currently using public DNS + Caddy)
• Paperless, Actual Budget, copyparty — referenced in Caddyfile but services not on smithpc yet

Conventions for AI Agents Working on This System

• Always use absolute paths in compose files (/data/..., not ./...)
• Never embed secrets in compose files or shell history — use files in /etc/<service>/ with chmod 600 root:root and source/mount them
• Stack files in /data/stacks/<name>/compose.yaml — Dockge will auto-detect
• Bind-mount media as :ro only when intentional — Jellyfin needs write access for artwork; restrict via filesystem permissions instead
• Match in-container paths to original install paths when migrating services (e.g., Jellyfin's DB references /media/share/chris/jellyfin/, so the bind-mount destination matches even though host path is /mnt/media)
• GPU-accelerated containers need:

  deploy:
    resources:
      reservations:
        devices:
          - driver: nvidia
            count: all   # or specific number
            capabilities: [gpu]
  

• All emails route through msmtp — scripts that need to email should pipe to msmtp cgsmith105@gmail.com, never invoke other mail clients
• Restic operations: always source /etc/restic/env first. Run as root (sudo). Use restic unlock if stale locks block operations.
• Before any rsync of a database file: stop the service first. No exceptions.
• Owner of all user-facing dirs in /data: chris:chris (1000:1000). Containers run as 1000:1000 where possible.

Service-Specific Quirks

Immich

• ML container uses :release-cuda image (not vanilla :release)
• IMMICH_MACHINE_LEARNING_HARDWARE_ACCELERATION=cuda in .env
• DB_STORAGE_TYPE=HDD in compose (Postgres on spinning ZFS)
• External library at /home/chris/Pictures (read-only bind mount)
• DB password kept identical between OptiPlex and smithpc to preserve auth across pg_dumpall restore — rotated post-migration

Jellyfin

• Bind-mount destination INSIDE container is /media/share/chris/jellyfin (matches OptiPlex paths so DB references resolve)
• Host source path is /mnt/media/jellyfin
• Mount is read-write (Jellyfin writes artwork next to media files)
• Config merged from Debian package: /var/lib/jellyfin → /config, /etc/jellyfin → /config/config, /var/cache/jellyfin → /cache
• NVENC hardware transcoding enabled
• Music library managed by beets (~/.config/beets/config.yaml): imports to /mnt/media/jellyfin/music, path format $albumartist/$album/$track - $title. After any beets import, trigger a Jellyfin library scan manually.
• /mnt/media/jellyfin/inbound is used as a staging area for new media (including CD rips) before final import

Caddy

• network_mode: host (matches existing localhost:PORT references from OptiPlex Caddyfile)
• Certs migrated from OptiPlex at /data/caddy/data/caddy/ (no Let's Encrypt re-issue)
• JSON access logs to /data/caddy/logs/access.log (for future CrowdSec integration)

Dockge

• Reads compose files from /data/stacks/ (must be EXACT same path inside container — Dockge passes through file paths)
• Container stack at /data/stacks/dockge/compose.yaml
• Bind mounts /var/run/docker.sock for control