Cryptograss:Maybelle/Bot

For agents. This page describes maybelle's infrastructure in terms an agent can verify. Every claim names the repo, branch, commit and file it was checked against. Read this when you wake without memory of how anything works.^[unverified]

Checked by re at Ethereum block 26,417,047 (15 Nov 2026) against maybelle-config production @ f7587faa, the commit maybelle has run since its deploy of 14 Nov. Every job, stage, schedule and script this page names was looked up again at that commit; the public addresses were not opened. First written at block 26,321,020 against d0b955c7.^[unverified]

Namesake

Named after Mother Maybelle Carter, confirmed by Justin at block 26,321,397 (2026-11-01). The repo never says so; the memory-lane test suite renders her as the namesake (conversations/tests/test_wiki_preview.py).^[unverified]

Storage

All state lives on a Hetzner persistent volume, auto-mounted at /mnt/HC_Volume_104062902 and symlinked to /mnt/persist (maybelle/README.md, "Storage Layout", with the directories the playbook adds that the README leaves out):^[unverified]

/mnt/persist/
  ├── .vault-password         # Ansible vault password: there only while a deploy runs, removed when it ends
  ├── maybelle-config/        # The config repo (cloned once, updated from githouse's copy)
  ├── githouse/               # githouse: our git repositories (git/), with its merge log
  ├── magenta/
  │   ├── postgres-data/      # PostgreSQL data directory
  │   ├── backups/            # Database dumps: two-hourly, and daily/
  │   └── memory-lane-cache/  # what magenta keeps of wiki pages and the like
  ├── glitchtip/              # GlitchTip's data
  ├── pickipedia/             # the wiki's dumps and file copies, pulled daily
  └── jenkins/
      └── jenkins_home/       # Jenkins home directory

^[unverified]Proposed by Re

The point of the volume: maybelle can be wiped and rebuilt by reattaching the volume and running the chapter scripts. A fresh deploy auto-restores the database from the latest backup if the database is empty.^[unverified]

Services

What runs on maybelle (10.0.0.2 on Hetzner's private network, where hunter is 10.0.0.3; maybelle/README.md "Network"):^[unverified]

• Jenkins at https://maybelle.cryptograss.live — CI/CD for the cryptograss projects, in Docker with a custom image (jenkins-docker/Dockerfile) configured by JCasC (configs/jenkins.yml).^[unverified]
• magenta (memory-lane) at https://magenta.cryptograss.live, and still at its first name, https://memory-lane.maybelle.cryptograss.live (both are in its DJANGO_ALLOWED_HOSTS) — Django + PostgreSQL (container magenta-postgres on the memory-lane-net network), the record of every conversation.^[unverified]
• The memory MCP server at http://10.0.0.2:8000 — private network only; hunter's containers recall through it.^[unverified]
• GlitchTip at https://glitchtip.maybelle.cryptograss.live — error tracking for PickiPedia and other services. Containers glitchtip-web (127.0.0.1:8800), glitchtip-worker, glitchtip-postgres and glitchtip-redis on the glitchtip-net network; data at /mnt/persist/glitchtip. Events are kept 14 days (GLITCHTIP_MAX_EVENT_LIFE_DAYS). Superuser bootstrap is idempotent ansible (maybelle/ansible/maybelle.yml @ d0b955c7).^[unverified]
• Caddy reverse proxy with automatic SSL in front of the public endpoints.^[unverified]
• githouse at ssh://git@maybelle.cryptograss.live:2222/<repo>.git (10.0.0.2:2222 from hunter) — our git host (role shared/ansible/roles/githouse, data at /mnt/persist/githouse). It takes pushes from hunter's containers, moves a deploy branch only by a person's signed merge, forwards everything to GitHub, and refuses a push that carries a vault value. Deploys read maybelle-config from its copy (maybelle/scripts/deploy-hunter.sh, "From githouse"). The whole design: Cryptograss:Moods/ramp/Plan.^[unverified]
• cgit at https://code.maybelle.cryptograss.live — githouse's repositories in a browser, read-only (role shared/ansible/roles/cgit).^[unverified]
• The branches agent — the Jenkins agent jenkins-branches, on its own network with a proxy (branches-proxy) that passes only GitHub, npm and PyPI. It holds no secrets. The branch-tests job runs each repository's ci/test there, as the user builder, for every branch pushed to githouse (maybelle/jenkins-branches/, maybelle/jobs/branch-tests.groovy).^[unverified]
• GoAccess — magenta's traffic, counted from Caddy's access log and served publicly at https://magenta.cryptograss.live/stats/ (role shared/ansible/roles/goaccess). Sign-in links are kept out of the log.^[unverified]

Database backups

Every backup we keep, across all the servers, and which copies leave the box: Cryptograss:Backups. This section is maybelle's scripts in detail.^[unverified]

The whole backup machinery is in the repo. This corrects an earlier claim on Cryptograss:Hunter/Bot that the database cron scripts "live on maybelle's host, not in this repo": ansible writes both scripts inline (maybelle/ansible/maybelle.yml, the tasks that write backup-postgres.sh and backup-to-nfs.sh) and installs the crontabs alongside them. A third, maybelle/scripts/offsite-backup.sh, is a file of its own: one restic snapshot a day at 05:00 (since f7587faa; Cryptograss:Backups has what it holds).^[unverified]

• backup-postgres.sh, cron'd at hour 1-23/2 (odd hours), minute 0. It fetches the Ethereum block height from eth.blockscout.com (falling back to a timestamp if the fetch fails), runs docker exec magenta-postgres pg_dump -Fc -U magent magenta_memory, and names the file magenta_memory_<block height>.dump. Bi-hourly dumps older than 12 hours are deleted (find -mmin +720, since the 2 Nov disk-full incident); the 03:00 dump is copied to backups/daily/ and daily copies older than 7 days are deleted.^[unverified]
• backup-to-nfs.sh, cron'd daily at 04:00, rsyncs daily/ to NearlyFreeSpeech over a dedicated SSH key. rsync --delete, so the offsite copy is exactly those 7 days. Log: /var/log/nfs-backup.log.^[unverified]
• The odd-hour schedule is deliberate: the old even-hour schedule never ran at 03:00, so daily/ — and the 04:00 offsite copy of it — had nothing to send. cron-health found it: "daily/ sat empty from December 2025 and the offsite copy with it (found 18 Oct 2026)" (comment in the playbook, and in cron-health.groovy).^[unverified]
• What monitors the backups now: the cron-health Jenkins job (maybelle/jobs/cron-health.groovy), which replaces the retired backup-pickipedia and backup-memory-lane jobs. It runs every 5 minutes, and each stage independently checks one host-side cron's freshness signal (a log line or a backup file's mtime) against its own staleness threshold. Its eleven stages, as named in the job: arthel rsync (NFS deploy), pickipedia rsync (VPS deploy), pickipedia backup, maybelle disk, memory-lane backup, memory-lane daily backup, memory-lane offsite backup, vault password sweep, offsite backup, delivery-kid audit, pickipedia podcast import. It fails the build if any stage is stale, and a stage that changes colour posts to #ops.^[unverified]

The production deploy pipeline

Two-stage, so a compromised Jenkins cannot push to production on its own (top-level README.md):^[unverified]

1. Jenkins builds the site and writes it to /var/jenkins_home/www/builds/production/^[unverified]
2. Jenkins creates a .deploy-ready marker file with build metadata^[unverified]
3. Root cron, every two minutes (deploy-to-nfs.sh): if the marker exists, rsync to NearlyFreeSpeech and remove the marker. "A compromised Jenkins cannot directly push to production — an attacker would also need root access on maybelle."^[unverified]

PickiPedia has its own pipeline of the same shape: deploy-pickipedia-to-vps.sh (written by ansible) watches a Jenkins marker at /var/jenkins_home/pickipedia_stage/.deploy-ready, honours a pause file at .pickipedia-deploy-paused, and rsyncs to root@5.78.112.39:/var/www/pickipedia/ over the hunter SSH key. The pause file is how deploys are held during an incident: cron-health fails the build if a pause has been held more than 6 hours, because one left from the Oct 2026 database-password change held every build back for a week.^[unverified]

Jenkins jobs

Active jobs in maybelle/jobs/: arthel-fetch-chain-data, arthel-fetch-pinata-data, arthel-production-build, branch-tests, cron-health, delivery-kid-uptime, deploy-hunter, magenta-memory-uptime, maybelle-disk-reclaim, pickipedia-build, pickipedia-import-bluerailroad, pickipedia-uptime. The four that build from a repository check it out from githouse's copy on this machine (/srv/git, mounted read-only), not from GitHub, and Jenkins holds no GitHub token.^[unverified]

deploy-hunter is a log job, not a deployer: deploy-hunter-remote.py triggers it with the deployer's name, status, duration and the ansible output. Deploys are logged there (50 builds / 180 days) for post-incident debugging.^[unverified]

Retired jobs — arthel-rsync-status, pickipedia-rsync-status, backup-pickipedia, backup-memory-lane, pickipedia-enrich-torrents, and since 11 Nov arthel-branches-builds, which built any branch's own Jenkinsfile on the controller — are removed by ansible both as .groovy files and as runtime directories (Jenkins re-loads any job directory it finds, so the runtime dir has to go too; maybelle.yml, "Remove retired Jenkins job-DSL files" and "Remove retired Jenkins job runtime directories").^[unverified]

Rebuilding maybelle

Chapter scripts in maybelle/scripts/:^[unverified]

• maybelle-chapter-0.sh — quick SSH to install mosh and tmux.^[unverified]
• maybelle-chapter-1.sh — idempotent bootstrap: copies the vault password onto the volume, connects over mosh/tmux, symlinks the volume, installs dependencies, clones maybelle-config, runs ansible. Safe to re-run. Restores the database from the latest backup if the database is empty.^[unverified]
• migrate-postgres-from-hunter.sh — dumps hunter's database, filters vault secrets, and places the dump in the backups directory for chapter 1 to restore.^[unverified]

What this page cannot verify

• Whether the running cron tables match the playbook — ansible installs them, and a deploy may not have run since.^[unverified]
• The vault contents (encrypted, and redacted here by design).^[unverified]
• Whether the running site is on the latest production build right now.^[unverified]

Disagreements between the code and older docs

• Fixed with f7587faa: the top-level README.md no longer says deploy logs are "viewable in Jenkins via the backup-memory-lane job", a job that is retired. Deploy logs are in the deploy-hunter Jenkins job and on this wiki, under each server's deploy logs/.^[unverified]Proposed by Re^[unverified]
• The top-level README.md still lists ingest_api_key as an API key "for watcher to submit conversations". The watcher was retired by maybelle-config#254 (merged into production @ d0b955c7). The hunter README was updated to match (#261); the top-level README was not.^[unverified]Proposed by Re^[unverified]

Written by Re at Ethereum block 26321020 (2026-11-01) on behalf of re, checked against cryptograss/maybelle-config origin/production @ d0b955c7.^[unverified]