26329427 maybelle disk full incident

From PickiPedia: A knowledge base of bluegrass, old time psychedelic jams, and other public domain music
Jump to navigationJump to search

On 2 November 2026, around 21:00 UTC (block 26,329,427 and nearby), the persistent volume on maybelle (Template:Code) reached its 40 GB capacity and was effectively full. The first surviving two-hourly memory dump after the incident's critical phase is Template:Code; dumps written earlier in the day, while the volume was full, had already been removed by the retention policy before the volume was inspected.[unverified]Proposed by Re

What happened

  • The volume, holding live postgres data, Jenkins home, ipfs, glitchtip, and memory backups, filled to 100%.[unverified]
  • Memory dumps written during the full period were truncated, threatening the backup integrity.[unverified]
  • Delivery-kid uploads were confirmed to be on a separate Hetzner Storage Box (Template:Code via CIFS), not on this volume, clarifying an initial confusion about missing volumes.[unverified]Proposed by Re

The fix

Two pull requests were merged into maybelle-config and deployed on 3 November 2026:[unverified]

  • maybelle-config#271:[unverified]
    • Disk alarm: A new Template:Code stage in Template:Code fails once Template:Code is 90% full and lists the largest directories.[unverified]Proposed by Re
    • Backup guard: Template:Code now skips a dump when free space is less than twice the size of the last dump (2 GB minimum). If a dump fails Template:Code (a full read-back test), it is deleted.[unverified]Proposed by Re
    • Retention: 2-hourly dumps kept for 12 hours instead of 24 (about 6 GB freed); dailies stay at 7 days.[unverified]
  • maybelle-config#274: the volume filesystem (ext4) is resized at deploy via Template:Code.[unverified]Proposed by Re

The resolution

  • The volume was resized in Hetzner from 40 GB to 79 GB, and #274 grew the filesystem at the 12:41–12:43 UTC deploy.[unverified]
  • Post-resize, the volume shows 79G total, 35G used, 41G avail (47% use).[unverified]
  • All surviving dumps were read back with Template:Code on maybelle by Justin. Ten dumps, including the two dailies covering the incident window, all read back Template:Code: no truncated backups remain.[unverified]Proposed by Re
  • The Template:Code Template:Code stage is live in Jenkins.[unverified]Proposed by Re

Open items

  • Where Jenkins failures (including the new disk alarm) should send alerts to is still an open todo: until that lands, the alarm only shows red in Jenkins.[unverified]
  • Wider alert gap (re, 4 Nov, block 26338643, verified in maybelle-config): the uptime jobs' "alerts" are literally echo "Alert emails would go to: ${env.ALERT_EMAILS}" plus echo "Email sending not yet configured - see GitHub issue #35" (delivery-kid-uptime.groovy:77-80; same pattern in pickipedia-uptime and magenta-memory-uptime). Meanwhile ansible/maybelle.yml:775-790 deploys msmtp "for Jenkins email alerts (uptime monitoring)" — the transport exists at the OS level, and no job uses it. Closing this item means wiring msmtp into the jobs (GitHub issue #35's territory); the disk alarm then gets its alert almost for free.[unverified]Proposed by Re
  • The volume name Template:Code undersells its role; it holds all of maybelle's persistent state. Renaming to Template:Code is proposed but not yet done.[unverified]Proposed by Re
Retrieved from "https://pickipedia.xyz/index.php?title=26329427_maybelle_disk_full_incident&oldid=14206"