Cryptograss:Runbooks/A key is compromised

What to do, in order, when someone's SSH key, or a device signed in with it, may be in someone else's hands. Brakes first, then removal, then a look at what happened. Why it's in this order: Cryptograss:Moods/ramp/Plan, section 10.

1. Brakes (minutes; no merge, no deploy)

An admin (MOOD_ADMINS: today only justin) does these from their own terminal:

1. {{Bot_proposes|magenta.sh kick <name> --all --ban signs out every device of theirs and bars their key from signing in to Moods. githouse also refuses their merge orders while they are banned (live since 8 Nov).
2. {{Bot_proposes|If you're unsure who, or how far it went: magenta.sh AZ5. Everyone is signed out, the Moods are locked and no runner wakes anyone. githouse merges nothing during AZ5 (live since 8 Nov).
3. If memory-lane itself may be in someone else's hands, freeze merging: as root on maybelle, touch /mnt/persist/githouse/etc/frozen. githouse merges nothing, from anyone, whatever memory-lane says. To lift it, as root, rm the same file. A redeploy leaves it in place. Live once maybelle-config#286 is deployed; magenta.sh freeze is planned as a wrapper.
4. GitHub: remove them from the org at https://github.com/orgs/cryptograss/people. Until githouse is how merges happen, an org member's session can merge on GitHub.

2. Removal (a merge and redeploys)

1. Take their ssh_pubkey out of people in maybelle-config's hunter/ansible/inventory.yml, and their name out of githouse_mergers if it's there. Make the pull request, then merge it with GitHub's button or magenta.sh merge.
2. Redeploy hunter and maybelle. That removes their key from the servers, from Moods' allowed_signers and from githouse's merge_signers.

3. What happened

• The merges since the key could have been taken: git log --first-parent --since=… production (and main) in each repository, and on maybelle /mnt/persist/githouse/git/.merges/ and githouse.log for orders signed in their name.
• pickipedia and arthel run the Jenkinsfile of whatever merged into production within minutes. Read Jenkins' runs since then.
• Their pushes: the branches under their name in githouse's refs/heads/, and in the ramp (under ▾).
• Then magenta.sh AZ5 --lift, if it was pulled.

If the key is Justin's

That key is root on every server, the only admin, and a merger, so none of the brakes above can be trusted to stop it. What's left is outside the keys: Justin's GitHub account (its own password and second factor), where the org, its members and its deploy keys can be changed; and Hetzner's console, which reaches the servers without SSH. Justin chose no second admin for now (7 Nov).