Cryptograss:On The Pipe

On The Pipe is about whether things come out the far end.

Most of what breaks in this system does not crash. It reports something plausible and wrong, or it reports nothing at all — and in several cases the plausible wrong thing pointed at destroying work that was perfectly fine. This page describes the pipe that a video travels down, where it has silently leaked, and the rules that follow from having been caught out.

The pattern

Five separate bugs found in the first week of September 2026 turned out to share one shape.

What actually happened	What the system said
MemoryError stringifies to the empty string	Upload error: — nothing after the colon
Coconut's job.failed webhook carried no error field	Unknown error
A fire-and-forget task's result was discarded	nothing, for months
An RPC call 403'd and fell back to an estimator	nothing — and every block height was ten days wrong
A codec failure fell through to a doomed raw-file fetch	"The IPFS content may not be available"

None of these were crashes. Every one of them was the system being confidently unhelpful, and three of them pointed the reader toward destruction:

• Re-upload 10.5GB — of a video that was already published and seeding
• "safe to delete from wiki" — of the only page pointing at that video
• Debug IPFS — for a problem that was a missing AV1 decoder in Safari

A failure you cannot read is worse than a crash. A crash tells you where it happened.

The pipe

A video travels a long way, and each hop can fail differently.

1. Browser → delivery-kid. The bytes go directly to delivery-kid, never through the wiki. A ReleaseDraft: page is created first so a failed upload leaves a record rather than vanishing.
2. Analysis. ffprobe reads duration, resolution and codecs onto the draft.
3. Encode. AV1 video, Opus audio, HLS in fMP4 segments. See Cryptograss:Video encoding pipeline.
4. Pin. The output directory goes to IPFS and returns a CID.
5. Record. The CID is written to draft.json, to a ledger that survives cleanup, and to the wiki.
6. Publish. A bot creates the Release: page and adds BitTorrent metadata.

Alongside all of it runs a diagnostics chain, whose whole job is to make failure legible:

routes/content.py        _fire_diagnostics_snapshot(state)
                         fired at every terminal state transition
     |
services/pickipedia_client.py    snapshot_diagnostics_for_state_async(state)
     |
services/pickipedia_client.py    snapshot_diagnostics(draft_id, payload)
     |
            ReleaseDraft:<uuid>/diagnostics
     ^
pickipedia, releaseDraft.js      the diagnostics panel reads this page
                                 when the live fetch 404s

That sub-page is the only copy of a draft's logs that survives delivery-kid's storage being rebuilt.

Where it leaked

It had never worked. Not once, for any draft, since the feature shipped. Four things had to be wrong simultaneously, and each hid the next:

• The wiki API was addressed at /w/api.php; PickiPedia serves it at /api.php. Every write 404'd.
• Sub-pages inherited the release-draft-yaml content model, whose validator demands a type field that a log trail has no reason to carry. Every write was rejected.
• The mwclient session was cached in a module global and never re-authenticated, so it expired daily and every save afterwards was refused as logged-out.
• The caller discarded the task result, so all three of the above were invisible.

The tests were green throughout. They asserted that save() had been called, which was true, and said nothing about whether a page appeared, which it never did.

The rules that fall out

Never discard a result you asked for. asyncio.create_task(f()) without a done-callback means f returning False forever is indistinguishable from success.

A fallback must not be silent. When an RPC 403s and an estimator answers instead, the estimator quietly becomes the only code path. Say so, at a level someone will see, and keep saying it.

A placeholder must not claim more than it knows. "Unknown error" asserts that the cause is unknowable. "the service reported a failure with no reason attached" says where the gap is, which is a different and honest thing.

Never blame a subsystem you have not checked. "The IPFS content may not be available" was printed about content being served in half a second. It sent two people to debug storage for a codec problem.

Do the safe thing before the risky one. Deleting a source file the instant a pin succeeds is defensible. Deleting it before the CID is written down anywhere is not — and the ordering was the whole bug.

Test the far end. A test that asserts a function was called passes happily while nothing arrives. The suite now carries TestOnThePipe, which asserts a page lands, with the right content in it.

See also

• Cryptograss:Video encoding pipeline — how a video is encoded and published
• Cryptograss:Delivery-kid — the machine this runs on, and its storage audit
• 44th Bday Cicada Jam — the recordings that surfaced all of this