Why implementing ActivityPub is hard, and why it doesn't have to be
A quiet failure
Picture the moment your server sends its first Follow activity to Mastodon. You read the spec, built the JSON, signed the HTTP request, and POSTed it with care. What comes back is a single line: 401 Unauthorized. No body. No explanation.
What went wrong? Maybe the clock behind your Date header drifted a few minutes. Maybe the hash in your Digest header is off. Maybe you uppercased the (request-target) pseudo-header while building the signing string, or published your public key as PEM where the other side wanted multibase. The remote server won't tell you. So you start reading someone else's server code to debug your own.
I know, because I've been there. Fedify began as a casualty of another project. I set out to build a single-user microblogging server, the one that would later become Hollo, and started implementing ActivityPub from scratch. Somewhere between the signature specs and the JSON-LD, the protocol work swallowed the product, and I put the whole thing down. What I picked back up wasn't the app. It was the framework the app should have had. Fedify shipped first; only then could Hollo exist, built on top of it. (I've told this story at more length in A year with the fediverse.)
ActivityPub development gets hard in a few very specific places. In this post I want to walk through five of them, then show what each one looks like with Fedify. If you've spent time in the fediverse, you'll probably nod along. If you haven't, you may wonder why anyone would do all of this by hand. Either way, the conclusion is the same: nobody has to anymore.
Five scenes
Scene 1: there is more than one standard
ActivityPub servers authenticate each other with HTTP signatures. Except there isn't one signature spec. Most of the fediverse runs on draft-cavage-http-signatures-12, an expired draft that never became a standard. The actual standard exists too: RFC 9421, HTTP Message Signatures. The problem is that you can't know which one a given server accepts until you try.
A real-world implementation therefore has to sign with one spec, see whether it gets rejected, re-sign with the other, and remember per server which one worked so it can skip the dance next time. The fediverse calls this double-knocking. Yes, you get to implement it yourself.
That's still not the end. HTTP signatures only prove who sent a request. For situations like inbox forwarding, where you relay an activity you received to a third party, you need signatures that live on the document itself: Linked Data Signatures and Object Integrity Proofs. Four signature mechanisms in total, and two kinds of keys to manage: RSA and Ed25519.
Scene 2: one document, many shapes
ActivityPub's wire format is JSON-LD, and in JSON-LD the same document can take many shapes. This is easier to show than to explain. Here is a Create activity one server might send:
{
"@context": "https://www.w3.org/ns/activitystreams",
"type": "Create",
"actor": "https://example.com/users/alice",
"to": "https://www.w3.org/ns/activitystreams#Public",
"object": {
"type": "Note",
"id": "https://example.com/notes/123",
"content": "Hello, fediverse!"
}
}
And here is a semantically identical activity from another server:
{
"@context": ["https://www.w3.org/ns/activitystreams"],
"type": "Create",
"actor": {
"type": "Person",
"id": "https://example.com/users/alice",
"preferredUsername": "alice"
},
"to": ["as:Public"],
"object": "https://example.com/notes/123"
}
actor turned from a URI string into an inline object. to turned from a string into an array. object went the other way, from an inline object to a URI. Even the address that means “public” has three valid spellings: https://www.w3.org/ns/activitystreams#Public, as:Public, and plain Public. Your parser has to accept every combination, and which one arrives depends on the sender's implementation.The spec-compliant answer is to normalize every document with a JSON-LD processor, expansion followed by compaction. In practice many implementations treat it all as “just JSON” and quietly break on whatever shape some server happens to emit. Either way, you end up with defensive code smeared across the whole codebase: is this a string? An array? An object? A URI I have to fetch?
Scene 3: the zombie post
A user publishes a post, spots a typo, and deletes it right away. Your server sends a Create, then a Delete. Thanks to network weather, some receiving server gets the Delete first and the Create second. It ignores the deletion of a post that doesn't exist yet, then dutifully processes the creation of a post that was already deleted. That post now lives on that server forever, while its author believes it's gone.
Then there's scale. With five thousand followers, one post means thousands of HTTP deliveries. Do that inline in the request handler and your publish button takes half a minute to respond, or the server falls over. Fine, use a queue. Deliveries fail, so retry them. On what schedule? Exponential backoff. How many times? And is a 500 Internal Server Error the same kind of failure as a 410 Gone? When do you clean up three thousand followers on a server that no longer exists? Should you keep hammering a host that has been down for days?
At some point it dawns on you that this is no longer protocol implementation. It's distributed systems engineering.
Scene 4: it's not a spec, it's an ecosystem
Even perfect spec compliance doesn't buy you interoperability. A few examples from the field:
- Mastodon's secure mode requires HTTP signatures on
GET requests too (so-called authorized fetch). Now suppose both servers run in that mode. To fetch the other side's public key you must sign your request; to verify your signature, the other side must first fetch your key. Deadlock. The community's workaround is to sign with an “instance actor” that represents the server itself. You won't find that in the spec. - Threads can't parse activities whose actor is embedded as an inline object. When sending to Threads, the actor has to be a URI.
- Lemmy silently rejects
Group actors that lack fields Mastodon never asks for, such as a moderators collection linked via attributedTo and a featured collection. - Misskey carries vocabulary extensions of its own; quote posts alone go by three different property names across implementations.
The list keeps growing. Interoperability here is not something you finish once and stop thinking about. It's maintenance, forever.
Scene 5: insecure by default
Build it from scratch, and you start out wide open. Skip signature verification on incoming activities and anyone can inject a forged Follow or Delete. Leave the document loader unrestricted and a malicious activity can point it at http://169.254.169.254/ or your internal network, turning your server into an SSRF proxy. Skip origin checks on embedded objects and any server can hand out a document claiming “here's what the Mastodon lead developer said.”
What these traps share is that nothing happens when you fall into them. Everything appears to work. Until someone exploits it.
Ghost ran into this too
If you're thinking “surely our team would manage,” consider Ghost: a leading open-source publishing platform used by thousands of journalists and creators, and a team that set out to build its own ActivityPub support.
We can definitely attest to the problems that Fedify is working hard to solve, because even in just a few weeks of early prototyping we were running into the issues described above right away.From Alright, let's Fedify
Ghost ended up building its ActivityPub layer on Fedify.
So I put all of it in a framework
Fedify is a TypeScript library for building federated server apps on ActivityPub and the standards around it. It runs on Deno, Node.js, and Bun, and supports edge runtimes like Cloudflare Workers. The design goal hasn't changed since the beginning: keep everything in those five scenes out of application code.
Here are the same five scenes again, this time with Fedify.
Scene 1, revisited: the signature war is the framework's job
Here is everything it takes to put one actor on the fediverse:
import { createFederation, generateCryptoKeyPair, MemoryKvStore } from "@fedify/fedify";
import { Endpoints, Person } from "@fedify/vocab";
const federation = createFederation<void>({
kv: new MemoryKvStore(), // Swap for Redis, PostgreSQL, etc. in production
});
federation
.setActorDispatcher("/users/{identifier}", async (ctx, identifier) => {
if (identifier !== "alice") return null;
const keyPairs = await ctx.getActorKeyPairs(identifier);
return new Person({
id: ctx.getActorUri(identifier),
preferredUsername: identifier,
name: "Alice",
inbox: ctx.getInboxUri(identifier),
endpoints: new Endpoints({ sharedInbox: ctx.getInboxUri() }),
publicKey: keyPairs[0].cryptographicKey,
assertionMethods: keyPairs.map((keyPair) => keyPair.multikey),
});
})
.setKeyPairsDispatcher(async (ctx, identifier) => {
// In real code you'd persist these in a database; this shows the gist
return [await generateCryptoKeyPair()];
});
The moment this code runs:- Every outgoing request gets signed. With an RSA key, Fedify emits HTTP Signatures and Linked Data Signatures; add an Ed25519 key and it attaches Object Integrity Proofs as well. All four mechanisms coexist on a single activity, and each receiver verifies with the strongest one it understands.
- Fedify does the double-knocking for you: first contact goes out as RFC 9421, a rejection triggers a draft-cavage retry, and the winning spec is cached per server. If the rejection carries an [
Accept-Signature challenge] (RFC 9421 §5), Fedify reads it and re-signs with exactly the components the server asked for. - Incoming signatures are verified before your code sees anything. An activity that fails verification never reaches your listeners.
- One bonus. Because you registered an actor dispatcher, you now have a WebFinger (RFC 7033) server, for free. Type
@alice@example.com into Mastodon's search box and your actor comes up. You never wrote a line of WebFinger code.
Scene 2, revisited: types instead of JSON-LD
Fedify ships about eighty classes covering the whole Activity Vocabulary plus the major vendor extensions. The classes are typed and immutable, and their accessors absorb the shape differences that JSON-LD allows.
const actor = await ctx.lookupObject("@hongminhee@hollo.social");
if (actor instanceof Person) {
console.log(actor.name); // Safe whether it's a string or langString
const followers = await actor.getFollowers(); // Fetches a URI, unwraps an object
}
[lookupObject()] takes a handle and runs the whole chain for you, WebFinger discovery included. Accessors like getFollowers() behave the same way whether the value is a URI reference or an inline object, and fetched values are cached.Vendor fragmentation gets stitched up here too. The three competing quote properties (quoteUri, _misskey_quote, quoteUrl) are unified behind one API, next to the emerging FEP-044f quote. Misskey's isCat property exists as a type, so your server can determine cat-ness with full type safety. It sounds like a joke, but a few dozen details of exactly this kind are what interoperability is actually made of.
Scene 3, revisited: the zombie post dies in one line
Delivery infrastructure first. Plug a message queue into createFederation() and delivery moves to the background, with automatic retries under exponential backoff (up to ten attempts by default). When a post goes to thousands of followers, two-stage fan-out kicks in: a single consolidated message enters the queue, and a background worker splits it into per-server delivery tasks. The publish button responds immediately.
Retries create a problem of their own: the same activity can arrive twice. Fedify keeps a 24-hour idempotence cache of processed activities, so duplicates get detected and skipped before they reach your handlers.
As for the zombie post, the fix is one option:
await ctx.sendActivity(
{ identifier: "alice" },
"followers", // Collects recipients from your followers collection
deleteActivity,
{ orderingKey: post.id }, // Same key = in-order delivery per server
);
[Activities that share an orderingKey] are delivered to each receiving server in the order they were sent. A Delete can no longer overtake its Create. Activities with different keys still go out in parallel, so throughput survives.Fedify also handles dead servers. On a 404 Not Found or 410 Gone, it stops retrying and calls a handler you register. If the delivery went to a shared inbox, you also get the list of followers behind it, so you can prune vanished accounts on the spot. Hosts that fail repeatedly trip a per-host circuit breaker that holds deliveries and probes periodically until the host recovers. It's on by default; there's nothing to configure.
Scene 4, revisited: we track the quirks so you don't
Here is how Fedify disarms the traps from scene 4:
- Authorized fetch: chain [
.authorize()] onto a dispatcher and the verified identity of the requester lands in your callback. Blocklists, private collections, whatever your app needs is plain application logic. The instance-actor deadlock has a supported pattern as well. - Threads and inline actors: an activity transformer, enabled by default, rewrites inline actors into URIs on the way out. You don't need to know Threads has this problem.
- Lemmy's requirements: the custom collection API exposes a moderators collection in a few lines, and Lemmy's JSON-LD context ships preloaded.
When a new quirk surfaces in the wild, the fix lands in Fedify, not in every application separately. Each interoperability lesson gets learned once.
Scene 5, revisited: becoming unsafe takes effort
Fedify's defaults point the other way.
- Signature verification is something you turn off (for tests), not something you remember to turn on.
- The document loader refuses private address ranges and loopback out of the box, with DNS rebinding accounted for. To open yourself up to SSRF you have to flip an option whose very name announces it's for testing.
- When an embedded object's origin differs from its parent document's, the accessor refuses to trust it and re-fetches from the source (based on FEP-fe34). Content spoofing is stopped at the property access level.
In a from-scratch implementation, you have to keep remembering to do things safely. In Fedify, the unsafe path is the one that takes deliberate effort. For a federated server, with its tangle of trust boundaries, that's the right way around.
Your stack stays your stack
“Fine, but what if it doesn't fit our stack?” Fedify was built to fit the stack you already have. There are thirteen web framework integrations: servers like Express, Hono, Fastify, Koa, NestJS, and Elysia, and meta-frameworks like Next.js, Nuxt, SvelteKit, Astro, SolidStart, and Fresh. Middleware handles content negotiation, so the same URL in your existing app serves HTML to browsers and JSON-LD to the fediverse.
Fedify doesn't dictate your database either. For its own storage it asks for one key–value interface, with seven adapters available (Redis, PostgreSQL, MySQL/MariaDB, SQLite, Deno KV, Cloudflare Workers KV, in-memory). Message queues come in eight flavors (PostgreSQL, Redis, AMQP/RabbitMQ, and so on), and you can implement the interface yourself if none fits. Your domain data stays in whatever database and ORM you already use.
Already running federation on another library? There are migration guides with data migration scripts for moving from activitypub-express and friends without losing your existing followers.
The core isn't the ceiling, either. Higher-level packages build on it: [@fedify/relay] gives you a complete ActivityPub relay server in a single function call, and [@fedify/backfill] reconstructs incomplete conversation threads by walking the rest of the fediverse for you.
Tools for the whole development loop
A quieter misery of federated development has always been the missing tooling. Fedify comes with tools for every stage of the loop.
[fedify init] scaffolds a project in one line, and fedify tunnel exposes your local server over HTTPS so you can test against real Mastodon. Activities your server sends can be received by fedify inbox, a disposable inbox server spun up on the spot; whatever other servers publish, you can inspect with fedify lookup. My personal favorite is fedify lookup --authorized-fetch, which generates a one-off key pair and stands up a temporary ActivityPub server just to make a signed request for an object behind secure mode. The CLI is also useful to ActivityPub developers who don't use Fedify at all.
While you write code, an ActivityPub-specific linter (@fedify/lint) catches twenty kinds of interoperability bugs, like an actor missing its inbox. Tests run without the network using mocks from [@fedify/testing]. Once the server is up, attach the debug dashboard (@fedify/debugger) with one line and watch activities and signature verification results in your browser, live. In production there's built-in OpenTelemetry instrumentation (28 span types, 37 metrics) plus a monitoring guide, and when performance matters, [fedify bench], a load-testing tool built for ActivityPub, catches regressions in CI.
As far as I know, no other ActivityPub framework ships even one of the tools in this section.
The documentation is part of the tooling. The official docs run to a thirty-chapter manual and five tutorials, and they go well past API listings. There's an operations chapter with ready-made PromQL queries and alerting rules for watching your queue backlog, and a field-guide chapter that documents de facto conventions, like which property makes your avatar show up in Mastodon, with screenshots. At two in the morning, when federation is broken and you don't know why, this is the difference between a bad night and a short one.
It's already running
Fedify is not a thought experiment. Ghost's ActivityPub service, mentioned above, is built on it. So are Encyclia, which bridges ORCID researcher records into the fediverse; SiliconBeest, running serverless on Cloudflare Workers; Typo Blue, a Korean blogging platform; Hollo, my own single-user microblogging platform; and Hackers' Pub, run by its community. Hollo, by the way, is the app from the beginning of this post: the project I once had to shelve, finished at last on the framework it forced into existence.
The tutorials give a concrete sense of scale. They walk you from a single-file server, a few dozen lines, that Mastodon can follow, through an image sharing service in roughly 750 lines that fully interoperates with Pixelfed (follows, likes, comments), up to a community platform federating both ways with the real lemmy.ml.
The fediverse needs more apps
I didn't build Fedify to mint more ActivityPub experts. Rather the opposite. I believe the fediverse will only grow beyond microblogging when developers can build federated apps without knowing ActivityPub's fine print. Signature spec transitions and JSON-LD compaction are problems that belong inside a framework, not barriers in front of someone with a new idea.
Starting takes one line:
npm init @fedify
Follow the first tutorial and by the end, Mastodon can find your server. If you get stuck, come find us in the Matrix room or GitHub Discussions. See you in the fediverse.
Problem: It's the end of a long week and your soul is tired from the endless monotony of slowly but surely conforming your behaviour to accommodate social acceptability as defined by our algorithmic overlords.
Ghost (Building ActivityPub)
Ich habe mal eine Frage betreffs Federation. Und zwar habe ich gesehen das ich in Sachen Beiträge vor allen via Lemmy Feedback bekomme, aber in den allermeisten Fällen kaum aus dem Rest des Fediverse. Dabei ist mir klar das ich wenn ich in eine Lemmy Community poste, ich recht viele Leute erreiche. Aber das erklärt eben nicht warum aus dem Rest des Fediverse wo ich sehr viel mehr Kontakte habe die mir folgen so wenig zurück bekomme.
