Every unfollow is a delete you have to remember

Two ways to build a feed

Fan-out on read: when a user opens the app, query everyone they follow, merge, sort, return. Simple, always correct, and it degrades badly — the work grows with how many people you follow and how much they post, and it happens while someone is staring at a loading spinner.

Fan-out on write: when something is created, immediately write a row into the feed table of every account that should see it. Reads become a single indexed query over one collection. The cost moves to write time, where nobody is waiting.

We built the second. It's the right call for a feed people open constantly and post to rarely — but it changes the shape of every subsequent problem.

The feed row is a denormalised copy

A materialised feed row doesn't just point at the source entity. It carries a snapshot: who created it, when, what kind of thing it is, which hashtag or group brought it into this feed, and whether the viewer is in the relevant circle.

AccountFeedItem {
  account          // whose feed this row belongs to
  entityId, type   // what it points at
  entityName
  entityCreatedAt  // denormalised so ordering needs no join
  entityCreatedBy
  hashtag          // why it's here, if via a tag
  inViewerCircle   // viewer's relationship at write time
  status
}

That denormalisation is the whole point — reads touch one collection with no joins. It's also the whole problem, because a copy can go stale in ways a pointer can't.

Every write has a paired delete

This is what the architecture actually costs. The workers come in pairs:

on-follow-account       ⇄  on-unfollow-account
on-follow-tag           ⇄  on-unfollow-tag
on-action               ⇄  on-undo-action
on-entity-creation      ⇄  on-entity-delete
on-adding-collaborator  ⇄  on-block

Follow someone and you should retroactively see their recent history — so following isn't just an edge in a graph, it's a backfill job writing their past entities into your feed. Unfollow and all of those rows have to come back out.

Miss one half of a pair and the failure is quiet and permanent. A missing create means a feed that looks emptier than it should, which nobody reports as a bug. A missing delete means content from someone you deliberately unfollowed, or blocked, still sitting in your timeline — which people absolutely do report, and which reads as broken trust rather than a stale cache.

The case that catches everyone

Visibility changing after the fact.

An entity is created public. It fans out to thousands of feeds. Later its author makes it private, or restricts it to a group. The source record updates in one write — and every materialised copy is now a leak.

So there has to be a path that walks the feed collection and removes rows for entities whose visibility no longer permits them. Same for blocking: block someone and their content must disappear from your feed in both directions, retroactively.

Doing it off the request path

None of this can happen inside the request that triggered it. Following someone must not block on writing a few hundred feed rows.

The fan-out runs on an event bus with a queue-backed implementation for production and an in-memory one for tests — same interface, so the test suite exercises the real code path without a broker.

// The request writes the edge and returns.
await follows.create({ follower, followed });
await eventBus.emit('account.followed', { follower, followed });

// A worker does the expensive part, and is allowed to fail loudly
// without taking the user's request down with it.
class CreateFeedItemsOnFollowWorker {
  async run({ follower, followed }) {
    try {
      await this.feedItems.backfillFrom(followed, follower);
    } catch (e) {
      this.logger.error(`feed backfill failed: ${e}`);
    }
  }
}

That catch deserves scrutiny — it's exactly the shape I've argued against in the outage that logged at debug level. Here it's defensible because a failed backfill degrades one user's feed rather than corrupting state, and the alternative is an unhandled rejection taking down the worker. But it should be emitting a metric, not just a log line. The same criticism applies.

What I'd want before scaling this further

• A reconciliation job. Fan-out on write drifts. Something should periodically compare a sample of feeds against what they ought to contain and report the delta — the same argument as reconciling payments.
• A ceiling on fan-out. Writing to every follower is fine at a few hundred and pathological at a million. High-follower accounts eventually need fan-out on read, blended at query time. Most large social products end up hybrid for exactly this reason.
• Idempotent workers. Queues redeliver. A backfill that runs twice should not duplicate rows.

Was it the right choice?

For this product, yes. Read volume dwarfs write volume, feeds are the most-opened surface, and the latency budget for opening an app is unforgiving.

But the honest summary is that fan-out on write trades a hard read problem for a large number of small write problems, each of which is easy and none of which you can skip. The architecture isn't difficult. Remembering every place the graph can change is.

← All engineering notes