Keeping embeddings fresh without blocking a request

Why freshness is the whole problem

Vector search matches on meaning rather than keywords, which is a real improvement when the underlying records are rich and current. The failure mode is invisible: if embeddings drift out of date, search keeps returning confident, well-ranked, wrong results. Nothing errors. Quality just decays.

The records being indexed were composites — a profile plus several related collections that all contributed signal. Any of those could change at any time, and each change invalidated the embedding.

One authoritative write path

The first decision was structural: exactly one place in the codebase writes an embedding.

Without that rule, embeddings get written from wherever someone happened to need one — a controller here, a migration script there — each assembling the source text slightly differently. You end up with vectors in the same index that were built from different inputs, and similarity comparisons between them become meaningless. Consolidating the write path makes the composition rule enforceable in one place.

Two modes, deliberately

Batch

A worker that walks the whole population to backfill new fields, recover from failures, and refresh periodically. This is the safety net: whatever the incremental path misses, the batch pass eventually corrects.

Real-time incremental

Changes enqueue an indexing job on a durable queue backed by Redis, processed outside the request path. The user's write returns as soon as their data is saved; embedding generation happens after.

This matters because embedding calls are slow and externally dependent. Putting one in a request handler means every profile save waits on a third-party API — and fails when that API does.

Debouncing is a cost control

Someone editing their profile generates a burst of writes — a dozen saves in two minutes as they work through a form. Naively that's a dozen embedding calls to produce one final state that a single call would have captured.

Debouncing collapses the burst into one job after activity settles. It's usually framed as a performance optimisation, but here the dominant motivation was different: every re-index is a billable API call. Without debouncing, cost scales with how indecisive your users are.

Most queue libraries give you this directly — a stable job id per entity plus a delay, so a re-enqueue inside the window replaces the pending job instead of adding to it:

await indexQueue.add(
  'reindex',
  { accountId },
  {
    jobId: `reindex:${accountId}`,   // same entity ⇒ same job
    delay: 30_000,                   // settle window
    removeOnComplete: true,
    attempts: 3,
    backoff: { type: 'exponential', delay: 5_000 },
  },
);

Twelve saves in two minutes become one embedding call. The window is a judgement call: too short and you're back to paying per keystroke, too long and search lags behind reality in a way users notice.

Failure handling

• Validation before enqueue — reject records that can't produce a meaningful embedding rather than discovering it inside the worker.
• Bounded retries — transient provider errors are common; infinite retries are how you turn one bad record into a permanent outage.
• Dead-letter queue — items that exhaust retries are kept and inspectable. A silently dropped index update is a search-quality bug nobody can trace back.

What I'd add

Observability on index staleness. The system was built for freshness but had no direct measurement of it. The metric I'd want is the distribution of time between a record changing and its embedding being updated — because that number degrading is the earliest possible signal that search quality is about to.

← All engineering notes