The one row that ate the queue

The worker

A scheduled job picked up user-submitted videos, sent them to a machine-learning API for analysis, and stored the feedback. It ran every few minutes and processed a small batch each time.

Selection was a query for items that had no feedback yet, with a batch limit. On success it wrote a feedback record and stamped the source item, which removed it from future selection. Clean enough to pass review.

The asymmetry

On failure it wrote nothing at all.

That single omission meant the item still matched the selection query on the next tick. And because the query had no ordering and a small batch window, a permanently unprocessable item — a corrupt file, one too large for the API, anything that would never succeed — sat at the front of that window forever.

• It was retried every few minutes, indefinitely.
• It consumed a slot in every batch, so healthy items behind it were never reached.
• Each retry was a billable API call against a large media file.

From the outside this looked like the feature had simply stopped. Queue depth was small. No errors escaped. The worker reported that it ran successfully every time — it had, after all, done exactly what it was told.

Reduced to its essentials, the worker was this:

const batch = await items.find({ feedback: { $exists: false } }).limit(3);

for (const item of batch) {
  const ok = await process(item);
  if (ok) {
    await items.update(item._id, { feedback: result });  // now excluded
  }
  // else: nothing written — matches the same query again in 10 minutes
}

The repair is small and the shape generalises — record the attempt, not just the success:

const batch = await items
  .find({ feedback: { $exists: false }, attempts: { $lt: MAX_ATTEMPTS } })
  .sort({ createdAt: 1 })              // oldest first: no permanent squatters
  .limit(3);

for (const item of batch) {
  try {
    await items.update(item._id, { feedback: await process(item) });
  } catch (err) {
    await items.update(item._id, {
      $inc: { attempts: 1 },
      $set: { lastError: String(err) },
    });
  }
}

attempts is what gives failure a terminal state. The sort is what stops any single row from owning the front of the window.

The fix, and the general shape

Immediately: track failures per item and bound the retries, so a terminally failing item drops out of selection instead of blocking the window.

The general pattern this belongs to:

• Failure needs a terminal state. If success is the only path that changes the selection criteria, failure loops by construction.
• Distinguish retryable from terminal. A network timeout deserves a retry; a file the API will never accept does not. Treating them identically guarantees you either give up too early or never give up at all.
• Bound retries and use backoff. Retrying a large media upload every few minutes forever is a cost incident as much as a reliability one.
• Dead-letter, don't discard. Items that exhaust their retries should land somewhere inspectable. "It silently vanished" is a worse outcome than "it silently retried."
• Order the selection. Without a sort, which items get picked is arbitrary — which is how one row acquired permanent squatting rights on the batch.

What the monitoring should have been

Queue depth looked fine throughout — items were being selected constantly. The metric that would have caught this in an hour is queue age: the timestamp of the oldest unprocessed item. That number was growing without bound the entire time.

Depth tells you how much work exists. Age tells you whether work is actually moving. For any worker where items can be skipped or reselected, age is the number that matters.

← All engineering notes