The outage that logged at debug level

Symptom

An AI-assisted search feature was returning results that looked plausible but were subtly wrong. Ranking fields were missing, an explanatory summary never appeared, and a user-specified result limit was being ignored. Nothing errored. No alert fired. The endpoint returned 200 with a well-formed response every time.

What was actually happening

The search ran in stages: interpret the query, retrieve candidates by vector similarity, then rerank them with a language model. Results were coming back in raw vector order, which meant the reranking stage wasn't running at all.

The cause was two layers down. A shared helper always sent a temperature parameter unless the model name matched a narrow pattern for one family of reasoning models. The configured model belonged to a newer family with the same restriction — it rejects any non-default temperature. So every completion failed with a 400.

And then the real bug: three separate call sites each wrapped that helper in a try/catch, fell back to unranked results, and logged the failure at debug level. Three independent, sensible- looking pieces of defensive code combined into a system that could fail completely while reporting perfect health.

The shape of it, repeated three times in three files:

async function rerank(candidates, query) {
  try {
    return await llm.rerank(candidates, query);
  } catch (err) {
    logger.debug('rerank failed, falling back', err);  // nobody reads debug
    return candidates;                                 // looks like a result
  }
}

Read on its own, that function is defensible. Search degrades instead of dying. Every reviewer who saw it — including me — read it as careful. What none of us did was ask what happens when it fires every single time, in three places at once.

The version I'd write now:

async function rerank(candidates, query) {
  try {
    return await llm.rerank(candidates, query);
  } catch (err) {
    metrics.increment('search.rerank.degraded');   // aggregates, alertable
    logger.warn('rerank failed, returning vector order', { err });
    return candidates;
  }
}

Same behaviour. The difference is that a counter going from zero to every-request is visible on a dashboard, and can page someone. A debug line is visible to whoever is already reading logs, which during a silent failure is nobody.

The second failure mode behind the first

Fixing the parameter surfaced another quiet path: the model sometimes wraps its JSON payload in markdown fences or a sentence of preamble. The parser expected clean JSON, threw, and hit the same fallback. One silent degradation was hiding another.

The fix was to extract the outermost brace-delimited object rather than trusting the response to be bare JSON — treating model output as untrusted input, which is what it is.

What I changed my mind about

I used to think a graceful fallback was strictly better than an error. It isn't. A fallback is a decision to continue in a degraded state, and that decision has to be visible to someone.

• A fallback should emit a metric, not a log line. Logs are read when you already suspect a problem. Metrics are what tell you to suspect one. A counter on "reranking skipped" would have made this obvious within a day.
• Debug level is where failures go to die. If a code path represents "the feature didn't do its job," it is not debug. It is at minimum a warning, and it should be aggregated.
• Integration failures need a single choke point. Three call sites each handling the same provider error independently guaranteed inconsistent behaviour. One wrapper that classifies and counts failures gives you one place to make the decision.
• Test the degraded path, not just the happy one. Every test passed, because every test exercised a mocked client that always succeeded.

The transferable lesson

Silent degradation is worse than an outage. An outage gets noticed and fixed within the hour. A feature that quietly stops working can run for weeks, producing output that is confidently wrong — and in a search or recommendation context, nobody can tell by looking.

When you write a fallback, ask what would have to be true for someone to find out it fired. If the answer is "somebody reads the debug logs," it will never happen.

← All engineering notes