Getting a data structure out of a language model

The task

Turn a résumé — a PDF written by a human with no schema in mind — plus enrichment data from a third-party profile API into a structured record: organisations, roles, dates, skills, all normalised against existing reference data.

The output feeds directly into a database. So "mostly right" isn't a grade; a hallucinated employer or a mangled date range is a data-quality bug that outlives the request.

Why an agent rather than one prompt

A single prompt has to do everything at once: parse the document, decide what's missing, guess at ambiguous entities, and emit valid JSON. It fails at all of them simultaneously and gives you no insight into which part went wrong.

A ReAct-style loop — reason, call a tool, observe, repeat — decomposes it. The model decides what it needs next and calls a tool to get it, rather than inventing it.

The tools were deliberately narrow: parse a document, look up an organisation in existing reference data, look up a job title, fetch enrichment for a profile URL. Each returns real data from the system or a real API. The model's job is orchestration and judgement, not recall.

Things that broke

Document parsing is not one problem

The initial parser handled one format acceptably and others badly. Real users upload whatever they have. Swapping in a parser with broader format coverage fixed more extraction failures than any prompt change did — the model was never the bottleneck; the input was garbage before it arrived.

Output is not JSON just because you asked for JSON

Models wrap payloads in markdown fences, add a sentence of preamble, or emit a trailing explanation. A parser that assumes a clean object throws on all three. Extracting the outermost brace-delimited region is unglamorous and necessary.

Models wrap payloads in markdown fences, add a sentence of preamble, or emit a trailing explanation. A parser that assumes a clean object throws on all three.

function parseObject(raw) {
  try {
    return JSON.parse(raw);              // the happy path, when it happens
  } catch {
    const start = raw.indexOf('{');
    const end   = raw.lastIndexOf('}');
    if (start === -1 || end <= start) {
      throw new Error('no JSON object in model output');
    }
    return JSON.parse(raw.slice(start, end + 1));   // outermost braces
  }
}

Unglamorous, and it eliminated a whole class of intermittent failure. Note it throws rather than returning null when there's genuinely nothing parseable — that's a real error and it should be retried or surfaced, not swallowed. Which is the same lesson as the outage that logged at debug level, learned the expensive way.

Model families have incompatible parameters

Reasoning-oriented models reject sampling parameters that ordinary chat models require. A shared helper that always sends temperature will hard-fail against them — quietly, if the call site catches the error. That specific failure is documented in the outage that logged at debug level.

Ordering isn't free

Extracted experience came back in whatever order the model emitted it, which was not chronological and not stable between runs. Anything a user will read needs an explicit sort applied afterwards — the model is not a source of ordering guarantees.

Design rules I'd keep

• Validate against a schema at the boundary and treat a validation failure as a retryable error, not a parse problem to paper over.
• Prefer lookups to generation for anything that exists in your database. Normalising against reference data beats free-text entity extraction every time.
• Make each tool do one thing with a narrow signature. Broad tools invite the model to pass plausible nonsense.
• Log the full trace — reasoning steps, tool calls, raw responses. When extraction is wrong, the transcript is the only way to find out where it went wrong.
• Treat model output as untrusted input, with the same suspicion you'd apply to a request body from the internet.

A related worker

A separate pipeline sent user-recorded video to a multimodal model for structured feedback. Same principles, different constraints: large payloads, slow calls, and per-call cost high enough that retry policy is a budget decision. The reliability lessons from that one are in the one row that ate the queue.

← All engineering notes