LLM annotation now: from preference pairs to agent traces

Five years ago the model-feedback request was almost always the same: here are two answers, tell us which is better. Pairwise preference was the whole job, and the people doing it needed careful reading and not much else.

That request has not disappeared, but it is no longer the interesting part of the queue. Most of what we now get asked for is judging something that has steps, against criteria somebody wrote down in advance. The change is worth describing plainly, because it changes who can do the work, what a project costs, and what can go wrong.

The shift in one sentence

Preference data asks which output is better. Rubric and trace work asks where a process went wrong. Those are different questions, and only the second one tells an engineer what to fix.

What is actually being asked for

Agent traces, judged step by step

This is the newest work and the fastest-growing part of what we are asked to staff. A model is given a goal, calls tools, reads what comes back, and calls more tools until it produces an answer. The output can be correct while the run was a mess: it searched three times for something it already had, it silently swallowed an error, it invented a parameter the API does not take, it did the right thing for the wrong reason.

Scoring only the final answer misses all of that. So the work is per-step: was this call necessary, was the argument right, did the model correctly read the result, was the recovery from that failure sensible. An annotator has to hold the whole run in their head and judge the reasoning, not the prose.

That is a genuinely harder task than reading two paragraphs and picking one. It needs someone who can read an API response and know whether the model understood it.

Rubrics instead of taste

The second shift is away from a single holistic judgement toward explicit criteria, scored separately: instruction-following, factual accuracy, tone, completeness, safety, and whatever else the product actually cares about.

The reason is not fashion. A holistic score is unauditable. When two annotators disagree you cannot tell whether they disagreed about the facts or about the tone, so you cannot fix the guideline. Split the criteria and disagreement becomes a specific, answerable question. It also means a model developer can see which dimension regressed after a change, rather than watching one number drift.

The practical consequence is that the rubric becomes the deliverable that matters most. A vague rubric produces expensive noise. We would rather spend a day arguing about the rubric than a month labelling against a bad one.

Factuality traced to a source, not asserted

"Is this true" is not a labelling task; it is an argument. "Is this claim supported by the source the model cited" is a labelling task, and it is the one worth paying for.

So the useful shape is claim-level: split the answer into its factual claims, and mark each one supported, contradicted, or unsupported by the cited source. Unsupported is not the same as false, and keeping those separate is most of the value. A model that is accidentally right while citing nothing relevant is still broken.

Safety against a written policy, with severity

Binary toxic-or-not labelling has largely given way to classification against the client's own policy, with a severity level. Two products with the same base model can have entirely different tolerances, and a label that says "unsafe" without saying which rule it breaks and how badly cannot be acted on.

This is also where the domain matters most. What counts as unsafe medical advice is a question for someone who knows the field, not for a generalist reading a policy document for the first time.

Written demonstrations, fewer and better

The last shift is quiet and, we think, the most under-appreciated: demand for written gold answers has held up, and the bar for who writes them has risen sharply. As base models improved, a mediocre demonstration stopped being useful. Writing a target answer that is better than what the model already produces now requires genuine domain knowledge.

The volume for this work is smaller than it was. The rate is higher, because the person has to actually know the subject.

What this changes about staffing

The old preference-pair work rewarded conscientiousness. Most of the new work rewards conscientiousness plus domain knowledge, and in the case of agent traces, technical literacy: the ability to read a tool call and know it is wrong.

That has two consequences worth being blunt about.

The first is that this work cannot be crowdsourced to the cheapest available pool without the quality collapsing, and anyone promising otherwise is selling volume rather than judgement. The second is that recruitment becomes the constraint. For a clinical rubric or a legal factuality task, finding the annotators is the hard part of the project, and it is worth saying so during scoping rather than discovering it in week three.

What has not changed

Two things, and they are the same two that mattered in 2020.

Guidelines still decide the outcome. Every ambiguity in the rubric becomes disagreement between annotators, and every unresolved disagreement becomes noise the model learns. A pilot batch labelled by more than one person is still the cheapest way to find out whether your criteria mean what you think they mean.

And disagreement is still information rather than a fault. When two people who know the subject score the same trace differently, the usual cause is that the rubric did not cover the case. Adjudicating and then amending the rubric is the work; blaming the annotator is not.

If you are scoping this kind of project

The questions worth answering before anyone labels anything: what decision will this data inform, what does the rubric say, who is qualified to apply it, and how will you measure whether two qualified people agree. If the answer to the last one is "we will see", that is the first thing to fix.

The label types we staff for list the specific shapes of this work, including agent trace review, rubric scoring and factuality checks. If your task is not among them, describe it and we will say honestly whether it fits.