LLM-as-a-Judge: Using One Model to Grade Another (and Where It Falls Apart)

An AI model does not ship with an instruction manual for grading its own output. Give it a task and it hands back an answer, but there is no deterministic test that tells you the answer is right. That is the gap the technique called LLM-as-a-judge tries to fill: you use one model as a grader for another model's work.

The name comes from automated software testing, where the idea got a careful treatment in a German-language interview between the consultant and podcaster Richard Seidl and Anupam Krishnamurthy, head of AI testing at TestSolutions. Krishnamurthy's central point is that the usual foundation of test automation, determinism, does not hold for large language models. Run the same prompt twice and you can get two different answers, so the classic assertion that output equals an expected value falls apart. You need a different way to check quality.

His second point is worth stealing even if you never write an automated test in your life. When an AI system fails, most of the error is usually not in the part that generates the answer. In a retrieval-augmented system, the mistakes tend to live in the retrieval step, the part that decides which source material the model is allowed to look at before it produces anything. Bad sources poison everything downstream, and no amount of clever generation fixes it.

The same logic maps onto image generation

When a generated render misses the brief, the failure is almost always earlier than you think. Either the model misread the brief in the first place, or it understood a perfectly good brief and rendered it badly. These are different problems with different fixes.

A prompt that was interpreted wrong needs rephrasing, not a new seed. A brief that was understood and then mangled needs a different model, a different aspect ratio, or more reference material. If you only ever react to the final image, you cannot tell which stage failed. That is the whole value of treating generation like a pipeline you can inspect stage by stage.

What a judge model actually does

A judge model takes the original brief and the model's output, and scores the output against a list of criteria you define ahead of time. Instead of asking "is this image good," you hand the judge a rubric and ask for a scored verdict on each line.

A useful rubric for an image brief might read something like:

• Does the output contain every required subject that the brief named?
• Is the composition the one requested (wide, portrait, close-up)?
• Are the lighting and color direction consistent with the style reference?
• Are there physical or anatomical failures that a human would notice at a glance?
• Does it read as a single coherent scene rather than a collage of unrelated pieces?

Because criteria like composition, lighting, anatomy, and scene coherence are visual, the judge has to be a vision-capable model (or a validated image evaluator) that actually receives the image — a text-only LLM grading from a description cannot assess them. The judge checks each line, cites where in the image it made its call, and returns a pass or a specific failure. That is the practical skill: a good judge prompt gives the grader real criteria, not a vibe check.

Where the judge itself is unreliable

The technique has a documented weak spot, and it is the same reason it works at all. A judge model rewards the same fluency it was trained to produce. It tends to grade polished, confident output favorably even when that output quietly misses the point, an echo of the reward-hacking pattern where a model games a proxy for quality instead of satisfying the real requirement.

That makes judge models good at catching literal failures and bad at deciding matters of taste. A judge will often flag a missing hand, wrong anatomy, a garbled composition, or a subject that simply is not there — but that is not a guarantee. Judge bias and uneven multimodal robustness can produce both false passes and false fails, so calibrate the judge against human-labeled examples and route low-confidence or high-impact decisions to a human reviewer. It will not tell you whether a style choice suits the piece, whether a palette feels right, or whether an idea landed emotionally. For those, your own eye is still the only judge that matters.

The lesson is to use each kind of check where it is strong. Route the objective, checkable criteria to a judge model so you stop burning attention on them. Keep the subjective calls for yourself. Automation handles the tasks where there is a verifiable right answer, and it leaves you with more energy for the decisions where there is not one.

One habit worth keeping

Whatever tool you use, name the failure before you blame the model. When a result is wrong, ask which part of the chain produced the error: the interpretation of the brief, or the rendering of it. That single question turns a vague sense that "the AI got it wrong" into a precise place to intervene, which is exactly what testing teaches and what ad hoc prompting rarely does.