A growing body of AI research relies on large language models to evaluate other AI outputs—a practice called LLM-as-a-Judge. According to a new paper on arXiv ("Does the Judge Prefer English?"), this approach is now widely used for open-ended instruction-following evaluation because it is convenient, scalable, and often more semantically aware than older reference-based scoring methods.
But researchers are raising a pointed concern: these AI judges may not be neutral arbiters. The study evaluates what it calls "language-switching invariance"—essentially, whether an LLM judge awards higher scores when a response is written in English versus another language, even if the original question was posed in that other language. The implication is that a model answering in English could receive a favorability bump that has nothing to do with the quality of its answer.
A separate paper, also on arXiv, compounds the concern from a different angle. The C2-Faith benchmark finds that LLM judges struggle to reliably assess whether a model's chain-of-thought reasoning is actually faithful—that is, whether the stated reasoning genuinely led to the answer—rather than simply checking whether the final answer sounds plausible.
Together, the two studies suggest that automated AI evaluation, while efficient, may carry systematic blind spots: a bias toward English-language outputs and a tendency to reward convincing-sounding answers over sound reasoning processes.
This matters because LLM-as-a-Judge is rapidly becoming the default way the AI industry measures model quality—meaning hidden biases in the judge could quietly shape which models get ranked highest and, ultimately, which ones get deployed.