LLM-as-a-Judge: Free-Form Prompts, Rubric Scoring, or a Typed Judge with Abstention

LLM-as-a-judge is an evaluation pattern: a model reads a task, the output of the system under test, and sometimes a reference answer, then returns a grade. The research that named the pattern (Zheng et al., 2023, “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”) describes three forms—pairwise comparison, single-answer grading, and reference-guided grading. Whatever the form, a judge measures agreement with the criteria you wrote, as interpreted by another model. It does not measure truth, user satisfaction, or business outcome unless your criteria and your human labels connect to those things. That is why the judge itself must be evaluated like any classifier. This page is about judging other AI outputs—chatbot answers, summaries, RAG responses, agent steps. To evaluate a JevModel workflow itself against labels, see evaluation.

The three options differ mainly in how stable the output is and whether uncertainty is visible. Free-form prompts are quick to start; rubrics make grades comparable across runs; a typed judge adds a fixed schema and probabilities you can threshold. The comparison is about engineering trade-offs, not a ranking of judge models.

Zheng et al. (2023) report three biases worth designing around. Position bias: in pairwise comparison the verdict can flip when the two answers swap places, so run each pair in both orders and count a win only when both orders agree; otherwise record a tie. Verbosity bias: judges can prefer a longer answer even when the extra length adds nothing, so write criteria that reward meeting the requirement, not coverage, and include short correct answers in your fixtures. Self-enhancement bias: a judge may favor outputs that resemble its own, so do not let the model being tested grade itself and hide which system produced each answer. Two more habits help: grade one criterion per question instead of an overall “quality” score, and for factual tasks give the judge a reference answer, because a judge can be misled by a confident wrong answer it would otherwise have solved.

Sample real outputs, including the hard ones, and have two people label them blind against the same written criteria. Where the two people disagree often, fix the criteria before blaming the judge—humans who cannot agree give the judge nothing stable to match. Then run the judge on the same items and report agreement per class, not only overall: a judge that passes everything looks accurate when most outputs are fine. A confusion table of judge versus human, plus a chance-corrected statistic such as Cohen’s kappa if your team uses one, shows where it fails. Next, check probability bands: among items the judge marks with high probability, how many did humans also mark that way? Pick an auto-accept band and a review band from that table. Freeze the sample as a regression set and re-run it whenever the judge prompt, the criteria, or the model changes. JevModel’s Console → Batch can run one fixed judge question over uploaded rows and export the answers for this join.

Put the task, an optional reference, and one candidate output in state, then ask one question per criterion. In JevModel, Noul is a yes/no question that returns a probability from 0 to 1, which suits binary checks such as “is every claim supported by the provided context?”. Score takes 2–10 ordered level descriptions and returns a value on that scale, which can fall between levels; it suits graded criteria such as helpfulness, but it is not a 0–1 probability, so set your own bands for it. A request can carry 1–8 named questions, and the serialized state and full request must each fit within 8,000 characters, so long answers or retrieved passages must be excerpted. For pairwise comparison, send two requests with the candidates in opposite order and accept only consistent results. A Noul probability near the middle, or a Score that lands between your pass and fail levels, is your abstention signal: route those items to a person rather than counting them. Calls go to POST /api/v1/systemone and use the fixed model typesafe/jev-1.13 through OpenRouter.

{
  "state": {
    "task": "Answer the customer using only the help-center context.",
    "context": "Refunds are available within 30 days of purchase for annual plans.",
    "candidate_answer": "You can get a refund within 30 days if you are on an annual plan."
  },
  "questions": {
    "grounded": {
      "type": "noul",
      "instructions": "Is every factual claim in candidate_answer supported by the context?",
      "criteria": {
        "true": "All claims appear in or follow directly from the context.",
        "false": "At least one claim is missing from or contradicts the context."
      }
    },
    "helpfulness": {
      "type": "score",
      "instructions": "How well does candidate_answer resolve the task? Do not reward length.",
      "criteria": [
        "Does not address the task or is misleading.",
        "Addresses part of the task or needs a follow-up.",
        "Fully resolves the task in the needed detail."
      ]
    }
  }
}

Skip the judge when code can check the answer: valid JSON, exact match against an expected label, a unit test passing, a SQL query returning the right rows. Deterministic checks are cheaper and do not drift. Do not use a judge as the only sign-off for legal, medical, financial or safety decisions; use it to prioritize human review. Do not trust a judge you have never compared with human labels, and do not let a model grade its own outputs. Be careful with math, code correctness and multi-step reasoning, where judges can be misled by the answer they are grading; prefer reference answers or execution-based tests. JevModel in particular is a decision model for text and JSON: it does not run code, browse, or read images, so it judges what is written in state, not what happens when the output is executed. It is not open source, and free verified accounts get 5 successful runs per UTC day—enough to try a judge, not to grade a large eval set daily.