Esta guía está disponible actualmente en inglés y chino.

Developers / JevModel

Evaluate a Jev workflow

Test labels, uncertainty, and real outcomes before automation.

Última actualización:

Build a small labeled set

Collect representative states from the actual workflow, including ambiguous and rare cases. Have reviewers label the intended branch independently. Keep test states separate from examples used to tune instructions.

Fix one question before comparing runs

Choose a question with known labels. After verifying your email and creating a Console API key, POST this JSON to https://jevmodel.app/api/v1/systemone with Authorization: Bearer YOUR_JEVMODEL_KEY and Content-Type: application/json. Replace only state for each test item and retain the exact question wording. The example tests billing versus technical routing; it is not a benchmark result.

{
  "state": "The customer was charged twice and wants an invoice correction.",
  "questions": {
    "queue": {
      "type": "choice",
      "instructions": "Which support queue should handle this request?",
      "criteria": {
        "billing": "Duplicate charges, refunds or invoice corrections.",
        "technical": "Application errors, API failures or broken functionality."
      }
    }
  }
}

Export answers and compare them with labels

In Console → Batch, upload CSV, TXT or JSONL inputs and use the same question for every row. Export the row-level answers, then join them to your independently reviewed labels in a spreadsheet or script. Keep input identifiers stable and record the request wording, date, model path and failures. Batch currently does not calculate an accuracy report or train a model. With an illustrative 24 correct labels out of 30, accuracy is 80%; that small example says little about rare classes or future traffic.

Check probabilities and flow outcomes separately

An illustrative Choice result with billing: 0.90 and technical: 0.10 prefers billing under your definitions; one answer cannot establish calibration. Across enough held-out cases, compare each probability band with the observed correct fraction. For a saved flow, also check which nodes ran, the final outcome and whether uncertain cases reached review. Each executed question uses the normal run and token rules, so report per-flow cost and latency as well as label quality. Error and fallback handling remain your application’s responsibility.

Measure decisions, not just outputs

Check accuracy by class, calibration by probability band, fallback rate, and downstream outcomes. Choose thresholds based on the cost of each error. Re-evaluate when the question wording, model version, or input distribution changes.

Jev probability, confidence and human-review thresholdsSentiment Analysis API in 2026: Cloud APIs vs LLM Prompts vs a Typed Decision ModelLLM-as-a-Judge: Free-Form Prompts, Rubric Scoring, or a Typed Judge with Abstention

JevModel es independiente y no está afiliado a TypeSafe AI.