Compare / JevModel

SemIf vs Jev: Benchmarks, Calibration, and Deployment

Compare SemIf vs Jev using primary sources: option scoring, benchmark limits, CPU and GPU deployment, calibration, and a practical workflow evaluation plan.

Last updated:

The short answer: choose who owns inference

SemIf vs Jev is a deployment and evaluation decision for teams building routing, moderation, or agent checks. A hosted Jev API is a reasonable first trial when your team wants to avoid operating a model. SemIf is worth evaluating when local inference, inspectable scoring, or existing hardware matters more. Neither choice removes the need to label representative cases and decide which errors your workflow can tolerate. This comparison reviews public sources checked on October 2, 2026; it does not report a new head-to-head test.

SemIf is an independent project, formerly OpenJev

Jev is TypeSafe AI’s hosted decision model. SemIf, maintained in TheoLeeCJ/SemIf-OpenJev, is an independent research implementation using open models. It does not release Jev’s weights or reproduce its undisclosed training. Its code has an MIT license; upstream model weights keep their own licenses. The old OpenJev name also appears in razorback16/openjev, a separate decision-server project. Check the repository owner before installing anything: a guide for that server is not a SemIf setup guide, and neither project is TypeSafe’s official Jev release.

Similar decisions, different scoring contracts

SemIf’s direct scorer reads option-token logits from a frozen language model and normalizes them over the declared alternatives. TypeSafe documents three Jev primitives: Choice selects an option, Score evaluates ordered levels, and Noul returns a yes probability. Choice and Score also include confidence. The practical distinction is the contract you integrate: SemIf gives you a scoring implementation to operate, while Jev gives you a hosted typed interface. Mapping a local option distribution into a Jev-shaped response requires an adapter and validation; similar field names alone do not establish interchangeable behavior.

What the 84.5% and 88.3% results actually measure

SemIf’s RESULTS document reports 0.845 for Qwen3.5-4B direct scoring and 0.883 for published Jev results on 102 aligned public rows across 20 cases. The metric is equal-case reference agreement, not a universal accuracy score. Jev’s value was taken from TypeSafe’s public records; SemIf’s authors did not run a live Jev endpoint. This selected subset is not the full 711-row TypeSafe aggregate. The reported gap is 3.8 percentage points within that scope. It cannot tell you the error rate on your tickets or establish identical probability quality.

The 5.21× speedup is against JSON generation

The project reports 1.023 seconds for direct readout versus 5.332 seconds for generating a compact JSON array: the same frozen Qwen3.5-4B, one RTX 3090, one state, 21 binary criteria, and medians of three runs. That is the source of 5.21×. It is not a SemIf-versus-Jev latency ratio. The two paths agreed on 18 of 21 choices, so even this experiment compares output methods with differing decisions. For your application, measure from request submission to usable result, including queues, network overhead, cold starts, and failures.

Calibration exists, but remains workload-specific

SemIf includes post-hoc temperature scaling fitted on labeled workload data. Its CALIBRATION document reports WANLI expected calibration error falling from 0.208 to 0.069 out of fold. Temperature scaling changes confidence without changing the winning option; the authored and Every workloads did not show clearly separated improvement intervals. TypeSafe describes Jev confidence as a statistic of its probability distribution, rather than a guarantee that a particular answer is correct. Do not transplant a 0.8 cutoff between the systems. Fit each operating threshold on calibration data, then compare results on untouched test cases.

Local deployment now includes CPU and Apple Silicon

The current SemIf README documents CUDA, a llama.cpp backend for local GGUF files, and a slower full-precision Torch CPU path. Apple Silicon has PyTorch/MPS and native MLX options. These broaden access, but a supported backend does not inherit the RTX 3090 benchmark timing. The browser demo uses quantized artifacts; its listed quality scores come from native BF16 checkpoints. Start with the backend you can actually maintain, pin the model and runtime, and evaluate that exact configuration. Model download size alone is not a reliable estimate of total runtime memory.

Compare data paths and total cost

Self-hosting can keep inference inside infrastructure you control, provided you also check telemetry, logging, and downloads. A hosted API sends state to its service providers. JevModel is an independent product that currently routes Jev requests through OpenRouter; its API, keys, and pricing are separate from TypeSafe’s direct service. Review the actual provider chain before sending sensitive records. For cost, include API usage and retries on one side, and hardware, idle capacity, power, updates, and engineering time on the other. Neither open code nor an existing GPU makes operations free.

A support-routing example with an explicit fallback

Consider a hypothetical support team choosing between account access, billing, and human review. A customer reports a failed password reset and also asks about an invoice. Include both facts in the state and define which queue owns mixed requests; otherwise either model may make a defensible choice that violates your workflow. The support lead owns labels and escalation rules, engineering owns timeouts and audit records, and security reviews the data path. Pilot in shadow mode: log proposed routes while the existing process remains in control. No customer result or measured improvement is claimed here.

{
  "state": "Password reset failed. The customer also needs an invoice copy.",
  "question": "Which queue should handle the request first?",
  "options": [
    {
      "id": "access",
      "description": "Account access support; owns mixed requests when access is blocked."
    },
    {
      "id": "billing",
      "description": "Billing support when account access is not blocked."
    },
    {
      "id": "review",
      "description": "Human review when the evidence is insufficient."
    }
  ]
}

Build an evaluation that can reject either system

Use the example above as a SemIf-shaped input fixture, not a request body for TypeSafe or this site. Translate it into each API’s schema while preserving its meaning. Create distinct development, calibration, and test sets; keep related or paraphrased tickets in the same split. Include missing evidence, mixed intents, reversed option order, and your costly error types. Compare per-queue errors, auto-routing coverage at an acceptable error rate, human-review volume, p50/p95 latency, and cost per correctly completed route. Shared numeric thresholds are not a fair comparison when confidence definitions differ.

When Jev fits, and when SemIf fits

Favor a Jev trial when your priority is a managed integration and you can accept the provider data path. Favor a SemIf trial when local operation or inspecting the scorer is a firm requirement and someone can own model serving and calibration. Keep your current rules if either pilot misses the agreed error budget or costs more after review work is counted. Before committing, write a short scorecard naming the test set, acceptable errors, data constraints, operating owner, and rollback path. Which requirement is binding in your workflow: data location, review workload, latency, or total cost?

JevModel is independent and not affiliated with TypeSafe AI.