Measured evidence
Inference and RAG benchmarks
Same-host local engine comparisons and end-to-end Java RAG measurements, including local and hosted inference.
Local inference
Models compared with native engines
The table contains the qualified artifacts with controlled cross-engine studies. Each row uses the same GGUF bytes and generation controls across the selected Models backend, llama.cpp, and Ollama; these engines are comparators, not ModelJars runtime dependencies.
| Model | Status | TTFT p95 | Prompt p50 | Decode p50 | Models decode ratio | Evidence | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Models | llama.cpp | Ollama | Models | llama.cpp | Ollama | Models | llama.cpp | Ollama | vs llama.cpp | vs Ollama | |||
| Loading inference evidence... | |||||||||||||
Controls
CPU-only retained runs pin processor limits, context, sampling, warmups, and trial counts in their evidence. Every row compares identical prompts and artifact bytes.
Qualification
Profiled rows are exact ModelJars launch profiles. A validated candidate passed output checks but is not an automatic runtime recommendation.
Launch qualification
Production RAG qualification by artifact
Every listed artifact completed the published Models Java workload with the exact ModelJar SHA on the controlled host. The summary and table are generated directly from the qualification ledger.
| Model and backend | RAG use | TTFT p95 | TPOT p95 | End to end p95 | Decode p50 | Raw quality | Final quality | Fallback | Peak RSS | Requests | Evidence |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Loading qualification evidence... | |||||||||||
Usability gate
Qualified rows completed every request with at least 90% grounded quality, 100% unsupported-query abstention, p95 TTFT at or below two seconds, and p95 end-to-end latency at or below ten seconds.
What Guarded RAG means
Guarded RAG uses the model when its answer passes citation and grounding checks. When it does not, Models returns a deterministic extract from the retrieved sources, or abstains when the sources do not support an answer. Raw model quality and fallback frequency remain visible in every row.
Evidence identity
Model bytes, raw report bytes, Models revision, backend version, policy, host, JVM, and every reported metric are preserved in versioned ModelJar metadata.
Application workload
RAG from the same Java application
Nine retrieval cases compare in-process Models, local native servers, and hosted providers under one Java 25 benchmark harness.
| Engine and model | Execution | Retrieval p95 | TTFT p95 | TPOT p95 | End to end p95 | Decode p50 | Strict quality | Audited quality | API cost / 1K | Data egress | Evidence |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Loading RAG evidence... | |||||||||||
Workload
One warmup and three measured iterations over nine cases at concurrency one.
Quality audit
Strict quality is the unchanged automated evaluator. Audited quality accepts two semantically correct referral paraphrases that did not contain the literal expected phrase.
Cost and privacy
Local rows keep retrieved context in the process or on the same host and incur no API charge. Hosted rows send context to a provider; costs reflect this measured token and cache mix, not a fixed price quote.