Qualify and submit a model

Three distinct states

Catalog candidate

Its upstream source, immutable revision, license, dimensions, byte size, and SHA-256 are recorded. This is an internal contribution queue, not a runtime claim.

Artifact verification

CI downloads those exact bytes and tests the GGUF parser, tokenizer, tensor layout, generation path, and deterministic output references needed by the model.

Production qualification

The exact artifact and Models backend first pass the complete RAG workload with library defaults, then satisfy the separate tuned performance gates.

Not yet qualified means the production run has not happened. Not qualified is used only after an executed artifact fails the policy.

Qualification workflow

1. Pin one immutable artifact. Record the upstream repository, commit, exact download URI, license, size, SHA-256, architecture, quantization, capabilities, and intended workload.
2. Prove Models compatibility. Add or run exact-artifact tests in integrallis/models. New parser, tokenizer, graph, or kernel support belongs there and must be covered test-first.
3. Run on a controlled Linux host. Use Java 25, a clean Models checkout, no competing inference process, and enough memory for the model. The harness records the CPU, memory, JVM, Models revision, artifact digest, prompt controls, and every request.
4. Pass with library defaults. Before tuning, the exact Models backend runs every workload case with the longest-common-prefix cache. Every generation must succeed and deterministic correctness must be 100%. A tuned result cannot override this gate.
5. Compare identical bytes separately. The harness measures Models and same-host Ollama and llama.cpp comparators with the same GGUF, prompts, context, sampling controls, thread count, warmups, and iterations. Ollama and llama.cpp are qualification comparators, not ModelJars runtime dependencies.
6. Submit reproducible evidence. Commit the raw reports to the Models repository, then update ModelJars catalog and qualification metadata, including defaultConfigurationSmoke, in a pull request. CI fetches the default report from the pinned Models commit, verifies its digest and controls, and rebuilds the marker and public catalog.

Run the controlled harness

The current harness lives in the Models repository. It fails closed when prerequisites, artifact identity, comparator controls, or policy gates do not match.

Host prerequisites

• Linux on a dedicated or otherwise idle host with enough RAM for all three engines.
• Java 25, a clean Models checkout, and the model artifact on a local filesystem.
• ollama, llama-server, curl, git, jq, and standard Linux utilities on PATH.
• No live Models, Ollama runner, llama-cli, llama-server, or llama-bench process.

git clone https://github.com/integrallis/models.git
cd models

scripts/run-controlled-rag-qualification.sh \
  /absolute/path/to/model.gguf \
  catalog_model_id \
  workload_id \
  chatml \
  build/reports/rag/qualification/catalog_model_id

Select the prompt template validated for the artifact, such as chatml, llama3, gemma, or zephyr. The output directory contains Models, Ollama, and llama.cpp reports plus qualification.json.

A contributor's local run is a preflight result, not publication authority. A maintainer must reproduce or review the run on the controlled host, pin the Models commit, and approve the exact report and artifact hashes before adding the qualification entry.

Acceptance policy

Qualification requires the absolute quality and usable-latency gates plus a qualifying same-host Ollama comparison. The llama.cpp comparison is retained as an additional performance diagnostic. A PRODUCTION_READY result uses the stricter latency ceilings of 100 ms retrieval, 1 second TTFT, 100 ms per output token, and 5 seconds end to end. These are ModelJars acceptance SLOs for the versioned workload, not guarantees for every application, prompt length, concurrency level, or machine.

Embedding acceptance policy

We test that a model produces the same vectors as llama.cpp, for a pinned probe set over the same model bytes.

A correct run measures 0.99950 agreement. Mean pooling in place of last-token measures 0.66156. Cosine is scale-invariant, so an unnormalized runtime agrees at exactly 1.0; vector length is gated separately. The gate takes the worst probe.

Evidence is recorded as ModelEmbeddingQualification, whose tier is SEMANTIC_SEARCH or UNQUALIFIED.

Pull request checklist

Runtime support and evidence are reviewed in integrallis/models first. The ModelJars pull request then adds the marker descriptor and qualification record that reference the accepted Models commit and report.

• One immutable marker version and no committed model weights.
• Attributable upstream source and explicit license metadata.
• Exact revision, byte size, SHA-256, dimensions, and backend declaration.
• Models exact-artifact tests and deterministic output evidence.
• Raw controlled reports committed to Models, the qualification verdict, and report SHA-256 values.
• A matching entry in catalog/qualifications.json only after maintainer approval.
• A clean local run of npm test and ./gradlew test verifyCatalog.

Start with the repository's contribution rules and open a pull request. Only maintainers can merge, and only protected GitHub Actions workflows can publish accepted marker artifacts.