Qualify and submit a model

Start with one CLI command

Give the CLI a public Hugging Face repository. It resolves the requested branch or tag to an immutable commit, selects the required GGUF, CACT, or Safetensors files, verifies their byte sizes and SHA-256 digests, and prepares the candidate report.

modeljars contribute Qwen/Qwen2.5-0.5B-Instruct --domain general

The CLI writes a new Markdown file and prints one copy-ready gh issue create command to submit it. It never overwrites an existing report. If a repository contains several weight files or formats, choose one with --file model-name.gguf or --file model-name.cact. Use --license, --capability, or --domain only when upstream metadata needs an explicit correction or addition.

This creates a verified catalog candidate, not a compatibility or qualification claim. Maintainers own the runtime tests, controlled measurements, metadata pull request, and publication steps that follow.

Three distinct states

Catalog candidate

Its upstream source, immutable revision, license, dimensions, byte size, and SHA-256 are recorded. This is an internal contribution queue, not a runtime claim.

Artifact verification

CI downloads those exact bytes and tests the applicable format reader, tokenizer, tensor manifest, execution graph, and deterministic output references needed by the model. GGUF, Safetensors, and CACT are supported catalog formats.

Production qualification

The exact artifact and Models backend pass the complete workload for its published capability: RAG generation, embedding equivalence, reranking, or tool-calling conformance.

Not yet qualified means the production run has not happened. Not qualified is used only after an executed artifact fails the policy.

What happens after submission

1. The CLI pins one immutable artifact. The candidate report records the upstream repository and commit, every required file, its size and SHA-256, the reported license and architecture, capabilities, and intended domain.
2. Prove Models compatibility. Add or run exact-artifact tests in integrallis/models. New parser, tokenizer, graph, or kernel support belongs there and must be covered test-first.
3. Maintainers run the applicable controlled workload. The generation, embedding, and reranking harnesses record the host, JVM, Models revision, complete artifact identity, prompt or probe controls, and every measured request.
4. Pass with library defaults. Before tuning, the exact Models backend runs every workload case with the longest-common-prefix cache. Every generation must succeed and deterministic correctness must be 100%. A tuned result cannot override this gate.
5. Use a compatible independent reference. GGUF studies can compare Models, Ollama, and llama.cpp against identical artifact bytes and controls. Safetensors studies use the pinned Transformers reference. CACT studies use the pinned upstream implementation only as a compatibility oracle, then execute the conformance suite entirely through Models. Comparators are qualification tools, not ModelJars runtime dependencies.
6. Publish reproducible evidence. Maintainers commit the raw reports to Models, then update the ModelJars catalog and qualification metadata in a reviewed pull request. CI fetches the pinned reports, verifies their digests and controls, and rebuilds the marker and public catalog.

How maintainers reproduce qualification

The Models repository owns the qualification tools. The all-in-one controlled driver below is the GGUF path; it fails closed when host prerequisites, artifact identity, comparator controls, or policy gates do not match.

GGUF host prerequisites

• Linux on a dedicated or otherwise idle host with enough RAM for all three engines.
• Java 25, a clean Models checkout, and the model artifact on a local filesystem.
• ollama, llama-server, curl, git, jq, and standard Linux utilities on PATH.
• No live Models, Ollama runner, llama-cli, llama-server, or llama-bench process.

git clone https://github.com/integrallis/models.git
cd models

scripts/run-controlled-rag-qualification.sh \
  /absolute/path/to/model.gguf \
  catalog_model_id \
  workload_id \
  chatml \
  build/reports/rag/qualification/catalog_model_id

Select the prompt template validated for the artifact, such as chatml, llama3, gemma, or zephyr. The output directory contains Models, Ollama, and llama.cpp reports plus qualification.json.

For Safetensors, maintainers run models-rag-bench against the pinned Hugging Face snapshot directory, then run the pinned deterministic Transformers reference against the same weights and controls. The Models RAG benchmark guide contains the exact commands for both runs.

For tool-calling candidates, the Models conformance command loads the checksum-pinned artifact directly into the Java backend. It does not install or run an upstream inference engine. The candidate key selects the reviewed artifact identity and its exact chat/tool protocol.

./gradlew :models-bench:run --args='tool-qualification \
  --candidate qwen3-1.7b \
  --model /absolute/path/to/Qwen3-1.7B-Q8_0.gguf \
  --models-revision 0123456789abcdef0123456789abcdef01234567 \
  --report /absolute/path/to/models/benchmark-results/qwen3-1.7b-tool-calling.json \
  --max-tokens 256'

A contributor's local run is a preflight result, not publication authority. A maintainer must reproduce or review the run on the controlled host, pin the Models commit, and approve the exact report and artifact hashes before adding the qualification entry.

Acceptance policy

Qualification requires the absolute quality and usable-latency gates plus a qualifying authoritative-engine comparison: Ollama for compatible GGUF artifacts or the pinned Transformers reference for formats those native engines cannot ingest. llama.cpp is retained as an additional GGUF performance diagnostic. A PRODUCTION_READY result uses the stricter latency ceilings of 100 ms retrieval, 1 second TTFT, 100 ms per output token, and 5 seconds end to end. These are ModelJars acceptance SLOs for the versioned workload, not guarantees for every application, prompt length, concurrency level, or machine.

Embedding acceptance policy

We test that a model produces the same vectors as llama.cpp, for a pinned probe set over the same model bytes.

A correct run measures 0.99950 agreement. Mean pooling in place of last-token measures 0.66156. Cosine is scale-invariant, so an unnormalized runtime agrees at exactly 1.0; vector length is gated separately. The gate takes the worst probe.

Evidence is recorded as ModelEmbeddingQualification, whose tier is SEMANTIC_SEARCH or UNQUALIFIED.

Reranking acceptance policy

Cross-encoder qualification compares a fixed query-document set with an independent implementation of the exact artifact, verifies the complete ranking, then measures a practical second-stage workload in fresh JVMs. Quantized artifacts also retain their comparison with the unquantized source when one is available.

The evidence pins the artifact, Models commit, reference implementations, pair set, machine, JDK, Vector API width, fresh-process controls, raw scores, and latency samples.

Tool-calling acceptance policy

The versioned conformance suite records every generated call, the exact Models commit, artifact digest, source-case digest, prompt template, generation controls, and host.

The upstream implementation supplies frozen reference cases only. Qualification executes tokenization, retrieval, constrained decoding, and generation in-process through Models.

Publication checklist

Contributors can track the candidate issue created by the CLI. Runtime support and evidence are reviewed in integrallis/models first. The maintainer pull request then adds the marker descriptor and qualification record that reference the accepted Models commit and report.

• One immutable marker version and no committed model weights.
• Attributable upstream source and explicit license metadata.
• Exact revision, byte size, SHA-256, dimensions, and backend declaration.
• Models exact-artifact tests and deterministic output evidence.
• Raw controlled reports committed to Models, the qualification verdict, and report SHA-256 values.
• A matching entry in the applicable RAG, embedding, reranking, or tool qualification manifest only after maintainer approval.
• The first public marker publication instant recorded as catalogPublishedAt, which drives the CLI's 48-hour NEW marker.
• A clean local run of npm test and ./gradlew spotlessCheck test verifyCatalog.

The repository's contribution rules remain authoritative. Only maintainers can merge, and only protected GitHub Actions workflows can publish accepted marker artifacts.