OPEN SOURCE

Validation science, published.

We open-source what makes our claims checkable: weights, benchmarks, harnesses, raw predictions. The training engineering stays private. Reproducible in use; honest about what that means.

THE FLAGSHIP

One model, one benchmark, one harness to check both.

The weights, the evaluation data, and the code that scores them, published so every CORE-68 number behind HivemindEval can be checked independently, not taken on faith.

Model

huggingface.co/Hypereum/HivemindEval

Apache-2.0 · full merged model in bfloat16 safetensors · full model card

View model card

Benchmark

huggingface.co/datasets/Hypereum/hivemind-eval-benchmark

68-item stratified subset · per-item gold · every model's raw predictions

View dataset card

Harness

github.com/hypereum-innovations/hivemind-eval

stdlib inference + scoring · methodology doc

View repository

Radical transparency includes our mistakes.

PRIOR-DEFECT DISCLOSURE

v2.0 was publicly visible on the Hub for a period beginning May 2026 and was later withdrawn; this release (v2.1) supersedes it.

An earlier version (v2.0) suffered complete rating collapse: it emitted the identical overall_score of 75, and an identical per-dimension score vector, for every input, regardless of quality (std 0.0 across a graded probe set), while producing perfectly well-formed JSON. It was syntactically valid and semantically inert.

The defect was caught in our internal validation program and root-caused to defective training labels (the labels carried a constant score vector rather than real judgments; no model trained on them could learn to discriminate). v2.1 is a full retrain on corrected labels and passes the anti-collapse acceptance gate; the benchmark items used to evaluate it are hash-proven disjoint from its training data.

Format validity is not judgment quality.

We publish our failures next to our results. That is what verifiable means.

REPRODUCE IT

Reproduce it.

PROVISIONAL · pending independent ≥3-expert validation

$ python3 eval/score_benchmark.py --gold data/gold_subset.json --predictions-dir data/predictions

⟩ every CORE-68 cell of the published board, bit-for-bit, bootstrap CIs included, seed 7, no GPU required

Prefer to regenerate predictions from the weights? One 24 GB GPU and the card's quickstart, verbatim.

FULL SET 4f2dbff7…
PUBLIC SUBSET b51e721e…
WHAT STAYS PRIVATE

What we don't publish, and why.

What's public is the validation science: how the benchmark is built, the gates a model has to clear, and the methodology behind every score. What stays private is the model's training pipeline: data generation, labeling, and hyperparameters. Open where it builds trust; private where it is the product.

EXPERT VALIDATION PROGRAMME

We're recruiting the panel that will grade us.

Every number we publish is provisional until independently validated. We are recruiting UK/EU compliance experts for a ≥3-reviewer blind validation panel: blind to model identity and to each other, per-dimension anchored rubrics, abstention allowed. Panel results set the human ceiling and convert provisional numbers into validated ones, whatever they show.

Join the panel
LICENSE & CITATION

License & citation.

License

Apache-2.0. Fine-tuned from Qwen/Qwen3-8B (Apache-2.0, © Alibaba Cloud); thanks to the Qwen team for the base model.

Citation

@software{hivemindeval2026,
  author  = {Hypereum},
  title   = {HivemindEval: an open-weight verifier for UK/EU compliance analysis},
  year    = {2026},
  url     = {https://huggingface.co/Hypereum/HivemindEval},
  note    = {Provisional benchmark results pending independent expert validation}
}