HYPEREUM

Model

HivemindEval v2.1

An open-weight verifier that scores a compliance finding on six quality dimensions for six UK and EU regulatory frameworks. JSON in, JSON out, with a reason for each score.

Provisional

Every benchmark number here is provisional. No result has been validated by an independent panel of human experts yet. A panel of at least three experts is planned, and its outcome will be published whatever it shows. Hypereum does not claim that this model is the best available, state of the art, or at expert parity.

Frameworks
PSD2 SCA-RTS. UK GDPR with the NHS DSPT. MOD JSP 440. Cyber Essentials Plus. DORA. EU AI Act.
Dimensions
Accuracy, completeness, regulatory alignment, actionability, coherence, evidence quality.
Runs on
One 24 GB GPU. About 40 tokens per second, single stream. About 23.4 GB of memory used with vLLM at --max-model-len 8192. A 24 GB deployment is single-stream with minimal headroom; use a larger card for concurrency.

Check it

The CORE-68 table recomputes from the published gold labels and predictions, bit for bit, confidence intervals included (seed 7, 400 resamples), with no GPU:

python3 eval/score_benchmark.py--gold data/gold_subset.json--predictions-dir data/predictions

The FULL-212 table cannot be rerun outside Hypereum. The full item set is withheld and committed by hash.

Serve it:

vllm serve Hypereum/HivemindEval--max-model-len 8192

Results, FULL-212

212 scored items, open models. Comparison models were run with a scoring rubric of about 10,500 tokens, at temperature 0, on identical items.

Results, FULL-212
ModelTier concordance (95% CI)Excellent concordanceDefect separationKendall τScored
Qwen2.5-72B-Instruct-AWQ (rubric)0.985 [0.977–0.993]0.9900.6020.801212/212
HivemindEval v2.1 (native)0.984 [0.973–0.993]0.9820.5910.817211/212
Qwen3-8B (rubric)0.973 [0.959–0.985]0.9730.7090.751208/212
Llama-3.3-70B (rubric)0.965 [0.950–0.980]0.9680.6160.764173/212
Qwen3-8B (native prompt, no fine-tune)nonenonenonenone0/212

The 72B model edges HivemindEval on tier concordance and on excellent concordance. HivemindEval leads on Kendall τ (0.817 against 0.801), which has no recorded confidence interval. All tier concordance gaps are within overlapping 95% confidence intervals. This is a statistical tie.

Results, CORE-68

The published subset, reproducible from the shipped data.

Results, CORE-68
ModelTier concordance (95% CI)Excellent concordanceDefect separationKendall τScored
Qwen3-8B (rubric)0.991 [0.977–1.000]0.9880.6390.78867/68
Qwen2.5-72B-AWQ (rubric)0.990 [0.980–1.000]0.9920.5490.83368/68
HivemindEval v2.1 (native)0.979 [0.958–0.998]0.9760.6150.84367/68
claude-sonnet-5 (rubric)0.976 [0.951–0.996]0.9750.5620.77766/68
Llama-3.3-70B (rubric)0.964 [0.932–0.988]0.9630.6670.79856/68

Where the model is weak

Defect separation by class, FULL-212. The value is the probability that a defective item scores below a clean good item. Higher is better.

Defect separation by class, FULL-212
Modelwrong_jurisdictionhallucinatedmissing_requirementband_edge
Qwen3-8B (rubric)1.000.590.650.60
Qwen2.5-72B-AWQ0.890.590.460.46
Llama-3.3-70B0.850.570.510.53
HivemindEval v2.10.900.670.420.36

HivemindEval is last among the scored models on overall defect separation (0.591). It is weakest on missing_requirement (0.42) and band_edge (0.36). A rubric-prompted base Qwen3-8B beats it on wrong_jurisdiction (1.00 against 0.90). If your primary need is defect detection rather than quality ranking, a rubric-prompted model is currently the better tool.

What the fine-tuning buys

On the same hardware and the same prompt, the base model produced 0 of 212 parseable outputs. HivemindEval produced valid output on 211 of 212. Format validity is a reliability property. It is reported separately from judgment quality.

Limits

  • It is trained for one input schema and one output schema in one domain. For general judging work a rubric-prompted frontier model is the right tool.
  • Its scores are decision support for human reviewers. They must never be the sole basis for a compliance decision.
  • It has not been evaluated on other jurisdictions, on non-English input or on free text.
  • The training pipeline is not published: data generation, labelling and hyperparameters.

History

History
VersionDateStateNote
2.12026-07-17ProvisionalFull retrain on corrected labels. Passes the acceptance gate published on the model card.
2.0Briefly public from May 2026WithdrawnIt returned the same overall score, 75, and the same score on every dimension, for every input, while producing valid JSON. Cause: defective training labels. At least one third-party catalogue captured a copy that outlived the withdrawal. Do not use v2.0 weights from any source.

Cite it

@software{hivemindeval2026,author  = {Hypereum},title   = {HivemindEval: an open-weight verifier for UK/EU compliance analysis},year    = {2026},url     = {https://huggingface.co/Hypereum/HivemindEval},note    = {Provisional benchmark results pending independent expert validation}}