Why this matters
Large RAG pipelines can return fluent answers that are not supported by retrieved evidence; detecting when a generated claim is actually grounded in the supplied context is essential for evaluation and downstream guardrails. This release exposes both the upstream reference annotations (from RAGTruth) and the separate automatic decisions made by jev-1.13.0 so researchers can inspect agreement, confident disagreements, and the exact context that led to each verdict.
What Sets It Apart
- Paired reference + verifier outputs: every decision row preserves the original upstream
gold_*annotation alongsidejev_*outputs (predictions, confidence and probability distributions), enabling direct comparison of teacher/reference labels with an automated decision layer. - Multi-task coverage and scale: 130,000 derived decisions across four task families (claim-span grounding, hallucination-type presence, whole-response grounding, response quality) drawn from ~12k upstream responses and ~2k distinct query/context pairs—big enough to analyze label imbalance and recurrent failure modes.
- Inspectable provenance and tooling: the release keeps full JSON state/request fields, 49 columns of provenance, Parquet shards sized for streaming, and example code to iterate claims without materializing repeated strings—designed for reproducible error analysis, threshold studies and reranker/probe experiments.
- Useful disagreement signal: 21,181 jev/reference disagreements are preserved (including 1,265 categorical disagreements with provider-reported jev confidence ≥ 0.90), which are ready for focused audits of high-confidence failures.
Who it's for — and tradeoffs
Great fit if you want a reproducible, inspectable corpus to study grounding and hallucination detection in RAG systems, to prototype evidence-checking classifiers, or to analyze confidence/thresholding behavior of a non-generative verifier. It is not a contamination-free held-out benchmark nor a production truth oracle: source annotations may be noisy, scores are uncalibrated, shared contexts/duplicates exist across splits, and the aggregate uses license: other. Rebuild splits and inspect provenance before training or reporting held-out accuracy; use this primarily for error analysis, model comparison and prototype research rather than as a definitive evaluation corpus.
