Draw
Bind a reproducible sample to one exact, finite trace population.
← All projectsPublic source · Beta · v0.1.0
03 · Govern how it is judged
Coeval
Governed evaluators for AI systems.
An LLM judge is easy to write. The difficult part is establishing what it measures, which human evidence supports it, how disagreements were resolved, and which exact evaluator version produced a result.
Coeval makes that work first-class. It turns representative traces into narrow criteria, governs human review and evaluator versions, validates them against protected truth, and emits evidence that downstream systems can verify without inheriting release policy.
HUMAN REVIEW · ILLUSTRATIVE INTERFACEITEM 07 / 24
TRACE EXCERPT
Agent “The deployment is complete and available at the production URL.”
Evidence Command returned before the deployment health check completed.
The interface is a code-native explanation of the review model—not a screenshot of customer data.
INDEPENDENT LABEL
The success claim exceeds the evidence available at the point it was made.
SEALED UNTIL SUBMISSION
“Human truth” is not one mutable score in a database. It is a governed chain of evidence.
Evaluator quality cannot be separated from the reference evidence used to validate it. If reviewers see one another’s labels, if ambiguous examples are forced into pass or fail, or if adjudication silently overwrites the original disagreement, the final metric becomes more confident and less trustworthy at the same time.
Coeval keeps independent labels, abstention, disagreement, alignment, and adjudication as append-only evidence. That history makes it possible to understand not only the final answer, but how stable the criterion was and where reasonable people differed.
A controlled path from real traces to a reusable evaluator.
Define
Turn observed failures into a taxonomy and narrow, versioned criteria.
Review
Collect independent labels, abstentions, disagreements, and adjudication.
Validate
Compare a pinned evaluator against protected human truth.
Emit
Produce policy-free evidence with exact provenance and completeness.
CRITERION IDENTITY
One evaluator measures one named question.
A broad “quality” judge obscures which behavior moved and why. Coeval keeps criteria narrow and versioned, then pins evaluator execution to that identity. Changing the criterion, prompt, model, or protected reference set creates new evidence rather than rewriting old results.
criterionunsupported_success_claim
criterion_version3
evaluator_versioneval_07
reference_setsealed_04
executionpinned
Evidence, not a shipping opinion.
ASSESSMENT RECEIPTVERIFIABLE
Populationexact digest
Drawreproducible
Criterionversioned
Evaluatorpinned
Coverageexplicit
Completenessvisible
Suite manifests and assessment receipts preserve criterion identity, execution provenance, coverage, and completeness. They intentionally do not say whether a release should ship. That decision depends on customer policy and belongs downstream—in a tool such as Dailies.
Coeval can import traces from LangSmith and LangFuse-compatible sources, integrate with Ironside as the system of record, and expose project-scoped machine access. The system is built around evidence portability rather than requiring every adjacent concern to live inside it.
Governance improves the claim; it does not make judgment objective.
Human labels can still be noisy, criteria can still be incomplete, and evaluator behavior can change with models and providers. Coeval’s job is to make those dependencies inspectable and versioned—not to erase uncertainty. It also does not encode release thresholds or compensation rules; those remain policy owned by the system consuming its evidence.
Read the v0.1.0 release ↗