Graders

Exact match, regex, JSON schema, semantic similarity, LLM judge and custom graders.

A grader scores one attempt. A case passes when every grader passes. Add graders inside graders { }.

Grader	Passes when	Needs
`exactMatch(ignoreCase)`	The trimmed output equals the expected value	`expect(...)` on the case
`regex(pattern)`	The pattern matches the whole trimmed output
`jsonSchema<T>()`	The output, after removing Markdown fences, decodes into `T`	A `@Serializable` type
`semanticSimilarity(embedder, threshold)`	Cosine similarity to the expected value reaches the threshold	An `EmbeddingProvider` and `expect(...)`
`llmJudge(rubric, judge)`	A judge model scores the output 3 or 4 on a 1 to 4 scale	A `Judge`

A grader that cannot do its job, such as exactMatch on a case without an expected value, returns “not graded”, not “failed”.

Custom graders

Implement Grader. A thrown exception becomes “not graded”.

class DetailsMatch : Grader {
    override val name = "detailsMatch"

    override suspend fun grade(context: GradeContext): Grade {
        val actual = Json.decodeFromString<TicketDetails>(extractJson(context.output))
        val expected = Json.decodeFromString<TicketDetails>(context.expected!!)
        val matching = listOf(actual.orderId == expected.orderId, actual.product == expected.product,
            actual.urgent == expected.urgent).count { it }
        val score = matching / 3.0
        return if (matching == 3) Grade.passed(score) else Grade.failed(score, "Fields differ: $actual")
    }
}

The judge runs on the host, not on the device, and is usually a cloud model. Proctor ships the grader and the prompt, not the client: you pass a Judge { prompt -> reply } that calls whatever model you use. Judge scores vary between runs and between judge models, so keep the judge fixed when you compare two runs, and prefer a deterministic grader when one exists. If the judge’s reply cannot be read, the attempt is “not graded”.

A judge sees the case input and the model output. If those contain user data, sending them to a cloud judge leaves the device. Proctor never does this for you.

Graders

Custom graders

LLM judges and variance