Introduction

Evaluation harness for the language models built into Android and iOS, written once in Kotlin Multiplatform.

Proctor runs an evaluation suite against the language model that ships with the phone: Gemini Nano on Android and Apple Foundation Models on iOS. You write the suite once in commonMain. Proctor runs it on the device, grades every output, saves the results as JSON, and builds a report that compares platforms, model versions and prompt versions.

The OS updates these models, not your app. A prompt that worked last month can answer differently after a system update, and two phones can run different model versions at the same time. Proctor makes that visible before your users see it.

The current version is 1.0.0. See the changelog for what each release changed.

val suite = evalSuite("ticket-classifier") {
    prompt(version = "v1") { input -> "Classify this support ticket as Billing, Bug, Account, Feature or Other: $input" }
    case("My card was charged twice", id = "double-charge") { expect("Billing") }
    case("The app crashes when I open the camera", id = "crash") { expect("Bug") }
    graders { exactMatch(ignoreCase = true) }
    repeat(3)
    passWhen { score >= 0.8 }
}

How it works

1. A suite is plain Kotlin: prompt versions, cases, graders, a repeat count and a pass condition.
2. A backend (MlKitBackend on Android, FoundationModelsBackend on iOS) sends each prompt to the system model. Requests run one at a time with a pause between them, because AICore meters inference per app.
3. The runner records the device, the OS, the model version, the backend settings and every attempt, and writes one *.proctor.json file per run.
4. proctorReport reads a folder of those files and writes an HTML and a Markdown report. It fails the build when a run misses its pass condition.

What it does not do

• It does not send anything anywhere. Datasets, prompts and outputs stay on the device and in your repository. See Privacy.
• It is not a cloud-model evaluation tool. For that, see prior art.
• It does not run on emulators. Android emulators have no AICore and no Gemini Nano.

Platform support

Each row says what has actually been run. A combination that was not run on real hardware is marked unverified.

See a sample report built from real iOS simulator runs.