Reading reports

The result files, the report sections, and what "not graded" means.

proctorReport reads every *.proctor.json file in the results folder and writes index.html (self-contained, no scripts, no network) and report.md. See a sample report.

Result files

One file per suite, platform, model version, prompt version and run time. The file has schemaVersion: 1, the environment (platform, device, OS, model version, backend settings), the sampling settings, availability, and every attempt with its output, grader results and latency. Runs started together share an optional groupId (set it with RunnerConfig(groupId = ...)), so a history view can show them as one batch. Files do not change after they are written; a new run is a new file.

Sections

• Checked on: device, OS, model version, backend and run time of every run. This is the table to copy into a README.
• Results: passed, failed and not graded per run, the mean score and the verdict.
• Compare: the oldest run of each suite (or the one you name as baseline) against every other run of that suite. The report says what differs: platform, model version, prompt version, or several at once. Per case it shows same, regressed or improved.
• Cases: input, expected value, first output, result, score and spread per case.

Passed, failed, not graded

• Failed: the model answered and a grader said no.
• Not graded: no verdict. Quota, lost foreground, an unavailable model or an unreadable judge reply. These cases say nothing about your prompt.

A run’s verdict comes from your passWhen. With a score threshold, a run can pass while single cases fail, as in the sample report. The report shows both.

Comparing across model versions

Run the same suite on each model, keep all result files, and run proctorReport. Change nothing else between runs: the same prompt version, sampling and cases. If the platform or model version differs the report labels the difference, but it cannot tell you why a score moved.