offset-smoker/laravel-llm-evals

An evaluation harness for the LLM features in a Laravel application.

A prompt change, a model upgrade or a new system message can quietly make an LLM feature worse, and a normal test suite will not notice, because the code still runs. This package checks the output: you write golden cases in JSON or YAML, each with code-based graders (contains, regex, JSON Schema, citations, refusal, reading level), and php artisan evals:run scores your feature against them and exits non-zero when it falls below a threshold. Every dataset must also declare a deliberately bad baseline, and the run fails if that baseline scores too well, because graders that a fixed answer can pass are not measuring anything. Runs can replay recorded responses, so CI needs no API key and spends nothing.

Requirements

• PHP 8.3 or 8.4
• Laravel 12 (12.69 or later) or 13 (13.30 or later)
• A PSR-18 HTTP client for the Claude driver. Laravel applications already have Guzzle.

Installation

composer require --dev offset-smoker/laravel-llm-evals

The service provider is discovered automatically. Publishing the config is optional:

php artisan vendor:publish --tag=llm-evals-config

Usage

The package ships a small synthetic dataset (plain-language rewrites of notices from a fictional "Example Township") and a recording of good responses to it. Copy both into your application and run it:

mkdir -p evals/recordings
cp vendor/offset-smoker/laravel-llm-evals/examples/evals/plain-language-notices.yaml evals/
cp vendor/offset-smoker/laravel-llm-evals/examples/evals/recordings/plain-language-notices.json evals/recordings/
php artisan evals:run

plain-language-notices  3 cases, threshold 100%
  control (constant): 1/3 passed, 33.3%, ceiling 34%  held
  subject (recorded):
    ✓ water-main-closure
    ✓ permit-hearing-json
    ✓ refuses-resident-address
  3/3 passed (100%), mean grader score 100%
  PASSED (exit 0)

1 dataset run, exit code 0.

Evaluate your own feature

Register the feature as a driver, in AppServiceProvider::boot() for example. The closure receives each case and returns the text your feature produced (or a Response, if you also want to pass a stop reason or token usage):

use App\Services\NoticeRewriter;
use OffsetSmoker\LlmEvals\Dataset\EvalCase;
use OffsetSmoker\LlmEvals\Drivers\CallbackDriver;
use OffsetSmoker\LlmEvals\Drivers\DriverRegistry;

$this->callAfterResolving(DriverRegistry::class, fn (DriverRegistry $drivers) => $drivers->extend('notice-rewriter', fn () => new CallbackDriver(
    fn (EvalCase $case): string => $this->app->make(NoticeRewriter::class)->rewrite($case->input),
    'notice-rewriter',
)));

Use callAfterResolving(), not $this->app->make(DriverRegistry::class). The package is a dev dependency, so after composer install --no-dev its classes do not exist, and resolving one in boot() would break every request. The callback runs only when the registry is resolved, which happens only where the package is installed.

php artisan evals:run --driver=notice-rewriter

If your feature cannot answer a case, throw OffsetSmoker\LlmEvals\Drivers\DriverFailed with a message that is safe to print. Any other exception also fails just that case, but the report shows only its class, file and line, because exception messages often quote the prompt.

A failing case lists the graders it failed and why:

  subject (notice-rewriter):
    ✗ water-main-closure
        contains "Elm Street": not found
        cites every source: cites 0 of the required 1
        reads at grade 8.0 or below: grade 28.2

Evaluate Claude directly, and record the result

The claude driver sends each case to Claude through the official Anthropic PHP SDK (anthropic-ai/sdk), with the dataset's system prompt as the system message. --record saves the responses as the dataset's recording, so the same outputs can be re-graded offline later:

ANTHROPIC_API_KEY=your-key php artisan evals:run --driver=claude --record

In Pest

With Pest installed, the package adds two expectations:

use OffsetSmoker\LlmEvals\Drivers\RecordedDriver;

it('passes the notices dataset', function () {
    expect(base_path('evals/plain-language-notices.yaml'))
        ->toPassEval(RecordedDriver::fromDirectory(base_path('evals/recordings')));
});

it('keeps rewrites readable', function () {
    expect('The water main under Elm Street is off on Tuesday at 9 a.m. [notice-101]')
        ->toPassGraders([['contains' => 'Elm Street'], ['reading_level' => 8], ['citations' => true]], ['notice-101']);
});

toPassEval() also takes a closure, fn (EvalCase $case): string, in place of a driver.

In CI

- run: mkdir -p build && php artisan evals:run --json=build/evals.json

With several datasets the highest code wins. An empty datasets directory is exit 3, not a pass. A dataset that stops with a setup error (exit 3), such as a missing recording, does not stop the others; the JSON report lists it under errors. Two dataset files with the same name are exit 3, because they would share one recording. With --record, existing recordings and the recordings directory are checked before anything is sent to the subject, so a bad recording never costs a live run.

Writing a dataset

name: plain-language-notices      # lowercase; also the recording's file name
threshold: 1.0                    # share of cases that must pass, above 0 and up to 1 (default 1)
system: |                         # optional; sent as the system prompt by the claude driver
  You rewrite public notices for residents...

control:                          # required, see below
  constant: "I can't help with that."
  ceiling: 0.34

cases:
  - id: water-main-closure
    sources: [notice-101]         # ids the output may cite, cited as [notice-101]
    input: |
      Rewrite notice-101 in plain language: ...
    expect:
      - contains: Elm Street
      - regex: /Tuesday/
      - citations: {min: 1, all: true}
      - reading_level: {max_grade: 8}
      - refusal: false

A case passes when all of its graders pass. A dataset passes when its share of passing cases is at or above threshold. Reports also show a mean grader score (the share of graders passed, averaged over cases) to show partial progress.

Everything in the dataset files is validated before anything runs. Unknown keys, wrong types, duplicate case ids, invalid regular expressions or schemas, a case with no graders, and a missing or incomplete control are all errors that name the file and the location, such as cases[2].expect[0].regex is not a valid PCRE pattern. Case ids may not be all digits.

Grader reasons describe the expectation and what was found without quoting the output: contains "Elm Street": not found, cites 1 id(s) that are not sources of this case, schema violations: pattern at #/properties/date.

Graders

Add your own with the GraderFactory. The closure gets the grader entry and its type key:

use OffsetSmoker\LlmEvals\Graders\GraderFactory;
use OffsetSmoker\LlmEvals\Support\Spec;

$this->callAfterResolving(GraderFactory::class, fn (GraderFactory $graders) => $graders->extend(
    'mentions_township',
    fn (Spec $entry, string $type) => new MentionsTownship,
));

MentionsTownship implements OffsetSmoker\LlmEvals\Graders\Grader (describe(): string and grade(Response $response, EvalCase $case): Grade).

The falsifiability control

Every dataset names a control: a deliberately bad baseline that its graders should reject.

Keep control recordings in a subdirectory such as controls/: every .json and .yaml file at the top level of the datasets directory is loaded as a dataset. A control recording must exist when the dataset loads, stay inside the dataset's directory (symlinks are resolved), and hold a response for every case. A control that cannot produce an answer would "fail" every case and look as if it held, so a control error is exit 3, never a pass.

The control's pass rate must be at or below ceiling (default 0), and the ceiling must be below the dataset's threshold. The control runs first. When it passes more cases than its ceiling allows, the run stops with exit code 2 and the subject is never called, so a dataset whose graders cannot tell good output from bad cannot report a pass, and does not spend money on a model call to find that out. The report lists the cases the control passed and the graders it satisfied, which are the ones to tighten.

The example dataset's control is a fixed refusal. It correctly passes the one case that expects a refusal, and fails the other two, so its ceiling is 0.34.

Configuration

config/llm-evals.php:

To send requests through another Anthropic client (a different base URL, or Claude on a cloud platform), bind an Anthropic\Client in the container; the claude driver uses that binding instead of building one from the key.

evals:run options: --driver=, --record, --json=path.json, and --with-outputs to include model outputs in the console and JSON reports (they are left out by default; see SECURITY.md).

Design notes

• No refusal fallback. The Claude driver does not configure a fallback model. An eval measures the model it names; if a refusal were silently answered by another model, the score would describe a model you did not choose. A refusal is reported as a stop reason and graded.
• Prompt caching. The system prompt is sent as one block with cache_control: ephemeral, so every case after the first can read it from the cache. The report shows cache reads and writes, which is how you can tell. Prompts shorter than the model's minimum cacheable length are not cached.
• Code-based graders only. There is no LLM-as-judge grader. Each grader is deterministic, so a recorded run grades the same way every time.

Testing

composer install
vendor/bin/pest
vendor/bin/phpstan analyse
vendor/bin/pint --test

The suite has 213 tests: unit tests for every grader, the dataset loader, the runner, recordings and the Claude driver; feature tests for evals:run and the Pest expectations on Orchestra Testbench; and architecture tests, including one that fails if anything in src/ logs or writes to output and one that fails if a CI action is not pinned to a commit SHA. A feature test boots a bare Laravel app with each README registration example, with the package installed and with its classes missing, as after composer install --no-dev. The Claude driver's tests go through the real SDK, with a PSR-18 client that replays recorded Messages API responses, so no test touches the network. Larastan runs at level max with phpstan-strict-rules, no baseline and no ignored errors. CI runs all of it on PHP 8.3 and 8.4, on Laravel 12 and 13, with the highest and the lowest allowed dependencies.

Three dev requirements exist only to set floors for that lowest-dependencies run; nothing in src/ or tests/ uses them. open-telemetry/api and open-telemetry/context (pulled in by the Anthropic SDK) are held at 1.1 or later because 1.0 raises PHP 8.4 deprecations, and paragonie/random_compat is held at 9.99, the empty release, because the 2.x polyfill (allowed by symfony/polyfill-uuid) declares a random_bytes() with no return type, which Larastan then reports in src/.

Status and limitations

Version 0.1.0. The core is complete and tested, but it is new, so expect the API to move before 1.0.

• The Claude driver sends one user message per case: no multi-turn conversations, images, documents or tools, and no streaming (hence the 32,000 max_tokens cap).
• Cases run one after another. There is no concurrency and no Message Batches support, so large live runs are slow.
• Each case is run once. There is no repeated sampling or statistical comparison between runs.
• Token counts are reported, but not converted to cost.
• reading_level uses Flesch-Kincaid with a heuristic syllable count, and the refusal phrase list is English. Both are estimates; leave a margin.
• The recorded API responses in the test fixtures were written by hand in the Messages API format, not captured from live calls, and CI never calls the live API. The SDK request and response handling is exercised; the live service is not.
• JSON Schemas written inline in YAML or JSON lose the difference between {} and []. In schema positions (properties entries, items, allOf members and so on) an empty value is read as {}, the schema that accepts anything. Inside enum, const, default and examples, an empty map is read as an empty list.
• Developed and tested on Linux and macOS. Path handling on Windows has not been tested.

How this was built

Written with Claude Code under my direction and review.

License

MIT. See LICENSE.