Laravel Observability

Developed and maintained by IntrAgile Kft.

An observability kit for Laravel on top of spatie/laravel-health and spatie/laravel-prometheus: Loki-ready JSON logging with SQL redaction, the health checks Spatie does not ship, fail-closed basic auth for the metrics endpoint, a grouped Slack error digest, and a doctor command that probes the whole chain live.

Magyar nyelvű leírás → README.hu.md

The package builds on Spatie instead of replacing it. The check framework, the result store, notifications, the /prometheus endpoint and its IP allowlist all stay Spatie's; this package adds what is missing around them, wires it up from one config file, and tells you what it could not wire.

composer require intragile/laravel-observability
php artisan observability:install
php artisan observability:doctor --live

What it adds on top of Spatie

Requirements

• PHP 8.3+, Laravel 12 or 13.
• spatie/laravel-health 1.37+ (installed as a dependency).

Optional packages

Everything that depends on an optional package is switched on when the package is installed and skipped, without errors, when it is not. observability:install and observability:doctor show each skipped part and the reason.

composer require spatie/laravel-prometheus spatie/laravel-backup laravel/reverb phrity/websocket laravel/doctor

No spatie/laravel-prometheus release supports PHP 8.3 together with Laravel 13 (its Laravel 13 releases need PHP 8.4). On that combination the package runs without it and the metrics rows are skipped; everything else works.

Installation

composer require intragile/laravel-observability
php artisan observability:install

The service provider is registered automatically. observability:install:

1. publishes config/observability.php (tag observability-config);
2. publishes the error report migration (tag observability-migrations) when the error digest is enabled, unless a copy of it already exists under any timestamp;
3. prints a readiness table: every component with ok, warning or skipped and the reason;
4. prints the two snippets you add by hand: the logging channel and the exception hook.

The readiness table only reflects configuration and installed packages. To prove that the chain actually works, run php artisan observability:doctor --live (see Doctor).

Structured logging for Loki

Add a channel to config/logging.php and include it in your stack:

'json' => [
    'driver' => 'monolog',
    'handler' => Monolog\Handler\StreamHandler::class,
    'with' => ['stream' => 'php://stderr'],
    'formatter' => \IntrAgile\Observability\Logging\LokiJsonFormatter::class,
],

Every record becomes one line of JSON. On top of Monolog's JsonFormatter output:

• level holds the lowercase level word (error), because Loki's level detection reads level and expects a word; the numeric level moves to level_value, and level_name (ERROR) is kept;
• service and env are written to every line, followed by the static fields in logging.extra (which can never overwrite the core fields);
• stack traces are included by default;
• interpolated SQL is redacted from the message, from context.exception and from every exception in its previous chain.

The constructor arguments includeStacktraces, redactSql, service and extra override the config for a single channel through Laravel's formatter_with key.

Take Loki labels from container labels rather than from the JSON: process output that does not go through Monolog (php-fpm, supervisord) never reaches this formatter.

SQL redaction

Laravel's QueryException message contains the full SQL with every binding substituted. Logged as it is, salaries, addresses or password hashes end up in log storage that usually has no retention limit. IntrAgile\Observability\Support\SqlMessageRedactor keeps the SQLSTATE code, the driver's error text and the connection details, and replaces the SQL tail:

SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'x' for key 'users_email_unique' (Connection: mysql, SQL: [redacted])

The driver's own message is kept on purpose, because without it the error loses its meaning. It may contain a value, as in the Duplicate entry example above.

Health checks

With observability.health.register set to true (the default) the enabled checks are registered with spatie/laravel-health automatically. A check whose name is already registered is left alone, so registering one of them yourself does not break boot. Set OBSERVABILITY_HEALTH_REGISTER=false to register them yourself; each check also works with its fluent setters, for example FailedJobsCheck::new()->windowHours(12).

Schedule Spatie's runner as usual, or the results go stale (the doctor warns about it):

// routes/console.php
use Spatie\Health\Commands\RunHealthChecksCommand;

Schedule::command(RunHealthChecksCommand::class)->everyMinute();

Or let the package schedule it, with a warmup after each deploy (see below).

All keys below live under observability.health.checks, and each check has an enabled key (default true).

Notes on individual checks:

• MailRelayCheck. Catcher hosts match whole dot- or hyphen-separated labels, so mail.example.com is not a catcher. Only the SMTP transport is inspected; other transports (SES, Postmark) are assumed to deliver. When the silence is deliberate, point hold_config_key at a config key: a truthy value turns the failure into ok, and a free-text value is shown as the reason.
• RequestTimingCheck. When the check is enabled, the provider pushes RecordRequestTiming into the listed middleware groups. The middleware adds an X-Response-Time header and keeps the slowest request in the cache. After a warning or failure the measurement is cleared, so the next run only sees requests served after the alert. Per-route limits are keyed by route name (the path for unnamed routes): 'routes' => ['reports.export' => ['warn' => 3000, 'fail' => 10000]]. Requests that never reach a route group (for example 404s) are not measured.
• BackupFreshnessCheck. A listener stores a timestamp whenever spatie/laravel-backup fires BackupWasSuccessful, so a normal run is a single cache read. The thresholds are compared in seconds: 26 hours and 1 minute is past a 26-hour limit. If no heartbeat exists yet, the newest archive on the backup disks seeds it once. Use a persistent cache store; the readiness table warns when the heartbeat store uses the array driver.
• BroadcastCheck. See Broadcast round trip.
• ContainerMemoryCheck. Reads memory.max and memory.stat (anon) on cgroup v2, or their v1 equivalents (rss). Page cache is excluded. It is not registered when no cgroup memory limit is found, for example on a development machine.

Scheduling with a deploy warmup

Right after a deploy, heartbeats such as Spatie's queue and schedule checks have not been written yet (often because the cache prefix changes with the release), so the first runs report failures on a healthy system. With OBSERVABILITY_HEALTH_SCHEDULE=true the package schedules health:check itself and keeps it off the schedule for warmup_seconds after the first scheduler tick of a deploy. Every other scheduled command keeps running.

OBSERVABILITY_HEALTH_SCHEDULE=true
# Optional: restarts the warmup on each release even when storage survives the deploy.
OBSERVABILITY_DEPLOY_ID="${GIT_COMMIT_SHA}"

The deploy marker lives in storage/app/observability/health-warmup.json (schedule.flag_path). On container platforms where storage is recreated on each deploy, no deploy ID is needed. If the marker cannot be written, the check is scheduled anyway, so monitoring never stops silently. Keys under observability.health.schedule: enabled false, cron * * * * *, on_one_server true, warmup_seconds 300, flag_path null. During the warmup the doctor shows the schedule row as skipped, with the seconds left.

JSON endpoint for deploy gates

Spatie's /health-json?fresh runs the checks through the command that also sends notifications, so a deploy gate polling it right after the switch causes alerts. With OBSERVABILITY_HEALTH_JSON=true the package registers HealthJsonController, which behaves the same but runs the checks with --no-notification. The scheduled run stays responsible for alerting.

// config/observability.php
'json_endpoint' => [
    'enabled' => env('OBSERVABILITY_HEALTH_JSON', false),
    'uri' => 'health-json',
    'middleware' => ['auth.basic'],
    'name' => 'observability.health-json',
],

The route has no middleware by default; protect it, because the results name your infrastructure. Spatie's health.json_results_failure_status and health.oh_dear_endpoint.always_send_fresh_results are honoured. To use the controller on your own route instead, leave the endpoint disabled and point the route at IntrAgile\Observability\Http\Controllers\HealthJsonController.

Notifications

IntrAgile\Observability\Health\CheckFailedNotification extends Spatie's notification. It sends only from the environments in health.notifications.environments (default ['production']), unless HEALTH_NOTIFICATIONS_ENABLED forces it on or off (a value that is not a boolean, such as a typo, is ignored rather than read as false). Its title carries the environment outside production, the instance name and the deploy ID, for example [staging] shop-eu [deploy: 4ad9758]. Register it in config/health.php:

'notifications' => [
    'enabled' => true,
    'notifications' => [
        IntrAgile\Observability\Health\CheckFailedNotification::class => ['slack'],
    ],
    // Spatie's throttle still applies.
    'throttle_notifications_for_minutes' => 15,
],

The instance name comes from health.notifications.instance (APP_ALIAS, then APP_NAME).

Prometheus

Requires spatie/laravel-prometheus. The package appends IntrAgile\Observability\Http\Middleware\PrometheusBasicAuth to prometheus.middleware, after Spatie's AllowIps, so both apply.

PROMETHEUS_USERNAME=prometheus
PROMETHEUS_PASSWORD=a-long-random-secret
# or, with a Docker or Kubernetes secret (a trailing newline is ignored):
PROMETHEUS_PASSWORD_FILE=/run/secrets/prometheus_password

The middleware fails closed: when neither PROMETHEUS_PASSWORD nor a readable, non-empty PROMETHEUS_PASSWORD_FILE yields a password, every request gets 401, including one with an empty password. Username and password are both compared with hash_equals. Configure the same credentials in the scrape job:

scrape_configs:
  - job_name: laravel
    metrics_path: /prometheus
    basic_auth:
      username: prometheus
      password_file: /etc/prometheus/laravel_password
    static_configs:
      - targets: ['app.example.com']

With basic_auth.enabled set to false the middleware is not attached, and the endpoint is protected only by prometheus.middleware and prometheus.allowed_ips.

Gauges. Inside a memory-limited container the package registers container_memory_limit_bytes and container_memory_used_bytes. Spatie prefixes the names with prometheus.default_namespace, so they appear as app_container_memory_limit_bytes and app_container_memory_used_bytes by default. Without a cgroup limit they are not registered.

Queue collectors. queue_collectors.enabled registers Spatie's queue size, pending, delayed, reserved and oldest pending job collectors for the listed queues.

Error digest

Records every reported exception and posts a grouped summary to a Slack incoming webhook. It is off by default.

OBSERVABILITY_ERROR_DIGEST=true
OBSERVABILITY_SLACK_WEBHOOK=https://hooks.slack.com/services/...

1. Enable it, then publish and run the migration (observability:install publishes it when the digest is already enabled):

   php artisan vendor:publish --tag=observability-migrations
   php artisan migrate
   

2. Hook the recorder into exception reporting. In bootstrap/app.php:

   use Illuminate\Foundation\Configuration\Exceptions;
   use IntrAgile\Observability\ErrorDigest\ExceptionHook;
   
   ->withExceptions(function (Exceptions $exceptions) {
       ExceptionHook::register($exceptions);
   })
   

   An application that still has a classic App\Exceptions\Handler registers it there:

   use IntrAgile\Observability\ErrorDigest\ErrorRecorder;
   use Throwable;
   
   public function register(): void
   {
       $this->reportable(function (Throwable $e) {
           app(ErrorRecorder::class)->record($e);
       });
   }
   

   The hook does not stop propagation, so regular logging continues. The recorder never throws: when the table is missing or the database is down it logs a warning and returns, and a static guard prevents recursion. While the digest is disabled it writes nothing.

3. Scheduling is automatic. While the digest is enabled, the provider schedules observability:error-digest with the schedule cron expression, withoutOverlapping() and onOneServer(). Your scheduler (schedule:run or schedule:work) has to be running.

Each run claims the IDs of at most batch_size unsent rows, groups them by exception type and redacted message, and posts one message: the max_groups largest groups, each with its count in the batch and in the last 24 hours, plus a "+N further error type(s)" line. Exactly the claimed rows are marked as sent, and only after Slack answered with a 2xx status. When Slack is down, every row stays unsent and the next run sends it. A cache lock stops two runs from sending the same rows. Sent rows older than retention_days are deleted. Outside production the instance name is prefixed with the environment, e.g. [staging] shop.

Messages are stored already redacted, so SQL bindings never reach the table. Treat the webhook URL as a secret: anyone who has it can post to the channel.

Broadcast round trip (Reverb)

Requires laravel/reverb and phrity/websocket. Instead of trusting that an event was dispatched, BroadcastCheck connects to Reverb as a client would, subscribes to a one-off public channel (observability-probe.<uuid>), broadcasts an observability.probe event carrying a random nonce through the application's broadcaster, and waits until that nonce arrives. The timeout bounds the whole exchange, so a down or silent Reverb fails the check instead of stalling the health run.

By default the WebSocket URL is built from broadcasting.connections.reverb.options (host, port, scheme). When clients reach Reverb through a different address, for example behind a reverse proxy or TLS terminator, set the client URL; only its scheme, host and port are used (https becomes wss, http becomes ws):

OBSERVABILITY_BROADCAST_URL=https://ws.example.com

Root guard

Artisan run as root (for example from a docker exec root shell) leaves cache and storage files owned by root, and the web server user can no longer write them. In the environments listed in root_guard.environments (default ['production']) Artisan refuses to run as root and explains how to run the command as the web server user instead.

For a deliberate one-off: ARTISAN_ALLOW_ROOT=1 php artisan <command>. In CI pipelines that run as root, set APP_ENV to a non-guarded value before composer install, because Composer's package:discover hook is already an Artisan call. The guard never runs under unit tests.

Doctor

php artisan observability:doctor
php artisan observability:doctor --live --strict
php artisan observability:doctor --live --json --timeout=10

laravel/doctor checks configuration. observability:doctor pushes something through each part of the chain and reports what came out. It starts with the readiness table from the installer, then runs the probes:

A probe that throws becomes a failed row and the run continues, so one broken component does not hide the others. Exception messages in the report are SQL-redacted.

As a deploy gate

Run it after migrations and before switching traffic:

php artisan migrate --force
php artisan observability:doctor --live --strict --json > observability-report.json

A non-zero exit stops the pipeline. Without --live and --send-test the command opens no WebSocket and sends no HTTP request outside the process (the Prometheus probe goes through the HTTP kernel in-process), so it is safe in any environment. It does query the database to check the error report table when the digest is enabled.

laravel/doctor integration

When laravel/doctor is installed, four diagnostics appear in its observability group: LoggingIsStructured, PrometheusIsProtected, HealthChecksPass and ErrorDigestIsDeliverable.

php artisan doctor --only=observability
php artisan doctor --only=observability --format=json

Only the in-process probes run there; no WebSocket connection is opened and no Slack message is sent. HealthChecksPass does not run the checks. It reports which ones are registered and scheduled, and reads the latest stored run from Spatie's result store: failed or crashed checks fail it, and a missing run or one older than health.stored_results.stale_minutes (default 10) warns. For the live probes use observability:doctor --live.

What is skipped when not installed

The application boots unchanged in each case; the readiness table and the doctor report show a skipped row with the reason.

Configuration reference

php artisan vendor:publish --tag=observability-config

Testing

composer test
OBSERVABILITY_REVERB_TEST=1 composer test   # also runs the round trip against a real Reverb server

License

MIT, see LICENSE.

Developed and maintained by IntrAgile Kft.