Steward
Sign in

Documentation › Maintainer guides

Debug a failed evaluation

When an evaluation reaches a non-done terminal state, this is the triage playbook. We'll cover the two failure modes — failed and skipped(budget_exhausted) — and walk through the specific causes the worker writes to the row's error_message.

Background reading: Evaluations + dimensions covers the full status lifecycle; this guide assumes you know the terminal states.

First: is this failed or skipped?

The two terminal-non-done states are different problems. Don't confuse them. From the per-repo PR list at /app/repos/<id>/pulls:

Status What it means Operator action
failed The worker tried to score and something broke. Investigate. The triage tree below.
skipped(budget_exhausted) The tenant hit the daily credit cap before the worker spent anything. Raise the tier, or wait for the daily reset. Not a bug.

A wave of skipped(budget_exhausted) means your tenant is running hot — likely a busy repo got pushed-to-Pro tier traffic on a Community tier subscription. Look at usage at /sponsor/usage and either upgrade or pause the offending repo.

A wave of failed means something is broken. Read on.

The triage tree for failed

Click into a failed row to see the error_message field (capped at 2000 chars). Match it against the categories below.

1. installation token: ...

The Steward GitHub App couldn't mint an installation token for the repo's installation. Common causes:

  • The App was uninstalled between when the PR webhook fired and when the worker picked the row up. Reinstall the App on the repo if you still want evaluations.
  • The installation was suspended by the org admin. Check https://github.com/organizations/<org>/settings/installations and re-enable.
  • The App's private key rotated without the runtime picking up the new value. Sponsor/staff issue; file a ticket.

After fixing, re-trigger the evaluation by pushing a commit to the PR head or commenting /steward evaluate on the PR.

2. collect pr: ...

The PR-corpus collector couldn't fetch the PR's diff + metadata. Common causes:

  • 404 Not Found — the PR was closed or the head branch deleted between webhook fire and worker pickup. Nothing to do; if the PR is reopened, a fresh evaluation will be triggered.
  • 403 Forbidden — the App's installation grants don't cover the repo (e.g. you removed the repo from the installation's allowlist). Add it back.
  • GitHub API rate-limit — 403 with a rate limit exceeded substring. Wait 60 seconds and re-trigger; if it persists, your tenant is generating more traffic than the App installation token can sustain. Talk to staff.

3. repo install context: ...

The DB lookup that resolves the repo's installation context failed. Almost always transient (Postgres hiccup) — the row's been moved to failed because the worker doesn't retry, but the next push or /steward evaluate comment will queue a fresh row that probably succeeds.

If you see this repeatedly for the same repo, file a ticket.

4. marshal report: ...

Internal — the synth.Report struct couldn't be serialized to JSON. This is a Steward-side bug. File a ticket with the row id

  • the truncated error_message. The score itself was computed correctly; only the persistence step broke.

5. context deadline exceeded

The worker enforces a 5-minute per-job timeout (JobTimeout). Common causes:

  • GitHub is slow. A handful of evaluations a day exceeding 5 minutes is normal API jitter. A sustained wave is a GitHub-side incident — check status.github.com before investigating further.
  • The Large Language Model (LLM) is slow. Steward already falls back to synth.PRDryReport (the deterministic-only report) when the LLM errors, so a deadline exceeded from the LLM side typically means the LLM timed out rather than returning an error. Verify your LLM provider's status; if it's a recurring problem with one model, raise it on the staff channel.

6. mark done: ...

The score landed, but the DB write that records "done + report_jsonb" failed. The score is lost; the row reads failed. Almost always transient — re-trigger the evaluation and it should land.

If you see this repeatedly, the issue is likely DB write-load saturation. Sponsor/staff issue.

7. Anything else

Includes parse errors on governance.pxf (rare — the file is validated when committed via the editor) and unexpected nil pointers (Steward-side bugs).

For an unknown error_message, the workflow is:

  1. Copy the row id and the full error_message to a ticket.
  2. Check whether the PR is still relevant. If the PR was closed, you can probably ignore the row.
  3. Re-trigger the evaluation via /steward evaluate and see whether the second attempt succeeds. Transient failures re-trigger fine; structural ones repeat.

The "stuck running" recovery

A running row that the worker abandoned (crash, OOM, mid-deploy SIGTERM) gets re-queued automatically at the next worker boot via recoverStuck. The match condition is "running with started_at older than max(2 × JobTimeout, 30 minutes)," so a transient hang has time to resolve before recovery kicks in.

You don't need to do anything for stuck rows — the boot-time sweep handles them. If you see a row that's been running for hours without recovery, the worker is wedged or hasn't restarted; that's a sponsor/staff issue.

Re-triggering an evaluation

Three ways to score a PR again after the original evaluation ended in failed or skipped:

Method Use when
/steward evaluate comment on the PR Quick one-shot. Works regardless of the PR's tenant tier.
Push a commit to the PR head Webhook auto-triggers a fresh evaluation. Natural workflow.
Manual run from /cc/admin/pr-evals/run Staff-only. For incident recovery sweeps over many PRs at once.

A re-trigger creates a new row in app.pr_evaluations. The original failed row stays — the audit trail is preserved. The per-repo PR list shows both; the most recent is the source of truth.

When to file a ticket

Self-resolvable:

  • installation token: ... — uninstall + reinstall fixes it.
  • collect pr: ... 404 — closed PR, harmless.
  • One-off repo install context: ... — transient.
  • One-off context deadline exceeded — likely a slow upstream.

Worth a ticket:

  • Repeated installation token: ... for the same repo (rotation issue you can't fix from your side).
  • Sustained context deadline exceeded waves (LLM provider or GitHub-side incident).
  • Any marshal report: ... — Steward-side bug.
  • Any mark done: ... that recurs.
  • Any unknown error_message.

When filing, include: row id, repo full name, PR number, head SHA, and the truncated error_message from the row. Staff have access to richer logs that include the untruncated stack trace keyed off the row id.

What's next