Steward
Sign in

Documentation › Maintainer guides

Read an evaluation report

The verdict shows up in two places: as a comment on the pull request, and as a row in the per-repo PR list. This guide walks through both surfaces and how to act on what you see. Background reading: Evaluations + dimensions — this guide assumes you know what the five dimensions are and what the status colours mean.

Surface 1: the PR comment

When the evaluation reaches done, Steward posts (or updates) a comment on the PR. The shape:

**Steward verdict: 78 / 100 — amber**

> Solid surgical change to the auth middleware with tests
> alongside, but the diff is on the heavy side for a single
> review pass.

### Correctness — 85
Source files changed alongside a test file per package; ratio
sits above the 0.33 threshold.

### Security — 65
Touched internal/auth/middleware.go with paired test changes;
amber rather than green because the sensitive surface still
warrants a focused review.

### Complexity delta — 55
432 LoC across 18 files. On the boundary between "medium" and
"large"; consider splitting.

### Docs alignment — 90
docs/auth/middleware.md updated alongside the code.

### Governance alignment — 75
governance.pxf present; touched files map to the `auth` domain.

Read it in this order:

  1. The number. Triage signal only. Green = move on; amber = look at one dimension you care about; red = open the PR and read it carefully.
  2. The headline. One-sentence summary the Large Language Model (LLM) wrote. If it feels off, the per-dimension paragraphs will say why.
  3. The dimensions you care about for THIS PR. A docs PR's security dimension is Unknown and you can ignore it; a backend refactor's complexity dimension is the one to read first.

What to do per status

Verdict Action
Green overall, all dimensions green or unknown Merge if the human review agrees.
Green overall, one amber/red dimension Open the dimension paragraph. The score is averaged — one outlier doesn't tank the result. Decide whether the outlier is acceptable for THIS change.
Amber overall Read every non-Unknown dimension. Usually the score is dragged down by one of: complexity, docs alignment, security-without-tests. Pick the one you'd ask a human reviewer to focus on.
Red overall The deterministic scorer found something hard. Almost always: source change with no tests, or a >1000-LoC PR, or a sensitive-surface change with no tests. Treat the report as a checklist.

When the LLM prose disagrees with the numbers

The LLM is instructed to defer to the deterministic scores and not invent higher or lower ratings. If you spot a paragraph that contradicts the number (says "great test coverage" next to a 40 correctness score), screenshot it and file a bug. It's usually a prompt-template regression we want to know about.

When the prose says "deterministic fallback"

If the report header includes [deterministic-only — LLM unavailable], the LLM call failed (timeout, API error, no key configured) and you're reading a template. The dimension scores are still accurate; the prose is just less context-aware. Verify your LLM provider config if this persists across PRs.

Surface 2: the per-repo PR list

For maintainers reviewing PR activity at the repo level:

/app/repos/<repo-id>/pulls

One row per PR-evaluation, newest first. Columns:

Column What you're looking at
PR Number + title, link to GitHub.
Author GitHub login + their reputation in the touched domains.
Head SHA Short SHA of the head commit evaluated. Lets you cross-reference an evaluation with a specific push.
Status queued / running / done / failed / skipped(budget_exhausted). See Evaluations + dimensions for what each means.
Overall Numeric score + colour pill.
Updated Last write to the row.

Filter by status to triage:

  • failed — your investigation queue. Click a row to see the underlying error; cross-reference with Debug a failed evaluation.
  • skipped(budget_exhausted) — a budget signal, not a bug. If you see a sustained wave, either raise the tier (sponsor surface) or wait for the daily reset.
  • done + red — PRs that scored badly. Worth a Monday- morning sweep even if you already approved them — the reputation feedback loop will weight the author's domain rep according to these outcomes.

When to disagree with a verdict

The verdict is advisory. There are three legitimate reasons to merge a red-overall PR anyway:

  1. The PR is a deliberate rule-bender — a 2000-LoC refactor that genuinely needed to land as one atomic change. complexity_delta will score red; that's the right signal but not the right blocker.
  2. The dimensions don't apply. A migration script that shouldn't have tests will fail the correctness gate. The Unknown heuristic catches the obvious cases (docs-only, config-only) but isn't exhaustive.
  3. You disagree with the LLM's prose. The deterministic scores stand on their own; the prose is interpretation. Read the dimensions table, ignore the paragraph, decide.

What you should not do:

  • Don't game the path heuristic. Renaming auth_check.go to validator.go will silence the security dimension but not the actual reviewer-attention need. Use the gate as a signal to request a review, not as something to silence.
  • Don't ignore a red correctness score because "tests aren't worth writing for this." If you genuinely believe that, the domain in governance.pxf should have its min_coverage threshold lowered. Encoding the disagreement in the configuration is more durable than ignoring the signal.

A worked example

A PR titled "feat(api): add /v2/exports endpoint with paginator":

**Steward verdict: 62 / 100 — amber**

> New endpoint with adjacent unit tests but no integration
> coverage and no docs update describing the response shape.

### Correctness — 75 (green)
3 source files, 1 test file; ratio at threshold.

### Security — 90 (green)
No security-sensitive paths touched.

### Complexity delta — 80 (green)
234 LoC across 6 files. Medium-sized; tracks with the scope
described in the title.

### Docs alignment — 50 (amber)
6 source files changed, no docs updates. Consider documenting
the response shape under docs/api/.

### Governance alignment — 75 (green)
governance.pxf present; touched files map to the `api` domain.

How to read this:

  • Amber overall, dragged down by docs. The headline calls it out in the first sentence.
  • Correctness is at threshold but not green-green — the test ratio is exactly at 0.33 which is the floor. If the author adds one more test the score jumps to 90; otherwise it's a legitimate "this could use one more test" signal.
  • Action: ask the author to update docs/api/. Don't merge until that lands.

This is the value loop. The verdict gives you a checklist; you either act on it or you commit to the disagreement by editing governance.pxf and merging once the rules match your judgment.

What's next