Steward
Sign in

Documentation › Concepts

Evaluations + dimensions

An evaluation is Steward's verdict on a single pull request: a numeric overall score, a status colour (green/amber/red), and five per-dimension breakdowns with a short headline each. This doc explains where those numbers come from so you can trust them when they agree with your gut and push back when they don't.

The pipeline in one paragraph

When a PR opens or pushes, GitHub fires a webhook. The trigger service decides whether to enqueue an evaluation (per the gates in Gates: DCO, CLA, complexity and the tenant's tier). A worker picks the row off the queue, fetches the diff + repo metadata via the installation token, runs a deterministic scorecard against the files, then asks a Large Language Model (LLM) to write a short report grounded in those numbers. The result lands as a JSON blob on app.pr_evaluations, surfaces as a PR comment, and shows up in the fleet dashboard's per-repo PR list.

The five PR dimensions

Each PR is scored on five orthogonal dimensions. Each dimension returns one of four states:

Status Score What it means
Green ≥ 75 This dimension looks healthy.
Amber 50–74 Worth a second look, not a blocker.
Red < 50 Something the deterministic scorer flagged hard.
Unknown — The dimension doesn't apply to this PR (e.g. correctness on a docs-only change).

The five dimensions, in display order:

1. Correctness

Test-to-source ratio across the changed files. Source files are the 10 languages Steward governs (.go .py .ts .tsx .java .rs .cs .cpp .dart .swift + their cousins). A test file is any file under /test/, /tests/, /__tests__/, or matching *_test.* / *.test.* / *.spec.* / test_*.py.

Pattern Score
Source changed, no test files updated 40 (red)
Source changed, test ratio ≥ 0.33 (one test per ~3 source files) 90 (green)
Source changed, test ratio ≥ 0.20 75 (green)
Anything else 60 (amber)
No source files changed (docs/config only) Unknown

The LLM layer reads "Unknown" as "no opinion here" and won't nag you about it.

2. Security

Path-based heuristic looking for sensitive substrings in the changed file paths: auth, authn, authz, oauth, oidc, crypto, encrypt, decrypt, jwt, token, password, secret, credential, kms, tls, ssl, webhook, signature, hmac, sso, saml.

Pattern Score
No sensitive paths touched 90 (green)
Sensitive paths + test changes alongside 65 (amber)
Sensitive paths, no tests 35 (red)

The dimension is path-only; a typo fix in auth.go flags the same as a real auth rewrite. The LLM is expected to disambiguate in the prose. The path signal just anchors the claim "this PR touched security surface."

3. Complexity delta

Diff churn (additions + deletions) and file count. The thresholds match how reviewers talk about PRs verbally:

Diff size Score
≤ 100 LoC 90 (green) — "small PR"
100–400 LoC, 10–20 files 75 (green) — "medium"
400–1000 LoC, 20–50 files 55 (amber) — "large"
> 1000 LoC or > 50 files 30 (red) — "huge"

A lopsided diff — huge additions with near-zero deletions (deletions < additions / 10 and total > 200) — drops the score by 10. Clean refactors balance the two; net-add-only PRs suggest scope creep.

4. Docs alignment

Were docs updated when source changed?

Pattern Score
Docs updated alongside source 90 (green)
≥ 5 source files changed, no docs 50 (amber)
< 5 source files, no docs 70 (amber/green)
No source changed Unknown

"Docs" means anything matching *.md, *.markdown, *.rst, *.adoc, paths under docs/, or top-level README*, CHANGELOG*, LICENSE*, CONTRIBUTING*, CODE_OF_CONDUCT*, SECURITY*.

5. Governance alignment

Does the repo have a governance.pxf on its default branch and do the touched files map cleanly to its domains?

Pattern Score
governance.pxf present 75 (green)
No governance.pxf Unknown

This scorecard dimension only notes whether governance exists — the rich per-path machinery lives on the PR-gate side (gates + enforcement modes, per-domain thresholds, presets), not in this score. The LLM layer uses the presence flag to decide how opinionated to be in its prose. A repo with no governance gets softer language; a repo with a deep governance.pxf gets prose that references its domains by name.

The overall score

overall = mean(scores of dimensions that are NOT Unknown)

That's it — simple unweighted mean over the non-Unknown dimensions. The overall status colour applies the same 75/50 thresholds.

There is no per-dimension override in the overall: a red correctness alongside four greens still rounds to amber if the arithmetic says so. The intent is: the dimension breakdown is where the nuance lives. The headline number is a triage signal, not a verdict.

The LLM-grounded prose

Once the deterministic scorecard exists, Steward calls an LLM (via synth.GeneratePR) with a structured prompt:

  • The five dimension scores + their evidence blobs (file counts, sensitive paths, additions/deletions).
  • The PR title + body.
  • The repo's governance.pxf content (when present).

The LLM writes a short summary headline + one paragraph per dimension. It's instructed to defer to the deterministic scores (no inventing higher or lower ratings) and to stay silent on Unknown dimensions.

If the LLM is unavailable (timeout, API error, no key configured), Steward falls back to synth.PRDryReport — a template-based summary using just the scorecard numbers. The report makes the fallback explicit so a reader knows whether they're seeing LLM prose or a template.

Status lifecycle

A row in app.pr_evaluations moves through these states:

queued → running → done
                 ↘ failed
                 ↘ skipped(budget_exhausted)
Status What it means Operator response
queued Worker hasn't claimed it yet. None — usually drains in seconds.
running Worker is fetching the PR + scoring. None — typical duration < 10s.
done Report written. PR comment posted. Read the verdict.
failed Something broke (GitHub API error, parse error). See Debug a failed evaluation.
skipped(budget_exhausted) Tenant hit the daily credit cap. Bump the tier or wait until the daily reset. Distinct from failed so dashboards can show "rate-limited" vs "broken" separately.

The terminal-state distinction matters: a wave of skipped(budget_exhausted) reads as "tenant maxed out for the day"; a wave of failed reads as "something is broken and needs investigation."

What it costs

Each done PR evaluation that actually called the LLM costs 0.05 credits (a flat per-call rate). Manual repo evaluations share the same daily cap. The skipped terminal state happens when the pre-check for that 0.05 finds the cap already hit — no LLM call is made; no credits are charged for a skipped row.

The PRDryReport fallback (deterministic scorer only) is also zero credits. So a tenant whose LLM provider is temporarily down still gets verdicts — just template-based ones — at no cost.

What's next