Integrate Steward with CI

A worked example of polling Steward from a CI job. Covers the actual integration patterns most teams reach for: gate your release on a clean evaluation, post the verdict to a chat channel, archive the report alongside your build artifacts.

Background reading: Using the Steward API covers the why-and-when; this guide covers the how-to in shell + Python.

Prerequisites

You need:

• A minted API token with the evaluations:read scope (Mint and manage API tokens).
• The token stored in your CI provider's secrets manager (e.g. STEWARD_TOKEN).
• A repo that's installed on Steward.
• A way to trigger evaluations — typically Steward fires on PR open/sync automatically; for a release-gate scenario you'll trigger from the browser before kicking off the release job.

The shape

Two CI patterns dominate:

1. Trigger-elsewhere, poll-in-CI — a human (or another system) triggers an evaluation via the browser; the CI job is handed the job_id as an input and polls until it lands done.
2. PR-auto-triggered, poll-by-commit — a pull_request event auto-fires the evaluation. The CI job polls EvaluationsService.List for the most recent eval on the head SHA, then waits for it to land done.

The first is cleaner for a release-gate scenario where the trigger is human-orchestrated. The second is the right default when CI is "the thing that runs on every PR."

This guide walks through the first pattern in detail; the second is a small variant covered at the end.

The polling loop, in shell

A self-contained Bash snippet you can drop into any CI job:

#!/usr/bin/env bash
set -euo pipefail

# Inputs from the CI job environment.
: "${STEWARD_TOKEN:?Missing STEWARD_TOKEN env var}"
: "${STEWARD_JOB_ID:?Missing STEWARD_JOB_ID env var}"

API="https://steward-dev.ai/api/steward.api.v1.EvaluationsService/Get"

# Tier-by-elapsed cadence: matches the recommendation in the
# Using-the-API concept doc. Initial poll is fast; we slow
# down as time passes.
cadence_seconds() {
    local elapsed=$1
    if   [[ $elapsed -lt   2 ]]; then echo 2
    elif [[ $elapsed -lt  60 ]]; then echo 5
    elif [[ $elapsed -lt 300 ]]; then echo 15
    elif [[ $elapsed -lt 900 ]]; then echo 60
    else echo 0  # signal: give up
    fi
}

start=$(date +%s)
while true; do
    response=$(curl -s -X POST "$API" \
        -H "Authorization: Bearer $STEWARD_TOKEN" \
        -H "Content-Type: application/json" \
        -d "{\"id\":\"$STEWARD_JOB_ID\"}")

    status=$(jq -r '.job.status // .error.code // "unknown"' <<< "$response")

    case "$status" in
        done)
            echo "Evaluation done"
            jq -r '.job.report_md' <<< "$response"
            exit 0
            ;;
        failed)
            echo "Evaluation failed:"
            jq -r '.job.error_message' <<< "$response"
            exit 1
            ;;
        skipped)
            echo "Evaluation skipped (typically budget_exhausted)"
            jq -r '.job.error_message' <<< "$response"
            exit 2
            ;;
        queued|running)
            elapsed=$(( $(date +%s) - start ))
            wait=$(cadence_seconds "$elapsed")
            if [[ "$wait" == 0 ]]; then
                echo "Giving up after $elapsed seconds (status: $status)"
                exit 3
            fi
            sleep "$wait"
            ;;
        *)
            echo "Unexpected status: $status"
            echo "$response"
            exit 4
            ;;
    esac
done

Drop it into your repo at .ci/steward-wait.sh, mark it executable, and call it from your CI job after the trigger. The STEWARD_TOKEN comes from your secrets manager; the STEWARD_JOB_ID is the value returned by the EvaluationsService.Create call you made out-of-band.

Exit codes

The script uses meaningful exit codes so your CI can route on the outcome:

Code	Meaning
0	Evaluation completed cleanly. report_md is on stdout.
1	Evaluation failed. Investigate via the audit log.
2	Skipped (typically budget_exhausted). Not a bug.
3	Polling timed out (15 minutes elapsed). Treat as stuck.
4	Unexpected response. Inspect the script output.

A typical CI workflow:

- name: Wait for Steward evaluation
  run: .ci/steward-wait.sh > evaluation.md
  env:
    STEWARD_TOKEN: ${{ secrets.STEWARD_TOKEN }}
    STEWARD_JOB_ID: ${{ inputs.steward_job_id }}

- name: Archive evaluation report
  uses: actions/upload-artifact@v4
  with:
    name: steward-evaluation
    path: evaluation.md

The same loop, in Python

For richer integrations (parsing the JSON report, posting to chat, branching on per-dimension scores):

#!/usr/bin/env python3
"""Wait for a Steward evaluation to land, then print the report."""
import json
import os
import sys
import time
import urllib.request
from urllib.error import HTTPError

API = "https://steward-dev.ai/api/steward.api.v1.EvaluationsService/Get"


def cadence_seconds(elapsed: int) -> int:
    """Tier-by-elapsed cadence. 0 = give up."""
    if elapsed < 2:    return 2
    if elapsed < 60:   return 5
    if elapsed < 300:  return 15
    if elapsed < 900:  return 60
    return 0


def fetch(token: str, job_id: str) -> dict:
    """One call to EvaluationsService.Get. Raises on HTTP error."""
    req = urllib.request.Request(
        API,
        data=json.dumps({"id": job_id}).encode("utf-8"),
        headers={
            "Authorization": f"Bearer {token}",
            "Content-Type": "application/json",
        },
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=10) as resp:
        return json.loads(resp.read())


def main() -> int:
    token = os.environ["STEWARD_TOKEN"]
    job_id = os.environ["STEWARD_JOB_ID"]
    start = time.monotonic()

    while True:
        try:
            response = fetch(token, job_id)
        except HTTPError as e:
            if e.code == 429:
                # Honor Retry-After. See the
                # handle-errors-and-rate-limits guide.
                retry_after = int(e.headers.get("Retry-After", "10"))
                print(f"Rate-limited; sleeping {retry_after}s", file=sys.stderr)
                time.sleep(retry_after)
                continue
            raise

        job = response.get("job") or {}
        status = job.get("status", "unknown")

        if status == "done":
            print(job["report_md"])
            return 0
        if status == "failed":
            print(f"Evaluation failed: {job.get('error_message','')}", file=sys.stderr)
            return 1
        if status == "skipped":
            print(f"Evaluation skipped: {job.get('error_message','')}", file=sys.stderr)
            return 2
        if status not in ("queued", "running"):
            print(f"Unexpected status: {status}", file=sys.stderr)
            print(json.dumps(response, indent=2), file=sys.stderr)
            return 4

        elapsed = int(time.monotonic() - start)
        wait = cadence_seconds(elapsed)
        if wait == 0:
            print(f"Giving up after {elapsed}s (status: {status})", file=sys.stderr)
            return 3
        time.sleep(wait)


if __name__ == "__main__":
    sys.exit(main())

The Python version handles 429 Too Many Requests correctly by honouring the Retry-After header (the shell version doesn't — for production, prefer the Python script or extend the shell version per Handle errors + rate limits).

Pattern 2: poll-by-commit (PR-auto-triggered)

When evaluations are fired automatically on pull_request events and your CI doesn't have a job_id to wait on, list recent evaluations and pick the one matching your commit SHA:

LIST_API = "https://steward-dev.ai/api/steward.api.v1.EvaluationsService/List"

def find_by_sha(token: str, sha: str) -> dict | None:
    """Return the most recent evaluation matching this commit, or None."""
    req = urllib.request.Request(
        LIST_API,
        data=json.dumps({}).encode("utf-8"),
        headers={
            "Authorization": f"Bearer {token}",
            "Content-Type": "application/json",
        },
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=10) as resp:
        body = json.loads(resp.read())
    for job in body.get("jobs", []):
        if job.get("commit_sha") == sha:
            return job
    return None

Then poll for the discovered job's id (or absence) instead of a known STEWARD_JOB_ID. The trade-off:

• Pro: no human-orchestrated trigger step.
• Con: a small window between push and Steward enqueueing the evaluation where the list query returns no match. Handle "no match yet" with a few retries before treating it as "evaluation never fired" — typically 5–10 seconds is enough.

Posting to chat

The report_md field on the done response is a human-readable markdown summary you can drop straight into Slack, Discord, or any chat surface that renders markdown. A sketch (Python):

import urllib.request

def post_to_slack(webhook_url: str, report_md: str) -> None:
    payload = json.dumps({"text": report_md}).encode("utf-8")
    req = urllib.request.Request(
        webhook_url,
        data=payload,
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    urllib.request.urlopen(req).read()

For Discord, swap the webhook URL and use content as the payload key instead of text; the rest of the request shape is identical.

Gating a release

The common release-gate pattern, full shape:

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - name: Trigger Steward evaluation
        id: trigger
        # This step runs from a browser-authed flow OR
        # uses the (currently reserved) evaluations:run
        # scope if/when it ships. For now, a human triggers
        # via /app/repos/<id> and pastes the job_id into
        # the workflow dispatch input.
        run: |
          echo "job_id=${{ inputs.steward_job_id }}" >> "$GITHUB_OUTPUT"

      - name: Wait for evaluation
        id: wait
        run: .ci/steward-wait.sh > evaluation.md
        env:
          STEWARD_TOKEN: ${{ secrets.STEWARD_TOKEN }}
          STEWARD_JOB_ID: ${{ steps.trigger.outputs.job_id }}

      - name: Gate the release on the verdict
        run: |
          # Extract overall_score from the JSON via a follow-up Get
          # (or have the wait script emit both md + json).
          # Simplistic example: fail if the report contains "Steward verdict: <50".
          if grep -qE "Steward verdict: [0-9]+ / 100 — red" evaluation.md; then
            echo "Verdict red — refusing to release"
            exit 1
          fi

  release:
    needs: evaluate
    runs-on: ubuntu-latest
    steps:
      - run: ./scripts/release.sh

The release job depends on evaluate, so a non-zero exit from the polling script stops the pipeline before the release runs.

Polling-cadence justification

Why the specific 2 / 5 / 15 / 60 / give-up tiers?

Window	Cadence	Cost in API calls
0–2s	2s	~1 call
2–60s	5s	~11 calls
60s–5min	15s	~16 calls
5min–15min	60s	~10 calls
Total worst case		~38 calls

Against the 60 req/min rate limit, this leaves comfortable headroom for other concurrent integrations under the same token. The cadence matches the median time-to-done (<20 seconds) at the high-resolution end and gracefully backs off when something's clearly stuck.

If your integration triggers many evaluations in parallel, budget for ~38 × concurrency calls per evaluation and consider minting separate tokens per integration to give each its own 60 req/min budget.

Common gotchas

• Hardcoding the token in a script. Always read from the CI secrets manager. A token in a script that lands in a public repo is a published credential.
• Not handling skipped. A skipped(budget_exhausted) result isn't an error — it's "the tenant hit the daily cap." Branch accordingly; don't fail the CI job and page on-call.
• Polling too fast. A 1-second cadence over a 15-minute worst case sends ~900 calls per evaluation. You'll hit the rate limit; the integration will be slower than polling every 5 seconds.
• Treating running as an error condition. It's normal. Keep polling.
• Not capturing report_md somewhere durable. The CI job's stdout disappears when the job's logs roll. Archive the report as an artifact (the workflow YAML above shows the shape) or post it to a durable surface.

What's next

• Handle errors + rate limits — the full error envelope catalogue + backoff strategy (forthcoming in slice 3).
• /docs/api — the per-RPC reference for the endpoints used in this guide.
• Using the Steward API — the concept doc covering when to reach for the API.