Handle errors + rate limits

The Steward API surfaces two error shapes — HTTP status errors at the auth/transport layer and envelope errors inside the response body for application-level failures. This guide catalogues both, walks through the retry decisions each one warrants, and shows the backoff pattern a production-grade integration should use.

Background: Using the Steward API covers the API shape. Integrate Steward with CI shows the polling pattern this guide hardens.

Two error shapes

Shape 1: HTTP status error

The auth / rate-limit / transport layers reject before the RPC handler runs. The response is a JSON object with status_code + message:

HTTP/1.1 401 Unauthorized
Content-Type: application/json

{
  "status_code": 401,
  "message": "Invalid token."
}

You get this shape for: missing/malformed/revoked tokens, rate-limit exceedances, server errors, maintenance windows.

Shape 2: Envelope error

The RPC handler returns successfully (HTTP 200), but the response body's error field is set:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "error": {
    "code": "NOT_FOUND",
    "message": "Evaluation not found."
  }
}

You get this shape for: missing resources, scope mismatches, invalid arguments, state-blocked operations, budget caps.

Always check both. A naive client that only checks HTTP status will miss the envelope errors entirely; one that only parses response.error will silently treat a 401 as "no error, response.error is undefined."

HTTP status catalogue

`401 Unauthorized`

Message variants:

"Invalid token." — the bearer token is missing, malformed (after the prefix check), revoked, or unknown to Steward.
"Malformed bearer token." — the Authorization header was present but didn't conform to Bearer <token> format.

Why the messages are vague: probes asking "is this a real token?" vs "is this revoked?" vs "is this rate-limited?" should all get the same answer at the auth layer. The ambiguity is deliberate.

Retry policy: don't. A 401 won't fix itself. Verify your token's plaintext + the Authorization header formatting; mint a fresh token if revoked or unknown.

`429 Too Many Requests`

HTTP/1.1 429 Too Many Requests
Retry-After: 23
Content-Type: application/json

{
  "status_code": 429,
  "message": "Rate limit exceeded (60 req/min); retry after 23s."
}

You exceeded the 60 req/min per-token budget within the current per-minute bucket.

The Retry-After value is honest — sleep for exactly that many seconds, then resume. Backing off longer is fine; backing off shorter earns another 429.

Retry policy: yes, after Retry-After seconds. This is the one HTTP error that's transient by design.

`403 Forbidden`

A narrow case from the bearer-auth layer: the rate-limit counter's IncrementApiTokenUsage query itself failed (the DB write that tracks per-minute calls). The request was authenticated; only the rate-limit infrastructure broke.

Retry policy: yes, but with a longer backoff (5–10 seconds). A persistent 403 on this path means the rate-limit DB infrastructure is unhappy on Steward's side.

`500 Internal Server Error`

{"status_code": 500, "message": "Server error."}

Steward had an unhandled error processing the request. The message is deliberately uninformative; details land in Steward's server logs.

Retry policy: yes, with exponential backoff. If it persists past your retry budget, file a ticket — include the approximate timestamp + the RPC you called so staff can correlate against logs.

`503 Service Unavailable` (with `Retry-After: 30`)

Steward is in maintenance mode. The Retry-After header is set to a fixed 30 seconds.

Retry policy: yes, honour Retry-After. Maintenance windows are short (typically <5 minutes total).

Envelope error catalogue

These come back as HTTP 200 with an error object in the body. The code field is the load-bearing identifier; the message is human-readable but its exact wording can change.

`UNAUTHENTICATED`

Returned for write RPCs called with token auth (writes are session-only) and for any RPC that requires browser-session state.

{"error": {"code": "UNAUTHENTICATED", "message": "Sign in first."}}

Retry policy: don't. You're calling a session-only endpoint from a token-authenticated context. Drive the operation from the browser surface instead.

`PERMISSION_DENIED`

Three sub-cases:

Scope missing. Your token authenticated but doesn't carry the scope this RPC requires. Re-mint with the right scope (Mint and manage API tokens).

Token-to-token mint refusal. Calling CreateApiToken with a Bearer token returns:

"API tokens can only be created from a signed-in browser
session, not from another API token."

Resource-level deny. You authenticated, your scope is correct, but SpiceDB says you don't have access to this specific resource. Verify your account's permissions on /app/account.

Retry policy: don't. None of these resolve themselves.

`NOT_FOUND`

The resource you asked about doesn't exist OR isn't visible to your token's owner. The two cases return the same envelope — a deliberate ambiguity so probes can't enumerate private resources.

{"error": {"code": "NOT_FOUND", "message": "Evaluation not found."}}

Retry policy: don't, except for a narrow race in the PR-auto-trigger pattern — see "the eval-enqueue race" below.

`INVALID_ARGUMENT`

Your request body has a problem. The message tells you what:

{"error": {"code": "INVALID_ARGUMENT", "message": "Missing name."}}

Common examples:

"Missing name." / "Name exceeds 100 characters." — CreateApiToken validation.
"Pick at least one scope." — empty or all-empty-string scopes list.
"Unknown scope: <s>" — typo in the scopes list.

Retry policy: don't. Fix the request body.

`FAILED_PRECONDITION`

The operation is blocked by state that the request itself can't fix. Examples:

Account deletion attempted while impersonated.
Account deletion attempted when blockers exist (corporate Contributor License Agreement (CLA) signatory, sole member_manager, active subscription).
Contributor report request with include_private_repos=true when the user hasn't granted private-repo scan consent.

{"error": {"code": "FAILED_PRECONDITION", "message": "Account deletion is blocked by rows you own that other parties depend on. See blockers for details."}}

Retry policy: don't. Resolve the precondition first.

`BUDGET_EXCEEDED`

Returned by manual EvaluationsService.Create when the tenant has hit their daily credit cap.

{"error": {"code": "BUDGET_EXCEEDED", "message": "Daily credit cap reached."}}

For the auto-trigger path (per-PR webhook), this surfaces differently — the row lands skipped(budget_exhausted) per Evaluations + dimensions. The token-driven manual path returns the envelope inline.

Retry policy: not within the same UTC day. The cap resets at UTC midnight; retry after that, or upgrade the tier (Tiers + credits).

The eval-enqueue race

A specific race worth handling explicitly: in the PR-auto-triggered polling pattern (Integrate Steward with CI), your CI script lists evaluations and filters by commit_sha.

Between the push and Steward enqueueing the evaluation, the list query returns no match. Treating that as NOT_FOUND and giving up means you'll race the system on every fast CI.

Recommended handling: retry the list query 5–10 times with a 2-second cadence before treating "no match" as a real absence. Typical eval-enqueue latency is under 5 seconds; if it hasn't appeared after 20 seconds, the trigger probably failed and you can surface that to your CI logs.

The retry pattern

A production-grade integrator handles transient errors with exponential backoff and a retry budget. A Python sketch:

import time
import random
from urllib.error import HTTPError

TRANSIENT_STATUSES = {429, 500, 502, 503, 504}
MAX_RETRIES = 5

def call_with_retry(func, *args, **kwargs):
    for attempt in range(MAX_RETRIES):
        try:
            return func(*args, **kwargs)
        except HTTPError as e:
            if e.code not in TRANSIENT_STATUSES:
                raise  # 401/403/etc don't get retried

            # Honour Retry-After when present (429/503)
            retry_after = e.headers.get("Retry-After")
            if retry_after is not None:
                sleep = int(retry_after)
            else:
                # Exponential backoff with full jitter
                # Cap at 60s; we don't want a runaway exponent
                base = min(60, 2 ** attempt)
                sleep = random.uniform(0, base)

            time.sleep(sleep)

    raise RuntimeError(f"Exhausted {MAX_RETRIES} retries")

Key properties:

Honour Retry-After first. The server told you when to retry; don't second-guess it.
Exponential backoff with jitter. When Retry-After isn't set (e.g., 500), back off with full jitter to avoid synchronised retry storms from concurrent integrations.
Cap the maximum backoff. A capped exponent (60 seconds here) prevents runaway sleeps on a persistently broken service.
Retry budget. Five retries is a reasonable starting point. Past that, surface the failure to your operator.

For envelope errors, none are transient — they all return immediately without retry. The error.code switch decides what to surface.

A complete handler

Stitched together, here's a single function that handles both shapes:

import json
from urllib.error import HTTPError

class StewardAPIError(Exception):
    """Raised for non-retriable API errors. Includes code + message."""
    def __init__(self, code: str, message: str):
        super().__init__(f"{code}: {message}")
        self.code = code
        self.message = message

def steward_call(token: str, rpc: str, body: dict) -> dict:
    """Call a Steward API RPC with full error handling."""
    url = f"https://steward-dev.ai/api/{rpc}"

    def attempt():
        req = urllib.request.Request(
            url,
            data=json.dumps(body).encode("utf-8"),
            headers={
                "Authorization": f"Bearer {token}",
                "Content-Type": "application/json",
            },
            method="POST",
        )
        with urllib.request.urlopen(req, timeout=10) as resp:
            return json.loads(resp.read())

    response = call_with_retry(attempt)

    # HTTP layer succeeded; check envelope.
    if "error" in response and response["error"]:
        err = response["error"]
        raise StewardAPIError(err.get("code", "UNKNOWN"),
                              err.get("message", ""))

    return response

Usage:

try:
    job = steward_call(token, "steward.api.v1.EvaluationsService/Get",
                       {"id": job_id})
    print(job["job"]["status"])
except StewardAPIError as e:
    if e.code == "BUDGET_EXCEEDED":
        notify_finance(e.message)
    elif e.code == "NOT_FOUND":
        print("Evaluation expired or not visible")
    else:
        raise  # Unknown application-level error
except HTTPError as e:
    print(f"HTTP {e.code}: gave up after retry budget")

Decision matrix at a glance

Error	Layer	Retry?	What to do
`401 Unauthorized`	HTTP	No	Verify / mint a fresh token.
`403 Forbidden`	HTTP	Yes (longer backoff)	Persistent: file a ticket.
`429 Too Many Requests`	HTTP	Yes (honour `Retry-After`)	—
`500 Internal Server Error`	HTTP	Yes (exponential)	Persistent: file a ticket.
`503 Service Unavailable`	HTTP	Yes (honour `Retry-After`)	—
`UNAUTHENTICATED`	Envelope	No	Use a browser session.
`PERMISSION_DENIED`	Envelope	No	Re-mint with the right scope.
`NOT_FOUND` (poll-by-commit)	Envelope	Yes (limited; eval-enqueue race)	After 20s, treat as real absence.
`NOT_FOUND` (other)	Envelope	No	The resource doesn't exist or isn't yours.
`INVALID_ARGUMENT`	Envelope	No	Fix request body.
`FAILED_PRECONDITION`	Envelope	No	Resolve precondition.
`BUDGET_EXCEEDED`	Envelope	No (today) / Yes (after UTC midnight)	Wait or upgrade.

What's next

Using the Steward API — the broader API concept this guide hardens.
Integrate Steward with CI — the CI integration pattern this guide makes production-ready.
/docs/api — the per-RPC reference for the endpoints that return these errors.

Handle errors + rate limits

Two error shapes

Shape 1: HTTP status error

Shape 2: Envelope error

HTTP status catalogue

401 Unauthorized

429 Too Many Requests

403 Forbidden

500 Internal Server Error

503 Service Unavailable (with Retry-After: 30)