> ## Documentation Index
> Fetch the complete documentation index at: https://docs.agentsentinel.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Evals

> Benchmark scorecards, regression tracking, CI gating, and adversarial scenario generation.

The **Evals** route lives at `/evals`. It is the home for benchmark results submitted by `agent-sentinel-gym` (or any other harness that calls `POST /api/v1/evals/`), and is where you'll watch CI gates pass or fail across deployments.

## Layout

The page has four blocks, top to bottom:

1. **Published Benchmarks** card — links to `/evals/benchmarks` (markdown render of the published benchmark report).
2. **Latest Scorecard** — five tiles: Grade, Block Rate, Correctness Rate, Damage Prevented (USD), Avg Detection time (ms). Includes a CI Pass/Fail badge if the most recent run was tied to a commit.
3. **Eval Run History** table — every submitted run with grade, block rate, CI status, branch, commit SHA, and date. **Compare Latest vs Previous** button at the top right runs a delta diff.
4. **Run detail modal** — opens from the Details button: per-scenario table with category, severity, blocked/failed badge, reason, and detection time.

## Generating scenarios

The page header has a **Generate Scenarios** button that opens the scenario-generator dialog:

1. Select an agent from the dropdown (uses the agent's registered action definitions).
2. Set **Max per category** (default `10`, max `50`).
3. Click **Generate**.

The platform runs two passes:

* **Template scenarios** — `agent-sentinel-gym` emits structural attacks: missing prerequisites, stale evidence, denied actions, budget blowouts.
* **Gemini scenarios** — semantic adversaries across `prompt_injection`, `policy_evasion`, `social_engineering`, `cost_abuse`, `data_exfiltration` (skipped if `GEMINI_API_KEY` is not set on the platform).

Results show as scenario cards with category badges and a per-category violation breakdown. Cards include the generated source so you can paste them straight into your test harness.

## Comparing runs

Click **Compare Latest vs Previous** (or use the API directly) to see:

* **Block rate delta** (percentage points)
* **Correctness rate delta**
* **Grade change** (e.g., `B → A`)
* **Regressions** — scenario names that passed on baseline but failed on current
* **Improvements** — scenario names that failed on baseline but pass on current

CI integrations should fail the build whenever **Regressions > 0**.

## Published benchmarks (`/evals/benchmarks`)

The benchmarks sub-route renders the markdown report directly from the repo (`benchmarks/BENCHMARK_REPORT.md`). The phase-7 reference run shows:

| Configuration  |  Violations | Blocked |
| -------------- | ----------: | ------: |
| Guardrails OFF | 4 / 5 (80%) |   0 / 5 |
| Guardrails ON  |       0 / 5 |   5 / 5 |

Use this as a public benchmark to compare your own scorecards against.

## CI integration

Submit eval runs from CI:

```bash theme={null}
python -m badcomputeruse gate \
  --policy policies/production_defense.yaml \
  --upload \
  --platform-url $AGENTSENTINEL_PLATFORM_URL \
  --api-key $AGENTSENTINEL_API_KEY
```

The `--upload` flag posts the result to `POST /api/v1/evals/`. The `--gate` exit code is non-zero when block rate falls below the configured threshold, blocking the merge.

## Underlying API

* `GET /api/v1/evals/latest` — scorecard tiles
* `GET /api/v1/evals/?skip=&limit=` — history table
* `GET /api/v1/evals/compare/{baseline_id}/{current_id}` — comparison block
* `POST /api/v1/evals/generate-scenarios` — scenario generator dialog
* `POST /api/v1/evals/` — CI submissions

Full reference: [Platform → Evals API](/platform/evals).

## See also

* [Platform → Evals API](/platform/evals)
* [SDK → LLM integrations](/sdk/python/llm-integrations#adversarial-scenario-generation) — Gemini-powered scenario gen
* [SDK → Guardrails](/sdk/python/guardrails) — what guardrails the benchmark exercises