Skip to main content

Overview

Agent Sentinel provides transparent instrumentation for major LLM providers. Simply call instrument_<provider>() once, and all LLM API calls are automatically tracked with:
  • Token usage (input and output tokens)
  • Costs (calculated from latest pricing data)
  • Duration and latency
  • Model information
  • Request/response metadata
No code changes required - just instrument and go.

Supported providers

  • OpenAI (GPT-3.5, GPT-4, GPT-4o, o1)
  • Anthropic (Claude 3 Opus, Sonnet, Haiku, Claude 3.5)
  • xAI/Grok (Grok models)
  • Google Gemini (Gemini models)

OpenAI instrumentation

Async OpenAI

Anthropic instrumentation

Async Anthropic

Grok instrumentation

Gemini instrumentation

Gemini-powered features

Beyond raw call instrumentation, Agent Sentinel uses Gemini (gemini-2.5-flash by default) to power three platform-side features. All three read GEMINI_API_KEY (or GOOGLE_API_KEY) from the platform environment and degrade gracefully when the key or google-genai package is missing.

Semantic moderation

GeminiModerator (in agent_sentinel.guardrails.moderation) classifies action arguments against intent-level categories that keyword matching can’t catch: prompt_injection, data_exfiltration, policy_evasion, plus the explicit violence / self_harm / hate_severe / sexual_minors set.
See Guardrails → Content moderation for full configuration.

Intervention enrichment

After every intervention is written to the platform DB, a FastAPI background task calls Gemini to replace terse rule-text fields with human-readable explanations: Enrichment runs out-of-band — the SDK’s POST /api/v1/ingest/ returns immediately and the enriched fields stream into the console asynchronously. See the Console → Interventions guardrail panels for how the enriched fields are surfaced.

Self-repair feedback

When a policy block returns to the LLM through a SentinelToolNode, the tool message includes the enriched suggested_rewrite along with the original retry_guidance. Most frontier models will read the rewrite and self-correct without any extra prompt-engineering.

Adversarial scenario generation

The platform exposes POST /api/v1/evals/generate-scenarios — a Gemini-driven generator that produces adversarial test scenarios across five categories (prompt_injection, policy_evasion, social_engineering, cost_abuse, data_exfiltration). Used by the console’s scenario-generator dialog to expand benchmark coverage. See Console → Evals.

Prose policy compilation

Compile English-language policy descriptions directly to the policy IR via POST /api/v1/policies/compile. See SDK → Prose policies.

Getting token costs

Retrieve cumulative costs across all instrumented LLM calls:

Pricing data

Agent Sentinel includes up-to-date pricing (as of December 2025):

OpenAI pricing

Anthropic pricing

Grok pricing (xAI)

Gemini pricing

Advanced: Custom pricing

Override default pricing for custom models:

Combining with @guarded_action

LLM instrumentation works alongside manual instrumentation:

Policy enforcement on LLM calls

LLM calls respect policy budgets:

Ledger output example

With LLM instrumentation enabled, your ledger includes detailed LLM call records:

Best practices

Instrument at startup: Call instrument_<provider>() once at application startup, before creating any LLM clients.
Use with budgets: Combine LLM instrumentation with session/run budgets to prevent runaway costs.
Pricing data may lag: Pricing is updated regularly but may not reflect same-day price changes. Override with custom pricing if needed.
Track costs by provider: Use get_token_costs() to see which providers are driving your costs.

Troubleshooting

”LLM calls not tracked”

Make sure to call instrument_<provider>() before creating the client:

“Incorrect costs”

Verify the model name matches our pricing data. Use normalize_model_name() to check:

See also