Overview
Agent Sentinel provides transparent instrumentation for major LLM providers. Simply callinstrument_<provider>() once, and all LLM API calls are automatically tracked with:
- Token usage (input and output tokens)
- Costs (calculated from latest pricing data)
- Duration and latency
- Model information
- Request/response metadata
Supported providers
- OpenAI (GPT-3.5, GPT-4, GPT-4o, o1)
- Anthropic (Claude 3 Opus, Sonnet, Haiku, Claude 3.5)
- xAI/Grok (Grok models)
- Google Gemini (Gemini models)
OpenAI instrumentation
Async OpenAI
Anthropic instrumentation
Async Anthropic
Grok instrumentation
Gemini instrumentation
Gemini-powered features
Beyond raw call instrumentation, Agent Sentinel uses Gemini (gemini-2.5-flash by default) to power three platform-side features. All three read GEMINI_API_KEY (or GOOGLE_API_KEY) from the platform environment and degrade gracefully when the key or google-genai package is missing.
Semantic moderation
GeminiModerator (in agent_sentinel.guardrails.moderation) classifies action arguments against intent-level categories that keyword matching can’t catch: prompt_injection, data_exfiltration, policy_evasion, plus the explicit violence / self_harm / hate_severe / sexual_minors set.
Intervention enrichment
After every intervention is written to the platform DB, a FastAPI background task calls Gemini to replace terse rule-text fields with human-readable explanations:
Enrichment runs out-of-band — the SDK’s
POST /api/v1/ingest/ returns immediately and the enriched fields stream into the console asynchronously. See the Console → Interventions guardrail panels for how the enriched fields are surfaced.
Self-repair feedback
When a policy block returns to the LLM through aSentinelToolNode, the tool message includes the enriched suggested_rewrite along with the original retry_guidance. Most frontier models will read the rewrite and self-correct without any extra prompt-engineering.
Adversarial scenario generation
The platform exposesPOST /api/v1/evals/generate-scenarios — a Gemini-driven generator that produces adversarial test scenarios across five categories (prompt_injection, policy_evasion, social_engineering, cost_abuse, data_exfiltration). Used by the console’s scenario-generator dialog to expand benchmark coverage. See Console → Evals.
Prose policy compilation
Compile English-language policy descriptions directly to the policy IR viaPOST /api/v1/policies/compile. See SDK → Prose policies.
Getting token costs
Retrieve cumulative costs across all instrumented LLM calls:Pricing data
Agent Sentinel includes up-to-date pricing (as of December 2025):OpenAI pricing
Anthropic pricing
Grok pricing (xAI)
Gemini pricing
Advanced: Custom pricing
Override default pricing for custom models:Combining with @guarded_action
LLM instrumentation works alongside manual instrumentation:Policy enforcement on LLM calls
LLM calls respect policy budgets:Ledger output example
With LLM instrumentation enabled, your ledger includes detailed LLM call records:Best practices
Troubleshooting
”LLM calls not tracked”
Make sure to callinstrument_<provider>() before creating the client:
“Incorrect costs”
Verify the model name matches our pricing data. Usenormalize_model_name() to check:
See also
- Framework Integrations - LangChain and CrewAI integrations
- Policies - Enforce budgets on LLM calls
- Instrumentation - Manual action wrapping
