Skip to main content

Overview

The Agent Sentinel SDK provides robust retry and resilience patterns to handle transient failures gracefully. These utilities are especially useful for:
  • Network operations that may fail temporarily
  • Remote sync to the platform
  • LLM API calls with rate limits
  • External service integrations

Retry with Exponential Backoff

The @with_retry decorator automatically retries failed operations with exponential backoff and jitter.

Basic usage

Custom configuration

Retry specific exceptions

Retry callbacks

Circuit Breaker Pattern

The circuit breaker prevents cascading failures by “opening” after repeated failures, rejecting requests until the service recovers.

States

  • CLOSED - Normal operation, requests pass through
  • OPEN - Too many failures, requests are rejected immediately
  • HALF_OPEN - Testing if service recovered, allows one request

Basic usage

Combining retry and circuit breaker

Monitoring circuit breaker state

Backoff Strategy

The retry module uses exponential backoff with jitter:
Example progression (initial_delay=1.0, base=2.0):
  • Attempt 1: ~1 second
  • Attempt 2: ~2 seconds
  • Attempt 3: ~4 seconds
  • Attempt 4: ~8 seconds
  • Attempt 5: ~16 seconds

Error Types

The SDK defines specific error types for retry logic:

Remote Sync Resilience

The background sync to the platform uses retry and circuit breaker internally:
If remote sync fails after retries:
  • Local ledger continues working
  • Logs warning but doesn’t crash your agent
  • Circuit breaker prevents repeated failed attempts
  • Automatically recovers when platform is available

Best Practices

Use retry for transient failures: Network glitches, rate limits, and temporary service outages benefit from retry logic.
Use circuit breaker for persistent failures: If a service is down, the circuit breaker prevents wasting resources on repeated failed attempts.
Don’t retry non-idempotent operations blindly: If your operation has side effects (e.g., charging a credit card), ensure it’s safe to retry or use idempotency tokens.
Combine with timeouts: Set reasonable timeouts on operations to prevent hanging indefinitely during retries.
Monitor circuit breaker state: Track when circuits open in production - it indicates underlying service issues that need attention.

Common Patterns

API call with full resilience

Database operation with retry

LLM call with exponential backoff

Troubleshooting

Retry not working

Make sure the exception is in retryable_exceptions:

Circuit breaker stuck open

If the circuit is stuck open:
  1. Check if the underlying service is actually down
  2. Verify recovery_timeout is reasonable
  3. Ensure the service is healthy before expecting recovery
  4. Consider manual reset in emergency:

Too many retries

If operations retry too many times:
  • Reduce max_attempts
  • Increase initial_delay or max_delay
  • Make exceptions more specific (don’t retry everything)
  • Add circuit breaker to fail fast on persistent issues

See Also