Determinism Analysis

Overview

Accessing the analyzer

1. Navigate to the Runs page
2. Click on any run to open its details modal
3. Switch to the Determinism tab (microscope icon) alongside Telemetry and Simulation
4. The analysis runs automatically against the run’s recorded actions

Determinism score

• 🟢 90-100 - High Reliability: Minimal issues, safe to replay
• 🟡 70-89 - Moderate Risk: Some non-deterministic patterns detected
• 🔴 0-69 - High Risk: Significant non-determinism, replay likely to fail

• Large numeric display
• Color-coded badge
• Progress bar visualization
• Severity label

Analysis summary

• Total actions analyzed
• Issues found count
• Summary description of findings
• Days elapsed since run

Patterns detected

Timestamp dependencies

• datetime.now()
• time.time()
• Date/time-based logic

Random generation

• random.random()
• random.randint()
• random.choice()

UUID generation

• uuid.uuid4()
• Dynamic ID creation
• Non-deterministic keys

Process/environment

• os.getpid()
• socket.gethostname()
• Platform details

API variability

• Request IDs
• Trace IDs
• Session tokens

Order-dependent

• Dict iteration (old Python)
• Set operations
• Concurrent operations

Detailed issues view

Action context

• Action name
• Action index (position in run)
• Action ID (for reference)

Issue details

• Type - Category of issue
• Severity - High/Medium/Low
• Description - What the problem is
• Recommendation - How to fix it

Action: generate_report (#3)
├─ Type: timestamp_dependency
├─ Severity: HIGH
├─ Description: Action uses timestamp or current time in inputs
└─ Recommendation: Use fixed timestamps for testing or mock time functions

Smart recommendations

Priority levels

• 🔴 High - Fix immediately (blocks reliable replay)
• 🟡 Medium - Fix soon (likely causes issues)
• 🟢 Low - Fix eventually (may cause issues)

Categories

• Determinism - Non-deterministic patterns
• Performance - Optimization opportunities
• Reliability - Error handling improvements

Detailed guidance

• Title - Brief summary
• Description - Full explanation
• Solution - Step-by-step fix
• Code example - Concrete implementation

🔴 HIGH PRIORITY | Determinism

Remove timestamp dependencies

Description:
2 actions use current timestamps which will vary on each execution

Solution:
Use fixed timestamps for testing or inject time via parameters

Code Example:
# Instead of:
report_time = datetime.now()

# Use:
report_time = datetime(2025, 1, 3, 12, 0, 0)

Warnings section

• Failed actions
• Error outcomes
• Incomplete executions
• Data quality issues

Visual features

Progress bar

• Green for high scores
• Yellow for moderate
• Red for low scores

Badge system

• Severity badges (info/warning/critical)
• Status badges (passed/failed)
• Category tags

Tabbed interface

• Recommendations tab - Prioritized fixes
• Issues tab - Detailed problems by action

Collapsible sections

• Individual issues
• Code examples
• Action groups

Common patterns and fixes

Pattern: Using current time

@guarded_action(name="log_event")
def log_event(message: str):
    timestamp = datetime.now()
    return {"message": message, "time": timestamp}

@guarded_action(name="log_event")
def log_event(message: str, event_time: datetime):
    return {"message": message, "time": event_time}

Pattern: Random sampling

@guarded_action(name="select_sample")
def select_sample(items: list):
    return random.sample(items, 5)

@guarded_action(name="select_sample")
def select_sample(items: list, seed: int = 42):
    random.seed(seed)
    return random.sample(items, 5)

Pattern: UUID generation

@guarded_action(name="create_task")
def create_task(name: str):
    task_id = str(uuid.uuid4())
    return {"id": task_id, "name": name}

@guarded_action(name="create_task")
def create_task(name: str, task_id: str):
    return {"id": task_id, "name": name}

Integration with replay

1. Run agent normally → generates ledger
2. Analyze determinism in console → identifies issues
3. Fix issues following recommendations
4. Replay with SDK → validates fixes
5. Compare original vs replay → confirms determinism

Exporting results

• Code review - Share with team
• Documentation - Track improvements
• CI/CD integration - Automated checks
• Reporting - Management visibility

• JSON (for automation)
• PDF (for reports)
• CSV (for tracking)

Best practices

Keyboard shortcuts

• a - Toggle analysis view
• r - Show recommendations
• i - Show issues
• e - Export results

Common questions

”Score seems low but my agent works fine”

”Can I ignore low-severity issues?”

• For testing/debugging: Fix them
• For production monitoring: Can usually ignore
• For compliance: May need to fix (check requirements)

“Analysis shows false positives”

”How often should I run analysis?”

• Development: Before each commit
• Staging: Before each deployment
• Production: Weekly or after incidents

Troubleshooting

Analysis fails to load

• Verify run exists and you have access
• Check run has completed (not in progress)
• Try refreshing the page
• Check browser console for errors

Issues seem incorrect

No recommendations shown

Analysis takes too long

See also

• Replay (SDK) - Actual replay execution
• Replay (Platform) - API documentation
• Runs - View run history
• Activity Ledger - Action details