Skip to main content

Observability & Monitoring

Table of Contents​

Automatic Observation​

Every workflow execution is automatically tracked and saved to the /observe/ directory with comprehensive metrics:

observe/
├── Example_Workflow_20240715_133132.json
├── Linkedin_Prospec_20240715_140521.json
└── Ai_Chat_Assistant_20240715_141203.json

Observation Data​

Each observation file contains:

{
  "workflow_info": {
    "workflow_id": "user-query-processor",
    "workflow_name": "User Query Processor",
    "entrypoint_id": "default",
    "mode": "production",
    "start_time": "2024-07-15T13:31:32.245Z",
    "end_time": "2024-07-15T13:31:35.891Z",
    "duration_ms": 3646,
    "status": "completed"
  },
  "execution_summary": {
    "total_steps_executed": 5,
    "successful_steps": 4,
    "failed_steps": 1,
    "skipped_steps": 0,
    "total_retries": 2,
    "slowest_step": "generate-response (2843.75ms)",
    "fastest_step": "validate-input (12.50ms)",
    "bottlenecks": ["generate-response"],
    "loops_detected": [],
    "error_steps": ["step-validation"],
    "error_patterns": {
      "Invalid input format": 1
    },
    "token_summary": {
      "total_tokens": 1524,
      "step_breakdown": [...],
      "model_breakdown": {...},
      "prediction_steps": 2,
      "execution_steps": 3
    }
  },
  "addon_summary": [
    {
      "addon_id": "openai",
      "addon_type": "openai",
      "class_type": "AddonType.AGENT",
      "package_name": "ai_rooms_openai",
      "enabled": true,
      "load_status": "success",
      "test_status": true,
      "registered_actions": ["openai::completion", "openai::chat"],
      "package_version": "1.0.0",
      "configuration": {
        "model": "gpt-4",
        "max_tokens": 1000
      }
    }
  ],
  "step_metrics": [...],
  "execution_paths": [...],
  "step_state_history": {...},
  "state_transition_timeline": [...],
  "performance_insights": {...},
  "execution_flow": {...}
}

Workflow Information Fields​

The workflow_info section contains core workflow execution metadata:

  • entrypoint_id: ID of the entrypoint used to start the workflow (e.g., "default", "api-handler")
  • mode: Execution mode of the workflow:
    • "production": Normal execution with real addons
    • "mock": Execution using mock handlers for testing
    • "dry_run": Simulation mode that analyzes workflow without executing actions

Execution Summary Fields​

The execution_summary includes additional tracking fields:

  • skipped_steps: Number of steps that were skipped due to conditional logic
  • error_steps: Array of step IDs that encountered errors during execution
  • error_patterns: Object mapping error messages to their occurrence count, useful for identifying common issues
  • token_summary: Comprehensive token usage tracking across all workflow steps including total tokens, per-step breakdown, model-specific usage, and prediction vs execution step counts

Execution Flow Data​

The execution_flow section provides workflow visualization data:

{
  "execution_flow": {
    "dag_structure": {
      "step-1": [
        {
          "to_step": "step-2",
          "condition_result": true,
          "timestamp": "2024-07-15T13:31:33.245Z"
        }
      ]
    },
    "execution_timeline": [
      {
        "step_id": "step-1",
        "step_name": "Validate Input",
        "start_time": "2024-07-15T13:31:32.245Z",
        "end_time": "2024-07-15T13:31:32.257Z",
        "duration_ms": 12.5,
        "status": "success",
        "state_transitions": [...]
      }
    ],
    "step_dependencies": {},
    "conditional_branches": [
      {
        "from_step": "step-1",
        "to_step": "step-2",
        "condition_result": true,
        "timestamp": "2024-07-15T13:31:33.245Z"
      }
    ]
  }
}

Addon Summary​

The addon_summary section provides detailed information about all addons used in the workflow execution:

Addon Status Information​

Each addon entry contains:

  • addon_id: Unique identifier for the addon
  • addon_type: Type of addon (e.g., "openai", "postgres", "rest_api")
  • class_type: Addon classification type (e.g., "AddonType.AGENT", "AddonType.STORAGE")
  • package_name: Python package name for the addon
  • enabled: Whether the addon was enabled in configuration
  • load_status: Result of addon loading ("success", "failed", "not_found")
  • test_status: Boolean indicating if addon connectivity test passed
  • registered_actions: List of actions registered by this addon
  • package_version: Version of the addon package (if available)
  • configuration: Non-sensitive configuration values used
  • error_message: Error details if loading or testing failed

Addon Health Monitoring​

Use the addon summary to:

  • Verify Addon Availability: Check if all required addons loaded successfully
  • Debug Integration Issues: Identify failed addon tests or missing packages
  • Track Action Registry: See which actions are available from each addon
  • Monitor Package Versions: Track addon versions across workflow executions
  • Audit Configuration: Review non-sensitive config values used

Example Addon Summary​

{
  "addon_summary": [
    {
      "addon_id": "openai",
      "addon_type": "openai",
      "class_type": "AddonType.AGENT",
      "package_name": "ai_rooms_openai",
      "enabled": true,
      "load_status": "success",
      "test_status": true,
      "registered_actions": ["openai::completion", "openai::chat", "openai::embeddings"],
      "package_version": "1.2.0"
    },
    {
      "addon_id": "database",
      "addon_type": "postgres",
      "class_type": "AddonType.STORAGE",
      "package_name": "ai_rooms_postgres", 
      "enabled": true,
      "load_status": "failed",
      "test_status": false,
      "registered_actions": [],
      "error_message": "Connection timeout: Could not connect to database server"
    }
  ]
}

Token Summary​

The token_summary section provides comprehensive token usage tracking:

  • total_tokens: Total token usage across all workflow steps
  • step_breakdown: Array of per-step token usage details including step info, token counts, model used, and prediction status
  • model_breakdown: Token usage grouped by AI model with totals and step counts
  • prediction_steps: Count of steps that performed token predictions
  • execution_steps: Count of steps that executed with token usage

Step State History & Transitions​

The system tracks detailed step state transitions through:

  • step_state_history: Object mapping step IDs to arrays of state transition records
  • state_transition_timeline: Chronologically sorted array of all state transitions across steps

Each transition record includes timestamp, from/to states, and reason for the transition.

Enhanced Step Metrics​

Step metrics now include additional tracking fields:

  • input_parameters: Raw parameters as defined in the workflow (before template resolution)
  • resolved_input_parameters: Parameters after template variable resolution (e.g., {{step.output.value}} replaced with actual values)
  • output_data: Result data returned by the step action
  • tool_usage: Array of detailed tool usage records for each tool used within the step
  • token_usage: Token consumption details for the step including input/output tokens, model, and prediction status
  • loop_iteration: Iteration index if the step is part of a loop (null otherwise)

Template Resolution Debugging​

The input_parameters and resolved_input_parameters fields help debug template resolution issues:

{
  "step_id": "process-user-data",
  "input_parameters": {
    "userId": "{{user-lookup.output.id}}",
    "name": "{{user-lookup.output.name}}"
  },
  "resolved_input_parameters": {
    "userId": "12345",
    "name": "John Doe"
  }
}

This makes it easy to see:

  • What templates were used in the workflow definition
  • What actual values were passed to the action
  • If template resolution is working correctly

Tool Usage Tracking​

Each step tracks individual tool usage with execution details, timing, success status, and retry information.

Performance Insights​

The system automatically generates:

  • Bottleneck Detection: Steps taking >5x average duration
  • Success Rates: Step and workflow success percentages
  • Error Patterns: Common errors and their frequencies
  • Execution Efficiency: Retry rates and timing distribution
  • Flow Analysis: Execution paths and conditional branches