Anomaly-Triggered Investigation

Configure pattern anomaly monitors to detect unusual log patterns and trigger autonomous AI investigations.

When a pattern anomaly monitor detects unusual log patterns, AI teammates autonomously investigate the root cause and propose remediation steps.

Environment Setup

ComponentPurpose
Edge Delta MCP ConnectorQuery logs, metrics, and traces from Edge Delta backend
GitHub ConnectorCorrelate anomalies with recent code deployments
Log to Pattern ProcessorAggregate logs into patterns in the Telemetry Pipeline
Pattern Anomaly MonitorDetect unusual log patterns and trigger investigations
Monitor NotificationsRoute alerts to an AI Team channel
AI Team ChannelReceive monitor notifications and route to OnCall AI

This workflow uses two pipelines. A Telemetry Pipeline with the Log to Pattern processor sends pattern data to the Edge Delta backend. A Pattern Anomaly Monitor watches for negative pattern spikes or new error patterns, and routes notifications to an AI Team channel. The Edge Delta MCP and GitHub connectors provision a separate AI Team ingestion pipeline, enabling teammates to query telemetry and correlate with code changes.

Data Flow

flowchart LR
    A[Application Logs] --> B[Telemetry Pipeline<br/>with Log to Pattern]
    B --> C[Edge Delta Backend]
    C --> D[Pattern Anomaly Monitor]
    D -->|Anomaly Detected| E[OnCall AI]
    E --> F[SRE Teammate]
    E --> G[Code Analyzer]
    F -->|Queries| H[Edge Delta MCP]
    H --> C
    G -->|Queries| I[GitHub]

The Telemetry Pipeline uses the Log to Pattern processor to aggregate logs into patterns using the Drain algorithm, replacing variable elements (IDs, timestamps, IP addresses) with wildcards. Pattern data flows to the Edge Delta backend, where the Pattern Anomaly Monitor compares current pattern distributions against historical baselines. When negative patterns spike or new patterns emerge, the monitor sends a notification to an AI Team channel configured in its Notifications section. OnCall AI receives the event in that channel and delegates analysis to specialized teammates. SRE queries the backend through the Edge Delta MCP connector to investigate.

Investigation Workflow

  1. OnCall AI receives the anomaly alert and initiates an investigation thread
  2. SRE pulls relevant logs, metrics, and traces from the affected service during the anomaly window
  3. SRE identifies the root cause (for example, detecting a surge in null pointer exceptions or connection timeout errors)
  4. Code Analyzer reviews recent commits to correlate the anomaly with deployment changes
  5. OnCall AI proposes a mitigation plan with specific remediation steps, including links to the relevant log patterns

Beyond the immediate investigation, teammates perform extended analysis: reviewing historical pattern trends, examining related services for correlated errors, and validating whether the failure matches known scenarios.

Learn More