Anomaly-Triggered Investigation
2 minute read
When a pattern anomaly monitor detects unusual log patterns, AI teammates autonomously investigate the root cause and propose remediation steps.
Environment Setup
| Component | Purpose |
|---|---|
| Edge Delta MCP Connector | Query logs, metrics, and traces from Edge Delta backend |
| GitHub Connector | Correlate anomalies with recent code deployments |
| Log to Pattern Processor | Aggregate logs into patterns in the Telemetry Pipeline |
| Pattern Anomaly Monitor | Detect unusual log patterns and trigger investigations |
| Monitor Notifications | Route alerts to an AI Team channel |
| AI Team Channel | Receive monitor notifications and route to OnCall AI |
This workflow uses two pipelines. A Telemetry Pipeline with the Log to Pattern processor sends pattern data to the Edge Delta backend. A Pattern Anomaly Monitor watches for negative pattern spikes or new error patterns, and routes notifications to an AI Team channel. The Edge Delta MCP and GitHub connectors provision a separate AI Team ingestion pipeline, enabling teammates to query telemetry and correlate with code changes.
Data Flow
flowchart LR
A[Application Logs] --> B[Telemetry Pipeline<br/>with Log to Pattern]
B --> C[Edge Delta Backend]
C --> D[Pattern Anomaly Monitor]
D -->|Anomaly Detected| E[OnCall AI]
E --> F[SRE Teammate]
E --> G[Code Analyzer]
F -->|Queries| H[Edge Delta MCP]
H --> C
G -->|Queries| I[GitHub]The Telemetry Pipeline uses the Log to Pattern processor to aggregate logs into patterns using the Drain algorithm, replacing variable elements (IDs, timestamps, IP addresses) with wildcards. Pattern data flows to the Edge Delta backend, where the Pattern Anomaly Monitor compares current pattern distributions against historical baselines. When negative patterns spike or new patterns emerge, the monitor sends a notification to an AI Team channel configured in its Notifications section. OnCall AI receives the event in that channel and delegates analysis to specialized teammates. SRE queries the backend through the Edge Delta MCP connector to investigate.
Investigation Workflow
- OnCall AI receives the anomaly alert and initiates an investigation thread
- SRE pulls relevant logs, metrics, and traces from the affected service during the anomaly window
- SRE identifies the root cause (for example, detecting a surge in null pointer exceptions or connection timeout errors)
- Code Analyzer reviews recent commits to correlate the anomaly with deployment changes
- OnCall AI proposes a mitigation plan with specific remediation steps, including links to the relevant log patterns
Beyond the immediate investigation, teammates perform extended analysis: reviewing historical pattern trends, examining related services for correlated errors, and validating whether the failure matches known scenarios.