Kafka Security Event Processing

When security events stream through Kafka topics and Edge Delta agents process them with severity classification, volume reduction, and content-based routing, AI Team teammates can investigate the processed telemetry to correlate threats across authentication, network, and endpoint domains.

Environment setup

Configure three Edge Delta pipelines with Kafka sources, each consuming from a separate security topic (auth-events, network-events, endpoint-events). See Kafka Security Monitoring for the pipeline architecture and processing patterns.

Data flow

flowchart LR
    subgraph "Kafka Topics"
        K1[auth-events]
        K2[network-events]
        K3[endpoint-events]
    end

    subgraph "Edge Delta Pipelines"
        P1[Auth Pipeline]
        P2[Network Pipeline]
        P3[Endpoint Pipeline]
    end

    subgraph "AI Team"
        M[Monitor Alert] --> C[AI Team Channel]
        C --> O[OnCall AI]
        O --> SE[Security Engineer]
        O --> SR[SRE Teammate]
        SE -->|Queries| ED[Edge Delta MCP]
        SR -->|Queries| ED
    end

    K1 --> P1
    K2 --> P2
    K3 --> P3

    P1 -->|Critical events| M
    P2 -->|Critical events| M
    P3 -->|Critical events| M

Edge Delta monitors detect anomalies in the processed security event streams, such as spikes in critical-severity events or new brute-force patterns. These alerts route to an AI Team channel, where OnCall AI triages and engages the appropriate specialists.

Investigation workflow

1. Alert triage

When a monitor fires on a spike in critical authentication events, OnCall AI queries the Edge Delta backend to characterize the alert:

• How many brute-force events in the last 15 minutes?
• Which source IPs and user accounts are involved?
• Is this a single source or distributed attack?

2. Cross-domain correlation

Security Engineer correlates the authentication events with network and endpoint telemetry:

• Network: Are the same source IPs involved in port scans or DNS anomalies?
• Endpoint: Did any targeted accounts show lateral movement or suspicious process execution after a successful login?
• Timeline: Do the events cluster in a window that suggests a coordinated attack?

3. Impact assessment

Based on the correlation, Security Engineer determines whether:

• A brute-force attempt succeeded (auth failure followed by auth success from the same IP)
• An attacker pivoted post-compromise (privilege escalation followed by lateral movement)
• The events are benign (automated scanner, penetration test, or misconfigured service)

4. Response coordination

For confirmed threats, the teammate can:

• Notify the SOC channel in Slack with a structured summary
• Create a PagerDuty incident for on-call response
• Document findings for the security incident timeline

Learn more

• Kafka Security Monitoring — Pipeline architecture and processing patterns
• Security Controls — Data boundaries and approval workflows for AI Team
• Security Degradation — Investigating service degradation with security implications
• Kafka Source — Configuring the Kafka input node