Edge Delta Large Message Pack

Edge Delta Pipeline Pack for Large Messages

Overview

The Edge Delta Large Message pack efficiently processes large messages, converting them to metrics for real-time monitoring and determining whether thresholds are met for alerting or further action. By handling large logs separately, it enhances scalability and monitoring capabilities in observability solutions.

Pack Description

1. Data Ingestion

The data flow begins with the Pack Source as the entry point for logs into the pack for processing.

2. Large Message Detection

Logs are passed to the Large Message Detector node, which is a Route node. This node uses OTTL expressions to identify logs that match a certain condition, specifically checking if there is a match in the attributes["ed.split.uid"].

- name: Large Message Detector
  type: route
  expression_type: ottl
  paths:
    - path: Large Message
      condition: IsMatch(attributes["ed.split.uid"],".*")
      exit_if_matched: true

The node analyzes whether the attribute in the logs matches the regex pattern. When a match is found, logs are routed to the Large Message path. The ones that do not match any condition are routed to the Normal Destination pack output.

3.1 Large Message Routing

On the Large Message path, logs are routed to the Large Destination pack output for further processing.

3.2 Convert to Metrics

On the Large Message path, logs are also sent to l2m_large_message, a Log to Metric node. This node transforms logs into metrics based on a defined pattern and reporting interval.

- name: l2m_large_message
  type: log_to_metric
  pattern: .*
  interval: 1m0s
  skip_empty_intervals: false
  only_report_nonzeros: false
  group_by:
    - item["attributes"]["ed.split.uid"]

This transformation produces metrics from large messages, allowing quantitative analysis and monitoring of such logs. Metrics are particularly useful for detecting trends or anomalies over time.

4. Metric Output

The metrics generated by l2m_large_message are routed to the Metric Destination pack output, making them available for further analysis or storage.

5. Threshold Evaluation

Metrics flow from l2m_large_message to threshold_l2m_large_message, a Threshold node. This node evaluates metrics against a defined threshold condition.

- name: threshold_l2m_large_message
  type: threshold
  condition: value > 0
  filter: item["name"]=="l2m-large-message.count"

The node checks if the metric count exceeds zero, which signifies the presence of large messages. When the condition is met, data is routed to the Threshold Destination pack output.

6. Normal Logs

Logs not matching the large message condition in Large Message Detector are sent to the Normal Destination pack output for processing outside of this pack.

Sample Input