Edge Delta Dynamic Flow Control

This is a pack that allows dynamic flow control of messages based on a lookup table supporting expiration.

4 minute read

Edge Delta Pipeline Pack for Dynamic Flow Control

Overview

The Dynamic Flow Control pipeline ensures effective, expected and consistent flow of the data pipeline. Any message that doesn’t have the prequisite values defined the default value will be applied. A message will flow through this Pack and if a message contains a matching key value attributes["ed.dfc"]["key"] to the lookup table, the path will be set to the lookup table path (attributes["ed.dfc"]["path"]) if the expiration is in the future. If the expiration (attributes["ed.dfc"]["expiration"]) is in the past the path will be set to the default path (attributes["ed.dfc"]["default_path"]).

Prerequisites

1. Default Path

Data must be tagged with a value in the attribute attributes["ed.dfc"]["default_path"] that will override the default value of unknown. If no path is defined a path of unknow is automatically applied.

2. Key

Data must be tagged with a value in the attribute attributes["ed.dfc"]["key"] that will be a case insensive match to the look up table.

Pack Description

1. Data Ingestion

The data flow starts with the compound_input_92ce node, which is a compound_input node. This node serves as the entry point into the pipeline, where it begins processing the incoming logs.

2. Multiprocessor

Next, logs flow into the Dynamic Flow Control node, which is a Multiprocessor. Multiprocessors are containers that contains all the nodes that perform the actual data manipulations.

  - name: sequence_7461
    type: sequence
    user_description: Dynamic Flow Control

2.1 Default Values

Next, the logs move to the DFC - Setup node, which is a custom ottl node. This node uses OTTL to manipulate the data using OTTL statements. This node ensures all default values are set.

    - type: ottl_transform
      metadata: '{"id":"5VGBw9ftYrZhPkTbag98s","type":"ottl_transform","name":"DFC
        - Setup"}'
      data_types:
      - log
      statements: |-
        set(attributes["ed.dfc"]["default_path"], "10") where attributes["ed.dfc"]["default_path"] == nil
        set(attributes["ed.dfc"]["key"], "unknown") where attributes["ed.dfc"]["key"] == nil
        set(attributes["ed.dfc"]["path"], attributes["ed.dfc"]["default_path"])

2.2 Lookup Table

Next, the logs move to the DFC - Lookup node, which is a lookup node. This node pulls down the lookup table from the Edge Delta SaaS and matches the row based on the attributes["ed.dfc"]["key"] field in the data and lookup table. Upon a match of the key fields the attributes["ed.dfc"]["expiration"] and attributes["ed.dfc"]["path"] are set in the message.

    - type: lookup
      metadata: '{"id":"tp1iHJKLTgxMN_GONdVo0","type":"lookup","name":"DFC - Lookup"}'
      data_types:
      - log
      location_path: ed://ed_dynamic_flow_control.csv
      reload_period: 30s
      ignore_case: true
      match_mode: exact
      key_fields:
      - event_field: attributes["ed.dfc"]["key"]
        lookup_field: key
      out_fields:
      - event_field: attributes["ed.dfc"]["path"]
        lookup_field: path
      - event_field: attributes["ed.dfc"]["expiration"]
        lookup_field: expiration
        default_value: 2020-01-01 00:01 UTC

2.3 Expiration Check

Next, the logs move to the DFC - Expiration node, which is a custom ottl node. This node uses OTTL to manipulate the data using OTTL statements. This node sets the expiration (attributes["ed.dfc"]["expiration"]) and rate (attributes["ed.dfc"]["path"]) after evaluating whether the expiration is in the past or future.

    - type: ottl_transform
      metadata: '{"id":"sYB2_YLp9LeSSOIu-DAL8","type":"ottl_transform","name":"DFC
        - Expiration Check"}'
      data_types:
      - log
      statements: |-
        set(attributes["ed.dfc"]["ts_current"],UnixSeconds(Now()))
        set(attributes["ed.dfc"]["ts_expire"], UnixSeconds(Time(attributes["ed.dfc"]["expiration"], "%Y-%m-%d %H:%M UTC")))
        set(attributes["ed.dfc"]["path"], attributes["ed.dfc"]["default_path"]) where Int(attributes["ed.dfc"]["ts_current"]) > Int(attributes["ed.dfc"]["ts_expire"])

3. Data Egress

The data flow starts with the compound_output_7de1 node, which is a compound_output node. This node serves as the egress point out to the pipeline.

Release Notes

See Get Updates for details about how to upgrade your deployed packs.

Version 1.0 - July, 1 2025

Initial release.

Sample Input

{"timestamp": "2025-07-01T20:52:19.010Z","level": "DEBUG","msg": "Quia velit qui quia non.","user": {"id": 51,"name": "Birdie Haag","email": "darrellschinner@wisozk.net"},"request": {"method": "GET","path": "/slategray/bee/eggplant","ip": "71.169.144.169"},"response_time_ms": 82}

{"timestamp": "2025-07-01T20:52:19.010Z","level": "ERROR","msg": "Nam nihil voluptatum qui officiis.","user": {"id": 18,"name": "Cathy Marks","email": "devinmarks@howe.org"},"request": {"method": "HEAD","path": "/navy/porcupine/bilberry","ip": "160.69.91.211"},"response_time_ms": 975}

{"timestamp": "2025-07-01T20:52:19.010Z","level": "ERROR","msg": "Et cupiditate numquam vitae vel.","user": {"id": 89,"name": "Christine Hansen","email": "allykoepp@adams.com"},"request": {"method": "PUT","path": "/lightpink/gerbil/legume","ip": "190.117.250.151"},"response_time_ms": 399}

{"timestamp": "2025-07-01T20:57:19.010Z","level": "INFO","msg": "Et eos ad officia voluptas.","user": {"id": 73,"name": "Damaris Bergstrom","email": "terrellshanahan@rosenbaum.net"},"request": {"method": "PUT","path": "/navy/moose/lychee","ip": "35.28.134.237"},"response_time_ms": 565}

{"timestamp": "2025-07-01T20:57:19.010Z","level": "DEBUG","msg": "Dolorem modi nihil commodi et.","user": {"id": 36,"name": "Joey Rippin","email": "emilyschuster@botsford.info"},"request": {"method": "GET","path": "/darkviolet/porcupine/lychee","ip": "158.235.41.204"},"response_time_ms": 887}

{"timestamp": "2025-07-01T20:57:19.010Z","level": "INFO","msg": "Velit iure rerum quidem id.","user": {"id": 94,"name": "Destiney Konopelski","email": "alimueller@conn.info"},"request": {"method": "PUT","path": "/lightskyblue/chimpanzee/currant","ip": "67.183.91.83"},"response_time_ms": 660}

{"timestamp": "2025-07-01T20:57:19.010Z","level": "DEBUG","msg": "Cumque quia nisi necessitatibus facilis.","user": {"id": 11,"name": "Lowell Kunze","email": "sandrastrosin@mosciski.com"},"request": {"method": "HEAD","path": "/chartreuse/pug/rock%20melon","ip": "128.64.86.207"},"response_time_ms": 118}

{"timestamp": "2025-07-01T20:57:19.010Z","level": "ERROR","msg": "Voluptatem consequatur accusamus quaerat commodi.","user": {"id": 98,"name": "Amy Spinka","email": "shanemiller@hoppe.biz"},"request": {"method": "POST","path": "/snow/mole/tangerine","ip": "193.203.126.225"},"response_time_ms": 948}

{"timestamp": "2025-07-01T20:57:19.010Z","level": "DEBUG","msg": "Perferendis dicta itaque dolore doloribus.","user": {"id": 33,"name": "Murl Pouros","email": "arturofranecki@lueilwitz.com"},"request": {"method": "POST","path": "/goldenrod/hare/bilberry","ip": "42.139.43.122"},"response_time_ms": 63}

{"timestamp": "2025-07-01T20:57:19.010Z","level": "ERROR","msg": "Voluptas quasi laboriosam libero voluptatibus.","user": {"id": 51,"name": "Michele Becker","email": "glendazboncak@hoeger.com"},"request": {"method": "GET","path": "/darkorchid/panda/pitaya","ip": "197.27.134.28"},"response_time_ms": 51}

{"timestamp": "2025-07-01T20:57:19.010Z","level": "DEBUG","msg": "Nemo aut et ut molestiae.","user": {"id": 67,"name": "Lafayette Rutherford","email": "tannerbreitenberg@hilpert.com"},"request": {"method": "GET","path": "/snow/shark/satsuma","ip": "115.20.202.189"},"response_time_ms": 278}

{"timestamp": "2025-07-01T20:57:20.010Z","level": "ERROR","msg": "Aut ea optio omnis enim.","user": {"id": 3,"name": "Rita Walsh","email": "watsoncarroll@bechtelar.name"},"request": {"method": "GET","path": "/mediumvioletred/porcupine/blackberry","ip": "13.182.66.182"},"response_time_ms": 683}