Edge Delta Dynamic Sample Control

This is a pack that allows dynamic sampling of messages based on a lookup table supporting expiration.

4 minute read

Edge Delta Pipeline Pack for Dynamic Sample Control

Overview

The Dynamic Sample Control pipeline ensures effective, expected and consistent sampling of the data flowing through the node. Any message that doesn’t have the prequisite values defined the default value will be applied. A message will flow through this Pack and if a message contains a matching key value attributes["ed.dsc"]["key"] to the lookup table, the sample rate will be set to the lookup table sample rate (attributes["ed.dsc"]["rate"]) if the expiration is in the future. If the expiration (attributes["ed.dsc"]["expiration"]) is in the past the rate will be set to the default rate (attributes["ed.dsc"]["default_rate"]). The message will be sampled based on that rate grouped by the key (attributes["ed.dsc"]["key"]).

Prerequisites

1. Default Sample Rate

Data must be tagged with a value in the attribute attributes["ed.dsc"]["default_rate"] that will override the default value of 10 percent sample rate. If no rate is defined a rate of 10% is automatically applied.

2. Key

Data must be tagged with a value in the attribute attributes["ed.dsc"]["key"] that will be a case insensive match to the look up table.

Pack Description

1. Data Ingestion

The data flow starts with the compound_input_da0b node, which is a compound_input node. This node serves as the entry point into the pipeline, where it begins processing the incoming logs.

2. Multiprocessor

Next, logs flow into the Dynamic Sample Control node, which is a Multiprocessor. Multiprocessors are containers that contains all the nodes that perform the actual data manipulations.

  - name: sequence_e651
    type: sequence
    user_description: Dynamic Sample Control

2.1 Default Values

Next, the logs move to the DSC - Setup node, which is a custom ottl node. This node uses OTTL to manipulate the data using OTTL statements. This node ensures all default values are set.

    - type: ottl_transform
      metadata: '{"id":"5VGBw9ftYrZhPkTbag98s","type":"ottl_transform","name":"DSC
        - Setup"}'
      data_types:
      - log
      statements: |-
        set(attributes["ed.dsc"]["default_rate"], Double("10")) where attributes["ed.dsc"]["default_rate"] == nil
        set(attributes["ed.dsc"]["key"], Double("10")) where attributes["ed.dsc"]["key"] == nil
        set(attributes["ed.dsc"]["rate"],Double(attributes["ed.dsc"]["default_rate"]))

2.2 Lookup Table

Next, the logs move to the DSC - Lookup node, which is a lookup node. This node pulls down the lookup table from the Edge Delta SaaS and matches the row based on the key field in the data and lookup table. Upon a match of the key fields the attributes["ed.dsc"]["expiration"] and attributes["ed.dsc"]["rate"] are set in the message.

    - type: lookup
      metadata: '{"id":"tp1iHJKLTgxMN_GONdVo0","type":"lookup","name":"DSC - Lookup"}'
      data_types:
      - log
      location_path: ed://ed_dynamic_sample_control.csv
      reload_period: 30s
      ignore_case: true
      match_mode: exact
      key_fields:
      - event_field: attributes["ed.dsc"]["key"]
        lookup_field: key
      out_fields:
      - event_field: attributes["ed.dsc"]["rate"]
        lookup_field: rate
      - event_field: attributes["ed.dsc"]["expiration"]
        lookup_field: expiration
        default_value: 2020-01-01 00:01 UTC

2.3 Expiration Check

Next, the logs move to the DSC - Expiration node, which is a custom ottl node. This node uses OTTL to manipulate the data using OTTL statements. This node sets the expiration (attributes["ed.dsc"]["expiration"]) and rate (attributes["ed.dsc"]["rate"]) after evaluating whether the expiration is in the past or future.

    - type: ottl_transform
      metadata: '{"id":"sYB2_YLp9LeSSOIu-DAL8","type":"ottl_transform","name":"DSC
        - Expiration Check"}'
      data_types:
      - log
      statements: |-
        set(attributes["ed.dsc"]["ts_current"],UnixSeconds(Now()))
        set(attributes["ed.dsc"]["ts_expire"], UnixSeconds(Time(attributes["ed.dsc"]["expiration"], "%Y-%m-%d %H:%M UTC")))
        set(attributes["ed.dsc"]["rate"], attributes["ed.dsc"]["default_rate"]) where Int(attributes["ed.dsc"]["ts_current"]) > Int(attributes["ed.dsc"]["ts_expire"])
        set(attributes["ed.dsc"]["rate"], Double(attributes["ed.dsc"]["rate"]))

2.4 Sample Message

Logs are subsequently processed by the DSC - Sample node, Sample node. This node samples messages based on the rate defined (attributes["ed.dsc"]["rate"]) and grouped by the key (attributes["ed.dsc"]["key"]).

    - type: sample
      metadata: '{"id":"Zayzdz3J2xV-OQJ4Hg3nT","type":"sample","name":"DSC - Sample"}'
      data_types:
      - log
      priority_field: attributes["ed.dsc"]["rate"]
      pass_through_on_failure: false

3. Data Egress

The data flow starts with the compound_output_7de1 node, which is a compound_output node. This node serves as the egress point out to the pipeline.

Release Notes

See Get Updates for details about how to upgrade your deployed packs.

Version 1.0 - July, 1 2025

Initial release.

Sample Input

{"timestamp": "2025-07-01T20:52:19.010Z","level": "DEBUG","msg": "Quia velit qui quia non.","user": {"id": 51,"name": "Birdie Haag","email": "darrellschinner@wisozk.net"},"request": {"method": "GET","path": "/slategray/bee/eggplant","ip": "71.169.144.169"},"response_time_ms": 82}

{"timestamp": "2025-07-01T20:52:19.010Z","level": "ERROR","msg": "Nam nihil voluptatum qui officiis.","user": {"id": 18,"name": "Cathy Marks","email": "devinmarks@howe.org"},"request": {"method": "HEAD","path": "/navy/porcupine/bilberry","ip": "160.69.91.211"},"response_time_ms": 975}

{"timestamp": "2025-07-01T20:52:19.010Z","level": "ERROR","msg": "Et cupiditate numquam vitae vel.","user": {"id": 89,"name": "Christine Hansen","email": "allykoepp@adams.com"},"request": {"method": "PUT","path": "/lightpink/gerbil/legume","ip": "190.117.250.151"},"response_time_ms": 399}

{"timestamp": "2025-07-01T20:57:19.010Z","level": "INFO","msg": "Et eos ad officia voluptas.","user": {"id": 73,"name": "Damaris Bergstrom","email": "terrellshanahan@rosenbaum.net"},"request": {"method": "PUT","path": "/navy/moose/lychee","ip": "35.28.134.237"},"response_time_ms": 565}

{"timestamp": "2025-07-01T20:57:19.010Z","level": "DEBUG","msg": "Dolorem modi nihil commodi et.","user": {"id": 36,"name": "Joey Rippin","email": "emilyschuster@botsford.info"},"request": {"method": "GET","path": "/darkviolet/porcupine/lychee","ip": "158.235.41.204"},"response_time_ms": 887}

{"timestamp": "2025-07-01T20:57:19.010Z","level": "INFO","msg": "Velit iure rerum quidem id.","user": {"id": 94,"name": "Destiney Konopelski","email": "alimueller@conn.info"},"request": {"method": "PUT","path": "/lightskyblue/chimpanzee/currant","ip": "67.183.91.83"},"response_time_ms": 660}

{"timestamp": "2025-07-01T20:57:19.010Z","level": "DEBUG","msg": "Cumque quia nisi necessitatibus facilis.","user": {"id": 11,"name": "Lowell Kunze","email": "sandrastrosin@mosciski.com"},"request": {"method": "HEAD","path": "/chartreuse/pug/rock%20melon","ip": "128.64.86.207"},"response_time_ms": 118}

{"timestamp": "2025-07-01T20:57:19.010Z","level": "ERROR","msg": "Voluptatem consequatur accusamus quaerat commodi.","user": {"id": 98,"name": "Amy Spinka","email": "shanemiller@hoppe.biz"},"request": {"method": "POST","path": "/snow/mole/tangerine","ip": "193.203.126.225"},"response_time_ms": 948}

{"timestamp": "2025-07-01T20:57:19.010Z","level": "DEBUG","msg": "Perferendis dicta itaque dolore doloribus.","user": {"id": 33,"name": "Murl Pouros","email": "arturofranecki@lueilwitz.com"},"request": {"method": "POST","path": "/goldenrod/hare/bilberry","ip": "42.139.43.122"},"response_time_ms": 63}

{"timestamp": "2025-07-01T20:57:19.010Z","level": "ERROR","msg": "Voluptas quasi laboriosam libero voluptatibus.","user": {"id": 51,"name": "Michele Becker","email": "glendazboncak@hoeger.com"},"request": {"method": "GET","path": "/darkorchid/panda/pitaya","ip": "197.27.134.28"},"response_time_ms": 51}

{"timestamp": "2025-07-01T20:57:19.010Z","level": "DEBUG","msg": "Nemo aut et ut molestiae.","user": {"id": 67,"name": "Lafayette Rutherford","email": "tannerbreitenberg@hilpert.com"},"request": {"method": "GET","path": "/snow/shark/satsuma","ip": "115.20.202.189"},"response_time_ms": 278}

{"timestamp": "2025-07-01T20:57:20.010Z","level": "ERROR","msg": "Aut ea optio omnis enim.","user": {"id": 3,"name": "Rita Walsh","email": "watsoncarroll@bechtelar.name"},"request": {"method": "GET","path": "/mediumvioletred/porcupine/blackberry","ip": "13.182.66.182"},"response_time_ms": 683}