Troubleshooting Patterns for Structured Logs

Overview

The Log to Pattern Metric processor finds patterns by comparing words in the same position across log lines. Words that stay the same become the pattern, and words that change become *. Plain-text logs work well because they contain fixed text such as connection refused.

Structured logs, such as compact JSON or key=value pairs, do not work well when the processor clusters the raw body. The fix is to build a short line from the fields that matter and cluster on that line instead. The original log continues downstream unchanged.

Note: The sample_field_path and max_token_count options used on this page require Edge Delta agent version v2.23.0 or higher. On older agents, clustering on a projection with field_path still works, but cluster samples show the projected line instead of the original log, and the processor keeps at most 50 words per line.

flowchart LR
    A["Raw JSON body"] --> B["OTTL: parse and project"]
    B --> C["attributes.pattern_input"]
    C --> D["Log to Pattern Metric
field_path: pattern_input
sample_field_path: body"]
    D --> E["Patterns with original log samples"]

Symptoms

How the processor reads a line

Before clustering, the processor prepares each line in three steps:

1. Select the field. The processor uses body, or the field set in field_path.
2. Mask and split. The processor reads the first 1 KB. It replaces timestamps, numbers, IP addresses, UUIDs, and any word that contains a digit with *. It then turns punctuation such as " , : { } [ ] into spaces and keeps the first 50 words.
3. Cluster. Lines with the same word count and similar words in the same positions join one pattern. similarity_threshold sets how similar they must be.

Masking runs on whitespace-separated words before punctuation is split. A compact JSON body such as {"route":"search","user_id":"1234567"} is a single word that contains digits, so step 2 replaces the entire line with *.

Fix: cluster on a projection of stable fields

Parse the body, choose the fields that describe what happened, and write them into one attribute as space-separated words. Point field_path at that attribute and sample_field_path at the body.

This example parses JSON logs, converts the numeric status code into a word, and clusters on route, method, and status class:

nodes:
- name: app_logs_multiprocessor
  type: sequence
  processors:
  - type: ottl_transform
    data_types:
    - log
    statements: |-
      merge_maps(attributes, ParseJSON(body), "upsert") where IsMatch(body, "^\\s*[{]")
      set(attributes["status_class"], "success") where attributes["status_code"] != nil and Int(attributes["status_code"]) < 400
      set(attributes["status_class"], "client_error") where attributes["status_code"] != nil and Int(attributes["status_code"]) >= 400 and Int(attributes["status_code"]) < 500
      set(attributes["status_class"], "server_error") where attributes["status_code"] != nil and Int(attributes["status_code"]) >= 500
      set(attributes["pattern_input"], Concat(["route", attributes["route"], "method", attributes["method"], "status", attributes["status_class"]], " "))
  - type: log_to_pattern_metric
    keep_item: true
    data_types:
    - log
    field_path: item["attributes"]["pattern_input"]
    sample_field_path: item["body"]
    samples_per_cluster: 3
    reporting_frequency: 1m
    similarity_threshold: 0.9

A log with "route":"search","method":"POST","status_code":200 produces the projection route search method POST status success. Each combination of route, method, and status class becomes its own pattern, and the cluster samples show the original JSON log.

Rules for a good projection

Follow these rules when you choose fields for the projection:

• Do not cluster a serialized map directly. Parse it first, then project fields or split it into words.
• Turn numbers into words. Any word with a digit becomes *, so a status code of 404 can never define a pattern. Map numbers to classes such as client_error first.
• Leave out IDs and other high-cardinality values. Fields such as user_id and request_id stay searchable as attributes on the original log.
• Keep the field order and field count fixed. The processor compares words by position and groups lines by word count first. A missing field or a different order creates a separate pattern.
• Prefer key: value in the source format. If you control how the application writes logs, the processor splits words on : but not on =.
• Match similarity_threshold to the projection length. With N words and threshold T, two lines that differ in one word join the same pattern only when (N - 1) / N is at least T. The example has 6 words, and (6 - 1) / 6 is about 0.83, which is below 0.9. Lines that differ in any field therefore stay in separate patterns.

Count by entity

Patterns group events by kind. To count events for each user or customer, use the Extract and Aggregate Metric processors with the entity field as a dimension, or set group_by on the pattern processor. Both create one series or cluster group per value, so check the cardinality of the field first.

Fix: split the whole body into words

When you cannot choose fields in advance, remove the JSON punctuation and cluster on the full key and value list. The keys become the fixed words of each pattern, and changing values become *:

  - type: ottl_transform
    data_types:
    - log
    statements: |-
      set(attributes["pattern_input"], body)
      replace_pattern(attributes["pattern_input"], "[{}\",:]+", " ")
  - type: log_to_pattern_metric
    keep_item: true
    data_types:
    - log
    field_path: item["attributes"]["pattern_input"]
    sample_field_path: item["body"]
    max_token_count: 80

This needs no field decisions, but the patterns are long and harder to read than a projection, and numeric fields such as status codes still become *. Each key and value pair is two words, so raise max_token_count (10 to 200) when logs carry more than 25 pairs.

Reproduce with sample data

Use this script to generate compact JSON logs with about 20 key and value pairs per line, then point a File source at the output file. The script writes 5 lines per second for 30 minutes.

#!/usr/bin/env python3
# Writes compact JSON analytics logs to /tmp/kv-demo/app.log.
import json, os, random, time, uuid

ROUTES = [
    ("get_order", "/api/v1/orders/{id}", "GET"),
    ("list_items", "/api/v1/carts/{id}/items", "GET"),
    ("search", "/api/v1/search", "POST"),
    ("checkout", "/api/v1/checkout", "POST"),
]
STATUS = [200] * 17 + [404, 403, 500]

def line():
    route, path, method = random.choice(ROUTES)
    entity = str(random.randint(10**9, 10**10 - 1))
    return json.dumps({
        "application": random.choice(["web", "ios", "android", "public-api"]),
        "created": int(time.time()),
        "datacenter": "region-a",
        "duration_ms": round(random.uniform(0.5, 900.0), 2),
        "tenant_id": random.choice(["tenant-a", "tenant-b", "tenant-c"]),
        "event_id": str(uuid.uuid4()),
        "host_name": "example-api-01",
        "ip_addr": f"10.0.{random.randint(0, 255)}.{random.randint(1, 254)}",
        "is_ssl": "1",
        "language_id": random.randint(1, 12),
        "method": method,
        "path": path.replace("{id}", entity),
        "request_id": f"{random.getrandbits(60):015x}",
        "route": route,
        "service": "example-api",
        "status_code": random.choice(STATUS),
        "trace_id": str(uuid.uuid4()),
        "user_agent": random.choice(["okhttp/5.3.2", "Mozilla/5.0", "python-requests/2.32"]),
        "user_id": entity,
        "version": "1.0.0",
    }, sort_keys=True, separators=(",", ":"))

os.makedirs("/tmp/kv-demo", exist_ok=True)
with open("/tmp/kv-demo/app.log", "a", buffering=1) as f:
    end = time.time() + 30 * 60
    while time.time() < end:
        for _ in range(5):
            f.write(line() + "\n")
        time.sleep(1)

With the raw body, all lines fall into one * pattern. With the projection above, the Patterns page shows one pattern per route, method, and status class, for example route search method POST status success.

Test the projection with edx

Before you deploy, dry-run the OTTL statements against sample logs with the edx CLI. Each line of samples.jsonl is one log item, for example {"body":"{\"route\":\"search\",\"method\":\"POST\",\"status_code\":200}","attributes":{},"resource":{},"timestamp":1}:

edx pipelines test ottl <pipeline-id> --file samples.jsonl --statements '<ottl statements>'

Check that each item has the expected pattern_input attribute. Pattern processors emit on reporting_frequency, so verify the patterns after deployment with Live Capture or the Patterns page.

If you use an AI coding assistant, the ed-pipeline-tuning skill includes this recipe. Update edx and reinstall the skills so the assistant has the current version:

edx update
edx skills update

Then ask the assistant to build the projection for your logs. For example: “Generate a sample data generator and a pipeline that extracts patterns from my structured JSON logs.”

See Also

• Log to Pattern Metric processor
• Extract and Aggregate Metrics
• edx command reference
• Explore Patterns