delete_matching_keys

Learn about the delete_matching_keys OTTL editor function.

  2 minute read  

This function is used to remove keys from a specified target that match a given pattern. It’s useful for eliminating multiple entries based on pattern matching, helping to clean up or censor log entries by dynamically selecting keys. If a batch of keys need to be deleted in a single operation and no regex pattern covers them all, use an Edge Delta extension function edx_delete_keys or edx_delete_matching_keys.

Syntax: delete_matching_keys(target, pattern)

  • Target: The target refers to the field or object within the log entry from which you wish to remove keys. It typically points to a parent container, such as a JSON object or associative array, which holds multiple key-value pairs.
  • Pattern: The pattern is a regular expression that specifies the keys to be removed. Keys that match this pattern will be deleted along with their associated values.

Input

{
  "_type": "log"
  "body": "..."
  "resource": {
    "container.id": "123456789"
    "container.image.name": "docker.io/edgedelta/loggen:latest"
    "ed.conf.id": "123456789"
    "ed.domain": "pipeline"
    "ed.filepath": "/var/log/pods/loggenlogs_loggen-123456789/loggen/0.log"
    "ed.org.id": "987654321"
    "ed.source.name": "Kubernetes Source"
    "ed.source.type": "kubernetes_input"
    "ed.tag": "loggen"
    "host.ip": "172.18.0.5"
    "host.name": "loggencluster-worker2"
    "k8s.container.name": "loggen"
    "k8s.deployment.name": "loggen"
    "k8s.namespace.name": "loggenlogs"
    "k8s.node.name": "loggencluster-worker2"
    "k8s.pod.name": "loggen-7cc748d75-xh8lq"
     "k8s.pod.uid": "123456789"
    "k8s.replicaset.name": "loggen-7cc748d75"
    "service.name": "loggen"
    "src_type": "K8s"
  }
  "timestamp": 1733375091978
}

Statement

delete_matching_keys(resource, ".*\\.name$")

See Understand Escaping Characters.

Output

{
  "_type": "log"
  "body": "..."
  "resource": {
    "container.id": "123456789"
    "ed.conf.id": "123456789"
    "ed.domain": "pipeline"
    "ed.filepath": "/var/log/pods/loggenlogs_loggen-123456789/loggen/0.log"
    "ed.org.id": "987654321"
    "ed.source.type": "kubernetes_input"
    "ed.tag": "loggen"
    "host.ip": "172.18.0.5"
     "k8s.pod.uid": "123456789"
    "src_type": "K8s"
  }
  "timestamp": 1733375091978
}

The log entry is edited to remove any keys in the resource object matching the pattern .*\\.name$, while preserving other fields within the resource.