Edge Delta Custom Attribute Filter

Identify logs with a specific field name and matching values.

5 minute read

A custom attribute filter is used to identify logs with a specific field name and matching values, including custom fields added by an Edge Delta configuration. You can set the filter to either allow only logs with matching fieldname value pairs, or to drop logs with matching field name value pairs. After being configured in the filters section of the agent yaml, customer attribute filters can be referenced in input definitions, processors or within workflows.

Example Scenarios

If you created custom attributes using an input configuration or enrichment, you can filter on those custom attributes using a custom attributes filter

This filter is particularly useful for filtering, for example, on a specific Kubernetes object type, such as a namespace, pod, or container etc. using the keys generated by an Edge Delta Kubernetes or Docker input configuration.

A Custom Attribute filter is configured using a simple key and value pair. The key specifies the field name to search and the value specifies the matching value for that field name. Alternatively, a regex pattern can be used to match values. In addition, the filter can be set to drop all except matching logs (default) or to drop only matching logs.

Example Configurations

Filter on Custom Attributes

The following example allows only logs where the service field is billing.

filters:
  - name: service_billing
    type: custom-attributes
    key: service
    value: billing

Create Source Specific pipelines

In the following example, all logs that contain a namespace_name value of openfaas (originally K8sNamespace) will be identified and all other logs will not be passed by the filter. This filter can be used to create a log pipeline specifically for that component.

filters:
  - name: openfaas_namespace
    type: custom-attributes
    key: "namespace_name"
    value: openfaas

The following keys can be used to identify specific sources using the keys assigned by Edge Delta inputs.

Edge Delta Assigned Key	Native Key
K8sNamespace	“namespace_name”
K8sControllerKind	“controllerKind”
K8sControllerName	“controllerName”
K8sControllerLogicalName	“controllerLogicalName”
K8sPodName	“pod_name”
K8sPodID	“pod_id”
K8sContainerName	“container_name”
K8sContainerImage	“container_image”
K8sDockerID	“docker_id”
DockerContainerName	“containerName”
DockerImageName	“imageName”
ECSClusterLabel	“com.amazonaws.ecs.cluster”
ECSClusterLabelInternalFormat	“labels.com.amazonaws.ecs.cluster”
ECSContainerNameLabel	“com.amazonaws.ecs.container-name”
ECSContainerNameLabelInternalFormat	“labels.com.amazonaws.ecs.container-name”
ECSTaskVersionLabel	“com.amazonaws.ecs.task-definition-version”
ECSTaskVersionLabelInternalFormat	“labels.com.amazonaws.ecs.task-definition-version”
ECSTaskFamilyLabel	“com.amazonaws.ecs.task-definition-family”
ECSTaskFamilyLabelInternalFormat	“labels.com.amazonaws.ecs.task-definition-family”
FileGlobPath	“globPath”
EnvironmentAttribute	“environment”
AppAttribute	“app”
RegionAttribute	region"

Drop filter

The default behavior of the custom attributes filter is to only allow matching logs to pass. The opposite behavior can be configured with the negate parameter. If it is set to true, the matching logs will be dropped with all other logs passing. In the following example, logs where the service is billing will be dropped.

filters:
  - name: service_billing_drop
    type: custom-attributes
    key: service
    value: billing
    negate: true

Match Conditions

In the following example, comma separated values are matched. If either one or both of them match, the log will be passed through.

  - name: two_values_filter
    type: custom-attributes
    key: component
    value: credithandler,debithandler

In the following example, logs where the level field’s value matches an error regex pattern will be passed.

  - name: error_level
    type: custom-attributes
    key: level
    pattern: "error|ERROR|problem|ERR|Err"

Required Parameters

name (required)

The name parameter specifies the name for the filter. You refer to this name in other places, for example to refer to a specific filter in a workflow or processor. Names must be unique within the filters: section. It is a yaml list element so it begins with a - and a space followed by the string. A name is a required parameter for a filter

filters:
  - name: <filter-name>

type: custom-attributes (required)

The type parameter in the filter context specifies the type of filter to apply. A type is a required parameter for a filter.

filters:
  - name: <filter-name>
    type: <filter-type>

key (required)

The key parameter defines the field name in which to search for matching values. It defines an existing source definition or a custom field that has been generated. It is a string value. A key is a required parameter for a custom attributes filter.

filters:
  - name: <filter-name>
    type: custom-attributes
    key: <field name>

value or pattern (required)

The value parameter defines the string to search for within the specified field name. It is defined as a string and it can be comma separated to specify multiple matching strings, any one of which will trigger the filter behavior. A value parameter is required for a custom attributes filter unless there is a pattern parameter to define matching values.

filters: 
  - name: <filter-name>
    type: custom-attributes
    key: <field name>
    value: <matching value string>,<matching value string>

The pattern parameter defines the matching pattern to search for within the specified field name. It is defined as a Golang regex pattern. A pattern parameter is required for a custom attributes filter unless there is a value parameter to define the exact matching value or values.

filters: 
  - name: <filter-name>
    type: custom-attributes
    key: <field name>
    pattern: <regex pattern>

Optional Parameters

negate

The negate parameter defines the dropping behavior of the custom-attributes filter. By default it is set to false to drop any logs that do not match the key value pair specified in the filter. You can set it to true to drop only logs that match the key value pair specified in the filter. It is a Boolean value and it is optional.

filters:
  - name: <filter name>
    type: custom-attributes
    key: <field name>
    value: <value string>
    negate: true|false