CEL Custom Macros

Use CEL Custom Macros to reference log fields.

CEL Macro Overview

There are several Common Expression Language (CEL) custom macros you can use to reference fields, for example in the Enrichment node’s field mappings parameter. Custom macros are defined as extensions to the CEL. As with any general CEL expression, references to fields that don’t exist will return an error. When used in a transformation or mapper, these CEL expressions are handled as best effort and expressions resulting in an error will be replaced with an empty string (“”). The inputs to these functions are CEL field path expressions referring to fields that have the given type.

CEL Macros

convert_timestamp(input location string, input format string, output format string)

This macro is used to convert timestamps. There are three options:

  • convert between datetime stamp and datetime stamp formats
  • convert a datetime stamp to a unix format
  • convert a unix format to a datetime stamp.

You specify the field location of the timestamp, the current format, and the desired format:

  • input location: Specify the location of the timestamp field using the field path.
  • input format: Provide an example of the format of the current timestamp. You can copy the the timestamp from an actual log. If the format does not match the incoming log’s timestamp format, the processor will fail.
  • output format: Provide an example of the desired format for the timestamp. You can enter an example in one of the following formats, or copy the format you require from this list:
    • “Unix Second”
    • “Unix Milli”
    • “Unix Nano”
    • “2006-01-02”
    • “2006-01-02T15:04:05Z”
    • “2006-01-02T15:04:05”
    • “2006-01-02T15:04:05.000Z”
    • “2006-01-02T15:04:05.000000Z”
    • “2006-01-02T15:04:05.000000000Z”
    • time.RFC1123
    • time.RFC1123Z
    • time.RFC3339
    • time.RFC3339Nano
    • “01/02/06”
    • “15:04”
    • “01/02/2006 15:04”
    • “January 2, 2006”
    • “15:04:05”
    • “January 2, 2006 15:04:05”
    • “January 2, 2006 15:04:05.000”
    • “January 2, 2006 15:04:05.000000”
    • “January 2, 2006 15:04:05.000000000”
    • “Mon, Jan 2, 2006 3:04 PM”
    • “2 January 2006 15:04”
    • “2 Jan 2006 15:04”

Example:

convert_timestamp(item["attributes"]["timestamp"], "2006-01-02T15:04:05.999Z", "Unix Milli")
convert_timestamp(item["attributes"]["timestamp"], "2024-01-02T15:03:06.000Z", "Unix Milli")

Both these examples will create the same configuration because the timestamp examples are in the same format even though they show different datetimes.

first_non_empty(listOfStrs []string)

  • Input: []string
  • Output: string

This macro returns the first non empty string from the input parameters.

Note hardcoded fallback values can not contains commas and the first_non_empty. In addition, this function can’t be nested within other CEL macros. However, you can apply the first_non_empty function, upsert it into the data item, and then apply any other cel macros on that new field.

regex_match(input string, regex string)

  • Input: string, string
  • Output: bool

Returns whether or not the input string matches the regex string.

regex_capture(input string, regexWithCaptureGroups string)

  • Input: string, string
  • Output: map[string]string

Returns one or more parts from the string using regex capture groups. The key for the returned map is the capture group and the value for the map is the value for that capture group.

env(envVarKey string)

  • Input: string
  • Output: string

Returns the value from the environment variables, if the environment variable doesn’t exist an empty string will be returned.

from_k8s(podID string, podAttributeName string)

  • Input: string, string
  • Output: string

Returns the attribute value of a K8s pod given a pod id, if the pod id is not found this will return an error.

json(jsonStr string)

  • Input: string
  • Output: Map[string]any

Un-marshalls a json string and returns the map representation of the json object. If the unmarshalling fails an error will be returned.

json(item["body"]).event

ec2_metadata(keyStr string)

  • Input: string
  • Output: string

Returns the value of given key from EC2 metadata service. If the key is not found an error is returned.

gcp_metadata(key)

  • Input: string
  • Output: string

Returns the value given a key from GCP metadata service. If the key is not found an error is returned.

merge(firstMap map[string]any, secondMap map[string]any)

  • Input: map[string]any, map[string]string
  • Output: map[string]string

Takes two maps and merges them together. If either map is empty the other will be returned. If both maps are empty, an empty map will be returned.

Testing CEL

On v3 configurations, the Visual Pipeline can be used in edit mode to test these CEL expressions. In the pipeline view:

  1. Click Edit Mode
  2. Select a node
  3. Click Test
  4. Open the CEL tab

You can expand the CEL Expression library section to view snippets of supported expressions:

In this example a CEL reference for the second field mapping item["resource"]["host.name"]is tested, the output shows only the resulting of the CEL expression ED_TEST:

You can view the log that would be output in the Processor tab of the test pane. Now the enriched attribute enriched-hostname and value ED_TESTis visible: