Testing Edge Delta Agent Configurations

Test a configuration using a sample of your logs.

  6 minute read  

Version 2 is no longer supported. This section is for historical reference only.

You can use Visual Pipelines to test agent configurations written in the v3 format (for agent versions 0.1.71 and later).

After creating a new agent configuration or modifying an existing agent you can test the configuration using a sample of your logs. This enables you to confirm the agent is behaving as expected before deploying it and without having to wait for logs to be generated. You can test the following agent configuration objects:

  • Workflows
  • Regex Processors
  • Clustering Processors
  • Regex filters
  • Mask filters
  • JSON extraction filters
  • JavaScript functions

Configuration Testing Prerequisites

To test workflows, you need any integrations mentioned in the agent configuration to be already set up on the integrations page. See the Outputs page for information on how to integrate a downstream destination.

Configuration Testing

To start testing a configuration, you make the required changes to an existing agent configuration:

  1. In the Data Pipeline section, click the kebab (⋮) button of the agent you want to reconfigure and test.
  2. Click Edit. The Edit Configuration page opens.
  1. Edit the configuration in the yaml code pane. You can right-click to use features such as Go To Definition and Change All Occurrences.

Don’t save the new configuration until you are finished testing. Alternatively, create a new default agent to use for testing.

  1. Create a sample of log data from your data sources. Include logs for events that you want the updated configuration to process in a certain manner.
  2. Paste the log data you created in the Test Input Data pane.
  3. Select a specific part of the configuration to test from the Object To Test menu. The options that are available in the menu reflect the configuration in the yaml code pane. The interface buttons also change depending on which object type you are testing, for example, regex processors allow to you test the regex pattern separately from the processor.
  4. Optionally, select a Test Input if the configuration that was selected in the previous step contains more than one input.
  5. Optionally, select a Streaming Destination if the configuration that was selected previously contains more than one output.

The following sections describe how to test specific objects within the agent configuration.

Test Workflows

To test a workflow

  1. Select the workflow name from the Object To Test menu.
  2. If the workflow contains more than one input label, select the input you want to test from the Test Input menu.
  3. If the workflow contains more than one streaming destination, select the destination you want to test from the Streaming Destination menu.

Only streaming destinations (those that capture data such as statistics, anomaly captures, etc.) will be listed in the Streaming Destinations menu. Raw log archives, for example, aren’t listed even if they are in the destinations object within the workflow.

  1. Click Test Workflow.

The results pane displays the output that would be sent to the Streaming Destination. If the output is not the expected behavior, you can test its components such as the processors and filters individually.

Test Regex Processors

There are two ways you can test a regex processor. You can test the regex pattern against your sample log data, and you can test the processor output:

  1. Select the processor name in the Object to Test menu.
  2. Select an input if more than one input has been defined in the Inputs section of the configuration.
  3. Optionally, click Test Regex. The entire sample log file is shown in the Results pane and any matches with the processor’s regex pattern are highlighted. The number of matching records is also shown.
  1. Click Test Processor. The output (metrics) of the regex processor is shown in the results pane.

After testing both the regex and the processor, you can tab between the two sets of results by selecting Regex or Processor in the results pane.

Test Clustering Processors

To test a clustering processor

  1. Select a clustering processor’s name from the Object To Test menu.
  2. Select an input if more than one input has been defined in the Inputs section of the configuration.
  3. Click Test Processor

The output (metrics) of the clustering processor is shown in the results pane, listing counts for clusters of patterns discovered in the test data.

Test Regex filters

To test a regex type filter

  1. Select the filter name in the Object to Test menu.
  2. Select an input if more than one input has been defined in the Inputs section of the configuration.
  3. Click Test Filter.

If the regex filter being tested has the negate parameter set to true, the results pane displays the full data set with regex matches highlighted and the records containing the regex matches struck through. The struck through records will not be passed by the filter.

If the regex filter being tested does not have the negate parameter, or if it is set to false, the results pane displays the full data set with regex matches highlighted. The records with no regex matches are struck through and these records will not be passed by the filter.

Test Mask filters

To test a mask type filter

  1. Select the filter name in the Object to Test menu.
  2. Select an input if more than one input has been defined in the Inputs section of the configuration.
  3. Click Test Filter.

The Results pane displays the entire test input data but it highlights where regex matches have been replaced with mask text, such as replacing card numbers with the word “REDACTED”.

If the mask configuration includes a capture group, changes are made only in the records where the entire group is matched. The key matches are highlighted and the masked values are assigned a color per capture group.

Test JSON extraction filters

To test a filter that extracts a JSON field

  1. Select the filter name in the Object to Test menu.
  2. Select an input if more than one input has been defined in the Inputs section of the configuration.
  3. Click Test Filter.

The Results pane displays only the extracted portion of the log, for example, the message field.

Test JavaScript functions

JavaScript functions are used to transform the message body of a log. If the JavaScript function was written as a filter, it must be tested using a workflow. There are three requirements for using a log-transformer-javascript type filter: the logs must contain JavaScript that matches the configuration, the raw log variable must be named log and the workflow destination must have the log Feature enabled. Consider the following filter configuration:

filters:
  - name: add_json_field
    type: log-transformer-javascript
    script: |
      var obj = JSON.parse(log);
      obj.version = "latest";
      log = JSON.stringify(obj);      
    required: true

This filter adds the version key with the value latest to matching logs. In the following example, the archive workflow is tested. It includes the add_json_field filter and specifies a streaming destination called sumo486, which is configured in the outputs section with the log feature. Given this log:

{"app": "alpine", "service": "nonprod", "source": "tomcat"}

The filter adds a JSON field as follows:

"body": "{\"app\":\"alpine\",\"service\":\"nonprod\",\"source\":\"tomcat\",\"version\":\"latest\"}"

This behavior is illustrated in the following image:

Deploying Configuration Changes

After completing the tests, click Save Edits to deploy the updated agent. If you used a new agent configuration for testing purposes rather than editing the production agent, copy the configuration you tested into the production agent and click Save Edits.