Test a Pipeline

This tutorial tests a Pipeline using the test bench.

  3 minute read  

Overview

In this tutorial you will test a Pipeline containing a route node processor using the test bench you created. You will configure the route node to send all logs from a particular service to the Edge Delta Archive. All other logs will be sent to a local storage destination.

Prerequisites

Before starting this tutorial, complete the steps in Tutorial 1: Create a Test Bench.

1. Create Route Node

  1. In the Edge Delta App, click Pipelines.
  2. Select the testbench fleet and click View/Edit Pipeline.
  3. Click Edit Mode.
  4. Click Add Processor, expand Filters and select Route.
  5. Specify a Name for the node route_test.
  6. Click Add New in the Paths section.
  7. Enter PaymentService_Path in the Path field.
  8. Specify the following CEL macro in the Condition field:
regex_match(item["body"], "PaymentService")
  1. Click Okay.

2. Connect the Route Node.

  1. Connect file_input_test to the input of the route_test node.
  2. Connect the PaymentService_Path output path on the route_test node to the ed_archive node.
  3. Click the link between the file_input_test node and the ed_archive node and click Delete.
  4. Click Review Changes.
  5. Click Save Changes.

3. Test the Pipeline

The pipeline currently pipes logs tailed from the input file through the route node to the ed_archive node. Test this functionality by dropping a log into the input file.

  1. In a terminal, navigate to the local mapped volume of the input file, such as /Users/path/testbench/inputlogs. You configured this location when you created a cluster definition in Tutorial 1: Create a Test Bench.
  2. Echo the following test messages into the file:
echo "{"timestamp": "2024-01-25T17:12:04.873419Z", "logLevel": "TRACE", "service": "PaymentService", "records": [{"data": {"user": "user650", "action": "data_fetch", "details": "Leaving method calculateShippingCost after 15ms, returning value $5.99"}}], "additionalInfo": {"sessionID": "2eaed606-933e-4373-9c67-2427aa3d3bd3", "transactionID": "trx346828"}}
" >> testbench_input_file.log

echo "{"timestamp": "2024-01-25T12:22:27.947573Z", "logLevel": "INFO", "serviceName": "AuthService", "message": "The user has logged in successfully.", "username": "Raider293", "event": "user_logged_in", "outcome": "success"}" >> testbench_input_file.log

This appends the logs one at a time to the file input.

  1. View the testbench_input_file.log file.
cat testbench_input_file.log
  1. After a few minutes, click Logs in the Edge Delta app.

Log search will open and it will show the most recent logs saved to the ed_archive. You may need to filter on your pipeline testbench in the Filters pane if you have more than one agent running. You should see the first log but not the second one because it did not contain PaymentService in the body, so it was not routed on the PaymentService_Path path to ed_archive. If the first log isn’t visible, wait a minute then click the Refresh button to see if it has been ingested and indexed in the archive.

The second log that didn’t match was sent to the unmatched path of the route node, which is not connected to any output so it is dropped. In the next tutorial you will see how to handle nodes’ unmatched or failure paths by routing them to their own destination.

Next Steps

Handle Node Processing Failures