Pipeline Quickstart Overview

This guide provides an overview of configuring a basic pipeline using the Telemetry Generator for test data generation and optimization tasks.

  2 minute read  

Overview

This Quick Start Guide illustrates some common pipeline configuration tasks. While a default pipeline will transmit data out of the box, you can configure it to do some optimization. This demonstration will use the Telemetry Generator, which generates dummy traffic.

This quick start will show you how to:

  1. Add a Telemetry Generator Node.
  2. Reduce noise and volume by dropping lower‑value severities (debug, info, trace, notice).
  3. Turn an unstructured log line into structured attributes so later processors can work with discrete fields.
  4. Remove unnecessary attributes.
  5. Normalize disparate severity words so dashboards and alerts can rely on a consistent set.
  6. Use the log timestamp as the ingestion timestamp.
  7. Save and Deploy the Changes.

Before You Begin

Deploy an Edge Delta pipeline. See Quickstart Kubernetes or Quickstart VM.

Note: This quickstart uses live capture to help you explore pipeline configuration with real-time data visibility. To enable live capture, set ED_DISABLE_SAMPLE_COLLECTOR=0 (see environment variables). If you experience UI performance issues with large payloads, you can configure truncation thresholds using the ED_CAPTURER_ITEM_MAX_BODY_SIZE and ED_CAPTURER_ITEM_TRUNCATION_SIZE environment variables to prevent UI freezing.

This demonstration starts after deploying a new pipeline. The Pipeline page is open on the new pipeline, and the pipeline builder interface is open with Edit Mode on.

Image Image

In this example, there is an existing source called Loggen. But we will work with a Telemetry Generator.

To navigate to this point:

  1. Click Pipelines.
  2. Select a pipeline.
  3. Click Edit Mode.

Next Step

Add a Telemetry Generator Node.