Pipeline and Processor Configuration

This page provides detailed information on configuring pipelines and processors, including traffic volume, resource usage, and Edge Delta agent management.

7 minute read

Overview

See Quickstart: Basic Pipeline Configuration.

This section describes the Pipelines Interface.

In the Edge Delta App, click Pipelines.
Select the pipeline you want to view.
The pipeline opens in the pipeline builder tab.

The configuration structure is a graph consisting of three types of nodes, the links between them, and the data handled by the pipeline:

Source, Processor, and Destination Nodes.
Links
Data Items

Pipeline Builder

The pipeline builder opens with Edit Mode off. Turn it on to build a pipeline.

Filter Pipeline

You can filters by name the Sources, Processors and Destinations to view only the connections between the selected objects. Note this changes the view only, not the pipeline configuration.

For example, you can view the pipeline with only kubernetes_logs, k8s_traffic_metrics, k8s_trace_input, and k8s_metrics nodes and their connections to only the edgedelta destination:

Traffic Volume

The pipeline builder lists the outgoing data rate for each node. Using this view you can quickly see how the configuration contributes to the overall pipeline efficiency.

The link weight indicates the relative volume of traffic on link between two nodes.

For example, the link from the kubernetes_logs node carrying 6GB of traffic is rendered thicker than the 896MB of traffic flowing on the link from the k8s_trace_input node.

Nodes

Nodes are configuration objects. Each node performs a function and you design a pipeline by creating a flow of traffic through a set of nodes. Select a node to view view its paths or double click to view details about it.

You can toggle between viewing traffic metrics as events or bytes and view a traffic graph for the node. Matched events indicates the input to successful output ratio in event numbers. Bear in mind that nodes that generate events might have a value of more than 100%. Nodes that fail to process traffic due to a configuration error will have a low matching rate.

The details pane also shows the configuration part of the node’s YAML.

Pipeline Options

Click the kebab icon to view settings for the pipeline.

Suspend: This option stops the Edge Delta agents associated with the pipeline from tailing data and collecting metrics or traces. However, they still emit small health events occasionally.
Duplicate: This option creates a new pipeline with the same configuration but it has a new Pipeline ID and name. This is not available for cloud pipelines.
Delete: This option deletes the pipeline from the back end. You still need to delete the agents on the edge. See Delete a Pipeline Configuration.

Cloud pipelines have a Settings option. See Edit Cloud Pipeline Resources.

Settings

The Settings tab displays configuration options for the pipeline. These settings can also be configured in YAML format - see Agent Settings for the YAML configuration reference.

The Pipeline Settings tab is organized into six sections: General, Logging, Persisting Cursor, Read, Live Tail, and Secrets.

General

The General section controls basic agent behavior.

Setting	Description	Default	YAML Field
Source Discovery Interval	Duration after which source discovery is invoked	5s	`source_discovery_interval`

Logging

The Logging section configures how the agent manages its internal logs. These logs are used for troubleshooting the agent itself.

Setting	Description	Default	YAML Field
Log Level	Severity level for agent logs (DEBUG, INFO, WARN, ERROR, FATAL)	INFO	`log.level`
Buffer Size	Size of the log buffer	-	-
Flush Bytesize	Size threshold that triggers a log flush	-	-
Flush Interval	Interval at which logs are flushed	-	`self_log_flush_interval`

After Config Change (Optional):

Enable faster flushing after configuration changes to surface issues faster. When enabled, the agent temporarily uses a faster flush interval after a config change before returning to the normal interval.

Setting	Description
Enable faster flushing	Toggle to enable temporary faster flushing after config changes
Flush every	The temporary flush interval to use after config changes
For duration	How long to use the temporary interval before returning to normal

Persisting Cursor

Configures persisting cursor for environments where no data can be lost during agent restarts. In YAML: persisting_cursor_settings

Setting	Description	Default
File Name	Name of the cursor file	cursor_file.json
File Path	Folder where the cursor file will be created	/var/lib/edgedelta
Flush Interval	Interval after which the file will be saved to disk from memory	1m

Kubernetes Deployments

In Kubernetes, the File Path must be within the mounted persistent volume for cursor data to survive pod restarts. The default Helm chart mounts /var/lib/edgedelta. Using a path outside this mount (such as /var/edgedelta/pos) will cause cursor data to be lost when pods restart. See Kubernetes Path Requirements for details.

Read

Controls buffer sizes for reading data.

Setting	Description	Default
Initial Read Size	Initial buffer size for reading data	1024 bytes
Total Read Capacity	Total read buffer capacity	5120 bytes

Live Tail

Minimum Agent Version: v2.11.0

These settings control the Live Capture feature, which enables in-stream debugging during pipeline design and testing. Previously, these settings were only configurable via environment variables. Starting with v2.11.0, they can be configured directly in the pipeline settings.

Setting	Description	Default	Environment Variable
Enabled	Enable or disable the live capture feature	true	`ED_DISABLE_LIVE_CAPTURE`
Max Items	Maximum number of items to capture	-	-
Polling Interval	How often the agent polls for capture tasks	-	-
Lower Threshold Size	Lower size threshold for captured items	-	-
Upper Threshold Size	Upper size threshold for captured items	-	-
Item Max Body Size	Maximum size of the body field in captured items before truncation	-	`ED_CAPTURER_ITEM_MAX_BODY_SIZE`
Item Truncation Size	Size threshold that triggers item truncation	-	`ED_CAPTURER_ITEM_TRUNCATION_SIZE`

Configuration Precedence:

Settings are applied in the following order of priority (highest to lowest):

Pipeline settings (configured in the Settings tab)
Environment variables (set during deployment)
Default values

This means pipeline settings always take precedence over environment variables, allowing you to centrally manage these configurations while environment variables serve as fallback defaults.

Use Cases:

Production environments: Disable live capture to save 15-20% memory overhead
Large payloads: Configure truncation sizes to prevent UI freezing
High-volume pipelines: Adjust thresholds to reduce capture overhead while maintaining visibility

For more information about live capture and its resource impact, see Live Capture and In-Stream Debugging.

Secrets

The Secrets section allows you to securely store sensitive credentials (such as API keys, passwords, and access tokens) at the pipeline level. Secrets are encrypted at rest and can be referenced in source and destination configurations using the '{{ SECRET secret-name }}' syntax.

The secrets table displays existing secrets with the following columns:

Column	Description
ID	The unique identifier used to reference the secret in configurations
Secret	The redacted secret value (never displayed in full)
Description	A description of the secret’s purpose

Click Add Secret to create a new secret.

For detailed instructions on creating and using secrets, see Secrets.

Throughput

The Throughput tab on the Pipeline view shows the traffic statistics for that particular pipeline only. Similarly to the Pipelines overview page you can view Bytes in and out as well as Events in and out.

You can filter the graphs by service, node type and name or data type:

Packs

If any packs are used in your pipeline they are listed on the Packs tab. Here you can determine if they are up to date with the latest pack version.

Resource Usage

On the Resource Usage tab you can view graphs for the CPU (agent_cpu_millicores.value and ed.agent.cpu.milicores metrics) and memory usage (agent_mem_alloc.value and ed.agent.memory.allocation metrics) of the Edge Delta agents belonging to this pipeline.

You can filter the graphs by Host:

Agents

On the Agents tab the Edge Delta agents belonging to the pipeline are listed on the Host Name table.

Click Upgrade Agents to view the upgrade commands for the latest version of the agents, and in the environment you selected when you created the pipeline.

Agent Details

Click an agent to view its details for troubleshooting an Edge Delta installation. It opens on the Health tab for the agent:

Here you can examine the health metrics for the agent internal components.

On the Logs tab you can view internal logs generated by that agent:

The metrics tab shows resource usage for the agent, similar to viewing pipeline resource usage filtered by Host Name.

On the Profiling tab you can view the agent’s performance profile. Select a dimension to view the flame graph. See Performance Profiling Edge Delta Agents for more information.

Logs

Back on the Pipeline view page, the Logs tab shows logs generated by Edge Delta agents. You can filter them by Severity: