Pipeline and Processor Configuration

Overview

See Quickstart: Basic Pipeline Configuration.

This section describes the Pipelines Interface.

1. In the Edge Delta App, click Pipelines.
2. Select the pipeline you want to view.
3. The pipeline opens in the pipeline builder tab.

The configuration structure is a graph consisting of three types of nodes, the links between them, and the data handled by the pipeline:

• Source, Processor, and Destination Nodes.
• Links
• Data Items

Pipeline Builder

The pipeline builder opens with Edit Mode off. Turn it on to build a pipeline.

Filter Pipeline

You can filters by name the Sources, Processors and Destinations to view only the connections between the selected objects. Note this changes the view only, not the pipeline configuration.

For example, you can view the pipeline with only kubernetes_logs, k8s_traffic_metrics, k8s_trace_input, and k8s_metrics nodes and their connections to only the edgedelta destination:

Traffic Volume

The pipeline builder lists the outgoing data rate for each node. Using this view you can quickly see how the configuration contributes to the overall pipeline efficiency.

The link weight indicates the relative volume of traffic on link between two nodes.

For example, the link from the kubernetes_logs node carrying 6GB of traffic is rendered thicker than the 896MB of traffic flowing on the link from the k8s_trace_input node.

Nodes

Nodes are configuration objects. Each node performs a function and you design a pipeline by creating a flow of traffic through a set of nodes. Select a node to view view its paths or double click to view details about it.

You can toggle between viewing traffic metrics as events or bytes and view a traffic graph for the node. Matched events indicates the input to successful output ratio in event numbers. Bear in mind that nodes that generate events might have a value of more than 100%. Nodes that fail to process traffic due to a configuration error will have a low matching rate.

The details pane also shows the configuration part of the node’s YAML.

Pipeline Options

Click the kebab icon to view settings for the pipeline.

• Suspend: This option stops the Edge Delta agents associated with the pipeline from tailing data and collecting metrics or traces. However, they still emit small health events occasionally.
• Duplicate: This option creates a new pipeline with the same configuration but it has a new Pipeline ID and name. This is not available for cloud pipelines.
• Delete: This option deletes the pipeline from the back end. You still need to delete the agents on the edge. See Delete a Pipeline Configuration.

Cloud pipelines have a Settings option. See Edit Cloud Pipeline Resources.

Settings

The Settings tab displays configuration options for the pipeline. These settings can also be configured in YAML format - see Agent Settings for the YAML configuration reference.

The Pipeline Settings tab displays the following configuration options:

General Settings:

• Source Discovery Interval: Configures the duration after which source discovery is invoked (default: 5s). In YAML: source_discovery_interval
• Log Level: Configures the severity level down to which the agent should populate its own log file for troubleshooting (INFO, DEBUG, WARN, ERROR, FATAL). In YAML: log.level

Persisting Cursor Settings: Configures persisting cursor for environments where no data can be lost during agent restarts. In YAML: persisting_cursor_settings

• File Name: Name of the cursor file (default: cursor_file.json)
• File Path: Folder where the cursor file will be created (default: /var/edgedelta/pos)
• Flush Interval: Interval after which the file will be saved to disk from memory (default: 1m)

Read Settings:

• Initial Read Size: Initial buffer size for reading data (default: 1024 bytes)
• Total Read Capacity: Total read buffer capacity (default: 5120 bytes)

Throughput

The Throughput tab on the Pipeline view shows the traffic statistics for that particular pipeline only. Similarly to the Pipelines overview page you can view Bytes in and out as well as Events in and out.

You can filter the graphs by service, node type and name or data type:

Packs

If any packs are used in your pipeline they are listed on the Packs tab. Here you can determine if they are up to date with the latest pack version.

Resource Usage

On the Resource Usage tab you can view graphs for the CPU (agent_cpu_millicores.value and ed.agent.cpu.milicores metrics) and memory usage (agent_mem_alloc.value and ed.agent.memory.allocation metrics) of the Edge Delta agents belonging to this pipeline.

You can filter the graphs by Host:

Agents

On the Agents tab the Edge Delta agents belonging to the pipeline are listed on the Host Name table.

Click Upgrade Agents to view the upgrade commands for the latest version of the agents, and in the environment you selected when you created the pipeline.

Agent Details

Click an agent to view its details for troubleshooting an Edge Delta installation. It opens on the Health tab for the agent:

Here you can examine the health metrics for the agent internal components.

On the Logs tab you can view internal logs generated by that agent:

The metrics tab shows resource usage for the agent, similar to viewing pipeline resource usage filtered by Host Name.

On the Profiling tab you can view the agent’s performance profile. Select a dimension to view the flame graph. See Performance Profiling Edge Delta Agents for more information.

Logs

Back on the Pipeline view page, the Logs tab shows logs generated by Edge Delta agents. You can filter them by Severity: