Build Pipelines in Edge Delta

Overview

A Fleet is one or more sets of agents all configured by a single Pipeline.

Visual Pipelines is a tool you can use to view and modify your Pipelines.

Edge Delta Fleets with v3 configurations (Agent v0.1.71 or higher) can be configured using Visual Pipelines. Fleets older than v0.1.71 use configuration v2. See here for more information.

View a Pipeline

1. In the Edge Delta App, click Pipelines.
2. Select the Fleet with the Pipeline you want to view.
3. Click View/Edit Pipeline.

The v3 configuration structure is a graph consisting of three types of nodes, the links between them, and the data handled by the pipeline:

• Source, Processor and Destination Nodes
• Links
• Data Items

View Mode

The Pipeline lists the outgoing data rate for each node. Using this view you can quickly see how the configuration contributes to the overall pipeline efficiency.

The link weight indicates the relative volume of traffic on link between two nodes.

For example, the paymentservice link to the mask-credit-card node with no traffic is rendered smaller than the 129MB of traffic flowing on the link to the Easytrade Team Pack node.

Nodes

Nodes are configuration objects. Each node performs a function and you design a pipeline by creating a flow of traffic through a set of nodes. Select a node to view details about it.

You can toggle between viewing traffic metrics as events or bytes and view a traffic graph for the node. Matched events indicates the input to successful output ratio in event numbers. Bear in mind that nodes that generate events might have a value of more than 100%. Nodes that fail to process traffic due to a configuration error will have a low matching rate.

The details pane also shows the configuration part of the node’s YAML.

Filter Pipeline

You can apply filters to the Sources, Processors and Destinations to view only the connections between the selected objects. Note this changes the view only, not the pipeline configuration.

For example, you can view the data flow along the pipeline between only the Kubernetes source and the Log destination:

Create a Pipeline Configuration

1. Click Pipelines.
2. Click New Fleet.
3. Select Edge Fleet.
4. Optionally, expand Advanced Fleet Configuration and select a pipeline with a configuration you want to duplicate.
5. Click Continue.
6. Select the appropriate template and click Continue.
7. Specify a name to identify the Fleet.
8. Click Generate Config.
9. Execute the installation commands, they include the unique ID for the Fleet.
10. Expand the namespaces and select the input sources you want to monitor.
11. Select the Destination Outputs you want to send processed data to, such as the Edge Delta Observability Platform.
12. Click Continue.
13. Click View Dashboard.

Create a Node

To create a node:

1. Open Visual Pipelines.
2. Click Edit Mode.

3. Click either Add Source, Add Processor, or Add Destination and select the type of node, for example Add Processor - Transformation - Mask Processor.

4. The node appears in the appropriate pane and the configuration pane opens.
5. Enter the configuration details and click Save Changes.

Connect Nodes

To connect nodes, click either the input or the output interface of the node and drag it to the input or output of another node. Some nodes differentiate between default and failure outputs so you can send different results of the node to different destinations.

Links between incompatible nodes can’t be configured - indicated by a red interface. As you release the link, the nodes will rearrange themselves to optimize readability of the data pipeline. To delete a link, select the link and click the Bin icon.

Alternatively, you can add a processor node directly onto an existing link - hover over the link, click the Add (+) icon, and select the node type. This creates the new processor node in the position where the add icon was, and also connects it to the upstream and downstream nodes on the link.

To delete a node, select it and hover over it, then select the Bin icon.

Test Node Configuration

Some nodes have a testing feature where you can specify an input log and see output based on the configuration of the node. See Designing Pipelines and Testing Nodes.

Applying Changes

After making changes to a Pipeline configuration using the Edge Delta Web UI, there is no need to restart or reinstall the Edge Delta Fleet. The configuration settings will be automatically applied after a few minutes.

Configure Agents using a Text Editor

Some configuration v3 parameters are advanced, requiring them to be configured only in YAML rather than Visual Pipelines. You may also prefer or be required to configure in YAML using a text editor, for example if you are passing in a local configuration file.

To edit a v3 configuration using YAML:

1. Click Pipelines and select the fleet you want to edit.
2. Click View/Edit Pipeline.
3. Click Edit YAML.

The Edit Configuration dialog box opens. You can edit the configuration and view the change diff.

3. Click Done.

You can also configure a link in the Pipeline configuration YAML file using the node names:

nodes:
- name: ed_component_health
  type: ed_component_health_input
- name: ed_health
  type: ed_health_output

links:
- from: ed_component_health
  to: ed_health

Delete a Pipeline Configuration:

1. Click Pipelines.
2. Select the fleet configured with the pipeline you want to delete.
3. Click Delete Pipeline.

Deleting a Pipeline will remove the Pipeline ID. Any active Fleets configured with this ID will no longer be able to communicate with the Edge Delta backend.

Legacy Pipelines

Starting at agent version 0.1.71, Edge Delta Fleets are configured with a pipeline that can be represented in a YAML file with a configuration v3 structure. This structure enables configuration using Visual Pipelines. You can also using a text editor to make changes directly to the YAML file. In addition, v0.1.71 agents support v2 configurations.

You can’t edit version 2 configurations using Visual Pipelines. Contact Support to migrate your configurations to the v3 format.

Identifying Your Version

The version is specified in the configuration YAML. Click Pipelines, select the pipeline and open the YAML tab.