Build Pipelines in Edge Delta

Overview

A pipeline is the configuration and representation of a set of agents.

Visual Builder is the tool you can use to view and modify your Pipelines.

Edge Delta agents with v3 configurations (Agent v0.1.71 or higher) can be configured using Visual Builder. Agents older than v0.1.71 use configuration v2. See here for more information.

Create a Pipeline Configuration

1. Click Pipelines.
2. Click New Pipeline.
3. Select Edge.
4. Select the appropriate environment template.
5. Specify a name to identify the Fleet.
6. Click Generate Deployment Command.
7. Execute the installation commands, they include the unique ID for the Fleet.
8. Click View Pipeline.
9. Click View Sources Found and select the sources you want to monitor.
10. Click Add Sources.

Alternatively, edit an existing pipeline:

1. In the Edge Delta App, click Pipelines.
2. Select the pipeline you want to view.
3. The pipeline opens in the Visual Builder tab.

Create a Node

To create a node:

1. Open Visual Builder.
2. Click Edit Mode.

When you add a source or a destination, a multiprocessor node is attached to it.

3. Click either Add Source, Add Processor, or Add Destination and select the type of node, for example Add Source - Collect - Demo Source.

4. Enter the configuration details and click Save Changes, Select Destination, or Select Source, depending on the node type.

5. You can select one or more existing nodes to automatically connect your new node to. Alternatively you can choose to create a new node, or connect it to no nodes for now.

The multiprocessor for the new node opens and you can configure it using live capture.

Connect Nodes

To connect nodes, click either the input or the output interface of the node and drag it to the input or output of another node. Some nodes differentiate between default and failure outputs so you can send different results of the node to different destinations.

Links between incompatible nodes can’t be configured - indicated by a red interface. As you release the link, the nodes will rearrange themselves to optimize readability of the data pipeline. To delete a link, select the link and click the Bin icon.

Alternatively, you can add a processor node directly onto an existing link - hover over the link, click the Add (+) icon, and select the node type. This creates the new processor node in the position where the add icon was, and also connects it to the upstream and downstream nodes on the link.

To delete a node, select it and hover over it, then select the Bin icon.

Test Node Configuration

To test a node, use the live capture feature to see actual data and the potential impact of the nodes configuration. See Designing Pipelines and Testing Nodes.

Applying Changes

While in Edit Mode, after making changes, click Review Changes then click Save Changes:

After making changes to a Pipeline configuration using the Edge Delta Web UI, there is no need to restart or reinstall the Edge Delta agents. The configuration settings will be automatically applied after a few minutes.

Configure Agents using a Text Editor

Some configuration v3 parameters are advanced, requiring them to be configured only in YAML rather than Visual Builder. You may also prefer or be required to configure in YAML using a text editor, for example if you are passing in a local configuration file.

To edit a v3 configuration using YAML:

1. Click Pipelines and select the fleet you want to edit.
2. Click View/Edit Pipeline.
3. Click the kebab (⋮) icon and select Edit YAML.

The Edit Configuration dialog box opens. You can edit the configuration and view the change diff.

4. Click Done.

You can also configure a link in the Pipeline configuration YAML file using the node names:

nodes:
- name: ed_component_health
  type: ed_component_health_input
- name: ed_health
  type: ed_health_output

links:
- from: ed_component_health
  to: ed_health

Delete a Pipeline Configuration:

1. Click Pipelines.
2. Select the pipeline you want to delete.
3. Expand the kebab menu and click Delete Pipeline.

Deleting a Pipeline will remove the Pipeline ID and the configuration. Any active agents configured with this ID will no longer be able to communicate with the Edge Delta backend. You still need to remove those agents from their hosts.