Edge Delta Cloud Pipelines

Installing Edge Delta Cloud Pipelines.

  3 minute read  

Beta

Overview

Cloud pipelines are Edge Delta agents that are hosted in cloud infrastructure owned and managed by the Edge Delta team. You might decide to use Cloud pipelines in the following scenarios:

  • data sources are serverless workloads such as AWS Lambda functions or events generated from Amazon Kinesis.
  • lightweight edge environments host thin data producers, such as Internet of Things.
  • you do not want to take on resource management associated with hosting an additional workload.
  • security limitations exist for deploying a pipeline in your environment.

You can create, configure, and remove pipeline configurations using the Edge Delta interface. The Cloud pipelines expose an HTTP, HTTPS, or GRPCS endpoint so you can push data to it.

Push or Pull Data Inputs

Cloud pipelines cater for both push or pull data input integrations. To push data to the Cloud pipeline configure your sources such as a CDN or Amazon Kinesis to push data to the pipeline endpoint. To pull data into the Cloud pipeline, add a source node such as HTTP pull configured with your data source’s endpoint to the Cloud pipeline’s configuration.

Managing a Cloud Pipeline

Click Pipelines and select Cloud from the filter by filter.

Creating a Cloud Pipeline

  1. Click Pipelines.
  2. Click New Pipeline.
  3. Select Cloud
  4. Specify a name to identify the pipeline.
  5. Select Compute Units based on your estimated traffic volume. This is the maximum bandwidth the agent can handle before signalling an error. The number of compute units used per hour counts towards your plan usage.
  6. Click Deploy Cloud Pipeline.

The new Cloud pipeline is added to the pipelines table.

Note: The endpoint is listed on the pipeline’s settings section. Click the kebab icon and select Settings.

Compute Units

You can change a Cloud pipeline’s resouce capability setting known as compute units. This is the maximum bandwidth the agent can handle before signalling an error.

Bear in mind the number of compute units you use per hour contribute to your plan usage in the form of compute units allocated per day. For example, running one Cloud pipeline with one compute unit (a maximum of 12MB per second or 1TB per day), and another Cloud pipeline with three compute units (36MB per second or 3TB per day) will result in a daily usage of 4 compute units. In addition, data flowing from your workloads to your Cloud pipelines will contribute to your Cloud Ingress plan allocation.

Edit Cloud Pipeline Resources

  1. Select the cloud pipeline on the Pipelines page.
  2. Click the kebab icon and select Settings.

The Edit Cloud Pipeline Settings page opens and you can make the changes to the resource settings and view the endpoints.

The Agent Version lists the most recent stable versions and most recent candidate version (containing rc). Choose the latest stable version. If this configuration doesn’t work you can contact Edge Delta support to experiment with the candidate.

Delete a Cloud Pipeline

To delete a Cloud pipeline: Select it on the Pipelines page. Click the kebab icon and select Delete.

Suspend a Cloud Pipeline

You can suspend a cloud pipeline to pause its resource consumption while saving its configuration. Select the pipeline you want to suspend on the Pipelines page. Click the kebab icon and select Suspend. You can resume a suspended cloud pipeline using the Resume option: