Hosted Agents

Overview

Hosted Agents are Edge Delta agents that are hosted in cloud infrastructure owned and managed by the Edge Delta team. You might decide to use hosted agents in the following scenarios:

• data sources are serverless workloads such as AWS Lambda functions or events generated from Amazon Kinesis.
• lightweight edge environments host thin data producers, such as Internet of Things.
• you do not want to take on resource management associated with hosting an additional workload.
• security limitations exist for deploying an agent in your environment.

You can create, configure, and remove agent configurations using the Edge Delta interface. The hosted agent can optionally expose an HTTPS endpoint so you can push data to it. If you don’t create an HTTPS endpoint, you can still pull data into the agent by specifying an input in the agent configuration. When you create a hosted agent, you should select a configuration that is logically close to your data source, for example, in the same region or hosted on the same provider.

Aggregator Agents

All hosted agents are Aggregator Agents. Aggregator agent configurations consist a single aggregator agent with multiple processor agents hosted in a Kubernetes cluster. Each processor agent processes different parts of the data while the aggregator agent orchestrates the aggregated data to reduce gaps or duplication. All the agents have the same configuration and no additional settings are required. This parallel architecture allows for high processor throughput as well as high availability with a FIFO buffer to cater for lost or dropped requests.

Push or Pull Data Inputs

Hosted agents cater for both push or pull data input integrations. Push based sources such as a CDN or Amazon Kinesis logs can be consumed on an HTTPS endpoint that the Edge Delta hosted agent exposes. In addition, hosted agents can pull logs from pull-based sources such as AWS Elastic Load Balancing, Kafka, Azure Event Hubs, Amazon S3 and SQS or Amazon CloudWatch.

Creating a Hosted Agent

To create a new hosted agent:

1. Click Data Pipeline in the Edge Delta Web App and select Hosted Agents.

The hosted agents table shows the details of any existing hosted agents for your organization.

2. Click + Create.
3. Enter a descriptive name for the hosted agent.
4. Select an agent version. The interface lists the current stable version (the lower version number) and most recent candidate version (the higher version number). Choose the current stable version. If this configuration doesn’t work you can contact Edge Delta support to experiment with the candidate.
5. Select a public cloud provider. If your preferred cloud provider is not listed, contact Edge Delta support.
6. Select the region where the hosted agent will be hosted.
7. For Config ID, there are 2 options:

• select Generate one for me if you want to create a new agent configuration.
• If you have already configured an agent and you want to reuse that configuration, select the configuration ID from the drop down.

8. Select the HTTPS checkbox if you want your data source to push data to a secure HTTP endpoint on the hosted agent.
9. Click Create.

The new hosted agent is added to the Hosted Agents table. If you configured an HTTPS endpoint, the URL is listed on the table and you use this to configure your push sources. If you created a hosted agent with a newly created configuration ID in step 7, the hosted agent is also automatically configured with an input path for the HTTPS endpoint and this input is added to a workflow by default:

inputs:
  files:
  - labels: captured_requests
    path: /var/captured_requests/body_*.json
   
workflows:
  sample_workflow:
    input_labels:
      - captured_requests

If you created a hosted agent with HTTPS based on an existing configuration ID, you need to add this input path to the hosted agent configuration manually.

Managing a Hosted Agent

Update a Hosted Agent’s Configuration

To adjust the configuration:

1. Click Data Pipeline - Agent Configs.
2. Select Edit from the hamburger menu for the agent you want to configure.

The Edit Configuration page opens where you can edit and test the configuration.

Hosted Agent Resources

You can change a Hosted Agent’s memory limit, CPU limit or agent count on the Hosted Agents page: Select Edit from the hamburger menu for the agent you want to update.

The Edit Hosted Agent page opens and you can make the changes to the resource settings.

Delete a Hosted Agent

To delete a hosted agent, open the Hosted Agent page and select Delete from the hamburger menu for the agent you want to remove. The configuration will remain listed in the agent configuration page until you delete it there too, although it will not have any active agents using it.