Edge Delta Rehydrations

Overview

Rehydration is the process of pushing already-archived data to a streaming destination, such as Splunk, Elasticsearch, etc. If you want to prevent duplicated logs, you can configure the Edge Delta rehydrator to check for past rehydrations and automatically create a filter to exclude logs that have already been rehydrated. However, this feature is only available when initiating a rehydration with the API.

Azure Blob and LocalStorage are not supported for rehydration.

If you use your own archive storage, then you can create a self-hosted rehydration.

Create a Rehydration

To create a rehydration, you must have an archived output and streaming output already configured in your account. To learn how to create an archived and streaming output, see Configure an Edge Delta Agent.

1. Click Data Pipeline - Rehydrations.
2. Click Create.
3. Select the tag for the agent configuration that handles the logs that you want to rehydrate. Based on the tag you select, the Source Type field will populate with options.
4. In From and To, select a date and time range of when data was collected to send to the streaming destination. This configuration represents the time frame for when data is collected, not when data is archived; however, the time difference between collection and archiving is a few minutes.
5. In Archive Source, select an existing archive output whose data you want to rehydrate and send to a streaming output.

Note For S3 Users: Before you created an S3 archive output, you first needed to have created an IAM user, and then attached a custom policy. That policy contained the 3 permissions, PutObject, GetObject, and ListBucket. If you created an S3 archive for rehydration purposes only, then at a minimum, your custom policy only needs to contain the GetObject permission. All other permissions are only required for archiving purposes. As a result, if you prefer, you can create 2 different S3 archive integrations with different custom policies.

6. In Destination, select an existing streaming output to send the archived data.
7. (Optional) In Keyword, enter a term to filter specific logs that will be sent. Only logs that match the entered keyword will be pushed to the streaming destination. This field is compatible with a valid Go regex. If you leave this field empty, then all logs will be pushed to the streaming destination.
8. In Host, select a host. This section is populated based on the agent configuration tag that you selected.
9. In Source Type, select a source type. This section is populated based on the agent configuration tag that you selected.
10. If your organization has a configured rehydration limit, then you must click Analyze.

You will not be able to click Create until you click Analyze. This action will display how much data will be pushed to the streaming destination. If your individual rehydration configuration is above the organization’s rehydration limit, then you will not be able to create the rehydration.

To troubleshoot, you can update the settings for the individual rehydration. Or, if you have the correct account permissions, you can update your organization’s rehydration settings. To do so, return to the Rehydrations page, click Settings, and then update the data limits.

If you click Analyze and you receive the following message, then please contact Edge Delta Support to troubleshoot: Failed to analyze rehydration data size. Failed to get rehydration analysis. Please have available your organization name and the time when the message appeared. 11. Click Create.

The app may take a few minutes to display your newly created rehydration. After you create a rehydration, you can click on the entry to view details.The Invalid Lines entry means that there may be empty files.

In the rehydrated data the name attribute is tagged as rehydrate and the host name is tagged as rehydrate-XXXX.

Rehydration API

You can use the Edge Delta API to trigger a rehydration. Contact Edge Delta for the agent API swagger.

POST /v1/orgs/{org_id}/rehydration

Rehydration Duplication

To prevent duplicate rehydrations, you set the exclude overlap parameter to true:

"exclude_overlap": true.

You must use the API if you want the option to prevent logs from being rehydrated more than once across multiple rehydrations. When this option is set to true, an exclude filter is automatically created to exclude logs that have been previously rehydrated. For example, if you rehydrate logs from a particular pod and then rehydrate logs from the entire namespace for the same point in time, the logs from the pod wont be included in the second rehydration.

Secrets

You can pass environment variables in for secret and key fields. Contact Edge Delta Support to enable this feature.

Configure Limits for Rehydrations

1. Click Data Pipeline - Rehydrations.
2. Click Settings.
3. Select false for Is On-Prem.
4. For Maximum Rehydration Size, enter a limit on the total size of the files to rehydrate, and then select the size type (byte, kilobyte, megabyte, gigabyte).
5. For Maximum Concurrent Rehydration Count, enter the maximum number of in-progress / non-completed rehydrations that can run at any time, per organization.
6. For Maximum Concurrent Rehydration Count Per User, enter the maximum number of in-progress / non-completed rehydrations that can run at any time, per user.
7. Click Save.

Whitelist the Edge Delta Rehydration Handlers

To ingest rehydrated logs, for example into your Elastic instance, you may need to whitelist the Edge Delta Rehydration Handler’s IP:

54.213.162.174/32, 34.215.114.77/32, 52.41.35.186/32