Rehydration of Self Hosted Archives
Rehydrate logs from Self hosted archive storage in the Edge Delta web application.
5 minute read
Rehydration is the process of pushing already-archived data to a streaming destination, such as Splunk, Elasticsearch, etc. If you want to prevent duplicated logs, you can configure the Edge Delta rehydrator to check for past rehydrations and automatically create a filter to exclude logs that have already been rehydrated. However, this feature is only available when initiating a rehydration with the API.
Azure Blob and LocalStorage are not supported for rehydration.
If you use your own archive storage, then you can create a self-hosted rehydration.
To create a rehydration, you must have an archived output and streaming output already configured in your account. To learn how to create an archived and streaming output, see Configure an Edge Delta Agent.
Note For S3 Users: Before you created an S3 archive output, you first needed to have created an IAM user, and then attached a custom policy. That policy contained the 3 permissions, PutObject, GetObject, and ListBucket. If you created an S3 archive for rehydration purposes only, then at a minimum, your custom policy only needs to contain the GetObject permission. All other permissions are only required for archiving purposes. As a result, if you prefer, you can create 2 different S3 archive integrations with different custom policies.
You will not be able to click Create until you click Analyze. This action will display how much data will be pushed to the streaming destination. If your individual rehydration configuration is above the organization’s rehydration limit, then you will not be able to create the rehydration.
To troubleshoot, you can update the settings for the individual rehydration. Or, if you have the correct account permissions, you can update your organization’s rehydration settings. To do so, return to the Rehydrations page, click Settings, and then update the data limits.
If you click Analyze and you receive the following message, then please contact Edge Delta Support to troubleshoot: Failed to analyze rehydration data size. Failed to get rehydration analysis.
Please have available your organization name and the time when the message appeared.
11. Click Create.
The app may take a few minutes to display your newly created rehydration. After you create a rehydration, you can click on the entry to view details.The Invalid Lines entry means that there may be empty files.
In the rehydrated data the name attribute is tagged as rehydrate and the host name is tagged as rehydrate-XXXX.
You can use the Edge Delta API to trigger a rehydration. Contact Edge Delta for the agent API swagger.
POST /v1/orgs/{org_id}/rehydration
To prevent duplicate rehydrations, you set the exclude overlap parameter to true:
"exclude_overlap": true.
You must use the API if you want the option to prevent logs from being rehydrated more than once across multiple rehydrations. When this option is set to true, an exclude filter is automatically created to exclude logs that have been previously rehydrated. For example, if you rehydrate logs from a particular pod and then rehydrate logs from the entire namespace for the same point in time, the logs from the pod wont be included in the second rehydration.
You can pass environment variables in for secret and key fields. Contact Edge Delta Support to enable this feature.
To ingest rehydrated logs, for example into your Elastic instance, you may need to whitelist the Edge Delta Rehydration Handler’s IP:
54.213.162.174/32, 34.215.114.77/32, 52.41.35.186/32
Rehydrate logs from Self hosted archive storage in the Edge Delta web application.
Troubleshooting rehydrations in the Edge Delta web application.