Deploying the Edge Delta Lambda Extension

Overview

Edge Delta’s AWS Lambda Extension is deployed as a layer within your Lambda Function. See Serverless AWS Monitoring. It is published in the AWS Serverless Application Repository and there are layers for ARM64 or AMD64. The Lambda extension is run on AWS as a part of the lambda execution and lambda function logs are sent to the extension. The extension in turn sends logs to an Edge Delta Hosted Agent.

To configure Lambda monitoring using the Lambda Extension:

1. Create an Agent Configuration.
2. Create an Edge Delta Hosted Agent.
3. Deploy the Lambda extension.
4. Specify the hosted agents endpoint.
5. Configure other environment variables as needed.

Create an Agent Configuration

Create an agent configuration that the hosted agent will use to ingest logs sent from lambda extension, identify function resources, and populate lambda tags.

1. Click Pipelines - Legacy Pipelines.
2. Click Create Configuration.
3. Click Continue.
4. Select Helm and click Continue.
5. Specify a name for the hosted agent configuration and click Create Configuration.
6. Add a File Input with the following path: /var/captured_requests/body_*.json
7. Connect the File Input to a Parse JSON Attributes node.
8. Configure the Parse JSON Attributes node with the process_field item.body.
9. Connect Parse JSON Attributes to an Extract JSON Field node.
10. Configure Extract JSON Field to extract the whole message field path to assign to the body, and set keep_log_if_failed: true.
11. Connect Extract JSON Field to a Log Transform node.
12. Configure the Log Transform node to delete attributes.message (they are in body). Also delete attributes.timestamp.
13. Click Pipelines - Legacy Pipelines.
14. Click the Actions column for the new agent row you just created and select Edit.
15. Add the following agent settings:

settings:
  multiline_max_byte_size: 11MB
  max_incomplete_line_buffer_size: 10MB

These values should match the max of lambda/cloudwatch so there are no OOM issues.

v3 Configuration Example

version: v3

settings:
  tag: <configuration name>
  log:
    level: debug
  archive_flush_interval: 5m0s
  archive_max_byte_limit: 10MB
  multiline_max_byte_size: 11MB
  max_incomplete_line_buffer_size: 10MB

links:
- from: file_input_lambda_logs
  to: parse_json_attributes
- from: ed_component_health
  to: ed_health
- from: ed_node_health
  to: ed_health
- from: parse_json_attributes
  to: extract_json_field
- from: parse_json_attributes
  path: failure
  to: extract_json_field
- from: log_transform
  to: ed_archive_output
- from: extract_json_field
  to: log_transform
- from: extract_json_field
  path: failure
  to: log_transform

nodes:
- name: file_input_lambda_logs
  type: file_input
  path: /var/captured_requests/body_*.json
- name: ed_component_health
  type: ed_component_health_input
- name: ed_agent_stats
  type: ed_agent_stats_input
- name: ed_node_health
  type: ed_node_health_input
- name: ed_pipeline_io_stats
  type: ed_pipeline_io_stats_input
- name: ed_health
  type: ed_health_output
- name: ed_archive_output
  type: ed_archive_output
- name: parse_json_attributes
  type: parse_json_attributes
  process_field: item.body
- name: log_transform
  type: log_transform
  transformations:
  - field_path: attributes.message
    operation: delete
  - field_path: attributes.timestamp
    operation: delete
- name: extract_json_field
  type: extract_json_field
  field_path: message
  keep_log_if_failed: true

Create an Edge Delta Hosted Agent

Create a new Edge Delta Hosted agent. Select the configuration you created and select the HTTPS endpoint option.

1. Click Pipelines in the Edge Delta Web App and select Hosted Agents.
2. Click + Create.
3. Enter a descriptive name for the hosted agent.
4. Select an agent version. The interface lists the current stable version (the lower version number) and most recent candidate version (the higher version number). Choose the current stable version. If this configuration doesn’t work you can contact Edge Delta support to experiment with the candidate.
5. Select a public cloud provider. If your preferred cloud provider is not listed, contact Edge Delta support.
6. Select the region where the hosted agent will be hosted.
7. For Config ID, there are 2 options:

• select Generate one for me if you want to create a new agent configuration.

Note: this will generate an agent with a v2 configuration.

• If you have already configured an agent and you want to reuse that configuration, select the configuration ID from the drop down.

Note: Specify a v3 configuration if you want to configure the hosted agent using Visual Pipelines.

8. Select the HTTPS checkbox if you want your data source to push data to a secure HTTP endpoint on the hosted agent.
9. Click Create.

Copy the HTTPS endpoint generated for the agent from the Hosted Agents table.

Deploying the Lambda Extension

Get the ARN from the AWS Serverless Application Repository

1. Open AWS Serverless Application Repository and click Available Applications.
2. Search for EdgeDelta and select the extension (either ARM64 or AMD64).
3. Confirm the layer template such as the application name and click Deploy.

The extension can be deployed multiple times if necessary by providing different names. Deploying with an existing application name upgrades the existing deployment if an upgrade is available.

4. On AWS Lambda, expand Additional Resources and click Layers.
5. Click the Layer you just created (edgedelta-lambda-extension-arm64 or edgedelta-lambda-extension-amd64)
6. Copy the ARN of the Lambda Layer.

Add the Layer to your Lambda Function

Add the layer to your Lambda Function using the CLI or using the Console.

CLI: Replace <Function Name> and <ARN>

aws lambda update-function-configuration -function-name <Function Name> \
 --layers “<ARN>"

Console

1. Navigate to the Lambda Function you want to monitor.
2. In the Layers section, click Add a Layer.
3. Click Specify an ARN, paste the ARN you copied earlier and click Verify.
4. Click Add.

Specify the Endpoint Environment Variable

To configure Environment Variables for the Lambda function:

1. Open the Functions page of the Lambda console.
2. Choose a function.
3. Choose Configuration, then choose Environment variables.
4. Under Environment variables, choose Edit.
5. Choose Add environment variable.
6. Enter a key and value.

Push Destination Variables

Configure the Pusher Mode as http and specify the hosted agent endpoint URL you copied earlier.

• PUSHER_MODE: Use http for hosted environments (default).
• ED_ENDPOINT: Specify the Edge Delta Hosted Agent endpoint.

Configuration Variables

Configure any other environment variables you need:

• ED_FORWARD_LAMBDA_TAGS: If set to true, logs are enriched with lambda tags (in the field “lambda_tags”). Lambda function role needs to have the lambda:ListTags permission.
• ED_LAMBDA_LOG_TYPES: The variable defines which types of logs you want to get from the Lambda Function. Options are function,platform,extension. Default is function,platform.
• ED_PUSH_TIMEOUT_SEC: Push timeout is the duration between sending batches of logs (in seconds). The default is 5.
• ED_RETRY_INTERVAL_MS: The initial interval to wait until the next retry (in milliseconds). It is increased exponentially until the Edge Delta process is shut down. The default is 100.
• ED_BUFFER_SIZE_MB: Maximum buffer size to keep logs (in MB). The default is 20. Some logs may be dropped if the buffer exceeds this limit.
• ED_FLUSH_AT_NEXT_INVOKE: If set to true, logs are flushed at the start of the next invocation of the function. Normally, logs are flushed after the function execution finishes, which can add to the total running time. For lambda functions that are executed periodically, ED_FLUSH_AT_NEXT_INVOKE can be turned on to reduce costs.
• ED_PRINT_EXTENSION_LOGS: If set to true, the extension’s own logs are sent to CloudWatch.

Lambda can buffer logs and deliver them to the subscriber. You can configure this behavior in the subscription request by specifying the following optional fields:

• ED_LAMBDA_MAX_ITEMS: The maximum number of events to buffer in memory. Default: 1000. Minimum: 1000. Maximum: 10000. This is also the size of the channel that the Edge Delta http server writes into and which inputs consume.
• ED_LAMBDA_MAX_BYTES: The maximum size (in bytes) of the logs to buffer in memory. Default: 262144. Minimum: 262144. Maximum: 1048576.
• ED_LAMBDA_TIMEOUT_MS: The maximum time (in milliseconds) to buffer a batch. Default: 1000. Minimum: 25. Maximum: 30000.