Deploying the Edge Delta Lambda Extension

Overview

Edge Delta’s AWS Lambda Extension is deployed as a layer within your Lambda Function. See Serverless AWS Monitoring. It is published in the AWS Serverless Application Repository and there are layers for ARM64 or AMD64. The Lambda extension is run on AWS as a part of the lambda execution and lambda function logs are sent to the extension. The extension in turn sends logs to an Edge Delta Cloud Fleet.

To configure Lambda monitoring using the Lambda Extension:

1. Create a Pipeline Configuration
2. Create an Edge Delta Cloud Fleet.
3. Deploy the Lambda extension.
4. Specify the Cloud Fleets endpoint.
5. Configure other environment variables as needed.

Create an Edge Delta Cloud Fleet

Create a new Edge Delta Cloud Fleet. Choose a new Pipeline configuration and ensure the HTTPS endpoint option is selected.

1. Click Pipelines.
2. Click New Fleet.
3. Select Cloud Fleet
4. Optionally, expand Advanced fleet configuration and choose a pipeline configuration to duplicate for the cloud fleet. If you don’t select one, a default configuration will be used.
5. Click Continue.
6. Specify a name to identify the Fleet.
7. Optionally, expand Advanced Settings and select Compute Units based on your estimated traffic volume. This is the maximum bandwidth the agent can handle before signalling an error. The number of compute units used per hour counts towards your plan usage.
8. Optionally, specify an agent version. The interface lists the current stable version (the latest version number) and most recent candidate version (containing rc). Choose the current stable version. If this configuration doesn’t work you can contact Edge Delta support to experiment with the candidate.
9. Optionally, select protocols for endpoints on the Cloud Fleet that your sources can push data to. The default is an HTTPS endpoint.
10. Click Deploy Cloud Fleet.

Copy the HTTPS endpoint from the Cloud Fleets table.

Modify the Pipeline

Modify the pipeline as follows:

1. Connect the HTTP source to a Parse JSON Attributes node.
2. Configure the Parse JSON Attributes node with the process_field item.body.
3. Connect Parse JSON Attributes to an Extract JSON Field node.
4. Configure Extract JSON Field to extract the whole message field path to assign to the body, and set keep_log_if_failed: true.
5. Connect Extract JSON Field to a Log Transform node.
6. Configure the Log Transform node to delete attributes.message (they are in body). Also delete attributes.timestamp.
7. Add the following Pipeline settings:

settings:
  multiline_max_byte_size: 11MB
  max_incomplete_line_buffer_size: 10MB

These values should match the max of lambda/cloudwatch so there are no OOM issues.

Configuration Example

version: v3

settings:
  tag: <configuration name>
  log:
    level: debug
  archive_flush_interval: 5m0s
  archive_max_byte_limit: 10MB
  multiline_max_byte_size: 11MB
  max_incomplete_line_buffer_size: 10MB

links:
- from: http_input_lambda_logs
  to: parse_json_attributes
- from: ed_component_health
  to: ed_health
- from: ed_node_health
  to: ed_health
- from: parse_json_attributes
  to: extract_json_field
- from: parse_json_attributes
  path: failure
  to: extract_json_field
- from: log_transform
  to: ed_archive_output
- from: extract_json_field
  to: log_transform
- from: extract_json_field
  path: failure
  to: log_transform

nodes:
- name: http_input_lambda_logs
  type: http_input
  port: 80
  read_timeout: 1m0s
- name: ed_component_health
  type: ed_component_health_input
- name: ed_agent_stats
  type: ed_agent_stats_input
- name: ed_node_health
  type: ed_node_health_input
- name: ed_pipeline_io_stats
  type: ed_pipeline_io_stats_input
- name: ed_health
  type: ed_health_output
- name: ed_archive_output
  type: ed_archive_output
- name: parse_json_attributes
  type: parse_json_attributes
  process_field: item.body
- name: log_transform
  type: log_transform
  transformations:
  - field_path: attributes.message
    operation: delete
  - field_path: attributes.timestamp
    operation: delete
- name: extract_json_field
  type: extract_json_field
  field_path: message
  keep_log_if_failed: true

Deploying the Lambda Extension

Get the ARN from the AWS Serverless Application Repository

1. Open AWS Serverless Application Repository and click Available Applications.
2. Search for EdgeDelta and select the extension (either ARM64 or AMD64).
3. Confirm the layer template such as the application name and click Deploy.

The extension can be deployed multiple times if necessary by providing different names. Deploying with an existing application name upgrades the existing deployment if an upgrade is available.

4. On AWS Lambda, expand Additional Resources and click Layers.
5. Click the Layer you just created (edgedelta-lambda-extension-arm64 or edgedelta-lambda-extension-amd64)
6. Copy the ARN of the Lambda Layer.

Add the Layer to your Lambda Function

Add the layer to your Lambda Function using the CLI or using the Console.

CLI: Replace <Function Name> and <ARN>

aws lambda update-function-configuration -function-name <Function Name> \
 --layers “<ARN>"

Console

1. Navigate to the Lambda Function you want to monitor.
2. In the Layers section, click Add a Layer.
3. Click Specify an ARN, paste the ARN you copied earlier and click Verify.
4. Click Add.

Specify the Endpoint Environment Variable

To configure Environment Variables for the Lambda function:

1. Open the Functions page of the Lambda console.
2. Choose a function.
3. Choose Configuration, then choose Environment variables.
4. Under Environment variables, choose Edit.
5. Choose Add environment variable.
6. Enter a key and value.

Push Destination Variables

Configure the Pusher Mode as http and specify the Cloud Fleet endpoint URL you copied earlier.

• PUSHER_MODE: Use http for hosted environments (default).
• ED_ENDPOINT: Specify the Edge Delta Cloud Fleet endpoint.

Configuration Variables

Configure any other environment variables you need:

• ED_FORWARD_LAMBDA_TAGS: If set to true, logs are enriched with lambda tags (in the field “lambda_tags”). Lambda function role needs to have the lambda:ListTags permission.
• ED_LAMBDA_LOG_TYPES: The variable defines which types of logs you want to get from the Lambda Function. Options are function,platform,extension. Default is function,platform.
• ED_PUSH_TIMEOUT_SEC: Push timeout is the duration between sending batches of logs (in seconds). The default is 5.
• ED_RETRY_INTERVAL_MS: The initial interval to wait until the next retry (in milliseconds). It is increased exponentially until the Edge Delta process is shut down. The default is 100.
• ED_BUFFER_SIZE_MB: Maximum buffer size to keep logs (in MB). The default is 20. Some logs may be dropped if the buffer exceeds this limit.
• ED_FLUSH_AT_NEXT_INVOKE: If set to true, logs are flushed at the start of the next invocation of the function. Normally, logs are flushed after the function execution finishes, which can add to the total running time. For lambda functions that are executed periodically, ED_FLUSH_AT_NEXT_INVOKE can be turned on to reduce costs.
• ED_PRINT_EXTENSION_LOGS: If set to true, the extension’s own logs are sent to CloudWatch.

Lambda can buffer logs and deliver them to the subscriber. You can configure this behavior in the subscription request by specifying the following optional fields:

• ED_LAMBDA_MAX_ITEMS: The maximum number of events to buffer in memory. Default: 1000. Minimum: 1000. Maximum: 10000. This is also the size of the channel that the Edge Delta http server writes into and which inputs consume.
• ED_LAMBDA_MAX_BYTES: The maximum size (in bytes) of the logs to buffer in memory. Default: 262144. Minimum: 262144. Maximum: 1048576.
• ED_LAMBDA_TIMEOUT_MS: The maximum time (in milliseconds) to buffer a batch. Default: 1000. Minimum: 25. Maximum: 30000.