Deploying the Edge Delta Lambda Forwarder

Deploy the Edge Delta Lambda Forwarder.

  5 minute read  

Overview

The Lambda Forwarder is a Lambda Function that collects AWS Lambda logs from Cloudwatch Log groups. See Serverless AWS Monitoring.

Note: the Edge Delta Lambda Extension is a preferred solution.

Create an Edge Delta Cloud Pipeline

Create a new Edge Delta Cloud pipeline.

  1. Click Pipelines.
  2. Click New Pipeline.
  3. Select Cloud
  4. Specify a name to identify the pipeline.
  5. Select Compute Units based on your estimated traffic volume. This is the maximum bandwidth the agent can handle before signalling an error. The number of compute units used per hour counts towards your plan usage.
  6. Click Deploy Cloud Pipeline.

Copy the HTTPS endpoint from the Cloud pipelines settings section.

Modify the Pipeline Configuration

Configure the pipeline for the Cloud pipeline as follows:

  1. Feed traffic from the HTTP source to a JSON Unroll node. This node will transform structured JSON logs by breaking down nested JSON array objects into individual log entries.
  • Set the new_field_name parameter to logEvents. This specifies the field under which the contents of each unrolled log will be placed in the resulting output. Each portion of the original array becomes a new log with a top-level field named logEvents.
  • Set the json_field_path also to logEvents. This parameter locates the exact JSON array field that needs to be processed - the incoming JSON object has a key logEvents whose array value will be unrolled.
  1. Connect the JSON Unroll node to a Parse JSON node. This node will parse each structured JSON log’s item["body"] into attributes.
  2. Connect the Parse JSON node to a Parse Timestamp. This node will use the log’s original timestamp as the data item timestamp, and also delete the logEvent attribute. Create an upsert operation group with the field path of item["timestamp"]. This field should be updated with a value from json(item["body"]).logEvent.timestamp. Then create a delete operation group with a delete operation for the item["attributes"]["logEvent"] field path.
  3. Connect the Log Transform node to an [Extract JSON Processor. This node will extract the content of the message field from within a logEvent JSON object and use it as the log’s body field. Specify the field_path as logEvent.message.

Deploying the Lambda Function

Get the ARN from the AWS Serverless Application Repository

  1. Open AWS Serverless Application Repository and click Available Applications.
  2. Select Show apps that create custom IAM roles or resource policies.
  3. Search for EdgeDelta and select the forwarder (either ARM64 or AMD64).
  4. Confirm the function template such as the application name
  5. Enter the HTTPS endpoint for the Cloud pipeline that you copied earlier.
  6. Click I acknowledge that this app creates custom IAM roles and resource policies.
  7. Click Deploy.

The function can be deployed multiple times if necessary by providing different names. Deploying with an existing application name upgrades the existing deployment if an upgrade is available.

The following environment variables can be specified in the form or specified in using the Lambda console:

  • ED_ENDPOINT: Edge Delta Cloud pipeline endpoint. (Required)
  • ED_FORWARD_FORWARDER_TAGS: If set to true, the forwarder Lambda’s own tags are fetched. This requires tag:GetResources and lambda:GetFunctionConfiguration permissions.
  • ED_FORWARD_LOG_GROUP_TAGS: If set to true, log group tags are fetched. Requires tag:GetResources permission.
  • ED_FORWARD_SOURCE_TAGS: If set to true, source log group’s tags are fetched. Forwarder tries to build an ARN of the source by using log group’s name. This requires tag:GetResources permission. If the source is lambda it also requires lambda:GetFunctionConfiguration permission and this only works if the log group name is in the correct format (i.e. /aws/lambda/<lambda_name>).
  • ED_PUSH_TIMEOUT_SEC: Push timeout is the total waiting duration between send batches of logs (in seconds). Default is 10.
  • ED_RETRY_INTERVAL_MS: RetryInterval is the initial interval to wait until the next retry (in milliseconds). It is increased exponentially until the Edge Delta process is shut down. Default is 100.

To configure Environment Variables for the Lambda function:

  1. Open the Functions page of the Lambda console.
  2. Choose a function.
  3. Choose Configuration, then choose Environment variables.
  4. Under Environment variables, choose Edit.
  5. Choose Add environment variable.
  6. Enter a key and value.

Assign Permissions

Assign Lambda Invoke Permission To AWS Logs Service in the CLI:

aws lambda add-permission \
    --function-name “<name_of_the_forwarder_lambda>” \
    --statement-id “<sid_for_policy>” \
    --principal “logs.amazonaws.com” \
    --action “lambda:InvokeFunction” \
    --source-arn “<arn_of_the_log_group_you_want_to_consume>” \
    --source-account ”<aws_account_id>” \

Subscribe the function to CloudWatch

Set up CloudWatch Logs subscription in the CLI:

aws logs put-subscription-filter \
    --log-group-name “<the_log_group_you_want_to_consume>” \
    --filter-name “<name_of_the_filter_just_for_display_purpose>” \
    --filter-pattern “<filter_pattern_for_logs_if_needed_to_send_logs_matching_with_pattern>” \
    --destination-arn “<arn_of_the_forwarder_lambda>”

Tag Fetching

Building a source ARN from log groups and log streams is not straightforward in AWS. Moreover, you can’t change log groups and stream names. The Forwarder is able to fetch lambda tags and SNS without any effort. However, Sagemaker log group and streams differ too much. In this instance the Forwarder is only able to fetch tags of the source of the log group and stream.

Building ARNs

The Forwarder builds ARNs with to these conventions:

  • ECS: The log configuration is defined in the Task Definition. A typical task definition is as follows:
        "logConfiguration": {
            "logDriver": "awslogs",
            "options": {
                "awslogs-create-group": "true",
                "awslogs-group": "/ecs/test-cluster/-test-service",
                "awslogs-region": "us-west-2",
                "awslogs-stream-prefix": "ecs"
            }
        }

The Forwarder can have two log group conventions:

  • /ecs/{cluster_name}: Forwarder fetches ECS cluster tags

  • /ecs/{cluster_name}/{service_name}: Forwarder fetches ECS cluster and service tags.

  • EC2: Typically you install Cloudwatch Agent to the EC2 instance to send EC2 logs to Cloudwatch. You can specify a log group name and a stream name in the Cloudwatch Agent configuration. The Forwarder expects the following log group name to fetch tags of the EC2 instance: /ec2/instance/{instanceID} Additionally, the Forwarder can fetch VPC logs and VPC log groups can be specified. The Forwarder expects the following log group to be able to fetch tags of the VPC: /ec2/vpc/{vpcID}

  • Other Services: For other services, The Forwarder assumes the following format and tries to build the ARNs:

/aws/<service>/<resource_name> or /aws/<service>/<resource_type>/<resource_name>/...

ARNs: arn:aws:{service}:{region}:{account}:{resource_name} or arn:aws:{service}:{region}:{account}:{resource_name}/{resource_type}.....

Benchmarks

The Edge Delta Forwarder can process 10MB per minute continuously with 1 Cloud pipeline.