Deploying the Edge Delta Lambda Forwarder

Overview

The Lambda Forwarder is a standalone Lambda function that collects logs from CloudWatch log groups and forwards them to an Edge Delta Cloud pipeline. It supports CloudWatch logs from any AWS service, including Lambda, ECS, EC2, WAF, RDS, and others. See Serverless AWS Monitoring for an overview and comparison with the Lambda Extension.

Use the Forwarder when you need to ingest CloudWatch logs from any AWS service. If you need to monitor a Lambda function’s own logs directly (without CloudWatch), see the Lambda Extension instead.

Prerequisites

Before you begin, make sure you have:

• An Edge Delta account with access to create Cloud pipelines
• AWS CLI configured with credentials that have permission to deploy Lambda functions
• The name or ARN of the CloudWatch log group you want to forward
• Your AWS account ID

Create a Cloud pipeline with the CloudWatch Pack

Create a new Edge Delta Cloud pipeline:

1. Click Pipelines.
2. Click New Pipeline.
3. Select Cloud
4. Specify a name to identify the pipeline.
5. Select Compute Units based on your estimated traffic volume. This is the maximum bandwidth the agent can handle before signalling an error. The number of compute units used per hour counts towards your plan usage.
6. Click Deploy Cloud Pipeline.

After the pipeline is deployed, apply the CloudWatch Pack to configure the pipeline for CloudWatch log processing. The Pack handles JSON unrolling, attribute parsing, and timestamp extraction automatically.

Copy the HTTPS endpoint from the Cloud pipeline settings. You need this endpoint when deploying the forwarder.

Deploy the Lambda Forwarder

The Lambda Forwarder is built from the edgedelta-forwarder repository. Clone the repo and follow the steps below to build and deploy the function.

Build the forwarder

1. Clone the repository:

git clone https://github.com/edgedelta/edgedelta-forwarder.git
cd edgedelta-forwarder

2. Build the executable:

GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -tags lambda.norpc -o bootstrap main.go

3. Zip the binary:

zip edgedelta-forwarder.zip bootstrap

Create an IAM role

Create an IAM role for the forwarder Lambda function. The role must have permission to be invoked by CloudWatch Logs. If the role also has lambda:GetFunction permission, the forwarder fetches additional metadata such as runtime, architecture, memory size, and version from Lambda functions.

Create the Lambda function

Run the following command, replacing the placeholder values with your own:

aws lambda create-function \
    --function-name "my-ed-forwarder" \
    --runtime provided.al2 \
    --handler bootstrap \
    --role "arn:aws:iam::123456789012:role/my-forwarder-role" \
    --zip-file "fileb://edgedelta-forwarder.zip"

After the function is created, set the ED_ENDPOINT environment variable to the Cloud pipeline HTTPS endpoint you copied earlier:

aws lambda update-function-configuration \
    --function-name "my-ed-forwarder" \
    --environment "Variables={ED_ENDPOINT=https://your-pipeline-endpoint}"

Assign Lambda invoke permission

The forwarder needs permission to be invoked by the CloudWatch Logs service.

CLI:

Run the following command, replacing the placeholder values with your own:

aws lambda add-permission \
    --function-name "my-ed-forwarder" \
    --statement-id "AllowCloudWatchInvoke" \
    --principal "logs.amazonaws.com" \
    --action "lambda:InvokeFunction" \
    --source-arn "arn:aws:logs:us-west-2:123456789012:log-group:/aws/lambda/my-app:*" \
    --source-account "123456789012"

Console:

1. Open the Lambda console and select your forwarder function.
2. Click Configuration, then click Permissions.
3. Scroll to Resource-based policy statements and click Add permissions.
4. Select AWS service, choose Other for the service, and enter logs.amazonaws.com as the principal.
5. Set the action to lambda:InvokeFunction.
6. Enter the log group ARN as the source ARN.

Create a subscription filter

A subscription filter triggers the forwarder whenever new logs are written to the target log group.

Note: The forwarder must be in the same AWS region as the log group. If you forward logs from multiple regions (for example, WAF logs in us-east-1 and application logs in us-west-2), deploy a separate forwarder in each region.

CLI:

Run the following command, replacing the placeholder values with your own:

aws logs put-subscription-filter \
    --log-group-name "/aws/lambda/my-app" \
    --filter-name "edge-delta-forwarder" \
    --filter-pattern "" \
    --destination-arn "arn:aws:lambda:us-west-2:123456789012:function:my-ed-forwarder"

Set --filter-pattern to "" (empty string) to forward all logs, or specify a CloudWatch filter pattern to forward only matching logs.

Console:

1. Open the CloudWatch console and click Log groups.
2. Select the log group you want to forward.
3. Click the Subscription filters tab, then click Create Lambda subscription filter.
4. Select your forwarder Lambda function as the destination.
5. Specify a filter name. Leave the filter pattern empty to forward all logs.
6. Click Start streaming.

Environment variables

Configure the following environment variables on the forwarder Lambda function. Set ED_ENDPOINT during SAR deployment. You can configure additional variables in the Lambda console under Configuration > Environment variables.

Supported resources and tag fetching

The Forwarder supports CloudWatch logs from any AWS service. It automatically fetches resource tags for the following services:

• Lambda
• Fargate
• ECS
• EC2
• SNS

The following services have experimental tag-fetching support:

• RDS (MariaDB, MySQL, PostgreSQL)
• API Gateway (REST and HTTP)
• Step Functions (/aws/vendedlogs/states)
• DMS
• FSx for Windows
• AppSync
• CodeBuild
• Kinesis
• DocumentDB
• EKS
• Network Firewall
• Route 53
• VPC Flow Logs
• CloudTrail
• MSK
• Elasticsearch
• Transit Gateway
• Verified Access

Log group naming conventions for tag fetching

The Forwarder builds resource ARNs from log group names to fetch tags. The following table shows the expected log group format for each service:

Verify the deployment

After completing the setup, verify that logs flow from CloudWatch to Edge Delta:

1. Generate test traffic in your source application, or wait for natural log activity.
2. In the AWS Lambda console, open the forwarder function and check the Monitor tab. Verify that the function is being invoked.
3. Check the forwarder’s own CloudWatch log group (/aws/lambda/<forwarder_name>) for errors.
4. In the Edge Delta console, open the Cloud pipeline and confirm that logs appear in the live view.
5. Optionally, test the Cloud pipeline endpoint directly:

curl -X POST "https://your-pipeline-endpoint" \
  -H "Content-Type: application/json" \
  -d '{"message": "test"}'

A successful response returns HTTP 202.

Troubleshooting

Benchmarks

A single Lambda Forwarder sends batches of 200 to 300 KB. One Cloud pipeline processes approximately 10 MB per minute of forwarder traffic. For higher throughput, deploy additional Cloud pipelines or use the Lambda Extension, which sends larger batches (10 to 20 MB).