Edge Delta Serverless AWS Monitoring

Overview

Modern workloads in AWS are typically run partially or fully on AWS Lambda or AWS Fargate with ECS to benefit from serverless compute solutions. However Monitoring such solutions proves costly and inefficient. For example, many solutions use Lambda native integration with Amazon CloudWatch Logs, which in turn integrates with the customer’s Centralized Observability Platform as follows:

While it is easy to leverage native integrations, this solution becomes less suitable as the adoption of serverless architecture with event-driven data grows. In particular, ingesting high data volume into the centralized observability platform becomes expensive. In addition, observability is delayed as raw data needs to first be indexed centrally.

While the Edge Delta agent works well in a distributed host architecture it shouldn’t be used for external push-based sources, such as consuming Lambda function logs over HTTPS. In addition, regulatory requirements may preclude EC2 deployment of the Edge Delta agent in a distributed architecture pattern.

Edge Delta offers two solutions to these scenarios:

• Lambda Extension with a Hosted Agent (preferred solution)
• Lambda Forwarder with a Hosted Agent

Hosted Agents

Hosted agents are hosted in cloud infrastructure owned and managed by Edge Delta. See Hosted Agents. The hosted agent can optionally pipe optimized data to a downstream Centralized Observability Platform.

In the following solutions Hosted Agents were tested with an instance limit of 60 lambda instances producing 6K lines or 105MB logs per minute. The default collective limit is 8 hosted agents.

Log batch size impacts Hosted Agent performance and therefore the solution selection. A single Hosted Agent processes 100MB per minute if logs are batched and sent in big batches of 10 to 20MB. However if batches are smaller (~1MB) it processes about 30 MB per minute. The Lambda Forwarder typically sends 200KB-300KB per batch resulting in 10MB per minute being processed by the hosted agent.

Lambda Extension

The preferred solution is to deploy the Edge Delta Lambda Extension with an Edge Delta Hosted Agent.

The Lambda Extension is deployed as a layer within your Lambda Function. It batches and sends logs to the hosted agent at the end of the function execution. In addition, it supports fetching AWS Tags from the lambda function and it collects some environment values such as runtime and architecture.

The Lambda Extension has been tested up to 20MB of total logs per function instance execution and it is fully scalable.

See Deploying the Lambda Extension for detailed instructions.

Lambda Forwarder

The Lambda Forwarder is a Lambda Function that collects AWS Lambda logs from Cloudwatch Log groups.

It batches and sends logs to the hosted agent in the configured log groups.

The Lambda Forwarder supports fetching AWS Tags for the original resource and it collects resources in addition to AWS Lambda:

• Fargate
• ECS
• EC2
• SNS
• Sagemaker

The following resources are experimental:

• /aws/rds/mariadb,
• /aws/rds/mysql,
• /aws/rds/mariadb/postgresql
• api-gateway
• /aws/api-gateway
• /aws/http-api
• /aws/vendedlogs/states
• dms-tasks
• sns/
• /aws/fsx/windows
• /aws/appsync/
• /aws/lambda
• /aws/codebuild
• /aws/kinesis
• /aws/docdb
• /aws/eks
• network-firewall
• route53
• vpc
• cloudtrail
• msk
• elasticsearch
• transitgateway
• verified-access

The Lambda Forwarder is fully scalable with respect to an instance or collective limit. Forwarder instances are invoked automatically by the cloudwatch queue as needed and they have been tested with 128 MB memory allocated.

See Deploying the Lambda Forwarder for detailed instructions.