Ingest Data from Amazon ECS with Edge Delta Agent

Deploy the Edge Delta agent as a sidecar container to collect and process telemetry from Amazon ECS Fargate workloads.

  11 minute read  

Overview

Amazon Elastic Container Service (ECS) with Fargate provides serverless compute for containers, eliminating the need to provision and manage servers. When running containerized workloads on ECS Fargate, monitoring and observability become critical for maintaining application health and performance. This guide demonstrates how to deploy the Edge Delta agent as a sidecar container alongside your application to collect and process telemetry directly at the source.

In this deployment pattern, the Edge Delta agent runs as a sidecar container within your ECS task definition. This architecture ensures that your application can send telemetry directly to the Edge Delta agent running in the same task, minimizing latency and network overhead. The agent processes data locally before forwarding it to Edge Delta SaaS or other configured destinations, providing full Edge Delta processing capabilities including AI-driven pattern detection, anomaly detection, and real-time data enrichment.

Note: For a truly agentless approach using AWS OpenTelemetry Collector, see Send ECS Data via AWS OpenTelemetry Collector.

When to Use This Integration

The Edge Delta agent sidecar deployment is optimal when you need full Edge Delta processing capabilities at the edge of your infrastructure. This approach excels at minimizing data egress costs through local processing and provides a streamlined single-vendor solution. Organizations requiring real-time pattern detection, anomaly detection, or processing of logs from non-OpenTelemetry sources will find this pattern particularly valuable. The agent’s ability to perform complex transformations and aggregations before data leaves your infrastructure can significantly reduce both costs and the volume of data sent to downstream systems.

Prerequisites

Before deploying this solution, you’ll need to ensure your environment is properly configured with the necessary AWS resources and Edge Delta account setup.

For AWS resources, you’ll need an ECS cluster configured for Fargate operation, along with a VPC containing private subnets where your ECS tasks will run. Proper IAM roles must be configured with the appropriate permissions for task execution and resource access. If you’re deploying web applications, you may also need an Application Load Balancer to distribute traffic across your containers.

Your Edge Delta setup requires:

  • An active Edge Delta account
  • A Pipeline ID and API token for authentication
  • Access to the Edge Delta web application for configuration

The deployment process relies on several tools:

  • AWS CLI configured with appropriate credentials
  • AWS SAM CLI or CloudFormation for infrastructure as code
  • Docker for building container images
  • Access to Amazon ECR or Docker Hub for container images

Architecture

ECS with Edge Delta Agent

The Edge Delta agent runs as a sidecar container in your ECS task, collecting logs from the application container’s filesystem, enriching them with ECS metadata, and processing them through a configurable pipeline before forwarding to destinations.

Step 1: Create the Edge Delta Configuration

First, create a configuration file for the Edge Delta agent that will run in your ECS environment. While many users build pipelines using the Edge Delta web interface, this example shows a complete configuration that can be deployed directly with your infrastructure as code.

This pipeline configuration implements a comprehensive telemetry processing strategy for ECS workloads. Application logs are collected from the container’s filesystem and enriched with ECS-specific metadata such as cluster name, service name, and task ARN. These enriched logs then flow through a processing sequence that performs three critical operations: parsing JSON-formatted log entries to extract structured fields, adding contextual information about the ECS environment, and masking sensitive data like credit card numbers, social security numbers, and email addresses to ensure compliance with data privacy regulations.

The processed telemetry is sent to two destinations simultaneously. The Edge Delta platform receives the enriched and sanitized data for real-time analysis, anomaly detection, and alerting. In parallel, raw logs are compressed and archived to S3, providing a cost-effective solution for long-term retention and compliance requirements. The configuration also includes self-monitoring capabilities, collecting metrics about the agent’s own performance and resource consumption to ensure the telemetry pipeline itself remains healthy and efficient.

version: v3

settings:
  tag: ecs-fargate-full
  log:
    level: info
  archive_flush_interval: 1m0s
  archive_max_byte_limit: 16MB

links:
- from: application_logs
  to: log_processing
- from: log_processing
  to: edge_delta_output
- from: log_processing
  to: s3_archive
- from: ed_system_stats
  to: edge_delta_output
- from: ed_self_telemetry
  to: edge_delta_output

nodes:
- name: application_logs
  type: file_input
  user_description: ECS Application Logs
  path: /var/log/app/*.log
- name: ed_system_stats
  type: ed_system_stats_input
  enable_granular_metrics: true
- name: ed_self_telemetry
  type: ed_self_telemetry_input
  enable_health_metrics: true
  enable_agent_stats_metrics: true
- name: log_processing
  type: sequence
  user_description: ECS Log Processing Pipeline
  processors:
  - type: ottl_transform
    metadata: '{"id":"parse-json","type":"parse-json","name":"Parse JSON"}'
    data_types:
    - log
    statements: |-
      merge_maps(attributes, ParseJSON(body), "upsert") where IsMap(attributes)
      set(attributes, ParseJSON(body)) where not IsMap(attributes)      
  - type: ottl_transform
    metadata: '{"id":"add-ecs-metadata","type":"add-field","name":"Add ECS Metadata"}'
    data_types:
    - log
    statements: |
      set(attributes["ecs.cluster"], "my-ecs-cluster")
      set(attributes["ecs.service"], "my-ecs-service")
      set(attributes["ecs.task_arn"], "arn:aws:ecs:us-west-2:123456789:task/abc123")
      set(attributes["environment"], "production")      
  - type: ottl_transform
    metadata: '{"id":"mask-sensitive","type":"mask","name":"Mask Sensitive Data"}'
    data_types:
    - log
    statements: |
      replace_pattern(body, "\\d{4}-\\d{4}-\\d{4}-\\d{4}", "[CARD]")
      replace_pattern(body, "\\d{3}-\\d{2}-\\d{4}", "[SSN]")
      replace_pattern(body, "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}", "[EMAIL]")      
- name: edge_delta_output
  type: ed_output
  user_description: Edge Delta Platform
- name: s3_archive
  type: s3_output
  user_description: S3 Archive Storage
  compression: gzip
  region: us-west-2
  bucket: my-ecs-logs-archive

Alternative Configuration Patterns

Depending on your application’s telemetry format and requirements, you may need to adapt the configuration to handle different data types. The following examples demonstrate common configuration patterns for various scenarios.

Metric Collection

If your application exposes Prometheus-style metrics, you can configure the Edge Delta agent to scrape these endpoints directly. This eliminates the need for a separate Prometheus server and allows you to correlate metrics with logs and traces in a single pipeline. The configuration below sets up a Prometheus scraper that collects metrics every 30 seconds and forwards them to Edge Delta.

- name: app_metrics
  type: prometheus_input
  endpoints:
    - http://localhost:9090/metrics
  interval: 30s

Trace Collection

For applications instrumented with OpenTelemetry, the Edge Delta agent can receive traces directly via the OTLP protocol. This configuration sets up an OTLP receiver that listens on port 4317 for gRPC connections. Your application’s OpenTelemetry SDK can send traces to this endpoint, allowing Edge Delta to correlate traces with logs and metrics for complete observability.

- name: otlp_traces
  type: otlp_input
  port: 4317
  protocols:
    - grpc

Step 2: Create the Task Definition

Create a CloudFormation template for your ECS task definition with the Edge Delta agent as a sidecar:

# ecs-task-definition.yaml
AWSTemplateFormatVersion: '2010-09-09'
Description: 'ECS Task Definition with Edge Delta Agent'

Parameters:
  EdgeDeltaApiKey:
    Type: String
    NoEcho: true
    Description: Edge Delta API Key
    
  EdgeDeltaPipelineId:
    Type: String
    Description: Edge Delta Pipeline ID
    
  ApplicationImage:
    Type: String
    Default: 'my-app:latest'
    Description: Docker image for your application

Resources:
  TaskExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: ecs-tasks.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
      Policies:
        - PolicyName: EdgeDeltaSecrets
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - secretsmanager:GetSecretValue
                Resource: !Ref EdgeDeltaApiKeySecret

  TaskRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: ecs-tasks.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: EdgeDeltaPermissions
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - s3:PutObject
                  - s3:PutObjectAcl
                Resource: 'arn:aws:s3:::my-ecs-logs-archive/*'
              - Effect: Allow
                Action:
                  - ecs:DescribeTasks
                  - ecs:DescribeServices
                  - ecs:DescribeClusters
                Resource: '*'

  EdgeDeltaApiKeySecret:
    Type: AWS::SecretsManager::Secret
    Properties:
      Name: !Sub 'edgedelta-api-key-${AWS::StackName}'
      SecretString: !Ref EdgeDeltaApiKey

  LogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub '/ecs/${AWS::StackName}'
      RetentionInDays: 7

  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Family: !Sub '${AWS::StackName}-task'
      RequiresCompatibilities:
        - FARGATE
      NetworkMode: awsvpc
      Memory: '2048'
      Cpu: '1024'
      ExecutionRoleArn: !GetAtt TaskExecutionRole.Arn
      TaskRoleArn: !GetAtt TaskRole.Arn
      ContainerDefinitions:
        # Application Container
        - Name: application
          Image: !Ref ApplicationImage
          Essential: true
          Memory: 1536
          PortMappings:
            - ContainerPort: 8080
              Protocol: tcp
          Environment:
            - Name: LOG_LEVEL
              Value: info
            - Name: OTEL_EXPORTER_OTLP_ENDPOINT
              Value: 'http://localhost:4317'
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-region: !Ref AWS::Region
              awslogs-group: !Ref LogGroup
              awslogs-stream-prefix: app
          DependsOn:
            - ContainerName: edgedelta-agent
              Condition: START

        # Edge Delta Agent Container
        - Name: edgedelta-agent
          Image: 'edgedelta/agent:latest'
          Essential: true
          Memory: 512
          Secrets:
            - Name: ED_API_KEY
              ValueFrom: !Ref EdgeDeltaApiKeySecret
          Environment:
            - Name: ED_PIPELINE_ID
              Value: !Ref EdgeDeltaPipelineId
            - Name: ED_AGENT_TYPE
              Value: 'ecs-fargate'
            - Name: ECS_CLUSTER
              Value: !Ref ECSCluster
            - Name: ECS_SERVICE
              Value: !Ref ECSService
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-region: !Ref AWS::Region
              awslogs-group: !Ref LogGroup
              awslogs-stream-prefix: edgedelta
          HealthCheck:
            Command:
              - CMD-SHELL
              - 'curl -f http://localhost:8088/health || exit 1'
            Interval: 30
            Timeout: 5
            Retries: 3
            StartPeriod: 60

Outputs:
  TaskDefinitionArn:
    Description: ARN of the task definition
    Value: !Ref TaskDefinition

This CloudFormation template defines a complete ECS task definition that deploys your application alongside the Edge Delta agent as a sidecar container. The task definition includes proper IAM roles for both execution and runtime permissions, ensuring the Edge Delta agent can access necessary AWS services like S3 for log archival and Secrets Manager for secure API key storage. The template configures both containers with appropriate resource allocations, with 1536 MB of memory for the application and 512 MB for the Edge Delta agent. The containers are configured with a dependency relationship ensuring the Edge Delta agent starts before the application, and both containers log to CloudWatch for operational visibility. Environment variables are passed to configure the Edge Delta agent with your pipeline ID and to provide ECS metadata for enrichment.

Step 3: Deploy the ECS Service

Create a service definition that uses the task definition:

# ecs-service.yaml
AWSTemplateFormatVersion: '2010-09-09'
Description: 'ECS Service with Edge Delta monitoring'

Parameters:
  TaskDefinitionArn:
    Type: String
    Description: ARN of the task definition
    
  ClusterName:
    Type: String
    Description: Name of the ECS cluster
    
  VpcId:
    Type: AWS::EC2::VPC::Id
    Description: VPC ID
    
  SubnetIds:
    Type: List<AWS::EC2::Subnet::Id>
    Description: Private subnet IDs

Resources:
  SecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Security group for ECS service
      VpcId: !Ref VpcId
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 8080
          ToPort: 8080
          SourceSecurityGroupId: !Ref ALBSecurityGroup
      SecurityGroupEgress:
        - IpProtocol: -1
          CidrIp: 0.0.0.0/0

  Service:
    Type: AWS::ECS::Service
    Properties:
      ServiceName: !Sub '${AWS::StackName}-service'
      Cluster: !Ref ClusterName
      TaskDefinition: !Ref TaskDefinitionArn
      LaunchType: FARGATE
      DesiredCount: 2
      NetworkConfiguration:
        AwsvpcConfiguration:
          SecurityGroups:
            - !Ref SecurityGroup
          Subnets: !Ref SubnetIds
          AssignPublicIp: DISABLED
      LoadBalancers:
        - ContainerName: application
          ContainerPort: 8080
          TargetGroupArn: !Ref TargetGroup
      HealthCheckGracePeriodSeconds: 60
      DeploymentConfiguration:
        MaximumPercent: 200
        MinimumHealthyPercent: 100
        DeploymentCircuitBreaker:
          Enable: true
          Rollback: true

  TargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      Port: 8080
      Protocol: HTTP
      TargetType: ip
      VpcId: !Ref VpcId
      HealthCheckEnabled: true
      HealthCheckPath: /health
      HealthCheckProtocol: HTTP
      HealthCheckIntervalSeconds: 30
      HealthCheckTimeoutSeconds: 5
      HealthyThresholdCount: 2
      UnhealthyThresholdCount: 3

Outputs:
  ServiceName:
    Description: Name of the ECS service
    Value: !GetAtt Service.Name

This service definition creates a highly available ECS service running on Fargate with automatic load balancing and health checks. The service deploys two instances of your task definition for redundancy, distributing them across multiple availability zones via the specified private subnets. The configuration includes a security group that allows inbound traffic from the Application Load Balancer on port 8080, while permitting all outbound connections for the Edge Delta agent to reach external endpoints. The service integrates with an Application Load Balancer target group that performs health checks every 30 seconds, ensuring only healthy containers receive traffic. Advanced deployment settings enable circuit breaker protection with automatic rollback if deployments fail, maintaining service availability during updates.

Step 4: Deploy Using AWS SAM

Deploy using SAM CLI:

# Build the application
sam build

# Deploy to AWS
sam deploy \
  --stack-name my-ecs-edgedelta-stack \
  --parameter-overrides \
    EdgeDeltaApiKey=YOUR_API_KEY \
    EdgeDeltaPipelineId=YOUR_PIPELINE_ID \
  --capabilities CAPABILITY_IAM \
  --region us-west-2

Step 5: Verify Deployment

  1. Check ECS Service Status:
aws ecs describe-services \
  --cluster my-cluster \
  --services my-ecs-edgedelta-stack-service \
  --region us-west-2
  1. View Container Logs:
aws logs tail /ecs/my-ecs-edgedelta-stack \
  --follow \
  --region us-west-2
  1. Verify in Edge Delta Web App:
    • Log into Edge Delta
    • Navigate to Pipelines
    • Select your pipeline
    • Verify data is flowing from your ECS tasks

Performance Considerations

Resource Allocation

Proper resource allocation is crucial for optimal performance of your telemetry pipeline. The Edge Delta agent requires a minimum of 256 CPU units and 512 MB of memory, though these values should be adjusted based on your actual data volumes and processing requirements. Your application containers should be allocated resources based on their specific requirements, keeping in mind that telemetry collection may add a small overhead to their operation.

Network Configuration

Network configuration plays a vital role in both performance and cost optimization. Implementing VPC endpoints can significantly minimize data transfer costs by keeping traffic within the AWS network. Security groups must be carefully configured to allow communication between containers while maintaining the principle of least privilege. For enhanced security and reliability, consider enabling AWS PrivateLink for connectivity to Edge Delta, which provides a private connection without exposing traffic to the public internet.

Data Processing Limits

Understanding the data processing limits helps in capacity planning and scaling decisions. The Edge Delta agent can process up to 10GB per minute per instance, making it suitable for high-volume applications. Cloud Pipelines support up to 100MB per minute with optimal batching, though this can be increased by deploying multiple pipeline instances or working with Edge Delta support to adjust limits based on your needs.

Troubleshooting

Agent Not Starting

Check the agent logs for configuration errors:

aws logs get-log-events \
  --log-group-name /ecs/my-stack \
  --log-stream-name edgedelta/task-id \
  --region us-west-2

No Data in Edge Delta

  1. Verify the API key is correct
  2. Check network connectivity from ECS tasks
  3. Ensure security groups allow outbound HTTPS traffic
  4. Verify the pipeline ID matches your configuration

High Memory Usage

Adjust the agent’s memory limits and buffer sizes in your configuration:

settings:
  memory_limit: 400MB
  buffer_size: 100MB

Best Practices

  1. Resource Allocation: Allocate at least 256 CPU units and 512MB memory to the Edge Delta agent
  2. Log Rotation: Configure log rotation to prevent disk space issues
  3. Monitoring: Set up CloudWatch alarms for agent health and resource usage
  4. Security: Store API keys in AWS Secrets Manager
  5. Networking: Use VPC endpoints to reduce data transfer costs
  6. Scaling: Configure auto-scaling based on your application’s load patterns

Alternative Approach

For organizations preferring a truly agentless solution or those already using AWS OpenTelemetry Collector, see Send ECS Data via AWS OpenTelemetry Collector which demonstrates how to use ADOT to forward telemetry to Edge Delta Cloud Pipelines without deploying the Edge Delta agent.