Send ECS Data via AWS OpenTelemetry Collector

Use AWS Distro for OpenTelemetry Collector to send Amazon ECS Fargate telemetry to Edge Delta Cloud Pipelines without deploying Edge Delta agents.

13 minute read

Overview

Amazon Elastic Container Service (ECS) with Fargate provides serverless compute for containers, eliminating the need to provision and manage servers. This guide demonstrates how to deploy the AWS Distro for OpenTelemetry (ADOT) Collector on Amazon ECS Fargate to collect telemetry from your applications and forward it to Edge Delta Cloud Pipelines. This agentless approach leverages AWS-native tooling and OpenTelemetry standards for vendor-neutral instrumentation.

The AWS OpenTelemetry Collector integration pattern leverages the AWS Distro for OpenTelemetry Collector (ADOT), which can be deployed either as a sidecar container or as a standalone service within your ECS cluster. Applications instrumented with the OpenTelemetry SDK send their telemetry data to ADOT, which then forwards it to Edge Delta Cloud Pipelines using the OTLP/gRPC protocol. This approach aligns with AWS-native tooling and embraces OpenTelemetry standards, making it an ideal choice for organizations already invested in the AWS ecosystem or those seeking vendor-neutral instrumentation.

Note: For organizations requiring full Edge Delta processing capabilities at the edge, see Ingest Data from Amazon ECS with Edge Delta Agent which deploys the Edge Delta agent as a sidecar container.

When to Use This Integration

The AWS OpenTelemetry Collector integration is the preferred choice for organizations with existing ADOT deployments or those following AWS-standardized architectures. This pattern embraces vendor-neutral OpenTelemetry standards and seamlessly integrates with AWS-native services like X-Ray and CloudWatch. If your requirements include sending data to multiple observability backends simultaneously, or if your organization mandates the use of AWS-managed collectors for compliance or standardization reasons, this approach provides the flexibility and compatibility you need while still benefiting from Edge Delta’s powerful cloud-based processing capabilities.

Architecture

In this deployment pattern:

ADOT Collector runs as a sidecar container or standalone service
Applications instrumented with OpenTelemetry SDK send telemetry to ADOT
ADOT processes and exports data to Edge Delta Cloud Pipelines via OTLP/gRPC
Edge Delta Cloud Pipelines process and route data to your observability platforms

Prerequisites

Before deploying this solution, you’ll need to ensure your environment is properly configured with the necessary AWS resources, Edge Delta Cloud Pipeline setup, and deployment tools.

Edge Delta Cloud Pipeline

Log into Edge Delta web application
Create a Cloud Pipeline with OTLP input
Note your OTLP endpoint (format: xxxxx-grpc-region.aws.edgedelta.com:443)

AWS Resources

For AWS resources, you’ll need an ECS cluster configured for Fargate operation, along with a VPC containing private subnets where your ECS tasks will run. You’ll also need a Service Discovery namespace (AWS Cloud Map) for service-to-service communication and proper IAM roles with appropriate permissions for task execution and resource access.

Application Setup

Your applications need to be instrumented with the OpenTelemetry SDK. For Java applications, you can use zero-code instrumentation with the OTEL Java agent. Other languages have their own auto-instrumentation options or require manual SDK integration.

Deployment Tools

The deployment process relies on several tools:

AWS CLI configured with appropriate credentials
AWS SAM CLI or CloudFormation for infrastructure as code
Docker for building container images with OpenTelemetry instrumentation

Step 1: Configure ADOT Collector

Create the ADOT Collector configuration that will forward data to Edge Delta:

# Store this configuration in AWS Systems Manager Parameter Store
# Parameter name: /aot/config

extensions:
  health_check:
  pprof:
    endpoint: 0.0.0.0:1777

receivers:
  # OTLP receiver for application telemetry
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:55681
  
  # AWS X-Ray receiver for trace data
  awsxray:
    endpoint: 0.0.0.0:2000
    transport: udp
  
  # StatsD receiver for metrics
  statsd:
    endpoint: 0.0.0.0:8125
    aggregation_interval: 60s
  
  # ECS container metrics
  awsecscontainermetrics:
    collection_interval: 60s

processors:
  # Filter ECS metrics to reduce volume
  filter:
    metrics:
      include:
        match_type: strict
        metric_names:
          - ecs.task.memory.utilized
          - ecs.task.memory.reserved
          - ecs.task.cpu.utilized
          - ecs.task.cpu.reserved
          - ecs.task.network.rate.rx
          - ecs.task.network.rate.tx
          - ecs.task.storage.read_bytes
          - ecs.task.storage.write_bytes
  
  # Add resource attributes
  resource:
    attributes:
      - key: ecs.cluster
        value: ${ECS_CLUSTER}
      - key: ecs.service
        value: ${ECS_SERVICE}
      - key: ecs.task.arn
        value: ${ECS_TASK_ARN}
      - key: cloud.provider
        value: aws
      - key: cloud.platform
        value: aws_ecs
      - key: cloud.region
        value: ${AWS_REGION}

  # Batch processor for efficiency
  batch:
    timeout: 10s
    send_batch_size: 1024

exporters:
  # Edge Delta Cloud Pipeline exporter
  otlp/edgedelta:
    endpoint: ${EDGEDELTA_OTLP_ENDPOINT}
    tls:
      insecure: false
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
  
  # Optional: CloudWatch Logs for debugging
  logging:
    loglevel: info

service:
  extensions: [health_check, pprof]
  pipelines:
    # Traces pipeline
    traces:
      receivers: [otlp, awsxray]
      processors: [resource, batch]
      exporters: [otlp/edgedelta]
    
    # Application metrics pipeline
    metrics/application:
      receivers: [otlp, statsd]
      processors: [resource, batch]
      exporters: [otlp/edgedelta]
    
    # ECS metrics pipeline
    metrics/ecs:
      receivers: [awsecscontainermetrics]
      processors: [filter, resource, batch]
      exporters: [otlp/edgedelta]
    
    # Logs pipeline
    logs:
      receivers: [otlp]
      processors: [resource, batch]
      exporters: [otlp/edgedelta, logging]

This ADOT Collector configuration establishes a comprehensive telemetry pipeline that receives data from multiple sources and forwards it to Edge Delta Cloud Pipelines. The configuration sets up receivers for OTLP, AWS X-Ray, StatsD, and ECS container metrics, ensuring compatibility with various instrumentation approaches. The processing layer enriches all telemetry with essential ECS metadata including cluster name, service name, and task ARN, while filtering ECS metrics to focus on key performance indicators like memory, CPU, network, and storage utilization. The batch processor optimizes data transmission by grouping telemetry before forwarding to Edge Delta via secure OTLP/gRPC connections. This configuration supports three distinct pipelines for traces, application metrics, and ECS infrastructure metrics, each optimized for their specific data characteristics while maintaining consistent resource attribution across all telemetry types.

Step 2: Store Configuration in Parameter Store

# Create the parameter in AWS Systems Manager
aws ssm put-parameter \
  --name "/aot/config" \
  --type "String" \
  --tier "Advanced" \
  --value "$(cat aot-config.yaml)" \
  --region us-west-2

Step 3: Create CloudFormation Stack

Deploy the complete infrastructure using CloudFormation:

# otel-service.yaml
AWSTemplateFormatVersion: '2010-09-09'
Description: 'AWS OpenTelemetry Collector Service for ECS'

Parameters:
  AppName:
    Type: String
    Default: otel-collector
    
  VpcId:
    Type: String
    Description: VPC ID
    
  Cluster:
    Type: String
    Description: ECS Cluster name
    
  Subnets:
    Type: List<AWS::EC2::Subnet::Id>
    Description: Private subnet IDs
    
  PrivateNamespaceId:
    Type: String
    Description: AWS Cloud Map namespace ID
    
  EdgeDeltaOtlpEndpoint:
    Type: String
    Description: Edge Delta OTLP gRPC endpoint
    Default: "your-pipeline-id-grpc-us-west2-cf.aws.edgedelta.com:443"

Mappings:
  Otel:
    Ports:
      grpc: "4317"
      http: "55681"
      xray: "2000"
      statsd: "8125"

Resources:
  # Store OTEL configuration
  AotConfigParameter:
    Type: AWS::SSM::Parameter
    Properties:
      Name: /aot/config
      Type: String
      Tier: Advanced
      Value: !Sub |
        extensions:
          health_check:
          pprof:
            endpoint: 0.0.0.0:1777

        receivers:
          otlp:
            protocols:
              grpc:
                endpoint: 0.0.0.0:4317
              http:
                endpoint: 0.0.0.0:55681
          awsxray:
            endpoint: 0.0.0.0:2000
            transport: udp
          statsd:
            endpoint: 0.0.0.0:8125
            aggregation_interval: 60s
          awsecscontainermetrics:
            collection_interval: 60s

        processors:
          filter:
            metrics:
              include:
                match_type: strict
                metric_names:
                  - ecs.task.memory.utilized
                  - ecs.task.memory.reserved
                  - ecs.task.cpu.utilized
                  - ecs.task.cpu.reserved
                  - ecs.task.network.rate.rx
                  - ecs.task.network.rate.tx
          resource:
            attributes:
              - key: service.name
                value: ${AppName}
              - key: service.namespace
                value: ${AppName}-namespace
          batch:
            timeout: 10s

        exporters:
          otlp/edgedelta:
            endpoint: ${EdgeDeltaOtlpEndpoint}
            tls:
              insecure: false
          logging:
            loglevel: info

        service:
          extensions: [health_check, pprof]
          pipelines:
            traces:
              receivers: [otlp, awsxray]
              processors: [resource, batch]
              exporters: [otlp/edgedelta]
            metrics/application:
              receivers: [otlp, statsd]
              processors: [resource, batch]
              exporters: [otlp/edgedelta]
            metrics/ecs:
              receivers: [awsecscontainermetrics]
              processors: [filter, resource, batch]
              exporters: [otlp/edgedelta]
            logs:
              receivers: [otlp]
              processors: [resource, batch]
              exporters: [otlp/edgedelta, logging]

  # Security Group
  SecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: !Sub ${AWS::StackName}-otel-sg
      VpcId: !Ref VpcId
      SecurityGroupIngress:
        - CidrIp: !GetAtt VPC.CidrBlock
          IpProtocol: TCP
          FromPort: !FindInMap [Otel, Ports, grpc]
          ToPort: !FindInMap [Otel, Ports, grpc]
        - CidrIp: !GetAtt VPC.CidrBlock
          IpProtocol: TCP
          FromPort: !FindInMap [Otel, Ports, http]
          ToPort: !FindInMap [Otel, Ports, http]
        - CidrIp: !GetAtt VPC.CidrBlock
          IpProtocol: UDP
          FromPort: !FindInMap [Otel, Ports, xray]
          ToPort: !FindInMap [Otel, Ports, xray]
        - CidrIp: !GetAtt VPC.CidrBlock
          IpProtocol: UDP
          FromPort: !FindInMap [Otel, Ports, statsd]
          ToPort: !FindInMap [Otel, Ports, statsd]
      Tags:
        - Key: Name
          Value: !Sub ${AppName}-otel-${AWS::StackName}

  # IAM Roles
  TaskExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: ecs-tasks.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
      Policies:
        - PolicyName: !Sub ${AppName}-task-exec-policy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - ssm:GetParameter
                  - ssm:GetParameters
                Resource: !Sub 'arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/aot/*'

  TaskRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: ecs-tasks.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: !Sub ${AppName}-task-policy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - logs:PutLogEvents
                  - logs:CreateLogGroup
                  - logs:CreateLogStream
                  - logs:DescribeLogStreams
                  - logs:DescribeLogGroups
                  - xray:PutTraceSegments
                  - xray:PutTelemetryRecords
                  - xray:GetSamplingRules
                  - xray:GetSamplingTargets
                  - xray:GetSamplingStatisticSummaries
                  - ssm:GetParameters
                  - ecs:DescribeTasks
                  - ecs:DescribeServices
                  - ecs:DescribeClusters
                  - ecs:ListTasks
                Resource: '*'

  # CloudWatch Log Group
  LogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub '/ecs/${AppName}'
      RetentionInDays: 14

  # Task Definition
  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Family: !Sub '${AppName}-taskdef'
      RequiresCompatibilities:
        - FARGATE
      Memory: '512'
      Cpu: '256'
      NetworkMode: awsvpc
      ExecutionRoleArn: !GetAtt TaskExecutionRole.Arn
      TaskRoleArn: !GetAtt TaskRole.Arn
      ContainerDefinitions:
        - Name: !Ref AppName
          Image: 'public.ecr.aws/aws-observability/aws-otel-collector:latest'
          Essential: true
          Memory: 512
          PortMappings:
            - ContainerPort: !FindInMap [Otel, Ports, grpc]
              Protocol: tcp
            - ContainerPort: !FindInMap [Otel, Ports, http]
              Protocol: tcp
            - ContainerPort: !FindInMap [Otel, Ports, xray]
              Protocol: udp
            - ContainerPort: !FindInMap [Otel, Ports, statsd]
              Protocol: udp
          Secrets:
            - Name: AOT_CONFIG_CONTENT
              ValueFrom: !Ref AotConfigParameter
          Environment:
            - Name: EDGEDELTA_OTLP_ENDPOINT
              Value: !Ref EdgeDeltaOtlpEndpoint
            - Name: AWS_REGION
              Value: !Ref AWS::Region
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-region: !Ref AWS::Region
              awslogs-group: !Ref LogGroup
              awslogs-stream-prefix: ecs
          HealthCheck:
            Command:
              - CMD-SHELL
              - 'curl -f http://localhost:13133/health || exit 1'
            Interval: 30
            Timeout: 5
            Retries: 3

  # Service Discovery
  DiscoveryService:
    Type: AWS::ServiceDiscovery::Service
    Properties:
      Description: Discovery Service for OTEL Collector
      DnsConfig:
        NamespaceId: !Ref PrivateNamespaceId
        RoutingPolicy: MULTIVALUE
        DnsRecords:
          - TTL: 60
            Type: A
      HealthCheckCustomConfig:
        FailureThreshold: 3
      Name: !Ref AppName

  # ECS Service
  FargateService:
    Type: AWS::ECS::Service
    Properties:
      ServiceName: !Sub '${AppName}-service'
      Cluster: !Ref Cluster
      DesiredCount: 1
      TaskDefinition: !Ref TaskDefinition
      LaunchType: FARGATE
      NetworkConfiguration:
        AwsvpcConfiguration:
          AssignPublicIp: DISABLED
          SecurityGroups:
            - !Ref SecurityGroup
          Subnets: !Ref Subnets
      ServiceRegistries:
        - RegistryArn: !GetAtt DiscoveryService.Arn
          ContainerName: !Ref AppName

Outputs:
  ServiceName:
    Description: OTEL Collector Service Name
    Value: !Ref FargateService
  OtelEndpoint:
    Description: OTEL Collector endpoint for applications
    Value: !Sub '${AppName}.${PrivateNamespace}:4317'

This CloudFormation stack creates a complete AWS OpenTelemetry Collector deployment on ECS Fargate with service discovery and proper networking configuration. The stack provisions all necessary infrastructure including IAM roles with appropriate permissions for accessing Parameter Store and CloudWatch, a security group that opens the required ports for OTLP, X-Ray, and StatsD protocols, and a task definition running the official AWS OpenTelemetry Collector image. The ECS service is configured with AWS Cloud Map service discovery, enabling applications to connect using a friendly DNS name rather than hardcoded IP addresses. The collector configuration is stored securely in AWS Systems Manager Parameter Store and injected at runtime, allowing for configuration updates without rebuilding containers. The service runs with minimal resource allocation (256 CPU units and 512 MB memory) suitable for most workloads, though these can be adjusted based on your telemetry volume requirements.

Step 4: Deploy Application with OpenTelemetry

Create a task definition for your application that sends telemetry to ADOT:

# application-task.yaml
TaskDefinition:
  Type: AWS::ECS::TaskDefinition
  Properties:
    Family: !Sub '${AppName}-app-taskdef'
    RequiresCompatibilities:
      - FARGATE
    Memory: '512'
    Cpu: '256'
    NetworkMode: awsvpc
    ExecutionRoleArn: !GetAtt TaskExecutionRole.Arn
    ContainerDefinitions:
      - Name: !Ref AppName
        Image: !Ref ApplicationImage
        Essential: true
        Memory: 512
        PortMappings:
          - ContainerPort: 8080
        Environment:
          # OpenTelemetry configuration
          - Name: OTEL_EXPORTER_OTLP_ENDPOINT
            Value: !Sub 'http://${OtelCollectorDomain}:4317'
          - Name: OTEL_RESOURCE_ATTRIBUTES
            Value: !Sub 'service.name=${AppName},service.namespace=${AppName}-namespace'
          - Name: OTEL_TRACES_EXPORTER
            Value: 'otlp'
          - Name: OTEL_METRICS_EXPORTER
            Value: 'otlp'
          - Name: OTEL_LOGS_EXPORTER
            Value: 'otlp'
          - Name: OTEL_IMR_EXPORT_INTERVAL
            Value: '10000'
          # Java-specific settings (if using Java)
          - Name: JAVA_TOOL_OPTIONS
            Value: '-javaagent:/opt/opentelemetry-javaagent.jar'
        LogConfiguration:
          LogDriver: awslogs
          Options:
            awslogs-region: !Ref AWS::Region
            awslogs-group: !Ref LogGroup
            awslogs-stream-prefix: app

This task definition configures your application container to send telemetry to the AWS OpenTelemetry Collector deployed in the previous step. The configuration establishes connectivity to the collector using AWS Cloud Map service discovery, allowing the application to resolve the collector’s address dynamically through the private DNS namespace. Environment variables configure the OpenTelemetry SDK to export traces, metrics, and logs via the OTLP protocol to the collector endpoint on port 4317. Resource attributes are added to identify the service name and namespace across all telemetry data, ensuring proper correlation and filtering in Edge Delta Cloud Pipelines. For Java applications, the configuration includes automatic instrumentation through the OpenTelemetry Java agent, which provides zero-code instrumentation for popular frameworks and libraries. The task runs on Fargate with minimal resources appropriate for microservices, though memory and CPU allocations should be adjusted based on your application’s requirements.

Step 5: Build and Deploy

Build Application with OTEL Agent (Java Example)

# Dockerfile
FROM openjdk:11-jre-slim

# Download OpenTelemetry Java agent
ADD https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar /opt/opentelemetry-javaagent.jar

# Copy application JAR
COPY target/my-app.jar /app/my-app.jar

# Set Java options to use OTEL agent
ENV JAVA_TOOL_OPTIONS="-javaagent:/opt/opentelemetry-javaagent.jar"

# Run application
ENTRYPOINT ["java", "-jar", "/app/my-app.jar"]

Deploy Using SAM CLI

# Build and push Docker image
docker build -t my-app:latest .
docker tag my-app:latest $AWS_ACCOUNT.dkr.ecr.$AWS_REGION.amazonaws.com/my-app:latest
docker push $AWS_ACCOUNT.dkr.ecr.$AWS_REGION.amazonaws.com/my-app:latest

# Deploy infrastructure
sam deploy \
  --template-file template.yml \
  --stack-name ecs-otel-edgedelta \
  --parameter-overrides \
    EdgeDeltaOtlpEndpoint=your-pipeline-grpc-region.aws.edgedelta.com:443 \
    ApplicationImage=$AWS_ACCOUNT.dkr.ecr.$AWS_REGION.amazonaws.com/my-app:latest \
  --capabilities CAPABILITY_IAM \
  --region us-west-2

Step 6: Verify Telemetry Flow

Check OTEL Collector Status:

# View collector logs
aws logs tail /ecs/otel-collector --follow --region us-west-2

# Check service health
aws ecs describe-services \
  --cluster my-cluster \
  --services otel-collector-service \
  --region us-west-2

Verify Application Telemetry:

# Check if application is sending data to collector
aws logs filter-log-events \
  --log-group-name /ecs/my-app \
  --filter-pattern "OTEL" \
  --region us-west-2

Confirm in Edge Delta:
- Navigate to your Cloud Pipeline in Edge Delta
- Check the Pipeline Overview for incoming data
- Verify traces, metrics, and logs are being received

Advanced Configuration

Multiple Applications

To support multiple applications, use service discovery:

Environment:
  - Name: OTEL_EXPORTER_OTLP_ENDPOINT
    Value: !Sub 'http://otel-collector.${ServiceDiscoveryNamespace}:4317'

Custom Sampling

Configure trace sampling in the OTEL collector:

processors:
  probabilistic_sampler:
    sampling_percentage: 10

Add Custom Attributes

Enrich telemetry with custom attributes:

processors:
  attributes:
    actions:
      - key: environment
        value: production
        action: insert
      - key: team
        value: platform
        action: insert

Performance Considerations

Resource Allocation

The ADOT Collector needs at least 256 CPU units and 512 MB of memory to function effectively. Your application containers should be allocated resources based on their specific requirements, keeping in mind that telemetry collection through OpenTelemetry SDK may add a small overhead to their operation.

Network Configuration

Network configuration plays a vital role in both performance and cost optimization. Implementing VPC endpoints can significantly minimize data transfer costs by keeping traffic within the AWS network. Security groups must be carefully configured to allow communication between containers while maintaining the principle of least privilege.

Data Processing Limits

The ADOT collector’s throughput varies based on configuration, the complexity of processing rules, and the destination endpoints. Edge Delta Cloud Pipelines support up to 100MB per minute with optimal batching, though this can be increased by deploying multiple pipeline instances or working with Edge Delta support to adjust limits based on your needs.

Monitoring and Alerting

CloudWatch Metrics

Monitor OTEL Collector performance:

CloudWatchAlarm:
  Type: AWS::CloudWatch::Alarm
  Properties:
    AlarmName: !Sub '${AppName}-otel-cpu-high'
    MetricName: CPUUtilization
    Namespace: AWS/ECS
    Statistic: Average
    Period: 300
    EvaluationPeriods: 2
    Threshold: 80
    ComparisonOperator: GreaterThanThreshold
    Dimensions:
      - Name: ServiceName
        Value: !Ref FargateService
      - Name: ClusterName
        Value: !Ref Cluster

Auto-scaling

Configure auto-scaling for the OTEL Collector:

ScalingTarget:
  Type: AWS::ApplicationAutoScaling::ScalableTarget
  Properties:
    ServiceNamespace: ecs
    ScalableDimension: ecs:service:DesiredCount
    ResourceId: !Sub 'service/${Cluster}/${FargateService}'
    MinCapacity: 1
    MaxCapacity: 10
    RoleARN: !Sub 'arn:aws:iam::${AWS::AccountId}:role/aws-service-role/ecs.application-autoscaling.amazonaws.com/AWSServiceRoleForApplicationAutoScaling_ECSService'

ScalingPolicy:
  Type: AWS::ApplicationAutoScaling::ScalingPolicy
  Properties:
    PolicyName: !Sub '${AppName}-cpu-scaling'
    ServiceNamespace: ecs
    ScalableDimension: ecs:service:DesiredCount
    ResourceId: !Sub 'service/${Cluster}/${FargateService}'
    PolicyType: TargetTrackingScaling
    TargetTrackingScalingPolicyConfiguration:
      PredefinedMetricSpecification:
        PredefinedMetricType: ECSServiceAverageCPUUtilization
      TargetValue: 70

Troubleshooting

No Data in Edge Delta

Verify Endpoint Configuration:
- Ensure the Edge Delta OTLP endpoint is correct
- Check that port 443 is used for secure connections
- Verify no grpcs:// prefix in the endpoint
Check Network Connectivity:

# Test connectivity from ECS task
aws ecs execute-command \
  --cluster my-cluster \
  --task task-id \
  --container otel-collector \
  --interactive \
  --command "nc -zv your-pipeline-grpc-region.aws.edgedelta.com 443"

Review Security Groups:
- Ensure outbound HTTPS (443) is allowed
- Verify inter-container communication ports are open

High Memory Usage

Adjust batch processor settings:

processors:
  batch:
    timeout: 5s
    send_batch_size: 512
    send_batch_max_size: 1024

Missing ECS Metadata

Ensure the task role has permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecs:DescribeTasks",
        "ecs:DescribeServices",
        "ecs:DescribeClusters"
      ],
      "Resource": "*"
    }
  ]
}

Best Practices

Use Service Discovery: Register OTEL Collector with AWS Cloud Map for easy service-to-service communication
Enable Health Checks: Configure health check endpoints for monitoring
Implement Retry Logic: Configure exponential backoff for transient failures
Batch Processing: Use batch processor to optimize network usage
Resource Limits: Set appropriate CPU and memory limits based on load
Secure Connections: Always use TLS for external connections
Log Sampling: Implement sampling for high-volume applications

Alternative Approach

For organizations requiring full Edge Delta processing capabilities at the edge with more control over data processing, see Ingest Data from Amazon ECS with Edge Delta Agent which demonstrates deploying the Edge Delta agent as a sidecar container for local processing before forwarding to Edge Delta.