Amazon Elastic Container Service (ECS)

Installing the Edge Delta Fleet on Amazon ECS.

  3 minute read  

Step 1: Create a Configuration

Select the Kubernetes template option in the following steps:

  1. Click Pipelines.
  2. Click New Fleet.
  3. Select Edge Fleet.
  4. Optionally, expand Advanced Fleet Configuration and select a pipeline with a configuration you want to duplicate.
  5. Click Continue.
  6. Select the appropriate template and click Continue.
  7. Specify a name to identify the Fleet.
  8. Click Generate Config.
  9. Execute the installation commands, they include the unique ID for the Fleet.
  10. Expand the namespaces and select the input sources you want to monitor.
  11. Select the Destination Outputs you want to send processed data to, such as the Edge Delta Observability Platform.
  12. Click Continue.
  13. Click View Dashboard.

Copy the Pipeline ID from the Key column.

Step 2: Create an ECS Task Definition

  1. Download the https://release.edgedelta.com/edgedelta-ecs.json task definition.
  2. In the task definition, replace <YOUR_ED_API_KEY> with your copied Pipeline ID.
  3. Create the fleet task definition via the AWS CLI or via the AWS console.

AWS CLI:

aws ecs register-task-definition --cli-input-jsonfile://path_to_edgedelta-ecs.json

AWS Console:

  1. Navigate to the Elastic Container Service (ECS) section.
  2. Click Task Definitions.
  3. Click Create new Task Definition.
  4. Select a launch type compatibility, and then click Next.
  5. Click Configure via JSON.
  6. Paste the paste task definition JSON, and then click Save.
  7. Click Create.

Step 3: Run the Fleet as an ECS Daemon Service

While you can use the fleet as an ordinary ECS task, you will only collect logs from the containers running on the EC2 instance the task is running. To properly collect all logs from containers on all instances in your cluster, you need to run the Fleet as a daemon Service.

  1. In the AWS Console, navigate to the Elastic Container Service (ECS) section, and then find your cluster.
  2. Under Services, click Create.
  3. Select EC2 Launch Type.
  4. In the drop-down menu, select the task definition you created.
  5. Select DAEMON service type, set a service name, and then click Next step.
  6. Select None for load balancer, click Next step.
  7. By default, auto scaling is disabled. click Next step.
  8. Review the summary, and then click Create Service.

Step 4: Verify Definitions and Configurations

  1. Verify that the container definition for other tasks or services does not have the logConfiguration.logDriver parameter. Without a logging driver, logs are written to standard output and collected by the agent service.
  2. In the Edge Delta App, click Pipelines. Select the fleet and and click View/Edit Pipeline.
  3. Verify that you have the Docker source node with image=.* specified in the Docker Include field.

(Optional) Step 5: Set Up File Monitoring

As an optional step, you can monitor for additional log files on your EC2 instances in your ECS cluster.

To do so, update the mountpoints and volumes section in the edgedelta-ecs.json.

The following example monitors log files in /var/log/ecs on an EC2 instance. Additionally, you must add a file input in the Pipeline YAML configuration file to monitor the mounted path. In this example, the path is /host/var/log/ecs/.

"mountPoints": [
        {
          "containerPath": "/var/run/docker.sock",
          "sourceVolume": "docker_sock",
          "readOnly": true
        },
        {
          "containerPath": "/host/var/log/ecs/",
          "sourceVolume": "ecs_logs",
          "readOnly": true
        },
      ],
 ...

"volumes": [
  {
    "host": {
      "sourcePath": "/var/run/docker.sock"
    },
    "name": "docker_sock"
  },
  {
    "host": {
      "sourcePath": "/var/log/ecs/"
    },
    "name": "ecs_logs"
  }
],
...