Amazon Elastic Container Service (ECS)

Installing the Edge Delta Fleet on Amazon ECS.

Step 1: Create a Configuration

Select the Kubernetes template option in the following steps:

  1. Click Pipelines.
  2. Click New Fleet.
  3. Ensure Visual Pipelines is selected and click Continue.
  4. Select the appropriate template and click Continue.
  5. Specify a tag to identify the Fleet and environment.
  6. Click Create Configuration.

To deploy the Fleet:

  1. Click Pipelines and select the Configuration.
  2. Click Add Agents.
  3. Select the appropriate template
  4. Execute the installation commands, they include the unique Pipeline ID for the configuration.

Copy the Pipeline ID from the Key column.

Step 2: Create an ECS Task Definition

  1. Download the https://release.edgedelta.com/edgedelta-ecs.json task definition.
  2. In the task definition, replace <YOUR_ED_API_KEY> with your copied Pipeline ID.
  3. Create the Fleet task definition via the AWS CLI or via the AWS console.

AWS CLI:

aws ecs register-task-definition --cli-input-jsonfile://path_to_edgedelta-ecs.json

AWS Console:

  1. Navigate to the Elastic Container Service (ECS) section.
  2. Click Task Definitions.
  3. Click Create new Task Definition.
  4. Select a launch type compatibility, and then click Next.
  5. Click Configure via JSON.
  6. Paste the paste task definition JSON, and then click Save.
  7. Click Create.

Step 3: Run the Fleet as an ECS Daemon Service

While you can use the Fleet as an ordinary ECS task, you will only collect logs from the containers running on the EC2 instance the task is running. To properly collect all logs from containers on all instances in your cluster, you need to run the Fleet as a Daemon Service.

  1. In the AWS Console, navigate to the Elastic Container Service (ECS) section, and then find your cluster.
  2. Under Services, click Create.
  3. Select EC2 Launch Type.
  4. In the drop-down menu, select the task definition you created.
  5. Select DAEMON service type, set a service name, and then click Next step.
  6. Select None for load balancer, click Next step.
  7. By default, auto scaling is disabled. click Next step.
  8. Review the summary, and then click Create Service.

Step 4: Verify Definitions and Configurations

  1. Verify that the container definition for other tasks or services does not have the logConfiguration.logDriver parameter. Without a logging driver, logs are written to standard output and collected by the Fleet service.
  2. In the Edge Delta App, click Pipelines. Select the Fleet and and click View/Edit Pipeline.
  3. Verify that you have the Docker input node with image=.* specified in the Docker Include field.

(Optional) Step 5: Set Up File Monitoring

As an optional step, you can monitor for additional log files on your EC2 instances in your ECS cluster.

To do so, update the mountpoints and volumes section in the edgedeleta-ecs.json.

The following example monitors log files in /var/log/ecs on an EC2 instance. Additionally, you must add a file input in the Pipeline YAML configuration file to monitor the mounted path. In this example, the path is /host/var/log/ecs/.

"mountPoints": [
        {
          "containerPath": "/var/run/docker.sock",
          "sourceVolume": "docker_sock",
          "readOnly": true
        },
        {
          "containerPath": "/host/var/log/ecs/",
          "sourceVolume": "ecs_logs",
          "readOnly": true
        },
      ],
 ...

"volumes": [
  {
    "host": {
      "sourcePath": "/var/run/docker.sock"
    },
    "name": "docker_sock"
  },
  {
    "host": {
      "sourcePath": "/var/log/ecs/"
    },
    "name": "ecs_logs"
  }
],
...