Edge Delta Pipeline Pack for AWS CloudTrail

Pre-designed pipeline configuration for AWS CloudTrail logs.

11 minute read

Overview

The CloudTrail pipeline pack ensures seamless ingestion and appropriate processing of AWS CloudTrail data. This pipeline efficiently manages the ingestion process by connecting structured JSON log data from CloudTrail to Edge Delta, providing insights into user activities and API calls across your AWS account. With this pipeline, you can monitor, detect anomalies, and generate metrics from CloudTrail logs, enabling proactive security and operational insights.

Pack Integration Process

The example complete pipeline YAML at the end of this page can be used as-is to deploy a pack in a single fleet. However, pipeline packs are best implemented as Compound Nodes for re-use and change control.

To create a new compound node containing the pack:

1. Create and deploy a new fleet:

Select the default options to deploy a temporary fleet. You will use this fleet to create the compound node and to test the pack.

Click Pipelines.
Click New Fleet.
Select Edge Fleet.
Optionally, expand Advanced Fleet Configuration and select a pipeline with a configuration you want to duplicate.
Click Continue.
Select the appropriate template and click Continue.
Specify a name to identify the Fleet.
Click Generate Config.
Execute the installation commands, they include the unique ID for the Fleet.
Expand the namespaces and select the input sources you want to monitor.
Select the Destination Outputs you want to send processed data to, such as the Edge Delta Observability Platform.
Click Continue.
Click View Dashboard.

2. Replace the Fleet Yaml

Open the yaml editor for the newly created fleet.

To edit a v3 configuration using YAML:

Click Pipelines and select the fleet you want to edit.
Click View/Edit Pipeline.
Click Edit YAML.

The Edit Configuration dialog box opens. You can edit the configuration and view the change diff.

Click Done.

Replace the links and nodes sections with the following YAML:


nodes:
- name: mask_access_key_id
  type: mask
  pattern: accessKeyId\":\s\"(?P<accesskeyid>[0-9A-Za-z]+)\"
- name: mask_assumed_role_id
  type: mask
  pattern: assumedRoleId\":\s\"(?P<assumedroleid>[0-9a-zA-Z-:_]+)\"
- name: mask_session_token
  type: mask
  pattern: \"sessionToken\":\s\"([^\"]+)\"
- name: unroll_records
  type: json_unroll
  field_path: Records
  new_field_name: Record
- name: parse_json_attributes
  type: parse_json_attributes
  field_path: item.attributes
- name: extract_event_name
  type: extract_json_field
  field_path: Record.eventName
  keep_log_if_failed: true
- name: route
  type: route
  paths:
  - path: not_important_event
    condition: regex_match(item["attributes"]["Records"]["eventName"], "^(Get|Describe|List)")
    exit_if_matched: true

links:
- from: mask_access_key_id
  to: mask_assumed_role_id
- from: mask_assumed_role_id
  to: mask_session_token
- from: mask_session_token
  to: unroll_records
- from: unroll_records
  to: parse_json_attributes
- from: parse_json_attributes
  to: extract_event_name
- from: extract_event_name
  to: route

3. Create a Compound Node

Click Edit Mode
Select the nodes using one of these methods:

hover over each node and check the selection checkbox,
hold Command select each node, or
hold Shift and drag a selection box over the nodes.

Select Create Compound Node.

The nodes are grouped in a single compound node and the new compound node is added to the compound node library.

Click Review Changes.
Click Deploy Changes.

4. Create Additional Outputs

Click Pipelines - Compound Nodes
Click the kebab (⋮) icon in the Actions column for the new Compound Node that was created and select Edit.
Double click the Output, rename it Processed and click Save Changes.
Click Add Output, name it Fails and click Save Changes.
Click Add Output, name it get describe list and click Save Changes.

5. Wire up the Compound Node

Connect the Fails output node to the Failure path on the parse_json_attributes node.
Connect the Fails output node to the Failure path on the extract_event_name node.
Connect the get describe list output node to the not_important_event path on the route node.
Connect the Processed output node to the Unmatched path on the route node.
Connect the Compound_input node to the mask_access_key node.

Edit the name and description appropriately, click Review Changes, click Save, then click Save.

The AWS CloudTrail Pipeline pack has now been added to the Compound node library for use across multiple fleets. You can now delete the temporary fleet you created earlier.

AWS CloudTrail Pack Contents

The CloudTrail Compound Node consists of a series of nodes, each of which will be discussed in turn:

Logs flow through each node in the following order:

1. mask_access_key_id

This Mask node masks sensitive information in CloudTrail logs by searching for patterns that match the given regular expression for accessKeyId. An access key ID is a unique identifier used to make programmatic requests to AWS services. Masking it helps prevent unauthorized access and ensures compliance with data protection regulations.

  - name: mask_access_key_id
    type: mask
    pattern: accessKeyId\":\s\"(?P<accesskeyid>[0-9A-Za-z]+)\"

2. mask_assumed_role_id

This Mask node masks the assumed role ID in the assumedRoleId field using the provided regular expression. An assumed role ID is used in AWS IAM to temporarily assume roles, providing limited access to resources. Masking it helps in maintaining the privacy and security of role usage.

  - name: mask_assumed_role_id
    type: mask
    pattern: assumedRoleId\":\s\"(?P<assumedroleid>[0-9a-zA-Z-:_]+)\"

3. mask_session_token

This Mask node masks session tokens by finding and replacing text that matches the provided regex pattern for sessionToken. A session token is part of temporary security credentials provided by AWS to access services securely. Masking it prevents unauthorized access and leakage of security credentials.

  - name: mask_session_token
    type: mask
    pattern: \"sessionToken\":\s\"([^\"]+)\"

4. unroll_records

The Unroll JSON node unrolls structured JSON data from the Records field in each CloudTrail log and creates a new log for each record found within the Records array. Each new log features a Record field at the top level, which contains the data of the original Records entry.

  - name: unroll_records
    type: json_unroll
    field_path: Records
    new_field_name: Record

5. parse_json_attributes

The Parse JSON node parses the JSON attributes from the specified field path in CloudTrail logs and converts them into standalone attributes. This transformation makes accessing and querying individual fields easier. Node failures are routed to the Fails output path of the Compound Node.

  - name: parse_json_attributes
    type: parse_json_attributes
    field_path: item.attributes

6. extract_event_name

The Extract JSON Field node extracts the value of the eventName field from the Record and creates a new field to hold this value. If extraction fails, the log is still kept (keep_log_if_failed: true). In CloudTrail logs, the eventName specifies the action that was performed, such as DescribeInstances or CreateBucket. Node failures are routed to the Fails output path of the Compound Node.

  - name: extract_event_name
    type: extract_json_field
    field_path: Record.eventName
    keep_log_if_failed: true

7. route

The Route node routes CloudTrail logs based on specified conditions. Logs are evaluated using a regex pattern to determine if they match the eventName for specific AWS operations (Get, Describe, List). If the condition is met, the log is routed to the not_important_event path, and processing stops for that log (exit_if_matched: true). The not important path sends logs to the get describe list output path of the Compound node. Logs that are not filtered out (i.e. important logs) are routed to the Processed output path. This route node helps in filtering out less critical events, focusing on more significant actions.

  - name: route
    type: route
    paths:
    - path: not_important_event
      condition: regex_match(item["attributes"]["Records"]["eventName"], "^(Get|Describe|List)")
      exit_if_matched: true

Testing the Pack

You can use the fleet in which you created the AWS CloudTrail Compound Node to test it.

1. Update the Fleet

To start, update the fleet with the latest version of the pack (which includes the additional nodes you created in the previous step).

Click Pipelines.
Select the test fleet.
Select Compound Nodes.
Click Review Changes.
Click View Updated Pipeline.

2. Add a Demo input

You can configure a demo input to generate AWS CloudTrail mock data, which you can use to test the Pipeline Pack.

Click Add Input.
Expand Standard and select Demo Input.
Configure an appropriate name such as CloudTrail demo.
Select CloudTrail in the Log Type field and click Save Changes.
Connect the demo input to the Compound Node’s input.

3. Add a Debug Output

Next, add a debug output node. At first, connect it directly to the Demo Input to view the baseline logs - the logs that will be ingested by the pack.

Click Add Output.
Expand Edge Delta and select ED Debug Output.
Click Save Changes.
Connect the Demo input directly to the debut output node, bypassing the compound node.
Click Review Changes.
Click Save Changes.

After a few minutes, click Pipelines - Debug Output to view the debug node’s output.

Ensure that the test fleet is selected to view the correct debug node’s output.

Copy a baseline sample of the log for comparison in the next step.

Compare Basedline and Processed logs

Next, get a sample of logs processed by the pack:

Click Pipelines.
Select the test fleet.
Click View/Edit Pipeline.
Click Edit Mode.
Delete the link between the demo input and the debug output.
Add a link between the debug output node and the Processed path on the Compound Node.
Click Review Changes.
Click Save Changes.

After a few minutes, click Pipelines - Debug Output to view the debug node’s output.

Ensure that the test fleet is selected to view the correct debug node’s output.

Compare the logs being sampled by the debug output node with the baseline sample you colleted in the previous step. This should validate the packs function such as masking fields appropriately.

Check for data flowing on the Fails path of the compound node. It should be 0KBs.

CloudTrail Pack Usage Example

Use the compound node in fleets that ingest AWS CloudTrail logs. In this instance, the pack has been connected to a Log to Pattern node to generate patterns, the Edge Delta Log Output node to capture the logs in the Edge Delta backend, and the Debug Output node to check the content of any logs that failed to be processed by the pack:

Note: Currently. pasting in the entire pipeline prevents the Compound Node from being added to the library.

links:
- from: ed_component_health
  to: ed_health
- from: ed_node_health
  to: ed_health
- from: ed_agent_stats
  to: ed_metrics
- from: ed_pipeline_io_stats
  to: ed_metrics
- from: CloudTrail
  to: CloudTrail_Pack->compound_input
- from: CloudTrail_Pack->Processed
  to: CloudTrail Patterns
- from: CloudTrail_Pack->Processed
  to: ed_archive
- from: CloudTrail_Pack->Fails
  to: CloudTrail Patterns
- from: CloudTrail_Pack->Fails
  to: ed_debug_output
- from: CloudTrail_Pack->get describe list
  to: CloudTrail Patterns
- from: CloudTrail Patterns
  to: ed_patterns_output

nodes:
- name: ed_component_health
  type: ed_component_health_input
- name: ed_node_health
  type: ed_node_health_input
- name: ed_agent_stats
  type: ed_agent_stats_input
- name: ed_pipeline_io_stats
  type: ed_pipeline_io_stats_input
- name: ed_archive
  type: ed_archive_output
- name: ed_metrics
  type: ed_metrics_output
- name: ed_health
  type: ed_health_output
- name: CloudTrail
  type: demo_input
  events_per_sec: 1
  log_type: cloudtrail
- name: CloudTrail_Pack
  type: compound
  nodes:
  - name: compound_input
    type: compound_input
  - name: Processed
    type: compound_output
  - name: mask_access_key_id
    type: mask
    pattern: accessKeyId\":\s\"(?P<accesskeyid>[0-9A-Za-z]+)\"
  - name: mask_assumed_role_id
    type: mask
    pattern: assumedRoleId\":\s\"(?P<assumedroleid>[0-9a-zA-Z-:_]+)\"
  - name: mask_session_token
    type: mask
    pattern: \"sessionToken\":\s\"([^\"]+)\"
  - name: unroll_records
    type: json_unroll
    field_path: Records
    new_field_name: Record
  - name: parse_json_attributes
    type: parse_json_attributes
    field_path: item.attributes
  - name: extract_event_name
    type: extract_json_field
    field_path: Record.eventName
    keep_log_if_failed: true
  - name: route
    type: route
    paths:
    - path: not_important_event
      condition: regex_match(item["attributes"]["Records"]["eventName"], "^(Get|Describe|List)")
      exit_if_matched: true
  - name: Fails
    type: compound_output
  - name: get describe list
    type: compound_output
  links:
  - from: compound_input
    to: mask_access_key_id
  - from: mask_access_key_id
    to: mask_assumed_role_id
  - from: mask_assumed_role_id
    to: mask_session_token
  - from: mask_session_token
    to: unroll_records
  - from: unroll_records
    to: parse_json_attributes
  - from: parse_json_attributes
    to: extract_event_name
  - from: parse_json_attributes
    path: failure
    to: Fails
  - from: extract_event_name
    to: route
  - from: extract_event_name
    path: failure
    to: Fails
  - from: route
    path: unmatched
    to: Processed
  - from: route
    path: not_important_event
    to: get describe list
- name: ed_debug_output
  type: ed_debug_output
- name: ed_patterns_output
  type: ed_patterns_output
- name: CloudTrail Patterns
  type: log_to_pattern

Example Input

Consider the following extract of a CloudTrail log, it only shows two events in the log: Record 0 and Record 1.

Note: Sensitive information has been replaced with dummy data.

{
  "Records": [
    "0": {
      "eventVersion": "1.08",
      "userIdentity": {
        "type": "AssumedRole",
        "invokedBy": "securityhub.amazonaws.com"
      },
      "eventTime": "2024-07-17T09:48:41Z",
      "eventSource": "config.amazonaws.com",
      "eventName": "DescribeEventAggregates",
      "awsRegion": "us-west-2",
      "sourceIPAddress": "13.71.17.166",
      "userAgent": "config.amazonaws.com",
      "requestParameters": {
        "roleArn": "arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789",
        "roleSessionName": "AWSConfig-BucketConfigCheck"
      },
      "responseElements": {
        "credentials": {
          "accessKeyId": "A1B2C3D4E5F6G7H8I9J0",
          "expiration": "Jul 10, 171717 8:10:24 AM",
          "sessionToken": "ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLM"
        },
        "assumedRoleUser": {
          "assumedRoleId": "A1B2C3D4E5F6G7H8I9J0:AWSConfig-BucketConfigCheck",
          "arn": "arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789/AWSConfig-BucketConfigCheck"
        }
      },
      "requestID": "abcd1234-efgh-5678-ijkl-9012mnopqrst",
      "eventID": "mnop5678-abcd-1234-efgh-5678ijklqrst",
      "readOnly": "true",
      "resources": [
        "0": {
          "accountId": "123456789012",
          "type": "AWS::IAM::Role",
          "ARN": "arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789"
        }
        "length": 1,
      ],
      "eventType": "AwsApiCall",
      "managementEvent": "true",
      "recipientAccountId": "123456789012",
      "sharedEventID": "01234567-89ab-cdef-edcb-a9876543210f",
      "eventCategory": "Management"
    },
    "1": {
      "eventVersion": "1.08",
      "userIdentity": {
        "type": "SAMLUser",
        "invokedBy": "config.amazonaws.com"
      },
      "eventTime": "2024-07-17T09:48:41Z",
      "eventSource": "ec2.amazonaws.com",
      "eventName": "GetBucketAcl",
      "awsRegion": "us-west-2",
      "sourceIPAddress": "78.28.127.254",
      "userAgent": "config.amazonaws.com",
      "requestParameters": {
        "roleArn": "arn:aws:iam::987654321098:role/ZYXWVUTSRQPONML9876543210",
        "roleSessionName": "AWSConfig"
      },
      "responseElements": {
        "credentials": {
          "accessKeyId": "B2C3D4E5F6G7H8I9J0A1",
          "expiration": "Jul 10, 171717 8:10:24 AM",
          "sessionToken": "ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGXYZ12"
        },
        "assumedRoleUser": {
          "assumedRoleId": "B2C3D4E5F6G7H8I9J0A1:AWSConfig",
          "arn": "arn:aws:iam::987654321098:role/ZYXWVUTSRQPONML9876543210/AWSConfig"
        }
      },
      "requestID": "wxyz9876-vuts-5432-rqpo-8765nmlkjihgfedc",
      "eventID": "qrst6789-efgh-1234-abcd-6789mnopuvwx",
      "readOnly": "true",
      "resources": [
        "0": {
          "accountId": "098765432109",
          "type": "AWS::IAM::Role",
          "ARN": "arn:aws:iam::987654321098:role/ZYXWVUTSRQPONML9876543210"
        }
        "length": 1,
      ],
      "eventType": "AwsApiCall",
      "managementEvent": "true",
      "recipientAccountId": "098765432109",
      "sharedEventID": "01234567-89ab-bcde-dcba-9876543210fe",
      "eventCategory": "Management"
    },
...

Example Output

Consider the following log emitted from the CloudTrail Compound Node.

Note: This example is not derived from the example above but the input log had the same structure.

It consists of a simple body:

PutEvaluations

As well as a detailed Attributes field:

{
  Record: {
    awsRegion: us-west-2
    eventCategory: Management
    eventID: uvwx7890-ghij-4321-bacd-7890qrstyz
    eventName: PutEvaluations
    eventSource: sts.amazonaws.com
    eventTime: 2024-07-17T09:44:25Z
    eventType: AwsApiCall
    eventVersion: 1.08
    managementEvent: true
    readOnly: true
    recipientAccountId: 001122334455
    requestID: ijkl4321-dcba-8765-zywx-5432vutsrqpo
    requestParameters: {
      roleArn: arn:aws:iam::112233445566:role/ABCDEF123456XYZ7890
      roleSessionName: AWSConfig-Describe
    }
    resources: {
      0: {
        ARN: arn:aws:iam::112233445566:role/ABCDEF123456XYZ7890
        accountId: 001122334455
        type: AWS::IAM::Role
      }
    }
    responseElements: {
      assumedRoleUser: {
        arn: arn:aws:iam::112233445566:role/ABCDEF123456XYZ7890/AWSConfig-Describe
        assumedRoleId: ******
      }
      credentials: {
        accessKeyId: ******
        expiration: Jul 10, 171717 8:10:24 AM
        sessionToken: ******
      }
    }
    sharedEventID: 12345678-9abc-cdef-f123-4567890abcde
    sourceIPAddress: 204.135.13.87
    userAgent: streams.metrics.cloudwatch.amazonaws.com
    userIdentity: {
      invokedBy: streams.metrics.cloudwatch.amazonaws.com
      type: AWSService
    }
  }
}