Edge Delta Pipeline Pack for AWS CloudTrail
11 minute read
Overview
The CloudTrail pipeline pack ensures seamless ingestion and appropriate processing of AWS CloudTrail data. This pipeline efficiently manages the ingestion process by connecting structured JSON log data from CloudTrail to Edge Delta, providing insights into user activities and API calls across your AWS account. With this pipeline, you can monitor, detect anomalies, and generate metrics from CloudTrail logs, enabling proactive security and operational insights.
Pack Integration Process
The example complete pipeline YAML at the end of this page can be used as-is to deploy a pack in a single fleet. However, pipeline packs are best implemented as Compound Nodes for re-use and change control.
To create a new compound node containing the pack:
1. Create and deploy a new fleet:
Select the default options to deploy a temporary fleet. You will use this fleet to create the compound node and to test the pack.
- Click Pipelines.
- Click New Fleet.
- Select Edge Fleet.
- Optionally, expand Advanced Fleet Configuration and select a pipeline with a configuration you want to duplicate.
- Click Continue.
- Select the appropriate template and click Continue.
- Specify a name to identify the Fleet.
- Click Generate Config.
- Execute the installation commands, they include the unique ID for the Fleet.
- Expand the namespaces and select the input sources you want to monitor.
- Select the Destination Outputs you want to send processed data to, such as the Edge Delta Observability Platform.
- Click Continue.
- Click View Dashboard.
2. Replace the Fleet Yaml
Open the yaml editor for the newly created fleet.
To edit a v3 configuration using YAML:
- Click Pipelines and select the fleet you want to edit.
- Click View/Edit Pipeline.
- Click Edit YAML.
The Edit Configuration dialog box opens. You can edit the configuration and view the change diff.
- Click Done.
Replace the links
and nodes
sections with the following YAML:
nodes:
- name: mask_access_key_id
type: mask
pattern: accessKeyId\":\s\"(?P<accesskeyid>[0-9A-Za-z]+)\"
- name: mask_assumed_role_id
type: mask
pattern: assumedRoleId\":\s\"(?P<assumedroleid>[0-9a-zA-Z-:_]+)\"
- name: mask_session_token
type: mask
pattern: \"sessionToken\":\s\"([^\"]+)\"
- name: unroll_records
type: json_unroll
field_path: Records
new_field_name: Record
- name: parse_json_attributes
type: parse_json_attributes
field_path: item.attributes
- name: extract_event_name
type: extract_json_field
field_path: Record.eventName
keep_log_if_failed: true
- name: route
type: route
paths:
- path: not_important_event
condition: regex_match(item["attributes"]["Records"]["eventName"], "^(Get|Describe|List)")
exit_if_matched: true
links:
- from: mask_access_key_id
to: mask_assumed_role_id
- from: mask_assumed_role_id
to: mask_session_token
- from: mask_session_token
to: unroll_records
- from: unroll_records
to: parse_json_attributes
- from: parse_json_attributes
to: extract_event_name
- from: extract_event_name
to: route
3. Create a Compound Node
- Click Edit Mode
- Select the nodes using one of these methods:
- hover over each node and check the selection checkbox,
- hold Command select each node, or
- hold Shift and drag a selection box over the nodes.
- Select Create Compound Node.
The nodes are grouped in a single compound node and the new compound node is added to the compound node library.
- Click Review Changes.
- Click Deploy Changes.
4. Create Additional Outputs
- Click Pipelines - Compound Nodes
- Click the kebab (⋮) icon in the Actions column for the new Compound Node that was created and select Edit.
- Double click the Output, rename it
Processed
and click Save Changes. - Click Add Output, name it
Fails
and click Save Changes. - Click Add Output, name it
get describe list
and click Save Changes.
5. Wire up the Compound Node
- Connect the Fails output node to the Failure path on the parse_json_attributes node.
- Connect the Fails output node to the Failure path on the extract_event_name node.
- Connect the get describe list output node to the not_important_event path on the route node.
- Connect the Processed output node to the Unmatched path on the route node.
- Connect the Compound_input node to the mask_access_key node.
Edit the name and description appropriately, click Review Changes, click Save, then click Save.
The AWS CloudTrail Pipeline pack has now been added to the Compound node library for use across multiple fleets. You can now delete the temporary fleet you created earlier.
AWS CloudTrail Pack Contents
The CloudTrail Compound Node consists of a series of nodes, each of which will be discussed in turn:
![](/images/cloudtrail-compound-node.png)
Logs flow through each node in the following order:
1. mask_access_key_id
This Mask node masks sensitive information in CloudTrail logs by searching for patterns that match the given regular expression for accessKeyId
. An access key ID is a unique identifier used to make programmatic requests to AWS services. Masking it helps prevent unauthorized access and ensures compliance with data protection regulations.
- name: mask_access_key_id
type: mask
pattern: accessKeyId\":\s\"(?P<accesskeyid>[0-9A-Za-z]+)\"
2. mask_assumed_role_id
This Mask node masks the assumed role ID in the assumedRoleId
field using the provided regular expression. An assumed role ID is used in AWS IAM to temporarily assume roles, providing limited access to resources. Masking it helps in maintaining the privacy and security of role usage.
- name: mask_assumed_role_id
type: mask
pattern: assumedRoleId\":\s\"(?P<assumedroleid>[0-9a-zA-Z-:_]+)\"
3. mask_session_token
This Mask node masks session tokens by finding and replacing text that matches the provided regex pattern for sessionToken
. A session token is part of temporary security credentials provided by AWS to access services securely. Masking it prevents unauthorized access and leakage of security credentials.
- name: mask_session_token
type: mask
pattern: \"sessionToken\":\s\"([^\"]+)\"
4. unroll_records
The Unroll JSON node unrolls structured JSON data from the Records
field in each CloudTrail log and creates a new log for each record found within the Records
array. Each new log features a Record
field at the top level, which contains the data of the original Records
entry.
- name: unroll_records
type: json_unroll
field_path: Records
new_field_name: Record
5. parse_json_attributes
The Parse JSON node parses the JSON attributes from the specified field path in CloudTrail logs and converts them into standalone attributes. This transformation makes accessing and querying individual fields easier. Node failures are routed to the Fails output path of the Compound Node.
- name: parse_json_attributes
type: parse_json_attributes
field_path: item.attributes
6. extract_event_name
The Extract JSON Field node extracts the value of the eventName
field from the Record and creates a new field to hold this value. If extraction fails, the log is still kept (keep_log_if_failed: true
). In CloudTrail logs, the eventName
specifies the action that was performed, such as DescribeInstances or CreateBucket. Node failures are routed to the Fails output path of the Compound Node.
- name: extract_event_name
type: extract_json_field
field_path: Record.eventName
keep_log_if_failed: true
7. route
The Route node routes CloudTrail logs based on specified conditions. Logs are evaluated using a regex pattern to determine if they match the eventName
for specific AWS operations (Get
, Describe
, List
). If the condition is met, the log is routed to the not_important_event
path, and processing stops for that log (exit_if_matched: true
). The not important path sends logs to the get describe list
output path of the Compound node. Logs that are not filtered out (i.e. important logs) are routed to the Processed
output path. This route node helps in filtering out less critical events, focusing on more significant actions.
- name: route
type: route
paths:
- path: not_important_event
condition: regex_match(item["attributes"]["Records"]["eventName"], "^(Get|Describe|List)")
exit_if_matched: true
Testing the Pack
You can use the fleet in which you created the AWS CloudTrail Compound Node to test it.
1. Update the Fleet
To start, update the fleet with the latest version of the pack (which includes the additional nodes you created in the previous step).
- Click Pipelines.
- Select the test fleet.
- Select Compound Nodes.
- Click Review Changes.
- Click View Updated Pipeline.
2. Add a Demo input
You can configure a demo input to generate AWS CloudTrail mock data, which you can use to test the Pipeline Pack.
- Click Add Input.
- Expand Standard and select Demo Input.
- Configure an appropriate name such as
CloudTrail demo
. - Select CloudTrail in the Log Type field and click Save Changes.
- Connect the demo input to the Compound Node’s input.
3. Add a Debug Output
Next, add a debug output node. At first, connect it directly to the Demo Input to view the baseline logs - the logs that will be ingested by the pack.
- Click Add Output.
- Expand Edge Delta and select ED Debug Output.
- Click Save Changes.
- Connect the Demo input directly to the debut output node, bypassing the compound node.
- Click Review Changes.
- Click Save Changes.
After a few minutes, click Pipelines - Debug Output to view the debug node’s output.
Ensure that the test fleet is selected to view the correct debug node’s output.
Copy a baseline sample of the log for comparison in the next step.
Compare Basedline and Processed logs
Next, get a sample of logs processed by the pack:
- Click Pipelines.
- Select the test fleet.
- Click View/Edit Pipeline.
- Click Edit Mode.
- Delete the link between the demo input and the debug output.
- Add a link between the debug output node and the Processed path on the Compound Node.
- Click Review Changes.
- Click Save Changes.
After a few minutes, click Pipelines - Debug Output to view the debug node’s output.
Ensure that the test fleet is selected to view the correct debug node’s output.
Compare the logs being sampled by the debug output node with the baseline sample you colleted in the previous step. This should validate the packs function such as masking fields appropriately.
Check for data flowing on the Fails path of the compound node. It should be 0KBs.
CloudTrail Pack Usage Example
Use the compound node in fleets that ingest AWS CloudTrail logs. In this instance, the pack has been connected to a Log to Pattern node to generate patterns, the Edge Delta Log Output node to capture the logs in the Edge Delta backend, and the Debug Output node to check the content of any logs that failed to be processed by the pack:
![](/images/cloudtrail-pack-fleet.png)
Note: Currently. pasting in the entire pipeline prevents the Compound Node from being added to the library.
links:
- from: ed_component_health
to: ed_health
- from: ed_node_health
to: ed_health
- from: ed_agent_stats
to: ed_metrics
- from: ed_pipeline_io_stats
to: ed_metrics
- from: CloudTrail
to: CloudTrail_Pack->compound_input
- from: CloudTrail_Pack->Processed
to: CloudTrail Patterns
- from: CloudTrail_Pack->Processed
to: ed_archive
- from: CloudTrail_Pack->Fails
to: CloudTrail Patterns
- from: CloudTrail_Pack->Fails
to: ed_debug_output
- from: CloudTrail_Pack->get describe list
to: CloudTrail Patterns
- from: CloudTrail Patterns
to: ed_patterns_output
nodes:
- name: ed_component_health
type: ed_component_health_input
- name: ed_node_health
type: ed_node_health_input
- name: ed_agent_stats
type: ed_agent_stats_input
- name: ed_pipeline_io_stats
type: ed_pipeline_io_stats_input
- name: ed_archive
type: ed_archive_output
- name: ed_metrics
type: ed_metrics_output
- name: ed_health
type: ed_health_output
- name: CloudTrail
type: demo_input
events_per_sec: 1
log_type: cloudtrail
- name: CloudTrail_Pack
type: compound
nodes:
- name: compound_input
type: compound_input
- name: Processed
type: compound_output
- name: mask_access_key_id
type: mask
pattern: accessKeyId\":\s\"(?P<accesskeyid>[0-9A-Za-z]+)\"
- name: mask_assumed_role_id
type: mask
pattern: assumedRoleId\":\s\"(?P<assumedroleid>[0-9a-zA-Z-:_]+)\"
- name: mask_session_token
type: mask
pattern: \"sessionToken\":\s\"([^\"]+)\"
- name: unroll_records
type: json_unroll
field_path: Records
new_field_name: Record
- name: parse_json_attributes
type: parse_json_attributes
field_path: item.attributes
- name: extract_event_name
type: extract_json_field
field_path: Record.eventName
keep_log_if_failed: true
- name: route
type: route
paths:
- path: not_important_event
condition: regex_match(item["attributes"]["Records"]["eventName"], "^(Get|Describe|List)")
exit_if_matched: true
- name: Fails
type: compound_output
- name: get describe list
type: compound_output
links:
- from: compound_input
to: mask_access_key_id
- from: mask_access_key_id
to: mask_assumed_role_id
- from: mask_assumed_role_id
to: mask_session_token
- from: mask_session_token
to: unroll_records
- from: unroll_records
to: parse_json_attributes
- from: parse_json_attributes
to: extract_event_name
- from: parse_json_attributes
path: failure
to: Fails
- from: extract_event_name
to: route
- from: extract_event_name
path: failure
to: Fails
- from: route
path: unmatched
to: Processed
- from: route
path: not_important_event
to: get describe list
- name: ed_debug_output
type: ed_debug_output
- name: ed_patterns_output
type: ed_patterns_output
- name: CloudTrail Patterns
type: log_to_pattern
Example Input
Consider the following extract of a CloudTrail log, it only shows two events in the log: Record 0
and Record 1
.
Note: Sensitive information has been replaced with dummy data.
{
"Records": [
"0": {
"eventVersion": "1.08",
"userIdentity": {
"type": "AssumedRole",
"invokedBy": "securityhub.amazonaws.com"
},
"eventTime": "2024-07-17T09:48:41Z",
"eventSource": "config.amazonaws.com",
"eventName": "DescribeEventAggregates",
"awsRegion": "us-west-2",
"sourceIPAddress": "13.71.17.166",
"userAgent": "config.amazonaws.com",
"requestParameters": {
"roleArn": "arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789",
"roleSessionName": "AWSConfig-BucketConfigCheck"
},
"responseElements": {
"credentials": {
"accessKeyId": "A1B2C3D4E5F6G7H8I9J0",
"expiration": "Jul 10, 171717 8:10:24 AM",
"sessionToken": "ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLM"
},
"assumedRoleUser": {
"assumedRoleId": "A1B2C3D4E5F6G7H8I9J0:AWSConfig-BucketConfigCheck",
"arn": "arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789/AWSConfig-BucketConfigCheck"
}
},
"requestID": "abcd1234-efgh-5678-ijkl-9012mnopqrst",
"eventID": "mnop5678-abcd-1234-efgh-5678ijklqrst",
"readOnly": "true",
"resources": [
"0": {
"accountId": "123456789012",
"type": "AWS::IAM::Role",
"ARN": "arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789"
}
"length": 1,
],
"eventType": "AwsApiCall",
"managementEvent": "true",
"recipientAccountId": "123456789012",
"sharedEventID": "01234567-89ab-cdef-edcb-a9876543210f",
"eventCategory": "Management"
},
"1": {
"eventVersion": "1.08",
"userIdentity": {
"type": "SAMLUser",
"invokedBy": "config.amazonaws.com"
},
"eventTime": "2024-07-17T09:48:41Z",
"eventSource": "ec2.amazonaws.com",
"eventName": "GetBucketAcl",
"awsRegion": "us-west-2",
"sourceIPAddress": "78.28.127.254",
"userAgent": "config.amazonaws.com",
"requestParameters": {
"roleArn": "arn:aws:iam::987654321098:role/ZYXWVUTSRQPONML9876543210",
"roleSessionName": "AWSConfig"
},
"responseElements": {
"credentials": {
"accessKeyId": "B2C3D4E5F6G7H8I9J0A1",
"expiration": "Jul 10, 171717 8:10:24 AM",
"sessionToken": "ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGXYZ12"
},
"assumedRoleUser": {
"assumedRoleId": "B2C3D4E5F6G7H8I9J0A1:AWSConfig",
"arn": "arn:aws:iam::987654321098:role/ZYXWVUTSRQPONML9876543210/AWSConfig"
}
},
"requestID": "wxyz9876-vuts-5432-rqpo-8765nmlkjihgfedc",
"eventID": "qrst6789-efgh-1234-abcd-6789mnopuvwx",
"readOnly": "true",
"resources": [
"0": {
"accountId": "098765432109",
"type": "AWS::IAM::Role",
"ARN": "arn:aws:iam::987654321098:role/ZYXWVUTSRQPONML9876543210"
}
"length": 1,
],
"eventType": "AwsApiCall",
"managementEvent": "true",
"recipientAccountId": "098765432109",
"sharedEventID": "01234567-89ab-bcde-dcba-9876543210fe",
"eventCategory": "Management"
},
...
Example Output
Consider the following log emitted from the CloudTrail Compound Node.
Note: This example is not derived from the example above but the input log had the same structure.
It consists of a simple body:
PutEvaluations
As well as a detailed Attributes field:
{
Record: {
awsRegion: us-west-2
eventCategory: Management
eventID: uvwx7890-ghij-4321-bacd-7890qrstyz
eventName: PutEvaluations
eventSource: sts.amazonaws.com
eventTime: 2024-07-17T09:44:25Z
eventType: AwsApiCall
eventVersion: 1.08
managementEvent: true
readOnly: true
recipientAccountId: 001122334455
requestID: ijkl4321-dcba-8765-zywx-5432vutsrqpo
requestParameters: {
roleArn: arn:aws:iam::112233445566:role/ABCDEF123456XYZ7890
roleSessionName: AWSConfig-Describe
}
resources: {
0: {
ARN: arn:aws:iam::112233445566:role/ABCDEF123456XYZ7890
accountId: 001122334455
type: AWS::IAM::Role
}
}
responseElements: {
assumedRoleUser: {
arn: arn:aws:iam::112233445566:role/ABCDEF123456XYZ7890/AWSConfig-Describe
assumedRoleId: ******
}
credentials: {
accessKeyId: ******
expiration: Jul 10, 171717 8:10:24 AM
sessionToken: ******
}
}
sharedEventID: 12345678-9abc-cdef-f123-4567890abcde
sourceIPAddress: 204.135.13.87
userAgent: streams.metrics.cloudwatch.amazonaws.com
userIdentity: {
invokedBy: streams.metrics.cloudwatch.amazonaws.com
type: AWSService
}
}
}