Google Cloud Audit Pack
4 minute read
Edge Delta Pipeline Pack for Google Cloud Audit Logs
Overview
The Edge Delta Google Cloud Audit Pack extracts the logs and transforms them to the schema compliant with OCSF requirements.
Pack Description
1. Data Ingestion
The processing begins with the Input as the entry point where all logs’ processing journey starts.
2. Multi Processor Node
Logs are processed by a multiprocessor node, managing transformations and field additions using the OTTL language.
- name: Multi Processor
type: sequence
processors:
2.1 Parse JSON Body
The logs are initially processed by Parse JSON, which is a Parse JSON Processor. It parses the JSON string body into a map.
- type: ottl_transform
metadata: '{"id":"QnWK5QOpfy2EzDNJLOVt0","type":"parse-json","name":"Parse JSON"}'
data_types:
- log
statements: |-
set(cache["parsed-json"], ParseJSON(body))
merge_maps(attributes, cache["parsed-json"], "upsert") where IsMap(attributes) and IsMap(cache["parsed-json"])
set(attributes, cache["parsed-json"]) where not (IsMap(attributes) and IsMap(cache["parsed-json"]))
The statement set(cache["parsed-json"], ParseJSON(body)) parses the JSON string in body and stores the resulting map in cache["parsed-json"]. The statement merge_maps(attributes, cache["parsed-json"], "upsert") where IsMap(attributes) and IsMap(cache["parsed-json"]) merges cache["parsed-json"] into attributes by updating or inserting keys, but only if both are maps. The statement set(attributes, cache["parsed-json"]) where not (IsMap(attributes) and IsMap(cache["parsed-json"])) replaces attributes entirely with cache[“parsed-json”] if either is not a map.
2.2 Extract UID
Following JSON parsing, the Extract UID processor, which is a Parse Regex Processor, extracts UID information using regex.
- type: ottl_transform
metadata: '{"id":"MbEHXePTBpMIkDZGGtpXs","type":"parse-regex","name":"Extract UID"}'
data_types:
- log
statements: |-
merge_maps(attributes, ExtractPatterns(attributes["protoPayload"]["authenticationInfo"]["principalSubject"], ":(?<uid>.*)@"), "upsert") where IsMap(attributes)
set(attributes, ExtractPatterns(attributes["protoPayload"]["authenticationInfo"]["principalSubject"], ":(?<uid>.*)@")) where not IsMap(attributes)
The first statement extracts a pattern from attributes["protoPayload"]["authenticationInfo"]["principalSubject"], specifically capturing the uid, and merges the result into attributes using an upsert operation if attributes is a map. The second statement extracts the uid pattern from attributes["protoPayload"]["authenticationInfo"]["principalSubject"] and sets attributes to this result if attributes is not a map.
2.3 Required Fields
Logs are further processed to populate required fields for OCSF compliance by Required Fields, an Add Field Processor.
- type: ottl_transform
metadata: '{"id":"TXyByCVDC16UHXHXblYuU","type":"add-field","name":"Required Fields"}'
data_types:
- log
statements: |-
set(attributes["target"]["class_uid"], 3002)
set(attributes["target"]["category_uid"], 3)
set(attributes["target"]["metadata"]["product"]["name"], "Google Cloud Audit Logs")
set(attributes["target"]["metadata"]["product"]["vendor_name"], "Google")
set(attributes["target"]["metadata"]["product"]["version"], "v1")
set(attributes["target"]["metadata"]["version"], "1.0.0-rc.3")
set(attributes["target"]["user"]["type_id"], 1)
set(attributes["target"]["activity_id"], 0)
set(attributes["target"]["severity_id"], 0)
These statements set specific attributes within a target object, including identifiers, category, and metadata details such as product name, vendor, version, and IDs for class, user type, activity, and severity level. These fields ensure standardized auditing compatible with OCSF models.
2.4 Recommended Fields
The Recommended Fields node populates additional fields that are not required but recommended for OCSF compliance, using an Add Field Processor.
- type: ottl_transform
metadata: '{"id":"F0LZXqX6838VUAU4zcCB5","type":"add-field","name":"Recommended Fields"}'
data_types:
- log
statements: |-
set(attributes["target"]["timezone_offset"], "")
set(attributes["target"]["auth_protocol_id"], 99)
set(attributes["target"]["logon_type_id"], 99)
These statements assign attributes within a target object, specifically setting an empty string for the timezone_offset, and setting the auth_protocol_id and logon_type_id both to 99. Recommended fields highlight key aspects like network protocols and time-sensitive pivots.
2.5 Optional Fields
Optional fields are populated through Optional Fields, some of which are enabled by default while others are disabled. This is executed using a Custom Processor.
- type: ottl_transform
metadata: '{"id":"8rjWuJfFJ96oW5dSRER3V","type":"ottl_transform","name":"Optional Fields"}'
data_types:
- log
statements: |
set(attributes["target"]["category_name"], "Audit Activity")
set(attributes["target"]["class_name"], 1)
set(attributes["target"]["class_name"], "Account Change") where attributes["target"]["class_uid"] == 3001
set(attributes["target"]["class_name"], "Authentication") where attributes["target"]["class_uid"] == 3002
set(attributes["target"]["count"], 1)
set(attributes["target"]["data"], "")
set(attributes["target"]["start_time"], "")
set(attributes["target"]["end_time"], "")
set(attributes["target"]["duration"], "")
set(attributes["target"]["enrichments"], [])
set(attributes["target"]["http_request"], {})
set(attributes["target"]["observables"], [{"name": attributes["insertId"],"type_id": 99}])
set(attributes["target"]["is_cleartext"], false)
set(attributes["target"]["is_mfa"], true)
set(attributes["target"]["session"], {})
set(attributes["target"]["src_endpoint"]["hostname"], attributes["protoPayload"]["serviceName"])
set(attributes["target"]["src_endpoint"]["ip"], attributes["protoPayload"]["requestMetadata"]["callerIp"])
set(attributes["target"]["src_endpoint"]["uid"], attributes["uid"])
set(attributes["target"]["unmapped"], {})
set(cache["severity"], {"6":"Fatal","5":"Critical","4":"High","3":"Medium","2":"Low","1":"Informational","0":"Unknown"})
set(attributes["target"]["severity"], "Unknown")
set(attributes["target"]["severity"], cache["severity"][Format("%d", [attributes["target"]["severity_id"]])])
set(cache["type"], {"300199":"Account Change Other","300109":"Account Change Lock","300108":"Account Change: Detach Policy","300107":"Account Change: Attach Policy","300106":"Account Change: Delete","300105":"Account Change: Disable","300104":"Account Change: Password Reset","300103":"Account Change: Password Change","300102":"Account Change: Enable","300101":"Account Change: Create"})
set(attributes["target"]["type_name"], "Account Change Unknown")
set(attributes["target"]["type_name"], cache["type"][Format("%d", [attributes["target"]["type_uid"]])])
set(cache["auth_protocol"], {"99":"Other","10":"RADIUS","9":"EAP","8":"CHAP","7":"PAP","6":"OAUTH 2.0","5":"SAML","4":"OpenID","3":"Digest","2":"Kerberos","1":"NTLM"})
set(attributes["target"]["auth_protocol"], "Unknown")
set(attributes["target"]["auth_protocol"], cache["auth_protocol"][Format("%d", [attributes["target"]["auth_protocol_id"]])])
set(cache["logon_type"], {"99":"Other","13":"Cached Unlock","12":"Cached Remote Interactive","11":"Cached Interactive","10":"Remote Interactive","9":"New Credentials","8":"Network Cleartext","7":"Unlock","5":"OS Service","4":"Batch","3":"Network","2":"Interactive","1":"System"})
set(attributes["target"]["logon_type"], "Unknown")
set(attributes["target"]["logon_type"], cache["logon_type"][Format("%d", [attributes["target"]["logon_type_id"]])])
These fields allow for further customization and enrichment based on the audit’s potential.
2.6 Overwrite Body
The Set Body Custom Processor overwrites the body with extracted information back as a map. The original data as a JSON string is preserved in the raw_data field, which is a recommended field.
- type: ottl_transform
metadata: '{"id":"UKGSNWEGuWhItiWydGLgN","type":"ottl_transform","name":"Set body"}'
data_types:
- log
statements: |
set(body, attributes["target"])
set(attributes, {})
This ensures all processed information forms a coherent output while retaining essential raw data.
3. Data Output
Once processed, logs are sent to the Output, marking the end of the processing journey within this pack. These are now routed to downstream nodes, ready for further analysis or storage.
Sample Input
{"insertId":"lleauAChZZ","logName":"projects/affable-alpha-386416/logs/cloudaudit.googleapis.com%2Factivity","protoPayload":{"@type":"type.googleapis.com/google.cloud.audit.AuditLog","authenticationInfo":{"principalEmail":"153750138338-compute@developer.gserviceaccount.com","principalSubject":"serviceAccount:153750138338-compute@developer.gserviceaccount.com","serviceAccountKeyName":"//iam.googleapis.com/projects/affable-alpha-386416/serviceAccounts/153750138338-compute@developer.gserviceaccount.com/keys/76b9aeba5cdb268d4fe4e0e9ddfeea35eea0647f"},"authorizationInfo":[{"granted":true,"permission":"pubsub.topics.attachSubscription","resource":"projects/affable-alpha-386416/topics/all4","resourceAttributes":{}}],"methodName":"google.pubsub.v1.Subscriber.CreateSubscription","request":{"@type":"type.googleapis.com/google.pubsub.v1.Subscription","name":"projects/affable-alpha-453629/subscriptions/all4sub","topic":"projects/affable-alpha-982691/topics/all4"},"requestMetadata":{"callerIp":"215.217.63.128","callerSuppliedUserAgent":"grpc-node-js/1.6.8,gzip(gfe)","destinationAttributes":{},"requestAttributes":{"auth":{},"time":"2025-06-13T10:26:58.741612000Z"}},"resourceName":"projects/affable-alpha-264335/topics/all4","response":{"@type":"type.googleapis.com/google.pubsub.v1.Subscription"},"serviceName":"pubsub.googleapis.com"},"receiveTimestamp":"2025-06-13T10:26:58.741612000Z","resource":{"labels":{"project_id":"affable-alpha-579151","topic_id":"projects/affable-alpha-062176/topics/all4"},"type":"pubsub_topic"},"severity":"NOTICE","timestamp":"2025-06-13T10:26:58.741612000Z"}
{"insertId":"jgFrWGndKl","logName":"projects/affable-alpha-386416/logs/cloudaudit.googleapis.com%2Factivity","protoPayload":{"@type":"type.googleapis.com/google.cloud.audit.AuditLog","authenticationInfo":{"principalEmail":"153750138338-compute@developer.gserviceaccount.com","principalSubject":"serviceAccount:153750138338-compute@developer.gserviceaccount.com","serviceAccountKeyName":"//iam.googleapis.com/projects/affable-alpha-386416/serviceAccounts/153750138338-compute@developer.gserviceaccount.com/keys/76b9aeba5cdb268d4fe4e0e9ddfeea35eea0647f"},"authorizationInfo":[{"granted":true,"permission":"pubsub.topics.attachSubscription","resource":"projects/affable-alpha-386416/topics/all4","resourceAttributes":{}}],"methodName":"google.pubsub.v1.Subscriber.CreateSubscription","request":{"@type":"type.googleapis.com/google.pubsub.v1.Subscription","name":"projects/affable-alpha-475746/subscriptions/all4sub","topic":"projects/affable-alpha-110819/topics/all4"},"requestMetadata":{"callerIp":"55.135.46.60","callerSuppliedUserAgent":"grpc-node-js/1.6.8,gzip(gfe)","destinationAttributes":{},"requestAttributes":{"auth":{},"time":"2025-06-13T10:25:53.754819000Z"}},"resourceName":"projects/affable-alpha-543335/topics/all4","response":{"@type":"type.googleapis.com/google.pubsub.v1.Subscription"},"serviceName":"pubsub.googleapis.com"},"receiveTimestamp":"2025-06-13T10:25:53.754819000Z","resource":{"labels":{"project_id":"affable-alpha-570729","topic_id":"projects/affable-alpha-712326/topics/all4"},"type":"pubsub_topic"},"severity":"INFO","timestamp":"2025-06-13T10:25:53.754819000Z"}