Microsoft Activity Pack

Edge Delta’s Microsoft Activity Pack

Overview

The Microsoft Activity Pack processes API logs relating to Microsoft services, extracting key information and transforming it to be more manageable and insightful. This pack is useful for structured data analysis, system monitoring, and operational intelligence.

Pack Description

1. Data Ingestion

The data flow starts with the Source as the entry point into the pack.

2. Extract Source

Logs are processed by the Extract source node, which is an OTTL Transform node.

- name: Extract source
  type: ottl_transform
  statements: |-
    set(attributes["audit"], ExtractPatterns(Decode(body, "utf-8"), "$(?<source_collector>audit_[^$]*)$"))
    set(attributes["source"], Concat(["o365:api", attributes["audit"]["source_collector"]], ":")) where attributes["audit"]["source_collector"] != nil
    delete_key(attributes, "audit")    

This node concatenates components to create a log source identifier and includes part of the collector’s name found in the pattern.

It sets the attributes["source"] attribute to a concatenated string of "o365:api" and the extracted pattern from body, with a colon separating them. The regex in ExtractPatterns is looking for a substring at the end of the body which matches the pattern audit_[^$]* and names it source_collector.

This transform operation identifies the source of the logs and forms a meaningful identifier.

3. Set Index and Sourcetype

Next, the Set index and sourcetype node further enriches the log data. It also is an OTTL Transform node.

- name: Set index and sourcetype
  type: ottl_transform
  statements: |-
    set(attributes["index"], "o365")
    set(attributes["sourcetype"], Concat(["o365:activity", ConvertCase(resource["service.name"], "lower")], ":"))    

It works by setting predefined indices and sourcetypes within the dataset.

It uses the set function to assign the string value "o365" to the attribute attributes["index"]. This is a straightforward assignment that designates the index of the telemetry data as "o365".

In the next statement, the Concat function is used to join strings together. Here, it concatenates two strings: "o365:activity" and the result of the ConvertCase function, with a colon in between. The ConvertCase function changes the case of a string. Here, it’s applied to resource["service.name"], converting its value to lowercase by specifying “lower” as the case type. Finally, the set function then assigns the concatenated result to attributes["sourcetype"].

By establishing consistent index and sourcetype attributes, you help in categorizing logs efficiently.

4. Reserialize Body

In the next phase, data is restructured with the Reserialize body OTTL Transform node.

- name: Reserialize body
  type: ottl_transform
  statements: |-
    set(attributes["parsed_body"], ParseJSON(Decode(body, "utf-8")))
    edx_delete_empty_values(attributes["parsed_body"], [], [""], ["deleteNull"])
    set(body, EDXEncode(attributes["parsed_body"], "utf-8", true))
    delete_key(attributes, "parsed_body")    

It parses JSON content into structured data and removes any empty array or null values as follows:

1. The first set statement decodes the body from a byte array to a UTF-8 encoded string and parses the decoded string into a JSON object, which is assigned to attributes["parsed_body"].
2. The edx_delete_empty_values custom function deletes keys from attributes["parsed_body"] where the values are empty strings or nulls, effectively cleaning the parsed JSON object of any empty values.
3. The second set statement uses the EDXEncode custom function to convert attributes["parsed_body"] back into a byte array encoded as UTF-8. This sets the body field with the newly encoded data.
4. The delete_key function deletes the temporary attributes["parsed_body"] field from attributes after it has been used and encoded back into the body field.

This step optimizes the JSON data structure by re-encoding after cleaning it, ensuring only necessary data proceeds.

5. Compounded Processing

Logs flow from the Reserialize body node to the Processed pack output.

Sample Input

{"records":[{"time":"2019-01-21T22:14:26.9792776Z","resourceId":"/subscriptions/s1/resourceGroups/MSSupportGroup/providers/microsoft.support/supporttickets/115012112305841","operationName":"microsoft.support/supporttickets/write","category":"Write","resultType":"Success","resultSignature":"Succeeded.Created","durationMs":2826,"callerIpAddress":"111.111.111.11","correlationId":"aaaa0000-bb11-2222-33cc-444444dddddd","identity":{"authorization":{"scope":"/subscriptions/s1/resourceGroups/MSSupportGroup/providers/microsoft.support/supporttickets/115012112305841","action":"microsoft.support/supporttickets/write","evidence":{"role":"Subscription Admin"}},"claims":{"aud":"https://management.core.windows.net/","iss":"https://sts.windows.net/72f988bf-86f1-41af-91ab-2d7cd011db47/","iat":"1421876371","nbf":"1421876371","exp":"1421880271","ver":"1.0","http://schemas.microsoft.com/identity/claims/tenantid":"ffffffff-eeee-dddd-cccc-bbbbbbbbbbb0","http://schemas.microsoft.com/claims/authnmethodsreferences":"pwd","http://schemas.microsoft.com/identity/claims/objectidentifier":"2468adf0-8211-44e3-95xq-85137af64708","http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn":"admin@contoso.com","puid":"20030000801A118C","http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier":"9vckmEGF7zDKk1YzIY8k0t1_EAPaXoeHyPRn6f413zM","http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname":"John","http://schemas.xmlsoap.org/ws/2005/05/identity/claims/surname":"Smith","name":"John Smith","groups":"cacfe77c-e058-4712-83qw-f9b08849fd60,7f71d11d-4c41-4b23-99d2-d32ce7aa621c,31522864-0578-4ea0-9gdc-e66cc564d18c","http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name":" admin@contoso.com","appid":"00001111-aaaa-2222-bbbb-3333cccc4444","appidacr":"2","http://schemas.microsoft.com/identity/claims/scope":"user_impersonation","http://schemas.microsoft.com/claims/authnclassreference":"1"}},"level":"Information","location":"global","properties":{"statusCode":"Created","serviceRequestId":"50d5cddb-8ca0-47ad-9b80-6cde2207f97c"}}]}