Search Logs in Edge Delta

Overview

Edge Delta’s Search is powered by the Common Query Language (CQL), a simple and powerful way to get answers from your logs.

Recap

Edge Delta Fleets send important logs and triggers to your configured data destinations, trigger endpoints, and archive destinations. To enable the Search function, it also archives logs in the Edge Delta back end, in customer specific s3 buckets. You can search this log archive using the search feature in Edge Delta. See here for more details about the architecture.

Flattening

When logs are archived, attributes are flattened to improve search performance. This may result in inconsistent application of a schema, particularly if custom field names containing dots have been used. Consider the following example:

"alpha": {
  "beta": "gamma"
}

This would be configured in the pipeline and testing as follows:

item["alpha"]["beta"]

Now consider a dot separated field name:

"delta.epsilon": "zeta"

This would be configured in the pipeline and testing as follows:

item["delta.epsilon"]

With flattening, these examples would be archived as follows:

alpha.beta: "gamma"
delta.epsilon: "zeta"

Sampling

From 3 hours of age, histogram and facet option counts are sampled so that log queries are performant. Logs themselves, however, are all stored and indexed.

Search for Logs

To search for logs you click Logs and select Search.

See Search Logs for more information.

Autocomplete suggests search strings based on your attribute names and values. By tabbing through the suggestions can quickly explore the data and see the frequency of values in the autocomplete suggestions.

You enter a search string in the Search box and click Search. You can cancel a query by clicking X.

You can also filter results using the Filters pane. Selecting all the checkboxes shows results from all the selected options. You can choose one specific filter, such as a Namespace to view results from only that namespace. If you select none of the filters, no results will be shown.

The query results are listed along with a chart. You can click and drag a time period on the histogram to filter the lookback period on it.

Search for Attributes

You can search for attributes using the @ character. For example, consider this log:

There is an attribute field anothernewfield. You can search for all logs with a particular value for this attribute by adding @anothernewfield:"parallelcluster-control-plane" to the search query:

Alternatively, you can click an attribute and select Add to Search:

Search Syntax

You can refine your search query using familiar search operators such as quotation marks and OR when using the basic search.

Full Text

By default, searches are run against the entirety of a log’s message and are not case sensitive. All queries below will return any log with a message containing the term ERROR:

error
Error
erroR

Full text search also supports partial matching, where a query for fail will return log messages containing:

FAILED
fails
failure

It is useful to begin with a broad search strategy before refining results using facet filters or field-based searching.

Key Value Pairs

You can search for values using a colon in between the key and the value. Keys can only be a term, whereas values can be a term, quoted term or a grouped expression. For the full text search (searching for the body of the log and not the facets), terms and quoted terms will be matched based on whether they are contained in the log or not. For the key value pair search, terms and quoted terms will be matched only if they exactly match. This behavior can be changed using wildcards.

ed.tag: default
host.name:(default OR admin)
namespace: "(this ) is : \n a \t= namespace"

Comparison Operators

You can use comparison operators <, >, <=, or >= on numerical attributes. The values must be in attributes to use comparison operators. You search for an attribute using @ and values can be integers or floating points.

for integer comparison search

@latency < 300.1 

for floating point comparison search

@latency < 300.1  

Hint: You can extract a value from the log body and use it an an attribute using the Grok parsing node or the Log Transform node with a regex_capture CEL macro.

Exact Match

If your search contains whitespace, you must wrap your search string in quotation marks. This makes the search treat the string explicitly and it only returns exact matches. Bear in mind wrapping a search string in quotes enforces case sensitivity.

"user postgres"

Without quotes this search would be treated as user AND postgres. If you need to search log contents that contain a search operator such as a bracket ) or the word OR, you can wrap them in quotation marks to be treated explicitly as search terms and not search operators.

Special characters include , \t, \n, \r, \u3000, !, (, ), :, ^, @, <, >, =, [, ], \, {, }, ~, \\, /. Terms also can’t start with the following two characters unless they are escaped by a backslash: +, -.

Inside quotation marks, three characters need escaping if they are needed in the search, ", * and \.

The asterisk cannot be treated explicitly using quotation marks because it is used to extend exact matches. See the Wildcard section.

You can wrap part of the query in quotes:

exception:"*object reference not set*"

This search will return all results with an exception field that includes the phrase “object reference not set” anywhere in its value

Boolean Operator AND

You can search for logs that must contain two or more specific keywords with the Boolean operator AND (case sensitive). The default behavior of the search engine when you leave a space between keywords is search for only logs with both words. Logs that contain both these words in any field and in any order are returned. Consider the following search strings:

failed post

All logs that contain both of these words are returned, such as the following:

failed password for invalid user postgres

error AND env:prod

Returns all results where error is present in log message and env attribute contains prod (exact match as there is no whitespace between env: and prod)

Boolean Operator OR

You can search for alternative words using the Boolean OR operator (case sensitive) to return results for either of the search keywords, but not necessarily both of them.

"invalid user" OR "user unknown"

This example will return all logs that contain the phrase invalid user as well as any logs that contain user unknown.

level:(error OR warn) database connection

Returns all results where level field contains error or warn, and the log message/body contains the terms database and connection, regardless of the ordering of or distance between terms.

Operator NOT

You can add an exclusion to your search string to ignore certain logs using the minus parameter or the NOT term. This search looks for failed HTTP processing requests except those that contain the word GET:

"Failed Processing HTTP request" -GET

env:prod NOT level:(debug OR info)

This returns all records where env attribute is equal to prod, exclude all records where level attribute is equal to debug or info.

-@my_attribute:""

This returns logs that contain a specific attribute my_attribute with any value (i.e. excluding those without the attribute, or where the attribute is empty).

Wildcards

You can use an asterisk character as a wildcard within quotation charaters to return a wider set of results. A wildcard is only useful when you are searching within quotation marks to extend an exact match. For example, statusCode:"*04" will return both 504 and 404 logs. Without quotation marks, the wildcard is implied. For example, post will return POST as well as postgres results. To search for the asterisk character you can escape the wildcard in the search string using a backslash \*.

For key value pair search, wildcards can only be used at the beginning or at the end of the value and they are used to change the behavior of key value pair search to contains search (instead of exact match). Some examples:

ed.tag:"def"

This will only match the logs that has the “ed.tag“ value of “def“.

ed.tag:"*def"

This will match any log that has the “ed.tag“ value that ends with “def”.

ed.tag:"*def*"

This will match any log that has the “ed.tag“ value that contains “def“.

@my_attribute:"*"

This will match any logs with the attribute my_attribute, including those without a value.

Processing Precedence with Brackets

By default the search engine will process search strings with more than one operator in the following order:

1. Exact Match
2. Exclude
3. And
4. OR

For example, given the following search string:

one two OR three 

Two types of logs will be returned:

1. any logs containing both one and two
2. any logs containing three, including those without any incidence of one.

You can adjust the processing order of precedence using brackets just the same as in mathematical equations. For example, given the following search string:

one (two OR three)

The logs returned will all contain a one. In addition they will all either contain a two or a three.