Edge Delta Log Search

Log search in the Edge Delta web application.

9 minute read

Overview

Edge Delta’s Search is powered by the Common Query Language (CQL), a simple and powerful way to get answers from your logs.

Recap

Edge Delta Fleets send important logs and triggers to your configured data destinations, trigger endpoints, and archive destinations. To enable the Search function, it also archives logs in the Edge Delta back end, in customer specific s3 buckets. You can search this log archive using the search feature in Edge Delta. See here for more details about the architecture.

Flattening

When logs are archived, attributes are flattened to improve search performance. This may result in inconsistent application of a schema, particularly if custom field names containing dots have been used. Consider the following example:

"alpha": {
  "beta": "gamma"
}

This would be configured in the pipeline and testing as follows:

item["alpha"]["beta"]

Now consider a dot separated field name:

"delta.epsilon": "zeta"

This would be configured in the pipeline and testing as follows:

item["delta.epsilon"]

With flattening, these examples would be archived as follows:

alpha.beta: "gamma"
delta.epsilon: "zeta"

Sampling

From 3 hours of age, histogram and facet option counts are sampled so that log queries are performant. Logs themselves, however, are all stored and indexed.

Search for Logs

To search for logs you click Logs and select Search.

Autocomplete suggests search strings based on your attribute names and values. By tabbing through the suggestions can quickly explore the data and see the frequency of values in the autocomplete suggestions.

You enter a search string in the Search box and click Search. You can cancel a query by clicking X.

See the Search Syntax.

You can also filter results using the Filters pane. Selecting all the checkboxes shows results from all the selected options. You can choose one specific filter, such as a Namespace to view results from only that namespace. If you select none of the filters, no results will be shown.

The query results are listed along with a chart. You can click and drag a time period on the histogram to filter the lookback period on it.

Search for Attributes

You can search for attributes using the @ character. For example, consider this log:

There is an attribute field anothernewfield. You can search for all logs with a particular value for this attribute by adding @anothernewfield:"parallelcluster-control-plane" to the search query:

Alternatively, you can click an attribute and select Add to Search:

Click the Share button to copy a shortened URL to the clipboard that you can share with others to view the same query. If you have a relative Lookback filter selected, such as the previous 15 minutes, the same relative duration will be shared. This means that someone viewing the shared query will see the previous 15 minutes from the moment they open the query. If you shared a query with a custom lookback period, the same period will be shared. Someone viewing this type of query will view the same log results from that particular time window, for example during a critical outage event.

Download Results

Click the Download button to download up to 20,000 rows of the log search page as a CSV file or a JSON formatted file.

If you select CSV, only the currently visible columns are downloaded. If you select JSON, all columns are downloaded.

Save Queries

Click the Save Query button next to the search button to save your query. You provide a name and click Save.

Saved queries and queries shared by users in your organization are listed in the Saved tab:

View Log Options

Click Settings and select the Expanded option to adjust the search results display method:

Expanded shows the search results in an expanded style with the default line breaks and indentation:

Click a log in the search results to see more details about it in a side pane.

You can add a custom facet to the filter widget. Click a log in the log search pane to open it. Select a tag and click Create facet.

The new facet is added to a Custom group in the filter widget.

Group by

You can group the graph by any of the facets listed in the Filter Pane using the Group By option. For example, you may want to see logs with a particular attribute from all sources, and viewed as series on a single graph.

With a group selected you can view the table as a pie chart:

Unique Count

You can view a count of unique dimension values by selecting the dimension in the Unique count by list. For example, 14 pods have generated logs in the past 15 minutes.

Add Column

You can add a column to the search results pane. Click a log in the log search pane to open it. Select a tag and click Add Column.

You can also add a column from the filters widget.

Added columns will be included if you share the search results, and they will be included if they are visible and you download the results.

You can toggle the visible columns using the configure button.

Search Syntax

You can refine your search query using familiar search operators such as quotation marks and OR when using the basic search.

Full Text

By default, searches are run against the entirety of a log’s message and are not case sensitive. All queries below will return any log with a message containing the term ERROR:

error
Error
erroR

Full text search also supports partial matching, where a query for fail will return log messages containing:

FAILED
fails
failure

It is useful to begin with a broad search strategy before refining results using facet filters or field-based searching.

Key Value Pairs

You can search for values using a colon in between the key and the value. Keys can only be a term, whereas values can be a term, quoted term or a grouped expression. For the full text search (searching for the body of the log and not the facets), terms and quoted terms will be matched based on whether they are contained in the log or not. For the key value pair search, terms and quoted terms will be matched only if they exactly match. This behavior can be changed using wildcards.

ed.tag: default
host.name:(default OR admin)
namespace: "(this ) is : \n a \t= namespace"

Comparison Operators

You can use comparison operators <, >, <=, or >= on numerical attributes. The values must be in attributes to use comparison operators. You search for an attribute using @ and values can be integers or floating points.

for integer comparison search

@latency < 300.1

for floating point comparison search

@latency < 300.1

Hint: You can extract a value from the log body and use it an an attribute using the Grok parsing node or the Log Transform node with a regex_capture CEL macro.

Exact Match

If your search contains whitespace, you must wrap your search string in quotation marks. This makes the search treat the string explicitly and it only returns exact matches. Bear in mind wrapping a search string in quotes enforces case sensitivity.

"user postgres"

Without quotes this search would be treated as user AND postgres. If you need to search log contents that contain a search operator such as a bracket ) or the word OR, you can wrap them in quotation marks to be treated explicitly as search terms and not search operators.

Special characters include , \t, \n, \r, \u3000, !, (, ), :, ^, @, <, >, =, [, ], \``, {, }, ~, \, /. Terms also can't start with the following two characters unless they are escaped by a backslash: +, -`.

Inside quotation marks, three characters need escaping if they are needed in the search, ", * and \.

The asterisk cannot be treated explicitly using quotation marks because it is used to extend exact matches. See the Wildcard section.

You can wrap part of the query in quotes:

exception:"*object reference not set*"

This search will return all results with an exception field that includes the phrase “object reference not set” anywhere in its value

Boolean Operator AND

You can search for logs that must contain two or more specific keywords with the Boolean operator AND (case sensitive). The default behavior of the search engine when you leave a space between keywords is search for only logs with both words. Logs that contain both these words in any field and in any order are returned. Consider the following search strings:

failed post

All logs that contain both of these words are returned, such as the following:

failed password for invalid user postgres

error AND env:prod

Returns all results where error is present in log message and env attribute contains prod (exact match as there is no whitespace between env: and prod)

Boolean Operator OR

You can search for alternative words using the Boolean OR operator (case sensitive) to return results for either of the search keywords, but not necessarily both of them.

"invalid user" OR "user unknown"

This example will return all logs that contain the phrase invalid user as well as any logs that contain user unknown.

level:(error OR warn) database connection

Returns all results where level field contains error or warn, and the log message/body contains the terms database and connection, regardless of the ordering of or distance between terms.

Operator NOT

You can add an exclusion to your search string to ignore certain logs using the minus parameter or the NOT term. This search looks for failed HTTP processing requests except those that contain the word GET:

"Failed Processing HTTP request" -GET

env:prod NOT level:(debug OR info)

This returns all records where env attribute is equal to prod, exclude all records where level attribute is equal to debug or info.

Wildcards

You can use an asterisk character as a wildcard to return a wider set of results. A wildcard is only useful when you are searching within quotation marks to extend an exact match. For example, "statusCode=*04" will return both 504 and 404 logs. Without quotation marks, the wildcard is implied. For example, post will return POST as well as postgres results. To search for the asterisk character you can escape the wildcard in the search string using a backslash \*.

For key value pair search, wildcards can only be used at the beginning or at the end of the value and they are used to change the behavior of key value pair search to contains search (instead of exact match). Some examples:

ed.tag:def

This will only match the logs that has the “ed.tag“ value of “def“.

ed.tag:*def

This will match any log that ends with “def”.

ed.tag:*def*

This will match any log that contains “def“.

Processing Precedence with Brackets

By default the search engine will process search strings with more than one operator in the following order:

Exact Match
Exclude
And
OR

For example, given the following search string:

one two OR three

Two types of logs will be returned:

any logs containing both one and two
any logs containing three, including those without any incidence of one.

You can adjust the processing order of precedence using brackets just the same as in mathematical equations. For example, given the following search string:

one (two OR three)

The logs returned will all contain a one. In addition they will all either contain a two or a three.