Multi-line Detection

Overview

You can group multiline logs into a single log message.

To better understand this concept, consider the following example of java stack trace:

Feb 09, 2021 3:23:29 PM com.google.devtools.search.cloud.feeder.MakeLog: RuntimeException: Run from this message!
    at com.my.app.Object.do$a1(MakeLog.java:50)
    at java.lang.Thing.call(Thing.java:10)
    at com.my.app.Object.help(MakeLog.java:40)
    at sun.javax.API.method(API.java:100)
    at com.jetty.Framework.main(MakeLog.java:30)

Without multiline processing, during collection, the above java stack trace will produce the following log lines for additional processing (parsing and ingesting):

log: {
Feb 09, 2021 3:23:29 PM com.google.devtools.search.cloud.feeder.MakeLog: RuntimeException: Run from this message!
}
log: {
    at com.my.app.Object.do$a1(MakeLog.java:50)
}
log: {
    at java.lang.Thing.call(Thing.java:10)
}
log: {
    at com.my.app.Object.help(MakeLog.java:40)
}
log: {
    at sun.javax.API.method(API.java:100)
}
log: {
    at com.jetty.Framework.main(MakeLog.java:30)
}

Enable Multiline Detection

There are 2 ways to convert multiple logs into a single log:

• Specify a line pattern or
• Enable auto line detection.

Specify a Line Pattern

To specify a line pattern, you can use the line_pattern regex rule in the agent configuration. The agent will process lines for this specific line separation rule, not for New Line("\n").

This rule accumulates the logs between these line patterns as multiline logs, and then further processes the logs based on the accumulated multiline logs.

  files:
    - path: "/var/log/service_a.log"
      labels: "app,service_a"
      line_pattern: "^MMM dd, yyyy hh:mm:ss"

Enable Auto Line Detection

This option detects line patterns automatically based on the Ragel FSM Based Lexical Recognition process.

There is no need to specify line_pattern explicitly.

This rule accumulates the logs between these line patterns as multiline logs, and then further processes the logs based on the accumulated multiline logs.

  kubernetes:
    - labels: "kubernetes_logs"
      include:
        - "namespace=.*"
      exclude:
        - "namespace=kube-system"
        - "namespace=kube-public"
        - "namespace=kube-node-lease"
        - "pod=edgedelta"
        - "kind=ReplicaSet"
      auto_detect_line_pattern: true

With this example configured, the following log lines will be collected and processed:

log: {Feb 09, 2021 3:23:29 PM com.google.devtools.search.cloud.feeder.MakeLog: RuntimeException: Run from this message!
    at com.my.app.Object.do$a1(MakeLog.java:50)
    at java.lang.Thing.call(Thing.java:10)
    at com.my.app.Object.help(MakeLog.java:40)
    at sun.javax.API.method(API.java:100)
    at com.jetty.Framework.main(MakeLog.java:30)
}