Multi-line Detection
2 minute read
Overview
You can group multiline logs into a single log message.
To better understand this concept, consider the following example of java stack trace:
Feb 09, 2021 3:23:29 PM com.google.devtools.search.cloud.feeder.MakeLog: RuntimeException: Run from this message!
at com.my.app.Object.do$a1(MakeLog.java:50)
at java.lang.Thing.call(Thing.java:10)
at com.my.app.Object.help(MakeLog.java:40)
at sun.javax.API.method(API.java:100)
at com.jetty.Framework.main(MakeLog.java:30)
Without multiline processing, during collection, the above java stack trace will produce the following log lines for additional processing (parsing and ingesting):
log: {
Feb 09, 2021 3:23:29 PM com.google.devtools.search.cloud.feeder.MakeLog: RuntimeException: Run from this message!
}
log: {
at com.my.app.Object.do$a1(MakeLog.java:50)
}
log: {
at java.lang.Thing.call(Thing.java:10)
}
log: {
at com.my.app.Object.help(MakeLog.java:40)
}
log: {
at sun.javax.API.method(API.java:100)
}
log: {
at com.jetty.Framework.main(MakeLog.java:30)
}
Enable Multiline Detection
There are 2 ways to convert multiple logs into a single log:
- Specify a line pattern or
- Enable auto line detection.
Specify a Line Pattern
To specify a line pattern, you can use the line_pattern
regex rule in the agent configuration. The agent will process lines for this specific line separation rule, not for New Line("\n").
This rule accumulates the logs between these line patterns as multiline logs, and then further processes the logs based on the accumulated multiline logs.
files:
- path: "/var/log/service_a.log"
labels: "app,service_a"
line_pattern: "^MMM dd, yyyy hh:mm:ss"
Enable Auto Line Detection
This option detects line patterns automatically based on the Ragel FSM Based Lexical Recognition process.
There is no need to specify line_pattern
explicitly.
This rule accumulates the logs between these line patterns as multiline logs, and then further processes the logs based on the accumulated multiline logs.
kubernetes:
- labels: "kubernetes_logs"
include:
- "namespace=.*"
exclude:
- "namespace=kube-system"
- "namespace=kube-public"
- "namespace=kube-node-lease"
- "pod=edgedelta"
- "kind=ReplicaSet"
auto_detect_line_pattern: true
With this example configured, the following log lines will be collected and processed:
log: {Feb 09, 2021 3:23:29 PM com.google.devtools.search.cloud.feeder.MakeLog: RuntimeException: Run from this message!
at com.my.app.Object.do$a1(MakeLog.java:50)
at java.lang.Thing.call(Thing.java:10)
at com.my.app.Object.help(MakeLog.java:40)
at sun.javax.API.method(API.java:100)
at com.jetty.Framework.main(MakeLog.java:30)
}