Edge Delta File inputs

Files for Edge Delta to monitor.

3 minute read

Version 2 is no longer supported. This section is for historical reference only.

See the latest version here.

Overview

This input type allows you to specify a set of files for Edge Delta to monitor.

Example

files:
    - labels: "billing,errorcheck"
      path: "/billing/logfolder1/*.log"
    - labels: "billing,errorcheck"
      path: "/etc/systemd/system/billingservice/*.log"
      exclude:
        - "/etc/systemd/system/billingservice/test.log"
        - "/etc/systemd/system/billingservice/dev.log"
      auto_detect_line_pattern: true
      boost_stacktrace_detection: true
      enable_persisting_cursor: true
      filters:
        - info
        - not_trace
        - mask_card
        - mask_password
    - labels: "docker,my_container"
      path: "/var/lib/docker/my_container/*.log"
      docker_mode: true
    - labels: "app,service_a"
      path: "/var/log/service_a.log"
      line_pattern: ^\d{4}-\d{2}-\d{2}
      late_arrival_handling:
        rule_metrics:
          ignore_after: 15m
        patterns:
          ignore_after: 4h
          report_with_original_timestamp: true
      filters:
      - source-detection-docker
    - labels: k8s_log
      path: /var/logs/anyDir/MyApp/users/MyPodID/transaction.log
      filters:
      - enrichment-full
    - labels: "billing,errorcheck"
      path: "/billing/logfolder1/*.log"
      separate_source: true  
    - labels: "billing,errorcheck"
      path: "/billing/logfolder1/*.log"
      add_ingestion_time: true
      skip_ingestion_time_on_failure: true

Required Parameters

labels

Enter a descriptive label for this input. When you create a workflow, you will use this label to enter your input into the workflow.

inputs:
  files:
    - labels: "<input name>"

path

Enter the full path to the file (or files) that you want Edge Delta to monitor. Wildcards are supported:

/etc/systemd/system/billingservice/*.log includes all .log files in the billingservice folder.
/etc/systemd/system/billingservice/**/*.log includes all .log files in the billingservice folder and any sub-directories.

inputs:
  files:
    - labels: "<input name>"
      path: "<path to files>"

Optional Parameters

exclude

Specify a list of regexes that will be run against glob path’s captures and will be discarded if there is a match.

inputs:
  files:
    - labels: "<input name>"
      path: "/etc/systemd/system/billingservice/*.log"
      exclude:
        - "/etc/systemd/system/billingservice/test.log"
        - "/etc/systemd/system/billingservice/dev.log"

auto_detect_line_pattern

Detects line patterns automatically based on the Ragel FSM Based Lexical Recognition process. There is no need to specify line_pattern explicitly.

inputs:
  files:
    - labels: "<input name>"
      path: "<path to files>"
      auto_detect_line_pattern: true

line_pattern

To specify a line pattern, you can use the line_pattern parameter with a regex rule to match the line break. The agent will process lines using this specific line separation rule, rather than using a new line("\n").

inputs:
  files:
    - labels: "<input name>"
      path: "<path to files>"
      line_pattern: <regex pattern>

boost_stracktrace_detection

Enter true to fallback (troubleshoot) missing content that resulted from the Auto Line Detection parameter. Stack trace detector only runs in auto line detection mode.

inputs:
  files:
    - labels: "<input name>"
      path: "<path to files>"
      auto_detect_line_pattern: true
      boost_stacktrace_detection: true

separate_source

Enter true to have the agent treat each file that matches the glob expression as a separate source. By default, this option is disabled; all files that match the glob path will be treated as a single source.

inputs:
  files:
    - labels: "<input name>"
      path: "<path to files>"
      separate_source: true

filters

Enter an existing filter to add to this input. To learn how to create a filter, see Filters.

inputs:
  files:
    - labels: "<input name>"
      path: "<path to files>"
      filters:
        - info

add_ingestion _time

Enter true to ingest a timestamp if the input format is in JSON.

inputs:
  files:
    - labels: "<input name>"
      path: "<path to files>"
      add_ingestion_time: true

skip_ingestion _time_on_failure

Enter true to skip the ingestion of the timestamp when the input is broken or in an invalid format.

inputs:
  files:
    - labels: "<input name>"
      path: "<path to files>"
      skip_ingestion_time_on_failure: true

late_arrival _handling

You can use this parameter to configure how to accommodate delayed data. Specifically, you can use the ignore_after parameter to configure when to ignore metrics or pattern logs. For example, for ignore_after, if you enter 15m, then the input will ignore logs whose timestamp is older than 15 minutes.

inputs:
  files:
    - labels: "<input name>"
      path: "<path to files>"
      late_arrival_handling:
        rule_metrics:
          ignore_after: 15m
        patterns:
          ignore_after: 4h
          report_with_original_timestamp: true

docker_mode

Enter true to collect Docker container standard output logs on a file with the JSON file logging driver. To learn more, review this document from Docker.