Configure Persisting Cursor in Edge Delta

Overview

A persisting cursor in Edge Delta is a feature that remembers the last read position in a log file even after an agent restart. This is useful in scenarios where data consistency is crucial, such as environments where data loss during agent restarts must be prevented.

Configuration

There are two methods of configuring persistent cursor. You can configure it for a single file source node, or for the entire fleet.

You enable a persisting cursor with the enable_persisting_cursor parameter in a file source node to allow the system to continue reading from the last processed entry in a log file upon restart. This parameter is a Boolean with the default value of false.

nodes:
  - name: <node name>
    type: file_input
    path: "/var/logs/app/*.log"
    enable_persisting_cursor: true    

Alternatively, you can configure a persisting cursor at the fleet level in the agent settings section. This means all inputs that tail a log file will benefit from a persisting cursor. This option also gives you control over additional settings:

settings:
  tag: prod
  persisting_cursor_settings:
    path: /var/edgedelta/pos
    file_name: cursor_file.json
    flush_interval: 1m

• path is the folder where the cursor file will be created.
• file_name is the name of the cursor file.
• flush_interval is the interval after which the file will be saved to from memory.