Persistence

Configure persistent cursor for maintaining state across agent restarts.

  3 minute read  

This page covers Helm values for persistent cursor configuration.

Overview

The persistent cursor feature remembers the last read position in log files across agent restarts, ensuring no data loss during pod failures, updates, or reschedules. When enabled, Edge Delta agents save file read positions to persistent storage (host path or PersistentVolumeClaim), allowing them to resume processing from where they left off rather than re-reading entire files or missing new data.

How it works:

  1. As the agent reads log files, it tracks the current position (byte offset) for each file
  2. These positions are periodically flushed to a cursor file in persistent storage
  3. When the agent restarts, it reads the cursor file and resumes from the saved positions
  4. This prevents duplicate processing and ensures data consistency

When to use persistent cursor:

  • Mission-critical environments where data loss during agent restarts is unacceptable
  • Compliance and audit scenarios requiring guaranteed log processing without gaps
  • High-value data streams where duplicate processing would cause issues (e.g., billing, transactions)
  • Frequent pod updates in environments with aggressive deployment schedules

When NOT to use persistent cursor:

  • Ephemeral or testing environments where data consistency is not critical
  • High-throughput scenarios where the I/O overhead of cursor persistence could impact performance
  • Stateless architectures preferring simpler agent deployments without persistent storage dependencies

See Troubleshooting Persisting Cursor for detailed configuration guidance and File Source for node-level cursor settings.

Configuration

Persisting Cursor Properties - Enabled

Variable: persistingCursorProps.enabled

Description: Enables or disables the persistent cursor feature.

Example: persistingCursorProps.enabled=false

Persisting Cursor Properties - Container Mount Path

Variable: persistingCursorProps.containerMountPath

Description: The container mount path to keep the persisting cursor state.

Example: /var/lib/edgedelta

Persisting Cursor Properties - Host Mount Path

Variable: persistingCursorProps.hostMountPath

Description: The host mount path to keep the persisting cursor state.

Example: /var/lib/edgedelta

Default Mount Paths

The Helm chart configures the following default mount paths:

PropertyDefault Value
persistingCursorProps.hostMountPath/var/lib/edgedelta
persistingCursorProps.containerMountPath/var/lib/edgedelta

These defaults create a volume mount that maps /var/lib/edgedelta on the host node to /var/lib/edgedelta inside the container.

Pipeline Configuration Requirements

When configuring persisting_cursor_settings.path in your pipeline configuration, the path must be within the container mount path for cursor data to persist across pod restarts.

Valid configuration: The path is within the mounted volume.

settings:
  persisting_cursor_settings:
    path: /var/lib/edgedelta

Invalid configuration: The path /var/edgedelta/pos is not within the mounted volume /var/lib/edgedelta, so cursor data will be lost when the pod restarts.

settings:
  persisting_cursor_settings:
    path: /var/edgedelta/pos  # Will not persist - outside mounted volume

See Troubleshooting Persisting Cursor for more details on Kubernetes-specific configuration and Pipeline Settings for the GUI configuration options.