Ingest Logs from a File

Overview

The File Input node captures log input from specific files. It does this by tailing the configured files and capturing the changes made to them. The changes are sent as logs into the pipeline as they arrive in the file. This node is useful were there’s a need to read and process logs in real time but the system logs are written to flat files on disk, it is also useful for testing and troubleshooting.

System logs are commonly written to flat files on disk. If the logs are not already being sent over the network, it is simpler to ingest them from the file than set up a new process to send them over TCP or HTTP.

Prerequisites

To use a file input node you need the following prerequisites:

• An Edge Delta account with an agent configuration already created. This is the configuration in which you will create the File Input node.
• An in-cluster volume where the logs are being saved, and the cluster might need to be configured to mount external files depending on your architecture.

1. Create a Helm Override File

In this step you create a helm override values file to enable Edge Delta to access the in-cluster file locations.

1. Create a yaml file and name it appropriately, such as custom-values.yaml.
2. Add the following yaml to the file:

volumeProps:
  volumeMounts:
    - name: input-file
      mountPath: /mnt/inputfile/logs  
      readOnly: true   
  volumes:
    - name: input-file
      hostPath:
        path: /mnt/inputfile/logs
        type: DirectoryOrCreate

2. Install the Edge Delta Agent

Run the Helm deployment commands for the configuration:

1. In the Edge Delta App, click Pipelines, and then select Pipelines.
2. Click the pipeline you want to deploy and select Deploy Pipeline.
3. Select Helm.
4. Copy and run the first command provided to create the edgedelta namespace in the cluster.

helm repo add edgedelta https://edgedelta.github.io/charts

4. Copy and run the second command provided.

helm repo update

5. Copy the third command provided in the UI, and append the override values file. The command will look similar to this but with a different API key, note the -f custom-values.yaml:

helm upgrade edgedelta edgedelta/edgedelta -i --version v0.1.92 --set secretApiKey.value=123456789987654321 -n edgedelta --create-namespace -f custom-values.yaml

In this example the custom yaml file was called custom-values.yaml, replace it with the name of the file you created in step 1.

3. Create the File Input Node

1. In the Edge Delta App, click Pipeline, and then click Pipelines.
2. Select the pipeline you want to add the file input to.
3. Click Edit Mode
4. Click Add Input and select File Input.
5. Specify a Name for the file input node.
6. Specify the path containing the files you want to tail for logs. The path should match the one you specified in the Edge Delta custom manitest. The file name can use wildcards to tail a number of files.
7. Optionally, specify a filename pattern for files in that folder that should not be tailed for logs.
8. Click Okay.
9. Connect the file input to the first processor in the pipeline.

In this example, the files you want to tail for logs is captured with the following pattern: /mnt/inputfile/logs/*.*.

10. Click Review Changes.
11. Click Deploy Changes.

The Edge Delta pipeline will now ingest logs saved to any file in the /mnt/inputfile/logs location. By default, it creates one log per line. If your logs are split with another character such as a semi-colon instead of a line break, you can configure and add a Split Delimiter node to the pipeline.