Troubleshooting Linux Installation

This page provides troubleshooting steps for the Edge Delta Pipeline on Linux, including process management and SELinux configuration.

  6 minute read  

This page provides a walkthrough for diagnosing and resolving issues related to the Edge Delta deployment on Linux platforms. It covers viewing and verifying the agent’s configuration, process status, and logging details for both Upstart and Systemd-based systems. It includes steps for restarting, starting, and stopping the Edge Delta process, ensuring smooth operations by checking system information through top or htop, and adjusting SELinux settings if necessary. The document also addresses advanced troubleshooting techniques such as analyzing memory limits and collecting heap dumps for performance profiling.

View the Agent’s Configuration

The agent fetches its pipeline configuration from the Edge Delta API at startup. To verify the agent is configured correctly, check the following local files:

API key:

cat /opt/edgedelta/agent/apikey

Environment variables:

cat /opt/edgedelta/agent/environment

To view the pipeline configuration, use the Edge Delta web application or the Configuration API.

View the Process Configuration

Use the less command to view the configuration file.

  • For Upstart-based systems such as RHEL, Amazon Linux, etc.:
less /etc/init/edgedelta.conf
  • For Systemd-based systems such as Debian, Ubuntu, etc.:
less /etc/systemd/system/edgedelta.service

Get the Process Status

Use the status command to view the edgedelta process status:

  • For Upstart-based systems such as RHEL, Amazon Linux, etc.:
initctl status edgedelta
  • For Systemd-based systems such as Debian, Ubuntu, etc.:
systemctl status edgedelta
  • For older systems with init:
sudo /etc/init.d/edgedelta status
  • For older Ubuntu systems:
sudo service edgedelta status

Restart the Process

To restart the edgedelta process use the restart command:

  • For Upstart-based systems such as RHEL, Amazon Linux, etc.:
initctl restart edgedelta
  • For Systemd-based systems such as Debian, Ubuntu, etc.:
systemctl restart edgedelta

To ensure the process was restarted, first capture its PID:

pgrep edgedelta

or

ps aux | grep edgedelta

Run the command to restart the process and check the PID of the process again.

Start and Stop the Process

To start or stop the edgedelta process use one of the following commands:

  • For Upstart-based systems such as RHEL, Amazon Linux, etc.:
initctl start edgedelta
initctl stop edgedelta
  • For Systemd-based systems such as Debian, Ubuntu, etc.:
systemctl start edgedelta
systemctl stop edgedelta

Get the Process’ System Information

Information can be gathered by either using top or htop. The top command will give you an isolated view of the process:

top -p $(pgrep edgedelta)

If htop is installed: htop then press / and type edgedelta.

For Upstart-based systems such as RHEL, Amazon Linux, etc. Messages from the initialization script can give you more information. They are logged every time the system is reloaded, the process is started or restarted, and when the configuration is changed.

tail -F /var/log/messages

If there is not enough information check the log level set:

initctl log-priority

You can check available levels:

initctl log-priority help

For troubleshooting, you should use the finest level of detail:

$ initctl log-priority debug

Permissions and Port Access Issues

Elevated Permissions Requirements

The Edge Delta agent requires elevated permissions (root/sudo) in the following scenarios:

  1. Low Port Numbers: When using TCP, UDP, or HTTP input nodes with port numbers between 0 and 1024, the agent must be installed with root privileges. Without elevated permissions, data will not flow through these ports.

  2. Cross-Account Source Detection: To detect and collect logs from all user and service accounts on the system, the agent must be installed with sudo/root privileges. Without elevated permissions, the agent can only access logs from the account that installed it.

Example Installation with Elevated Permissions:

sudo ED_API_KEY=<your-api-key> bash -c "$(curl -L https://release.edgedelta.com/release/install.sh)"

If you encounter port binding errors or missing log sources, verify that:

  • The agent was installed with appropriate privileges
  • The systemd service is running with the correct permissions
  • Port numbers above 1024 are used if root access is not available

SELinux Troubleshooting

Understanding SELinux Execution Errors

If the Edge Delta service fails to start with status=203/EXEC and you see errors like the following, SELinux is blocking the binary from executing:

edgedelta.service: Failed to execute command: Permission denied
Failed at step EXEC spawning /opt/edgedelta/agent/edgedelta: Permission denied

This commonly occurs after agent upgrades because the new binary inherits an incorrect SELinux context.

Diagnose SELinux Issues

Run the following commands to confirm SELinux is causing the problem:

  1. Check if SELinux is enforcing:
getenforce
  1. Inspect the binary’s SELinux context:
ls -Z /opt/edgedelta/agent/edgedelta

If the type is not bin_t, SELinux will block execution.

  1. Check for recent SELinux denials:
ausearch -m avc -ts recent | grep edgedelta
  1. Get human-readable denial explanations:
ausearch -m avc -ts recent | audit2why

Fix SELinux Context

Apply the correct SELinux type to allow execution:

sudo chcon -t bin_t /opt/edgedelta/agent/edgedelta
sudo systemctl restart edgedelta.service

Verify the fix:

ls -Z /opt/edgedelta/agent/edgedelta
systemctl status edgedelta

Prevent SELinux Issues After Upgrades

The chcon fix is temporary. When the binary is replaced during upgrades, it loses its SELinux context. To make the fix permanent:

  1. Create a persistent file context rule:
sudo semanage fcontext -a -t bin_t "/opt/edgedelta/agent/edgedelta"
  1. Apply the context:
sudo restorecon -v /opt/edgedelta/agent/edgedelta

After creating this rule, running restorecon will always set the correct context, even after upgrades.

Note: Add restorecon -rv /opt/edgedelta/ to your upgrade scripts to ensure the correct context is applied after each upgrade.

Create a Custom SELinux Policy Module

If the standard bin_t context does not resolve the issue, create a custom policy module:

  1. Generate a policy from the denials:
ausearch -m avc -ts recent | audit2allow -M edgedelta_policy
  1. Install the policy module:
sudo semodule -i edgedelta_policy.pp
  1. Restart the service:
sudo systemctl restart edgedelta.service

Confirm SELinux Is the Cause

To temporarily disable SELinux enforcement and confirm it is causing the issue:

sudo setenforce 0
sudo systemctl restart edgedelta.service
systemctl status edgedelta.service
sudo setenforce 1

If the service starts successfully with SELinux in permissive mode, SELinux is the cause. Apply one of the permanent fixes described above.

Additional Resources

For more information about SELinux administration and troubleshooting, see the official documentation:

Restart the Operating System

For Upstart-based systems such as RHEL, Amazon Linux, etc. The operating system needs to be restarted for any change to the configuration to take effect. For Systemd-based systems such as Debian, Ubuntu, etc. use

systemctl daemon-reload

View the Agent Log File

Check the agent’s log file for any errors that may indicate an issue with the agent, configuration, or deployment settings:

cat /var/log/edgedelta/edgedelta.log

Troubleshoot Memory Limits

The edgedelta service will restart if it reaches its memory limits.

If you notice that the service restarts frequently, then you may want to increase the limits. If you do not want to change your memory limits, then you can enable memory profiling and contact support with a heapdump from the profiler.

Run the following command for the current agent version (replace x.x.xx) to enable a more aggressive Garbage Collection and pprof profiler on port 6060. If another service uses port 6060, then you must use another port number in the bash and curl commands.

`sudo ED_API_KEY=$API_KEY ED_ENV_VARS="GOGC=30,PROFILER_PORT=6060" bash -c "$(curl -L https://release.edgedelta.com/vx.x.xx/install.sh)"`

Run the following command to obtain heapdump from the profiler. GOGC is 30. The default value is 100 for Golang. This setting will initiate a garbage collection job once the heap size has grown 30% since the last garbage collection.

curl http://localhost:6060/debug/pprof/heap --output heap