Troubleshooting Secrets

Troubleshoot common issues with Edge Delta secrets management, including GUI and CLI-based workflows.

  4 minute read  

This page covers common issues with Edge Delta secrets management and their solutions.

GUI Secrets Issues

Secret Reference Not Resolving

Symptom: Source or destination shows authentication errors, or the literal string '{{ SECRET secret-name }}' appears in logs.

Resolution:

  • Verify the secret name matches exactly (case-sensitive)
  • Ensure the secret reference is enclosed in single quotes: '{{ SECRET secret-name }}'
  • Check that the secret exists in the correct pipeline
  • Verify the agent has received the updated configuration

Authentication Failures After Secret Update

Symptom: Source or destination authentication fails after updating a secret value.

Resolution:

  • Verify the new secret value is correct
  • Check the integration’s authentication requirements (format, encoding, etc.)
  • Review agent logs for specific error messages
  • Ensure the agent has reloaded the configuration (may take 1-2 minutes in coordinator mode)

Secret Not Available in Dropdown

Symptom: Secret doesn’t appear in the GUI dropdown when configuring a field.

Resolution:

  • Ensure the secret was created and saved successfully
  • Refresh the browser page
  • Verify you’re viewing the correct pipeline
  • Check that you have sufficient permissions to view secrets

Secret Changes Not Taking Effect

Symptom: Updated or new secrets are not being used by the agent.

Resolution:

  • Deploy the pipeline configuration after making secret changes
  • Wait 1-2 minutes for the agent to receive the updated configuration
  • Check coordinator logs to verify configuration distribution
  • Restart the agent if the issue persists

edctl CLI Issues

Connection Issues

Problem: failed to connect to coordinator

Resolution:

Verify the port-forward is active:

lsof -i :5555

Check the coordinator pod is running:

kubectl get pods -n edgedelta | grep coordinator

Verify port-forward command syntax:

kubectl port-forward -n <namespace> <pod-name> 5555:5555

If the port-forward keeps disconnecting, check for network issues or use a stable connection.

Authentication Errors

Problem: org_id parameter is required or pipeline-id is required

Resolution:

  • Verify you have the correct organization ID
  • Confirm the pipeline ID (API key) is valid
  • Check that you’re using the correct flags or environment variables:
# Using flags
edctl secrets create \
  --org-id=<organization-id> \
  --pipeline-id=<pipeline-id> \
  --file=encrypted.jsonl

# Using environment variable
export ED_API_KEY=<pipeline-id>
edctl secrets create \
  --org-id=<organization-id> \
  --file=encrypted.jsonl

Secret Already Exists

Problem: Secret with ID 'xxx' already exists

Resolution:

Use the update command instead of create when modifying existing secrets:

edctl secrets update \
  --org-id=<org-id> \
  --pipeline-id=<pipeline-id> \
  --file=encrypted.jsonl

Secret Not Found

Problem: Secret with ID 'xxx' does not exist

Resolution:

  • Check for typos in the secret ID
  • Use the create command instead of update for new secrets:
edctl secrets create \
  --org-id=<org-id> \
  --pipeline-id=<pipeline-id> \
  --file=encrypted.jsonl

Invalid File Format

Problem: unsupported file format: .csv, expected one of: .jsonl

Resolution:

Ensure your input file:

  • Uses the .jsonl extension
  • Contains valid JSON (one object per line)
  • Has the correct field names:
{"secret_id":"my-secret","value":"plain-text-value"}

For encrypted files:

{"secret_id":"my-secret","encrypted":"edk1_co6MwY568h..."}

Invalid Secret Format

Problem: Invalid secret format for secret 'xxx'

Resolution:

  • Verify the encrypted value starts with edk1_
  • Re-encrypt the secret using edctl
  • Check that the encrypted value wasn’t truncated or modified
  • Ensure you copied the complete encrypted value

Decryption Fails

Problem: decryption failed when the coordinator processes secrets

Resolution:

  • Verify the secret was encrypted with the correct coordinator’s master key
  • Confirm the secret wasn’t manually edited after encryption
  • Check that you’re using the same coordinator that generated the encryption
  • Re-encrypt and update the secret:
# Re-encrypt
edctl secrets encrypt \
  --coordinator-addr=localhost:5555 \
  --secret-id=<secret-id> \
  --value="<new-value>"

# Update
edctl secrets update \
  --org-id=<org-id> \
  --pipeline-id=<pipeline-id> \
  --file=encrypted.jsonl

Master Key Export Issues

Problem: Cannot export the master key

Resolution:

  • Ensure you have an active port-forward to the coordinator
  • Verify the coordinator has a PVC attached and is persisting the master key
  • Check coordinator pod logs for errors:
kubectl logs -n edgedelta <coordinator-pod> | grep -i "master key"

Offline Mode Not Working

Problem: Encryption fails when using --key-file

Resolution:

  • Verify the master key file exists and is readable
  • Check file permissions (should be 0400 or similar)
  • Ensure the master key was exported from the correct coordinator
  • Re-export the master key if necessary

Coordinator Issues

Coordinator Pod Not Running

Problem: No coordinator pod found when running kubectl get pods

Resolution:

  • Check the Helm deployment includes the coordinator component
  • Verify the namespace is correct
  • Check for pod errors:
kubectl describe pods -n edgedelta -l app=edgedelta-coordinator

Coordinator PVC Issues

Problem: Master key not persisting across coordinator restarts

Resolution:

  • Verify the PVC is properly configured in the Helm chart
  • Check PVC status:
kubectl get pvc -n edgedelta
  • Ensure the storage class supports persistent volumes
  • Check coordinator pod logs for storage-related errors

See Also