Secrets

Securely store and manage sensitive credentials for use in Edge Delta pipelines.

  5 minute read  

Every pipeline that connects to external systems needs credentials. API keys for Datadog, HEC tokens for Splunk, access keys for AWS—these sensitive values must live somewhere in your configuration. The question is how to store them securely.

Hardcoding credentials directly in configuration files creates obvious problems. Anyone with access to the configuration can see the credentials. Version control systems preserve them in history forever. Log files might accidentally expose them during debugging.

Edge Delta’s secrets management solves this by storing credentials separately from configuration. You create a secret once, give it a name, and reference that name wherever you need the credential. The actual value is encrypted at rest and only decrypted at runtime when the agent needs it.

How secrets work

When you create a secret, Edge Delta encrypts the value using AES-256 encryption. The encrypted value is stored in the pipeline configuration, but the original plaintext is never retrievable—not through the API, not in logs, not anywhere.

To use a secret, you reference it by name when configuring a node. At runtime, the agent resolves these references and passes the decrypted values directly to the integration. The plaintext never appears in logs, even at debug level.

The pipeline YAML contains a secrets section where encrypted values are stored:

secrets:
  splunk-hec-token:
    value: <encrypted-value>
    description: Production Splunk HEC token

Because the values are encrypted, this section is safe to store in version control. You cannot derive the original plaintext from the encrypted value.

Note: The secrets section is managed by the Edge Delta platform. In default encryption mode, secrets must be created through the GUI—the platform encrypts values when you save them. If you manually add a plaintext value to the secrets section, the agent will fail to start with a “no secret version found” error. To manage secrets outside the GUI, use master key mode with the edctl CLI to encrypt values before adding them to your configuration.

Where secrets are stored

Regardless of which encryption mode you use, the encrypted secret values are stored in the same place: the pipeline configuration YAML. The secrets section of your pipeline contains the encrypted values, and this configuration lives in the Edge Delta platform.

What differs between encryption modes is not where secrets are stored, but how they are encrypted—specifically, where the encryption key comes from.

Encryption modes

Edge Delta supports two secret encryption modes designed to meet different operational needs. Both modes securely protect sensitive values and store them in a consistent pipeline configuration format, but they differ in how encryption is managed and where it occurs.

Default mode provides a fully managed experience where secrets are created and managed through the Edge Delta web interface. Encryption and decryption are handled transparently by the platform, requiring no additional pipelines or infrastructure. This mode is ideal for simple, UI-driven workflows with minimal setup.

Master key mode offers a stronger security model and is designed for advanced or security-sensitive environments. The coordinator pipeline must be deployed and available for encryption and decryption operations in this mode. While this introduces additional setup compared to default mode, it enables a higher level of security and operational control. This approach enables the edctl CLI tool to encrypt secrets outside the GUI, which opens up possibilities that default mode cannot support: batch operations that create dozens of secrets at once, CI/CD pipelines that manage secrets programmatically, and air-gapped environments where the GUI is not accessible.

In both modes, encrypted secrets are stored in the same pipeline configuration format. The difference is operational and architectural: default mode emphasizes ease of use, while master key mode prioritizes security, isolation, and automation.

The choice between modes depends on your requirements. If ease of management through the UI is the primary concern, default mode is sufficient. If stronger security guarantees, reduced exposure, or automated secret management are required, master key mode is the recommended approach.

What to store as secrets

Not every configuration field supports secrets—only fields specifically designed for sensitive values. In the pipeline builder, these fields display a Use Secret option. Common examples include AWS access keys and secret keys, API tokens for platforms like Datadog and Splunk, database passwords, and authentication credentials.

The principle is straightforward: if a field offers secret support and the value would compromise security if exposed, store it as a secret.

Important: Using the {{ SECRET secret-name }} syntax on a field that doesn’t support secrets will not work. The agent only resolves secret references on fields that display the Use Secret option in the GUI. On unsupported fields, the literal string {{ SECRET secret-name }} is passed through unchanged, which typically causes configuration errors at runtime.

Security model

Secrets are scoped to individual pipelines. A secret created for one pipeline cannot be accessed by another, providing isolation between environments and teams.

Access control follows pipeline permissions. Only users who can edit a pipeline can view or manage its secrets. Even then, they see only that secrets exist—the actual values are never displayed after creation.

The runtime behavior is designed to prevent accidental exposure. Secret values are resolved only when needed, passed directly to integrations without intermediate storage, and excluded from all logging regardless of log level.

When not to use secrets

While secrets are the recommended approach for most credentials, alternatives exist for specific scenarios.

For agents running in AWS environments like EKS or EC2, IAM Roles for Service Accounts (IRSA) or instance profiles can provide credentials without storing static keys. This approach is often preferable because credentials rotate automatically and never exist as storable values.

Some integrations can read credentials from environment variables or mounted files. These approaches make sense when your deployment infrastructure already manages secrets through systems like Kubernetes Secrets or HashiCorp Vault.

The specific authentication options available depend on the integration. Check the source or destination documentation for details.

See Also