Edge Delta DigitalOcean Spaces Output

Overview

The DigitalOcean Spaces Node send items to a DigitalOcean Spaces destination. The items are raw archive bytes buffered with the archive buffer processor.

Example Configuration

nodes:
  - name: my_dos
    type: digitalocean_spaces_output
    bucket: <REDACTED>
    access_key: <REDACTED>
    secret_key: <REDACTED>
    compression: zstd
    encoding: parquet
    use_native_compression: true
    path_prefix:
      order:
      - Year
      - Month
      - Day
      - Hour
      - 2 Minute
      - tag
      - host
      format: ver=parquet/year=%s/month=%s/day=%s/hour=%s/min=%s/tag=%s/host=%s/

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the yaml using the name. It must be unique across all nodes. It is a yaml list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: digitalocean_spaces_output

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

endpoint

The endpoint parameter defines the DigitalOcean Spaces URI. It is specified as a string and is required.

nodes:
  - name: <node name>
    type: digitalocean_spaces_output
    endpoint: <endpoint>
    bucket: <bucket>
    access_key: <key>
    secret_key: <secret key>

bucket

The bucket parameter defines the DigitalOcean Spaces target bucket to use. It is specified as a string and is required.

nodes:
  - name: <node name>
    type: digitalocean_spaces_output
    endpoint: <endpoint>
    bucket: <bucket>
    access_key: <key>
    secret_key: <secret key>

access_key

The access_key parameter defines the access key to authorize access to the bucket. It is specified as a string and is a required parameter.

nodes:
  - name: <node name>
    type: digitalocean_spaces_output
    endpoint: <endpoint>
    bucket: <bucket>
    access_key: <key>
    secret_key: <secret key>

secret_key

The secret_key parameter defines the secret key to authorize access to the bucket. It is specified as a string and is a required parameter.

nodes:
  - name: <node name>
    type: digitalocean_spaces_output
    endpoint: <endpoint>
    bucket: <bucket>
    access_key: <key>
    secret_key: <secret key>

Optional Parameters

archiver_enabled

The archiver_enabled parameter configures whether archiver agents will be used for sending archive bytes. It is specified as a Boolean with the default of false and it is optional.

nodes:
  - name: <node name>
    type: digitalocean_spaces_output
    endpoint: <endpoint>
    bucket: <bucket>
    access_key: <key>
    secret_key: <secret key>
    archiver_enabled: true

compression

The compression parameter specifies the compression format. It can be gzip, zstd, snappy or uncompressed. It is specified as a string, has a default of gzip, and it is optional.

nodes:
  - name: <node name>
    type: digitalocean_spaces_output
    endpoint: <endpoint>
    bucket: <bucket>
    access_key: <key>
    secret_key: <secret key>
    compression: gzip | zstd | snappy | uncompressed

disable_ssl

The disable_ssl parameter defines whether to use HTTP instead of HTTPS for accessing the bucket. It is specified as a Boolean and the default is false indicating that SSL is required. It is optional.

nodes:
  - name: <node name>
    type: digitalocean_spaces_output
    endpoint: <endpoint>
    bucket: <bucket>
    access_key: <key>
    secret_key: <secret key>
    disable_ssl: true

encoding

The encoding parameter specifies the encoding format. It can be json or parquet. It is specified as a string, has a default of json, and it is optional.

nodes:
  - name: <node name>
    type: digitalocean_spaces_output
    endpoint: <endpoint>
    bucket: <bucket>
    access_key: <key>
    secret_key: <secret key>
    encoding: json | parquet

path_prefix

The path_prefix parameter configures the path prefix using order and format child parameters. It is optional.

The order child parameter lists the formatting items that will define the path prefix:

• You can refer to Year, Month, Day, <any number that can divide 60> Minute, Hour, tag, host, OtherTags.<item related tags> and LogFields.<log related tags>.
• For ECS, ecs_cluster, ecs_container_name, ecs_task_family and ecs_task_version are available.
• For K8s, k8s_namespace, k8s_controller_kind, k8s_controller_logical_name, k8s_pod_name, k8s_container_name and k8s_container_image are available.
• For Docker, docker_container_name and docker_image_name are available

The format child parameter specifies a format string that has %s as placeholders per each order item.

nodes:
  - name: <node name>
    type: digitalocean_spaces_output
    endpoint: <endpoint>
    bucket: <bucket>
    access_key: <key>
    secret_key: <secret key>
    path_prefix:
      order:
      - Year
      - Month
      - Day
      - Hour
      - 2 Minute
      - tag
      - host
    format: ver=parquet/year=%s/month=%s/day=%s/hour=%s/min=%s/tag=%s/host=%s/

s3_force_path_style

The s3_force_path_style parameter forces the node to use the {endpoint}/{bucket} format instead of the {bucket}.{endpoint}/ format when reaching buckets. It is specified as a Boolean and the default is false indicating the node will use the {bucket}.{endpoint}/ format. It is optional.

nodes:
  - name: <node name>
    type: digitalocean_spaces_output
    endpoint: <endpoint>
    bucket: <bucket>
    access_key: <key>
    secret_key: <secret key>
    s3_force_path_style: true

tls

The tls parameter is a dictionary type that enables a number of options to be set using sub-parameters. It is optional.

nodes:
  - name: <node name>
    type: digitalocean_spaces_output
    endpoint: <endpoint>
    bucket: <bucket>
    access_key: <key>
    secret_key: <secret key>
    tls:
      <tls options>: 

ca_file

The ca_file parameter is a child of the tls parameter. It specifies the CA certificate file. It is specified as a string and is optional.

nodes:
  - name: <node name>
    type: digitalocean_spaces_output
    endpoint: <endpoint>
    bucket: <bucket>
    access_key: <key>
    secret_key: <secret key>
    tls:
      ca_file: /certs/ca.pem  

ca_path

The ca_path parameter is a child of the tls parameter. It specifies the location of the CA certificate files. It is specified as a string and is optional.

nodes:
  - name: <node name>
    type: digitalocean_spaces_output
    endpoint: <endpoint>
    bucket: <bucket>
    access_key: <key>
    secret_key: <secret key>
    tls:
      ca_path: <path>

crt_file

The crt_file parameter is a child of the tls parameter. It specifies the certificate file. It is specified as a string and is optional.

nodes:
  - name: <node name>
    type: digitalocean_spaces_output
    endpoint: <endpoint>
    bucket: <bucket>
    access_key: <key>
    secret_key: <secret key>
    tls:
      crt_file: /certs/server-cert.pem   

disable_verify

The disable_verify parameter is a child of tls. It specifies whether to disable the certificate check for remote endpoints. It is specified as a Boolean and the default is false. It is optional.

nodes:
  - name: <node name>
    type: digitalocean_spaces_output
    endpoint: <endpoint>
    bucket: <bucket>
    access_key: <key>
    secret_key: <secret key>
    tls:
      disable_verify: true 

key_file

The key_file parameter is a child of the tls parameter. It specifies the private key file. It is specified as a string and is optional.

nodes:
  - name: <node name>
    type: digitalocean_spaces_output
    endpoint: <endpoint>
    bucket: <bucket>
    access_key: <key>
    secret_key: <secret key>
    tls:
      key_file: /certs/server-key.pem

key_password

The key_password parameter is a child of the tls parameter. It specifies the key password. When the private key_file location is provided, this file can also be provided to get the password of the private key. It is specified as a string and is optional.

nodes:
  - name: <node name>
    type: digitalocean_spaces_output
    endpoint: <endpoint>
    bucket: <bucket>
    access_key: <key>
    secret_key: <secret key>
    tls:
      key_password: <password>

use_native_compression

The use_native_compression parameter configures whether, for parquet encoding, to only compress data segments for each archive file, not the whole file. It is specified as a Boolean, has a default of false, and it is optional.

nodes:
  - name: <node name>
    type: digitalocean_spaces_output
    endpoint: <endpoint>
    bucket: <bucket>
    access_key: <key>
    secret_key: <secret key>
    use_native_compression: true