Edge Delta IBM Object Storage Destination

Send items to an IBM Object Storage destination.

  9 minute read  

Beta

Overview

The IBM Object Storage destination node sends items to an IBM Object Storage destination. These items are raw archive bytes that are buffered with the archive buffer processor.

  • incoming_data_types: log

Example Configuration

nodes:
  - name: my_ibm_os
    type: ibm_object_storage_output
    endpoint: <REDACTED>
    bucket: <REDACTED>
    access_key: <REDACTED>
    secret_key: <REDACTED>
    compression: zstd
    encoding: parquet
    use_native_compression: true
    path_prefix:
      order:
      - Year
      - Month
      - Day
      - Hour
      - 2 Minute
      - tag
      - host
      format: ver=parquet/year=%s/month=%s/day=%s/hour=%s/min=%s/tag=%s/host=%s/

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the YAML using the name. It must be unique across all nodes. It is a YAML list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: ibm_object_storage_output

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

endpoint

The endpoint parameter specifies the IBM Object Storage endpoint. It is specified as a string and is required.

nodes:
- name: <node name>
  type: ibm_object_storage_output
  endpoint: <IBM Object Storage endpoint>
  bucket: <target bucket>
  access_key: <access key>
  secret_key: <key secret>

bucket

The bucket parameter defines the target bucket to use. It is specified as a string and is required.

nodes:
- name: <node name>
  type: ibm_object_storage_output
  endpoint: <IBM Object Storage endpoint>
  bucket: <target bucket>
  access_key: <access key>
  secret_key: <key secret>

access_key

The access_key parameter defines the access key to authorize access to the bucket. It is specified as a string and is a required parameter.

nodes:
- name: <node name>
  type: ibm_object_storage_output
  endpoint: <IBM Object Storage endpoint>
  bucket: <target bucket>
  access_key: <access key>
  secret_key: <key secret>

secret_key

The secret_key parameter defines the secret key to authorize access to the bucket. It is specified as a string and is a required parameter.

nodes:
- name: <node name>
  type: ibm_object_storage_output
  endpoint: <IBM Object Storage endpoint>
  bucket: <target bucket>
  access_key: <access key>
  secret_key: <key secret>

Optional Parameters

buffer_max_bytesize

The buffer_max_bytesize parameter configures the maximum byte size for total unsuccessful items. If the limit is reached, the remaining items are discarded until the buffer space becomes available. It is specified as a datasize.Size, has a default of 0 indicating no size limit, and it is optional.

nodes:
- name: <node name>
  type: ibm_object_storage_output
  endpoint: <IBM Object Storage endpoint>
  bucket: <target bucket>
  access_key: <access key>
  secret_key: <key secret>
  buffer_max_bytesize: 2048

buffer_path

The buffer_path parameter configures the path to store unsuccessful items. Unsuccessful items are stored there to be retried back (exactly once delivery). It is specified as a string and it is optional.

nodes:
- name: <node name>
  type: ibm_object_storage_output
  endpoint: <IBM Object Storage endpoint>
  bucket: <target bucket>
  access_key: <access key>
  secret_key: <key secret>
  buffer_path: <path to unsuccessful items folder>

buffer_ttl

The buffer_ttl parameter configures the time-to-Live for unsuccessful items, which indicates when to discard them. It is specified as a duration, has a default of 10m, and it is optional.

nodes:
- name: <node name>
  type: ibm_object_storage_output
  endpoint: <IBM Object Storage endpoint>
  bucket: <target bucket>
  access_key: <access key>
  secret_key: <key secret>
  buffer_ttl: 20m

compression

The compression parameter specifies the compression format. It can be gzip, zstd, snappy or uncompressed. It is specified as a string, has a default of gzip, and it is optional.

nodes:
- name: <node name>
  type: ibm_object_storage_output
  endpoint: <IBM Object Storage endpoint>
  bucket: <target bucket>
  access_key: <access key>
  secret_key: <key secret>
  compression: gzip | zstd | snappy | uncompressed

disable_compaction

YAML Only

This parameter configures whether to disable compaction by the Compactor Agent for data from this node before it is sent to the data destination. It is specified as a boolean, the default is false and it is optional.

nodes:
- name: <node name>
  type: ibm_object_storage_output
  endpoint: <IBM Object Storage endpoint>
  bucket: <target bucket>
  access_key: <access key>
  secret_key: <key secret>
  disable_compaction: true

disable_ssl

The disable_ssl parameter defines whether to use HTTP instead of HTTPS for accessing the bucket. It is specified as a Boolean and the default is false indicating that SSL is required. It is optional.

nodes:
- name: <node name>
  type: ibm_object_storage_output
  endpoint: <IBM Object Storage endpoint>
  bucket: <target bucket>
  access_key: <access key>
  secret_key: <key secret>
  disable_ssl: true

encoding

The encoding parameter specifies the encoding format. It can be json or parquet. It is specified as a string, has a default of json, and it is optional.

nodes:
- name: <node name>
  type: ibm_object_storage_output
  endpoint: <IBM Object Storage endpoint>
  bucket: <target bucket>
  access_key: <access key>
  secret_key: <key secret>
  encoding: json | parquet

flush_interval

YAML Only

The flush_interval parameter specifies the duration to flush (or force) data to the destination, including buffered data. It is specified as a duration and is optional.

nodes:
- name: <node name>
  type: ibm_object_storage_output
  endpoint: <IBM Object Storage endpoint>
  bucket: <target bucket>
  access_key: <access key>
  secret_key: <key secret>
  flush_interval: 10m

max_byte_limit

YAML Only

The max_byte_limit parameter specifies the maximum bytes before flushing buffered raw data to archive destination. It is specified with a data size and is optional. If not specified for this node the setting in the agent settings is used.

nodes:
- name: <node name>
  type: ibm_object_storage_output
  endpoint: <IBM Object Storage endpoint>
  bucket: <target bucket>
  access_key: <access key>
  secret_key: <key secret>
  max_byte_limit: 32MB

path_prefix

The path_prefix parameter configures the path prefix using order and format child parameters. It is optional.

The order child parameter lists the formatting items that will define the path prefix:

  • You can refer to Year, Month, Day, <any number that can divide 60> Minute, Hour, tag, host, OtherTags.<item related tags> and LogFields.<log related tags>.
  • For ECS, ecs_cluster, ecs_container_name, ecs_task_family and ecs_task_version are available.
  • For K8s, k8s_namespace, k8s_controller_kind, k8s_controller_logical_name, k8s_pod_name, k8s_container_name and k8s_container_image are available.
  • For Docker, docker_container_name and docker_image_name are available

The format child parameter specifies a format string that has %s as placeholders per each order item.

nodes:
- name: <node name>
  type: ibm_object_storage_output
  endpoint: <IBM Object Storage endpoint>
  bucket: <target bucket>
  access_key: <access key>
  secret_key: <key secret>
  path_prefix:
      order:
      - Year
      - Month
      - Day
      - Hour
      - 2 Minute
      - tag
      - host
    format: ver=parquet/year=%s/month=%s/day=%s/hour=%s/min=%s/tag=%s/host=%s/

s3_force_path_style

The s3_force_path_style parameter forces the node to use the {endpoint}/{bucket} format instead of the {bucket}.{endpoint}/ format when reaching buckets. It is specified as a Boolean and the default is false indicating the node will use the {bucket}.{endpoint}/ format. It is optional.

nodes:
- name: <node name>
  type: ibm_object_storage_output
  endpoint: <IBM Object Storage endpoint>
  bucket: <target bucket>
  access_key: <access key>
  secret_key: <key secret>
  s3_force_path_style: true

tls

The tls parameter is a dictionary type that enables a number of options to be set using sub-parameters. It is optional.

nodes:
- name: <node name>
  type: ibm_object_storage_output
  endpoint: <IBM Object Storage endpoint>
  bucket: <target bucket>
  access_key: <access key>
  secret_key: <key secret>
  tls:
      <tls options>:

ca_file

The ca_file parameter is a child of the tls parameter. It specifies the CA certificate file. It is specified as a string and is optional.

nodes:
- name: <node name>
  type: ibm_object_storage_output
  endpoint: <IBM Object Storage endpoint>
  bucket: <target bucket>
  access_key: <access key>
  secret_key: <key secret>
  tls:
      ca_file: /certs/ca.pem

ca_path

The ca_path parameter is a child of the tls parameter. It specifies the location of the CA certificate files. It is specified as a string and is optional.

nodes:
- name: <node name>
  type: ibm_object_storage_output
  endpoint: <IBM Object Storage endpoint>
  bucket: <target bucket>
  access_key: <access key>
  secret_key: <key secret>
  tls:
      ca_path: <path>

client_auth_type

The client_auth_type parameter is a child of the tls parameter. It specifies the authentication type to use for the connection. It is specified as a string from a closed list and is optional.

The following authentication methods are available:

  • noclientcert indicates that no client certificate should be requested during the handshake, and if any certificates are sent they will not be verified.
  • requestclientcert indicates that a client certificate should be requested during the handshake, but does not require that the client send any certificates.
  • requireanyclientcert indicates that a client certificate should be requested during the handshake, and that at least one certificate is required from the client, but that certificate is not required to be valid.
  • verifyclientcertifgiven indicates that a client certificate should be requested during the handshake, but does not require that the client sends a certificate. If the client does send a certificate it is required to be valid.
  • requireandverifyclientcert indicates that a client certificate should be requested during the handshake, and that at least one valid certificate is required to be sent by the client
nodes:
- name: <node name>
  type: ibm_object_storage_output
  endpoint: <IBM Object Storage endpoint>
  bucket: <target bucket>
  access_key: <access key>
  secret_key: <key secret>
  tls:
      client_auth_type: <auth type>

crt_file

The crt_file parameter is a child of the tls parameter. It specifies the certificate file. It is specified as a string and is optional.

nodes:
- name: <node name>
  type: ibm_object_storage_output
  endpoint: <IBM Object Storage endpoint>
  bucket: <target bucket>
  access_key: <access key>
  secret_key: <key secret>
  tls:
      crt_file: /certs/server-cert.pem

ignore_certificate_check

The ignore_certificate_check parameter is a child of tls. It specifies whether to disable the certificate check for remote endpoints. It is specified as a Boolean and the default is false. It is optional.

nodes:
- name: <node name>
  type: ibm_object_storage_output
  endpoint: <IBM Object Storage endpoint>
  bucket: <target bucket>
  access_key: <access key>
  secret_key: <key secret>
  tls:
      ignore_certificate_check: true

key_file

The key_file parameter is a child of the tls parameter. It specifies the key file. It is specified as a string and is optional.

nodes:
- name: <node name>
  type: ibm_object_storage_output
  endpoint: <IBM Object Storage endpoint>
  bucket: <target bucket>
  access_key: <access key>
  secret_key: <key secret>
  tls:
      key_password: <password>
      key_file: <path to file>

key_password

The key_password parameter is a child of the tls parameter. It specifies the key password. When the private key_file location is provided, this file can also be provided to get the password of the private key. It is specified as a string and is optional.

nodes:
- name: <node name>
  type: ibm_object_storage_output
  endpoint: <IBM Object Storage endpoint>
  bucket: <target bucket>
  access_key: <access key>
  secret_key: <key secret>
  tls:
      key_password: <password>
      key_file: <path to file>

max_version

The max_version parameter is a child of the tls parameter. It specifies the maximum version of TLS to accept. It is specified as a string and is optional.

You can select one of the following options:

  • TLSv1_0
  • TLSv1_1
  • TLSv1_2
  • TLSv1_3
nodes:
- name: <node name>
  type: ibm_object_storage_output
  endpoint: <IBM Object Storage endpoint>
  bucket: <target bucket>
  access_key: <access key>
  secret_key: <key secret>
  tls:
      max_version: <TLS version>

min_version

The min_version parameter is a child of the tls parameter. It specifies the minimum version of TLS to accept. It is specified as a string and is optional. The default is TLSv1_2.

You can select one of the following options:

  • TLSv1_0
  • TLSv1_1
  • TLSv1_2
  • TLSv1_3
nodes:
- name: <node name>
  type: ibm_object_storage_output
  endpoint: <IBM Object Storage endpoint>
  bucket: <target bucket>
  access_key: <access key>
  secret_key: <key secret>
  tls:
      min_version: <TLS version>

use_native_compression

The use_native_compression parameter configures whether, for parquet encoding, to only compress data segments for each archive file, not the whole file. It is specified as a Boolean, has a default of false, and it is optional.

nodes:
- name: <node name>
  type: ibm_object_storage_output
  endpoint: <IBM Object Storage endpoint>
  bucket: <target bucket>
  access_key: <access key>
  secret_key: <key secret>
  use_native_compression: true