Send Data from Edge Delta to OpenSearch

Configure Edge Delta to stream telemetry data to OpenSearch using the Elastic destination node.

  5 minute read  

Overview

OpenSearch is an open-source search and analytics engine based on Elasticsearch 7.10.2, maintained by the OpenSearch Project under the Apache License 2.0. While OpenSearch has diverged from Elasticsearch, it maintains API compatibility that allows Edge Delta to send data using the Elastic destination node.

Edge Delta can stream logs, metrics, signals, cluster patterns & samples, and custom data to OpenSearch indices in real-time, providing:

  • Pre-index optimization: Process, filter, and enrich data before indexing
  • Volume reduction: Aggregate and summarize data to reduce storage costs
  • Schema normalization: Apply consistent field mappings across data types
  • Flexible routing: Send different data types to specific indices

Prerequisites

  • OpenSearch cluster (self-hosted or AWS OpenSearch Service)
  • OpenSearch user with index creation privileges
  • Network connectivity between Edge Delta and OpenSearch

Step 1: Prepare OpenSearch

Create an Index

Create an index to receive Edge Delta data. You can customize the index settings based on your retention and performance requirements.

In the OpenSearch Dev Tools console or via API:

PUT /ed-logs
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1,
    "refresh_interval": "1s"
  },
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date",
        "format": "epoch_millis"
      },
      "body": {
        "type": "keyword"
      },
      "msg": {
        "type": "text"
      },
      "host": {
        "type": "keyword"
      },
      "tag": {
        "type": "keyword"
      },
      "src_name": {
        "type": "keyword"
      }
    }
  }
}

Note: Edge Delta automatically converts _timestamp to @timestamp when sending data to OpenSearch/Elasticsearch destinations.

Create an Index Pattern (OpenSearch Dashboards)

To visualize data in OpenSearch Dashboards:

  1. Navigate to Management > Dashboard Management
  2. Click Index patterns
  3. Click Create index pattern
  4. Enter ed-logs as the pattern name
  5. Select @timestamp as the time field
  6. Click Create index pattern

Configure User Access

If using fine-grained access control, ensure your user has the necessary permissions:

PUT _opendistro/_security/api/roles/edgedelta_writer
{
  "cluster_permissions": [
    "cluster:monitor/main",
    "indices:data/write/bulk"
  ],
  "index_permissions": [{
    "index_patterns": ["ed-*"],
    "allowed_actions": [
      "indices:admin/create",
      "indices:data/write/*",
      "indices:admin/mapping/put"
    ]
  }]
}

Step 2: Configure Edge Delta

Use the pipeline builder to add an Elastic destination node configured for OpenSearch, then route the data you want (logs, metrics, signals, cluster patterns & samples, custom).

Basic Configuration

nodes:
  - name: elastic_opensearch
    type: elastic_output
    user_description: opensearch-main
    index: ed-logs
    address: 
      - https://your-opensearch-domain.region.es.amazonaws.com
    user: opensearch_user
    password: your_password

AWS OpenSearch Service Configuration

For AWS-hosted OpenSearch with IAM authentication:

nodes:
  - name: elastic_opensearch_aws
    type: elastic_output
    user_description: opensearch-aws
    index: ed-logs
    address:
      - https://search-domain.region.es.amazonaws.com
    region: us-east-1
    role_arn: arn:aws:iam::12345abcde:role/EdgeDeltaOpenSearchRole

Secure Configuration with TLS

nodes:
  - name: elastic_opensearch_secure
    type: elastic_output
    user_description: opensearch-secure
    index: ed-logs
    address:
      - https://opensearch.example.com:9200
    user: opensearch_user
    password: your_password
    tls:
      ca_file: /path/to/ca.pem
      ignore_certificate_check: false
      min_version: TLSv1_2

Multi-Index Configuration

Route different data types to separate indices:

nodes:
  - name: elastic_opensearch_logs
    type: elastic_output
    user_description: opensearch-logs
    index: ed-logs
    address:
      - https://opensearch.example.com:9200
    user: opensearch_user
    password: your_password

  - name: elastic_opensearch_metrics
    type: elastic_output
    user_description: opensearch-metrics
    index: ed-metrics
    address:
      - https://opensearch.example.com:9200
    user: opensearch_user  
    password: your_password

Step 3: Validate Data Flow

Check OpenSearch

Query your index to verify data is being received:

curl -u username:password -X GET "https://opensearch.example.com:9200/ed-logs/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "match_all": {}
  },
  "size": 5,
  "sort": [
    {
      "@timestamp": {
        "order": "desc"
      }
    }
  ]
}'

OpenSearch Dashboards

  1. Navigate to OpenSearch Dashboards > Discover
  2. Select your index pattern (e.g., ed-logs)
  3. Verify logs are appearing with proper timestamps and fields

Advanced Configuration

Index Lifecycle Management

For production deployments, implement index lifecycle policies to manage retention:

PUT _plugins/_ism/policies/ed-logs-policy
{
  "policy": {
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [
          {
            "rollover": {
              "min_size": "5gb",
              "min_index_age": "1d"
            }
          }
        ],
        "transitions": [
          {
            "state_name": "delete",
            "conditions": {
              "min_index_age": "30d"
            }
          }
        ]
      },
      {
        "name": "delete",
        "actions": [
          {
            "delete": {}
          }
        ]
      }
    ]
  }
}

Performance Tuning

For high-volume ingestion, adjust the Edge Delta node configuration:

nodes:
  - name: elastic_opensearch_optimized
    type: elastic_output
    user_description: opensearch-optimized
    index: ed-logs
    address:
      - https://opensearch.example.com:9200
    user: opensearch_user
    password: your_password
    parallel_worker_count: 5
    buffer_max_bytesize: 10485760  # 10MB buffer
    buffer_ttl: 5m                  # Retry for 5 minutes
    buffer_path: /var/edgedelta/buffer/opensearch

Troubleshooting

Connection Issues

Cannot connect to OpenSearch

  • Verify network connectivity and firewall rules
  • Check OpenSearch domain endpoint is correct
  • Ensure proper authentication credentials
  • For AWS OpenSearch, verify IAM role has necessary permissions

Data Not Appearing

No data in OpenSearch index

  • Check Edge Delta agent logs for errors
  • Verify index name matches in both OpenSearch and Edge Delta config
  • Ensure user has write permissions to the index
  • Check timestamp field mapping is correct

Authentication Failures

401 Unauthorized errors

  • Verify username and password are correct
  • For AWS OpenSearch with fine-grained access control, ensure master user credentials are used
  • Check IAM role trust relationship if using role-based authentication

Timestamp Parsing Issues

Timestamp field not recognized

  • Ensure index mapping includes proper timestamp format
  • Edge Delta converts _timestamp to @timestamp automatically
  • Use epoch_millis format for best compatibility

Production Hardening

For production deployments, implement security best practices to protect your OpenSearch cluster. Use TLS encryption for all connections and store credentials securely in a secrets manager. Configure VPC endpoints or private connectivity when possible to avoid exposing OpenSearch to the public internet.

To optimize performance and costs, implement index lifecycle policies to automatically manage data retention. Use index templates to ensure consistent mappings across indices, and consider using OpenSearch’s ISM (Index State Management) for automated index management. Monitor index size and shard allocation to maintain cluster health.

For high availability, deploy OpenSearch across multiple availability zones with appropriate replica settings. Configure Edge Delta with buffer paths to ensure data durability during network interruptions. Regularly backup your OpenSearch indices and test recovery procedures to ensure business continuity.