Sizing and Capacity Estimations

Sizing and Capacity Estimations

A pipeline is a set of agents with a single configuration. There are different types of pipelines that you can deploy depending on your architecture.

• See Architecture for an overview.
• See Scale Deployments for detailed instructions.

Except where called out, the following guidance applies to all pipelines.

Sizing

The resources your Edge Delta components need depend primarily on:

1. Throughput: raw volume per hour.
2. Pipeline Configuration features: parsing, alerting, transforms, buffering, etc.
3. Replication factor: number of pods (node pipeline) or Deployments (gateway pipeline).

We strongly recommend:

• Prototype first: test your configuration with a narrow slice of real traffic.
• Measure: capture CPU and memory peaks with kubectl top pods or the Edge Delta app.
• Extrapolate: scale the observed telemetry up to match production load.

Estimations

The following estimations can form a baseline for your prototype testing:

*Benchmarks include ~10 % safety head-room. Always validate with your own workload.

Recommendations

Instances with at least 8 vCPUs and 16 GiB RAM strike a good balance between vertical scalability and cost. For high availability, deploy at least one node per cloud Availability Zone.

CPUs

Compute resources drive nearly every stage of the Edge Delta data path, from extraction and parsing through roll-up and final compaction. Allocate cores deliberately so that routine bursts don’t trigger scaling events the instant traffic ticks upward.

The following table lists safe minimum vCPU counts; start here and monitor real-world usage before tightening budgets.

Edge Delta images are multi-architecture; ARM 64 (Graviton / Ampere) often delivers higher work-per-dollar than x86.

Memory

Start with ≈ 2 GiB per vCPU. Bump memory for configurations that:

• Parse large JSON payloads
• Buffer high-frequency metrics
• Perform heavy transformations

Scaling

Vertical Scaling

Each Edge Delta binary is multithreaded and will happily consume all vCPUs you give it. Vertical scaling is simplest for Node Pipeline agents because the DaemonSet already spreads one pod per node.

Caution: Even with vertical scaling, keep at least two nodes (and therefore two agents) behind your ingest endpoint to avoid single-point failure.

Horizontal Scaling

Node Pipeline

Horizontal scaling is usually unnecessary. One DaemonSet pod per node is sufficient. Scale Rollup and Compactor Deployments independently via replicas or an HPA.

Gateway Pipeline

Gateway components are classic Deployments and must be scaled horizontally to meet burst demand.

  --set deployment.autoscaling.enabled=true \
  --set deployment.autoscaling.minReplicas=2 \
  --set deployment.autoscaling.maxReplicas=10 \
  --set deployment.autoscaling.targetForCPUUtilizationPercentage=80 \
  --set deployment.autoscaling.targetForMemoryUtilizationPercentage=80

• Scale Rollup first when metric volume surges.
• Scale Compactor to match the rolled-up output rate.

Autoscaling Settings

Below is a clear, step-by-step recipe for setting up an HPA on any Deployment (Gateway Processor, Rollup, or Compactor).

Example HPA behavior block

behavior:
  scaleUp:
    stabilizationWindowSeconds: 300          # Wait 5 min before adding pods
    policies:
    - type: Percent
      value: 50                              # Grow by ≤ 50 % each step
      periodSeconds: 60
  scaleDown:
    stabilizationWindowSeconds: 300          # Wait 5 min before removing pods

After deploying, watch the HPA for a full traffic cycle (peak and off-peak) and fine-tune targets or delays if you still see oscillations.

Quick Reference

Need deeper guidance? Check:

• Scaling guide: See scale deployments for step-by-step HPA setup
• Benchmark calculator: Contact Edge Delta support for a sizing worksheet tailored to your data profile