Sizing and Capacity Estimations

Sizing and Capacity Estimations

  3 minute read  

Sizing and Capacity Estimations

Sizing

The sizing of your Edge Delta agents are directly related to how you configure your agents which means that having more nodes in the configuration will result in more computation power and/or memory needed for these operations. We recommend testing your configurations in small scale of data and doing estimations based on actual data and test data differences.

Estimations

The following recommendations are based on testing of our onboarding configs for Helm and Kubernetes with exception of features stated under Restrictions and used HTTP source for pushing data to our agents. Based on event size of ~256 bytes, one agent (with one rollup and two compactor) can sustain 35 MiB/min/vCPU load.

Recommendations

Instances with at least 8 vCPUs and 16 GiB of memory are better in terms of horizontal scaling. Also deploying at least one instance per availability zone is desired to provide high availability to our agents. Some examples from cloud providers about recommended node types: AWS - c7g.2xlarge Azure - Standard_D8pls_v5 Google Cloud - t2a-standard-8

CPUs

We generally recommend:

  • For processor agents, allocate at least 2 vCPU
  • For Compactor Agents, allocate at least 0.5 vCPU
  • For Rollup Agents, allocate at least 0.5 vCPU

Our agents can run in both Arm and x86_64 processors so users are free to choice their favorite instances. However, we find that Arm instances perform better with similar cost aspects.

Memory

For typical operations, 2GiB memory per vCPU would work however in some memory intensive operations such as parsing JSON from raw bytes can create huge demand for memory so please feel free to increase it according to your needs.

Scaling

Vertical Scaling

Our agent can be managed by vertical scaling since it utilizes concurrent execution through implementation. With that, we can utilize all vCPUs. Beware that you are utilizing load balancing techniques so that in an event of failure, data loss is minimized.

Horizontal Scaling

Having a load balancer in front of processor agent instances would allow you to achieve horizontal scaling seamlessly. To deploy Edge Delta agents with horizontal scaling support, please set the corresponding HPA values found in helm values.

Choosing a Load Balancer

Rule of a thumb, use a load balancer that you are comfortable with. If you have doubts, cloud providers provide managed load balancing solutions such as:

  • AWS ALB and AWS NLB for AWS
  • Azure LB for Azure
  • Network Load Balancer for Google Cloud

Configuring a Load Balancer

The following are best practices when configuring a load balancer with Edge Delta agents.

  • Use a protocol that allows for better load balancing such as an HTTP-based protocol.
  • Enable keep alive feature (if exists) to reuse established connections.
  • Use a load-balancing algorithm that supports even load distribution (such as round-robin).

Avoiding Hot Spots

Some connections can cause more data coming in which will make things hard for load balancing.

  • Use protocols such as HTTP to have a control over message terminations. Using plain TCP connections for that reason.
  • Ensure all of your agents can handle the connection with highest volume so you can enjoy vertical scaling.

Autoscaling

For majority of deployments with Deployment kind, CPU should be decision factor for scaling. Hence, autoscaling should be done based on CPU utilization. Edge Delta agents are generally constrained with CPU (however in some heavy data throughput together with buffering scenarios) so it is a great metric to achieve autoscaling without any false-positives. These are the recommended settings:

  • Average CPU utilization over 5 minutes with a 90% target
  • A 5 minute stabilization period for scaling up
  • (Optional - for heavy memory intensive applications) Average memory utilization over 5 minutes with a 75% target