Sizing and Capacity Estimations
3 minute read
Sizing and Capacity Estimations
Sizing
The sizing of your Edge Delta agents are directly related to how you configure your agents which means that having more nodes in the configuration will result in more computation power and/or memory needed for these operations. We recommend testing your configurations in small scale of data and doing estimations based on actual data and test data differences.
Estimations
The following recommendations are based on testing of our onboarding configs for Helm and Kubernetes with exception of features stated under Restrictions
and used HTTP input for pushing data to our agents. Based on event size of ~256 bytes, one agent (with one rollup and two compactor) can sustain 35 MiB/min/vCPU load.
Recommendations
Instances with at least 8 vCPUs and 16 GiB of memory are better in terms of horizontal scaling. Also deploying at least one instance per availability zone is desired to provide high availability to our agents. Some examples from cloud providers about recommended node types: AWS - c7g.2xlarge Azure - Standard_D8pls_v5 Google Cloud - t2a-standard-8
CPUs
We generally recommend:
- For processor agents, allocate at least 2 vCPU
- For compactor agents, allocate at least 0.5 vCPU
- For rollup agents, allocate at least 0.5 vCPU
Our agents can run in both Arm and x86_64 processors so users are free to choice their favorite instances. However, we find that Arm instances perform better with similar cost aspects.
Memory
For typical operations, 2GiB memory per vCPU would work however in some memory intensive operations such as parsing JSON from raw bytes can create huge demand for memory so please feel free to increase it according to your needs.
Scaling
Vertical Scaling
Our agent can be managed by vertical scaling since it utilizes concurrent execution through implementation. With that, we can utilize all vCPUs. Beware that you are utilizing load balancing techniques so that in an event of failure, data loss is minimized.
Horizontal Scaling
Having a load balancer in front of processor agent instances would allow you to achieve horizontal scaling seamlessly. To deploy Edge Delta agents with horizontal scaling support, please set the corresponding HPA values found in helm values.
Choosing a Load Balancer
Rule of a thumb, use a load balancer that you are comfortable with. If you have doubts, cloud providers provide managed load balancing solutions such as:
- AWS ALB and AWS NLB for AWS
- Azure LB for Azure
- Network Load Balancer for Google Cloud
Configuring a Load Balancer
The following are best practices when configuring a load balancer with Edge Delta agents.
- Use a protocol that allows for better load balancing such as an HTTP-based protocol.
- Enable keep alive feature (if exists) to reuse established connections.
- Use a load-balancing algorithm that supports even load distribution (such as round-robin).
Avoiding Hot Spots
Some connections can cause more data coming in which will make things hard for load balancing.
- Use protocols such as HTTP to have a control over message terminations. Using plain TCP connections for that reason.
- Ensure all of your agents can handle the connection with highest volume so you can enjoy vertical scaling.
Autoscaling
For majority of deployments with Deployment kind, CPU should be decision factor for scaling. Hence, autoscaling should be done based on CPU utilization. Edge Delta agents are generally constrained with CPU (however in some heavy data throughput together with buffering scenarios) so it is a great metric to achieve autoscaling without any false-positives. These are the recommended settings:
- Average CPU utilization over 5 minutes with a 90% target
- A 5 minute stabilization period for scaling up
- (Optional - for heavy memory intensive applications) Average memory utilization over 5 minutes with a 75% target