Scaling Down EKS Clusters at night

Scaling down workloads at night or at the weekends is a common implementation task for companies building on AWS. By running only the applications that need to be available at any point in time, the total consumption of infrastructure resources can be reduced, and thus customers can benefit from the pay-by-use pricing models of cloud providers.

For EC2-based workloads, this is a fairly simple task: Using Scheduled Scaling for Auto Scaling Groups, compute resources can be scaled up and down at a time schedule which is defined as a cron expression.

For EKS-based workloads, there is however an additional challenge. While usually one single application is running on an EC2 instance in the first setup, many application are running on the same EC2 instance when using a container orchestrator like kubernetes. Hence, simply shutting down an EC2 instance from the infrastructure side is not a viable solution for the following reasons:

  • It will shut down random applications
  • These applications will be rescheduled using managing controllers of the pods (e.g. a deployment)
  • A cluster-autoscaler will spin up new EC2 instances to meet the demand

Instead, a solution is needed that will scale down the applications first, and the infrastructure will follow.

Introducing Keda

Keda is an event-driven autoscaler for workloads on kubernetes. It has many integrations (so-called scalers) such as SNS, SQS or Kafka. For a full list, refer to Keda Scalers. The operator implements a custom resource called ScaledObject which creates and updates Horizontal Pod Autoscalers (HPAs) inside the cluster. HPAs are updated by Keda whenever the event defined in the ScaledObject resource trigger.

Setting up keda is fairly simple: We only need to install the helm chart.

helm install keda kedacore/keda --namespace keda --create-namespace

Solving the problem

To solve the problem described above, Keda’s cron scaler can be utilized in order to scale applications (deployments) up and down based on time schedules.

The ScaledObject has a start and an end time, defined by cron expressions. At the start time, Keda will scale out the deployment to the desiredReplicas count. At the end time, it will scale back in to zero, unless a minReplicaCount is defined.

Refer to the documentation to see how to use the cron scaler in a ScaledObject.

Example Code

The following example will scale the deployment scalable-deployment-example to 5 replicas every morning at 7am and scale it back in to 1 replica at 7pm.

kind: ScaledObject
  name: cron-scaledobject
  namespace: keda
    name: scalable-deployment-example
  minReplicaCount: 1
  - type: cron
      timezone: Europe/Zurich
      start: "0 7 * * *"
      end: "0 19 * * *"
      desiredReplicas: "5"

Title Photo by Benjamin Voros on Unsplash

Similar Posts You Might Enjoy

Assigning EKS Namespaces to Node Groups

In AWS EKS clusters, there are a couple of use cases for which all pods of a namespace should be automatically scheduled to specific nodes in Kubernetes, including: Clear allocation of data plane infrastructure (and costs) to teams in large organizations, Running critical workloads on on-demand nodes and not on spot nodes, or Using specific hardware, such as GPU, only by workloads that actually require it. In this post, we will explore how to facilitate that in EKS. - by Benjamin Wagner

EKS Backup with Velero

Velero is a tool to backup the kubernetes cluster state and its persistent volumes. It can be used for disaster recovery or cluster migration. Please refer to the official documentation for a more comprehensive description of use cases. This article describes the baseline setup for the backup to ease the start of backing up your EKS clusters. - by Benjamin Wagner

Using AWS Security Hub for EKS Security

kube-bench is a tool for checking kubernetes clusters against requirements defined in the CIS Benchmark. The tool runs locally on a kubernetes node, performs its checks and prompts the outputs to the shell or to files. This is quite unhandy, because it means that a user needs to pick up the logs, store them somewhere and analyze them. A deployment of the tool via kubernetes can ease the process for example with the kubectl logs command, but it is still far from perfect. Luckily, there is an integration in AWS Security Hub. - by Benjamin Wagner