Skip to main content

Kubernetes

Rationale#

Kubernetes is the system we use for hosting, deploying and managing our applications. It comprises infrastructure solutions like RBAC Authorization, distributed persistent storage, managing resource quotas, managing DNS records, managing load balancers, autoscaling, blue-Green deployments, rollbacks among many others. It allows us to serve and scale our applications in an easy, secure and automated way.

The main reasons why we chose it over other alternatives are:

  1. It is capable of deploying complex applications, including related Servers, DNS records, and load balancers in an automated way, allowing us to focus more on the application development and less on the infrastructure supporting it.
  2. It can be fully managed using Terraform.
  3. It supports Blue-Green deployments, allowing us to deploy applications many times a day without service interruptions.
  4. It supports Rollbacks, allowing us to revert applications to previous versions in case the need arise.
  5. It supports Horizontal autoscaling, allowing us to easily adapt our applications to the loads they're getting.
  6. It supports Service accounts, RBAC Authorization, and IRSA, allowing to give applications permissions to external resources using a least privilege approach.
  7. It supports resource quotas, allowing to easily distribute containers among physical machines using a granular cpu/memory per container approach.
  8. It has its own package manager, which makes deploying services very easy.
  9. It has its own local reproducibility tool for simulating clusters in local environments.
  10. It is Open source.
  11. It is not platform-bounded.
  12. Azure AKS, AWS EKS, GCP GKE, support it.
  13. It can be IaaS when implemented under a cloud provider.
  14. Migrating it from one cloud provider to another is, although not a simple task, at least possible.
  15. It is widely used by the community.
  16. It has many open source extensions.

Alternatives#

The following alternatives were considered but not chosen for the following reasons:

  1. AWS ECS: It is a serverless service for running containers. It is expensive as only one container exists within an entire physical machine. It does not support extensions. It is platform-bounded. It is not Open source.
  2. AWS Fargate: It is a serverless service for running containers without administering the infrastructure they run upon. It is expensive as only one container exists within an entire physical machine. It does not support extensions. It is platform-bounded. It is not Open source.
  3. AWS EC2: It is a service for cloud computing. AWS EKS actually uses it for setting up cluster workers. It does not support extensions. It is platform-bounded. It is not Open source.
  4. HashiCorp Nomad: Currently, no cloud provider supports it, which means that having to manage both managers and workers is required. It takes a simpler approach to orchestrating applications, with the downside of losing flexibility.
  5. Docker Swarm: Currently, no cloud provider supports it, which means that having to manage both managers and workers is required. It takes a simpler approach to orchestrating applications, with the downside of losing flexibility.

Usage#

We use Kubernetes for:

  1. Hosting our ASM.
  2. Automatically deploying ephemeral environments on CI/CD workflows.
  3. Automatically deploying DNS records for applications.
  4. Automatically deploying load balancers for applications.
  5. Automatically scaling worker nodes based on application load.
  6. Running application performance monitoring using New Relic.

We do not use Kubernetes for:

  1. Metrics-based autoscaling: Our autoscaling is based on the number of replicas we specify. It should instead be based on application load.
  2. Rollbacks: We should version production artifacts in order to be able to automatically return to a previous working version of our applications.
  3. Gitlab Runner: It was slow, unreliable and added too much overhead to workers. We decided to go back to Autoscaling Runner.
  4. Chaos Engineering: In order to harden ourselves against errors, we should create a little chaos in our infrastructure.

Guidelines#

Connect to cluster#

In order to connect to the Kubernetes Cluster, you must:

  1. Login as an Integrates developer using this guide.
  2. Install kubectl with nix-env -i kubectl.
  3. Select cluster by running aws eks update-kubeconfig --name makes-k8s --region us-east-1.
  4. Run kubectl get node.

Your input should be similar to this:

$ kubectl get node
NAME STATUS ROLES AGE VERSION
ip-192-168-5-112.ec2.internal Ready <none> 58d v1.17.9-eks-4c6976
ip-192-168-5-144.ec2.internal Ready <none> 39d v1.17.11-eks-cfdc40
ip-192-168-5-170.ec2.internal Ready <none> 20d v1.17.11-eks-cfdc40
ip-192-168-5-35.ec2.internal Ready <none> 30d v1.17.11-eks-cfdc40
ip-192-168-5-51.ec2.internal Ready <none> 30d v1.17.11-eks-cfdc40
ip-192-168-6-109.ec2.internal Ready <none> 30d v1.17.11-eks-cfdc40
ip-192-168-6-127.ec2.internal Ready <none> 18d v1.17.11-eks-cfdc40
ip-192-168-6-135.ec2.internal Ready <none> 31d v1.17.11-eks-cfdc40
ip-192-168-6-151.ec2.internal Ready <none> 30d v1.17.11-eks-cfdc40
ip-192-168-6-221.ec2.internal Ready <none> 13d v1.17.11-eks-cfdc40
ip-192-168-7-151.ec2.internal Ready <none> 30d v1.17.11-eks-cfdc40
ip-192-168-7-161.ec2.internal Ready <none> 33d v1.17.11-eks-cfdc40
ip-192-168-7-214.ec2.internal Ready <none> 61d v1.17.9-eks-4c6976
ip-192-168-7-48.ec2.internal Ready <none> 30d v1.17.11-eks-cfdc40
ip-192-168-7-54.ec2.internal Ready <none> 39d v1.17.11-eks-cfdc40

Common commands#

These are the most commonly used kubectl commands for debugging:

CommandExampleDescription
kubectl get pod -ANAGet all running pods
kubectl get node -ANAGet all cluster nodes
kubectl describe pod -n <namespace> <pod>kubectl describe pod -n ephemeral app-dsalazaratfluid-7c485cf565-w9gwgDescribe pod configurations
kubectl logs -n <namespace> <pod> -c <container>kubectl logs -n ephemeral app-dsalazaratfluid-7c485cf565-w9gwg -c appGet container logs from a pod
kubectl exec -it -n ephemeral <pod> -c <container> -- <command>kubectl exec -it -n ephemeral app-dsalazaratfluid-7c485cf565-w9gwg -c app -- bashAccess container within pod