Kubernetes Scaler

Our primary cluster is hosted on AWS EKS, and is managed with Terraform: see _infra/terraform in this repo.

We also operate GPU clusters, where we run our inference engines. The Kubernetes configs for these clusters will generally be available in 1Password.

First-time Setup

Use kubie to manage cluster contexts.

We recommend kubie for switching between clusters (kubie ctx <cluster>) and namespaces (kubie ns <namespace>). It scopes each context to a shell session instead of mutating your global kubeconfig, and is handy alongside kubectl and k9s.

Connect to the primary (AWS EKS) cluster.

# After setting up AWS credentials (eg. with `aws sso login`)
aws eks update-kubeconfig --region us-east-1 --name synthetic-eks

# Switch to the primary cluster context.
kubie ctx synthetic-eks

note

The rest of this page assumes synthetic-eks is the default context. Run kubie ctx synthetic-eks first in any new shell.

Set up a GPU cluster context.

Download the cluster's kubeconfig from 1Password and save it somewhere out of the way, eg. ~/kube/gpu.kubeconfig.

Then merge it into your main kubeconfig:

# Merge the new kubeconfig with your existing one.
cp ~/.kube/config ~/.kube/config.bak  # Make a backup
KUBECONFIG=$HOME/.kube/config:$HOME/kube/gpu.kubeconfig kubectl config view --merge --flatten > ~/.kube/config

# Verify that the new config contains the cluster's context.
kubectl config get-contexts

# Switch to the GPU cluster when you need to work with it.
kubie ctx <context>

Feel free to rename the context or cluster in your ~/.kube/config to be more descriptive.

(Optional) Set up a local Kubernetes cluster.

These instructions are for setting up a local Kubernetes GPU cluster.

Some tools for this include minikube, kind, and k3s.

We use k3s in this guide.

Prerequisites:

• Machine with a NVIDIA GPU with proper drivers installed. (eg. nvidia-smi should work)
• Docker w/ NVIDIA Container Toolkit installed. (eg. sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi should work)
• kubectl, helm

warning

This will create a new context in your .kube/config and set it to default.

To see all contexts, kubectl config get-contexts, and to set a new default, kubectl config use-context <context>

# Configure containerd to support the nvidia runtime.
sudo nvidia-ctk runtime configure --runtime=containerd

# Install k3s
curl -fsL https://get.k3s.io | sh -

# Verify the NVIDIA container runtime has been recognized by k3s
sudo grep nvidia /var/lib/rancher/k3s/agent/etc/containerd/config.toml

# Add "--default-runtime nvidia" to the server runtime command
sudo vi /etc/systemd/system/k3s.service
sudo systemctl daemon-reload
sudo systemctl restart k3s

# (optional) Merge the new k3s kube config with your original one.
cp /home/$(whoami)/.kube/config /home/($whoami)/.kube/config.bak
sudo KUBECONFIG=/home/$(whoami)/.kube/config:/etc/rancher/k3s/k3s.yaml kubectl config view --flatten > /home/$(whoami)/.kube/config

# Select the new k3s context.
kubectl config use-context default

# Verify k3s is working
kubectl cluster-info

# Install the NVIDIA GPU operator
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
# DISABLE_DEV_CHAR_SYMLINK_CREATION: https://github.com/NVIDIA/gpu-operator/issues/569
helm install gpu-operator --wait -n gpu-operator nvidia/gpu-operator \
  --set driver.enabled=false \
  --set toolkit.enabled=false \
  --set "validator.driver.env[0].name=DISABLE_DEV_CHAR_SYMLINK_CREATION" \
  --set-string "validator.driver.env[0].value=true"

# Wait for all pods to finish initializing (~seconds)
kubectl get pods --namespace gpu-operator

# Verify our node has capability nvidia.com/gpu
kubectl get node -o json | jq '.items[0].status.capacity'
# {
#   ...
#   "nvidia.com/gpu": "1",
#   ...
# }

# Create a pod with a CUDA workload to test our configuration.
kubectl apply -f - << EOF
apiVersion: v1
kind: Pod
metadata:
  name: cuda-vectoradd
spec:
  restartPolicy: OnFailure
  runtimeClassName: nvidia
  containers:
  - name: cuda-vectoradd
    image: "nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04"
    resources:
      limits:
        nvidia.com/gpu: 1
EOF

# Verify everything is running!
kubectl logs cuda-vectoradd
# [Vector addition of 50000 elements]
# Copy input data from the host memory to the CUDA device
# CUDA kernel launch with 196 blocks of 256 threads
# Copy output data from the CUDA device to the host memory
# Test PASSED
# Done

Appendix

Run nvidia-smi in a pod

kubectl apply -f - << EOF
apiVersion: v1
kind: Pod
metadata:
  name: nvidia-version-check
spec:
  restartPolicy: OnFailure
  runtimeClassName: nvidia
  containers:
  - name: nvidia-version-check
    image: "nvidia/cuda:12.6.0-base-ubuntu22.04"
    command: ["nvidia-smi"]
    resources:
      limits:
         nvidia.com/gpu: 8
EOF