Pod Scheduling: Node Affinity, Tolerations, and Security Contexts

The pod template: shared spec for all pods in a workload

Every Deployment, StatefulSet, DaemonSet, and Job embeds a pod template — the spec that gets instantiated into actual pods. The template defines containers, but also scheduling constraints, security settings, and network configuration that determine where and how those containers run.

apiVersion: apps/v1
kind: Deployment
spec:
  template:           # ← everything inside here is the pod template
    metadata:
      labels:
        app: api
    spec:
      containers: [...]
      # scheduling constraints, security, volumes all go here

nodeAffinity: expressive scheduling rules

spec:
  affinity:
    nodeAffinity:
      # Hard requirement: must be satisfied for scheduling
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: node.kubernetes.io/instance-type
            operator: In
            values: ["m6i.xlarge", "m6i.2xlarge", "c6i.xlarge"]
          - key: topology.kubernetes.io/zone
            operator: In
            values: ["us-east-1a", "us-east-1b"]

      # Soft preference: scheduler prefers but doesn't require
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        preference:
          matchExpressions:
          - key: node-group
            operator: In
            values: ["high-memory"]

requiredDuringSchedulingIgnoredDuringExecution — "ignored during execution" means if a node's labels change after the pod is scheduled, the pod stays running. There's no requiredDuringExecution variant that would evict running pods.

Taints and tolerations: dedicating nodes

Taints mark nodes as reserved for specific workloads. Without a matching toleration, pods won't be scheduled there.

# Taint a node group for GPU workloads
kubectl taint nodes node-gpu-1 workload=gpu:NoSchedule

# List all taints on a node
kubectl describe node node-gpu-1 | grep Taints

Only pods with matching tolerations can run on tainted nodes:

spec:
  tolerations:
  - key: "workload"
    operator: "Equal"
    value: "gpu"
    effect: "NoSchedule"

  # Also add nodeAffinity to actively target these nodes
  # (tolerations allow scheduling on tainted nodes, but don't require it)
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: workload
            operator: In
            values: ["gpu"]

Taint effects:

• NoSchedule: new pods without toleration won't be scheduled. Existing pods unaffected.
• PreferNoSchedule: scheduler avoids the node but will use it if necessary.
• NoExecute: existing pods without toleration are evicted. New pods without toleration won't schedule.

EKS applies system taints automatically — the node.kubernetes.io/not-ready:NoExecute taint is applied when a node goes unhealthy and removed when it recovers. This is how Kubernetes evicts pods from failed nodes.

Security contexts: pod and container privilege settings

Security contexts define the privilege model at the pod level (applies to all containers) and container level (overrides pod-level):

spec:
  # Pod-level security context
  securityContext:
    runAsNonRoot: true         # reject pods trying to run as root
    runAsUser: 1000
    runAsGroup: 1000
    fsGroup: 2000              # supplemental group for volume mounts
    seccompProfile:
      type: RuntimeDefault     # seccomp filtering

  containers:
  - name: api
    image: myapp:1.2.3
    # Container-level overrides
    securityContext:
      allowPrivilegeEscalation: false   # prevent sudo/setuid escalation
      readOnlyRootFilesystem: true      # prevents writes to container root FS
      capabilities:
        drop: ["ALL"]                   # drop all Linux capabilities
        add: ["NET_BIND_SERVICE"]       # add only what's needed (port < 1024)

readOnlyRootFilesystem: true is a strong defense-in-depth measure. If a container is compromised, attackers can't write to the filesystem. Applications that write to local files need emptyDir or PVC mounts for their writable paths.

spec:
  affinity:
    podAntiAffinity:
      # Hard: never two api pods on the same node
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values: ["api"]
        topologyKey: "kubernetes.io/hostname"

      # Soft: prefer spreading across zones
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values: ["api"]
          topologyKey: "topology.kubernetes.io/zone"

spec:
  topologySpreadConstraints:
  - maxSkew: 1                                  # max difference in pod count across zones
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: api