Kubernetes_statefulsets

Kubernetes StatefulSets and DaemonSets

Overview

StatefulSets and DaemonSets are two specialized workload resources in Kubernetes that address specific use cases beyond the standard Deployment.

When to Use Each

Feature	Deployment	StatefulSet	DaemonSet
Use Case	Stateless apps	Stateful apps	Node-level daemons
Pod Identity	Random	Stable, ordinal	Unique per node
Storage	Shared/EmptyDir	Persistent volumes	HostPath/Local PV
Scaling	Any order	Ordered	One per node
Networking	Cluster DNS	Headless + stable	Cluster DNS

StatefulSet

StatefulSets provide:

• Stable, unique network identifiers: Pods retain their identity across rescheduling
• Stable, persistent storage: PersistentVolume claims persist across pod rescheduling
• Ordered, graceful deployment and scaling: Pods are created, updated, and deleted in order
• Ordered, automated rolling updates: Updates proceed in reverse ordinal order

Basic StatefulSet

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: my-statefulset
spec:
  serviceName: my-headless-service
  replicas: 3
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: myapp:latest
        ports:
        - containerPort: 80
          name: http
        volumeMounts:
        - name: data
          mountPath: /data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: standard
      resources:
        requests:
          storage: 1Gi

Headless Service

StatefulSets require a headless service for stable network identity:

apiVersion: v1
kind: Service
metadata:
  name: my-headless-service
spec:
  clusterIP: None  # Makes it headless
  selector:
    app: myapp
  ports:
  - name: http
    port: 80
    targetPort: 80

Pod Naming Convention

Pods in a StatefulSet get stable names:

• my-statefulset-0
• my-statefulset-1
• my-statefulset-2

These names persist across rescheduling, enabling:

• Stable DNS entries: my-statefulset-0.my-headless-service.default.svc.cluster.local
• Stable hostnames: Used in pod manifests

StatefulSet Scaling

# Scale up
kubectl scale statefulset my-statefulset --replicas=5

# Scale down (pods are terminated in reverse order: 4, 3, 2...)
kubectl scale statefulset my-statefulset --replicas=2

# View current replicas
kubectl get statefulset my-statefulset

Update Strategies

spec:
  updateStrategy:
    type: RollingUpdate  # Default
    rollingUpdate:
      partition: 2  # Only update pods with ordinal >= 2

Or use OnDelete strategy - pods are only updated when manually deleted:

spec:
  updateStrategy:
    type: OnDelete

Pod Management Policy

spec:
  podManagementPolicy: Parallel  # Default is OrderedReady

• OrderedReady: One pod at a time, in order
• Parallel: All pods created/terminated simultaneously

PersistentVolumeClaim Templates

Each pod gets its own PVC from the template:

spec:
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: fast-storage
      resources:
        requests:
          storage: 10Gi

StatefulSet Example: Database

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres-statefulset
spec:
  serviceName: postgres
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:14
        env:
        - name: POSTGRES_DB
          value: mydb
        - name: POSTGRES_USER
          value: admin
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: password
        ports:
        - containerPort: 5432
          name: postgres
        volumeMounts:
        - name: postgres-data
          mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
  - metadata:
      name: postgres-data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: fast-storage
      resources:
        requests:
          storage: 20Gi
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
spec:
  clusterIP: None
  selector:
    app: postgres
  ports:
  - port: 5432
    targetPort: 5432

DaemonSet

DaemonSets ensure that all (or specific) nodes run a copy of a specific pod.

Basic DaemonSet

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: logging-agent
  labels:
    app: logging
spec:
  selector:
    matchLabels:
      app: logging-agent
  template:
    metadata:
      labels:
        app: logging-agent
    spec:
      containers:
      - name: fluentd
        image: fluentd:v1.16
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

DaemonSet with Node Selection

Run pods only on specific nodes:

spec:
  nodeSelector:
    disktype: ssd  # Only run on nodes with this label
  tolerations:
  - key: "node-role.kubernetes.io/master"
    effect: NoSchedule
  - key: "critical"
    effect: NoSchedule
    operator: Exists

DaemonSet Update Strategies

spec:
  updateStrategy:
    type: RollingUpdate  # or OnDelete
    rollingUpdate:
      maxUnavailable: 1  # Max pods unavailable during update

Monitoring Agent Example

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      hostNetwork: true
      hostPID: true
      containers:
      - name: node-exporter
        image: prom/node-exporter:v1.6.1
        args:
        - --path.procfs=/host/proc
        - --path.sysfs=/host/sys
        - --path.rootfs=/host
        volumeMounts:
        - name: proc
          mountPath: /host/proc
          readOnly: true
        - name: sys
          mountPath: /host/sys
          readOnly: true
        - name: root
          mountPath: /host
          readOnly: true
      volumes:
      - name: proc
        hostPath:
          path: /proc
      - name: sys
        hostPath:
          path: /sys
      - name: root
        hostPath:
          path: /

Running on Control Plane Nodes

To run DaemonSet pods on master nodes:

spec:
  tolerations:
  - key: "node-role.kubernetes.io/control-plane"
    operator: Exists
    effect: NoSchedule
  - key: "node-role.kubernetes.io/master"
    operator: Exists
    effect: NoSchedule

Working with StatefulSets

Debugging StatefulSets

# List StatefulSets
kubectl get statefulset

# Describe StatefulSet
kubectl describe statefulset my-statefulset

# Get pods (note stable naming)
kubectl get pods -l app=myapp

# Check PersistentVolumeClaims
kubectl get pvc

# View pod logs (use stable pod name)
kubectl logs my-statefulset-0

# Delete pod (will be recreated with same identity)
kubectl delete pod my-statefulset-0

# Manually force delete a stuck pod
kubectl delete pod my-statefulset-0 --grace-period=0 --force

Troubleshooting Common Issues

1. Pod stuck in Pending:

   kubectl describe pvc <pvc-name>
   # Check storage class, available storage

2. Pod stuck in ContainerCreating:

   kubectl describe pod <pod-name>
   # Check volume mounting issues

3. Scaling stuck:

   kubectl get events --sort-by='.lastTimestamp'
   # Check for resource constraints or PVC issues

Working with DaemonSets

Debugging DaemonSets

# List DaemonSets
kubectl get daemonset

# Describe DaemonSet
kubectl describe daemonset logging-agent

# Check which nodes have the daemon
kubectl get pods -o wide | grep <daemonset-name>

# View DaemonSet rollout status
kubectl rollout status daemonset logging-agent

Common DaemonSet Use Cases

1. Log collectors: Fluentd, Logstash
2. Monitoring agents: Prometheus node exporter, Datadog
3. Storage daemons: Ceph, GlusterFS
4. Networking: CNI plugins, kube-proxy
5. Security: Falco, Auditbeat

Best Practices

StatefulSet Best Practices

1. Use for truly stateful applications: Databases, message queues
2. Always use headless service: Required for stable network identity
3. Configure proper storage: Use appropriate StorageClass
4. Set appropriate resources: Memory/CPU requests for scheduling
5. Plan your scaling strategy: Consider data rebalancing
6. Use readiness probes: Ensure pods are ready before traffic

DaemonSet Best Practices

1. Use nodeSelector/tolerations: Target specific nodes
2. Set appropriate resources: Since they run on every node
3. Use hostNetwork wisely: Consider port conflicts
4. Handle upgrades carefully: Use updateStrategy appropriately
5. Consider pod disruption budgets: Prevent service interruption

Summary

StatefulSets are ideal for:

• Databases (PostgreSQL, MySQL, MongoDB)
• Message queues (Kafka, RabbitMQ)
• Distributed systems requiring stable identity
• Applications needing persistent storage with stable names

DaemonSets are ideal for:

• Log aggregation (Fluentd, Filebeat)
• Monitoring (Prometheus node exporter)
• Cluster storage (Ceph, Rook)
• Network plugins (CNI providers)
• System daemons (kube-proxy)

Both resources provide specialized behavior that Deployments cannot replicate, making them essential tools for specific Kubernetes workloads.