Kubernetes troubleshooting

• Kubernetes troubleshooting steps

Commands

• kubectl logs your_pod
• kubectl get events -A
• kubectl describe pod your_pod
• kubectl describe nodes, review conditions:

• Tools: kubectl top, K9s and crictl

Events

• FailedScheduling: Insufficient cpu, Insufficient memory, timed out waiting for the condition, unbound immediate PersistentVolumeClaims
• kubectl get events, OOMKilling, FailedKillPod, SuccessfulDelete, SuccessfulCreate, NoPods, Warning, NodeSysctlChange, FailedAttachVolume, FailedMount, UnAvailableLoadBalancer, FailedCreatePodSandBox, InvalidDiskCapacity, Scheduled, NetworkNotReady, Evict, Killing, SuccessfulReconcilied, FailedToUpdateEndpointSlices, BackendNotFound, FailedScheduling
• Kubernetes node events: Kubernetes node events, NodeNotSchedulable, NodeAllocatableEnforced, NodeHasNoDiskPressure, DiskPressure, NodeHasSufficientMemory, NodeHasSufficientPID, RegisteredNode, InvalidDiskCapacity, Starting, NodeReady, RemovingNode

Kubernetes components

Load Balancer

• UnAvailableLoadBalancer

Kubelet

• PLEG is not healthy
• /var/log/kubelet.log

Scheduling

• Kubernetes scheduling
• Kubernetes Pod Topology Spread Constraints
• Kubernetes pod affinity and anti affinity
• Karpenter
• ttlSecondsUntilExpired, controller.node Triggering termination for expired node after 168h0m0s .../...

Log

• Karpenter logs
• Kubelet logs
• /var/log/kubelet.log

Related

• Readiness, Liveness, Readiness probe errored
• Reason: ProbeWarning
• Kubernetes Pod Disruptions
• Unable to connect to the server, ~/.kube/config
• DiskPressure
• CalculateExpectedPodCountFailed
• aws eks create-cluster --logging
• Node-pressure Eviction
• karpenter.sh/do-not-evict: true
• NodeNotReady
• kubectl-node-shell
• kubectl exec
• kubectl attach

Activities

• Review: https://learnk8s.io/troubleshooting-deployments
• Kubernetes debugging with an ephemeral debug container: kubectl debug

See also

• kubectl get events, OOMKilling, FailedKillPod, SuccessfulDelete, SuccessfulCreate, NoPods, Warning, NodeSysctlChange, FailedAttachVolume, FailedMount, UnAvailableLoadBalancer, FailedCreatePodSandBox, InvalidDiskCapacity, Scheduled, NetworkNotReady, Evict, Killing, SuccessfulReconcilied, FailedToUpdateEndpointSlices, BackendNotFound, FailedScheduling
• Kubernetes node events: Kubernetes node events, NodeNotSchedulable, NodeAllocatableEnforced, NodeHasNoDiskPressure, DiskPressure, NodeHasSufficientMemory, NodeHasSufficientPID, RegisteredNode, InvalidDiskCapacity, Starting, NodeReady, RemovingNode
• K8s troubleshooting: kubectl logs, kubectl get events -A, kubectl describe pod, Liveness, Readiness, Kubernetes events, Pulling image, OOMKilled, ProbeWarning, Reason, FailedScheduling, errImagePull, ImagePullBackOff, Kubelet conditions: MemoryPressure, DiskPressure, KubeletHasSufficientPID, KubeletReady, kubectl [ debug | attach | exec ]