Difference between revisions of "Kubectl get events"
Jump to navigation
Jump to search
(→Events) |
|||
(69 intermediate revisions by 3 users not shown) | |||
Line 1: | Line 1: | ||
{{lc}} | {{lc}} | ||
+ | TOMERGE: [[Kubernetes node events]] | ||
+ | |||
+ | * <code>[[kubectl get events --help]]</code> | ||
+ | * <code>[[kubectl get events -A]]</code> | ||
+ | * <code>[[kubectl get events -A]] | grep [[Warning]]</code> | ||
+ | * <code>[[kubectl get events -A -o wide]] | grep [[Warning]]</code> | ||
* <code>[[kubectl get]] events</code> | * <code>[[kubectl get]] events</code> | ||
− | * <code>[[kubectl get events - | + | * <code>[[kubectl get events -o wide]]</code> |
* <code>kubectl get events -o yaml</code> | * <code>kubectl get events -o yaml</code> | ||
Line 10: | Line 16: | ||
* <code>[[kubectl -n gitlab-runner get events --field-selector type!=Normal]]</code> | * <code>[[kubectl -n gitlab-runner get events --field-selector type!=Normal]]</code> | ||
− | kubectl get events -A | grep [[Warning]] | + | |
− | [[kubectl get events -A]] | grep Normal | + | kubectl get events -A | grep [[Warning]] | egrep "[[FailedMount]]|[[FailedAttachVolume]]|[[Unhealthy]]|[[ClusterUnhealthy]]|[[FailedScheduling]]" |
+ | |||
+ | [[kubectl get events -A]] | grep [[Normal]] | ||
== Events examples == | == Events examples == | ||
+ | * [[Normal]], [[Warning]], [[Critical]] | ||
+ | |||
=== Warning === | === Warning === | ||
+ | |||
+ | [[BackoffLimitExceeded]] | ||
+ | [[CalculateExpectedPodCountFailed]] | ||
+ | [[ClusterUnhealthy]] | ||
+ | [[FailedMount]] | ||
+ | [[FailedScheduling]] | ||
+ | [[InvalidDiskCapacity]] | ||
+ | [[Unhealthy]] | ||
+ | .../... | ||
+ | |||
+ | |||
your_namespace 28s Warning [[FailedScheduling]] pod/kibana-kibana-654ccb45bd-pbp4r 0/2 nodes are available: 2 [[Insufficient cpu]]. | your_namespace 28s Warning [[FailedScheduling]] pod/kibana-kibana-654ccb45bd-pbp4r 0/2 nodes are available: 2 [[Insufficient cpu]]. | ||
Line 37: | Line 58: | ||
default 107s Warning [[ProvisioningFailed]] persistentvolumeclaim/myprometheus-server (combined from similar events): failed to provision volume with StorageClass "[[gp2]]": rpc error: code = Internal desc = [[Could not create volume]] "pvc-4e14416c-c9c2-4d39-b749-9ce0fa98d597": could not create volume in EC2: [[UnauthorizedOperation]]: [[You are not authorized to perform this operation]]. Encoded authorization failure message: Goz6E3qExxxxx.../... | default 107s Warning [[ProvisioningFailed]] persistentvolumeclaim/myprometheus-server (combined from similar events): failed to provision volume with StorageClass "[[gp2]]": rpc error: code = Internal desc = [[Could not create volume]] "pvc-4e14416c-c9c2-4d39-b749-9ce0fa98d597": could not create volume in EC2: [[UnauthorizedOperation]]: [[You are not authorized to perform this operation]]. Encoded authorization failure message: Goz6E3qExxxxx.../... | ||
+ | |||
+ | [[kube-system]] 9m44s Warning [[FailedMount]] pod/[[kube-dns]]-85df8994db-v8qdg [[MountVolume]].SetUp failed for volume "kube-dns-config" : failed to sync [[configmap cache]]: [[timed out waiting for the condition]]</code> | ||
+ | |||
+ | |||
+ | |||
+ | [[kube-system]] 43m [[Warning]] [[ClusterUnhealthy]] [[configmap/]] | ||
+ | [[cluster-autoscaler-status]] | ||
+ | |||
+ | |||
+ | LAST SEEN TYPE REASON OBJECT MESSAGE | ||
+ | 28s Warning FailedScheduling pod/deployment-1230/3 nodes | ||
+ | are available: 3 [[persistentvolumeclaim]] "your" bound to non-existent | ||
+ | persistentvolume "". | ||
+ | 19m Warning FailedScheduling pod/deployment-123 0/3 nodes | ||
+ | are available: 3 persistentvolumeclaim "your" not found. | ||
+ | 10m Warning FailedScheduling pod/deployment-91234 0/3 nodes | ||
+ | are available: 3 persistentvolumeclaim "your" bound to non-existent | ||
+ | persistentvolume "". | ||
+ | 17m Warning [[ClaimLost]] persistentvolumeclaim/yourclaim [[Bound claim has lost reference to PersistentVolume]]. Data on the volume is lost! [[Cluster has no ready nodes]]. | ||
=== Normal === | === Normal === | ||
+ | Started | ||
+ | Created | ||
+ | Pulled | ||
+ | [[Pulling]] | ||
+ | [[Scheduled]] | ||
+ | [[Killing]] | ||
+ | [[Evict]] | ||
+ | [[SandboxChanged]] | ||
+ | [[SuccessfulCreate]] - [[ReplicaSet]] | ||
+ | [[SuccessfulDelete]] | ||
+ | [[NodeNotSchedulable]] | ||
+ | [[RemovingNode]] | ||
+ | [[TaintManagerEviction]] | ||
+ | [[WaitForFirstConsumer]] | ||
+ | [[ExternalProvisioning]] | ||
+ | [[TaintManagerEviction: Cancelling deletion of pod]] | ||
+ | |||
+ | |||
+ | |||
default 4s Normal [[Provisioning]] persistentvolumeclaim/myprometheus-alertmanager External provisioner is provisioning volume for claim "default/myprometheus-alertmanager" | default 4s Normal [[Provisioning]] persistentvolumeclaim/myprometheus-alertmanager External provisioner is provisioning volume for claim "default/myprometheus-alertmanager" | ||
Related: <code>[[kubectl get pvc]]</code> | Related: <code>[[kubectl get pvc]]</code> | ||
+ | |||
+ | |||
+ | ingress-nginx 53m Normal [[UpdatedLoadBalancer]] service/nginx-ingress-controller Updated load balancer with new hosts | ||
+ | ingress-nginx 54m Warning [[UnAvailableLoadBalancer]] service/nginx-ingress-controller There are no available nodes for LoadBalancer | ||
== Events == | == Events == | ||
− | + | * <code>[[BackOff]]</code> | |
− | + | * <code>[[Completed]]</code> | |
− | + | * <code>[[Created]]</code> | |
− | + | * <code>[[DeadlineExceeded]]</code> | |
− | + | * <code>[[Failed]]</code> | |
− | + | * <code>[[FailedAttachVolume]]</code> | |
− | + | * <code>[[FailedCreatePodSandBox]]</code> | |
− | + | * <code>[[FailedMount]]</code> | |
− | + | * <code>[[FailedKillPod]]</code> | |
− | + | * <code>[[FailedScheduling]]</code> | |
− | + | * <code>[[FailedToUpdateEndpoint]]</code> | |
− | + | * <code>[[FailedToUpdateEndpointSlices]]</code> | |
− | + | * <code>[[Generated]]</code> | |
+ | * <code>[[PresentError]]</code> | ||
+ | * <code>[[Pulled]]</code> | ||
+ | * <code>[[Pulling]]</code> | ||
+ | * <code>[[Requested]]</code> | ||
+ | * <code>[[SawCompletedJob]]</code> | ||
+ | * <code>[[Scheduled]]</code> | ||
+ | * <code>[[Started]]</code> | ||
* <code>[[SuccessfulCreate]]</code> | * <code>[[SuccessfulCreate]]</code> | ||
* <code>[[SuccessfulDelete]]</code> | * <code>[[SuccessfulDelete]]</code> | ||
− | + | * <code>[[NetworkNotReady]]</code> | |
− | * <code>[[ | + | * <code>[[NodeNotReady]]</code> |
− | + | * <code>[[NodeAllocatableEnforced]]</code> | |
* <code>[[NoPods]]</code> | * <code>[[NoPods]]</code> | ||
− | |||
− | |||
* <code>[[NodeHasNoDiskPressure]]</code> | * <code>[[NodeHasNoDiskPressure]]</code> | ||
+ | * <code>[[UnAvailableLoadBalancer]]</code> | ||
+ | * <code>[[Unhealthy]]</code> | ||
+ | * <code>[[VolumeFailedDelete]]</code> | ||
== Related == | == Related == | ||
+ | * <code>[[kubectl events]]</code> | ||
* <code>[[kubectl top]]</code> | * <code>[[kubectl top]]</code> | ||
* <code>[[kubectl logs]]</code> | * <code>[[kubectl logs]]</code> | ||
− | * <code>[[ | + | * <code>[[gcloud logging read resource.labels.cluster_name]]</code> |
− | |||
* [[job-controller]] | * [[job-controller]] | ||
− | |||
* [[GCP Node logs]] | * [[GCP Node logs]] | ||
− | |||
* <code>[[gcloud logging read]] projects/yourproject/logs/[[kubelet]]</code> | * <code>[[gcloud logging read]] projects/yourproject/logs/[[kubelet]]</code> | ||
+ | * <code>[[kubectl describe nodes (conditions:)]]</code> | ||
+ | * <code>[[kubectl describe nodes]] | grep [[KubeletReady]]</code> | ||
+ | * <code>[[--event-ttl]]</code> defines the amount of time to retain events, default 1h. | ||
== See also == | == See also == | ||
* {{kubectl events}} | * {{kubectl events}} | ||
− | |||
* {{K8s troubleshooting}} | * {{K8s troubleshooting}} | ||
+ | * {{K8s monitoring}} | ||
[[Category:K8s]] | [[Category:K8s]] |
Latest revision as of 17:53, 10 April 2024
TOMERGE: Kubernetes node events
kubectl get events --help
kubectl get events -A
kubectl get events -A | grep Warning
kubectl get events -A -o wide | grep Warning
kubectl get events
kubectl get events -o wide
kubectl get events -o yaml
kubectl get events --sort-by=.metadata.creationTimestamp
kubectl get events --sort-by=.metadata.creationTimestamp -A
kubectl get events --sort-by='.lastTimestamp
kubectl get events -A | grep Warning | egrep "FailedMount|FailedAttachVolume|Unhealthy|ClusterUnhealthy|FailedScheduling"
kubectl get events -A | grep Normal
Events examples[edit]
Warning[edit]
BackoffLimitExceeded CalculateExpectedPodCountFailed ClusterUnhealthy FailedMount FailedScheduling InvalidDiskCapacity Unhealthy .../...
your_namespace 28s Warning FailedScheduling pod/kibana-kibana-654ccb45bd-pbp4r 0/2 nodes are available: 2 Insufficient cpu.
your_namespace 4m53s Warning ProbeWarning pod/metabase-prod-f8f4b765b-h4pgs Readiness probe warning:
your_namespace 30m Warning BackoffLimitExceeded job/your-job27740460 Job has reached the specified backoff limit
your_namespace 26m Warning Unhealthy pod/elasticsearch-master-1 Readiness probe failed: Waiting for elasticsearch cluster to become ready (request params: "wait_for_status=green&timeout=1s" )...
your_namespace 99s Warning BackOff pod/elasticsearch-master-0 Back-off restarting failed container your_namespace 108s Warning BackOff pod/elasticsearch-master-1 Back-off restarting failed container
your_namespace 12m Warning PresentError challenge/prod-admin-tls-cert-dzmbt-2545 Error presenting challenge: error getting clouddns service account: secret "clouddns-dns01-solver-svc-acct" not found
your_namespace 27m Warning OOMKilling node/gke-you-pool4 Memory cgroup out of memory: Killed process 2768158 (python) total-vm:5613088kB, anon-rss:3051580kB, file-rss:65400kB, shmem-rss:0kB, UID:0 pgtables:7028kB oom_score_adj:997
your_namespace 8m51s Warning FailedScheduling pod/myprometheus-alertmanager-5967d4ff85-5glkh running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition default 4m58s Normal ExternalProvisioning persistentvolumeclaim/myprometheus-alertmanager waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Solution: Install aws-ebs-csi-driver
default 107s Warning ProvisioningFailed persistentvolumeclaim/myprometheus-server (combined from similar events): failed to provision volume with StorageClass "gp2": rpc error: code = Internal desc = Could not create volume "pvc-4e14416c-c9c2-4d39-b749-9ce0fa98d597": could not create volume in EC2: UnauthorizedOperation: You are not authorized to perform this operation. Encoded authorization failure message: Goz6E3qExxxxx.../...
kube-system 9m44s Warning FailedMount pod/kube-dns-85df8994db-v8qdg MountVolume.SetUp failed for volume "kube-dns-config" : failed to sync configmap cache: timed out waiting for the condition
kube-system 43m Warning ClusterUnhealthy configmap/ cluster-autoscaler-status
LAST SEEN TYPE REASON OBJECT MESSAGE 28s Warning FailedScheduling pod/deployment-1230/3 nodes are available: 3 persistentvolumeclaim "your" bound to non-existent persistentvolume "". 19m Warning FailedScheduling pod/deployment-123 0/3 nodes are available: 3 persistentvolumeclaim "your" not found. 10m Warning FailedScheduling pod/deployment-91234 0/3 nodes are available: 3 persistentvolumeclaim "your" bound to non-existent persistentvolume "". 17m Warning ClaimLost persistentvolumeclaim/yourclaim Bound claim has lost reference to PersistentVolume. Data on the volume is lost! Cluster has no ready nodes.
Normal[edit]
Started Created Pulled Pulling Scheduled Killing Evict SandboxChanged SuccessfulCreate - ReplicaSet SuccessfulDelete NodeNotSchedulable RemovingNode TaintManagerEviction WaitForFirstConsumer ExternalProvisioning TaintManagerEviction: Cancelling deletion of pod
default 4s Normal Provisioning persistentvolumeclaim/myprometheus-alertmanager External provisioner is provisioning volume for claim "default/myprometheus-alertmanager"
Related: kubectl get pvc
ingress-nginx 53m Normal UpdatedLoadBalancer service/nginx-ingress-controller Updated load balancer with new hosts ingress-nginx 54m Warning UnAvailableLoadBalancer service/nginx-ingress-controller There are no available nodes for LoadBalancer
Events[edit]
BackOff
Completed
Created
DeadlineExceeded
Failed
FailedAttachVolume
FailedCreatePodSandBox
FailedMount
FailedKillPod
FailedScheduling
FailedToUpdateEndpoint
FailedToUpdateEndpointSlices
Generated
PresentError
Pulled
Pulling
Requested
SawCompletedJob
Scheduled
Started
SuccessfulCreate
SuccessfulDelete
NetworkNotReady
NodeNotReady
NodeAllocatableEnforced
NoPods
NodeHasNoDiskPressure
UnAvailableLoadBalancer
Unhealthy
VolumeFailedDelete
Related[edit]
kubectl events
kubectl top
kubectl logs
gcloud logging read resource.labels.cluster_name
- job-controller
- GCP Node logs
gcloud logging read projects/yourproject/logs/kubelet
kubectl describe nodes (conditions:)
kubectl describe nodes | grep KubeletReady
--event-ttl
defines the amount of time to retain events, default 1h.
See also[edit]
- Kubernetes events, Kubernetes node events:
[ Normal | Warning ]
,kubectl events, kubectl get events
:BackOff
,FailedMount, FailedAttachVolume, TaintManagerEviction, InvalidDiskCapacity
,FailedScheduling, kubectl describe events, Unhealthy
,conditions:, UpdatedLoadBalancer, BackoffLimitExceeded
- K8s troubleshooting:
kubectl logs, kubectl top, kubectl get events -A, kubectl describe pod
, Liveness, Readiness,Kubernetes events
, Pulling image, OOMKilled, ProbeWarning, Reason,FailedScheduling
,errImagePull, ImagePullBackOff
, Kubelet conditions:MemoryPressure, DiskPressure, KubeletHasSufficientPID, KubeletReady, kubectl [ debug | attach | exec ] kubectl cluster-info dump, SimKube, KWOK
- Kubernetes monitoring, node conditions, Kube-state-metrics (KSM), Prometheus, VictoriaMetrics,
node-problem-detector, Thanos
, log collection,ProbeWarning
, Kubernetes node-problem-detector, Pixie, OpenMetrics,kind: PodMonitor
, Jaeger
Advertising: