Difference between revisions of "Out of Memory (OOM)"
Jump to navigation
Jump to search
| (11 intermediate revisions by 2 users not shown) | |||
| Line 5: | Line 5: | ||
* [[Docker]]: <code>--oom-kill-disable=false</code> To enable or disable [[OOM Killer]] for the [[container]]. | * [[Docker]]: <code>--oom-kill-disable=false</code> To enable or disable [[OOM Killer]] for the [[container]]. | ||
Use <code>[[docker inspect]] CONTANINER_NAME | grep OOM</code> to show options. For example: | Use <code>[[docker inspect]] CONTANINER_NAME | grep OOM</code> to show options. For example: | ||
| − | + | {{docker inspect OOMM}} | |
| − | |||
| − | |||
| − | |||
== Kubernetes == | == Kubernetes == | ||
| Line 16: | Line 13: | ||
== OOM Linux kernel logs == | == OOM Linux kernel logs == | ||
| − | Jan 26 04:40:29 SERVER_NAME kernel: [[runc]]:[2:INIT] invoked [[oom-killer]]: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0 | + | Jan 26 04:40:29 SERVER_NAME kernel: [[runc]]:[2:INIT] invoked [[oom-killer]]: [[gfp_mask]]=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, [[oom_score_adj]]=0 |
Jan 26 04:40:29 SERVER_NAME kernel: [[oom_reaper]]: reaped process 22572 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB | Jan 26 04:40:29 SERVER_NAME kernel: [[oom_reaper]]: reaped process 22572 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB | ||
| + | |||
| + | May 22 14:02:59 ip-10-10-10-10 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=init.scope,mems_allowed=0,global_oom,task_memcg=/system.slice/system-postgresql.slice/[email protected],task=postgres,pid=9120,uid=115 | ||
| + | |||
| + | May 22 14:02:59 ip-10-10-10-10 kernel: Out of memory: Killed process 9120 (postgres) total-vm:1493744kB, anon-rss:280412kB, file-rss:2052kB, shmem-rss:963576kB, UID:115 pgtables:2856kB oom_score_adj:0 | ||
== Redis == | == Redis == | ||
| Line 27: | Line 28: | ||
* [[Out of space]] | * [[Out of space]] | ||
* <code>[[/proc/sys/vm/]]panic_on_oom</code> | * <code>[[/proc/sys/vm/]]panic_on_oom</code> | ||
| + | * <code>[[/proc/sys/vm/oom_kill_allocating_task]]</code> | ||
* [[yarn]]: <code>[[Error Command failed with exit code 137]]</code> | * [[yarn]]: <code>[[Error Command failed with exit code 137]]</code> | ||
* <code>[[systemd-oomd]]</code> | * <code>[[systemd-oomd]]</code> | ||
* <code>[[pm.max_requests]]</code> | * <code>[[pm.max_requests]]</code> | ||
* <code>[[EarlyOOM]]</code> | * <code>[[EarlyOOM]]</code> | ||
| + | * <code>[[FailedScheduling]] .../... [[Insufficient memory]]</code> | ||
| + | * <code>[[oom_kill_process.cold]]</code> | ||
| + | * <code>[[sar -r]]</code> | ||
== See also == | == See also == | ||
Latest revision as of 15:11, 22 May 2023
OOM in Docker[edit]
- Docker:
--oom-kill-disable=falseTo enable or disable OOM Killer for the container.
Use docker inspect CONTANINER_NAME | grep OOM to show options. For example:
docker inspect grafana | grep -i OOM "OOMKilled": false, "OomScoreAdj": 0, "OomKillDisable": false,
Kubernetes[edit]
Pod The node had condition: [MemoryPressure].kubectl describe nodes | grep MemoryPressureCrashLoopBackOff
OOM Linux kernel logs[edit]
Jan 26 04:40:29 SERVER_NAME kernel: runc:[2:INIT] invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
Jan 26 04:40:29 SERVER_NAME kernel: oom_reaper: reaped process 22572 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
May 22 14:02:59 ip-10-10-10-10 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=init.scope,mems_allowed=0,global_oom,task_memcg=/system.slice/system-postgresql.slice/[email protected],task=postgres,pid=9120,uid=115
May 22 14:02:59 ip-10-10-10-10 kernel: Out of memory: Killed process 9120 (postgres) total-vm:1493744kB, anon-rss:280412kB, file-rss:2052kB, shmem-rss:963576kB, UID:115 pgtables:2856kB oom_score_adj:0
Redis[edit]
Related terms[edit]
Java.lang.OutOfMemoryError: Java heap space- Out of space
/proc/sys/vm/panic_on_oom/proc/sys/vm/oom_kill_allocating_task- yarn:
Error Command failed with exit code 137 systemd-oomdpm.max_requestsEarlyOOMFailedScheduling .../... Insufficient memoryoom_kill_process.coldsar -r
See also[edit]
- Memory: memory pages, RAM, virsh Memory Commands, OOM, meminfo,
vmstat, NAND, DDR,lsmem,/dev/shm,/proc/meminfo,sar -r, IOMMU,pmem, Memory management, Garbage collector, THP, Linux Huge Page TLB - OOM, EarlyOOM, OOM Killer, MemoryPressure (K8s),
systemd-oomd, OOMKilled, oom-killer, memcpy,sar -r,Exit code: 137 - ENOMEM
Advertising:
