问题定位

1.服务挂的现象出现!
2.检查发现并未出现OOM的溢出DUMP文件产生
3.检查是否是被Linux系统给杀了

服务器环境查询步骤

  1. 服务启动脚本查看
// 查看Shell脚本
[vevor@v169 bmp-crm-manager]$ cat start.sh 
# 脚本内容如下
#!/bin/bash
DATE=$(date +%Y%m%d)
source /etc/profile
DIR=/home/vevor/vevor-bmp-packages/bmp-crm-manager/
cd $DIR
# 获取最新的jar包
JAR=`ls -lt *-spring-boot.jar | head -1 | awk '{print $9}'`
# 获取当前项目的进程的进程号
PID=`ps -ef | grep bmp-crm-service | grep -v grep | awk '{print $2}'`
# 项目dump文件根据启动时候的日期确定
DUMPNAME=`date "+%Y_%m_%d_%H_%M_%S_DUMP"`
# 有进程号就杀进程后再启动新的,没有就直接启动
if [ "$PID" != "" ];then
  kill -9 $PID
  nohup java  -jar -XX:+HeapDumpOnOutOfMemoryError  -XX:HeapDumpPath=$DIR$DUMPNAME \
  -Dspring.cloud.consul.host=172.19.225.168 \
  -Dspring.cloud.consul.port=8500 \
  -Dspring.cloud.consul.config.default-context=test \
  -Dspring.cloud.consul.config.data-key=lichenghuai $DIR$JAR  > /dev/null 2>&1 &
else
  nohup java  -jar -XX:+HeapDumpOnOutOfMemoryError  -XX:HeapDumpPath=$DIR$DUMPNAME \
  -Dspring.cloud.consul.host=172.19.225.168 \
  -Dspring.cloud.consul.port=8500 \
  -Dspring.cloud.consul.config.default-context=test \
  -Dspring.cloud.consul.config.data-key=lichenghuai $DIR$JAR  > /dev/null 2>&1 &
fi

2.启动后的服务进程号查看并记录
在这里插入图片描述
3. 等待服务挂后查看日志【1.么有DUMP产生】
在这里插入图片描述
4. 扩大日志搜索范围【看看是不是Linux干的好事】
命令:dmesg | egrep -i -B100 ‘Kill process’
我们的进程号:9445
在这里插入图片描述
发现咱的进程确实被杀,可是到底啥时候被杀的呢?
命令:date -d “1970-01-01 UTC echo "$(date +%s)-$(cat /proc/uptime|cut -f 1 -d' ')+25094516.126539"|bcseconds”
时间转化成可以看懂的!!!
在这里插入图片描述
total-vm是指所有可用内存,约8G
anon-rss + file-rss 是该进程实际使用的内存,约2.5G,

这个进程实际使用了3.5GB左右内存,为什么会导致被oom killer杀掉呢?

应该是其他进程导致的,我注意到其中有个vim进程(7788),占用的rss内存页个数是814178,内存页大小为4KB,那么这个vim进程占用了814178*4/1024 = 3180MB,那么原因就很明显了,有人用vim打开了一个很大的文件,导致内存耗尽,最终oom killer登场杀死了java进程。
6. 死亡原因报告说明
oom killer是如何来选择一个被终止掉的进程?
原来oom killer巡视一遍所有进程后就给每个进程分配一个分数,也就是"badness score",坏蛋分,分数最高的那个将被kill掉,打分的原则简单讲就是占用内存越高的那个进程得分就越高,很显然我们的进程就是得分最高的那个,然后就被kill掉了,没有留下任何线索。
Top命令查看进程占用资源

【处理方案】

限制内存使用参数,避免被进程杀死:nohup java -Xms1g -Xmx1g -jar

【完整日志】

[25094516.126455] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
[25094516.126462] [  628]     0   628    13962      100      32        0             0 systemd-journal
[25094516.126465] [  654]     0   654    11165      118      22        0         -1000 systemd-udevd
[25094516.126467] [  987]     0   987    13882      113      28        0         -1000 auditd
[25094516.126470] [ 1052]   999  1052   153086     1658      62        0             0 polkitd
[25094516.126472] [ 1058]     0  1058     6701      219      19        0             0 systemd-logind
[25094516.126475] [ 1059]    81  1059    14530      158      32        0          -900 dbus-daemon
[25094516.126477] [ 1063]   998  1063    29482      133      28        0             0 chronyd
[25094516.126479] [ 1296]     0  1296    25724      515      48        0             0 dhclient
[25094516.126481] [ 1354]     0  1354   143549     3329      99        0             0 tuned
[25094516.126483] [ 1500]     0  1500   177430      446     156        0             0 rsyslogd
[25094516.126485] [ 1509]     0  1509     6477       53      17        0             0 atd
[25094516.126488] [ 1511]     0  1511    31573      162      20        0             0 crond
[25094516.126490] [ 1517]     0  1517    27527       34      12        0             0 agetty
[25094516.126492] [ 1518]     0  1518    27527       33      10        0             0 agetty
[25094516.126494] [ 2038]     0  2038     5717       88      15        0             0 argusagent
[25094516.126497] [ 2040]     0  2040   253445     2881      54        0             0 /usr/local/clou
[25094516.126499] [ 3104]     0  3104    28230      256      57        0         -1000 sshd
[25094516.126501] [27621]  1000 27621  1189200   180693     458        0             0 java
[25094516.126503] [ 1690]  1000  1690  1248350   398534     925        0             0 java
[25094516.126505] [ 5052]     0  5052   109229      148      27        0             0 AliSecGuard
[25094516.126507] [  947]     0   947   201651     1497      13        0             0 aliyun-service
[25094516.126509] [ 1069]     0  1069     4470      122      12        0             0 assist_daemon
[25094516.126511] [30180]  1000 30180  1187045   168005     436        0             0 java
[25094516.126513] [21557]  1000 21557  1229125   219482     551        0             0 java
[25094516.126516] [ 1364]  1000  1364  1233480   222741     560        0             0 java
[25094516.126518] [12203]     0 12203    20685     8176      38        0             0 AliYunDunUpdate
[25094516.126521] [22307]  1000 22307  1244476   270965     668        0             0 java
[25094516.126523] [ 2117]  1000  2117  1238545   255050     666        0             0 java
[25094516.126525] [17180]  1000 17180  1292373   245993     771        0             0 java
[25094516.126527] [  915]  1000   915  1391175   269320     962        0             0 java
[25094516.126529] [11040]  1000 11040  1192170   180695     468        0             0 java
[25094516.126531] [14645]     0 14645    32641     1370      64        0             0 AliYunDun
[25094516.126533] [ 3603]  1000  3603  1233254   219157     587        0             0 java
[25094516.126535] [28583]  1000 28583  1201707   171353     483        0             0 java
[25094516.126537] [ 9415]  1000  9415  2073112   923377    2090        0             0 java
[25094516.126539] Out of memory: Kill process 9415 (java) score 237 or sacrifice child
[25094516.127920] Killed process 9415 (java), UID 1000, total-vm:8292448kB, anon-rss:3693380kB, file-rss:128kB, shmem-rss:0kB
[25140556.796701] java invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
[25140556.796705] java cpuset=/ mems_allowed=0
[25140556.796708] CPU: 1 PID: 1549 Comm: java Tainted: G           OE  ------------   3.10.0-1062.18.1.el7.x86_64 #1
[25140556.796710] Hardware name: Alibaba Cloud Alibaba Cloud ECS, BIOS e623647 04/01/2014
[25140556.796711] Call Trace:
[25140556.796720]  [<ffffffff8597b416>] dump_stack+0x19/0x1b
[25140556.796728]  [<ffffffff85975d8d>] dump_header+0x90/0x229
[25140556.796733]  [<ffffffff85306392>] ? ktime_get_ts64+0x52/0xf0
[25140556.796737]  [<ffffffff853c196e>] oom_kill_process+0x25e/0x3f0
[25140556.796740]  [<ffffffff853330c1>] ? cpuset_mems_allowed_intersects+0x21/0x30
[25140556.796743]  [<ffffffff853c13cd>] ? oom_unkillable_task+0xcd/0x120
[25140556.796745]  [<ffffffff853c1476>] ? find_lock_task_mm+0x56/0xc0
[25140556.796747]  [<ffffffff853c21c6>] out_of_memory+0x4b6/0x4f0
[25140556.796749]  [<ffffffff859768aa>] __alloc_pages_slowpath+0x5db/0x729
[25140556.796751]  [<ffffffff853c8636>] __alloc_pages_nodemask+0x436/0x450
[25140556.796755]  [<ffffffff85416c58>] alloc_pages_current+0x98/0x110
[25140556.796757]  [<ffffffff853bd877>] __page_cache_alloc+0x97/0xb0
[25140556.796760]  [<ffffffff853c0430>] filemap_fault+0x270/0x420
[25140556.796774]  [<ffffffffc0577356>] ext4_filemap_fault+0x36/0x50 [ext4]
[25140556.796778]  [<ffffffff853ebdba>] __do_fault.isra.61+0x8a/0x100
[25140556.796780]  [<ffffffff853ec36c>] do_read_fault.isra.63+0x4c/0x1b0
[25140556.796782]  [<ffffffff853f3bd0>] handle_mm_fault+0xa20/0xfb0
[25140556.796787]  [<ffffffff85312110>] ? futex_wake+0x90/0x180
[25140556.796792]  [<ffffffff85988653>] __do_page_fault+0x213/0x500
[25140556.796794]  [<ffffffff85988a26>] trace_do_page_fault+0x56/0x150
[25140556.796797]  [<ffffffff85987fa2>] do_async_page_fault+0x22/0xf0
[25140556.796799]  [<ffffffff859847a8>] async_page_fault+0x28/0x30
[25140556.796800] Mem-Info:
[25140556.796805] active_anon:3757967 inactive_anon:78 isolated_anon:0
 active_file:999 inactive_file:2553 isolated_file:99
 unevictable:0 dirty:19 writeback:0 unstable:0
 slab_reclaimable:15833 slab_unreclaimable:11246
 mapped:277 shmem:230 pagetables:10853 bounce:0
 free:33154 free_pcp:949 free_cma:0
[25140556.796808] Node 0 DMA free:15908kB min:68kB low:84kB high:100kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[25140556.796814] lowmem_reserve[]: 0 2813 15178 15178
[25140556.796816] Node 0 DMA32 free:61844kB min:12516kB low:15644kB high:18772kB active_anon:2739356kB inactive_anon:64kB active_file:448kB inactive_file:2688kB unevictable:0kB isolated(anon):0kB isolated(file):328kB present:3111608kB managed:2881176kB mlocked:0kB dirty:12kB writeback:0kB mapped:40kB shmem:208kB slab_reclaimable:22436kB slab_unreclaimable:11368kB kernel_stack:11536kB pagetables:5840kB unstable:0kB bounce:0kB free_pcp:2120kB local_pcp:632kB free_cma:0kB writeback_tmp:0kB pages_scanned:5702 all_unreclaimable? yes
[25140556.796821] lowmem_reserve[]: 0 0 12364 12364
[25140556.796824] Node 0 Normal free:54864kB min:54996kB low:68744kB high:82492kB active_anon:12292512kB inactive_anon:248kB active_file:3548kB inactive_file:7524kB unevictable:0kB isolated(anon):0kB isolated(file):68kB present:12931072kB managed:12661380kB mlocked:0kB dirty:64kB writeback:76kB mapped:1068kB shmem:712kB slab_reclaimable:40896kB slab_unreclaimable:33616kB kernel_stack:30512kB pagetables:37572kB unstable:0kB bounce:0kB free_pcp:1676kB local_pcp:68kB free_cma:0kB writeback_tmp:0kB pages_scanned:17171 all_unreclaimable? yes
[25140556.796828] lowmem_reserve[]: 0 0 0 0
[25140556.796831] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
[25140556.796840] Node 0 DMA32: 190*4kB (UE) 246*8kB (UEM) 422*16kB (UEM) 194*32kB (UEM) 240*64kB (UEM) 144*128kB (UEM) 39*256kB (UEM) 1*512kB (M) 2*1024kB (M) 0*2048kB 0*4096kB = 62024kB
[25140556.796850] Node 0 Normal: 13699*4kB (UE) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 54796kB
[25140556.796857] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[25140556.796858] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[25140556.796859] 4105 total pagecache pages
[25140556.796861] 0 pages in swap cache
[25140556.796862] Swap cache stats: add 0, delete 0, find 0/0
[25140556.796863] Free swap  = 0kB
[25140556.796864] Total swap = 0kB
[25140556.796865] 4014668 pages RAM
[25140556.796866] 0 pages HighMem/MovableOnly
[25140556.796867] 125052 pages reserved
[25140556.796868] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
[25140556.796875] [  628]     0   628    13962      102      33        0             0 systemd-journal
[25140556.796878] [  654]     0   654    11165      118      22        0         -1000 systemd-udevd
[25140556.796880] [  987]     0   987    13882      113      28        0         -1000 auditd
[25140556.796882] [ 1052]   999  1052   153086     1658      62        0             0 polkitd
[25140556.796885] [ 1058]     0  1058     6701      213      19        0             0 systemd-logind
[25140556.796887] [ 1059]    81  1059    14530      156      32        0          -900 dbus-daemon
[25140556.796889] [ 1063]   998  1063    29482      145      28        0             0 chronyd
[25140556.796891] [ 1296]     0  1296    25724      515      48        0             0 dhclient
[25140556.796893] [ 1354]     0  1354   143549     3329      99        0             0 tuned
[25140556.796896] [ 1500]     0  1500   177430      443     157        0             0 rsyslogd
[25140556.796898] [ 1509]     0  1509     6477       53      17        0             0 atd
[25140556.796900] [ 1511]     0  1511    31573      163      20        0             0 crond
[25140556.796902] [ 1517]     0  1517    27527       34      12        0             0 agetty
[25140556.796904] [ 1518]     0  1518    27527       33      10        0             0 agetty
[25140556.796906] [ 2038]     0  2038     5717       88      15        0             0 argusagent
[25140556.796908] [ 2040]     0  2040   253446     2893      54        0             0 /usr/local/clou
[25140556.796910] [ 3104]     0  3104    28230      256      57        0         -1000 sshd
[25140556.796913] [27621]  1000 27621  1189200   180693     458        0             0 java
[25140556.796915] [ 1690]  1000  1690  1248350   398534     925        0             0 java
[25140556.796917] [ 5052]     0  5052   109229      148      27        0             0 AliSecGuard
[25140556.796919] [  947]     0   947   201651     1512      13        0             0 aliyun-service
[25140556.796921] [ 1069]     0  1069     4470      122      12        0             0 assist_daemon
[25140556.796923] [30180]  1000 30180  1187045   168008     436        0             0 java
[25140556.796925] [21557]  1000 21557  1229125   219506     551        0             0 java
[25140556.796927] [ 1364]  1000  1364  1233480   222750     560        0             0 java
[25140556.796929] [12203]     0 12203    20685     8176      38        0             0 AliYunDunUpdate
[25140556.796931] [22307]  1000 22307  1244476   271006     668        0             0 java
[25140556.796934] [ 2117]  1000  2117  1238545   262222     678        0             0 java
[25140556.796936] [17180]  1000 17180  1292373   247051     773        0             0 java
[25140556.796938] [  915]  1000   915  1391175   269373     962        0             0 java
[25140556.796940] [11040]  1000 11040  1192170   181123     469        0             0 java
[25140556.796942] [14645]     0 14645    32641     1349      64        0             0 AliYunDun
[25140556.796944] [ 3603]  1000  3603  1233254   219423     587        0             0 java
[25140556.796946] [28583]  1000 28583  1201707   172433     483        0             0 java
[25140556.796948] [ 9445]  1000  9445  2070967   618667    1498        0             0 java
[25140556.796950] [14314]     0 14314    38677      346      78        0             0 sshd
[25140556.796951] [14317]  1000 14317    38677      326      75        0             0 sshd
[25140556.796953] [14318]  1000 14318    28895      140      15        0             0 bash
[25140556.796955] [15150]     0 15150    38677      346      77        0             0 sshd
[25140556.796957] [15153]  1000 15153    38907      550      75        0             0 sshd
[25140556.796959] [15154]  1000 15154    28895      137      13        0             0 bash
[25140556.796960] [15273]  1000 15273   317269   280581     578        0             0 vim
[25140556.796962] Out of memory: Kill process 9445 (java) score 159 or sacrifice child
[25140556.798288] Killed process 9445 (java), UID 1000, total-vm:8283868kB, anon-rss:2474472kB, file-rss:196kB, shmem-rss:0kB
[25140578.512344] AliYunDunUpdate invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
[25140578.512348] AliYunDunUpdate cpuset=/ mems_allowed=0
[25140578.512351] CPU: 3 PID: 12203 Comm: AliYunDunUpdate Tainted: G           OE  ------------   3.10.0-1062.18.1.el7.x86_64 #1
[25140578.512353] Hardware name: Alibaba Cloud Alibaba Cloud ECS, BIOS e623647 04/01/2014
[25140578.512354] Call Trace:
[25140578.512364]  [<ffffffff8597b416>] dump_stack+0x19/0x1b
[25140578.512368]  [<ffffffff85975d8d>] dump_header+0x90/0x229
[25140578.512373]  [<ffffffff85306392>] ? ktime_get_ts64+0x52/0xf0
[25140578.512378]  [<ffffffff853c196e>] oom_kill_process+0x25e/0x3f0
[25140578.512381]  [<ffffffff853330c1>] ? cpuset_mems_allowed_intersects+0x21/0x30
[25140578.512384]  [<ffffffff853c13cd>] ? oom_unkillable_task+0xcd/0x120
[25140578.512386]  [<ffffffff853c1476>] ? find_lock_task_mm+0x56/0xc0
[25140578.512389]  [<ffffffff853c21c6>] out_of_memory+0x4b6/0x4f0
[25140578.512391]  [<ffffffff859768aa>] __alloc_pages_slowpath+0x5db/0x729
[25140578.512394]  [<ffffffff853c8636>] __alloc_pages_nodemask+0x436/0x450
[25140578.512399]  [<ffffffff85416c58>] alloc_pages_current+0x98/0x110
[25140578.512401]  [<ffffffff853bd877>] __page_cache_alloc+0x97/0xb0
[25140578.512404]  [<ffffffff853c0430>] filemap_fault+0x270/0x420
[25140578.512420]  [<ffffffffc0577356>] ext4_filemap_fault+0x36/0x50 [ext4]
[25140578.512424]  [<ffffffff853ebdba>] __do_fault.isra.61+0x8a/0x100
[25140578.512428]  [<ffffffff852e6601>] ? put_prev_entity+0x31/0x400
[25140578.512430]  [<ffffffff853ec36c>] do_read_fault.isra.63+0x4c/0x1b0
[25140578.512433]  [<ffffffff853f3bd0>] handle_mm_fault+0xa20/0xfb0
[25140578.512438]  [<ffffffff852ca42d>] ? hrtimer_start_range_ns+0x1fd/0x3c0
[25140578.512444]  [<ffffffff85988653>] __do_page_fault+0x213/0x500
[25140578.512447]  [<ffffffff85988a26>] trace_do_page_fault+0x56/0x150
[25140578.512450]  [<ffffffff85987fa2>] do_async_page_fault+0x22/0xf0
[25140578.512452]  [<ffffffff859847a8>] async_page_fault+0x28/0x30
[25140578.512454] Mem-Info:
[25140578.512460] active_anon:3764487 inactive_anon:76 isolated_anon:0
 active_file:929 inactive_file:2890 isolated_file:83
 unevictable:0 dirty:21 writeback:14 unstable:0
 slab_reclaimable:15787 slab_unreclaimable:11254
 mapped:293 shmem:222 pagetables:10595 bounce:0
 free:33118 free_pcp:430 free_cma:0
[25140578.512464] Node 0 DMA free:15908kB min:68kB low:84kB high:100kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[25140578.512470] lowmem_reserve[]: 0 2813 15178 15178
[25140578.512474] Node 0 DMA32 free:61748kB min:12516kB low:15644kB high:18772kB active_anon:2745676kB inactive_anon:64kB active_file:916kB inactive_file:4044kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:3111608kB managed:2881176kB mlocked:0kB dirty:52kB writeback:36kB mapped:172kB shmem:208kB slab_reclaimable:22340kB slab_unreclaimable:11368kB kernel_stack:10688kB pagetables:5836kB unstable:0kB bounce:0kB free_pcp:608kB local_pcp:236kB free_cma:0kB writeback_tmp:0kB pages_scanned:9201 all_unreclaimable? yes
[25140578.512480] lowmem_reserve[]: 0 0 12364 12364
[25140578.512482] Node 0 Normal free:54816kB min:54996kB low:68744kB high:82492kB active_anon:12312272kB inactive_anon:240kB active_file:2800kB inactive_file:7516kB unevictable:0kB isolated(anon):0kB isolated(file):204kB present:12931072kB managed:12661380kB mlocked:0kB dirty:32kB writeback:20kB mapped:1000kB shmem:680kB slab_reclaimable:40808kB slab_unreclaimable:33648kB kernel_stack:25488kB pagetables:36544kB unstable:0kB bounce:0kB free_pcp:1112kB local_pcp:120kB free_cma:0kB writeback_tmp:0kB pages_scanned:16466 all_unreclaimable? yes
[25140578.512488] lowmem_reserve[]: 0 0 0 0
[25140578.512491] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kB
[25140578.512502] Node 0 DMA32: 380*4kB (UEM) 243*8kB (UEM) 468*16kB (UEM) 297*32kB (UE) 220*64kB (UEM) 109*128kB (UE) 24*256kB (UEM) 10*512kB (M) 2*1024kB (M) 0*2048kB 0*4096kB = 61800kB
[25140578.512514] Node 0 Normal: 13710*4kB (UE) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 54840kB
[25140578.512522] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[25140578.512524] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[25140578.512525] 4250 total pagecache pages
[25140578.512527] 0 pages in swap cache
[25140578.512528] Swap cache stats: add 0, delete 0, find 0/0
[25140578.512529] Free swap  = 0kB
[25140578.512531] Total swap = 0kB
[25140578.512532] 4014668 pages RAM
[25140578.512533] 0 pages HighMem/MovableOnly
[25140578.512534] 125052 pages reserved
[25140578.512535] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
[25140578.512543] [  628]     0   628    13962       98      33        0             0 systemd-journal
[25140578.512546] [  654]     0   654    11165      118      22        0         -1000 systemd-udevd
[25140578.512549] [  987]     0   987    13882      113      28        0         -1000 auditd
[25140578.512552] [ 1052]   999  1052   153086     1658      62        0             0 polkitd
[25140578.512555] [ 1058]     0  1058     6701      197      19        0             0 systemd-logind
[25140578.512557] [ 1059]    81  1059    14530      139      32        0          -900 dbus-daemon
[25140578.512560] [ 1063]   998  1063    29482      141      28        0             0 chronyd
[25140578.512562] [ 1296]     0  1296    25724      515      48        0             0 dhclient
[25140578.512565] [ 1354]     0  1354   143549     3329      99        0             0 tuned
[25140578.512567] [ 1500]     0  1500   177430      443     157        0             0 rsyslogd
[25140578.512570] [ 1509]     0  1509     6477       53      17        0             0 atd
[25140578.512573] [ 1511]     0  1511    31573      160      20        0             0 crond
[25140578.512575] [ 1517]     0  1517    27527       34      12        0             0 agetty
[25140578.512578] [ 1518]     0  1518    27527       33      10        0             0 agetty
[25140578.512580] [ 2038]     0  2038     5717       88      15        0             0 argusagent
[25140578.512583] [ 2040]     0  2040   253445     2893      54        0             0 /usr/local/clou
[25140578.512585] [ 3104]     0  3104    28230      256      57        0         -1000 sshd
[25140578.512588] [27621]  1000 27621  1189200   180693     458        0             0 java
[25140578.512592] [ 1690]  1000  1690  1248350   398534     925        0             0 java
[25140578.512611] [ 5052]     0  5052   109229      148      27        0             0 AliSecGuard
[25140578.512613] [  947]     0   947   201651     1512      13        0             0 aliyun-service
[25140578.512616] [ 1069]     0  1069     4470      122      12        0             0 assist_daemon
[25140578.512618] [30180]  1000 30180  1187045   168008     436        0             0 java
[25140578.512621] [21557]  1000 21557  1229125   219506     551        0             0 java
[25140578.512623] [ 1364]  1000  1364  1233480   222750     560        0             0 java
[25140578.512625] [12203]     0 12203    20685     8176      38        0             0 AliYunDunUpdate
[25140578.512628] [22307]  1000 22307  1244476   271006     668        0             0 java
[25140578.512630] [ 2117]  1000  2117  1238545   262222     678        0             0 java
[25140578.512632] [17180]  1000 17180  1292373   247051     773        0             0 java
[25140578.512634] [  915]  1000   915  1391175   269373     962        0             0 java
[25140578.512637] [11040]  1000 11040  1192170   181123     469        0             0 java
[25140578.512639] [14645]     0 14645    32641     1349      64        0             0 AliYunDun
[25140578.512642] [ 3603]  1000  3603  1233254   219423     587        0             0 java
[25140578.512644] [28583]  1000 28583  1201707   172483     483        0             0 java
[25140578.512646] [14314]     0 14314    38677      339      78        0             0 sshd
[25140578.512648] [14317]  1000 14317    38677      336      75        0             0 sshd
[25140578.512650] [14318]  1000 14318    28895      135      15        0             0 bash
[25140578.512653] [15150]     0 15150    38677      339      77        0             0 sshd
[25140578.512655] [15153]  1000 15153    38907      543      75        0             0 sshd
[25140578.512657] [15154]  1000 15154    28895      132      13        0             0 bash
[25140578.512659] [15273]  1000 15273   950884   914042    1818        0             0 vim
[25140578.512661] Out of memory: Kill process 15273 (vim) score 235 or sacrifice child
[25140578.514172] Killed process 15273 (vim), UID 1000, total-vm:3803536kB, anon-rss:3656016kB, file-rss:152kB, shmem-rss:0kB

【神坑】

当系统内存不足时oom killer会基于一套算法选出一个进程并杀掉!不要问我啥算法!也不要问我为啥挂的都是俺!

【参考资料】

内存溢出(OOM)问题分析
一个嘴贱引发的内存异常bug

GitHub 加速计划 / li / linux-dash
10.39 K
1.2 K
下载
A beautiful web dashboard for Linux
最近提交(Master分支:2 个月前 )
186a802e added ecosystem file for PM2 4 年前
5def40a3 Add host customization support for the NodeJS version 4 年前
Logo

旨在为数千万中国开发者提供一个无缝且高效的云端环境,以支持学习、使用和贡献开源项目。

更多推荐