Linux服务器EDAC CE memory read error
linux-dash
A beautiful web dashboard for Linux
项目地址：https://gitcode.com/gh_mirrors/li/linux-dash
免费下载资源
·
之前在大数据集群中，有一台服务器的CPU占用总是莫名其妙飙高，就算执行简单任务也会耗费很长时间，且reboot不能解决问题。检查了各种可能的问题之后，最终在查看dmesg命令的设备信息时，发现大量如下的日志：
......
[1180532.573917] EDAC MC1: 128 CE memory read error on CPU_SrcID#1_Ha#0_Chan#1_DIMM#1 (channel:1 slot:1 page:0x2b67afa offset:0xb40 grain:32 syndrome:0x0 - area:DRAM err_code:0001:0092 socket:1 ha:0 channel_mask:2 rank:5)
[1180532.573925] EDAC MC1: 147 CE memory read error on CPU_SrcID#1_Ha#0_Chan#1_DIMM#1 (channel:1 slot:1 page:0x30b6e9a offset:0x540 grain:32 syndrome:0x0 - area:DRAM err_code:0001:0092 socket:1 ha:0 channel_mask:2 rank:5)
[1180532.573930] EDAC MC1: 236 CE memory read error on CPU_SrcID#1_Ha#0_Chan#0_DIMM#1 (channel:0 slot:1 page:0x236e7bb offset:0xd00 grain:32 syndrome:0x0 - area:DRAM err_code:0001:0092 socket:1 ha:0 channel_mask:1 rank:5)
[1180532.573935] EDAC MC1: 189 CE memory read error on CPU_SrcID#1_Ha#0_Chan#1_DIMM#1 (channel:1 slot:1 page:0x365669d offset:0xf40 grain:32 syndrome:0x0 - area:DRAM err_code:0001:0092 socket:1 ha:0 channel_mask:2 rank:5)
[1180532.573940] EDAC MC1: 255 CE memory read error on CPU_SrcID#1_Ha#0_Chan#1_DIMM#1 (channel:1 slot:1 page:0x3d2953a offset:0x540 grain:32 syndrome:0x0 - area:DRAM err_code:0001:0092 socket:1 ha:0 channel_mask:2 rank:5)
[1180532.573944] EDAC MC1: 177 CE memory read error on CPU_SrcID#1_Ha#0_Chan#1_DIMM#1 (channel:1 slot:1 page:0x212b93c offset:0x540 grain:32 syndrome:0x0 - area:DRAM err_code:0001:0092 socket:1 ha:0 channel_mask:2 rank:5)
[1180532.573949] EDAC MC1: 136 CE memory read error on CPU_SrcID#1_Ha#0_Chan#1_DIMM#1 (channel:1 slot:1 page:0x31d099d offset:0xb40 grain:32 syndrome:0x0 - area:DRAM err_code:0001:0092 socket:1 ha:0 channel_mask:2 rank:5)
[1180532.573953] EDAC MC1: 88 CE memory read error on CPU_SrcID#1_Ha#0_Chan#0_DIMM#1 (channel:0 slot:1 page:0x228e5be offset:0x100 grain:32 syndrome:0x0 - area:DRAM err_code:0001:0092 socket:1 ha:0 channel_mask:1 rank:5)
[1180532.573958] EDAC MC1: 111 CE memory read error on CPU_SrcID#1_Ha#0_Chan#0_DIMM#1 (channel:0 slot:1 page:0x34e5e7b offset:0xf00 grain:32 syndrome:0x0 - area:DRAM err_code:0001:0092 socket:1 ha:0 channel_mask:1 rank:5)
[1180532.573962] EDAC MC1: 149 CE memory read error on CPU_SrcID#1_Ha#0_Chan#0_DIMM#1 (channel:0 slot:1 page:0x314bb9f offset:0xd00 grain:32 syndrome:0x0 - area:DRAM err_code:0001:0092 socket:1 ha:0 channel_mask:1 rank:5)
[1180532.611035] EDAC sbridge MC1: HANDLING MCE MEMORY ERROR
[1180532.611045] EDAC sbridge MC1: HANDLING MCE MEMORY ERROR
[1180533.460534] EDAC sbridge MC1: HANDLING MCE MEMORY ERROR
[1180533.460544] EDAC sbridge MC1: HANDLING MCE MEMORY ERROR
[1180533.547047] EDAC sbridge MC1: HANDLING MCE MEMORY ERROR
......
EDAC即error detection and correction（错误检测与纠正），是Linux系统内部的机制。在上面的日志中，可以清楚地看出是内存读错误。MC即memory controller（内存控制器）。CE则代表correctable error，是ECC内存中可以纠正的错误，相对地还有UE（uncorrectable error）。
为了摸清是哪些内存出了问题，找出所有内存的CE计数：
~ grep "[0-9]" /sys/devices/system/edac/mc/mc*/csrow*/ch*_ce_count
/sys/devices/system/edac/mc/mc0/csrow0/ch0_ce_count:0
/sys/devices/system/edac/mc/mc0/csrow0/ch1_ce_count:0
/sys/devices/system/edac/mc/mc0/csrow0/ch2_ce_count:0
/sys/devices/system/edac/mc/mc0/csrow0/ch3_ce_count:0
/sys/devices/system/edac/mc/mc0/csrow1/ch0_ce_count:0
/sys/devices/system/edac/mc/mc0/csrow1/ch1_ce_count:0
/sys/devices/system/edac/mc/mc0/csrow1/ch2_ce_count:0
/sys/devices/system/edac/mc/mc0/csrow1/ch3_ce_count:0
/sys/devices/system/edac/mc/mc1/csrow0/ch0_ce_count:0
/sys/devices/system/edac/mc/mc1/csrow0/ch1_ce_count:0
/sys/devices/system/edac/mc/mc1/csrow0/ch2_ce_count:0
/sys/devices/system/edac/mc/mc1/csrow0/ch3_ce_count:0
/sys/devices/system/edac/mc/mc1/csrow1/ch0_ce_count:198276530
/sys/devices/system/edac/mc/mc1/csrow1/ch1_ce_count:233266714
/sys/devices/system/edac/mc/mc1/csrow1/ch2_ce_count:0
/sys/devices/system/edac/mc/mc1/csrow1/ch3_ce_count:0
可见是mc1/csrow1/ch0和mc1/csrow1/ch1两个通道上的内存出了问题。mc、csrow（chip-select row）与ch之间的关系类似下图：
mc0 ch0 ch1
==============================
csrow0 | DIMM_A0 | DIMM_B0 |
csrow1 | DIMM_A0 | DIMM_B0 |
==============================
mc1 ch0 ch1
==============================
csrow2 | DIMM_A1 | DIMM_B1 |
csrow3 | DIMM_A1 | DIMM_B1 |
==============================
通过dmidecode -t memory命令，可以查看每个DIMM（也就是常说的内存条）对应的位置。然后根据映射关系，去服务器上将出现问题的两根内存条更换，问题就解决了。
GitHub 加速计划 / li / linux-dash
10.39 K
1.2 K
下载
A beautiful web dashboard for Linux
最近提交(Master分支：2 个月前 )
186a802e
added ecosystem file for PM2 4 年前
5def40a3
Add host customization support for the NodeJS version 4 年前
旨在为数千万中国开发者提供一个无缝且高效的云端环境，以支持学习、使用和贡献开源项目。
更多推荐
3
0- 0
已为社区贡献1条内容
所有评论(0)