我们在 Kdump-内核崩溃转储服务中，介绍了kdump服务可以在系统崩溃时，生成崩溃转储文件vmcore，便于我们分析内核崩溃原因，那么，下面我们就来看下如何使用crash工具可以帮助我们分析系统崩溃原因。

crash工具，跟gdb很类似，它可以交互的分析运行中的内核，也可以分析由kdump、netdump、diskdump、xendump产生的core dump文件。

本文实验环境：

[root@yglocal ~]# cat /etc/redhat-release 
CentOS Linux release 7.6.1810 (Core) 
[root@yglocal ~]# uname -r
3.10.0-957.el7.x86_64

一、调试环境准备

使用crash工具分析vmcore，需要：

• crash工具
• 崩溃转储文件(vmcore)
• 发生崩溃的内核映像文件(vmlinux)，包含调试内核所需调试信息

一般系统在安装后在/boot目录下，也有个内核映像文件，vmlinuxz开头的文件，但是它是压缩过后的，无法完成调试工作，如下图：

所以我们需要下载带有完整调试信息的内核映像文件(编译时带-g选项)，内核调试信息包kernel-debuginfo有两个：

• kernel-debuginfo
• kernel-debuginfo-common

对于centos系统，可以在http://debuginfo.centos.org/上下载到各发行版本所需的调试包。
对于centos7.x，安装对应内核版本的内核调试包，执行如下即可：

# wget http://debuginfo.centos.org/7/x86_64/kernel-debuginfo-common-x86_64-`uname -r`.rpm
# wget http://debuginfo.centos.org/7/x86_64/kernel-debuginfo-`uname -r`.rpm

注意：如果系统为centos6.x，则将debuginfo.centos.org/后面的7改成6即可。

对于oracle linux系统，可以在https://oss.oracle.com/上下载内核调试包，比如现在要找oracle linux6.6的：

# wget https://oss.oracle.com/ol6/debuginfo/kernel-uek-debuginfo-3.8.13-44.1.1.el6uek.x86_64.rpm
# wget https://oss.oracle.com/ol6/debuginfo/kernel-uek-debuginfo-common-3.8.13-44.1.1.el6uek.x86_64.rpm

下载完后，安装内核调试包：

rpm -ivh *.rpm

安装完成后，可以在/lib/debug/lib/modules/3.10.0-957.el7.x86_64/(或者是/usr/lib/…下面)目录下看到vmlinux内核映像文件：

[root@yglocal ~]# ll -th /lib/debug/lib/modules/3.10.0-957.el7.x86_64/
total 419M
drwxr-xr-x.  2 root root  119 Mar 26 13:13 vdso
drwxr-xr-x. 12 root root  128 Mar 26 13:13 kernel
-rwxr-xr-x.  2 root root 419M Nov  9  2018 vmlinux

再来安装crash工具，先查看下是否已安装

[root@yglocal ~]# rpm -q crash
crash-7.2.3-8.el7.x86_64

若没有安装过，则执行以下命令安装：

yum install crash

二、使用crash分析vmcore

分析vmcore文件，执行命令：

crash /lib/debug/lib/modules/3.10.0-957.el7.x86_64/vmlinux /var/crash/127.0.0.1-2020-04-04-14\:10\:45/vmcore

其中下面这些信息，就是导致系统崩溃的直接原因及进程相关信息：

 PANIC: "BUG: unable to handle kernel NULL pointer dereference at 000000000000000c"
    PID: 27005
COMMAND: "pickup"
   TASK: ffff997b388ae180  [THREAD_INFO: ffff997b37f64000]
    CPU: 1
  STATE: TASK_RUNNING (PANIC)

1、bt命令

backtrace打印内核栈回溯信息，bt pid 打印指定进程栈信息。

最重要的信息：[exception RIP: my_openat+36]，指出发生异常的指令信息

RIP: ffffffffc07c5024，可以得知，发生crash的函数是my_openat中，偏移36字节处的指令。

这里，对应x86-64汇编，应用层下来的系统调用对应的6个参数存放的寄存器依次对应：rdi、rsi、rdx、rcx、r8、r9。对于多于6个参数的，仍存储在栈上。

2、log命令

打印vmcore所在的系统内核日志信息

可以分析出当时，内核启动的各项配置，内核最后崩溃时异常日志信息。

3、dis命令

dis -l (function+offset) 10 反汇编出指令所在代码开始，10行代码

幸运的话，反汇编出来会直接对应源码，我们这里只能看出执行mov 0x70(%rdi),%r13时系统崩溃。

4、mod命令

mod 查看当时内核加载的所有内核模块信息

重装加载进改内核模块：

crash> mod -s my_test_lkm /mnt/hgfs/test_ko/lkm-test05/my_test_lkm.ko
     MODULE       NAME                            SIZE  OBJECT FILE
ffffffffc07c7000  my_test_lkm                    12740  /mnt/hgfs/test_ko/lkm-test05/my_test_lkm.ko

5、sym命令

sym 转换指定符号为其虚拟地址，显示系统中符号信息

如上面bt打印的RIP: ffffffffc07c5024，使用sym转换查看系统符号信息
sym addr 通过虚拟地址，显示symbol和源码位置

crash> sym ffffffffc07c5024
ffffffffc07c5024 (t) my_openat+36 [my_test_lkm] /mnt/hgfs/test_ko/lkm-test05/my_lkm.c: 25

这时就可以看出对应到my_test_lkm模块的源码，/mnt/hgfs/test_ko/lkm-test05/my_lkm.c文件里第25行，查看代码可以发现：

原因：这种hook写法在centos8.x上的最新系统调用约定，是内核版本4.17及之后采用的调用约定。而我当前环境是centos7.6，内核版本为3.10。0，其调用约定并不是这样的，所以这样取参数是有问题的。

sym symbol 通过symbol，显示虚拟地址和源码位置：

crash> sym vfs_fstat
ffffffff81196f30 (T) vfs_fstat ../debug/kernel-3.8.13/linux-3.8.13-44.1.1.el6uek/fs/stat.c: 59

6、ps命令

ps 打印内核崩溃时，正常的进程信息

带 > 标识代表是活跃的进程，ps pid打印某指定进程的状态信息：

crash> ps 27005
   PID    PPID  CPU       TASK        ST  %MEM     VSZ    RSS  COMM
> 27005   7783   1  ffff997b388ae180  RU   0.2   91732   4124  pickup

查看指定进程的进程树（ps -p pid）

crash> ps -p 85151
PID: 0      TASK: ffffffff818b6420  CPU: 0   COMMAND: "swapper/0"
 PID: 1      TASK: ffff881f91dae040  CPU: 28  COMMAND: "init"
  PID: 14544  TASK: ffff881f8d7b05c0  CPU: 11  COMMAND: "init.tfa"
   PID: 85138  TASK: ffff880bab01a400  CPU: 8   COMMAND: "tfactl"
    PID: 85151  TASK: ffff880b7a728380  CPU: 17  COMMAND: "perl"

7、files命令

files pid 打印指定进程所打开的文件信息

8、vm命令

vm pid 打印某指定进程当时虚拟内存基本信息

9、task命令

task 查看当前进程或指定进程task_struct和thread_info的信息

10、kmem命令

kmen 查看当时系统内存使用信息

kmem [-f|-F|-c|-C|-i|-v|-V|-n|-z|-o|-h] [-p | -m member[,member]]

       [[-s|-S] [slab] [-I slab[,slab]]] [-g [flags]] [[-P] address]]

我们是要kmem -i查看：

11、struct命令

1、查看结构体成员变量，直接struct后面跟结构体名，比如我要看下内核中dentry结构体的定义，可以这样：

crash> struct dentry
struct dentry {
    unsigned int d_flags;
    seqcount_t d_seq;
    struct hlist_bl_node d_hash;
    struct dentry *d_parent;
    struct qstr d_name;
    struct inode *d_inode;
    unsigned char d_iname[32];
    unsigned int d_count;
    spinlock_t d_lock;
    const struct dentry_operations *d_op;
    struct super_block *d_sb;
    long unsigned int d_time;
    void *d_fsdata;
    struct list_head d_lru;
    union {
        struct list_head d_child;
        struct callback_head d_rcu;
    } d_u;
    struct list_head d_subdirs;
    struct hlist_node d_alias;
}
SIZE: 192

2、查看成员变量在结构体中的偏移量，struct加-o选项：

crash> struct dentry -o
struct dentry {
    [0] unsigned int d_flags;
    [4] seqcount_t d_seq;
    [8] struct hlist_bl_node d_hash;
   [24] struct dentry *d_parent;
   [32] struct qstr d_name;
   [48] struct inode *d_inode;
   [56] unsigned char d_iname[32];
   [88] unsigned int d_count;
   [92] spinlock_t d_lock;
   [96] const struct dentry_operations *d_op;
  [104] struct super_block *d_sb;
  [112] long unsigned int d_time;
  [120] void *d_fsdata;
  [128] struct list_head d_lru;
        union {
            struct list_head d_child;
            struct callback_head d_rcu;
  [144] } d_u;
  [160] struct list_head d_subdirs;
  [176] struct hlist_node d_alias;
}
SIZE: 192

crash> struct dentry.d_inode
struct dentry {
   [48] struct inode *d_inode;
}

3、查看内存中指定结构体的值，struct后跟指针地址

crash> struct dentry ffff881fbdc02c80
struct dentry {
  d_flags = 136, 
  d_seq = {
    sequence = 4
  }, 
  d_hash = {
    next = 0x0, 
    pprev = 0xffffc900031bd8a0
  }, 
  d_parent = 0xffff881fbdc02ec0, 
  d_name = {
    {
      {
        hash = 1819047278, 
        len = 4
      }, 
      hash_len = 18998916462
    }, 
    name = 0xffff881fbdc02cb8 "null"
  }, 
  d_inode = 0xffff881f9011f5d0, 
  d_iname = "null\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000", 
  d_count = 2974, 
  d_lock = {
    {
      rlock = {
        raw_lock = {
          {
            head_tail = 2290387076, 
            tickets = {
              head = 34948, 
              tail = 34948
            }
          }
        }
      }
    }
  }, 
  d_op = 0xffffffff815cce40, 
  d_sb = 0xffff881f9011e800, 
  d_time = 0, 
  d_fsdata = 0x0, 
  d_lru = {
    next = 0xffff881fbdc02d00, 
    prev = 0xffff881fbdc02d00
  }, 
  d_u = {
    d_child = {
      next = 0xffff881fbdc02dd0, 
      prev = 0xffff881fbdfe1f50
    }, 
    d_rcu = {
      next = 0xffff881fbdc02dd0, 
      func = 0xffff881fbdfe1f50
    }
  }, 
  d_subdirs = {
    next = 0xffff881fbdc02d20, 
    prev = 0xffff881fbdc02d20
  }, 
  d_alias = {
    next = 0x0, 
    pprev = 0xffff881f9011f6e8
  }
}

注：如果要查看二阶指针的值，可以通过rd命令需要先获取一级指针的值，然后再用struct 结构体名 + addr获取具体的值，比如：

下面将通过struct file**fd的地址，获取fd指针数组中各个file结构体成员地址（二阶指针存放的是file指针的地址，所以可以通过rd获取一级指针的地址；二阶指针+8，则指向下一个file结构体指针）：

对比可以看出，这种方式获取的file指针跟files命令读取到的文件指针的值相同。

13、p命令

p命令可以用来打印出表达式或者变量的值

crash> p jiffies
jiffies = $7 = 4314305182
crash>
crash> p old_close_func
old_close_func = $5 = (close_t) 0xffffffff811ffd60 <SyS_close>
crash> p SyS_close
SyS_close = $6 = 
 {long (long)} 0xffffffff811ffd60 <SyS_close>
crash> sym SyS_close
ffffffff811ffd60 (T) SyS_close /usr/src/debug/kernel-3.10.0-693.el7/linux-3.10.0-693.el7.x86_64/fs/open.c: 1113
crash> 

14、查看某个命令使用方法及使用示例

使用help+某个命令，可以查看这个命令的使用方法及使用示例：
比如我们想看struct命令如何使用，可以执行 help struct命令：

crash> help struct

NAME
  struct - structure contents

SYNOPSIS
  struct struct_name[.member[,member]][-o][-l offset][-rfuxdp][address | symbol]
         [count | -c count]

DESCRIPTION
  This command displays either a structure definition, or a formatted display
  of the contents of a structure at a specified address.  When no address is
  specified, the structure definition is shown along with the structure size.
  A structure member may be appended to the structure name in order to limit
  the scope of the data displayed to that particular member; when no address
  is specified, the member's offset and definition are shown.

    struct_name  name of a C-code structure used by the kernel.
        .member  name of a structure member; to display multiple members of a
                 structure, use a comma-separated list of members.
             -o  show member offsets when displaying structure definitions; 
                 if used with an address or symbol argument, each member will
                 be preceded by its virtual address.
      -l offset  if the address argument is a pointer to a structure member that
                 is contained by the target data structure, typically a pointer
                 to an embedded list_head, the offset to the embedded member may
                 be entered in either of the following manners:
                   1. in "structure.member" format.
                   2. a number of bytes. 
             -r  raw dump of structure data.
             -f  address argument is a dumpfile offset.
             -u  address argument is a user virtual address in the current
                 context.
             -x  override default output format with hexadecimal format.
             -d  override default output format with decimal format.
             -p  if a structure member is a pointer value, show the member's
                 data type on the output line; and on the subsequent line(s),
                 dereference the pointer, display the pointer target's symbol
                 value in brackets if appropriate, and if possible, display the
                 target data; requires an address argument.
        address  hexadecimal address of a structure; if the address points
                 to an embedded list_head structure contained within the
                 target data structure, then the "-l" option must be used.
         symbol  symbolic reference to the address of a structure.
          count  count of structures to dump from an array of structures;
                 if used, this must be the last argument entered.
       -c count  "-c" is only required if "count" is not the last argument
                 entered or if a negative number is entered; if a negative
                 value is entered, the (positive) "count" structures that
                 lead up to and include the target structure will be displayed.

  Structure data, sizes, and member offsets are shown in the current output
  radix unless the -x or -d option is specified.

再比如，想查看ps命令在crash工具中怎么使用，可以help ps查看：

crash> help ps

NAME
  ps - display process status information

SYNOPSIS
  ps [-k|-u|-G][-s][-p|-c|-t|-l|-a|-g|-r] [pid | taskp | command] ...

DESCRIPTION
  This command displays process status for selected, or all, processes
  in the system.  If no arguments are entered, the process data is
  is displayed for all processes.  Specific processes may be selected
  by using the following identifier formats:

       pid  a process PID.
     taskp  a hexadecimal task_struct pointer.
   command  a command name.  If a command name is made up of letters that
            are all numerical values, precede the name string with a "\".
 
  The process list may be further restricted by the following options:

        -k  restrict the output to only kernel threads.
        -u  restrict the output to only user tasks.
        -G  display only the thread group leader in a thread group.
 
  The process identifier types may be mixed.  For each task, the following
  items are displayed:
 
    1. the process PID.
    2. the parent process PID.
    3. the CPU number that the task ran on last.
    4. the task_struct address or the kernel stack pointer of the process.
       (see -s option below)
    5. the task state (RU, IN, UN, ZO, ST, TR, DE, SW).
    6. the percentage of physical memory being used by this task.
    7. the virtual address size of this task in kilobytes.
    8. the resident set size of this task in kilobytes.
    9. the command name.
 
  The default output shows the task_struct address of each process under a
  column titled "TASK".  This can be changed to show the kernel stack 
  pointer under a column titled "KSTACKP".
 
       -s  replace the TASK column with the KSTACKP column.
 
  On SMP machines, the active task on each CPU will be highlighted by an
  angle bracket (">") preceding its information.
 
  Alternatively, information regarding parent-child relationships,
  per-task time usage data, argument/environment data, thread groups,
  or resource limits may be displayed:
 
       -p  display the parental hierarchy of selected, or all, tasks.

15、其它命令

可以通过help查看到如下：

如：可以使用rd memory读取内存内容；
struct命令显示结构体定义及指定地址的结构体内容；
irq查看中断信息；
vtop查看地址页表信息等等。

三、写在最后

如果我们确定是某个内核模块导致的问题，可以反汇编出该模块的源代码：

objdump -S -D my_test_lkm.ko > lkm.S

然后vim查看lkm.S文件，查看对应的源码，我们上面调试的bt信息中**[exception RIP: my_openat+36]**，也即是my_openat中偏移量为0x24的地方，如下图:

如果崩溃处对应有c代码的话，这样排查起来就简单多了。

最后，可以关注我的微信公众号 大胖聊编程 ，一起交流学习，共同进步。

参考资料：
[1]: Red_Hat_Enterprise_Linux-7-Kernel_Administration_Guide-en-US 7.5. ANALYZING A CORE DUMP