按照SystemTap Beginners Guide的Installation and Setup部分安装了SystemTap,但是在安装过程中还是出现问题,在这里记录下解决方法。
环境
- Linux发行版本:CentOS Linux release 7.4.1708 (Core)
- 内核版本:3.10.0-693.2.2.el7.x86_64
- uname -a: Linux hostname 3.10.0-693.2.2.el7.x86_64 #1 SMP Tue Sep 12 22:26:13 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
安装
安装SystemTap
先安装如下两个RPM包:
-
systemtap
-
systemtap-runtime
以root运行如下命令:
在运行SystemTap之间,还需要装必要的内核信息包。在现代系统上,可以运行如下stap-prep来安装这些包,如下:
# stap-prep Need to install the following packages: kernel-devel-3.10.0-693.2.2.el7.x86_64 kernel-debuginfo-3.10.0-693.2.2.el7.x86_64 Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile No package kernel-devel-3.10.0-693.2.2.el7.x86_64 available. No package kernel-debuginfo-3.10.0-693.2.2.el7.x86_64 available. Error: Nothing to do Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile Could not find debuginfo for main pkg: kernel-3.10.0-693.2.2.el7.x86_64 No debuginfo packages available to install package kernel-devel-3.10.0-693.2.2.el7.x86_64 is not installed package kernel-debuginfo-3.10.0-693.2.2.el7.x86_64 is not installed problem installing rpm(s) kernel-devel-3.10.0-693.2.2.el7.x86_64 kernel-debuginfo-3.10.0-693.2.2.el7.x86_64
运行stap-prep的时候,它探测出还要安装kernel-devel-3.10.0-693.2.2.el7.x86_64包和kernel-debuginfo-3.10.0-693.2.2.el7.x86_64包 (实际上还有kernel-debuginfo-common包),但是自动安装失败。我们可以按照如下方法手动安装。
手动安装必要的内核信息包
SystemTap需要安装内核内核符号文件来probe内核。必要的内核信息包含在如下三个包中:
-
kernel-debuginfo
-
kernel-debuginfo-common
-
kernel-devel
一定要安装与当前内核版本一致的包。当前环境的内核版本是3.10.0-693.2.2.el7.x86_64,所以需要安装的包为:
- kernel-debuginfo-3.10.0-693.2.2.el7.x86_64
- kernel-debuginfo-common-3.10.0-693.2.2.el7.x86_64
- kernel-devel-3.10.0-693.2.2.el7.x86_64
注意:其实在上一步运行stap-prep时,已经把需要的包的名称及其内核精准地打印在屏幕上了。
接下来安装这三个包,注意不要直接yum install kernel-debuginfo kernel-debuginfo-common kernel-devel, 即使能找到相应的包,也是安装的最新版本,不会自动匹配当前版本。所以我们下载RPM包,再用rpm命令依次安装。
对于CentOS来说,内核符号文件一版在http://debuginfo.centos.org上有各个版本非常完整的包,但是一般从境内下载都比较慢,特别是kernel-debuginfo,比较大下载可能非常慢。所以在debuginfo.centos.org上下了kernel-debuginfo-common包,另外两个包在Google上搜了一把,分别找了两个镜像。下了之后才发现这个地方有坑,这个坑在后面展开讲。
wget https://ftp.sjtu.edu.cn/scientific/7/archive/debuginfo/kernel-debuginfo-3.10.0-693.2.2.el7.x86_64.rpm wget http://debuginfo.centos.org/7/x86_64/kernel-debuginfo-common-x86_64-3.10.0-693.2.2.el7.x86_64.rpm wget ftp://mirror.switch.ch/pool/4/mirror/scientificlinux/7.0/x86_64/updates/security/kernel-devel-3.10.0-693.2.2.el7.x86_64.rpm
下载之后,直接用rpm命令安装就好:
# rpm -ivh kernel-debuginfo-common-x86_64-3.10.0-693.2.2.el7.x86_64.rpm # rpm -ivh kernel-debuginfo-3.10.0-693.2.2.el7.x86_64.rpm # rpm -ivh kernel-devel-3.10.0-693.2.2.el7.x86_64.rpm
至此安装步骤完毕,下面来测试SystemTap能不能正常运行。
运行SystemTap
为了测试stap是否能正常运行,用如下简单命令打印:
# stap -e 'probe begin{printf("Hello, World"); exit();}'
运行失败…结果如下:
# stap -e 'probe begin{printf("Hello, World"); exit();}' ERROR: module version mismatch (#1 SMP Tue Sep 12 10:10:26 CDT 2017 vs #1 SMP Tue Sep 12 22:26:13 UTC 2017), release 3.10.0-693.2.2.el7.x86_64 WARNING: /usr/bin/staprun exited with status: 1 Pass 5: run failed. [man error::pass5]
错误信息是:”ERROR: module version mismatch (#1 SMP Tue Sep 1210:10:26 CDT 2017 vs #1 SMP Tue Sep 1222:26:13 UTC 2017)“。
解决”ERROR: module version mismatch”问题
stap运行的时候加上-v参数,打印更多信息看看还有没有更多线索:
# stap -e 'probe begin{printf("Hello, World"); exit();}' -v Pass 1: parsed user script and 470 library scripts using 228224virt/41280res/3348shr/38020data kb, in 330usr/20sys/346real ms. Pass 2: analyzed script: 1 probe, 1 function, 0 embeds, 0 globals using 229148virt/42332res/3536shr/38944data kb, in 0usr/0sys/6real ms. Pass 3: using cached /root/.systemtap/cache/0b/stap_0bc9e27aef7a1de50ea41889a27fc524_1010.c Pass 4: using cached /root/.systemtap/cache/0b/stap_0bc9e27aef7a1de50ea41889a27fc524_1010.ko Pass 5: starting run. ERROR: module version mismatch (#1 SMP Tue Sep 12 10:10:26 CDT 2017 vs #1 SMP Tue Sep 12 22:26:13 UTC 2017), release 3.10.0-693.2.2.el7.x86_64 WARNING: /usr/bin/staprun exited with status: 1 Pass 5: run completed in 0usr/10sys/38real ms. Pass 5: run failed. [man error::pass5]
查看c文件,vi /root/.systemtap/cache/0b/stap_0bc9e27aef7a1de50ea41889a27fc524_1010.c,搜错误信息”module version mismatch“,能搜到报错发生在下面的第13行,至于UTS_RELEASE和UTS_VERSION是在哪里设置的,直接Google一把。
#ifndef STP_NO_VERREL_CHECK
const char* release = UTS_RELEASE;
#ifdef STAPCONF_GENERATED_COMPILE
const char* version = UTS_VERSION;
#endif
might_sleep();
if (strcmp (release, “3.10.0-693.2.2.el7.x86_64”)) {
_stp_error (“module release mismatch (%s vs %s)”, release, “3.10.0-693.2.2.el7.x86_64”);
rc = -EINVAL;
}
#ifdef STAPCONF_GENERATED_COMPILE
if (strcmp (utsname()->version, version)) {
_stp_error (“module version mismatch (%s vs %s), release %s”, version, utsname()->version, release);
rc = -EINVAL;
}
#endif
#endif
有两篇文章里面提到了同样的坑,文章连接在底部的参考中。在kernel-devel包的所以文件中搜以下变量UTS_VERSION,
# rpm -ql kernel-devel | xargs grep UTS_VERSION /usr/src/kernels/3.10.0-693.2.2.el7.x86_64/include/generated/compile.h:#define UTS_VERSION "#1 SMP Tue Sep 12 10:10:26 CDT 2017"
可以看到在compile.h中有#define UTS_VERSION “#1 SMP Tue Sep 12 10:10:26 CDT 2017”. 这个是不是很熟悉… 对比下上面运行stap的报错信息, module mismatch的时间就是这个。文件compile.h是自动生成的,可能和当时编译时的时间相关。但是stap要求这个也和当前系统uname -a里面的时间完全一直,如果下个CentOS原生的kernel-devel应该就没这个问题。
解决问题的另一个简单方法就是直接修改这个compile.h文件,原来的文件如下:
# cat /usr/src/kernels/3.10.0-693.2.2.el7.x86_64/include/generated/compile.h /* This file is auto generated, version 1 */ /* SMP */ #define UTS_MACHINE "x86_64" #define UTS_VERSION "#1 SMP Tue Sep 12 10:10:26 CDT 2017" #define LINUX_COMPILE_BY "mockbuild" #define LINUX_COMPILE_HOST "sl7-uefisign.fnal.gov" #define LINUX_COMPILER "gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) "
修改define UTS_VERSION那一行,如下:
#define UTS_VERSION “#1 SMP Tue Sep 12 10:10:26 CDT 2017” -> #define UTS_VERSION “#1 SMP Tue Sep 12 22:26:13 UTC 2017”
再次运行stap:
# stap -e 'probe begin{printf("Hello, World"); exit();}' -v Pass 1: parsed user script and 470 library scripts using 228220virt/41276res/3348shr/38016data kb, in 350usr/10sys/355real ms. Pass 2: analyzed script: 1 probe, 1 function, 0 embeds, 0 globals using 229144virt/42328res/3536shr/38940data kb, in 0usr/0sys/6real ms. Pass 3: using cached /root/.systemtap/cache/0b/stap_0bc9e27aef7a1de50ea41889a27fc524_1010.c Pass 4: using cached /root/.systemtap/cache/0b/stap_0bc9e27aef7a1de50ea41889a27fc524_1010.ko Pass 5: starting run. ERROR: module version mismatch (#1 SMP Tue Sep 12 10:10:26 CDT 2017 vs #1 SMP Tue Sep 12 22:26:13 UTC 2017), release 3.10.0-693.2.2.el7.x86_64 WARNING: /usr/bin/staprun exited with status: 1 Pass 5: run completed in 0usr/10sys/38real ms. Pass 5: run failed. [man error::pass5]
因为中间生成的C文件和ko模块都是用的cache (蓝色标注的部分),我们把上面的cache文件删除,再重新运行,这次可以成功了。
# stap -e 'probe begin{printf("Hello, World"); exit();}' Hello, World
参考
https://sourceware.org/systemtap/SystemTap_Beginners_Guide/using-systemtap.html#using-setup
ERROR: module version mismatch
https://groups.google.com/forum/#!topic/openresty/nlEc3qlDyOc