Linux cpu占用率居高不下 偵錯

今天偵錯程式，使用top命令後，發現程式的cpu占用率很高，一直在99，這很可怕，所以來偵錯。

使用top命令，得如下結果

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1997 root 20 0 358m 71m 3208 S 99.1 7.2 81:53.50 test
1 root 20 0 24332 2044 1176 S 0.0 0.2 0:01.36 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:46.51 ksoftirqd/0
4 root 20 0 0 0 0 S 0.0 0.0 0:38.53 kworker/0:0
6 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0

可知test的PID為1997

然後具體檢視test裡執行緒的cpu使用情況

使用 top -H -p 1997 命令

root@linuxidc:~# top -H -p 1997

top - 17:19:47 up 15 days, 34 min, 4 users, load average: 1.02, 1.06, 1.06
Tasks: 8 total, 1 running, 7 sleeping, 0 stopped, 0 zombie
Cpu(s): 99.3%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.3%si, 0.3%st
Mem: 1017924k total, 887500k used, 130424k free, 85928k buffers
Swap: 0k total, 0k used, 0k free, 351280k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2005 root 20 0 358m 71m 3208 R 99.3 7.2 83:44.47 test
2001 root 20 0 358m 71m 3208 S 0.3 7.2 0:08.66 test 
2004 root 20 0 358m 71m 3208 S 0.3 7.2 0:19.61 test 
1997 root 20 0 358m 71m 3208 S 0.0 7.2 0:26.24 test 
1999 root 20 0 358m 71m 3208 S 0.0 7.2 0:00.06 test 
2000 root 20 0 358m 71m 3208 S 0.0 7.2 0:00.00 test
2002 root 20 0 358m 71m 3208 S 0.0 7.2 0:05.86 Server Listen(2
2003 root 20 0 358m 71m 3208 S 0.0 7.2 0:00.00 Server Accept(2

可得執行緒中CPU佔有率最高的執行緒的PID是2005

使用gdb icdn 2005 命令

root@linuxidc:~# gdb icdn 2005
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>...
icdn: No such file or directory.
Attaching to process 2005

warning: process 2005 is a cloned process
Reading symbols from /home/linuxidc/test/snmptrapd...done.
Reading symbols from /usr/lib/libnetsnmp.so.30...done.
Loaded symbols for /usr/lib/libnetsnmp.so.30
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Loaded symbols for /lib/x86_64-linux-gnu/libpthread.so.0
Reading symbols from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18
Reading symbols from /usr/local/lib/libstd.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libstd.so
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/librt.so.1
Reading symbols from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib/x86_64-linux-gnu/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libz.so.1
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6
Reading symbols from /usr/local/lib/libudt.so...(no debugging symbols found)...done.
Loaded symbols for /usr/local/lib/libudt.so
Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libgcc_s.so.1
Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2
execute_search () at test.c:310

從最後一行可知，執行緒2005執行的是execute_search這個函數，然後檢視該函數，發現有這樣一段

while(g_search_state == 1)
{
　　tmp = g_data_list->phead->next;

　　if (tmp)
　　{
　　　　pthread_mutex_lock(&g_mysql_mutex);
　　　　res.result = search_mysql(tmp->sn, &(res.machine));
　　　　pthread_mutex_unlock(&g_mysql_mutex);

　　　　snprintf(res.sn, sizeof(res.sn), "%s", tmp->sn);
　　}
}

由於tmp長期為NULL，所以這變成了一個while的死迴圈，而死迴圈極為佔用記憶體，於是再下面加上usleep(50000)，問題解決。

本文永久更新連結地址：http://www.linuxidc.com/Linux/2016-04/130528.htm