重点是查看进程的线程中,哪个线程占用cpu过高,然后用gdb附加到进程,调试线程,看是否有死循环或者死锁等问题,步骤如下:
先用ps + grep找出该死的进程pid,比如 1706
然后 top -H -p 1706,(top然后shift+H可以看出某个线程,左上角有提示:thread on 则为可查看线程)所有该进程的线程都列出来, 看看哪个线程pid占用最多,记下对应的线程号,如:1723
1.gdb attach 到进程号码(1706)
2.(仍然在gdb中) info threads 结果大致如下:
(gdb) info threads 8 Thread 0x7f9fa9366700 (LWP 1716) 0x0000003cec00b98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 7 Thread 0x7f9fa8965700 (LWP 1720) 0x0000003cec00b98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 6 Thread 0x7f9fa7f64700 (LWP 1721) 0x0000003cec00f4b5 in sigwait () from /lib64/libpthread.so.0 5 Thread 0x7f9fa7563700 (LWP 1722) 0x0000003cec00b98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 4 Thread 0x7f9fa6b62700 (LWP 1723) 0x0000003cec00b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 3 Thread 0x7f9fa6161700 (LWP 1724) 0x0000003cebce9163 in epoll_wait () from /lib64/libc.so.6 2 Thread 0x7f9fa1159700 (LWP 1887) 0x0000003cebce9163 in epoll_wait () from /lib64/libc.so.6 * 1 Thread 0x7f9fa95ad820 (LWP 1706) 0x0000003cec00b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
找到线程号码对应的thread(LWP1723)即是我们刚刚记下的线程号
3.(仍然在gdb中)thread 线程号码切换到线程(4)–这里在info threads显示出来的序号需要使用gdb能识别的线程序号,即执行:thread 4切换到我们刚刚记下的线程号:1723的对应线程,如下:
(gdb) thread 4 [Switching to thread 4 (Thread 0x7f9fa6b62700 (LWP 1723))]#0 0x0000003cec00b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
4.(仍然在gdb中)bt 查看线程调用堆栈
(gdb) bt
#0 0x0000003cec00b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f9fa9f7144d in IceUtil::Cond::waitImpl (this=0x263f4c8, mutex=...) at ../../include/IceUtil/Cond.h:215 #2 0x00007f9fa9f9a4b1 in IceUtil::Monitor::wait (this=0x263f4c8) at ../../include/IceUtil/Monitor.h:152 #3 0x00007f9fa9fd7567 in IceInternal::EndpointHostResolver::run (this=0x263f480) at EndpointI.cpp:161 #4 0x00007f9fa9b1b975 in startHook (arg=0x263f480) at Thread.cpp:413 #5 0x0000003cec0079d1 in start_thread () from /lib64/libpthread.so.0 #6 0x0000003cebce8b6d in clone () from /lib64/libc.so.6
5.从上面输出的信息,基本上可以查看线程对应的代码断,是否有死循环等,如果是死锁的话,需要多次查看当前线程堆栈,或者查看全部线程的堆栈,总是会有某些个线程跟其他线程不一致,然后再对应到代码来进行定位解决