java进程占用cpu高排查－数据库连接过多

Kingson_Wu

浏览: 119493 次

最近访客更多访客>>

ningzong

liuqu11

maxingfang

yubachang2012

博主相关

博客

微博

相册

留言

关于我

文章分类

全部博客 (548)

社区版块

存档分类

原文：http://blog.163.com/xiangfei209@126/blog/static/98695674201622095414808/

物理环境：

线上服务器配置是16g内存，单个物理cpu，四核四线程

系统环境：

线上有一千多个机器24小时不间断调用服务器，该服务器没有做任何缓存，直接操作数据库，

jvm参数：

/usr/share/tomcat7/bin/catalina.sh

JAVA_OPTS="$JAVA_OPTS -server -Xms1024M -Xmx2048M -XX:PermSize=256M - XX:MaxPermSize=512M"

<div>问题场景：</div> <div> 调用服务器接口很慢，后台也打不开</div> <div>问题分析：</div> <div> 业务没有进行密集型IO计算，排查此情况</div> <div> 操作数据库频繁，该情况有可能</div> <div> </div> <div>排查步骤</div> <div>第一种排查问题方式：</div> <div>1、通过top命令，观察发现tomcat pid 8509 cpu基本超过125%</div> <div>2、通过lsof 命令查看nginx和mysql连接数</div> <div>lsof -i:80 | awk '{print $2}' | uniq -c 查看nginx数量，默认开启了1024个连接，发现不是nginx问题</div> <div>lsof -i:3306 | awk '{print $2}' | uniq -c 查看数据库连接默认开启了100个，发现数据库连接已经到达95个，可能是数据库太频繁导致</div> <div> </div> <div>3、接下来必须抓取heap dump 和 thread dump 来确定问题所在</div> <div> 命令行方式： </div> <div> jmap -dump:file=/var/log/jvm-log/heapdump.hprof,format=b8509生成heap dump</div> <div> jstack -l8509> /var/log/jvm-log/threaddump.log 生成thread dump</div> <div> visualVm方式： </div> <div> 配置jmx：</div> <div> vim /usr/share/tomcat7/bin/catalina.sh</div> <div> 在其中“# ----- Execute The Requested Command -------------”之前插入一行（中间没有换行），</div> <div> 内容如下(其中9999为监控端口，若要更换端口请自己填写)：CATALINA_OPTS="$CATALINA_OPTS</div> <div> -Dcom.sun.management.jmxremote</div> <div> -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.ssl=false - Dcom.sun.management.jmxremote.authenticate=true"</div> <div> 配置完jmx，然后通过visualVM连接过去，且安装memory pools插件，该插件主要来分析，heap（2新生代＋老年代），perm（持久代）内存信息</div> <div> 看thread heap发现操作数据库一个方法出现Locked ownable synchronizers</div> <div> </div> <div> </div> <div>第二种排查问题方式：</div> <div>1、通过top命令，观察发现tomcat pid 8509 cpu已经到达125% 2、找到该进程后，如何定位具体线程或代码呢，首先显示线程列表,并按照CPU占用高的线程排序： ＃ps -mp 8509 -o THREAD,tid,time | sort -rn</div> <div>结果如下： USER %CPU PRI SCNT WCHAN USER SYSTEM TID TIME <div>tomcat7 10.3 19 - futex_ - - 8522 00:10:18</div> <div>tomcat7 0.3 19 - futex_ - - 8521 00:00:17</div> <div>tomcat7 0.2 19 - futex_ - - 8622 00:00:10</div> 找到了耗时最高的线程8522，占用CPU时间有10分钟了 将需要的线程ID转换为16进制格式： # printf "%x\n" 8522 214a 最后打印线程的堆栈信息： # jstack8509|grep 214a -A 30</div> <div> </div> <div>第三种排查问题方式：</div> <div>vimshow-busy-java-threads.sh</div> <div> <pre class="prettyprint" style="white-space:pre-wrap; padding:2px; border:1px solid rgb(136,136,136)"><pre style="line-height:normal; white-space:pre-wrap; word-wrap:break-word">#!/bin/bash # @Function # Find out the highest cpu consumed threads of java, and print the stack of these threads. # # @Usage # $ ./show-busy-java-threads.sh # # @author Jerry Lee readonly PROG=`basename $0` readonly -a COMMAND_LINE=("$0" "$@") usage() { cat <<EOF Usage: ${PROG} [OPTION]... Find out the highest cpu consumed threads of java, and print the stack of these threads. Example: ${PROG} -c 10 Options: -p, --pid find out the highest cpu consumed threads from the specifed java process, default from all java process. -c, --count set the thread count to show, default is 5 -h, --help display this help and exit EOF exit $1 } readonly ARGS=`getopt -n "$PROG" -a -o c:p:h -l count:,pid:,help -- "$@"` [ $? -ne 0 ] && usage 1 eval set -- "${ARGS}" while true; do case "$1" in -c|--count) count="$2" shift 2 ;; -p|--pid) pid="$2" shift 2 ;; -h|--help) usage ;; --) shift break ;; esac done count=${count:-5} redEcho() { [ -c /dev/stdout ] && { # if stdout is console, turn on color output. echo -ne "\033[1;31m" echo -n "$@" echo -e "\033[0m" } || echo "$@" } yellowEcho() { [ -c /dev/stdout ] && { # if stdout is console, turn on color output. echo -ne "\033[1;33m" echo -n "$@" echo -e "\033[0m" } || echo "$@" } blueEcho() { [ -c /dev/stdout ] && { # if stdout is console, turn on color output. echo -ne "\033[1;36m" echo -n "$@" echo -e "\033[0m" } || echo "$@" } # Check the existence of jstack command! if ! which jstack &> /dev/null; then [ -z "$JAVA_HOME" ] && { redEcho "Error: jstack not found on PATH!" exit 1 } ! [ -f "$JAVA_HOME/bin/jstack" ] && { redEcho "Error: jstack not found on PATH and $JAVA_HOME/bin/jstack file does NOT exists!" exit 1 } ! [ -x "$JAVA_HOME/bin/jstack" ] && { redEcho "Error: jstack not found on PATH and $JAVA_HOME/bin/jstack is NOT executalbe!" exit 1 } export PATH="$JAVA_HOME/bin:$PATH" fi readonly uuid=`date +%s`_${RANDOM}_$$ cleanupWhenExit() { rm /tmp/${uuid}_* &> /dev/null } trap "cleanupWhenExit" EXIT printStackOfThreads() { local line local count=1 while IFS=" " read -a line ; do local pid=${line[0]} local threadId=${line[1]} local threadId0x="0x`printf %x ${threadId}`" local user=${line[2]} local pcpu=${line[4]} local jstackFile=/tmp/${uuid}_${pid} [ ! -f "${jstackFile}" ] && { { if [ "${user}" == "${USER}" ]; then jstack ${pid} > ${jstackFile} else if [ $UID == 0 ]; then sudo -u ${user} jstack ${pid} > ${jstackFile} else redEcho "[$((count++))] Fail to jstack Busy(${pcpu}%) thread(${threadId}/${threadId0x}) stack of java process(${pid}) under user(${user})." redEcho "User of java process($user) is not current user($USER), need sudo to run again:" yellowEcho " sudo ${COMMAND_LINE[@]}" echo continue fi fi } || { redEcho "[$((count++))] Fail to jstack Busy(${pcpu}%) thread(${threadId}/${threadId0x}) stack of java process(${pid}) under user(${user})." echo rm ${jstackFile} continue } } blueEcho "[$((count++))] Busy(${pcpu}%) thread(${threadId}/${threadId0x}) stack of java process(${pid}) under user(${user}):" sed "/nid=${threadId0x} /,/^$/p" -n ${jstackFile} done } ps -Leo pid,lwp,user,comm,pcpu --no-headers | { [ -z "${pid}" ] && awk '$4=="java"{print $0}' || awk -v "pid=${pid}" '$1==pid,$4=="java"{print $0}' } | sort -k5 -r -n | head --lines "${count}" | printStackOfThreads</pre></pre> chmod +show-busy-java-threads.sh</div> <div>./show-busy-java-threads.sh</div> <div>参考：https://github.com/oldratlee/useful-scripts/blob/master/docs/java.md#beer-show-busy-java-threadssh</div> <div> </div> <div>修复问题：</div> <div>请求频繁的sql语句放到redis，观察一段时间cpu静下来</div>

分享到：