使用Docker容器时需要更改GC并发参数配置

最近在做统一服务优化时发现使用容器时垃圾回收偏长，而我们的服务是一个响应时间优先的应用，需要对GC进行一些调优，在对容器内Java应用进行GC时发现：

 

（8核）如下是8核机器的GC，CMS，young gc基本在100ms左右

[GC (Allocation Failure) 1540815K->708720K(1992320K), 0.1204106 secs]

[GC (Allocation Failure) 1547632K->718037K(1992320K), 0.1039874 secs]

[GC (Allocation Failure) 1568245K->734990K(1992320K), 0.0979778 secs]

[GC (Allocation Failure) 1573902K->739338K(1992320K), 0.1270793 secs]

[GC (Allocation Failure) 1578250K->743388K(1992320K), 0.1074495 secs]

[GC (Allocation Failure) 1582300K->749564K(1992320K), 0.1199881 secs]

[GC (Allocation Failure) 1588476K->752993K(1992320K), 0.1134528 secs]

[GC (Allocation Failure) 1591905K->761514K(1992320K), 0.1208703 secs]

[GC (Allocation Failure) 1600426K->762095K(1992320K), 0.1214570 secs]

[GC (Allocation Failure) 1601007K->764275K(1992320K), 0.1035069 secs]

[GC (CMS Initial Mark) 775325K(1992320K), 0.0122687 secs]

[GC (CMS Final Remark) 1174093K(1992320K), 0.2219463 secs]

[GC (Allocation Failure) 1075577K->241414K(1992320K), 0.0959227 secs]

[GC (Allocation Failure) 1080326K->246942K(1992320K), 0.1073993 secs]

 

（8核）经发现与GC线程有关，通过指定并发GC线程数 -XX:ParallelGCThreads=8 -XX:ConcGCThreads=4，得到

[GC (Allocation Failure) 1631622K->798323K(1992320K), 0.0408915 secs]

[GC (Allocation Failure) 1637235K->797560K(1992320K), 0.0423643 secs]

[GC (Allocation Failure) 1636472K->803864K(1992320K), 0.0422181 secs]

[GC (Allocation Failure) 1642776K->813965K(1992320K), 0.0429882 secs]

[GC (Allocation Failure) 1686836K->850230K(1992320K), 0.0521587 secs]

[GC (Allocation Failure) 1689142K->858658K(1992320K), 0.0496138 secs]

[GC (Allocation Failure) 1697570K->858148K(1992320K), 0.0439780 secs]

[GC (Allocation Failure) 1697059K->865706K(1992320K), 0.0517013 secs]

[GC (CMS Initial Mark) 865828K(1992320K), 0.0184015 secs]

[GC (CMS Final Remark) 1198017K(1992320K), 0.1831652 secs]

[GC (Allocation Failure) 1132594K->301826K(1992320K), 0.0444428 secs]

[GC (Allocation Failure) 1140738K->305794K(1992320K), 0.0428419 secs]

[GC (Allocation Failure) 1144679K->312688K(1992320K), 0.0545760 secs]

[GC (Allocation Failure) 1151600K->320109K(1992320K), 0.0410423 secs]

[GC (Allocation Failure) 1159021K->320191K(1992320K), 0.0411587 secs]

[GC (Allocation Failure) 1159103K->330729K(1992320K), 0.0387938 secs]

[GC (Allocation Failure) 1169641K->331256K(1992320K), 0.0393934 secs]

[GC (Allocation Failure) 1170163K->343068K(1992320K), 0.0419092 secs]

 

（16核）而下边是-XX:ParallelGCThreads=16 -XX:ConcGCThreads=8

[GC (Allocation Failure) 1551548K->713703K(1992320K), 0.0400264 secs]

[GC (Allocation Failure) 1552615K->716536K(1992320K), 0.0463409 secs]

[GC (Allocation Failure) 1555448K->723921K(1992320K), 0.0313725 secs]

[GC (Allocation Failure) 1562833K->728720K(1992320K), 0.0333375 secs]

[GC (Allocation Failure) 1567629K->734094K(1992320K), 0.0352418 secs]

[GC (Allocation Failure) 1573006K->739880K(1992320K), 0.0368524 secs]

[GC (Allocation Failure) 1578792K->741607K(1992320K), 0.0418331 secs]

[GC (Allocation Failure) 1580519K->749161K(1992320K), 0.0379511 secs]

[GC (Allocation Failure) 1588073K->755358K(1992320K), 0.0347147 secs]

[GC (Allocation Failure) 1594270K->761399K(1992320K), 0.0379215 secs]

[GC (Allocation Failure) 1600311K->765948K(1992320K), 0.0348537 secs]

[GC (Allocation Failure) 1604858K->774675K(1992320K), 0.0371332 secs]

[GC (CMS Initial Mark) 774724K(1992320K), 0.0120999 secs]

[GC (CMS Final Remark) 1067302K(1992320K), 0.1657415 secs]

[GC (Allocation Failure) 1130442K->286546K(1992320K), 0.0299338 secs]

[GC (Allocation Failure) 1125458K->298132K(1992320K), 0.0328611 secs]

[GC (Allocation Failure) 1137044K->304370K(1992320K), 0.0357785 secs]

 

（8核）而如果换成UseParallelGC，young gc不错，而full gc偏慢，线上如果没有配置CMS的话，默认是用这个

[GC (Allocation Failure) 1993894K->1000058K(2072576K), 0.0396914 secs]

[GC (Allocation Failure) 1999482K->1007021K(2072576K), 0.0403703 secs]

[GC (Allocation Failure) 2006445K->1011597K(2068480K), 0.0432666 secs]

[GC (Allocation Failure) 2006925K->1016275K(2070528K), 0.0450037 secs]

[GC (Allocation Failure) 2011603K->1020929K(2070528K), 0.0490808 secs]

[GC (Allocation Failure) 2016257K->1025389K(2070528K), 0.0395652 secs]

[GC (Allocation Failure) 2020717K->1029915K(2070528K), 0.0415800 secs]

[GC (Allocation Failure) 2025243K->1035623K(2070528K), 0.0534249 secs]

[GC (Allocation Failure) 2030951K->1041520K(2072576K), 0.0473277 secs]

[GC (Allocation Failure) 2040944K->1046783K(2072576K), 0.0447377 secs]

[GC (Allocation Failure) 2046207K->1052829K(2068480K), 0.0466185 secs]

[GC (Allocation Failure) 2048157K->1057749K(2070528K), 0.0435571 secs]

[GC (Allocation Failure) 2053077K->1063253K(2070528K), 0.0434343 secs]

[GC (Allocation Failure) 2058581K->1067759K(2070528K), 0.0402552 secs]

[Full GC (Ergonomics) 1067759K->235608K(2070528K), 0.4808497 secs]

[GC (Allocation Failure) 1230936K->247560K(2070528K), 0.0318850 secs]

[GC (Allocation Failure) 1242888K->254827K(2070528K), 0.0340009 secs]

[GC (Allocation Failure) 1250155K->259247K(2066432K), 0.0337921 secs]

[GC (Allocation Failure) 1250479K->264965K(2068480K), 0.0585058 secs]

 

虽然GC不如物理机，但是可以满足我们的需要了。

注：

 

1、因为容器不是物理隔离的，比如使用Runtime.getRuntime().availableProcessors() ，会拿到物理CPU个数，而不是容器申请时的个数，

2、CMS在算GC线程时默认是根据物理CPU算的：（http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2013-November/008952.html ）

年轻代的并行线程数（ParallelGCThreads）：

return (ncpus <= 8) ? ncpus : MAX2(8, ncpus / 2)

CMS并行线程数（ConcGCThreads）：

 if (AdjustGCThreadsToCores) {
   FLAG_SET_DEFAULT(ConcGCThreads, ParallelGCThreads / 2);
 } else {
   FLAG_SET_DEFAULT(ConcGCThreads, (3 + ParallelGCThreads) / 4);
 }

因为我们升级到了JDK8/Tomcat8， 遇到以下问题：

1、Tomcat8使用Nio2时有时候会遇到如下警告，暂时没有升级上去

org.apache.tomcat.util.net.AbstractEndpoint countDownConnection

WARNING: Incorrect connection count, multiple socket.close called on the same socket.

 

2、使用G1垃圾回收时，在我们的小堆上没有多大优势，暂时没有使用。

 

我们线上某服务的JVM参数

export JAVA_OPTS="-Djava.library.path=/usr/local/lib -server -XX:ReservedCodeCacheSize=64m -XX:TLABWasteTargetPercent=10 -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -XX:+CMSClassUnloadingEnabled -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UnlockDiagnosticVMOptions -XX:ParallelGCThreads=8 -XX:ConcGCThreads=4 -Xss256k -server -Xms2g -Xmx2g -XX:MaxDirectMemorySize=256m -XX:MaxTenuringThreshold=3 -XX:NewRatio=1 -XX:SurvivorRatio=8 -XX:+UnlockDiagnosticVMOptions -XX:ParGCCardsPerStrideChunk=32768 -XX:+AlwaysPreTouch

 

评论

1 楼 waykingeye 2015-11-27  

服务器调优，该怎么入手呢？
可不可以给些学习路线，比如了解jvm参数，了解系统实时监控工具（有哪些）。
求指点。