性能测试之:操作系统性能指标观测及分析

<p class="MsoNormal" style=""><span style="" lang="EN-US"><span style=""><span style="font-family: Times New Roman;"><span style="font-size: small;">1.</span><span style='font: 7pt "Times New Roman";'> </span></span></span></span><span style="font-size: small;"><span style="">对于</span><span lang="EN-US"><span style="font-family: Times New Roman;">Linux</span></span><span style="">系统来说，</span><span lang="EN-US"><span style="font-family: Times New Roman;">Free Mem</span></span><span style="">很少并不一定代表内存不足，成为瓶颈。实际上这是由于</span><span lang="EN-US"><span style="font-family: Times New Roman;">LINUX</span></span><span style="">的内存管理机制造成的，</span><span lang="EN-US"><span style="font-family: Times New Roman;">LINUX</span></span><span style="">会把</span><span lang="EN-US"><span style="font-family: Times New Roman;">Free</span></span><span style="">的内存（完全没使用过的内存）尽可能的利用起来，作为</span><span lang="EN-US"><span style="font-family: Times New Roman;">buffers/cached</span></span><span style="">（当有进程申请内存时，如果没有</span><span lang="EN-US"><span style="font-family: Times New Roman;">Free</span></span><span style="">，会把</span><span lang="EN-US"><span style="font-family: Times New Roman;">buffers/cached</span></span><span style="">分配给进程），以提高系统的性能，尤其是在有大量</span><span lang="EN-US"><span style="font-family: Times New Roman;">I/O</span></span><span style="">的情况下，增大</span><span lang="EN-US"><span style="font-family: Times New Roman;">buffers/cached</span></span><span style="">会很大地提高</span><span lang="EN-US"><span style="font-family: Times New Roman;">I/O</span></span><span style="">，可以提高多达</span><span lang="EN-US"><span style="font-family: Times New Roman;">20%</span></span><span style="">的</span><span lang="EN-US"><span style="font-family: Times New Roman;">I/O</span></span><span style="">性能，因此实际是</span><span lang="EN-US"><span style="font-family: Times New Roman;">LINUX</span></span><span style="">的这一特性造成</span><span lang="EN-US"><span style="font-family: Times New Roman;">Free Mem</span></span><span style="">几乎没有。当服务器存在大量</span><span lang="EN-US"><span style="font-family: Times New Roman;">I/O</span></span><span style="">时，比较容易出现</span><span lang="EN-US"><span style="font-family: Times New Roman;">Free Mem</span></span><span style="">几乎没有这样的情况。</span><span lang="EN-US"><span style="font-family: Times New Roman;">used-buffers-cached</span></span><span style="">得到的结果才是应用程序用掉的内存，如果这个数字已经接近或超过总的内存数，那么内存成为瓶颈。如下图所示：</span></span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt; text-indent: 15.75pt; text-align: center;" align="center"><span lang="EN-US"></span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt 15.75pt;"><span style="font-size: small;"><span style="">第二行（</span><span lang="EN-US"><span style="font-family: Times New Roman;">-/+ buffers/cache</span></span><span style="">）的</span><span lang="EN-US"><span style="font-family: Times New Roman;">used</span></span><span style="">是扣除了</span><span lang="EN-US"><span style="font-family: Times New Roman;">buffers/cache</span></span><span style="">后的</span><span lang="EN-US"><span style="font-family: Times New Roman;">used</span></span><span style="">内存，这才是真正的被应用程序使用掉的内存（包括内核级进程和用户级进程）。也可以通过</span><span lang="EN-US"><span style="font-family: Times New Roman;">swap</span></span><span style="">来判断内存是否是瓶颈，如果</span><span lang="EN-US"><span style="font-family: Times New Roman;">swap</span></span><span style="">大量</span><span lang="EN-US"><span style="font-family: Times New Roman;">used</span></span><span style="">，并且频繁的发生</span><span lang="EN-US"><span style="font-family: Times New Roman;">swap in</span></span><span style="">和</span><span lang="EN-US"><span style="font-family: Times New Roman;">swap out</span></span><span style="">，那么内存一定是瓶颈；有时用</span><span lang="EN-US"><span style="font-family: Times New Roman;">free</span></span><span style="">会看到少量的</span><span lang="EN-US"><span style="font-family: Times New Roman;">swap used</span></span><span style="">，这并不影响性能，内存不是瓶颈。</span></span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt 15.75pt;"><span style="font-size: small;"><span lang="EN-US"><span style="font-family: Times New Roman;">PS</span></span><span style="">：</span><span lang="EN-US"><span style="font-family: Times New Roman;">buffers</span></span><span style="">和</span><span lang="EN-US"><span style="font-family: Times New Roman;">cached</span></span><span style="">的区别。两者都是缓存，都是为了提高</span><span lang="EN-US"><span style="font-family: Times New Roman;">I/O</span></span><span style="">的性能，不同的是</span><span lang="EN-US"><span style="font-family: Times New Roman;">buffers</span></span><span style="">是块设备的缓存，而</span><span lang="EN-US"><span style="font-family: Times New Roman;">cached</span></span><span style="">是字符设备的缓存，简单地说，</span><span lang="EN-US"><span style="font-family: Times New Roman;">buffers</span></span><span style="">用来存储目录下面有什么内容、什么权限等等，</span><span lang="EN-US"><span style="font-family: Times New Roman;">cached</span></span><span style="">则存储文件的内容，例如，</span><span lang="EN-US"><span style="font-family: Times New Roman;">ls</span></span><span style="">一下，可以发现</span><span lang="EN-US"><span style="font-family: Times New Roman;">buffers</span></span><span style="">有明显增长（</span><span lang="EN-US"><span style="font-family: Times New Roman;">buffers</span></span><span style="">其实一直在缓慢增长），而打开一个文件后，可以发现</span><span lang="EN-US"><span style="font-family: Times New Roman;">cached</span></span><span style="">明显增长。</span></span></p>
<p class="MsoNormal" style=""><span style="font-size: small;"><span lang="EN-US"><span style="font-family: Times New Roman;">2. </span></span><span style="">现象：还没有开始性能测试，服务器的</span><span lang="EN-US"><span style="font-family: Times New Roman;">Load</span></span><span style="">达到</span><span lang="EN-US"><span style="font-family: Times New Roman;">1</span></span><span style="">以上，</span><span lang="EN-US"><span style="font-family: Times New Roman;">CPU </span></span><span style="">使用</span><span lang="EN-US"><span style="font-family: Times New Roman;"> user</span></span><span style="">接近</span><span lang="EN-US"><span style="font-family: Times New Roman;">0</span></span><span style="">，</span><span lang="EN-US"><span style="font-family: Times New Roman;">sys</span></span><span style="">有些情况下也接近</span><span lang="EN-US"><span style="font-family: Times New Roman;">0</span></span><span style="">，有些情况下会达到</span><span lang="EN-US"><span style="font-family: Times New Roman;">10%</span></span><span style="">以上，用</span><span lang="EN-US"><span style="font-family: Times New Roman;">vmstat</span></span><span style="">查看，</span><span lang="EN-US"><span style="font-family: Times New Roman;">r</span></span><span style="">（运行队列中的等待进程数）</span><span lang="EN-US"><span style="font-family: Times New Roman;">=0</span></span><span style="">，</span><span lang="EN-US"><span style="font-family: Times New Roman;">b</span></span><span style="">（等待</span><span lang="EN-US"><span style="font-family: Times New Roman;">I/O</span></span><span style="">的进程数）</span><span lang="EN-US"><span style="font-family: Times New Roman;">&gt;0</span></span></span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt 18pt;"><span style="font-size: small;"><span style="">原因：产生这种现象的原因很多，比如，操作系统中存在状态为</span><span lang="EN-US"><span style="font-family: Times New Roman;">D</span></span><span style="">（</span><span style="" lang="EN-US"><span style="font-family: Times New Roman;">Uninterruptible sleep</span></span><span style="">）或状态为</span><span lang="EN-US"><span style="font-family: Times New Roman;">Z</span></span><span style="">（</span><span lang="EN-US"><span style="font-family: Times New Roman;">Zombie</span></span><span style="">）的进程，这两种进程会把系统</span><span lang="EN-US"><span style="font-family: Times New Roman;">Load</span></span><span style="">弄高（因为这两种状态的进程会滞留在</span><span lang="EN-US"><span style="font-family: Times New Roman;">CPU </span></span><span style="">运行队列中）；或者系统对同一个目录</span><span lang="EN-US"><span style="font-family: Times New Roman;">mount</span></span><span style="">了多次，会导致</span><span lang="EN-US"><span style="font-family: Times New Roman;">load</span></span><span style="">升高，</span><span lang="EN-US"><span style="font-family: Times New Roman;">sys CPU&gt;10%</span></span><span style="">；一般都是跟</span><span lang="EN-US"><span style="font-family: Times New Roman;">I/O</span></span><span style="">相关。</span></span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt 18pt;"><span style="font-size: small;"><span style="">判断及解决方法：</span><span lang="EN-US"><span style="font-family: Times New Roman;">top</span></span><span style="">或者</span><span lang="EN-US"><span style="font-family: Times New Roman;">ps afux</span></span><span style="">一下，看看进程的状态；</span><span lang="EN-US"><span style="font-family: Times New Roman;">df</span></span><span style="">一下看看是否有重复的</span><span lang="EN-US"><span style="font-family: Times New Roman;">mount</span></span><span style="">；</span><span lang="EN-US"><span style="font-family: Times New Roman;">vmstat</span></span><span style="">查看一下；可以根据不同的情况采用不同的办法来解决，由于是测试环境，最简单有效的办法就是</span><span lang="EN-US"><span style="font-family: Times New Roman;">reboot</span></span><span style="">。不过有时</span><span lang="EN-US"><span style="font-family: Times New Roman;">reboot</span></span><span style="">前也要做一些工作，比如已经</span><span lang="EN-US"><span style="font-family: Times New Roman;">mount</span></span><span style="">了</span><span lang="EN-US"><span style="font-family: Times New Roman;">NFS</span></span><span style="">服务端上的一个目录，并写入了</span><span lang="EN-US"><span style="font-family: Times New Roman;">fstab</span></span><span style="">中，而</span><span lang="EN-US"><span style="font-family: Times New Roman;">NFS</span></span><span style="">服务端关闭了，那么需要把</span><span lang="EN-US"><span style="font-family: Times New Roman;">fstab</span></span><span style="">中的相应内容删除后再</span><span lang="EN-US"><span style="font-family: Times New Roman;">reboot</span></span><span style="">，否则</span><span lang="EN-US"><span style="font-family: Times New Roman;">reboot</span></span><span style="">后仍然会自动</span><span lang="EN-US"><span style="font-family: Times New Roman;">mount</span></span><span style="">。</span></span></p>