【汇总压力测试】linux+nginx+php-fpm+mysql优化

====================== 压测前：

netstat -ntlp
ulimit -SHn 65535  （加入开机启动，nginx启动前）

检查优化：
nginx.conf
php-fpm.conf
my.cnf

 

优化/etc/sysctl.conf

增加：
 kernel.shmmax = 68719476736
 # Controls the maximum number of shared memory segments, in pages
 kernel.shmall = 4294967296
   
        #syncookie
        net.ipv4.tcp_max_syn_backlog = 65536
        net.core.netdev_max_backlog =  32768
        net.core.somaxconn = 32768

        net.core.wmem_default = 8388608
        net.core.rmem_default = 8388608
        net.core.rmem_max = 16777216
        net.core.wmem_max = 16777216

        net.ipv4.tcp_timestamps = 0
        net.ipv4.tcp_synack_retries = 2
        net.ipv4.tcp_syn_retries = 2
        #time_wait
        net.ipv4.tcp_syncookies = 1
        net.ipv4.tcp_tw_recycle = 1
        net.ipv4.tcp_fin_timeout = 5
        net.ipv4.tcp_tw_reuse = 1

        net.ipv4.tcp_mem = 94500000 915000000 927000000
        net.ipv4.tcp_max_orphans = 3276800
        net.ipv4.ip_local_port_range = 1024  65535

 

 

 

 

 

查看mysql连接

#netstat -anp | grep mysql | wc -l

netstat -n | awk '/^tcp/ {++state[$NF]} END {for(key in state) print key,"\t",state[key]}'

查看TIME_WAIT连接数
netstat -ae|grep "TIME_WAIT" |wc -l

netstat -n | awk '/^tcp/ {++state[$NF]} END {for(key in state) print key,"\t",state[key]}'

查找较多time_wait连接
netstat -n|grep TIME_WAIT|awk '{print $5}'|sort|uniq -c|sort -rn|head -n20

 

 

 

 

 

===================================实例测试

1 ） 系统说明

服务器说明：

两台 CentOS release 5.4 (Final) 32 位， 1 颗 CPU 8 核

 

a.       服务器 1

服务： Nginx + php (fpm)

nginx 开启的进程数 8

FastCGI 进程数 128

 

b.       服务器 2

服务 :Memcache+mysql

memcache 替代 session ，开启两个端口，分别分配 1024M;

 

 

2) 模拟场景 : 首页，登录，退出

并发量： 900

 

 

 

=============== 系统说明

 

#cat /etc/issue

CentOS release 5.4 (Final)

Kernel \r on an \m

 

 

# cat /proc/cpuinfo | grep physical | uniq -c

      8 physical id     : 1

 

# ulimit -a

core file size          (blocks, -c) 0  需要修改

data seg size            (kbytes, -d) unlimited

scheduling priority             (-e) 0

file size               (blocks, -f) unlimited

pending signals                 (-i) 143360

max locked memory       (kbytes, -l) 32

max memory size         (kbytes, -m) unlimited

open files                       (-n) 1024   需要修改 (因为是tcp协议  要打开套接字，要打开文件句柄，而单进程的最大打开文件句柄操作系统是有限制的，默认是1024)

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 10240

cpu time               (seconds, -t) unlimited

max user processes              (-u) 143360

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

 

 

 

#getconf LONG_BIT

32

 

 

两台服务器：

Nginx + php (fpm)   113.106.90.80

Memcache+mysql   113.106.90.78

 

 

 

 

nginx 开启的进程数 8

FastCGI 进程数 128

worker_connections 65535;
每个工作进程允许最大的同时连接数   （ Maxclient = work_processes * 　 worker_connections ）

 

 

――― memcache 替代 session

Mysql 服务器：

/usr/local/bin/memcached -d -m 10 -u root -l 192.168.12.203 -p 13001 -c 1024 -P /tmp/memcached.pid

/usr/local/bin/memcached -d -m 10 -u root -l 192.168.12.203 -p 13002 -c 1024 -P /tmp/memcached2.pid

 

 

session.save_handler = memcache
session.save_path = "tcp://192.168.12.203:13001,tcp://192.168.12.203:13002"

 

 

 

＝＝＝＝＝＝＝＝＝＝＝优化处

 

―――――― 1 linux 默认 core file size 为 0

何谓 core 文件，当一个程序崩溃时，在进程当前工作目录的 core 文件中复制了该进程的存储图像。 core 文件仅仅是一个内存映象（同时加上调试信息），主要是用来调试的。 core 文件是个二进制文件，需要用相应的工具来分析程序崩溃时的内存映像。

系统默认 core 文件的大小为 0 ，所以没有创建。可以用 ulimit 命令查看和修改 core 文件的大小。

 

$ulimit -c

0

$ ulimit -c 1000

$ ulimit -c

1000

-c 指定修改 core 文件的大小， 1000 指定了 core 文件大小。也可以对 core 文件的大小不做限制，如：

# ulimit -c unlimited

#ulimit -c

Unlimited

 

如果想让修改永久生效，则需要修改配置文件

 

Vim /etc/profile 中的：

ulimit -S -c 0 > /dev/null 2>&1

修改为

ulimit -S -c 1000 > /dev/null 2>&1

 

#source /etc/profile  重启生效

 

 

参考资料： http://hi.baidu.com/jrckkyy/blog/item/2562320a5bdbc534b1351d95.html

 

 

―――――― 2 linux 默认值 open files 和 max user processes 为 1024

#ulimit -n

1024

#ulimit –u

1024

问题描述： 说明 server 只允许同时打开 1024 个文件，处理 1024 个用户进程

使用ulimit -a 可以查看当前系统的所有限制值，使用ulimit -n 可以查看当前的最大打开文件数。

新装的linux 默认只有1024 ，当作负载较大的服务器时，很容易遇到error: too many open files 。因此，需要将其改大。

 

解决方法：

使用 ulimit –n 65535 可即时修改，但重启后就无效了。（注ulimit -SHn 65535 等效 ulimit -n 65535 ，-S 指soft ，-H 指hard)

有如下三种修改方式：

在/etc/rc.local 中增加一行 ulimit -SHn 65535
2. 在/etc/profile 中增加一行 ulimit -SHn 65535
3. 在/etc/security/limits.conf 最后增加：

* soft nofile 65535
* hard nofile 65535
* soft nproc 65535
* hard nproc 65535

具体使用哪种，在 CentOS 中使用第1 种方式无效果，使用第3 种方式有效果，而在Debian 中使用第2 种有效果

 

# ulimit -n

65535

# ulimit -u

65535

 

备注：ulimit 命令本身就有分软硬设置，加-H 就是硬，加-S 就是软默认显示的是软限制

soft 限制指的是当前系统生效的设置值。 hard 限制值可以被普通用户降低。但是不能增加。 soft 限制不能设置的比 hard 限制更高。 只有 root 用户才能够增加 hard 限制值。

 

 

―――――― 3 发现存在大量 TIME_WAIT 状态的连接

 

问题描述： 发现大量 time_wait 状态的连接 , 有时甚至达到 7000 以上

netstat -n | awk '/^tcp/ {++state[$NF]} END {for(key in state) print key,"\t",state[key]}'

 

查找较多 time_wait 连接

netstat -n|grep TIME_WAIT|awk '{print $5}'|sort|uniq -c|sort -rn|head -n20

发现是 DB 服务器的 mysql 和 memcache 的未释放导致。

 

建议：在 PHP 程序开发时，建议大家尽量 mysql_close ，减少 msyql 消耗。

 

 

解决方法： 对于大量使用 tcp 连接的应用来说，也需要对 /etc/sysctl.conf 中的参数进行相应优化 :

vim /etc/sysctl.conf

编辑文件，加入以下内容：

net.ipv4.tcp_syncookies = 1

net.ipv4.tcp_tw_reuse = 1

net.ipv4.tcp_tw_recycle = 1

net.ipv4.tcp_fin_timeout = 30

 

然后执行 /sbin/sysctl -p 让参数生效。

 

优化后：

发现大量的 TIME_WAIT 已不存在， mysql 进程的占用率很快就降下来的。

 

参考资料： http://space.itpub.net/9653957/viewspace-626059

 

―――――― 4 nginx.conf 优化

 

 

 

－－－－－－－5 php优化

 

open_basedir = /data/www/  #将用户可操作的文件限制在某目录
safe_mode = on #打开php的安全模式，开启重要文件的权限控制
safe_mode_exec_dir = /data/www/ #执行某些程序目录设置。不要执行任何程序，那么就可以指向我们网页目录
disable_functions = chdir,getcwd,scandir,chgrp,chmod,chown #禁用函数
expose_php = Off #关闭PHP版本信息在http头中的泄漏
display_errors = Off #禁止错误
关闭error_reporting
log_errors = On
post_max_size = 8M
upload_max_filesize = 10M

apc.enabled = 1
apc.cache_by_default = on
apc.shm_segments = 1
apc.shm_size = 32
apc.ttl = 600
apc.user_ttl = 600
apc.num_files_hint = 0
apc.write_lock = On

 

 

 

――――――php-fpm.conf


<?xml version="1.0" ?>
<configuration>

All relative paths in this config are relative to php's install prefix

<section name="global_options">

Pid file
<value name="pid_file">/usr/local/services/php/logs/php-fpm.pid</value>

Error log file
<value name="error_log">/data/logs/php/php-fpm.log</value>

Log level
<value name="log_level">notice</value>

When this amount of php processes exited with SIGSEGV or SIGBUS ...
<value name="emergency_restart_threshold">10</value>

... in a less than this interval of time, a graceful restart will be initiated.
Useful to work around accidental curruptions in accelerator's shared memory.
<value name="emergency_restart_interval">1m</value>

Time limit on waiting child's reaction on signals from master
<value name="process_control_timeout">5s</value>

Set to 'no' to debug fpm
<value name="daemonize">yes</value>

</section>

<workers>

<section name="pool">

Name of pool. Used in logs and stats.
<value name="name">default</value>

Address to accept fastcgi requests on.
Valid syntax is 'ip.ad.re.ss:port' or just 'port' or '/path/to/unix/socket'
<value name="listen_address">127.0.0.1:9000</value>

<value name="listen_options">

Set listen(2) backlog
<value name="backlog">-1</value>

Set permissions for unix socket, if one used.
In Linux read/write permissions must be set in order to allow connections from web server.
Many BSD-derrived systems allow connections regardless of permissions.
<value name="owner"></value>
<value name="group"></value>
<value name="mode">0666</value>
</value>

Additional php.ini defines, specific to this pool of workers.
<value name="php_defines">
<value name="sendmail_path">/usr/sbin/sendmail -t -i</value>
<value name="display_errors">1</value>
</value>

Unix user of processes
<value name="user">www</value>

Unix group of processes
<value name="group">www</value>

Process manager settings
<value name="pm">

Sets style of controling worker process count.
Valid values are 'static' and 'apache-like'
<value name="style">static</value>

Sets the limit on the number of simultaneous requests that will be served.
Equivalent to Apache MaxClients directive.
Equivalent to PHP_FCGI_CHILDREN environment in original php.fcgi
Used with any pm_style.
<value name="max_children">128</value>

Settings group for 'apache-like' pm style
<value name="apache_like">

Sets the number of server processes created on startup.
Used only when 'apache-like' pm_style is selected
<value name="StartServers">20</value>

Sets the desired minimum number of idle server processes.
Used only when 'apache-like' pm_style is selected
<value name="MinSpareServers">5</value>

Sets the desired maximum number of idle server processes.
Used only when 'apache-like' pm_style is selected
<value name="MaxSpareServers">35</value>

</value>

</value>

The timeout (in seconds) for serving a single request after which the worker process will be terminated
Should be used when 'max_execution_time' ini option does not stop script execution for some reason
'0s' means 'off'
<value name="request_terminate_timeout">0s</value>

The timeout (in seconds) for serving of single request after which a php backtrace will be dumped to slow.log file
'0s' means 'off'
<value name="request_slowlog_timeout">0s</value>

The log file for slow requests
<value name="slowlog">/data/logs/php/php-fpm-slow.log</value>

Set open file desc rlimit
<value name="rlimit_files">65535</value>

Set max core size rlimit
<value name="rlimit_core">0</value>

Chroot to this directory at the start, absolute path
<value name="chroot"></value>

Chdir to this directory at the start, absolute path
<value name="chdir"></value>

Redirect workers' stdout and stderr into main error log.
If not set, they will be redirected to /dev/null, according to FastCGI specs
<value name="catch_workers_output">yes</value>

How much requests each process should execute before respawn.
Useful to work around memory leaks in 3rd party libraries.
For endless request processing please specify 0
Equivalent to PHP_FCGI_MAX_REQUESTS
<value name="max_requests">102400</value>

Comma separated list of ipv4 addresses of FastCGI clients that allowed to connect.
Equivalent to FCGI_WEB_SERVER_ADDRS environment in original php.fcgi (5.2.2+)
Makes sense only with AF_INET listening socket.
<value name="allowed_clients">127.0.0.1</value>

Pass environment variables like LD_LIBRARY_PATH
All $VARIABLEs are taken from current environment
<value name="environment">
<value name="HOSTNAME">$HOSTNAME</value>
<value name="PATH">/usr/local/bin:/usr/bin:/bin</value>
<value name="TMP">/tmp</value>
<value name="TMPDIR">/tmp</value>
<value name="TEMP">/tmp</value>
<value name="OSTYPE">$OSTYPE</value>
<value name="MACHTYPE">$MACHTYPE</value>
<value name="MALLOC_CHECK_">2</value>
</value>

</section>

</workers>

</configuration>

－－－－－－－－－－－－－－－－6 mysql优化

 

 

[client]
#password       = your_password
default-character-set = utf8
port            = 33306
socket          = /data/mysql/mysql.sock

# Here follows entries for some specific programs

# The MySQL server
[mysqld]
port            = 33306
socket          = /data/mysql/mysql.sock
skip-external-locking
key_buffer_size = 16M
max_allowed_packet = 32M
table_open_cache = 64
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M

open_files_limit    = 10240
max_connections = 3000
max_connect_errors = 6000
table_cache = 1024
back_log = 600
max_allowed_packet = 32M
sort_buffer_size = 2M
join_buffer_size = 2M
thread_cache_size = 64
thread_concurrency = 16
query_cache_size = 32M
query_cache_limit = 2M
query_cache_min_res_unit = 2k
#default-storage-engine = MyISAM
#default_table_type = MyISAM
thread_stack = 192K
transaction_isolation = READ-COMMITTED
tmp_table_size = 256M
max_heap_table_size = 256M
binlog_cache_size = 8M
max_binlog_cache_size = 8M
max_binlog_size = 512M
expire_logs_days = 7
key_buffer_size = 256M
read_buffer_size = 1M
read_rnd_buffer_size = 16M
bulk_insert_buffer_size = 64M

myisam_sort_buffer_size = 128M
myisam_max_sort_file_size = 10G
myisam_repair_threads = 1
myisam_recover
skip-name-resolve
#master-connect-retry = 10
#slave-skip-errors = 1032,1062,126,1114,1146,1048,1396
innodb_additional_mem_pool_size = 16M
innodb_buffer_pool_size = 1024M
innodb_file_io_threads = 4
innodb_thread_concurrency = 16
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 16M
innodb_max_dirty_pages_pct = 90
innodb_lock_wait_timeout = 120
innodb_file_per_table = 0




[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates

[myisamchk]
key_buffer_size = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout