keepalived实现故障自动切换

测试环境

写道

A:192.168.0.219 (ubuntu)
B:192.168.0.8 (FreeBsd)

不知道一个unix,一个linux行不行.keepalived好像对linux核心有要求.
二台机器都要做keepalived的配置呀..


keepalived是一个的IPVS包装和健康检查服务.FreeBSD的端口不支持keepalived的VRRP协议栈. :(

 

用linux双机测试，将来可以布署双linux主机

 

更改测试环境

写道

A:192.168.0.219 (ubuntu)
B:192.168.0.19 (CentOs)

这二台机已经互为主从

 

keepalived安装

写道

现在keepalived的最新版本是1.2.2

wget http://www.keepalived.org/software/keepalived-1.2.2.tar.gz

tar zxvf keepalived-1.2.2.tar.gz


A(ubuntu)安装keepalived:

xiaofei@xiaofei-desktop:~/keepalived-1.2.2$ ./configure
configure: error:
!!! OpenSSL is not properly installed on your system. !!!
!!! Can not include OpenSSL headers files.
配置出错,提示openssl安装得不正确.

执行:sudo apt-get install libssl-dev 这个错误解决.又出现新的错误.

checking for poptGetContext in -lpopt... no
configure: error: Popt libraries is required

执行:sudo apt-get install libpopt-dev 解决

configure: creating ./config.status
config.status: creating Makefile
config.status: creating genhash/Makefile
config.status: WARNING: 'genhash/Makefile.in' seems to ignore the --datarootdir setting
config.status: creating keepalived/core/Makefile
config.status: creating keepalived/include/config.h
config.status: creating keepalived.spec
config.status: creating keepalived/Makefile
config.status: WARNING: 'keepalived/Makefile.in' seems to ignore the --datarootdir setting
config.status: creating lib/Makefile
config.status: creating keepalived/vrrp/Makefile

Keepalived configuration
------------------------
Keepalived version : 1.2.2
Compiler : gcc
Compiler flags : -g -O2
Extra Lib : -lpopt -lssl -lcrypto
Use IPVS Framework : No
IPVS sync daemon support : No
Use VRRP Framework : Yes
Use Debug flags : No

然后:make && sudo make install

xiaofei@xiaofei-desktop:~$ whereis keepalived
keepalived: /usr/local/sbin/keepalived /usr/local/etc/keepalived


B(CentOs)安装keepalived:

由于这台CentOs是台新机,需要安装的东西估计多了.
[root@centos keepalived-1.2.2]# ./configure
checking for gcc... no
checking for cc... no
checking for cl.exe... no
configure: error: in `/root/keepalived/keepalived-1.2.2':
configure: error: no acceptable C compiler found in $PATH

GCC错误　yum install gcc 解决

checking openssl/ssl.h usability... no
checking openssl/ssl.h presence... no
checking for openssl/ssl.h... no
configure: error:
!!! OpenSSL is not properly installed on your system. !!!
!!! Can not include OpenSSL headers files. !!!

OpenSSL错误 yum install openssl openssl-devel 解决

checking for poptGetContext in -lpopt... no
configure: error: Popt libraries is required

Popt错误 yum install popt popt-devel 解决

configure: creating ./config.status
config.status: creating Makefile
config.status: creating genhash/Makefile
config.status: WARNING: 'genhash/Makefile.in' seems to ignore the --datarootdir setting
config.status: creating keepalived/core/Makefile
config.status: creating keepalived/include/config.h
config.status: creating keepalived.spec
config.status: creating keepalived/Makefile
config.status: WARNING: 'keepalived/Makefile.in' seems to ignore the --datarootdir setting
config.status: creating lib/Makefile
config.status: creating keepalived/vrrp/Makefile

Keepalived configuration
------------------------
Keepalived version : 1.2.2
Compiler : gcc
Compiler flags : -g -O2
Extra Lib : -lpopt -lssl -lcrypto
Use IPVS Framework : No
IPVS sync daemon support : No
Use VRRP Framework : Yes
Use Debug flags : No


然后 make && make install 即可

[root@centos keepalived-1.2.2]# whereis keepalived
keepalived: /usr/local/sbin/keepalived /usr/local/etc/keepalived

 

 

keepalived再次安装

写道

从先前安装keepalived我们看到,Use IPVS Framework:NO 应该要安装lvs

A:sudo apt-get install ipvsadm

B:yum install ipvsadm

A:
./configure --with-kernel-dir=/usr/src/linux-headers-2.6.32-41-generic

发现还有一个警告:
checking for nl_handle_alloc in -lnl... no
configure: WARNING: keepalived will be built without libnl support.

sudo apt-get install libnl-dev

重新编译了一次 包括make &&　make install

Keepalived configuration
------------------------
Keepalived version : 1.2.2
Compiler : gcc
Compiler flags : -g -O2
Extra Lib : -lpopt -lssl -lcrypto -lnl
Use IPVS Framework : Yes
IPVS sync daemon support : Yes
IPVS use libnl : Yes
Use VRRP Framework : Yes
Use Debug flags : No


不知道有没有效..

B:
./configure --with-kernel-dir=/usr/src/kernels/2.6.32-220.el6.x86_64

checking for nl_handle_alloc in -lnl... no
configure: WARNING: keepalived will be built without libnl support.

yum install libnl-devel

还有错误
checking net/ip_vs.h usability... no
checking net/ip_vs.h presence... no
checking for net/ip_vs.h... no
configure: WARNING: keepalived will be built without LVS support.

yum install kernel-devel　　

安装完后kernel也变了 : 2.6.32-220.17.1.el6.x86_64

./configure --with-kernel-dir=/usr/src/kernels/2.6.32-220.17.1.el6.x86_64

Keepalived configuration
------------------------
Keepalived version : 1.2.2
Compiler : gcc
Compiler flags : -g -O2
Extra Lib : -lpopt -lssl -lcrypto -lnl
Use IPVS Framework : Yes
IPVS sync daemon support : Yes
IPVS use libnl : Yes
Use VRRP Framework : Yes
Use Debug flags : No

接着make && make install

 

keepalived配置

我们自己在新建一个配置文件，默认情况下keepalived启动时会去/etc/keepalived目录下找配置文件

A:

cd /etc/keepalived
sudo vim keepalived.conf

我的内容:
!Configuration File for keepalived

global_defs {
         notification_email {
         [e-mail]
           }
         notification_email_from [e-mail]
         smtp_server 127.0.0.1
         smtp_connect_timeout 30
         router_id MySQL-ha
         }

vrrp_instance VI_1 {
        state BACKUP   #两台配置此处均是BACKUP
        interface eth0
        virtual_router_id 51
        priority 100   #优先级，另一台改为90
        advert_int 1
        nopreempt  #不抢占，只在优先级高的机器上设置即可，优先级低的机器不设置
        authentication {
          auth_type PASS
          auth_pass 1111
        }
        virtual_ipaddress {
          192.168.0.200
        }
        }

virtual_server 192.168.0.200 3306 {
        delay_loop 2   #每个2秒检查一次real_server状态
        lb_algo wrr   #LVS算法
        lb_kind DR    #LVS模式
        persistence_timeout 60   #会话保持时间
        protocol TCP
        real_server 192.168.0.219 3306 {
        weight 3
        notify_down /usr/local/MySQL/bin/MySQL.sh  #检测到服务down后执行的脚本
        TCP_CHECK {
          connect_timeout 10    #连接超时时间
          nb_get_retry 3       #重连次数
          delay_before_retry 3   #重连间隔时间
          connect_port 3306   #健康检查端口
        }
        }
        }

编写检测服务down后所要执行的脚本
 
 cd /usr/local
 mkdir MySQL
 cd MySQL
 mkdir bin

 #sudo vim /usr/local/MySQL/bin/MySQL.sh  
 内容:
 #!/bin/sh  
 pkill keepalived

 放开权限:
 #chmod +x /usr/local/MySQL/bin/MySQL.sh

注：此脚本是上面配置文件notify_down选项所用到的，keepalived使用notify_down选项来检查real_server的服务状态，当发现real_server服务故障时，便触发此脚本；
   我们可以看到，脚本就一个命令，通过pkill keepalived强制杀死keepalived进程，从而实现了MySQL故障自动转移。

启动keepalived

#sudo /usr/local/sbin/keepalived –D

接着我们来ping一下我们设置的虚拟ip(192.168.0.200)

[root@centos etc]# ping 192.168.0.200
PING 192.168.0.200 (192.168.0.200) 56(84) bytes of data.
64 bytes from 192.168.0.200: icmp_seq=1 ttl=64 time=0.951 ms
64 bytes from 192.168.0.200: icmp_seq=2 ttl=64 time=0.173 ms
64 bytes from 192.168.0.200: icmp_seq=3 ttl=64 time=0.204 ms
64 bytes from 192.168.0.200: icmp_seq=4 ttl=64 time=0.175 ms
64 bytes from 192.168.0.200: icmp_seq=5 ttl=64 time=0.138 ms

此时keepalived状态是正在运行中
xiaofei@xiaofei-desktop:~$ ps ax | grep keepalived
17617 ?        Ss     0:00 /usr/local/sbin/keepalived -D
17618 ?        S      0:00 /usr/local/sbin/keepalived -D
17619 ?        S      0:00 /usr/local/sbin/keepalived -D
17622 pts/1    S+     0:00 grep --color=auto keepalived

我停掉该机的mysql服务:  sudo /etc/init.d/mysql stop

再来看看keepallived的状态
xiaofei@xiaofei-desktop:~$ ps ax | grep keepalived
17669 pts/1    S+     0:00 grep --color=auto keepalived

我们发现当keepalived发现服务不响应时,自动杀死了keepalived进程...


B: B的配置跟Ａ一样,只是keepalived.conf文件有三个地方与Ａ不同 : 优先级为90、无抢占设置、real_server为本机IP

测试一下

一直不成功,查看日志  tail -f /var/log/message

Jun  6 16:15:59 centos Keepalived_healthcheckers: Opening file '/etc/keepalived/keepalived.conf'.
Jun  6 16:15:59 centos Keepalived_healthcheckers: Configuration is using : 11671 Bytes
Jun  6 16:15:59 centos Keepalived: Healthcheck child process(20866) died: Respawning
Jun  6 16:15:59 centos Keepalived: Starting Healthcheck child process, pid=20868
Jun  6 16:15:59 centos Keepalived_healthcheckers: IPVS: Can't initialize ipvs: Protocol not available
Jun  6 16:15:59 centos Keepalived_healthcheckers: Registering Kernel netlink reflector
Jun  6 16:15:59 centos Keepalived_healthcheckers: Registering Kernel netlink command channel


Ａ服务器中的日志是这样:
Jun  6 16:18:02 xiaofei-desktop Keepalived_vrrp: VRRP_Instance(VI_1) Received lower prio advert, forcing new election
Jun  6 16:18:02 xiaofei-desktop Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.0.200
Jun  6 16:18:03 xiaofei-desktop Keepalived_vrrp: VRRP_Instance(VI_1) Received lower prio advert, forcing new election
Jun  6 16:18:03 xiaofei-desktop Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.0.200


IPVS: Can't initialize ipvs: Protocol not available,应该是这个原因...

GG一下:打个补丁  modprobe -q ip_vs||true

再看日志:
Jun  6 16:30:30 centos Keepalived_healthcheckers: Configuration is using : 11651 Bytes
Jun  6 16:30:30 centos Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE
Jun  6 16:30:30 centos Keepalived_vrrp: VRRP sockpool: [ifindex(2), proto(112), fd(10,11)]
Jun  6 16:30:30 centos Keepalived_healthcheckers: Using LinkWatch kernel netlink reflector...
Jun  6 16:30:30 centos Keepalived_healthcheckers: Activating healtchecker for service [192.168.0.19]:3306
Jun  6 16:30:34 centos Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Jun  6 16:30:35 centos Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Jun  6 16:30:35 centos Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Jun  6 16:30:35 centos Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.0.200
Jun  6 16:30:35 centos avahi-daemon[1417]: Registering new address record for 192.168.0.200 on eth0.IPv4.
Jun  6 16:30:40 centos Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.0.200


好像没提示错误了.

停掉Ｂ服务器的mysql服务之后,也自动杀死了keepalived进程...

 

keepalived测试

写道

测试一:
当我停掉Ａ服务器的mysql服务之后,ping 192.168.0.200还是正常的;
当我再停Ｂ服务器的mysql服务之后,ping 192.2168.0.200就不通了;

当我打开Ｂ服务器的mysql服务之后,又通了;

测试二:
用第三台机(192.168.0.6)mysql远程登陆192.168.0.200

首先还是得在219及19的机器上设置允许远程登陆;

grant select on *.* to 'remote_guest'@'%' identified by '123456';

暂时先在这二台机器上创建remote_guest这个帐号,给select权限测试;

ssh 192.168.0.6

> mysql -h192.168.0.200 -uremote_guest -p123456
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2271
Server version: 5.5.24-log MySQL Community Server (GPL)

Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

通过查看我知道这时进的mysql是192.168.0.219的..

接着我们来进行故障测试,停掉Ａ的mysql服务

mysql -h192.168.0.200 -uremote_guest -p123456
ERROR 2003 (HY000): Can't connect to MySQL server on '192.168.0.200' (60)

有错误,没能连到19的服务器

直接通过　mysql -h192.168.0.19 -uremote_guest -p123456 -P3306
是可以连到19的mysql的,帐号建立的是没有问题.

Ａ当掉后,Ｂ没有接管

[root@centos ~]# tail -f /var/log/messages
Jun 6 17:30:41 centos Keepalived_healthcheckers: Using LinkWatch kernel netlink reflector...
Jun 6 17:30:41 centos Keepalived_healthcheckers: Activating healtchecker for service [192.168.0.19]:3306
Jun 6 17:30:45 centos Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Jun 6 17:30:46 centos Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Jun 6 17:30:46 centos Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Jun 6 17:30:46 centos Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.0.200
Jun 6 17:30:46 centos avahi-daemon[1417]: Registering new address record for 192.168.0.200 on eth0.IPv4.
Jun 6 17:30:51 centos Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.0.200
Jun 6 17:57:45 centos dmeventd[20127]: dmeventd ready for processing.
Jun 6 17:57:45 centos lvm[20127]: Monitoring snapshot 2wd1gdisk-vm--snapshot

什么lvm,不会是你也在这台机上搞,搞得我搞坏了吧　跟我先前的日志都不一样了


今天又试了一下　　在192.168.0.6 上　mysql -h192.168.0.200 -uremote_guest -p123456
居然连上了,进的就是19的mysql

我再接着把Ａ服务器的mysql服务启用,这个时候应该还是只会一直连接Ｂ数据库....... OK

接着再断掉Ｂ服务器,看能不能自动切到Ａ服务器.......OK

-------

虽然Ａ,Ｂ是互为主从,但还是以Ａ为主,只是这种方式切换,当切换到Ｂ后,Ａ恢复正常后不能再自动切回Ａ.

keepalived只能做到对3306的健康检查，但是做不到比如像MySQL复制中的slave-SQL、slave-IO进程的检查。
所以要想做到一些细致的健康检查，还得需要借助额外的监控工具，比如nagios，然后用nagios实现短信、邮件报警，从而能够有效地解决问题。