Linux 下 Redis Cluster 搭建

Redis集群演变过程：

单机版----->主从复制------>哨兵（小公司）----->集群（大公司）

Redis cluster集群：

(1)所有的redis节点彼此互联(PING-PONG机制),内部使用二进制协议优化传输速度和带宽.

(2)节点的fail是通过集群中超过半数的节点检测失效时才生效.通过投票机制

(3)客户端与redis节点直连,不需要中间proxy层.客户端不需要连接集群所有节点,连接集群中任何一个可用节点即可

(4)redis-cluster把所有的物理节点映射到[0-16383]slot上,cluster 负责维护node<->slot<->value

(5)redis 集群中内置了 16384 个哈希槽，当需要在 Redis 集群中放置一个 key-value 时，redis 先对 key 使用 crc16 算法算出一个结果，然后把结果对 16384 求余数，这样每个 key 都会对应一个编号在 0-16383 之间的哈希槽，redis 会根据节点数量大致均等的将哈希槽映射到不同的节点

由于Redis的容错投票机制是集群中过半数的节点认为某个节点检测失效时才生效，所以搭建最小集群模式至少需要三个节点，但是为了集群的高可用，就需要为每一个节点增加一个备份机，所以就需要六台服务器。

多个主节点、每个主节点可附带多个从节点、多个主节点共分16384（0---16383）个槽位（slot）


Redis cluster分片特点：

1）无中心架构，支持动态扩容
2）具备Sentinel的监控和自动切换能力
3）客户端不需要连接集群所有节点,连接集群中任何一个可用节点即可
4）高性能，客户端直连redis服务，不需要使用代理

Redis cluster有固定16384个slot，每次写入，会对key计算CRC16值，然后对16384求模，计算出应该存放slot位置，然后会找到对应的master节点。

假设有三主三从，那么slot在master划分：[0－5460]，[5461－10922]， [10923－16383]，每个key都会找到属于自己的master节点，在master节点中进行主从复制

集群中某个master节点宕机，会进行主备切换，如果主从全部宕机，由于分片，只会影响相邻节点，引发数据迁移，迁移代价较小，而其它节点正常工作，不会因为缓存大面积失效造成缓存雪崩，造成数据库压力过大甚至宕机。

选举策略：每个从节点，都根据自己对 master 复制数据的 offset，来设置一个选举时间，offset 越大（复制数据越多）的从节点，选举时间越靠前，优先进行选举，半数以上投票通过，该从节点可以切换成为master。


5.0.9 版本搭建：

主节点M1,从节点S2 192.101.11.153:7001,192.101.11.153:7002
主节点M2,从节点S3 192.101.11.154:7003,192.101.11.154:7004
主节点M3,从节点S1 192.101.11.155:7005,192.101.11.155:7006


M1：192.101.11.153:7001 S1：192.101.11.155:7006
M2：192.101.11.154:7003 S2：192.101.11.153:7002
M3：192.101.11.155:7005 S3：192.101.11.154:7004


一、下载安装包

http://download.redis.io/releases/
http://download.redis.io/releases/redis-5.0.9.tar.gz

创建目录:
mkdir -p /opt/redis-cluster

提示：redis-5.0.9.tar.gz 上传到三个服务器该目录：/opt/redis-cluster

二、解压编译

安装编译环境:

yum install -y gcc gcc-c++ make
yum -y update

创建目录:

mkdir -p /usr/local/redis-cluster/

解压安装包:
cd /opt/redis-cluster

tar -zxvf redis-5.0.5.tar.gz

mv /opt/redis-cluster/redis-5.0.9 /usr/local/redis-cluster/

编译安装:
cd /usr/local/redis-cluster/redis-5.0.9/

make PREFIX=/usr/local/redis install

检测:

make test

三、集群分配

主节点M1,从节点S2 192.101.11.153:
mkdir -p /usr/local/redis-cluster/redis-m1
mkdir -p /usr/local/redis-cluster/redis-s2
cp /usr/local/redis/bin /usr/local/redis-cluster/redis-m1 -r
cp /usr/local/redis-cluster/redis-5.0.9/redis.conf  /usr/local/redis-cluster/redis-m1
cp /usr/local/redis/bin /usr/local/redis-cluster/redis-s2 -r
cp /usr/local/redis-cluster/redis-5.0.9/redis.conf  /usr/local/redis-cluster/redis-s2

主节点M2,从节点S3 192.101.11.154:
mkdir -p /usr/local/redis-cluster/redis-m2
mkdir -p /usr/local/redis-cluster/redis-s3
cp /usr/local/redis/bin /usr/local/redis-cluster/redis-m2 -r
cp /usr/local/redis-cluster/redis-5.0.9/redis.conf  /usr/local/redis-cluster/redis-m2
cp /usr/local/redis/bin /usr/local/redis-cluster/redis-s3 -r
cp /usr/local/redis-cluster/redis-5.0.9/redis.conf  /usr/local/redis-cluster/redis-s3

主节点M3,从节点S1 192.101.11.155:
mkdir -p /usr/local/redis-cluster/redis-m3
mkdir -p /usr/local/redis-cluster/redis-s1
cp /usr/local/redis/bin /usr/local/redis-cluster/redis-m3 -r
cp /usr/local/redis-cluster/redis-5.0.9/redis.conf  /usr/local/redis-cluster/redis-m3
cp /usr/local/redis/bin /usr/local/redis-cluster/redis-s1 -r
cp /usr/local/redis-cluster/redis-5.0.9/redis.conf  /usr/local/redis-cluster/redis-s1


四、配置文件

主节点M1 192.101.11.153:
mkdir -p /data/redis/m1/logs/
vi /usr/local/redis-cluster/redis-m1/redis.conf

bind 192.101.11.153  ##绑定IP，把 bind 127.0.0.1 给注释掉，这里的bind指的是只有指定的网段才能远程访问这个redis，注释掉后，就没有这个限制了，监听所有地址
port 7001 #端口
cluster-enabled yes #启用集群模式
cluster-node-timeout 5000 #超时时间5s
appendonly yes # 开启AOF日志持久化
daemonize yes #后台运行
cluster-config-file /usr/local/redis-cluster/redis-m1/nodes-7001.conf # redis群集名称文件设置
pidfile /var/run/redis_7001.pid
dbfilename 7001-dump.rdb
dir /data/redis/m1
logfile /data/redis/m1/logs/redis-7001.log


从节点S2 192.101.11.153:
mkdir -p /data/redis/s2/logs/
vi /usr/local/redis-cluster/redis-s2/redis.conf

bind 192.101.11.153  ##绑定IP，把 bind 127.0.0.1 给注释掉，这里的bind指的是只有指定的网段才能远程访问这个redis，注释掉后，就没有这个限制了，监听所有地址
port 7002 #端口
cluster-enabled yes #启用集群模式
cluster-node-timeout 5000 #超时时间5s
appendonly yes # 开启AOF日志持久化
daemonize yes #后台运行
cluster-config-file /usr/local/redis-cluster/redis-s2/nodes-7002.conf  # redis群集名称文件设置
pidfile /var/run/redis_7002.pid
dbfilename 7002-dump.rdb
dir /data/redis/s2
logfile /data/redis/s2/logs/redis-7002.log


主节点M2 192.101.11.154:
mkdir -p /data/redis/m2/logs/
vi /usr/local/redis-cluster/redis-m2/redis.conf

bind 192.101.11.154  ##绑定IP，把 bind 127.0.0.1 给注释掉，这里的bind指的是只有指定的网段才能远程访问这个redis，注释掉后，就没有这个限制了，监听所有地址
port 7003 #端口
cluster-enabled yes #启用集群模式
cluster-node-timeout 5000 #超时时间5s
appendonly yes # 开启AOF日志持久化
daemonize yes #后台运行
cluster-config-file /usr/local/redis-cluster/redis-m2/nodes-7003.conf # redis群集名称文件设置
pidfile /var/run/redis_7003.pid
dbfilename 7003-dump.rdb
dir /data/redis/m2
logfile /data/redis/m2/logs/redis-7003.log


从节点S3 192.101.11.154:
mkdir -p /data/redis/s3/logs/
vi /usr/local/redis-cluster/redis-s3/redis.conf

bind 192.101.11.154  ##绑定IP，把 bind 127.0.0.1 给注释掉，这里的bind指的是只有指定的网段才能远程访问这个redis，注释掉后，就没有这个限制了，监听所有地址
port 7004 #端口
cluster-enabled yes #启用集群模式
cluster-node-timeout 5000 #超时时间5s
appendonly yes # 开启AOF日志持久化
daemonize yes #后台运行
cluster-config-file /usr/local/redis-cluster/redis-s3/nodes-7004.conf   # redis群集名称文件设置
pidfile /var/run/redis_7004.pid 
dbfilename 7004-dump.rdb
dir /data/redis/s3
logfile /data/redis/s3/logs/redis-7004.log


主节点M3 192.101.11.155:
mkdir -p /data/redis/m3/logs/
vi /usr/local/redis-cluster/redis-m3/redis.conf

bind 192.101.11.155  ##绑定IP，把 bind 127.0.0.1 给注释掉，这里的bind指的是只有指定的网段才能远程访问这个redis，注释掉后，就没有这个限制了，监听所有地址
port 7005 #端口
cluster-enabled yes #启用集群模式
cluster-node-timeout 5000 #超时时间5s
appendonly yes # 开启AOF日志持久化
daemonize yes #后台运行
cluster-config-file /usr/local/redis-cluster/redis-m3/nodes-7005.conf   # redis群集名称文件设置
pidfile /var/run/redis_7005.pid
dbfilename 7005-dump.rdb
dir /data/redis/m3
logfile /data/redis/m3/logs/redis-7005.log


从节点S1 192.101.11.155:
mkdir -p /data/redis/s1/logs/
vi /usr/local/redis-cluster/redis-s1/redis.conf

bind 192.101.11.155  ##绑定IP，把 bind 127.0.0.1 给注释掉，这里的bind指的是只有指定的网段才能远程访问这个redis，注释掉后，就没有这个限制了，监听所有地址
port 7006 #端口
cluster-enabled yes #启用集群模式
cluster-node-timeout 5000 #超时时间5s
appendonly yes # 开启AOF日志持久化
daemonize yes #后台运行
cluster-config-file /usr/local/redis-cluster/redis-s1/nodes-7006.conf   # redis群集名称文件设置
pidfile /var/run/redis_7006.pid
dbfilename 7006-dump.rdb
dir /data/redis/s1
logfile /data/redis/s1/logs/redis-7006.log

其他可选设置：
1）开启远程访问：protected-mode no
2）设置redis集群密码：masterauth 123456 和 requirepass 123456     ##密码是123456 
   注意：所有节点的密码都必须一致，masterauth也要加

五、启动节点

配置好了之后，就可以启动节点了，三台机器分别启动。使用redis-server命令启动，启动时注意指定配置文件

主节点M1 192.101.11.153:
/usr/local/redis-cluster/redis-m1/bin/redis-server /usr/local/redis-cluster/redis-m1/redis.conf
从节点S2 192.101.11.153:
/usr/local/redis-cluster/redis-s2/bin/redis-server /usr/local/redis-cluster/redis-s2/redis.conf

[root@hadoop001 redis-s2]# ps -ef|grep redis
root      2609     1  0 00:43 ?        00:00:00 /usr/local/redis-cluster/redis-m1/bin/redis-server 192.101.11.153:7001 [cluster]
root      2660     1  0 00:44 ?        00:00:00 /usr/local/redis-cluster/redis-s2/bin/redis-server 192.101.11.153:7002 [cluster]


主节点M2 192.101.11.154:
/usr/local/redis-cluster/redis-m2/bin/redis-server /usr/local/redis-cluster/redis-m2/redis.conf
从节点S3 192.101.11.154:
/usr/local/redis-cluster/redis-s3/bin/redis-server /usr/local/redis-cluster/redis-s3/redis.conf

[root@hadoop002 redis-m2]# ps -ef|grep redis
root     21229     1  0 00:47 ?        00:00:00 /usr/local/redis-cluster/redis-m2/bin/redis-server 192.101.11.154:7003 [cluster]
root     21234     1  0 00:48 ?        00:00:00 /usr/local/redis-cluster/redis-s3/bin/redis-server 192.101.11.154:7004 [cluster]


主节点M3 192.101.11.155:
/usr/local/redis-cluster/redis-m3/bin/redis-server /usr/local/redis-cluster/redis-m3/redis.conf
从节点S1 192.101.11.155:
/usr/local/redis-cluster/redis-s1/bin/redis-server /usr/local/redis-cluster/redis-s1/redis.conf

[root@hadoop003 redis-m3]# ps -ef|grep redis
root      2239     1  0 00:51 ?        00:00:00 /usr/local/redis-cluster/redis-m3/bin/redis-server 192.101.11.155:7005 [cluster]
root      2261     1  0 00:51 ?        00:00:00 /usr/local/redis-cluster/redis-s1/bin/redis-server 192.101.11.155:7006 [cluster]


六、创建集群

三个节点都启动成功后，就可以创建集群了。Redis官方提供了命令，我们敲一下就可以了，非常方便。

提示：在其中一台服务器执行即可，选择主节点M1（192.101.11.153）

./redis-cli --cluster create ip1:port1 ip2:port2 ip3:port3

cd /usr/local/redis/bin

./redis-cli --cluster create 192.101.11.153:7001 192.101.11.154:7003 192.101.11.155:7005 192.101.11.153:7002 192.101.11.154:7004 192.101.11.155:7006 --cluster-replicas 1

提示：--cluster-replicas 1 命令的意思：一主一从配置，六个节点就是：三主三从

./redis-cli --cluster create 192.101.11.153:7001 192.101.11.154:7003 192.101.11.155:7005 192.101.11.153:7002 192.101.11.154:7004 192.101.11.155:7006 --cluster-replicas 1 -a 123456

提示：如果我们设置了redis集群的密码，那么就要带上 -a 123456 (密码)，其他同理

>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.101.11.154:7004 to 192.101.11.153:7001
Adding replica 192.101.11.155:7006 to 192.101.11.154:7003
Adding replica 192.101.11.153:7002 to 192.101.11.155:7005
M: e69af493737ff8fe6ced6a9edd41814c79fe4c54 192.101.11.153:7001
   slots:[0-5460] (5461 slots) master
M: 46683ba127a4c7ca9ecb5d56316b925a575d1741 192.101.11.154:7003
   slots:[5461-10922] (5462 slots) master
M: ff08cdb6ab20ec6649639a28481f3dcacdb83e6d 192.101.11.155:7005
   slots:[10923-16383] (5461 slots) master
S: af1887d5eef14a65793fd19fb0f9a24fb8c43f3c 192.101.11.153:7002
   replicates ff08cdb6ab20ec6649639a28481f3dcacdb83e6d
S: d3964275adfba68de04f9b75ece8a42b94207ed6 192.101.11.154:7004
   replicates e69af493737ff8fe6ced6a9edd41814c79fe4c54
S: 8971bd6b6071b6bcd16a9b03e71217d01555e3ab 192.101.11.155:7006
   replicates 46683ba127a4c7ca9ecb5d56316b925a575d1741
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
...
>>> Performing Cluster Check (using node 192.101.11.153:7001)
M: e69af493737ff8fe6ced6a9edd41814c79fe4c54 192.101.11.153:7001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: ff08cdb6ab20ec6649639a28481f3dcacdb83e6d 192.101.11.155:7005
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 8971bd6b6071b6bcd16a9b03e71217d01555e3ab 192.101.11.155:7006
   slots: (0 slots) slave
   replicates 46683ba127a4c7ca9ecb5d56316b925a575d1741
S: af1887d5eef14a65793fd19fb0f9a24fb8c43f3c 192.101.11.153:7002
   slots: (0 slots) slave
   replicates ff08cdb6ab20ec6649639a28481f3dcacdb83e6d
S: d3964275adfba68de04f9b75ece8a42b94207ed6 192.101.11.154:7004
   slots: (0 slots) slave
   replicates e69af493737ff8fe6ced6a9edd41814c79fe4c54
M: 46683ba127a4c7ca9ecb5d56316b925a575d1741 192.101.11.154:7003
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.


七、测试

查看帮助命令：

./redis-cli --help

检查集群状态：

./redis-cli --cluster check 192.101.11.153:7001

./redis-cli --cluster check 192.101.11.153:7001 -a 123456

提示：如果我们设置了redis集群的密码，那么就要带上 -a 123456 (密码)，其他同理


[root@hadoop001 bin]# ./redis-cli --cluster check 192.101.11.153:7001
192.101.11.153:7001 (e69af493...) -> 0 keys | 5461 slots | 1 slaves.
192.101.11.155:7005 (ff08cdb6...) -> 0 keys | 5461 slots | 1 slaves.
192.101.11.154:7003 (46683ba1...) -> 0 keys | 5462 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.101.11.153:7001)
M: e69af493737ff8fe6ced6a9edd41814c79fe4c54 192.101.11.153:7001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: ff08cdb6ab20ec6649639a28481f3dcacdb83e6d 192.101.11.155:7005
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 8971bd6b6071b6bcd16a9b03e71217d01555e3ab 192.101.11.155:7006
   slots: (0 slots) slave
   replicates 46683ba127a4c7ca9ecb5d56316b925a575d1741
S: af1887d5eef14a65793fd19fb0f9a24fb8c43f3c 192.101.11.153:7002
   slots: (0 slots) slave
   replicates ff08cdb6ab20ec6649639a28481f3dcacdb83e6d
S: d3964275adfba68de04f9b75ece8a42b94207ed6 192.101.11.154:7004
   slots: (0 slots) slave
   replicates e69af493737ff8fe6ced6a9edd41814c79fe4c54
M: 46683ba127a4c7ca9ecb5d56316b925a575d1741 192.101.11.154:7003
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.


./redis-cli -c -h 192.101.11.153 -p 7001

./redis-cli -c -h 192.101.11.153 -p 7001 -a 123456

提示：如果我们设置了redis集群的密码，那么就要带上 -a 123456 (密码)，其他同理

>cluster info

[root@hadoop001 bin]# ./redis-cli -c -h 192.101.11.153 -p 7001
192.101.11.153:7001> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:352
cluster_stats_messages_pong_sent:321
cluster_stats_messages_sent:673
cluster_stats_messages_ping_received:316
cluster_stats_messages_pong_received:352
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:673

192.101.11.153:7001> set hi redis
-> Redirected to slot [16140] located at 192.101.11.155:7005
OK

192.101.11.155:7005> get hi
"redis"

测试性能：

redis-benchmark -h 192.101.11.153 -p 7001 -c 100 -n 100000     ##100个并发连接，100000个请求，检测host为localhost 端口为7002的redis服务器性能


查看复制信息：
./redis-cli -c -h 192.101.11.153 -p 7001 -a 123456 info replication


八、手动添加slave方式：

以上创建集群方式是自动分配的 master 和 slave 的组合关系，而有时我们想按照规划好的组合关系，手动匹配他们的对应关系。

创建集群（创建不含slaver 的集群）：
./redis-cli --cluster create 192.101.11.153:7001 192.101.11.154:7003 192.101.11.155:7005 --cluster-replicas 0

提示：--cluster-replicas 0 命令的意思：一主零从配置，三个节点就是：三主零从

./redis-cli --cluster create 192.101.11.153:7001 192.101.11.154:7003 192.101.11.155:7005 --cluster-replicas 0 -a 123456
提示：如果我们设置了redis集群的密码，那么就要带上 -a 123456 (密码)，其他同理

[root@hadoop001 bin]# ./redis-cli --cluster create 192.101.11.153:7001 192.101.11.154:7003 192.101.11.155:7005 --cluster-replicas 0
>>> Performing hash slots allocation on 3 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
M: 0182025e5d7a7e248c0bf53bc91c1054b7f53122 192.101.11.153:7001
   slots:[0-5460] (5461 slots) master
M: ac978c3dd9ba47b1b83cbbd1ea4404735b1841b3 192.101.11.154:7003
   slots:[5461-10922] (5462 slots) master
M: e9b9eca0e106dcb12e74a71d231a3fdaf622bada 192.101.11.155:7005
   slots:[10923-16383] (5461 slots) master
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 192.101.11.153:7001)
M: 0182025e5d7a7e248c0bf53bc91c1054b7f53122 192.101.11.153:7001
   slots:[0-5460] (5461 slots) master
M: ac978c3dd9ba47b1b83cbbd1ea4404735b1841b3 192.101.11.154:7003
   slots:[5461-10922] (5462 slots) master
M: e9b9eca0e106dcb12e74a71d231a3fdaf622bada 192.101.11.155:7005
   slots:[10923-16383] (5461 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.


查看集群运行状态：
[root@hadoop001 bin]# ./redis-cli --cluster check 192.101.11.153:7001
192.101.11.153:7001 (0182025e...) -> 0 keys | 5461 slots | 0 slaves.
192.101.11.154:7003 (ac978c3d...) -> 0 keys | 5462 slots | 0 slaves.
192.101.11.155:7005 (e9b9eca0...) -> 0 keys | 5461 slots | 0 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.101.11.153:7001)
M: 0182025e5d7a7e248c0bf53bc91c1054b7f53122 192.101.11.153:7001
   slots:[0-5460] (5461 slots) master
M: ac978c3dd9ba47b1b83cbbd1ea4404735b1841b3 192.101.11.154:7003
   slots:[5461-10922] (5462 slots) master
M: e9b9eca0e106dcb12e74a71d231a3fdaf622bada 192.101.11.155:7005
   slots:[10923-16383] (5461 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.


查看集群运行状态（如果是通过--cluster-replicas 1 则创建了集群的slave,并自动进行分组匹配）：

./redis-cli --cluster check 192.101.11.153:7001
[root@hadoop001 bin]# ./redis-cli --cluster check 192.101.11.153:7001
192.101.11.153:7001 (e69af493...) -> 0 keys | 5461 slots | 1 slaves.
192.101.11.155:7005 (ff08cdb6...) -> 1 keys | 5461 slots | 1 slaves.
192.101.11.154:7003 (46683ba1...) -> 0 keys | 5462 slots | 1 slaves.
[OK] 1 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.101.11.153:7001)
M: e69af493737ff8fe6ced6a9edd41814c79fe4c54 192.101.11.153:7001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: ff08cdb6ab20ec6649639a28481f3dcacdb83e6d 192.101.11.155:7005
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 8971bd6b6071b6bcd16a9b03e71217d01555e3ab 192.101.11.155:7006
   slots: (0 slots) slave
   replicates 46683ba127a4c7ca9ecb5d56316b925a575d1741
S: af1887d5eef14a65793fd19fb0f9a24fb8c43f3c 192.101.11.153:7002
   slots: (0 slots) slave
   replicates ff08cdb6ab20ec6649639a28481f3dcacdb83e6d
S: d3964275adfba68de04f9b75ece8a42b94207ed6 192.101.11.154:7004
   slots: (0 slots) slave
   replicates e69af493737ff8fe6ced6a9edd41814c79fe4c54
M: 46683ba127a4c7ca9ecb5d56316b925a575d1741 192.101.11.154:7003
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.


删除当前的slave（如果是通过--cluster-replicas 1 创建了集群的slave,则删除重新匹配对应关系）：
./redis-cli -a 123456 --cluster del-node 192.101.11.155:7006 '8971bd6b6071b6bcd16a9b03e71217d01555e3ab'
./redis-cli -a 123456 --cluster del-node 192.101.11.153:7002 'af1887d5eef14a65793fd19fb0f9a24fb8c43f3c'
./redis-cli -a 123456 --cluster del-node 192.101.11.154:7004 'd3964275adfba68de04f9b75ece8a42b94207ed6'

提示：
slave1 id:8971bd6b6071b6bcd16a9b03e71217d01555e3ab
slave2 id:af1887d5eef14a65793fd19fb0f9a24fb8c43f3c
slave3 id:d3964275adfba68de04f9b75ece8a42b94207ed6


[root@hadoop001 bin]# ./redis-cli --cluster del-node 192.101.11.155:7006 '8971bd6b6071b6bcd16a9b03e71217d01555e3ab'
>>> Removing node 8971bd6b6071b6bcd16a9b03e71217d01555e3ab from cluster 192.101.11.155:7006
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.

[root@hadoop001 bin]# ./redis-cli --cluster del-node 192.101.11.154:7004 'd3964275adfba68de04f9b75ece8a42b94207ed6'
>>> Removing node d3964275adfba68de04f9b75ece8a42b94207ed6 from cluster 192.101.11.154:7004
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.

[root@hadoop001 bin]# ./redis-cli --cluster del-node 192.101.11.153:7002 'af1887d5eef14a65793fd19fb0f9a24fb8c43f3c'
>>> Removing node af1887d5eef14a65793fd19fb0f9a24fb8c43f3c from cluster 192.101.11.153:7002
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.

查看集群当前状态：
[root@hadoop001 bin]# ./redis-cli --cluster check 192.101.11.153:7001
192.101.11.153:7001 (e69af493...) -> 0 keys | 5461 slots | 0 slaves.
192.101.11.155:7005 (ff08cdb6...) -> 1 keys | 5461 slots | 0 slaves.
192.101.11.154:7003 (46683ba1...) -> 0 keys | 5462 slots | 0 slaves.
[OK] 1 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.101.11.153:7001)
M: e69af493737ff8fe6ced6a9edd41814c79fe4c54 192.101.11.153:7001
   slots:[0-5460] (5461 slots) master
M: ff08cdb6ab20ec6649639a28481f3dcacdb83e6d 192.101.11.155:7005
   slots:[10923-16383] (5461 slots) master
M: 46683ba127a4c7ca9ecb5d56316b925a575d1741 192.101.11.154:7003
   slots:[5461-10922] (5462 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

启动salve节点Redis：
从节点S1 192.101.11.155:
/usr/local/redis-cluster/redis-s1/bin/redis-server /usr/local/redis-cluster/redis-s1/redis.conf

从节点S2 192.101.11.153:
/usr/local/redis-cluster/redis-s2/bin/redis-server /usr/local/redis-cluster/redis-s2/redis.conf

从节点S3 192.101.11.154:
/usr/local/redis-cluster/redis-s3/bin/redis-server /usr/local/redis-cluster/redis-s3/redis.conf


集群主从对应关系：
M1：192.101.11.153:7001 S1：192.101.11.155:7006
M2：192.101.11.154:7003 S2：192.101.11.153:7002
M3：192.101.11.155:7005 S3：192.101.11.154:7004

./redis-cli --cluster add-node 192.101.11.155:7006 192.101.11.153:7001 --cluster-slave --cluster-master-id e69af493737ff8fe6ced6a9edd41814c79fe4c54
或者
./redis-cli -a 123456 --cluster add-node 192.101.11.155:7006 192.101.11.153:7001 --cluster-slave --cluster-master-id e69af493737ff8fe6ced6a9edd41814c79fe4c54

./redis-cli --cluster add-node 192.101.11.153:7002 192.101.11.153:7001 --cluster-slave --cluster-master-id 46683ba127a4c7ca9ecb5d56316b925a575d1741
或者
./redis-cli -a 123456 --cluster add-node 192.101.11.153:7002 192.101.11.153:7001 --cluster-slave --cluster-master-id 46683ba127a4c7ca9ecb5d56316b925a575d1741

./redis-cli --cluster add-node 192.101.11.154:7004 192.101.11.153:7001 --cluster-slave --cluster-master-id ff08cdb6ab20ec6649639a28481f3dcacdb83e6d
或者
./redis-cli -a 123456 --cluster add-node 192.101.11.154:7004 192.101.11.153:7001 --cluster-slave --cluster-master-id ff08cdb6ab20ec6649639a28481f3dcacdb83e6d

提示：
192.101.11.155:7006 为slave node, master1 id: e69af493737ff8fe6ced6a9edd41814c79fe4c54
192.101.11.153:7002 为slave node, master2 id: 46683ba127a4c7ca9ecb5d56316b925a575d1741
192.101.11.154:7004 为slave node, master3 id: ff08cdb6ab20ec6649639a28481f3dcacdb83e6d

提示：这一步出现如下的报错问题：解决见最后（九、问题及解决）
[root@hadoop001 bin]# ./redis-cli --cluster add-node 192.101.11.155:7006 192.101.11.153:7001 --cluster-slave --cluster-master-id e69af493737ff8fe6ced6a9edd41814c79fe4c54
>>> Adding node 192.101.11.155:7006 to cluster 192.101.11.153:7001
>>> Performing Cluster Check (using node 192.101.11.153:7001)
M: e69af493737ff8fe6ced6a9edd41814c79fe4c54 192.101.11.153:7001
   slots:[0-5460] (5461 slots) master
M: ff08cdb6ab20ec6649639a28481f3dcacdb83e6d 192.101.11.155:7005
   slots:[10923-16383] (5461 slots) master
M: 46683ba127a4c7ca9ecb5d56316b925a575d1741 192.101.11.154:7003
   slots:[5461-10922] (5462 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[ERR] Node 192.101.11.155:7006 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.


[root@hadoop001 bin]# ./redis-cli --cluster add-node 192.101.11.155:7006 192.101.11.153:7001 --cluster-slave --cluster-master-id 0182025e5d7a7e248c0bf53bc91c1054b7f53122
>>> Adding node 192.101.11.155:7006 to cluster 192.101.11.153:7001
>>> Performing Cluster Check (using node 192.101.11.153:7001)
M: 0182025e5d7a7e248c0bf53bc91c1054b7f53122 192.101.11.153:7001
   slots:[0-5460] (5461 slots) master
M: ac978c3dd9ba47b1b83cbbd1ea4404735b1841b3 192.101.11.154:7003
   slots:[5461-10922] (5462 slots) master
M: e9b9eca0e106dcb12e74a71d231a3fdaf622bada 192.101.11.155:7005
   slots:[10923-16383] (5461 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 192.101.11.155:7006 to make it join the cluster.
Waiting for the cluster to join
..
>>> Configure node as replica of 192.101.11.153:7001.
[OK] New node added correctly.
You have mail in /var/spool/mail/root

[root@hadoop001 bin]# ./redis-cli --cluster add-node 192.101.11.153:7002 192.101.11.153:7001 --cluster-slave --cluster-master-id ac978c3dd9ba47b1b83cbbd1ea4404735b1841b3
>>> Adding node 192.101.11.153:7002 to cluster 192.101.11.153:7001
>>> Performing Cluster Check (using node 192.101.11.153:7001)
M: 0182025e5d7a7e248c0bf53bc91c1054b7f53122 192.101.11.153:7001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: ac978c3dd9ba47b1b83cbbd1ea4404735b1841b3 192.101.11.154:7003
   slots:[5461-10922] (5462 slots) master
S: 9015969881858f7952263ba4ede6cb4e4b70715f 192.101.11.155:7006
   slots: (0 slots) slave
   replicates 0182025e5d7a7e248c0bf53bc91c1054b7f53122
M: e9b9eca0e106dcb12e74a71d231a3fdaf622bada 192.101.11.155:7005
   slots:[10923-16383] (5461 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 192.101.11.153:7002 to make it join the cluster.
Waiting for the cluster to join
...
>>> Configure node as replica of 192.101.11.154:7003.
[OK] New node added correctly.

[root@hadoop001 bin]# ./redis-cli --cluster add-node 192.101.11.154:7004 192.101.11.153:7001 --cluster-slave --cluster-master-id e9b9eca0e106dcb12e74a71d231a3fdaf622bada
>>> Adding node 192.101.11.154:7004 to cluster 192.101.11.153:7001
>>> Performing Cluster Check (using node 192.101.11.153:7001)
M: 0182025e5d7a7e248c0bf53bc91c1054b7f53122 192.101.11.153:7001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 279dfba90e40732c6b9bf9fdbb55ba6ab54a0028 192.101.11.153:7002
   slots: (0 slots) slave
   replicates ac978c3dd9ba47b1b83cbbd1ea4404735b1841b3
M: ac978c3dd9ba47b1b83cbbd1ea4404735b1841b3 192.101.11.154:7003
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
M: e9b9eca0e106dcb12e74a71d231a3fdaf622bada 192.101.11.155:7005
   slots:[10923-16383] (5461 slots) master
S: 9015969881858f7952263ba4ede6cb4e4b70715f 192.101.11.155:7006
   slots: (0 slots) slave
   replicates 0182025e5d7a7e248c0bf53bc91c1054b7f53122
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 192.101.11.154:7004 to make it join the cluster.
Waiting for the cluster to join

>>> Configure node as replica of 192.101.11.155:7005.
[OK] New node added correctly.


[root@hadoop001 bin]#  ./redis-cli --cluster check 192.101.11.153:7001
192.101.11.153:7001 (0182025e...) -> 0 keys | 5461 slots | 1 slaves.
192.101.11.154:7003 (ac978c3d...) -> 0 keys | 5462 slots | 1 slaves.
192.101.11.155:7005 (e9b9eca0...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.101.11.153:7001)
M: 0182025e5d7a7e248c0bf53bc91c1054b7f53122 192.101.11.153:7001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 279dfba90e40732c6b9bf9fdbb55ba6ab54a0028 192.101.11.153:7002
   slots: (0 slots) slave
   replicates ac978c3dd9ba47b1b83cbbd1ea4404735b1841b3
M: ac978c3dd9ba47b1b83cbbd1ea4404735b1841b3 192.101.11.154:7003
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 8c143cba38d0978d6202785502925062f4bac6b8 192.101.11.154:7004
   slots: (0 slots) slave
   replicates e9b9eca0e106dcb12e74a71d231a3fdaf622bada
M: e9b9eca0e106dcb12e74a71d231a3fdaf622bada 192.101.11.155:7005
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 9015969881858f7952263ba4ede6cb4e4b70715f 192.101.11.155:7006
   slots: (0 slots) slave
   replicates 0182025e5d7a7e248c0bf53bc91c1054b7f53122
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.


[root@hadoop001 bin]# ./redis-cli -c -h 192.101.11.155 -p 7005 cluster nodes
e9b9eca0e106dcb12e74a71d231a3fdaf622bada 192.101.11.155:7005@17005 myself,master - 0 1597759574000 3 connected 10923-16383
8c143cba38d0978d6202785502925062f4bac6b8 192.101.11.154:7004@17004 slave e9b9eca0e106dcb12e74a71d231a3fdaf622bada 0 1597759580000 3 connected
279dfba90e40732c6b9bf9fdbb55ba6ab54a0028 192.101.11.153:7002@17002 slave ac978c3dd9ba47b1b83cbbd1ea4404735b1841b3 0 1597759577000 2 connected
0182025e5d7a7e248c0bf53bc91c1054b7f53122 192.101.11.153:7001@17001 master - 0 1597759578000 1 connected 0-5460
ac978c3dd9ba47b1b83cbbd1ea4404735b1841b3 192.101.11.154:7003@17003 master - 0 1597759580856 2 connected 5461-10922
9015969881858f7952263ba4ede6cb4e4b70715f 192.101.11.155:7006@17006 slave 0182025e5d7a7e248c0bf53bc91c1054b7f53122 0 1597759579000 1 connected

 
九、问题及解决：

[root@hadoop001 bin]# ./redis-cli --cluster add-node 192.101.11.155:7006 192.101.11.153:7001 --cluster-slave --cluster-master-id e69af493737ff8fe6ced6a9edd41814c79fe4c54
>>> Adding node 192.101.11.155:7006 to cluster 192.101.11.153:7001
>>> Performing Cluster Check (using node 192.101.11.153:7001)
M: e69af493737ff8fe6ced6a9edd41814c79fe4c54 192.101.11.153:7001
   slots:[0-5460] (5461 slots) master
M: ff08cdb6ab20ec6649639a28481f3dcacdb83e6d 192.101.11.155:7005
   slots:[10923-16383] (5461 slots) master
M: 46683ba127a4c7ca9ecb5d56316b925a575d1741 192.101.11.154:7003
   slots:[5461-10922] (5462 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[ERR] Node 192.101.11.155:7006 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.

解决方法：

1)、将需要新增的节点下aof、rdb等本地备份文件删除；

2)、同时将新Node的集群配置文件删除,即：删除你redis.conf里面cluster-config-file所在的文件；

3)、再次添加新节点如果还是报错，则登录新Node,./redis-cli -c –h –p对数据库进行清除：

172.168.63.201:7001>  flushdb      #清空当前数据库

查看集群节点：
redis-cli -p 7000 -h 192.168.1.147 cluster nodes

连接到每个master节点上进行数据删除:

./redis-cli -c -h 192.101.11.153 -p 7001

./redis-cli -c -h 192.101.11.155 -p 7005

./redis-cli -c -h 192.101.11.154 -p 7003

> flushall ##清除所有redis中的数据

[root@hadoop003 bin]# ./redis-cli -c -h 192.101.11.155 -p 7005 cluster nodes
46683ba127a4c7ca9ecb5d56316b925a575d1741 192.101.11.154:7003@17003 master - 0 15                            97755991713 2 connected 5461-10922
ff08cdb6ab20ec6649639a28481f3dcacdb83e6d 192.101.11.155:7005@17005 myself,master                             - 0 1597755990000 3 connected 10923-16383
e69af493737ff8fe6ced6a9edd41814c79fe4c54 192.101.11.153:7001@17001 master - 0 15                            97755990711 1 connected 0-5460
[root@hadoop003 bin]# ./redis-cli -c -h 192.101.11.155 -p 7005
192.101.11.155:7005> flushall
OK
192.101.11.155:7005>exit
[root@hadoop003 bin]# ./redis-cli -c -h 192.101.11.153 -p 7001
192.101.11.153:7001> flushall
OK
192.101.11.153:7001> exit
[root@hadoop003 bin]# ./redis-cli -c -h 192.101.11.154 -p 7003
192.101.11.154:7003> flushall
OK
192.101.11.154:7003> exit

查看集群当前状态：
./redis-cli --cluster check 192.101.11.153:7001
192.101.11.153:7001 (e69af493...) -> 0 keys | 5461 slots | 0 slaves.
192.101.11.155:7005 (ff08cdb6...) -> 0 keys | 5461 slots | 0 slaves.
192.101.11.154:7003 (46683ba1...) -> 0 keys | 5462 slots | 0 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.101.11.153:7001)
M: e69af493737ff8fe6ced6a9edd41814c79fe4c54 192.101.11.153:7001
   slots:[0-5460] (5461 slots) master
M: ff08cdb6ab20ec6649639a28481f3dcacdb83e6d 192.101.11.155:7005
   slots:[10923-16383] (5461 slots) master
M: 46683ba127a4c7ca9ecb5d56316b925a575d1741 192.101.11.154:7003
   slots:[5461-10922] (5462 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.