redis-cluster构建与JedisCluster测试连接

最近公司想用redis集群，正好看见redis3.0 release出来了，就准备上手试一试。
下文主要是搭建时遇到的一些问题和心里历程。

引用

原创：http://langmnm.iteye.com/admin/blogs/2286186

一、关于redis-cluster配置与说明
因为有现成的文章，我就不在多说。

引用

http://hot66hot.iteye.com/blog/2050676?page=2#comments

二、搭建过程中所遇到的坑
因搭建过程中，使用上文引用的链接中操作步骤，遇到一些问题，所以自己重新写了一下搭建过程（可能使用的版本有差异造成）。遇到以下问题：
1）ruby安装失败。ruby2.3.0换成了ruby2.2.4安装成功，原因没弄清楚。
2）rubygems在ruby中已内置
3）redis.gem手动下载安装失败，gem install redis时出错
因为安装ruby之后，安装ruby-zlib

no such file to load -- redis (LoadError)

1.测试环境
linux ip：192.168.71.188
node_1 port：6380 master
node_2 port：6381 master
node_3 port：6382 master
node_4 port：7380 slave
node_5 port：7381 slave
node_6 port：7382 slave

redis：3.0.7
zlib：1.2.8
ruby：2.2.4
redis.gem：3.2.1

关于各结点port占用问题：

引用

http://redis.io/topics/cluster-spec
The Cluster bus
Every Redis Cluster node has an additional TCP port for receiving incoming connections from other Redis Cluster nodes. This port is at a fixed offset from the normal TCP port used to receive incoming connections from clients. To obtain the Redis Cluster port, 10000 should be added to the normal commands port. For example, if a Redis node is listening for client connections on port 6379, the Cluster bus port 16379 will also be opened.
Node-to-node communication happens exclusively using the Cluster bus and the Cluster bus protocol: a binary protocol composed of frames of different types and sizes. The Cluster bus binary protocol is not publicly documented since it is not intended for external software devices to talk with Redis Cluster nodes using this protocol. However you can obtain more details about the Cluster bus protocol by reading the cluster.h and cluster.c files in the Redis Cluster source code.

注：各结点通信是通过特点port来进行信息交互的。如结点port为6380，则与其他结点的tcp连接port为16380。


2.redis-cluster集群搭建
1）redis3.0

cd /opt/redis-x.x.x
make  
sudo cp /opt/redis/src/redis-server /usr/local/bin  
sudo cp /opt/redis/src/redis-cli /usr/local/bin  
sudo cp /opt/redis/src/redis-trib.rb /usr/local/bin

2）zlib

cd /path/zlib-x.x.x
./configure
make
make install

3）ruby

cd /path/ruby-x.x.x
./configure -prefix=/usr/local/ruby
make
make install
sudo cp ruby /usr/local/bin

4）ruby-zlib ruby内置

cd /rubypath/
cd ext/zlib
cd ruby ./extconf.rb
make
make install

5）rubygems ruby内置 依赖zlib、ruby-zlib

cd /rubypath/
sudo cp bin/gem /usr/local/bin

6）redis.gem 依赖rubygems

cd /rubypath/bin
#gem source -r=remove -a=add -l=curr source
#gem source -l
#gem source -r https://rubygems.org/
#gem source -a http://rubygems.org/

#gem install redis --source http://rubygems.org
#gem install redis --version 3.0.0
gem install redis

7）redis-cluster结点启动

redis-server /opt/redis-cluster/conf/redis-6380.conf > /opt/redis-cluster/logs/redis-6380.log 2>&1 &
redis-server /opt/redis-cluster/conf/redis-6381.conf > /opt/redis-cluster/logs/redis-6381.log 2>&1 &
redis-server /opt/redis-cluster/conf/redis-6382.conf > /opt/redis-cluster/logs/redis-6382.log 2>&1 &
redis-server /opt/redis-cluster/conf/redis-7380.conf > /opt/redis-cluster/logs/redis-7380.log 2>&1 &
redis-server /opt/redis-cluster/conf/redis-7381.conf > /opt/redis-cluster/logs/redis-7381.log 2>&1 &
redis-server /opt/redis-cluster/conf/redis-7382.conf > /opt/redis-cluster/logs/redis-7382.log 2>&1 &

8）redis-trib.rb 依赖ruby、redis.gem
使用redis自带的redis-trib.rb来构建集群，完事slots分配等。

#redis-trib.rb的create子命令构建  
#--replicas 则指定了为Redis Cluster中的每个Master节点配备几个Slave节点  
#节点角色由顺序决定,先master之后是slave(为方便辨认,slave的端口比master大1000)  
redis-trib.rb create --replicas 1 192.168.71.188:6380 192.168.71.188:6381 192.168.71.188:6382 192.168.71.188:7380 192.168.71.188:7381 192.168.71.188:7382

9）检查集群状态 check redis-cluster

#redis-trib.rb的check子命令构建  
#ip:port可以是集群的任意节点  
redis-trib.rb check 192.168.71.188:6380

[OK] All nodes agree about slots configuration.  
>>> Check for open slots...  
>>> Check slots coverage...  
[OK] All 16384 slots covered. 

10）redis-cli -c 可自动MOVED到对应slot节点上

redis-cli -c -p 6380
192.168.71.188:7381> set aa a
-> Redirected to slot [1180] located at 127.0.0.1:7380
OK
127.0.0.1:7380> set bb b
-> Redirected to slot [8620] located at 127.0.0.1:7381
OK
127.0.0.1:6382> get aa
-> Redirected to slot [1180] located at 127.0.0.1:7380
"a"
127.0.0.1:7380> get bb
-> Redirected to slot [8620] located at 127.0.0.1:7381
"b"

11）Redis-rb-cluster

#Redis Cluster client work in progress. It wraps Redis-rb, and eventually should be part of it.  
ruby example.rb
1
2
3
4
5
6
7
8
9
^C (I stopped the program here)

ruby consistency-test.rb 192.168.71.188 6380
925 R (0 err) | 925 W (0 err) |
5030 R (0 err) | 5030 W (0 err) |
9261 R (0 err) | 9261 W (0 err) |
13517 R (0 err) | 13517 W (0 err) |
17780 R (0 err) | 17780 W (0 err) |
22025 R (0 err) | 22025 W (0 err) |
25818 R (0 err) | 25818 W (0 err) |

引用

https://github.com/antirez/redis-rb-cluster
http://blog.csdn.net/zalion/article/details/45192507

3.JedisCluster测试
用jedis2.8使用JedisCluster访问，一访问就报错：

Exception in thread "main" redis.clients.jedis.exceptions.JedisClusterMaxRedirectionsException:Too many Cluster redirections?

最开始以后是集群搭建的有问题，但是各处也排查了，集群状态是OK的。
同时测试了一下集群的容错功能，一切正常。
那么我就想，肯定不是集群的问题。又用Jedis单结点访问，一切正常。
集群正常，单结点访问也正常，这就奇怪了，为什么用JedisCluster就不行呢？
难道是jedis2.8对redis-cluster支持的不好？上jedis一看，是支持redis-cluster的。看jedis2.8.1发布了，死马当活马医吧，maven改下version，问题依就存在。
头疼啊，上网找资料吧。关于cluster的还是太少，没找到中文可以的。没办法看英文文档吧，英文又太差，将就吧。文档看了不少，还是没找到问题。
没办法了，就想着要不换个服务器重新搭建一下试试？
所以就开始看看自己搭建之初记录下来的操作步骤。哎呀，有发现啊。

redis-trib.rb create --replicas 1 127.0.0.1:6380 127.0.0.1:6381 127.0.0.1:6382 127.0.0.1:7380 127.0.0.1:7381 127.0.0.1:7382

难道是这的问题？
kill掉所以结点，删除data下的所有数据（dump、aof、nodes）文件，重新配置

redis-trib.rb create --replicas 1 192.168.71.188:6380 192.168.71.188:6381 192.168.71.188:6382 192.168.71.188:7380 192.168.71.188:7381 192.168.71.188:7382

立马执行一下JedisCluster，OK，问题解决了~
那个心情啊...超好了
但是回过头来一看之前看的文档，心情不好了~~~都是眼泪啊~
就这么个问题，快弄了两天了~~~只能说干这行没学好英语苦啊...

引用

注：redis-trib.rb中的IP是根据conf中bind的IP来决定的，保持一致，bind默认是127.0.0.1。如果是部署到服务器，则需要配置bind为服务器IP。
https://github.com/xetorthio/jedis/issues/943

4.Jedis测试代码
Redis cluster specification (still under development) is implemented

Set<HostAndPort> jedisClusterNodes = new HashSet<HostAndPort>();
//Jedis Cluster will attempt to discover cluster nodes automatically
//只需要配置集群中的一个结点，连接成功后，自动获取点集群中其他结点信息
jedisClusterNodes.add(new HostAndPort("192.168.71.188", 7380));
JedisCluster jc = new JedisCluster(jedisClusterNodes);
jc.set("foo", "bar");
String value = jc.get("foo");

引用

https://github.com/xetorthio/jedis

三、redis-cluster对Lua的支持
目前redis-cluster只支持单KEYS[1]操作，因为不同key存储的slot不同。

        Set<HostAndPort> jedisClusterNodes = new HashSet<HostAndPort>();
        jedisClusterNodes.add(new HostAndPort("192.168.71.188", 7380));
        System.out.println("jcn set initialised");
        JedisCluster jc = new JedisCluster(jedisClusterNodes);
        String sha = jc.scriptLoad(
            "redis.call('set', KEYS[1], '100') ;"
            + "redis.call('set', KEYS[2], '200') ;"
            + "return true;", "test");
        System.out.println(sha);
        List<String> listKey = new ArrayList<String>();
        List<String> listParam = new ArrayList<String>();
        listKey.add("aa");
        listKey.add("cc");
        System.out.println(jc.evalsha(sha, listKey, listParam));
        System.out.println(jc.get("aa"));
        System.out.println(jc.get("cc"));

throw new JedisClusterException("No way to dispatch this command to Redis Cluster because keys have different slots.");

// Referenced classes of package redis.clients.jedis:
//            Jedis, JedisClusterConnectionHandler

public abstract class JedisClusterCommand
{
    public Object run(String key)
    {
        if(key == null)
            throw new JedisClusterException("No way to dispatch this command to Redis Cluster.");
        else
            return runWithRetries(SafeEncoder.encode(key), redirections, false, false);
    }

    public transient Object run(int keyCount, String keys[])
    {
        if(keys == null || keys.length == 0)
            throw new JedisClusterException("No way to dispatch this command to Redis Cluster.");
        if(keys.length > 1)
        {
            int slot = JedisClusterCRC16.getSlot(keys[0]);
            for(int i = 1; i < keyCount; i++)
            {
                int nextSlot = JedisClusterCRC16.getSlot(keys[i]);
                if(slot != nextSlot)
                    throw new JedisClusterException("No way to dispatch this command to Redis Cluster because keys have different slots.");
            }

        }
        return runWithRetries(SafeEncoder.encode(keys[0]), redirections, false, false);
    }
}