Redis Cluster 2019(1)Set Up 3 Masters and 3 Replicas Version 5.0.5

Redis Cluster 2019(1)

Install the latest version 5.0.5 currently
> wget http://download.redis.io/releases/redis-5.0.5.tar.gz

Unzip and go to that directory
> make distclean
> make

Prepare the installation directory
> mkdir ~/tool/redis-5.0.5

> mkdir ~/tool/redis-5.0.5/bin
> cp src/redis-server ~/tool/redis-5.0.5/bin/
> cp src/redis-cli ~/tool/redis-5.0.5/bin/
> cp src/redis-benchmark ~/tool/redis-5.0.5/bin/

> mkdir ~/tool/redis-5.0.5/conf
> cp redis.conf ~/tool/redis-5.0.5/conf/

Soft link
> sudo ln -s /home/carl/tool/redis-5.0.5 /opt/redis-5.0.5
> sudo ln -s /opt/redis-5.0.5 /opt/redis

Prepare the cluster conf
> mkdir cluster-conf
> mkdir cluster-conf/7001
> mkdir cluster-conf/7002

Prepare the default configuration to the cluster
> cp conf/redis.conf cluster-conf/7001/
> cp conf/redis.conf cluster-conf/7002/

Change the cluster-conf/7001/redis.conf as follow:
port 7001
appendonly yes
cluster-enabled yes
cluster-config-file nodes-7001.conf
cluster-node-timeout 15000
bind 0.0.0.0

Prepare the same thing for ubuntu-master:7001, ubuntu-master:7002, ubuntu-dev2:7001, ubuntu-dev2:7002

Command to start them one by one
> bin/redis-server ./cluster-conf/7001/redis.conf
> bin/redis-server ./cluster-conf/7002/redis.conf

I saw something like this in the logging
No cluster configuration found, I'm 8b65491322a3c5d650a6027b1e47465fb5a03c32

Run this command to initiate the cluster
> redis-cli --cluster create ubuntu-master:7001 ubuntu-master:7002 ubuntu-dev2:7001 ubuntu-dev2:7002 --cluster-replicas 1

Error Message as follow:
*** ERROR: Invalid configuration for cluster creation.
*** Redis Cluster requires at least 3 master nodes.
*** This is not possible with 4 nodes and 1 replicas per node.
*** At least 6 nodes are required.

I will try to create 7003 on ubuntu-dev2 and ubuntu-master.

Another error message:
Node ubuntu-master:7002 replied with error:
ERR Invalid node address specified: ubuntu-master:7001

Solution:
Will use IP instead of hostname
> redis-cli --cluster create 192.168.56.101:7001 192.168.56.101:7002 192.168.56.101:7003 192.168.56.3:7001 192.168.56.3:7002 192.168.56.3:7003 --cluster-replicas 1

[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

Connect and Verify
> redis-cli -c -h ubuntu-master -p 7001

set foo bar
-> Redirected to slot [12182] located at 192.168.56.101:7002
OK
192.168.56.101:7002> get foo
"bar"

Show all the nodes information
> cluster nodes
4aaa84498db791f6865d095c1beb59c1c2d476c7 192.168.56.101:7003@17003 slave fb1174b72aeb5ffd0152720c21e2690a5380cd28 0 1560192317000 4 connected
8b65491322a3c5d650a6027b1e47465fb5a03c32 192.168.56.3:7002@17002 slave e96b06f852023e64546e7566fd2ae417731c78c3 0 1560192316000 5 connected
fb1174b72aeb5ffd0152720c21e2690a5380cd28 192.168.56.3:7001@17001 master - 0 1560192319919 4 connected 5461-10922
36235b5b5954b15ad90d6d9acdee7d83eacb1a2f 192.168.56.101:7001@17001 master - 0 1560192319000 1 connected 0-5460
eae59392de289ead846ab08be2893deda52ed3d3 192.168.56.3:7003@17003 slave 36235b5b5954b15ad90d6d9acdee7d83eacb1a2f 0 1560192318897 6 connected
e96b06f852023e64546e7566fd2ae417731c78c3 192.168.56.101:7002@17002 myself,master - 0 1560192318000 2 connected 10923-16383

Info will show more information
> info
# Server
redis_version:5.0.5
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:7d80a20d45e418a3
redis_mode:cluster

It seems that this one is good tool
https://github.com/LittlePeng/redis-monitor
https://github.com/steelThread/redmon

> sudo apt install ruby
> sudo apt install ruby-dev
> sudo gem install redmon

Check the installation
> redmon -h

Not working, I think should be version issues.

Try this one
https://github.com/nkrode/RedisLive
> sudo apt-get install python python-dev
> sudo apt install python-pip

> pip install tornado
> pip install redis
> pip install python-dateutil

> python -V
Python 2.7.16

> git clone https://github.com/kumarnitin/RedisLive.git

Copy to working directory
> mv RedisLive ~/tool/
> sudo ln -s /home/carl/tool/RedisLive /opt/redislive
> cd /opt/redislive

> cp src/redis-live.conf.example src/redis-live.conf

After I configure that, I get error as follow
AttributeError: 'str' object has no attribute 'iteritems'

I think it is python version issues.
I used to use this very old tool
https://github.com/junegunn/redis-stat

> redis-stat ubuntu-dev2:7001 ubuntu-dev2:7002 ubuntu-dev2:7003 ubuntu-master:7001 ubuntu-master:7002 ubuntu-master:7003

It is ok to use it. And it seems that we can send the logging to Elastisearch as well.
> redis-stat ubuntu-dev2:7001 ubuntu-dev2:7002 ubuntu-dev2:7003 ubuntu-master:7001 ubuntu-master:7002 ubuntu-master:7003 --server=7000

It will open a webUI
http://ubuntu-dev2:7000/

We can put that into daemon as well

> redis-stat ubuntu-dev2:7001 ubuntu-dev2:7002 ubuntu-dev2:7003 ubuntu-master:7001 ubuntu-master:7002 ubuntu-master:7003 --server=7000 --daemon

We can send the data to Elasticsearch as well. I start to run the elastic and Kibana. Let me try to logging.

There is a known issue after this command
> redis-stat ubuntu-dev2:7001 ubuntu-dev2:7002 ubuntu-dev2:7003 ubuntu-master:7001 ubuntu-master:7002 ubuntu-master:7003 --es=http://ubuntu-master:9200/redisstat

Exception:
[2019/06/10 22:18:57@elasticsearch] [406] {"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}

A workaround, we can gather all the information to CSV file
> redis-stat ubuntu-dev2:7001 ubuntu-dev2:7002 ubuntu-dev2:7003 ubuntu-master:7001 ubuntu-master:7002 ubuntu-master:7003 --csv=./redis-stat.csv

I think Filebeat can gather all these informations to Logstash, and Logstash convert CSV to JSON in Elasticsearch with data format

References:
https://redis.io/topics/cluster-tutorial
Useful Redis Tool
https://github.com/AlexStocks/redis-tool-set