2.4：Redis Cluster 部署
2.4.1：部署 Redis Cluster 的前提
- 时间同步；
- Redis Node 均采用相同的硬件配置、相同的密码、相同的 Redis 版本；
- Redis Node 必须开启的参数：
- cluster-enabled yes：启用集群，开启后 Redis 进程会有 Cluster 显示；
- cluster-config-file nodes-6380.conf：指定集群配置文件，此文件由 Redis Cluster 自动创建和维护，开启即可，不需要任何手动配置；
- 所有 Redis Node 必须没有任何数据；
2.4.2：拓扑图
共 6 台 Redis 服务器组成 Redis Cluster，三主三从，即集群中的 Redis Node 为 3 个；
奇数为 Master，偶数为 Slave，方便后期维护管理时辨别；
-
Masters：Redis1、Redis3、Redis5；
-
Slaves：Redis2、Redis4、Redis6；
各 Redis 服务器编译安装 redis-4.0.14 和 redis-5.0.10（为了测试 redis 5 版本的集群创建和维护）；
2.4.3：各节点配置并启动 Redis
编辑各节点的 Redis 配置文件：
配置 masterauth（如果有 requirepass）；
开启集群；
指定集群配置文件；
masterauth 123456
cluster-enabled yes
cluster-config-file nodes-6379.conf
各节点启动为单机 Redis：
systemctl start redis && systemctl enable redis
2.4.4：创建集群（Redis-3/4）
准备 redis-trib.rb 使用环境
Redis 3 和 4 版本创建集群时，需要使用集群管理工具 redis-trib.rb，这个工具是 Redis 官方推出的管理 Redis 集群的工具，集成在 Redis 源码的 src 目录下，是基于 Redis 提供的集群命令而封装的简单、便捷、实用的操作工具；
redis-trib.rb 是 Redis 作者用 Ruby 开发完成的，CentOS yum 安装的 Ruby 存在版本较低问题，需要编译安装 Ruby：
只需在一台 Redis 服务器上准备环境即可；
[root@redis1 ~]# yum remove ruby rubygems -y
[root@redis1 ~]# cd /usr/local/src
[root@s1 src]# wget https://cache.ruby-lang.org/pub/ruby/2.5/ruby-2.5.5.tar.gz
[root@redis1 src]# tar xf ruby-2.5.5.tar.gz
[root@redis1 src]# cd ruby-2.5.5/
[root@redis1 ruby-2.5.5]# ./configure
[root@redis1 ruby-2.5.5]# make -j 2
[root@redis1 ruby-2.5.5]# make install
安装 Ruby 的 Redis 模块：
[root@redis1 ruby-2.5.5]# gem install redis
如果无法在线安装 Ruby 的 Redis 模块，可以先下载 Redis 模块（https://rubygems.org/gems/redis），再离线安装：
gem install -l redis-3.3.0.gem
创建 redis-trib.rb 软链接：
[root@redis1 ~]# ln -sv /usr/local/src/redis-4.0.14/src/redis-trib.rb /usr/bin/redis-trib
‘/usr/bin/redis-trib’ -> ‘/usr/local/src/redis-4.0.14/src/redis-trib.rb’
查看 redis-trib.rb 使用帮助：
[root@redis1 ~]# redis-trib
Usage: redis-trib <command> <options> <arguments ...>
create host1:port1 ... hostN:portN # 创建集群，指定集群中的节点；
--replicas <arg> # 指定每个Master的副本数量（即一个Master有几个Slave）；
check host:port # 检查集群状态
info host:port # 查看集群信息
fix host:port # 修复集群
--timeout <arg>
reshard host:port # 重新分片（热迁移slots）；
--from <arg>
--to <arg>
--slots <arg>
--yes
--timeout <arg>
--pipeline <arg>
rebalance host:port # 平衡集群中各节点的slot数量；
--weight <arg>
--auto-weights
--use-empty-masters
--timeout <arg>
--simulate
--pipeline <arg>
--threshold <arg>
add-node new_host:new_port existing_host:existing_port
# 向集群中添加节点；
--slave
--master-id <arg>
del-node host:port node_id # 从集群中删除节点；
set-timeout host:port milliseconds # 设置节点的超时时间（单位毫秒）；
call host:port command arg arg .. arg
# 在集群的所有节点上执行命令；
import host:port # 导入外部redis服务器的数据到当前集群；
--from <arg>
--copy
--replace
help (show this help)
For check, fix, reshard, del-node, set-timeout you can specify the host and port of any working node in the cluster.
为 Ruby Redis 模块配置 Redis 连接密码
[root@redis1 ~]# vim /usr/local/lib/ruby/gems/2.5.0/gems/redis-4.2.5/lib/redis/client.rb
class Redis
class Client
# Defaults are also used for converting string keys to symbols.
DEFAULTS = {
……
password: 123456,
创建 Redis Cluster
创建时，配置在前的节点会成为 Master：
[root@redis1 ~]# redis-trib create --replicas 1 192.168.1.201:6379 192.168.1.203:6379 192.168.1.205:6379 192.168.1.202:6379 192.168.1.204:6379 192.168.1.206:6379
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
192.168.1.201:6379
192.168.1.203:6379
192.168.1.205:6379
Adding replica 192.168.1.204:6379 to 192.168.1.201:6379
Adding replica 192.168.1.206:6379 to 192.168.1.203:6379
Adding replica 192.168.1.202:6379 to 192.168.1.205:6379
M: 469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:6379
slots:0-5460 (5461 slots) master
M: 4dd0b78edebcfe93c93518b5817c00cc21bef07e 192.168.1.203:6379
slots:5461-10922 (5462 slots) master
M: 1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:6379
slots:10923-16383 (5461 slots) master
S: 674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:6379
replicates 1b897eda6acfb806eb27326f151e4067f0d47f7b
S: 6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:6379
replicates 469e0c16495598b5fadba28f856203e206464b99
S: 9c4b8d3a34266ba89cacb80ccf170c34169a0dd9 192.168.1.206:6379
replicates 4dd0b78edebcfe93c93518b5817c00cc21bef07e
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join........
>>> Performing Cluster Check (using node 192.168.1.201:6379)
M: 469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:6379
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: 674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:6379
slots: (0 slots) slave
replicates 1b897eda6acfb806eb27326f151e4067f0d47f7b
M: 1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:6379
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: 6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:6379
slots: (0 slots) slave
replicates 469e0c16495598b5fadba28f856203e206464b99
M: 4dd0b78edebcfe93c93518b5817c00cc21bef07e 192.168.1.203:6379
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 9c4b8d3a34266ba89cacb80ccf170c34169a0dd9 192.168.1.206:6379
slots: (0 slots) slave
replicates 4dd0b78edebcfe93c93518b5817c00cc21bef07e
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
2.4.5：创建集群（Redis-5）
Redis 5 版本可以直接使用 redis-cli 创建 Redis Cluster：
[root@redis1 ~]# redis-cli -a 123456 --cluster create 192.168.1.201:6379 192.168.1.203:6379 192.168.1.205:6379 192.168.1.202:6379 192.168.1.204:6379 192.168.1.206:6379 --cluster-replicas 1
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.168.1.204:6379 to 192.168.1.201:6379
Adding replica 192.168.1.206:6379 to 192.168.1.203:6379
Adding replica 192.168.1.202:6379 to 192.168.1.205:6379
M: feb6b43233fdcc01e0d5425fd03e0116b74f0833 192.168.1.201:6379
slots:[0-5460] (5461 slots) master
M: c2b48542344ece56e69d8ca9404ee29a48ae7b8e 192.168.1.203:6379
slots:[5461-10922] (5462 slots) master
M: df921d013a6cbf1588c1ad51809435fe39f6c25c 192.168.1.205:6379
slots:[10923-16383] (5461 slots) master
S: d7797d3410c1d43b918ee2e7c8cec8810a3fd519 192.168.1.202:6379
replicates df921d013a6cbf1588c1ad51809435fe39f6c25c
S: 90d1475f064839143bbaea911ac1e449bee174da 192.168.1.204:6379
replicates feb6b43233fdcc01e0d5425fd03e0116b74f0833
S: b11dfd69bd23ae0d987a136061d7fd15b500800a 192.168.1.206:6379
replicates c2b48542344ece56e69d8ca9404ee29a48ae7b8e
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 192.168.1.201:6379)
M: feb6b43233fdcc01e0d5425fd03e0116b74f0833 192.168.1.201:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
M: df921d013a6cbf1588c1ad51809435fe39f6c25c 192.168.1.205:6379
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
M: c2b48542344ece56e69d8ca9404ee29a48ae7b8e 192.168.1.203:6379
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: b11dfd69bd23ae0d987a136061d7fd15b500800a 192.168.1.206:6379
slots: (0 slots) slave
replicates c2b48542344ece56e69d8ca9404ee29a48ae7b8e
S: 90d1475f064839143bbaea911ac1e449bee174da 192.168.1.204:6379
slots: (0 slots) slave
replicates feb6b43233fdcc01e0d5425fd03e0116b74f0833
S: d7797d3410c1d43b918ee2e7c8cec8810a3fd519 192.168.1.202:6379
slots: (0 slots) slave
replicates df921d013a6cbf1588c1ad51809435fe39f6c25c
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
redis-cli -a 123456 --cluster create 192.168.7.101:6379
192.168.7.101:6380 192.168.7.102:6379 192.168.7.102:6380 1
92.168.7.103:6379 192.168.7.103:6380 --cluster-replicas 1
2.4.6：验证集群创建
集群中任一节点均可查看集群状态和信息；
CLUSTER 指令
查看集群状态：
127.0.0.1:6379> CLUSTER INFO
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:3
cluster_stats_messages_ping_sent:127
cluster_stats_messages_pong_sent:125
cluster_stats_messages_meet_sent:3
cluster_stats_messages_sent:255
cluster_stats_messages_ping_received:121
cluster_stats_messages_pong_received:130
cluster_stats_messages_meet_received:4
cluster_stats_messages_received:255
查看 Node 对应关系：
127.0.0.1:6379> CLUSTER NODES
4dd0b78edebcfe93c93518b5817c00cc21bef07e 192.168.1.203:6379@16379 master - 0 1609749076000 2 connected 5461-10922
9c4b8d3a34266ba89cacb80ccf170c34169a0dd9 192.168.1.206:6379@16379 myself,slave 4dd0b78edebcfe93c93518b5817c00cc21bef07e 0 1609749076000 6 connected
469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:6379@16379 master - 0 1609749077996 1 connected 0-5460
674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:6379@16379 slave 1b897eda6acfb806eb27326f151e4067f0d47f7b 0 1609749075000 4 connected
6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:6379@16379 slave 469e0c16495598b5fadba28f856203e206464b99 0 1609749076973 5 connected
1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:6379@16379 master - 0 1609749075960 3 connected 10923-16383
redis-trib.rb（Redis-3/4）
指定集群中的任一节点均可查看集群的状态信息；
查看集群信息：
[root@redis1 ~]# redis-trib info 192.168.1.206:6379
192.168.1.203:6379 (4dd0b78e...) -> 0 keys | 5462 slots | 1 slaves.
192.168.1.201:6379 (469e0c16...) -> 0 keys | 5461 slots | 1 slaves.
192.168.1.205:6379 (1b897eda...) -> 0 keys | 5461 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
检查集群状态：
[root@redis1 ~]# redis-trib check 192.168.1.203:6379
>>> Performing Cluster Check (using node 192.168.1.203:6379)
M: 4dd0b78edebcfe93c93518b5817c00cc21bef07e 192.168.1.203:6379
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:6379
slots: (0 slots) slave
replicates 469e0c16495598b5fadba28f856203e206464b99
M: 469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:6379
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: 9c4b8d3a34266ba89cacb80ccf170c34169a0dd9 192.168.1.206:6379
slots: (0 slots) slave
replicates 4dd0b78edebcfe93c93518b5817c00cc21bef07e
M: 1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:6379
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: 674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:6379
slots: (0 slots) slave
replicates 1b897eda6acfb806eb27326f151e4067f0d47f7b
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
redis-cli（Redis-5）
[root@redis1 ~]# redis-cli -a 123456 --cluster check 192.168.1.201:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
192.168.1.201:6379 (feb6b432...) -> 0 keys | 5461 slots | 1 slaves.
192.168.1.205:6379 (df921d01...) -> 0 keys | 5461 slots | 1 slaves.
192.168.1.203:6379 (c2b48542...) -> 0 keys | 5462 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.1.201:6379)
M: feb6b43233fdcc01e0d5425fd03e0116b74f0833 192.168.1.201:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
M: df921d013a6cbf1588c1ad51809435fe39f6c25c 192.168.1.205:6379
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
M: c2b48542344ece56e69d8ca9404ee29a48ae7b8e 192.168.1.203:6379
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: b11dfd69bd23ae0d987a136061d7fd15b500800a 192.168.1.206:6379
slots: (0 slots) slave
replicates c2b48542344ece56e69d8ca9404ee29a48ae7b8e
S: 90d1475f064839143bbaea911ac1e449bee174da 192.168.1.204:6379
slots: (0 slots) slave
replicates feb6b43233fdcc01e0d5425fd03e0116b74f0833
S: d7797d3410c1d43b918ee2e7c8cec8810a3fd519 192.168.1.202:6379
slots: (0 slots) slave
replicates df921d013a6cbf1588c1ad51809435fe39f6c25c
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
查看集群配置文件
Redis-4.0.14：
[root@redis1 ~]# cat /apps/redis/data/nodes-6379.conf
674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:6379@16379 slave 1b897eda6acfb806eb27326f151e4067f0d47f7b 0 1609748891000 4 connected
1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:6379@16379 master - 0 1609748892000 3 connected 10923-16383
6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:6379@16379 slave 469e0c16495598b5fadba28f856203e206464b99 0 1609748892000 5 connected
4dd0b78edebcfe93c93518b5817c00cc21bef07e 192.168.1.203:6379@16379 master - 0 1609748892922 2 connected 5461-10922
9c4b8d3a34266ba89cacb80ccf170c34169a0dd9 192.168.1.206:6379@16379 slave 4dd0b78edebcfe93c93518b5817c00cc21bef07e 0 1609748891000 6 connected
469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:6379@16379 myself,master - 0 1609748890000 1 connected 0-5460
vars currentEpoch 6 lastVoteEpoch 0
Redis-5.0.10：
[root@redis1 ~]# cat /apps/redis5/data/nodes-6379.conf
df921d013a6cbf1588c1ad51809435fe39f6c25c 192.168.1.205:6379@16379 master - 0 1609811160000 3 connected 10923-16383
c2b48542344ece56e69d8ca9404ee29a48ae7b8e 192.168.1.203:6379@16379 master - 0 1609811161000 2 connected 5461-10922
b11dfd69bd23ae0d987a136061d7fd15b500800a 192.168.1.206:6379@16379 slave c2b48542344ece56e69d8ca9404ee29a48ae7b8e 0 1609811162000 6 connected
90d1475f064839143bbaea911ac1e449bee174da 192.168.1.204:6379@16379 slave feb6b43233fdcc01e0d5425fd03e0116b74f0833 0 1609811162884 5 connected
d7797d3410c1d43b918ee2e7c8cec8810a3fd519 192.168.1.202:6379@16379 slave df921d013a6cbf1588c1ad51809435fe39f6c25c 0 1609811162000 4 connected
feb6b43233fdcc01e0d5425fd03e0116b74f0833 192.168.1.201:6379@16379 myself,master - 0 1609811160000 1 connected 0-5460
vars currentEpoch 6 lastVoteEpoch 0
2.4.7：测试集群读写
集群数据写入
Redis1 节点上尝试写入：
提示 foo1 这个 key 的 CRC 结果被调度到了 13431 槽位，该槽位在 192.168.1.205:6379 节点上；
[root@redis1 ~]# redis-cli
127.0.0.1:6379> auth 123456
OK
127.0.0.1:6379> SET foo1 bar1
(error) MOVED 13431 192.168.1.205:6379
Redis5 节点 写入 foo1：
[root@redis5 ~]# redis-cli
127.0.0.1:6379> auth 123456
OK
127.0.0.1:6379> SET foo1 bar1
OK
各节点只保存各自槽位的数据，所以 Redis1 和 Redis3 都没有刚刚写入的 foo1：
[root@redis3 ~]# redis-cli
127.0.0.1:6379> auth 123456
OK
127.0.0.1:6379> KEYS foo1
(empty list or set)
集群数据读取
因为各节点只保存各自槽位的数据，所以数据的读取也只能到相应的节点上进行；
而且集群中的 Slave 节点，读写服务均不提供；
查看 Redis5 的 Slave 节点：
Redis5 的 ID 为：1b897eda6acfb806eb27326f151e4067f0d47f7b；
可以查到对应的 Slave 为 Redis2（192.168.1.202）；
[root@redis1 ~]# cat /apps/redis/data/nodes-6379.conf
674a2a0e6a4c9a88db493c649bbd584bc6ba1edc 192.168.1.202:6379@16379 slave 1b897eda6acfb806eb27326f151e4067f0d47f7b 0 1609748891000 4 connected
1b897eda6acfb806eb27326f151e4067f0d47f7b 192.168.1.205:6379@16379 master - 0 1609748892000 3 connected 10923-16383
6ec84046944b4b82af12d8fbc2977c54fa5f670c 192.168.1.204:6379@16379 slave 469e0c16495598b5fadba28f856203e206464b99 0 1609748892000 5 connected
4dd0b78edebcfe93c93518b5817c00cc21bef07e 192.168.1.203:6379@16379 master - 0 1609748892922 2 connected 5461-10922
9c4b8d3a34266ba89cacb80ccf170c34169a0dd9 192.168.1.206:6379@16379 slave 4dd0b78edebcfe93c93518b5817c00cc21bef07e 0 1609748891000 6 connected
469e0c16495598b5fadba28f856203e206464b99 192.168.1.201:6379@16379 myself,master - 0 1609748890000 1 connected 0-5460
vars currentEpoch 6 lastVoteEpoch 0
到 Redis2 读取刚刚写入到 Redis5 的 foo1：
可以查到有 foo1 的数据，但是读取还是得到 Redis5 上；
127.0.0.1:6379> KEYS foo1
1) "foo1"
127.0.0.1:6379> GET foo1
(error) MOVED 13431 192.168.1.205:6379