Edu/Support

オンライン定期点検
サービス開始

クバネティス(Kubernetes) 環境で Redis-Cluster テスト実戦ガイド ③

Failover(障害対応)テスト

🖧 1. Redis-Cluster ポッド(Pod) ダイアグラム (初期状態)

┌────────────────┐ ┌────────────────┐ ┌───────────────┐ │🟢Redis-0(231) │ │🟢Redis-1(233) │ │🟢Redis-2(235) │ │ Master-1 │ │ Master-2 │ │ Master-3 │ └──────┬─────────┘ └──────┬─────────┘ └──────┬────────┘ │ │ │ │ ┌──────────────────│────────────────────┘ │ │ └────────────────────┐ └──────────────────┐ │ │ │ │ ▼ ▼ ▼ ┌────────────────┐ ┌────────────────┐ ┌───────────────┐ │🟡Redis-3(237) │ │🟡Redis-4(239) │ │🟡Redis-5(241) │ │ Replica-1 │ │ Replica-2 │ │ Replica-3 │ └────────────────┘ └────────────────┘ └───────────────┘

1. Failover(障害対応)テスト

Redis-0 ポッド(Pod)をダウンさせ、Redis-4 が Replica から Master へ転換(昇格)するか確認します
redis.conf の replica-priority 値は Redis-Cluster では使われません。 Redis-Sentinel で使われます。代わりに、最も新しいデータを受け取った複製がマスターになります。ここではマスターごとに複製が1つなので、Redis-0 の複製である Redis-4 がマスターになります。
💥 ポッド(Pod)をダウン後に再起動すると IP が変わります。

Redis-0 💥 ダウン(delete)

💻 Terminal

$ kubectl delete pod redis-cluster-0
  pod "redis-cluster-0" deleted

💻 Redis-0 LOG: Master -> Replica

① 24秒 kill シグナルを受けて Redis ダウン (+0秒)
② 26秒 Redis 自動再起動 (+2秒)
③ 35秒 Master -> Replica(Slave) へ転換 (+11秒)

💻 Terminal

1:M 14:42:24.187 #📍kill シグナルを受けてダウン: Redis is now ready to exit, bye bye...
1:C 14:42:26.706 *📍2秒後に再起動: oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 
1:S 14:42:35.038 *📍Master -> Replica(Slave)へ転換: Before turning into a replica, ...

Redis-0 全ログ ▼

📄 redis.log

$ kubectl exec -it redis-cluster-0 -- tail -f /data/redis.log
1:signal-handler (1786167744) Received SIGTERM scheduling shutdown... 
1:M 14:42:24.182 * User requested shutdown...
1:M 14:42:24.187 #📍kill シグナルを受けてダウン: Redis is now ready to exit, bye bye...
1:C 14:42:26.706 *📍2秒後に再起動: oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 
1:M 14:42:26.708 * Node configuration loaded, I'm 66390566e7f48211e8bed3a6a0e49155cbe8b54b
1:M 14:42:26.709 * Server initialized
1:M 14:42:26.709 * Reading RDB base file on AOF loading...
1:M 14:42:26.709 * Loading RDB produced by version 8.8.1
1:M 14:42:26.709 * RDB age 3615 seconds
1:M 14:42:26.709 * RDB memory usage when created 2.16 Mb
1:M 14:42:26.710 * RDB is base AOF
1:M 14:42:26.710 * Done loading RDB, keys loaded: 0, keys expired: 0.
1:M 14:42:26.710 * DB loaded from base file aof.1.base.rdb: 0.001 seconds
1:M 14:42:35.027 * DB loaded from incr file aof.1.incr.aof: 8.317 seconds
1:M 14:42:35.027 *➡8.3秒かかった: DB loaded from append only file: 8.318 seconds	
1:M 14:42:35.031 * Opening AOF incr file aof.1.incr.aof on server start
1:M 14:42:35.031 * Ready to accept connections tcp
1:M 14:42:35.038 * Configuration change detected. Reconfiguring myself as a replica of ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4)
1:S 14:42:35.038 *📍Master -> Replica(Slave)へ転換: Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
1:S 14:42:35.038 *➡新マスターと接続: Connecting to MASTER 10.42.0.239:18504 
1:S 14:42:35.038 *➡新マスターからのデータ受信を開始: MASTER <-> REPLICA sync started	
1:S 14:42:35.039 * Cluster state changed: ok
1:S 14:42:35.042 * Non blocking connect for SYNC fired the event.
1:S 14:42:35.046 * Master replied to PING, replication can continue...
1:S 14:42:35.046 * Trying a partial resynchronization (request b95c76e4720380a0cdced96e236842d144aeb5b8:918944394).
1:S 14:42:35.047 * PSYNC is not possible, initialize RDB channel.
1:S 14:42:38.638 * Starting to receive RDB and replication stream in parallel.
1:S 14:42:38.676 * MASTER <-> REPLICA sync: receiving streamed RDB from master with EOF to disk
1:S 14:42:42.179 * Discarding previously cached master state.
1:S 14:42:42.179 * MASTER <-> REPLICA sync: Loading DB in memory
1:S 14:42:42.188 * MASTER <-> REPLICA sync: Flushing old data
1:S 14:42:43.300 * Loading RDB produced by version 8.8.1
1:S 14:42:43.300 * RDB age 5 seconds
1:S 14:42:43.300 * RDB memory usage when created 650.33 Mb
1:S 14:42:45.600 * Done loading RDB, keys loaded: 3053053, keys expired: 0.
1:S 14:42:45.600 * Removing the RDB file obtained from the master. This replica has persistence disabled
1:S 14:42:45.600 *➡新マスターからのデータ受信・ロード完了: MASTER <-> REPLICA sync: Finished with success 
1:S 14:42:45.600 *➡AOF rewrite 開始: MASTER <-> REPLICA sync: Starting AOF after a successful sync  
1:S 14:42:45.600 * Creating AOF incr file temp-aof.incr on background rewrite
1:S 14:42:45.613 * Background append only file rewriting started by pid 19
1:S 14:42:45.613 * MASTER <-> REPLICA sync: Starting to stream replication buffer into the db (0 bytes).
1:S 14:42:45.613 * MASTER <-> REPLICA sync: Successfully streamed replication buffer into the db (0 bytes in total)
19:C 14:42:48.921 * BGSAVE done, 3053053 keys saved, 0 keys skipped, 384717496 bytes written.
19:C 14:42:48.927 * Successfully created the temporary AOF base file temp-rewriteaof-bg-19.aof
19:C 14:42:48.933 * Fork CoW for AOF rewrite: current 2 MB, peak 2 MB, average 1 MB
1:S 14:42:49.027 * Background AOF rewrite terminated with success
1:S 14:42:49.027 * Successfully renamed the temporary AOF base file temp-rewriteaof-bg-19.aof into aof.2.base.rdb
1:S 14:42:49.027 * Successfully renamed the temporary AOF incr file temp-aof.incr into aof.2.incr.aof
1:S 14:42:49.031 * Removing the history file aof.1.incr.aof in the background
1:S 14:42:49.031 * Removing the history file aof.1.base.rdb in the background
1:S 14:42:49.036 *➡AOF rewrite 完了: Background AOF rewrite finished successfully

💻 Redis-1 LOG: Master

① 28秒 Redis-0 ダウン確定(Fail/odown) (+4秒) -> cluster-node-timeout 3000ms
② 29秒 Redis-4 が新マスターへ昇格 (+5秒)

📄 redis.log

$ kubectl exec -it redis-cluster-1 -- tail -f /data/redis.log
1:M 14:42:28.507 *📍Redis-0 ダウン確定: Marking node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0) as failing (quorum reached).
1:M 14:42:29.608 *📍Redis-4 が新マスターへ昇格: Failover auth granted to ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4) for epoch 7	
1:M 14:42:35.033 *➡Redis-0 ポッド再起動 ➔ 新IP(10.42.0.242)を検知: Address updated for node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0), now 10.42.0.242:18504
1:M 14:42:35.078 *➡Redis-0 がレプリカノードとして再接続され FAIL 状態解除: Clear FAIL state for node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0):replica is reachable again.

💻 Redis-4 LOG: Replica -> Master

① 24秒 Redis-0 ダウン検知 (+0秒)
② 29秒マスターへ昇格 (+5秒)

📄 redis.log

$ kubectl exec -it redis-cluster-4 -- tail -f /data/redis.log
1:S 14:42:24.235 *📍Redis-0 ダウン検知: Connection with master lost.	 
1:M 14:42:29.613 *📍マスターへ昇格: Discarding previously cached master state.

Redis-4 全ログ ▼

📄 redis.log

$ kubectl exec -it redis-cluster-4 -- tail -f /data/redis.log
1:S 14:42:24.235 *📍Redis-0 ダウン検知: Connection with master lost.	 
1:S 14:42:24.235 * Caching the disconnected master state.
1:S 14:42:24.235 *➡接続試行 Reconnecting to MASTER 10.42.0.231:18504 
1:S 14:42:24.235 * MASTER <-> REPLICA sync started
1:S 14:42:24.235 #➡接続失敗 Error condition on socket for SYNC: Connection refused	
1:S 14:42:24.579 * Connecting to MASTER 10.42.0.231:18504
1:S 14:42:24.579 * MASTER <-> REPLICA sync started
1:S 14:42:28.507 * FAIL message received from 3b690009fb9a90ff6b43765363c4c0b81b63d103 (redis-cluster-1) about 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0)
1:S 14:42:28.598 * Start of election delayed for 972 milliseconds (rank #0, offset 918944393).
1:S 14:42:29.602 * Starting a failover election for epoch 7.
1:S 14:42:29.613 * Failover election won: I'm the new master.
1:S 14:42:29.613 * configEpoch set to 7 after successful failover					
1:M 14:42:29.613 *📍マスターへ昇格: Discarding previously cached master state.	
1:M 14:42:29.613 * Setting secondary replication ID to c76d2514db200f6d13ba070d8ef349efe33bc2cf, valid up to offset: 918944394. New replication ID is caab310287582352de716fc92f8542ef435b125d
1:M 14:42:35.033 * Address updated for node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0), now 10.42.0.242:18504
1:M 14:42:35.042 * A failover occurred in shard 14464984890143edab6e29a8503e3c93981c8809; node 66390566e7f48211e8bed3a6a0e49155cbe8b54b () lost 0 slot(s) to node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4) with a config epoch of 7
1:M 14:42:35.047 *➡旧マスター(現在はレプリカ)がデータ同期を要求: Replica 10.42.0.242:18504 asks for synchronization 
1:M 14:42:35.047 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for 'b95c76e4720380a0cdced96e236842d144aeb5b8', my replication IDs are 'caab310287582352de716fc92f8542ef435b125d' and 'c76d2514db200f6d13ba070d8ef349efe33bc2cf')
1:M 14:42:35.047 * Replica 10.42.0.242:18504 is capable of rdb channel synchronization, and partial sync isn't possible. Full sync will continue with dedicated rdb channel.
1:M 14:42:35.047 * Replica 10.42.0.242:18504 asks for synchronization
1:M 14:42:35.047 * Full resync requested by replica 10.42.0.242:18504 (rdb-channel)
1:M 14:42:35.047 * Delay next BGSAVE for diskless SYNC
1:M 14:42:35.124 * Clear FAIL state for node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0):replica is reachable again.
1:M 14:42:38.638 *➡送信データの作成開始: Starting BGSAVE for SYNC with target: replicas sockets (rdb-channel)
1:M 14:42:38.638 * Starting to deliver RDB and replication stream to replica: 10.42.0.242:18504
1:M 14:42:38.652 * Background RDB transfer started by pid 20 to replica socket
20:C 14:42:42.174 * BGSAVE done, 3053053 keys saved, 0 keys skipped, 384717629 bytes written.
20:C 14:42:42.183 * Fork CoW for RDB: current 2 MB, peak 2 MB, average 1 MB
1:M 14:42:42.267 * Background RDB transfer terminated with success
1:M 14:42:42.267 *➡新レプリカへのデータ送信(同期)完了: Synchronization with replica 10.42.0.242:18504 succeeded 
1:M 14:42:42.267 * Streamed RDB transfer with replica 10.42.0.242:18504 succeeded (socket). Waiting for REPLCONF ACK from replica to enable streaming
1:M 14:42:42.267 * RDB transfer completed, rdb only replica (10.42.0.242:18504) should be disconnected asap
1:M 14:42:42.267 * Connection with replica (rdbchannel) 10.42.0.242:18504 lost.

🧮 Failover(障害対応)後の状況

Redis-0 の IP が '10.42.0.231' から '10.42.0.242' に変わりました。

💻 Terminal

[redis@Rocky9 kube]$ k get pods -o wide
NAME              READY   STATUS    RESTARTS   AGE    IP            NODE  
redis-cluster-0   1/1     Running   0          3m3s   10.42.0.242   rocky9	IP:231->242
redis-cluster-1   1/1     Running   0          63m    10.42.0.233   rocky9
redis-cluster-2   1/1     Running   0          63m    10.42.0.235   rocky9
redis-cluster-3   1/1     Running   0          63m    10.42.0.237   rocky9
redis-cluster-4   1/1     Running   0          63m    10.42.0.239   rocky9
redis-cluster-5   1/1     Running   0          63m    10.42.0.241   rocky9

🖧 Redis-Cluster ポッド(Pod) ダイアグラム (Failover 後)

┌────────────────┐ ┌────────────────┐ ┌───────────────┐ │🟡Redis-0(242) │ │🟢Redis-1(233) │ │🟢Redis-2(235) │ │ Replica-2 │ │ Master-2 │ │ Master-3 │ └──────┬─────────┘ └──────┬─────────┘ └──────┬────────┘ ▲ │ │ │ ┌──────────────────│────────────────────┘ │ │ └────────────────────┐ └──────────────────┐ │ │ │ │ ▼ │ ▼ ┌────────────────┐ ┌────────────────┐ ┌───────────────┐ │🟡Redis-3(237) │ │🟢Redis-4(239) │ │🟡Redis-5(241) │ │ Replica-1 │ │ Master-1 │ │ Replica-3 │ └────────────────┘ └────────────────┘ └───────────────┘

2. Failback(原状回復)テスト

現在のマスターである Redis-4 をダウンさせ、Redis-0 が Replica から Master へ転換(昇格)するか確認します
📚 'Failback(原状回復)テスト' は、Redis-0 ポッド(Pod)が再びマスターになるという意味で使いました。
(Failover テストと区別するためです)

💻 Redis-4 💥ダウン(delete)

💻 Terminal

$ kubectl delete pod redis-cluster-4
  pod "redis-cluster-4" deleted

💻 Redis-0 LOG: Replica -> Master

① 52秒 Redis-4 接続切断 (+0秒)
② 56秒 Redis-4 ダウン確定(Fail/odown) (+4秒)
③ 57秒 Replica -> Master へ転換(昇格) (+5秒)

📄 redis.log

1:S 15:19:52.289 *📍Redis-4 接続切断: Connection with master lost.	
1:S 15:19:56.549 *📍Redis-4 ダウン確定: Marking node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4) as failing (quorum reached). 
1:M 15:19:57.276 *📍Replica -> Masterへ転換(昇格): Discarding previously cached master state.

Redis-0 全ログ ▼

📄 redis.log

1:S 15:19:52.289 *📍Redis-4 接続切断: Connection with master lost.	
1:S 15:19:52.289 * Caching the disconnected master state.
1:S 15:19:52.289 *➡Redis-4へ接続試行: Reconnecting to MASTER 10.42.0.239:18504	
1:S 15:19:52.289 * MASTER <-> REPLICA sync started
1:S 15:19:52.289 # Error condition on socket for SYNC: Connection refused
1:S 15:19:52.644 * Connecting to MASTER 10.42.0.239:18504
1:S 15:19:52.644 * MASTER <-> REPLICA sync started
1:S 15:19:56.549 *📍Redis-4 ダウン確定(odown): Marking node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4) as failing (quorum reached).
1:S 15:19:56.560 * Start of election delayed for 639 milliseconds (rank #0, offset 918947529).
1:S 15:19:57.267 * Starting a failover election for epoch 8.
1:S 15:19:57.276 * Failover election won: I'm the new master.
1:S 15:19:57.276 * configEpoch set to 8 after successful failover                              
1:M 15:19:57.276 *📍Replica(Slave) -> Masterへ転換(昇格): Discarding previously cached master state.
1:M 15:19:57.276 * Setting secondary replication ID to caab310287582352de716fc92f8542ef435b125d, valid up to offset: 918947530. New replication ID is 2e46928d6c500b1c71c1038b815e61b9c059cc82
1:M 15:20:02.615 * Address updated for node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4), now 10.42.0.243:18504
1:M 15:20:02.632 * A failover occurred in shard 14464984890143edab6e29a8503e3c93981c8809; node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 () lost 0 slot(s) to node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0) with a config epoch of 8
1:M 15:20:02.637 * Replica 10.42.0.243:18504 asks for synchronization
1:M 15:20:02.637 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for 'f0e297826ed9c693bb4b5abccd5c415d8319ac10', my replication IDs are '2e46928d6c500b1c71c1038b815e61b9c059cc82' and 'caab310287582352de716fc92f8542ef435b125d')
1:M 15:20:02.637 * Replica 10.42.0.243:18504 is capable of rdb channel synchronization, and partial sync isn't possible. Full sync will continue with dedicated rdb channel.
1:M 15:20:02.638 * Replica 10.42.0.243:18504 asks for synchronization
1:M 15:20:02.638 * Full resync requested by replica 10.42.0.243:18504 (rdb-channel)
1:M 15:20:02.638 * Delay next BGSAVE for diskless SYNC
1:M 15:20:02.689 *➡Clear FAIL state for node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4):replica is reachable again.
1:M 15:20:05.701 * Starting BGSAVE for SYNC with target: replicas sockets (rdb-channel)
1:M 15:20:05.701 * Starting to deliver RDB and replication stream to replica: 10.42.0.243:18504
1:M 15:20:05.708 * Background RDB transfer started by pid 28 to replica socket
28:C 15:20:08.931 * BGSAVE done, 3053053 keys saved, 0 keys skipped, 384717629 bytes written.
28:C 15:20:08.938 * Fork CoW for RDB: current 2 MB, peak 2 MB, average 1 MB
1:M 15:20:09.019 * Background RDB transfer terminated with success
1:M 15:20:09.019 * Synchronization with replica 10.42.0.243:18504 succeeded
1:M 15:20:09.019 * Streamed RDB transfer with replica 10.42.0.243:18504 succeeded (socket). Waiting for REPLCONF ACK from replica to enable streaming
1:M 15:20:09.019 * RDB transfer completed, rdb only replica (10.42.0.243:18504) should be disconnected asap
1:M 15:20:09.020 * Connection with replica (rdbchannel) 10.42.0.243:18504 lost.

💻 Redis-1 LOG: Master

① 56秒 Redis-4 ダウン確定 (+4秒)
② 57秒 Redis-0 が新マスターへ昇格 (+5秒)

📄 redis.log

1:M 15:19:56.549 *📍Redis-4 ダウン確定: FAIL message received from 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0) about ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4)
1:M 15:19:57.271 *📍Redis-0 が新マスターへ昇格: Failover auth granted to 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0) for epoch 8
1:M 15:20:02.615 *➡Redis-4 ポッド再起動 -> 新IP(10.42.0.243)を検知: Address updated for node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4), now 10.42.0.243:18504
1:M 15:20:02.692 *➡Redis-4 がレプリカノードとして再接続され FAIL 状態解除: Clear FAIL state for node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4):replica is reachable again.

💻 Redis-4 LOG: Master -> Replica

① 19分52秒 kill シグナルを受けてダウン (+0秒)
② 19分54秒ポッド(Pod)(Redis) 再起動 (+2秒)
③ 20分02秒 Master -> Replica(Slave) へ転換 (+10秒)

📄 redis.log

1:M 15:19:52.237 #📍kill シグナルを受けてダウン: Redis is now ready to exit, bye bye...
1:M 15:19:54.305 *📍ポッド(Redis)再起動: Server initialized
1:S 15:20:02.629 *📍Master -> Replica(Slave)へ転換: Before turning into a replica, ...

Redis-4 全ログ ▼

📄 redis.log

1:signal-handler (1786169992) Received SIGTERM scheduling shutdown...
1:M 15:19:52.232 * User requested shutdown...
1:M 15:19:52.237 #📍kill シグナルを受けてダウン: Redis is now ready to exit, bye bye...
1:M 15:19:54.305 *📍ポッド(Redis)再起動: Server initialized
1:M 15:19:54.306 * Reading RDB base file on AOF loading...
1:M 15:19:54.306 * Loading RDB produced by version 8.8.1
1:M 15:19:54.306 * RDB age 5681 seconds
1:M 15:19:54.306 * RDB memory usage when created 2.39 Mb
1:M 15:19:54.306 * RDB is base AOF
1:M 15:19:54.306 * Done loading RDB, keys loaded: 0, keys expired: 0.
1:M 15:19:54.306 * DB loaded from base file aof.2.base.rdb: 0.001 seconds
1:M 15:20:02.608 * DB loaded from incr file aof.2.incr.aof: 8.302 seconds
1:M 15:20:02.609 *➡Data Load 8.3秒かかった: DB loaded from append only file: 8.303 seconds 
1:M 15:20:02.613 * Opening AOF incr file aof.2.incr.aof on server start
1:M 15:20:02.613 * Ready to accept connections tcp
1:M 15:20:02.629 * Configuration change detected. Reconfiguring myself as a replica of 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0)
1:S 15:20:02.629 *📍Master -> Replica(Slave)へ転換: Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
1:S 15:20:02.629 *➡新マスターに接続: Connecting to MASTER 10.42.0.242:18504	
1:S 15:20:02.629 * MASTER <-> REPLICA sync started
1:S 15:20:02.629 * Cluster state changed: ok
1:S 15:20:02.632 * Non blocking connect for SYNC fired the event.
1:S 15:20:02.637 * Master replied to PING, replication can continue...
1:S 15:20:02.637 * Trying a partial resynchronization (request f0e297826ed9c693bb4b5abccd5c415d8319ac10:918947530).
1:S 15:20:02.637 * PSYNC is not possible, initialize RDB channel.
1:S 15:20:05.701 * Starting to receive RDB and replication stream in parallel.
1:S 15:20:05.716 * MASTER <-> REPLICA sync: receiving streamed RDB from master with EOF to disk
1:S 15:20:08.938 * Discarding previously cached master state.
1:S 15:20:08.938 * MASTER <-> REPLICA sync: Loading DB in memory
1:S 15:20:08.947 * MASTER <-> REPLICA sync: Flushing old data
1:S 15:20:10.042 * Loading RDB produced by version 8.8.1
1:S 15:20:10.042 * RDB age 5 seconds
1:S 15:20:10.042 * RDB memory usage when created 522.06 Mb
1:S 15:20:12.351 * Done loading RDB, keys loaded: 3053053, keys expired: 0.
1:S 15:20:12.351 * Removing the RDB file obtained from the master. This replica has persistence disabled
1:S 15:20:12.351 *➡同期完了: MASTER <-> REPLICA sync: Finished with success  
1:S 15:20:12.351 * MASTER <-> REPLICA sync: Starting AOF after a successful sync
1:S 15:20:12.351 * Creating AOF incr file temp-aof.incr on background rewrite
1:S 15:20:12.370 * Background append only file rewriting started by pid 19
1:S 15:20:12.370 * MASTER <-> REPLICA sync: Starting to stream replication buffer into the db (0 bytes).
1:S 15:20:12.370 * MASTER <-> REPLICA sync: Successfully streamed replication buffer into the db (0 bytes in total)
19:C 15:20:15.668 * BGSAVE done, 3053053 keys saved, 0 keys skipped, 384717496 bytes written.
19:C 15:20:15.673 * Successfully created the temporary AOF base file temp-rewriteaof-bg-19.aof
19:C 15:20:15.682 * Fork CoW for AOF rewrite: current 2 MB, peak 2 MB, average 1 MB
1:S 15:20:15.790 * Background AOF rewrite terminated with success
1:S 15:20:15.790 * Successfully renamed the temporary AOF base file temp-rewriteaof-bg-19.aof into aof.3.base.rdb
1:S 15:20:15.790 * Successfully renamed the temporary AOF incr file temp-aof.incr into aof.3.incr.aof
1:S 15:20:15.795 * Removing the history file aof.2.incr.aof in the background
1:S 15:20:15.795 * Removing the history file aof.2.base.rdb in the background
1:S 15:20:15.798 *➡AOF Rewrite 完了: Background AOF rewrite finished successfully

🧮 Failback(原状回復)後の状況

Redis-4 の IP が '10.42.0.239' から '10.42.0.243' に変わりました。

💻 Terminal

$ k get pods -o wide
NAME              READY   STATUS    RESTARTS   AGE     IP            NODE  
redis-cluster-0   1/1     Running   0          47m     10.42.0.242   rocky9	📍IP:231->242
redis-cluster-1   1/1     Running   0          107m    10.42.0.233   rocky9
redis-cluster-2   1/1     Running   0          107m    10.42.0.235   rocky9
redis-cluster-3   1/1     Running   0          107m    10.42.0.237   rocky9
redis-cluster-4   1/1     Running   0          9m34s   10.42.0.243   rocky9 📍IP:239->243
redis-cluster-5   1/1     Running   0          106m    10.42.0.241   rocky9

🧮 Cluster info 確認

💻 Terminal

$ ./cluster_info.sh
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384	➡📍すべてのスロット OK
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3

🧮 Cluster nodes 確認

💻 Terminal

$ ./cluster_nodes.sh
66390566e7f48211e8bed3a6a0e49155cbe8b54b 10.42.0.242:18504@28504,redis-cluster-0.redis-cluster-service.default.svc.cluster.local master - 0 1786170813036 8 connected 0-5460
3b690009fb9a90ff6b43765363c4c0b81b63d103 10.42.0.233:18504@28504,redis-cluster-1.redis-cluster-service.default.svc.cluster.local myself,master - 0 0 2 connected 5461-10922
02069ae6b4f98c31d5f4e70f243bc9a006b68443 10.42.0.235:18504@28504,redis-cluster-2.redis-cluster-service.default.svc.cluster.local master - 0 1786170813539 3 connected 10923-16383
3722046802e006615ac1d976e62e4c94c58af586 10.42.0.237:18504@28504,redis-cluster-3.redis-cluster-service.default.svc.cluster.local slave 02069ae6b4f98c31d5f4e70f243bc9a006b68443 0 1786170813237 3 connected
ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 10.42.0.243:18504@28504,redis-cluster-4.redis-cluster-service.default.svc.cluster.local slave 66390566e7f48211e8bed3a6a0e49155cbe8b54b 0 1786170814041 8 connected
541fa7cc95095c4b0390bb8ae9d787aa1201c9e2 10.42.0.241:18504@28504,redis-cluster-5.redis-cluster-service.default.svc.cluster.local slave 3b690009fb9a90ff6b43765363c4c0b81b63d103 0 1786170813539 2 connected

🖧 Redis-Cluster ポッド(Pod) ダイアグラム (Failback 後)

┌────────────────┐ ┌────────────────┐ ┌───────────────┐ │🟢Redis-0(242) │ │🟢Redis-1(233) │ │🟢Redis-2(235) │ │ Master-1 │ │ Master-2 │ │ Master-3 │ └──────┬─────────┘ └──────┬─────────┘ └──────┬────────┘ │ │ │ │ ┌──────────────────│────────────────────┘ │ │ └────────────────────┐ └──────────────────┐ │ │ │ │ ▼ ▼ ▼ ┌────────────────┐ ┌────────────────┐ ┌───────────────┐ │🟡Redis-3(237) │ │🟡Redis-4(243) │ │🟡Redis-5(241) │ │ Replica-1 │ │ Replica-2 │ │ Replica-3 │ └────────────────┘ └────────────────┘ └───────────────┘

🧑‍💻 結論

Redis-Cluster Failover(障害対応)、Failback(原状回復)テスト成功
ポッド(Pod)をダウン、再起動すると IP は変わりますが、問題なく処理されます。

Failover(障害対応)テスト: Redis-0 SLEEP(停止)

SLEEP テストは、Redis が何らかの理由で応答できない状況を作り、この場合も正しく Failover(障害対応)されるかを確認するテストです。
💥 'Sleep 10' は Redis を10秒間止めるので、IP は変わりません。

1. Failover(障害対応)テスト

• Failover(障害対応)テスト: Redis-0 Sleep 10 -> Redis-4 が Replica から Master へ転換(昇格)するか確認

Redis-0 SLEEP 10

💻 Terminal

$ kubectl exec -it redis-cluster-0 -- redis-cli -p 18504 -a redisgate DEBUG sleep 10

💻 Redis-0 LOG: Master -> Replica

① 17秒 SLEEP 開始 (+0秒)
② 27秒 SLEEP 終了 (+10秒)
③ 27秒 Master -> Replica へ転換 (+10秒)

📄 redis.log

1:M 17:08:27.199 *📍SLEEP 終了: Connection with replica 10.42.0.243:18504 lost.
1:S 17:08:27.200 *📍Master -> Replicaへ転換: Before turning into a replica, ...

Redis-0 全ログ ▼

📄 redis.log

1:M 17:08:27.199 *📍SLEEP 終了: Connection with replica 10.42.0.243:18504 lost.
1:M 17:08:27.200 * Configuration change detected. Reconfiguring myself as a replica of ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4)
1:S 17:08:27.200 *📍 Master -> Replicaへ転換: Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
1:S 17:08:27.200 * Connecting to MASTER 10.42.0.243:18504
1:S 17:08:27.200 * MASTER <-> REPLICA sync started
1:S 17:08:27.205 * Non blocking connect for SYNC fired the event.
1:S 17:08:27.210 * Master replied to PING, replication can continue...
1:S 17:08:27.211 * Trying a partial resynchronization (request 2e46928d6c500b1c71c1038b815e61b9c059cc82:918956630).
1:S 17:08:27.211 * Successful partial resynchronization with master.
1:S 17:08:27.211 * Master replication ID changed to 8d3577deeb6b8113718c642303195a34c4b78d09
1:S 17:08:27.211 *➡部分同期完了: MASTER <-> REPLICA sync: Master accepted a Partial Resynchronization.

💻 Redis-1 LOG: Master

① 21秒 Redis-0 ダウン確定(Fail/odown) (+4秒) -> cluster-node-timeout 3000ms
② 22秒 Redis-4 Replica -> Master へ転換 (+5秒)
③ 27秒 Redis-0 再起動 (+10秒)

📄 redis.log

💻 Redis-4 LOG: Replica -> Master

① 21秒 Redis-0 ダウン確定(Fail/odown) (+4秒)
② 22秒 Replica -> Master へ転換 (+5秒)

📄 redis.log

1:S 17:08:21.073 *📍Redis-0 ダウン確定: Marking node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0) as failing (quorum reached).
1:S 17:08:22.144 *📍Replica(Slave) -> Masterへ転換: Failover election won: I'm the new master.

Redis-4 ログ全体 ▼

📄 redis.log

1:S 17:08:21.073 *📍Redis-0 ダウン確定: Marking node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0) as failing (quorum reached).
1:S 17:08:21.131 * Start of election delayed for 944 milliseconds (rank #0, offset 918956629).
1:S 17:08:22.134 * Starting a failover election for epoch 9.
1:S 17:08:22.144 *📍Replica(Slave) -> Masterへ転換: Failover election won: I'm the new master.
1:S 17:08:22.144 * configEpoch set to 9 after successful failover                               
1:M 17:08:22.144 * Connection with master lost.
1:M 17:08:22.144 * Caching the disconnected master state.
1:M 17:08:22.144 * Discarding previously cached master state.
1:M 17:08:22.144 * Setting secondary replication ID to 2e46928d6c500b1c71c1038b815e61b9c059cc82, valid up to offset: 918956630. New replication ID is 8d3577deeb6b8113718c642303195a34c4b78d09
1:M 17:08:27.205 * Clear FAIL state for node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0):master without slots is reachable again.
1:M 17:08:27.205 * A failover occurred in shard 14464984890143edab6e29a8503e3c93981c8809; node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0) lost 0 slot(s) to node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4) with a config epoch of 9
1:M 17:08:27.211 * Replica 10.42.0.242:18504 asks for synchronization
1:M 17:08:27.211 * Partial resynchronization request from 10.42.0.242:18504 accepted. Sending 0 bytes of backlog starting from offset 918956630.

🧮 Failover(障害対応)後の状況

IP は変わりません。ポッド(Pod)自体は生き続けています。'AGE' 項目を参照してください。

💻 Terminal

$ k get pods -o wide
NAME              READY   STATUS    RESTARTS   AGE     IP            NODE  
redis-cluster-0   1/1     Running   0          146m    10.42.0.242   rocky9
redis-cluster-1   1/1     Running   0          3h26m   10.42.0.233   rocky9
redis-cluster-2   1/1     Running   0          3h26m   10.42.0.235   rocky9
redis-cluster-3   1/1     Running   0          3h26m   10.42.0.237   rocky9
redis-cluster-4   1/1     Running   0          108m    10.42.0.243   rocky9
redis-cluster-5   1/1     Running   0          3h26m   10.42.0.241   rocky9

🧮 Cluster info 照会

💻 Terminal

cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3

🧮 Cluster nodes 照会

💻 Terminal

66390566e7f48211e8bed3a6a0e49155cbe8b54b 10.42.0.242:18504@28504,redis-cluster-0.redis-cluster-service.default.svc.cluster.local 🟡slave ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 0 1786176794190 9 connected
3b690009fb9a90ff6b43765363c4c0b81b63d103 10.42.0.233:18504@28504,redis-cluster-1.redis-cluster-service.default.svc.cluster.local myself,🟢master - 0 0 2 connected 5461-10922
02069ae6b4f98c31d5f4e70f243bc9a006b68443 10.42.0.235:18504@28504,redis-cluster-2.redis-cluster-service.default.svc.cluster.local 🟢master - 0 1786176794089 3 connected 10923-16383
3722046802e006615ac1d976e62e4c94c58af586 10.42.0.237:18504@28504,redis-cluster-3.redis-cluster-service.default.svc.cluster.local 🟡slave 02069ae6b4f98c31d5f4e70f243bc9a006b68443 0 1786176793184 3 connected
ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 10.42.0.243:18504@28504,redis-cluster-4.redis-cluster-service.default.svc.cluster.local 🟢master - 0 1786176794000 9 connected 0-5460
541fa7cc95095c4b0390bb8ae9d787aa1201c9e2 10.42.0.241:18504@28504,redis-cluster-5.redis-cluster-service.default.svc.cluster.local 🟡slave 3b690009fb9a90ff6b43765363c4c0b81b63d103 0 1786176793586 2 connected

🖧 Redis-Cluster ポッド(Pod) ダイアグラム (Failover 後)

┌────────────────┐ ┌────────────────┐ ┌───────────────┐ │🟡Redis-0(242) │ │🟢Redis-1(233) │ │🟢Redis-2(235) │ │ Replica-2 │ │ Master-2 │ │ Master-3 │ └────────────────┘ └──────┬─────────┘ └──────┬────────┘ ▲ │ │ │ ┌──────────────────│────────────────────┘ │ │ └────────────────────┐ └──────────────────┐ │ │ │ │ ▼ │ ▼ ┌────────────────┐ ┌────┴───────────┐ ┌───────────────┐ │🟡Redis-3(237) │ │🟢Redis-4(243) │ │🟡Redis-5(241) │ │ Replica-1 │ │ Master-1 │ │ Replica-3 │ └────────────────┘ └────────────────┘ └───────────────┘

2. Failback(原状回復)テスト

• Failback(原状回復)テスト: Redis-4 Sleep 10 -> Redis-0 が Replica から Master へ転換(昇格)するか確認

💻 Redis-4 SLEEP 10

💻 Terminal

$ kubectl exec -it redis-cluster-4 -- redis-cli -p 18504 -a redisgate DEBUG sleep 10

💻 Redis-0 LOG: Replica -> Master

① 26秒 SLEEP コマンド実行 (+0秒)
② 30秒 Redis-4 ダウン確定(Fail/odown) (+4秒)
③ 31秒 Redis-0 Replica -> Master へ転換 (+5秒)
④ 36秒 Redis-4 再起動 (+10秒)

📄 redis.log

1:S 20:19:30.481 *📍Redis-4 ダウン確定: FAIL message received from 3b690009fb9a90ff6b43765363c4c0b81b63d103 (redis-cluster-1) about ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4)
1:S 20:19:31.588 *📍Replica(Slave) -> Masterへ転換: Failover election won: I'm the new master.
1:M 20:19:36.349 *📍Redis-4 再起動: Clear FAIL state for node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4):master without slots is reachable again.

Redis-0 全ログ ▼

📄 redis.log

1:S 20:19:30.481 *📍Redis-4 ダウン確定: FAIL message received from 3b690009fb9a90ff6b43765363c4c0b81b63d103 (redis-cluster-1) about ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4)
1:S 20:19:30.568 * Start of election delayed for 971 milliseconds (rank #0, offset 918972673).
1:S 20:19:31.573 * Starting a failover election for epoch 10.
1:S 20:19:31.588 *📍Replica(Slave) -> Masterへ転換: Failover election won: I'm the new master.
1:S 20:19:31.588 * configEpoch set to 10 after successful failover
1:M 20:19:31.588 * Connection with master lost.
1:M 20:19:31.588 * Caching the disconnected master state.
1:M 20:19:31.588 * Discarding previously cached master state.
1:M 20:19:31.588 * Setting secondary replication ID to 8d3577deeb6b8113718c642303195a34c4b78d09, valid up to offset: 918972674. New replication ID is 806c6073fffb5fe2ac6d318f6aceb9d2aa090c18
1:M 20:19:36.349 *📍Redis-4 再起動: Clear FAIL state for node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4):master without slots is reachable again.
1:M 20:19:36.349 * A failover occurred in shard 14464984890143edab6e29a8503e3c93981c8809; node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4) lost 0 slot(s) to node 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0) with a config epoch of 10
1:M 20:19:36.354 * Replica 10.42.0.243:18504 asks for synchronization
1:M 20:19:36.354 * Partial resynchronization request from 10.42.0.243:18504 accepted. Sending 0 bytes of backlog starting from offset 918972674.

💻 Redis-1 LOG: Master

① 30秒 Redis-4 ダウン確定(Fail/odown) (+4秒)
② 31秒 Redis-0 Replica -> Master へ転換(昇格) (+5秒)
③ 36秒 Redis-4 再起動 (+10秒)

📄 redis.log

1:M 20:19:30.481 *📍Redis-4 ダウン(FAIL)確定: Marking node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4) as failing (quorum reached).
1:M 20:19:31.577 *📍Replica -> Masterへ転換(昇格): Failover auth granted to 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0) for epoch 10
1:M 20:19:36.349 *📍Clear FAIL state for node ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 (redis-cluster-4):master without slots is reachable again.

💻 Redis-2 LOG: Redis-1 ログと同じため省略

💻 Redis-4 LOG: Master -> Replica

① 26秒 SLEEP コマンド実行 (+0秒)
① 36秒 SLEEP コマンド終了 -> 複製(Redis-0)との接続切断 (+10秒)
② 36秒 Master -> Replica へ転換 (+10秒)

📄 redis.log

1:M 20:19:36.342 *📍レプリカ(Redis-0)との接続切断: Connection with replica 10.42.0.242:18504 lost.
1:S 20:19:36.343 *📍Master -> Replica(Slave)へ転換: Before turning into a replica, ...

Redis-4 ログ全体 ▼

📄 redis.log

1:M 20:19:36.342 *📍レプリカ(Redis-0)との接続切断: Connection with replica 10.42.0.242:18504 lost.
1:M 20:19:36.342 # Failover auth denied to 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0): its master is up
1:M 20:19:36.343 * Configuration change detected. Reconfiguring myself as a replica of 66390566e7f48211e8bed3a6a0e49155cbe8b54b (redis-cluster-0)
1:S 20:19:36.343 *📍Master -> Replica(Slave)へ転換: Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
1:S 20:19:36.343 * Connecting to MASTER 10.42.0.242:18504
1:S 20:19:36.343 * MASTER <-> REPLICA sync started
1:S 20:19:36.349 * Non blocking connect for SYNC fired the event.
1:S 20:19:36.354 * Master replied to PING, replication can continue...
1:S 20:19:36.354 * Trying a partial resynchronization (request 8d3577deeb6b8113718c642303195a34c4b78d09:918972674).
1:S 20:19:36.354 * Successful partial resynchronization with master.
1:S 20:19:36.354 * Master replication ID changed to 806c6073fffb5fe2ac6d318f6aceb9d2aa090c18
1:S 20:19:36.354 * MASTER <-> REPLICA sync: Master accepted a Partial Resynchronization.

🧮 Failback(原状回復)後の状況

💻 Terminal

$ k get pods -o wide	-> IP変更なし、再起動しない
NAME              READY   STATUS    RESTARTS   AGE     IP            NODE  
redis-cluster-0   1/1     Running   0          5h37m   10.42.0.242   rocky9
redis-cluster-1   1/1     Running   0          6h38m   10.42.0.233   rocky9
redis-cluster-2   1/1     Running   0          6h37m   10.42.0.235   rocky9
redis-cluster-3   1/1     Running   0          6h37m   10.42.0.237   rocky9
redis-cluster-4   1/1     Running   0          5h      10.42.0.243   rocky9
redis-cluster-5   1/1     Running   0          6h37m   10.42.0.241   rocky9

🧮 Cluster info 照会

💻 Terminal

cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3

🧮 Cluster nodes 照会

💻 Terminal

66390566e7f48211e8bed3a6a0e49155cbe8b54b 10.42.0.242:18504@28504,redis-cluster-0.redis-cluster-service.default.svc.cluster.local 🟢master - 0 1786188079000 10 connected 0-5460
3b690009fb9a90ff6b43765363c4c0b81b63d103 10.42.0.233:18504@28504,redis-cluster-1.redis-cluster-service.default.svc.cluster.local myself,🟢master - 0 0 2 connected 5461-10922
02069ae6b4f98c31d5f4e70f243bc9a006b68443 10.42.0.235:18504@28504,redis-cluster-2.redis-cluster-service.default.svc.cluster.local 🟢master - 0 1786188078548 3 connected 10923-16383
3722046802e006615ac1d976e62e4c94c58af586 10.42.0.237:18504@28504,redis-cluster-3.redis-cluster-service.default.svc.cluster.local 🟡slave 02069ae6b4f98c31d5f4e70f243bc9a006b68443 0 1786188079552 3 connected
ee5c1d85271bb86d2c0bb2e943fc9f546e4321c1 10.42.0.243:18504@28504,redis-cluster-4.redis-cluster-service.default.svc.cluster.local 🟡slave 66390566e7f48211e8bed3a6a0e49155cbe8b54b 0 1786188079552 10 connected
541fa7cc95095c4b0390bb8ae9d787aa1201c9e2 10.42.0.241:18504@28504,redis-cluster-5.redis-cluster-service.default.svc.cluster.local 🟡slave 3b690009fb9a90ff6b43765363c4c0b81b63d103 0 1786188079000 2 connected

🖧 Redis-Cluster ポッド(Pod) ダイアグラム (Failback 後)

🧑‍💻 結論

SLEEP コマンドを使った Redis-Cluster Failover(障害対応)、Failback(原状回復)テスト成功
SLEEP コマンドを使うと Redis はダウンせず、IP も変わりませんが、 3秒以上応答がなければ障害対応が実行されます。

➡ ここまでは Redis-Cluster 単独でテストしました。
Java(Spring) App と接続して動くところまで見るには、こちらをクリックしてください。

<< K8s Redis-Cluster 構築 ②	K8s Redis-Cluster App ① >>

Email 返事がかかってなれば、メールでお知らせします。

クバネティス(Kubernetes) 環境で Redis-Cluster テスト 実戦ガイド ③

Failover(障害対応)テスト

🖧 1. Redis-Cluster ポッド(Pod) ダイアグラム (初期状態)

1. Failover(障害対応)テスト

Redis-0 💥 ダウン(delete)

💻 Redis-0 LOG: Master -> Replica

💻 Redis-1 LOG: Master

💻 Redis-4 LOG: Replica -> Master

🧮 Failover(障害対応)後の状況

🖧 Redis-Cluster ポッド(Pod) ダイアグラム (Failover 後)

2. Failback(原状回復)テスト

💻 Redis-4 💥ダウン(delete)

💻 Redis-0 LOG: Replica -> Master

💻 Redis-1 LOG: Master

💻 Redis-4 LOG: Master -> Replica

🧮 Failback(原状回復)後の状況

🧮 Cluster info 確認

🧮 Cluster nodes 確認

🖧 Redis-Cluster ポッド(Pod) ダイアグラム (Failback 後)

🧑‍💻 結論

Failover(障害対応)テスト: Redis-0 SLEEP(停止)

1. Failover(障害対応)テスト

Redis-0 SLEEP 10

💻 Redis-0 LOG: Master -> Replica

💻 Redis-1 LOG: Master

💻 Redis-4 LOG: Replica -> Master

🧮 Failover(障害対応)後の状況

🧮 Cluster info 照会

🧮 Cluster nodes 照会

🖧 Redis-Cluster ポッド(Pod) ダイアグラム (Failover 後)

2. Failback(原状回復)テスト

💻 Redis-4 SLEEP 10

💻 Redis-0 LOG: Replica -> Master

💻 Redis-1 LOG: Master

💻 Redis-2 LOG: Redis-1 ログと同じため省略

💻 Redis-4 LOG: Master -> Replica

🧮 Failback(原状回復)後の状況

🧮 Cluster info 照会

🧮 Cluster nodes 照会

🖧 Redis-Cluster ポッド(Pod) ダイアグラム (Failback 後)

🧑‍💻 結論

クバネティス(Kubernetes) 環境で Redis-Cluster テスト実戦ガイド ③