Edu/Support

オンライン定期点検
サービス開始

クバネティス(Kubernetes) 環境で Redis-Sentinel 構築実戦ガイド ③

Failover(障害対応) テスト

🖧 Redis-Sentinel ポッド(Pod) ダイアグラム（初期状態）

┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │Sentinel-0(98) │ │Sentinel-1(100)│ │Sentinel-2(102)│ └───────┬───────┘ └───────┬───────┘ └───────┬───────┘ └───────────────────┼───────────────────┘ ▼ ┌───────────────┐ │ Redis-0(92) │ │ 🟢Master │ └──────┬────────┘ ┌──────────┴───────────┐ ▼ ▼ ┌───────────────┐ ┌───────────────┐ │ Redis-1(94) │ │ Redis-2(96) │ │ 🟡Replica-1 │ │ 🟡Replica-2 │ └───────────────┘ └───────────────┘

1. Failover(障害対応) テスト

Redis-0 をダウンさせ、Redis-1 が Replica から Master へ切り替わる(昇格する)ことを確認します
レプリカ Redis-1,2 のうち Redis-1 がマスターになる理由: redis.conf replica-priority 値: Redis-0 100, Redis-1 110, Redis-2 120 -> 値が小さい Redis-1 がマスターへ切り替わります。
💥 ポッド(Pod) ダウン後に再起動すると IP が変わります。

Redis-0 💥 ダウン(delete)

💻 Terminal

$ kubectl delete pod redis-0
  pod "redis-0" deleted

💻 Redis-0 LOG: Master -> Replica

① 03秒 kill シグナルを受けて Redis ダウン (+0秒)
② 05秒 Redis 自動再起動 (+2秒)
③ 17秒 Sentinel から命令を受けて Master -> Replica(Slave) へ切替 (+14秒)

📄 redis.log

1:M 11:03:03.043 #📍kill シグナルを受けてダウン: Redis is now ready to exit, bye bye...
1:C 11:03:05.808 *📍2秒後に再起動: oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:S 11:03:17.398 *📍Sentinelからの信号を受けて Master->Replica(Slave)へ転換: Before turning into a replica,...

Redis-0 全ログ ▼

📄 redis.log

1:signal-handler (1786413782) Received SIGTERM scheduling shutdown... 
1:M 11:03:03.037 * User requested shutdown...
1:M 11:03:03.043 #📍kill シグナルを受けてダウン: Redis is now ready to exit, bye bye...				
1:C 11:03:05.808 *📍2秒後に再起動: oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 11:03:05.808 * Redis version=8.8.2, bits=64, commit=00000000, modified=1, pid=1, just started
1:C 11:03:05.808 * Configuration loaded				
1:M 11:03:05.808 *➡いったん Master として起動: monotonic clock: POSIX clock_gettime
1:M 11:03:05.810 * Server initialized
1:M 11:03:05.811 * Reading RDB base file on AOF loading...
1:M 11:03:05.811 * DB loaded from base file aof.1.base.rdb: 0.001 seconds
1:M 11:03:10.458 * DB loaded from incr file aof.1.incr.aof: 4.648 seconds
1:M 11:03:10.458 *➡4.6秒所要: DB loaded from append only file: 4.648 seconds 
1:M 11:03:10.462 * Opening AOF incr file aof.1.incr.aof on server start
1:M 11:03:10.462 * Ready to accept connections tcp				
1:S 11:03:17.398 *📍Sentinelからの信号を受けて Master -> Replica(Slave)へ転換: Before turning into a replica, ... 
1:S 11:03:17.398 * Connecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504
1:S 11:03:17.399 * MASTER <-> REPLICA sync started
1:S 11:03:17.399 *➡Sentinelから 'REPLICAOF redis-1' 命令を受信: REPLICAOF redis-1.redis-service.default.svc.cluster.local:18504 enabled (user request from 'id=6 name=sentinel-...')
1:S 11:03:17.403 *➡redis.conf 記録: CONFIG REWRITE executed with success.
1:S 11:03:17.404 * Master replied to PING, replication can continue...
1:S 11:03:27.929 * Done loading RDB, keys loaded: 2955803, keys expired: 0.
1:S 11:03:27.930 * MASTER <-> REPLICA sync: Finished with success
1:S 11:03:27.930 * MASTER <-> REPLICA sync: Starting AOF after a successful sync
1:S 11:03:31.263 * Background AOF rewrite finished successfully

💻 Redis-1 LOG: Replica -> Master

① 03秒 Redis-0 ダウン検知 (+0秒)
② 06秒 Sentinel-1 から命令(replicaof noone)を受けてレプリカ -> マスターへ切替 (+3秒)

📄 redis.log

1:S 11:03:03.090 *📍Redis-0 ダウン検知: Connection with master lost. 
1:M 11:03:06.504 *📍Sentinelからの命令を受けて Replica(Slave) -> Masterへ転換: Discarding previously cached master state.

Redis-1 全ログ ▼

📄 redis.log

1:S 11:03:03.090 *📍Redis-0 ダウン検知: Connection with master lost. 
1:S 11:03:03.090 * Caching the disconnected master state.
1:S 11:03:03.091 *➡再接続試行: Reconnecting to MASTER redis-0.redis-service.default.svc.cluster.local:18504  
1:S 11:03:03.091 * MASTER <-> REPLICA sync started
1:S 11:03:03.091 #➡接続失敗: Error condition on socket for SYNC: Connection refused 
1:S 11:03:03.905 * Connecting to MASTER redis-0.redis-service.default.svc.cluster.local:18504
1:S 11:03:03.905 * MASTER <-> REPLICA sync started					 
1:M 11:03:06.504 *📍Sentinelからの命令を受けて Replica(Slave) -> Masterへ転換: Discarding previously cached master state.
1:M 11:03:06.504 * MASTER MODE enabled (user request from 'id=8 addr=10.42.0.100:49936 laddr=10.42.0.94:18504 fd=16 name=sentinel-bde0e191-cmd...')
1:M 11:03:06.508 *➡redis.conf 記録('REPLICAOF' 削除): CONFIG REWRITE executed with success.
1:M 11:03:07.355 *➡redis-2から同期リクエストを受信: Replica redis-2.redis-service.default.svc.cluster.local:18504 asks for synchronization
1:M 11:03:07.356 * Partial resynchronization request from redis-2.redis-service.default.svc.cluster.local:18504 accepted. Sending 596 bytes of backlog starting from offset 432382052.
1:M 11:03:17.404 * Replica redis-0.redis-service.default.svc.cluster.local:18504 asks for synchronization
1:M 11:03:17.404 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '27e7d7f2495d430017d646aef6776a55870fac45', my replication IDs are 'd735e0345181173ecc516d24ffb58e11e54cc224' and '80d34a3b7754b6bc41d8e95c4645a44540ec73b2')  
1:M 11:03:17.404 * Replica redis-0.redis-service.default.svc.cluster.local:18504 is capable of rdb channel synchronization, and partial sync isn't possible. Full sync will continue with dedicated rdb channel.
1:M 11:03:17.404 *➡redis-0から同期リクエストを受信: Replica redis-0.redis-service.default.svc.cluster.local:18504 asks for synchronization
1:M 11:03:17.405 * Full resync requested by replica redis-0.redis-service.default.svc.cluster.local:18504 (rdb-channel)
1:M 11:03:17.405 * Delay next BGSAVE for diskless SYNC
1:M 11:03:20.981 * Starting BGSAVE for SYNC with target: replicas sockets (rdb-channel)
1:M 11:03:20.981 * Starting to deliver RDB and replication stream to replica: redis-0.redis-service.default.svc.cluster.local:18504
1:M 11:03:20.994 * Background RDB transfer started by pid 20 to replica socket
20:C 11:03:24.223 * BGSAVE done, 2955803 keys saved, 0 keys skipped, 354696590 bytes written.
20:C 11:03:24.231 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
1:M 11:03:24.309 * Background RDB transfer terminated with success
1:M 11:03:24.309 * Synchronization with replica redis-0.redis-service.default.svc.cluster.local:18504 succeeded
1:M 11:03:24.309 * Streamed RDB transfer with replica redis-0.redis-service.default.svc.cluster.local:18504 succeeded (socket). Waiting for REPLCONF ACK from replica to enable streaming
1:M 11:03:24.309 * RDB transfer completed, rdb only replica (redis-0.redis-service.default.svc.cluster.local:18504) should be disconnected asap
1:M 11:03:24.309 * Connection with replica (rdbchannel) redis-0.redis-service.default.svc.cluster.local:18504 lost.

💻 Redis-2 LOG

① 03秒 Redis-0 ダウン検知 (+0秒)
② 07秒新しいマスター(Redis-1) に接続 (+4秒)

📄 redis.log

1:S 11:03:03.090 *📍Redis-0 ダウン検知: Connection with master lost. 
1:S 11:03:07.349 *📍新しいマスターへ接続: Connecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504

Redis-2 全ログ ▼

📄 redis.log

1:S 11:03:03.090 *📍Redis-0 ダウン検知: Connection with master lost. 
1:S 11:03:03.090 * Caching the disconnected master state.
1:S 11:03:03.091 *➡再接続試行: Reconnecting to MASTER redis-0.redis-service.default.svc.cluster.local:18504  
1:S 11:03:03.091 * MASTER <-> REPLICA sync started
1:S 11:03:03.091 #➡接続失敗: Error condition on socket for SYNC: Connection refused 
1:S 11:03:03.965 * Connecting to MASTER redis-0.redis-service.default.svc.cluster.local:18504
1:S 11:03:03.965 * MASTER <-> REPLICA sync started
1:S 11:03:07.349 *📍新しいマスターへ接続: Connecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504  
1:S 11:03:07.350 * MASTER <-> REPLICA sync started
1:S 11:03:07.350 *➡Sentinelから 'REPLICAOF redis-1' 命令を受信: REPLICAOF redis-1.redis-service.default.svc.cluster.local:18504 enabled (user request from 'id=8 addr=10.42.0.100:33902 laddr=10.42.0.96:18504 fd=16 name=sentinel-bde0e191-cmd ...')  
1:S 11:03:07.355 *➡redis.conf 記録: CONFIG REWRITE executed with success.
1:S 11:03:07.355 * Non blocking connect for SYNC fired the event.
1:S 11:03:07.355 * Master replied to PING, replication can continue...
1:S 11:03:07.355 * Trying a partial resynchronization (request 80d34a3b7754b6bc41d8e95c4645a44540ec73b2:432382052).
1:S 11:03:07.356 * Successful partial resynchronization with master.
1:S 11:03:07.356 * Master replication ID changed to d735e0345181173ecc516d24ffb58e11e54cc224
1:S 11:03:07.356 * MASTER <-> REPLICA sync: Master accepted a Partial Resynchronization.

💻 Sentinel-0 LOG

① 06.1秒 redis-0 ダウン検知(+sdown) (+3秒) -> sentinel.conf down-after-milliseconds mymaster 3000 ms
② 06.2秒 redis-0 ダウン確定(+odown) (+3秒) -> 他の Sentinel と通信し過半数が同意すれば確定

📄 sentinel.log

1:X 11:03:06.131 #📍redis-0 ダウン検知: +sdown master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:06.242 #📍redis-0 ダウン確定: +odown master mymaster redis-0.redis-service.default.svc.cluster.local 18504 #quorum 2/2

Sentinel-0 全ログ ▼

📄 sentinel.log

1:X 11:03:06.131 #📍redis-0 ダウン検知: +sdown master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:06.231 * Sentinel new configuration saved on disk
1:X 11:03:06.231 # +new-epoch 1
1:X 11:03:06.242 * Sentinel new configuration saved on disk
1:X 11:03:06.242 # +vote-for-leader bde0e1917a4b878a3d48742d44f932639a1de44a 1
1:X 11:03:06.242 #📍redis-0 ダウン確定: +odown master mymaster redis-0.redis-service.default.svc.cluster.local 18504 #quorum 2/2
1:X 11:03:06.242 * Next failover delay: I will not start a failover before Tue Aug 11 11:05:06 2026
1:X 11:03:07.350 # +config-update-from sentinel bde0e1917a4b878a3d48742d44f932639a1de44a sentinel-1.sentinel-service 28504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.350 #➡redis-1をマスターへ転換確定: +switch-master mymaster redis-0.redis-service.default.svc.cluster.local 18504 redis-1.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.351 * +slave slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504  
1:X 11:03:07.351 * +slave slave redis-0.redis-service.default.svc.cluster.local:18504 redis-0.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.356 *➡変更内容を sentinel.conf に記録: Sentinel new configuration saved on disk

💻 Sentinel-1 LOG

① 06.1秒 redis-0 ダウン検知(+sdown) (+3秒) -> sentinel.conf down-after-milliseconds mymaster 3000 ms
② 06.2秒 redis-0 ダウン確定(+odown) (+3秒) -> 他の Sentinel と通信し過半数が同意すれば確定
③ 06.3秒 Sentinel 同士で投票し 'Sentinel-1' を Leader に選出(+elected-leader)
④ 06.4秒 redis-1 をマスターへ変更(+failover-state-send-slaveof-noone)

📄 sentinel.log

1:X 11:03:06.138 #📍redis-0 ダウン検知: +sdown master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:06.222 #📍redis-0 ダウン確定: +odown master mymaster redis-0.redis-service.default.svc.cluster.local 18504 #quorum 3/2
1:X 11:03:06.327 #📍Leaderに選出: +elected-leader master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:06.403 *📍redis-1をマスターに変更: +failover-state-send-slaveof-noone slave redis-1.redis-service.default.svc.cluster.local:18504 redis-1.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504

Sentinel-1 全ログ ▼

📄 sentinel.log

1:X 11:03:06.138 #📍redis-0 ダウン検知: +sdown master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:06.222 #📍redis-0 ダウン確定: +odown master mymaster redis-0.redis-service.default.svc.cluster.local 18504 #quorum 3/2
1:X 11:03:06.222 # +new-epoch 1
1:X 11:03:06.222 # +try-failover master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:06.226 * Sentinel new configuration saved on disk
1:X 11:03:06.226 # +vote-for-leader bde0e1917a4b878a3d48742d44f932639a1de44a 1
1:X 11:03:06.243 * 984e3350ee6299ced794cd4fb6820e0d202f0a6e voted for bde0e1917a4b878a3d48742d44f932639a1de44a 1
1:X 11:03:06.243 * b2919c14e1a99a3ca317cf7f91ba75f1d40b50d1 voted for bde0e1917a4b878a3d48742d44f932639a1de44a 1
1:X 11:03:06.327 #📍Leaderに選出: +elected-leader master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:06.327 # +failover-state-select-slave master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:06.403 # +selected-slave slave redis-1.redis-service.default.svc.cluster.local:18504 redis-1.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:06.403 *📍redis-1をマスターに変更: +failover-state-send-slaveof-noone slave redis-1.redis-service.default.svc.cluster.local:18504 redis-1.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:06.503 * +failover-state-wait-promotion slave redis-1.redis-service.default.svc.cluster.local:18504 redis-1.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.295 * Sentinel new configuration saved on disk
1:X 11:03:07.296 # +promoted-slave slave redis-1.redis-service.default.svc.cluster.local:18504 redis-1.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.296 # +failover-state-reconf-slaves master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.349 *➡redis-2へ redis-1がマスターに変更されたと通知: +slave-reconf-sent slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504  
1:X 11:03:08.322 * +slave-reconf-inprog slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:08.322 * +slave-reconf-done slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:08.381 # -odown master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:08.381 # +failover-end master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:08.381 # +switch-master mymaster redis-0.redis-service.default.svc.cluster.local 18504 redis-1.redis-service.default.svc.cluster.local 18504
1:X 11:03:08.381 * +slave slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 11:03:08.382 * +slave slave redis-0.redis-service.default.svc.cluster.local:18504 redis-0.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 11:03:08.394 * Sentinel new configuration saved on disk

💻 Sentinel-2 LOG

① 06.1秒 redis-0 ダウン検知(+sdown) (+3秒) -> sentinel.conf down-after-milliseconds mymaster 3000 ms
② 07.2秒 redis-0 ダウン確定(+odown) (+4秒) -> 他の Sentinel と通信し過半数が同意すれば確定

📄 sentinel.log

1:X 11:03:06.114 #📍+sdown master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.239 #📍+odown master mymaster redis-0.redis-service.default.svc.cluster.local 18504 #quorum 3/2

Sentinel-2 全ログ ▼

📄 sentinel.log

1:X 11:03:06.114 #📍+sdown master mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:06.232 * Sentinel new configuration saved on disk
1:X 11:03:06.232 # +new-epoch 1
1:X 11:03:06.243 * Sentinel new configuration saved on disk
1:X 11:03:06.243 # +vote-for-leader bde0e1917a4b878a3d48742d44f932639a1de44a 1
1:X 11:03:07.239 #📍+odown master mymaster redis-0.redis-service.default.svc.cluster.local 18504 #quorum 3/2
1:X 11:03:07.239 * Next failover delay: I will not start a failover before Tue Aug 11 11:05:06 2026
1:X 11:03:07.350 # +config-update-from sentinel bde0e1917a4b878a3d48742d44f932639a1de44a sentinel-1.sentinel-service 28504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.350 #➡redis-1をマスターへ転換確定+switch-master mymaster redis-0.redis-service.default.svc.cluster.local 18504 redis-1.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.351 * +slave slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.351 * +slave slave redis-0.redis-service.default.svc.cluster.local:18504 redis-0.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 11:03:07.356 * Sentinel new configuration saved on disk
1:X 11:03:17.398 *➡redis-0へ redis-1がマスターに変更されたと通知: +convert-to-slave slave redis-0.redis-service.default.svc.cluster.local:18504 redis-0.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504

🧮 Failover(障害対応) 後の状況

redis-0 の IP が '10.42.0.92' から '10.42.0.103' に変わりました。

💻 Terminal

[redis@Rocky9 kube]$ k get pods -o wide
NAME         READY   STATUS    RESTARTS   AGE     IP            NODE  
redis-0      1/1     Running   0           7s     10.42.0.103   rocky9 📍IP:92->103
redis-1      1/1     Running   0          23m     10.42.0.94    rocky9
redis-2      1/1     Running   0          23m     10.42.0.96    rocky9
sentinel-0   1/1     Running   0          23m     10.42.0.98    rocky9
sentinel-1   1/1     Running   0          22m     10.42.0.100   rocky9
sentinel-2   1/1     Running   0          22m     10.42.0.102   rocky9

🧮 redis.conf 確認

Redis LOG: 'CONFIG REWRITE' で変更しました。

📄 redis.conf

Redis-0(103): replicaof redis-1.redis-service.default.svc.cluster.local 18504
Redis-1(94):  replicaof なし -> Redis-1がマスターであるため
Redis-2(96):  replicaof redis-1.redis-service.default.svc.cluster.local 18504

🧮 sentinel.conf 確認

Sentinel LOG: 'Sentinel new configuration saved on disk' で変更しました。
redis-0,2 がレプリカと記録され、sentinel-0,1,2 の内容は同一です。

📄 sentinel.conf

# Generated by CONFIG REWRITE
sentinel monitor mymaster redis-1.redis-service.default.svc.cluster.local 18504 2
sentinel known-replica mymaster redis-0.redis-service.default.svc.cluster.local 18504
sentinel known-replica mymaster redis-2.redis-service.default.svc.cluster.local 18504

🖧 Redis-Sentinel ポッド(Pod) ダイアグラム（Failover 後）

┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │Sentinel-0(98) │ │Sentinel-1(100)│ │Sentinel-2(102)│ └───────┬───────┘ └───────┬───────┘ └───────┬───────┘ └─────┬─────────────┴───────────────────┘ │ │ ┌───────────────┐ │ │ Redis-0(103) │ │ │ 🟡Replica-1 │ │ └───────────────┘ │ ▲ │ ┌──────────┴───────────┐ ▼ │ ▼ ┌──────┴────────┐ ┌───────────────┐ │ Redis-1(94) │ │ Redis-2(96) │ │ 🟢Master │ │ 🟡Replica-2 │ └───────────────┘ └───────────────┘

2. Failback(原状復帰) テスト

現在のマスターである Redis-1 をダウンさせ、Redis-0 が Replica から Master へ切り替わる(昇格する)ことを確認します
📚 'Failback(原状復帰) テスト' は Redis-0 ポッド(Pod) が再びマスターになる、という意味で使いました。
(Failover テストと区別するために使いました)

💻 Redis-1 💥ダウン(delete)

💻 Terminal

$ kubectl delete pod redis-1
  pod "redis-1" deleted

💻 Redis-0 LOG: Replica -> Master

① 29分 55秒 Redis-1 接続切断 (+0秒)
② 29分 58秒 Sentinel から命令を受けて Replica(Slave) -> Master へ切替(昇格) (+3秒)

📄 redis.log

1:S 13:29:55.437 *📍Redis-1 接続切断: Connection with master lost.
1:M 13:29:58.714 *📍Sentinelからの信号を受けて Replica->Masterへ転換: Discarding previously cached master state.

Redis-0 全ログ ▼

📄 redis.log

1:S 13:29:55.437 *📍Redis-1 接続切断: Connection with master lost.
1:S 13:29:55.437 * Caching the disconnected master state.
1:S 13:29:55.437 *➡再接続試行: Reconnecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504
1:S 13:29:55.438 * MASTER <-> REPLICA sync started
1:S 13:29:55.438 #➡接続失敗: Error condition on socket for SYNC: Connection refused
1:S 13:29:55.765 * Connecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504
1:S 13:29:55.766 # Unable to connect to MASTER: Operation now in progress
1:S 13:29:56.771 * Connecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504
1:S 13:29:56.772 # Unable to connect to MASTER: Operation now in progress
1:S 13:29:57.776 * Connecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504
1:S 13:29:57.777 # Unable to connect to MASTER: Operation now in progress
1:M 13:29:58.714 *📍Sentinelからの信号を受けて Replica(Slave) -> Masterへ転換: Discarding previously cached master state.
1:M 13:29:58.714 * Setting secondary replication ID to d735e0345181173ecc516d24ffb58e11e54cc224, valid up to offset: 434858832. New replication ID is b9c98fef85ebf2b43197de7c0505082ef78ccef1
1:M 13:29:58.714 * MASTER MODE enabled (user request from 'id=15 addr=10.42.0.98:55870 laddr=10.42.0.103:18504 fd=18 name=sentinel-984e3350-cmd ...')
1:M 13:29:58.720 * CONFIG REWRITE executed with success.
1:M 13:29:59.655 * Replica redis-2.redis-service.default.svc.cluster.local:18504 asks for synchronization
1:M 13:29:59.655 * Partial resynchronization request from redis-2.redis-service.default.svc.cluster.local:18504 accepted. Sending 596 bytes of backlog starting from offset 434858832.
1:M 13:30:11.770 * Replica redis-1.redis-service.default.svc.cluster.local:18504 asks for synchronization
1:M 13:30:11.770 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '1d3bd6057b7649148d5200d8a78f296834efc05e', my replication IDs are 'b9c98fef85ebf2b43197de7c0505082ef78ccef1' and 'd735e0345181173ecc516d24ffb58e11e54cc224')
1:M 13:30:11.770 * Replica redis-1.redis-service.default.svc.cluster.local:18504 is capable of rdb channel synchronization, and partial sync isn't possible. Full sync will continue with dedicated rdb channel.
1:M 13:30:11.770 * Replica redis-1.redis-service.default.svc.cluster.local:18504 asks for synchronization
1:M 13:30:11.771 * Full resync requested by replica redis-1.redis-service.default.svc.cluster.local:18504 (rdb-channel)
1:M 13:30:11.771 * Delay next BGSAVE for diskless SYNC
1:M 13:30:14.852 * Starting BGSAVE for SYNC with target: replicas sockets (rdb-channel)
1:M 13:30:14.852 * Starting to deliver RDB and replication stream to replica: redis-1.redis-service.default.svc.cluster.local:18504
1:M 13:30:14.860 * Background RDB transfer started by pid 14 to replica socket
14:C 13:30:17.948 * BGSAVE done, 2955803 keys saved, 0 keys skipped, 354696590 bytes written.
14:C 13:30:17.956 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
1:M 13:30:17.973 * Background RDB transfer terminated with success
1:M 13:30:17.973 * Synchronization with replica redis-1.redis-service.default.svc.cluster.local:18504 succeeded
1:M 13:30:17.973 * Streamed RDB transfer with replica redis-1.redis-service.default.svc.cluster.local:18504 succeeded (socket). Waiting for REPLCONF ACK from replica to enable streaming
1:M 13:30:17.973 * RDB transfer completed, rdb only replica (redis-1.redis-service.default.svc.cluster.local:18504) should be disconnected asap
1:M 13:30:17.973 * Connection with replica (rdbchannel) redis-1.redis-service.default.svc.cluster.local:18504 lost.

💻 Redis-1 LOG: Master -> Replica

① 29分 55秒 kill シグナルを受けてダウン (+0秒)
② 29分 57秒ポッド(Pod) 再起動 (+2秒)
③ 30分 11秒 Sentinel から命令を受けて Master -> Replica(Slave) へ切替 (+16秒)

📄 redis.log

1:M 13:29:55.389 #📍kill シグナルを受けてダウン: Redis is now ready to exit, bye bye...
1:M 13:29:57.787 *📍2秒後に再起動: Server initialized
1:S 13:30:11.754 *📍Sentinelからの信号を受けて Master->Replicaへ転換: Before turning into a replica,...

Redis-1 全ログ ▼

📄 redis.log

1:signal-handler (1786422594)➡kill シグナル: Received SIGTERM scheduling shutdown...
1:M 13:29:54.480 * User requested shutdown...
1:M 13:29:55.389 #📍kill シグナルを受けてダウン: Redis is now ready to exit, bye bye...
1:C 13:29:57.785 *oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 13:29:57.785 * Redis version=8.8.2, bits=64, commit=00000000, modified=1, pid=1, just started
1:C 13:29:57.785 * Configuration loaded
1:M 13:29:57.785 *➡いったん Master として起動: monotonic clock: POSIX clock_gettime
1:M 13:29:57.787 *📍2秒後に再起動: Server initialized
1:M 13:29:57.787 * Reading RDB base file on AOF loading...
1:M 13:29:57.788 * Loading RDB produced by version 8.8.2
1:M 13:29:57.788 * RDB age 10198 seconds
1:M 13:29:57.788 * RDB memory usage when created 0.70 Mb
1:M 13:29:57.788 * RDB is base AOF
1:M 13:29:57.788 * Done loading RDB, keys loaded: 0, keys expired: 0.
1:M 13:29:57.788 * DB loaded from base file aof.2.base.rdb: 0.001 seconds
1:M 13:30:02.391 * DB loaded from incr file aof.2.incr.aof: 4.604 seconds
1:M 13:30:02.391 * DB loaded from append only file: 4.604 seconds
1:M 13:30:02.397 * Opening AOF incr file aof.2.incr.aof on server start
1:M 13:30:02.397 * Ready to accept connections tcp
1:S 13:30:11.754 *📍Sentinelからの信号を受けて Master -> Replica(Slave)へ転換: Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
1:S 13:30:11.754 * Connecting to MASTER redis-0.redis-service.default.svc.cluster.local:18504
1:S 13:30:11.755 * MASTER <-> REPLICA sync started
1:S 13:30:11.755 * REPLICAOF redis-0.redis-service.default.svc.cluster.local:18504 enabled (user request from 'id=6 addr=10.42.0.100:51752 laddr=10.42.0.104:18504 fd=13 name=sentinel-bde0e191-cmd ...')
1:S 13:30:11.769 * CONFIG REWRITE executed with success.
1:S 13:30:11.769 * Non blocking connect for SYNC fired the event.
1:S 13:30:11.770 * Master replied to PING, replication can continue...
1:S 13:30:11.770 * Trying a partial resynchronization (request 1d3bd6057b7649148d5200d8a78f296834efc05e:434858832).
1:S 13:30:11.770 * PSYNC is not possible, initialize RDB channel.
1:S 13:30:14.852 * Starting to receive RDB and replication stream in parallel.
1:S 13:30:14.871 * MASTER <-> REPLICA sync: receiving streamed RDB from master with EOF to disk
1:S 13:30:17.954 * Discarding previously cached master state.
1:S 13:30:17.954 * MASTER <-> REPLICA sync: Loading DB in memory
1:S 13:30:17.960 * MASTER <-> REPLICA sync: Flushing old data
1:S 13:30:18.983 * Loading RDB produced by version 8.8.2
1:S 13:30:18.983 * RDB age 4 seconds
1:S 13:30:18.983 * RDB memory usage when created 476.64 Mb
1:S 13:30:21.623 * Done loading RDB, keys loaded: 2955803, keys expired: 0.
1:S 13:30:21.623 * Removing the RDB file obtained from the master. This replica has persistence disabled
1:S 13:30:21.623 * MASTER <-> REPLICA sync: Finished with success
1:S 13:30:21.623 * MASTER <-> REPLICA sync: Starting AOF after a successful sync
1:S 13:30:21.623 * Creating AOF incr file temp-aof.incr on background rewrite
1:S 13:30:21.634 * Background append only file rewriting started by pid 13
1:S 13:30:21.634 * MASTER <-> REPLICA sync: Starting to stream replication buffer into the db (1933 bytes).
1:S 13:30:21.635 * MASTER <-> REPLICA sync: Successfully streamed replication buffer into the db (1933 bytes in total)
13:C 13:30:24.866 * BGSAVE done, 2955803 keys saved, 0 keys skipped, 354696457 bytes written.
13:C 13:30:24.872 * Successfully created the temporary AOF base file temp-rewriteaof-bg-13.aof
13:C 13:30:24.881 * Fork CoW for AOF rewrite: current 1 MB, peak 1 MB, average 1 MB
1:S 13:30:24.950 * Background AOF rewrite terminated with success
1:S 13:30:24.950 * Successfully renamed the temporary AOF base file temp-rewriteaof-bg-13.aof into aof.3.base.rdb
1:S 13:30:24.950 * Successfully renamed the temporary AOF incr file temp-aof.incr into aof.3.incr.aof
1:S 13:30:24.953 * Removing the history file aof.2.incr.aof in the background
1:S 13:30:24.953 * Removing the history file aof.2.base.rdb in the background
1:S 13:30:24.960 * Background AOF rewrite finished successfully

💻 Redis-2 LOG

① 29分 55秒 Redis-1 との接続切断 (+0秒)
② 29分 59秒 Sentinel から Redis-0 が新しいマスターになった信号を受けて接続 (+4秒)

📄 redis.log

1:S 13:29:55.437 *📍Redis-1との接続切断: Connection with master lost.
1:S 13:29:59.650 *📍Sentinelから Redis-0 が新しいマスターになった信号を受けて接続: Connecting to MASTER redis-0.redis-service.default.svc.cluster.local:18504

Redis-2 全ログ ▼

📄 redis.log

1:S 13:29:55.437 *📍Redis-1との接続切断: Connection with master lost.
1:S 13:29:55.437 * Caching the disconnected master state.
1:S 13:29:55.437 *➡Redis-1へ接続試行: Reconnecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504
1:S 13:29:55.437 * MASTER <-> REPLICA sync started
1:S 13:29:55.438 #➡Redis-1へ接続失敗: Error condition on socket for SYNC: Connection refused
1:S 13:29:56.341 * Connecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504
1:S 13:29:56.342 # Unable to connect to MASTER: Operation now in progress
1:S 13:29:57.347 * Connecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504
1:S 13:29:57.347 # Unable to connect to MASTER: Operation now in progress
1:S 13:29:58.352 * Connecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504
1:S 13:29:58.353 # Unable to connect to MASTER: Operation now in progress
1:S 13:29:59.357 * Connecting to MASTER redis-1.redis-service.default.svc.cluster.local:18504
1:S 13:29:59.357 # Unable to connect to MASTER: Success
1:S 13:29:59.650 *📍Sentinelから Redis-0 が新しいマスターになった信号を受けて接続: Connecting to MASTER redis-0.redis-service.default.svc.cluster.local:18504
1:S 13:29:59.650 * MASTER <-> REPLICA sync started
1:S 13:29:59.650 * REPLICAOF redis-0.redis-service.default.svc.cluster.local:18504 enabled (user request from 'id=16 addr=10.42.0.98:54564 laddr=10.42.0.96:18504 fd=17 name=sentinel-984e3350-cmd ...')
1:S 13:29:59.654 * CONFIG REWRITE executed with success.
1:S 13:29:59.654 * Non blocking connect for SYNC fired the event.
1:S 13:29:59.655 * Master replied to PING, replication can continue...
1:S 13:29:59.655 * Trying a partial resynchronization (request d735e0345181173ecc516d24ffb58e11e54cc224:434858832).
1:S 13:29:59.655 * Successful partial resynchronization with master.
1:S 13:29:59.655 * Master replication ID changed to b9c98fef85ebf2b43197de7c0505082ef78ccef1
1:S 13:29:59.655 * MASTER <-> REPLICA sync: Master accepted a Partial Resynchronization.

💻 Sentinel-0 LOG

① 58.4秒 redis-1 ダウン検知(+sdown) (+3秒) -> sentinel.conf down-after-milliseconds mymaster 3000 ms
② 58.5秒 redis-1 ダウン確定(+odown) (+3秒) -> 他の Sentinel と通信し過半数が同意すれば確定
③ 58.5秒 Sentinel 同士で投票し 'Sentinel-0' を Leader に選出(+elected-leader)
④ 58.6秒 redis-0 をマスターへ変更(+failover-state-send-slaveof-noone)

📄 sentinel.log

1:X 13:29:58.457 #📍Redis-1 ダウン検知: +sdown master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:58.534 #📍Redis-1 ダウン確定: +odown master mymaster redis-1.redis-service.default.svc.cluster.local 18504 #quorum 2/2
1:X 13:29:58.593 #📍リーダーに選出: +elected-leader master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:58.652 *📍redis-0をマスターに変更: +failover-state-send-slaveof-noone slave redis-0.redis-service.default.svc.cluster.local:18504 redis-0.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504

Sentinel-0 全ログ ▼

📄 sentinel.log

1:X 13:29:56.546 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.302 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.457 #📍Redis-1 ダウン検知: +sdown master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:58.521 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.534 #📍Redis-1 ダウン確定: +odown master mymaster redis-1.redis-service.default.svc.cluster.local 18504 #quorum 2/2
1:X 13:29:58.534 # +new-epoch 2
1:X 13:29:58.534 # +try-failover master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:58.538 * Sentinel new configuration saved on disk
1:X 13:29:58.538 # +vote-for-leader 984e3350ee6299ced794cd4fb6820e0d202f0a6e 2
1:X 13:29:58.548 * bde0e1917a4b878a3d48742d44f932639a1de44a voted for 984e3350ee6299ced794cd4fb6820e0d202f0a6e 2
1:X 13:29:58.549 * b2919c14e1a99a3ca317cf7f91ba75f1d40b50d1 voted for 984e3350ee6299ced794cd4fb6820e0d202f0a6e 2
1:X 13:29:58.593 #📍リーダーに選出: +elected-leader master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:58.593 # +failover-state-select-slave master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:58.652 # +selected-slave slave redis-0.redis-service.default.svc.cluster.local:18504 redis-0.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:58.652 *📍redis-0をマスターに変更: +failover-state-send-slaveof-noone slave redis-0.redis-service.default.svc.cluster.local:18504 redis-0.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:58.714 * +failover-state-wait-promotion slave redis-0.redis-service.default.svc.cluster.local:18504 redis-0.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.354 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:59.587 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:59.596 * Sentinel new configuration saved on disk
1:X 13:29:59.596 # +promoted-slave slave redis-0.redis-service.default.svc.cluster.local:18504 redis-0.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.596 # +failover-state-reconf-slaves master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.649 * +slave-reconf-sent slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:30:00.611 * +slave-reconf-inprog slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:30:00.611 * +slave-reconf-done slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:30:00.687 #➡Redis-1 再起動確認: -odown master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:30:00.687 # +failover-end master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:30:00.687 #➡Redis-1へ Redis-0がマスターだと通知: +switch-master mymaster redis-1.redis-service.default.svc.cluster.local 18504 redis-0.redis-service.default.svc.cluster.local 18504
1:X 13:30:00.688 * +slave slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 13:30:00.689 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:30:00.689 * +slave slave redis-1.redis-service.default.svc.cluster.local:18504 redis-1.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 13:30:00.693 * Sentinel new configuration saved on disk
1:X 13:30:00.747 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'

💻 Sentinel-1 LOG

① 58.5秒 redis-1 ダウン検知(+sdown) (+3秒) -> sentinel.conf down-after-milliseconds mymaster 3000 ms
② 58.6秒 redis-1 ダウン確定(+odown) (+3秒) -> 他の Sentinel と通信し過半数が同意すれば確定

📄 sentinel.log

1:X 13:29:58.519 #📍+sdown master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:58.620 #📍+odown master mymaster redis-1.redis-service.default.svc.cluster.local 18504 #quorum 3/2

Sentinel-1 全ログ ▼

📄 sentinel.log

1:X 13:29:56.497 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.302 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.459 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.519 #📍+sdown master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:58.539 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.543 * Sentinel new configuration saved on disk
1:X 13:29:58.543 # +new-epoch 2
1:X 13:29:58.548 * Sentinel new configuration saved on disk
1:X 13:29:58.548 # +vote-for-leader 984e3350ee6299ced794cd4fb6820e0d202f0a6e 2
1:X 13:29:58.620 #📍+odown master mymaster redis-1.redis-service.default.svc.cluster.local 18504 #quorum 3/2
1:X 13:29:58.620 * Next failover delay: I will not start a failover before Tue Aug 11 13:31:59 2026
1:X 13:29:59.354 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:59.592 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:59.650 # +config-update-from sentinel 984e3350ee6299ced794cd4fb6820e0d202f0a6e sentinel-0.sentinel-service 28504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.650 #➡redis-0をマスターへ転換確定: +switch-master mymaster redis-1.redis-service.default.svc.cluster.local 18504 redis-0.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.651 * +slave slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.652 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:59.652 * +slave slave redis-1.redis-service.default.svc.cluster.local:18504 redis-1.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.657 *➡変更内容を sentinel.conf に記録: Sentinel new configuration saved on disk
1:X 13:29:59.664 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:30:00.612 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:30:00.689 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:30:11.754 * +convert-to-slave slave redis-1.redis-service.default.svc.cluster.local:18504 redis-1.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504

💻 Sentinel-2 LOG

① 58.3秒 redis-1 ダウン検知(+sdown) (+3秒) -> sentinel.conf down-after-milliseconds mymaster 3000 ms
② 59.4秒 redis-1 ダウン確定(+odown) (+3秒) -> 他の Sentinel と通信し過半数が同意すれば確定

📄 sentinel.log

1:X 13:29:58.301 #📍+sdown master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.454 #📍+odown master mymaster redis-1.redis-service.default.svc.cluster.local 18504 #quorum 3/2

Sentinel-2 全ログ ▼

📄 sentinel.log

1:X 13:29:56.552 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.301 #📍+sdown master mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:58.459 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.521 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.539 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:58.543 * Sentinel new configuration saved on disk
1:X 13:29:58.543 # +new-epoch 2
1:X 13:29:58.549 * Sentinel new configuration saved on disk
1:X 13:29:58.549 # +vote-for-leader 984e3350ee6299ced794cd4fb6820e0d202f0a6e 2
1:X 13:29:59.454 #📍+odown master mymaster redis-1.redis-service.default.svc.cluster.local 18504 #quorum 3/2
1:X 13:29:59.454 * Next failover delay: I will not start a failover before Tue Aug 11 13:31:58 2026
1:X 13:29:59.587 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:59.592 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:59.650 # +config-update-from sentinel 984e3350ee6299ced794cd4fb6820e0d202f0a6e sentinel-0.sentinel-service 28504 @ mymaster redis-1.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.650 #➡redis-0をマスターへ転換確定: +switch-master mymaster redis-1.redis-service.default.svc.cluster.local 18504 redis-0.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.651 * +slave slave redis-2.redis-service.default.svc.cluster.local:18504 redis-2.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504  
1:X 13:29:59.652 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:29:59.652 * +slave slave redis-1.redis-service.default.svc.cluster.local:18504 redis-1.redis-service.default.svc.cluster.local 18504 @ mymaster redis-0.redis-service.default.svc.cluster.local 18504
1:X 13:29:59.657 *➡変更内容を sentinel.conf に記録: Sentinel new configuration saved on disk
1:X 13:29:59.694 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:30:00.612 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'
1:X 13:30:00.708 # Failed to resolve hostname 'redis-1.redis-service.default.svc.cluster.local'

🧮 Failback(原状復帰) 後の状況

💻 Terminal

$ kubectl get pods -o wide
redis-0      1/1     Running   0          147m    10.42.0.103   rocky9 📍IP:92->103
redis-1      1/1     Running   0           39s    10.42.0.104   rocky9 📍IP:94->104
redis-2      1/1     Running   0          170m    10.42.0.96    rocky9
sentinel-0   1/1     Running   0          170m    10.42.0.98    rocky9
sentinel-1   1/1     Running   0          170m    10.42.0.100   rocky9
sentinel-2   1/1     Running   0          170m    10.42.0.102   rocky9

🧮 redis.conf 確認

📄 redis.conf

Redis-0: replicaof なし	-> Master
Redis-1: replicaof redis-0.redis-service.default.svc.cluster.local 18504
Redis-2: replicaof redis-0.redis-service.default.svc.cluster.local 18504

🧮 sentinel.conf 確認

Redis-0 マスター、Redis-1,2 レプリカ

📄 sentinel.conf

sentinel monitor mymaster redis-0.redis-service.default.svc.cluster.local 18504 2
sentinel known-replica mymaster redis-1.redis-service.default.svc.cluster.local 18504
sentinel known-replica mymaster redis-2.redis-service.default.svc.cluster.local 18504

🖧 Redis-Sentinel ポッド(Pod) ダイアグラム（原状復帰）

┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │Sentinel-0(98) │ │Sentinel-1(100)│ │Sentinel-2(102)│ └───────┬───────┘ └───────┬───────┘ └───────┬───────┘ └───────────────────┼───────────────────┘ ▼ ┌───────────────┐ │ Redis-0(103) │ │ 🟢Master │ └──────┬────────┘ ┌──────────┴───────────┐ ▼ ▼ ┌───────────────┐ ┌───────────────┐ │ Redis-1(104) │ │ Redis-2(96) │ │ 🟡Replica-1 │ │ 🟡Replica-2 │ └───────────────┘ └───────────────┘

🧑‍💻 結論

Redis-Sentinel Failover(障害対応)、Failback(原状復帰) テスト成功
ポッド(Pod) をダウン、再起動すると IP は変わりますが、問題なく処理されます。
障害対応(復旧) 時間は約 4秒です。

➡ ここまでは Redis-Sentinel 単体でテストしました。
Java(Spring) App と接続して動くところまで見るには、こちらをクリックしてください。

<< K8s Redis-Sentinel 構築 ②	K8s Redis-Sentinel App ① >>

Email 返事がかかってなれば、メールでお知らせします。