How to Build a Highly Available Redis Cluster Using Tidis?

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 如何使用Tidis构建高可用的Redis集群？

| username: abelard2008

I have recently been using the latest version: Tidis + three-node TiKV + three-node PD, aiming to replace my original single-node Redis to achieve a highly available Redis setup. However, when I use one Tidis node + three-node TiKV + three-node PD, I encounter write failures (I only have one type of sorted set read/write operation). I suspect there might be an issue with my Tidis setup because the three-node TiKV, along with three PD nodes and three TiDB nodes, has been running with read/write operations for nearly 20 days without any problems.

Here is my Tidis + three-node architecture diagram:

Image
Image480×649 20.6 KB

Part of my Tidis config.toml configuration is as follows:

[server]
listen = “0.0.0.0”
port = 6666
pd_addrs = “172.216.9.81:2379,172.216.9.82:2379,172.216.9.83:2379”

I hope someone who is currently using Tidis can provide some guidance. Thank you very much!

2
| username: WalterWj | Original post link

I haven’t used it :thinking: Can you send the error message first so we can take a look?

3
| username: abelard2008 | Original post link

Thank you for your enthusiasm, there were no errors! My client is implemented based on hiredis in C language, and I can only see that hiredis failed to write and then exited. In the past few days, after reverting to using standalone Redis, everything has been normal. I am now also rebuilding Tidis and trying to capture the scenarios when Tidis fails.

4
| username: Billmay表妹 | Original post link

Here are some practical solutions you can check out:

5
| username: abelard2008 | Original post link

Thank you, but it seems that none of these are using Tidis? :slight_smile:

6
| username: Billmay表妹 | Original post link

I mistook it for Redis.

Currently, I don’t have any practical examples to share, but I look forward to your practical articles.

7
| username: abelard2008 | Original post link

My current idea is to build the following architecture to test it:

Image
Image515×729 60.6 KB

8
| username: abelard2008 | Original post link

Suddenly remembered, after my C client failed, I used the redis-cli command line to connect and encountered the following error:

[abelard@fedora redis-6.2.6]$ ./src/redis-cli -h 172.216.9.83 -p 6666
172.216.9.83:6666> scan *
Error: Server closed the connection
(8.19s)
172.216.9.83:6666>

Furthermore, I saw that Tidis has a log file: tikv-service.log, which continuously outputs:

2023/01/03 15:02:43.760 +08:00 ERRO [GC] failed to begin txn
2023/01/03 15:02:46.268 +08:00 ERRO error to begin new transaction: [/root/.cargo/git/checkouts/client-rust-cbb958c150bdea6a/2648fe6/tikv-
client-pd/src/cluster.rs:264]: failed to connect to [Member { name: "pd-172.216.9.83-2379", member_id: 7916348439350824078, peer_urls: ["ht
tp://172.216.9.83:2380"], client_urls: ["http://172.216.9.83:2379"], leader_priority: 0, deploy_path: "", binary_version: "", git_hash: "", 
dc_location: "" }, Member { name: "pd-172.216.9.82-2379", member_id: 15138125198525324009, peer_urls: ["http://172.216.9.82:2380"], client_u
rls: ["http://172.216.9.82:2379"], leader_priority: 0, deploy_path: "", binary_version: "", git_hash: "", dc_location: "" }, Member { name:
 "pd-172.216.9.81-2379", member_id: 17883531146897579492, peer_urls: ["http://172.216.9.81:2380"], client_urls: ["http://172.216.9.81:2379"],
 leader_priority: 0, deploy_path: "", binary_version: "", git_hash: "", dc_location: "" }]

And 2379 and 2380 are the ports of the pd-server. My TiDB has been working, but I don’t know why these two ports are causing errors in Tidis?

9
| username: Billmay表妹 | Original post link

Let’s give it a try~ Looking forward to your output. Our current content in this area is still relatively scarce~

10
| username: abelard2008 | Original post link

Hmm, the real cause of my problem this time should be the logs in TiKV-service.log. Can you help me take a look? Thanks.

11
| username: Billmay表妹 | Original post link

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version]
[Reproduction Path] What operations were performed when the issue occurred
[Encountered Issue: Issue Symptoms and Impact]
[Resource Configuration]
[Attachments: Screenshots / Logs / Monitoring]

If convenient, please describe the issue again according to this format. Also, are you using pure TiKV without TiDB?

12
| username: abelard2008 | Original post link
  1. It’s not pure TiKV; I am also running TiDB on TiKV, and now TiDB is continuously working.
  2. For re-describing, should I start a new topic or modify this one?
13
| username: tidb菜鸟一只 | Original post link

I’ve never used TiDis before, can’t we just use a Redis cluster directly?

14
| username: abelard2008 | Original post link

I thought about using a Redis cluster directly, but my project also needs a MySQL cluster, so I considered using TiDB to replace MySQL. Running Tidis on top of TiKV to replace the single-point Redis, solving the original single-point and high-availability issues altogether.

15
| username: 裤衩儿飞上天 | Original post link

Waiting for the expert’s great article :crazy_face:

16
| username: Billmay表妹 | Original post link

Send a new one, it’s better. We’ve gone off-topic too much.

17
| username: abelard2008 | Original post link

Okay, I am reading the code for starting the tidis service and using the tikv-client to connect to the pd-server. Once I understand it better, I will post another question. Thanks for the reminder!

18
| username: Billmay表妹 | Original post link

Got it~

19
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.