Real-time Writing Under High Concurrency

This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 高并发下的实时写入

| username: TiDBer_U71iNHJh

The project involves real-time writing and reading under high concurrency. I would like to ask, regarding real-time writing, is it better to directly insert into TiDB under 2000 concurrent requests, or to buffer through Kafka and Flink for batch writing? How is TiDB’s performance if we directly insert point inserts?

| username: 像风一样的男子 | Original post link

You can conduct a stress test yourself to understand the performance of your database cluster. 2000 concurrent connections are not considered high, and building a cluster according to the server configuration on the official website will definitely support it.

| username: TiDBer_U71iNHJh | Original post link

Time is a bit tight, and I want to avoid detours as much as possible. If TiDB’s high-concurrency point-insert performance is good, then there’s no need to set up Kafka and Flink. The architecture is heavy, and maintenance is also difficult.

| username: Kongdom | Original post link

There should be no problem with the standard configuration. Performance largely depends on the hardware environment.

| username: ziptoam | Original post link

No problem, I asked my colleague who works in the production environment, and 2000 concurrent writes to TiDB is not an issue.

| username: TiDBer_rvITcue9 | Original post link

If the hardware is not bad, then there is no problem.

| username: xfworld | Original post link

Regardless of the type of database, accumulating small batches and writing them all at once will definitely improve performance by reducing the number of transaction waits and interactions.

2k concurrency is no pressure for TiDB. For convenience, you can directly use insert into …

It is recommended to optimize~

| username: 濱崎悟空 | Original post link

2000 is not a big problem.

| username: 呢莫不爱吃鱼 | Original post link

2000 should be no problem.

| username: zhaokede | Original post link

Directly write. You can use the built database resources to perform stress testing and fatigue testing.

| username: vincentLi | Original post link

I’ve heard that 3 TiDB servers can support 2000 concurrent connections. If that’s not enough, just add more TiDB servers.

| username: zhanggame1 | Original post link

There is no problem with 2000 concurrent connections; the advantage of TiDB is its support for high concurrency.

| username: 有猫万事足 | Original post link

Don’t accumulate batches; having too many rows can turn into a large transaction, which is more likely to cause issues. Many optimizations for small transactions have already been implemented. It’s best if you test it yourself: one insert per row versus one insert per 500 rows.

| username: zhaokede | Original post link

Submitting 500-1000 lines at a time should be more appropriate.