TPC-C Benchmark Test Failed, 100 Threads, 1400 Warehouses

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TPC-C压测失败，100线程，1400warehouse

| username: magic

[TiDB Usage Environment] Testing
[TiDB Version] v650
[Reproduction Path] tpcc stress test, 100 threads, warehouse 1400, time 15min; tikv configuration uses official stress test configuration
[Encountered Problem: Problem Phenomenon and Impact] Errors occurred during stress tests with 100 and 200 threads, while the 50-thread stress test completed normally;
[Resource Configuration]
tikv 16c128G 5 nodes, with stress test threads on tikv node a
tidb 8c16G 5 nodes, configured with SLB

[Attachments: Screenshots/Logs/Monitoring]
Stress test starts, 100 threads…
tiup is checking updates for component bench …
Starting component bench: /root/.tiup/components/bench/v1.12.0/tiup-bench tpcc run -H192.168.120.17 -P4000 -Dtpcc14 -Uroot -pQaz123… --time 900s --warehouses 1400 --interval 90s --threads 100
Omitting normal output
[Current] NEW_ORDER - Takes(s): 90.0, Count: 191420, TPM: 127613.2, Sum(ms): 4689326.4, Avg(ms): 24.5, 50th(ms): 24.1, 90th(ms): 33.6, 95th(ms): 37.7, 99th(ms): 46.1, 99.9th(ms): 60.8, Max(ms): 260.0
[Current] ORDER_STATUS - Takes(s): 90.0, Count: 16915, TPM: 11278.0, Sum(ms): 184683.3, Avg(ms): 10.9, 50th(ms): 10.0, 90th(ms): 17.8, 95th(ms): 19.9, 99th(ms): 25.2, 99.9th(ms): 31.5, Max(ms): 218.1
[Current] PAYMENT - Takes(s): 90.0, Count: 182659, TPM: 121773.0, Sum(ms): 2543295.7, Avg(ms): 13.9, 50th(ms): 13.1, 90th(ms): 21.0, 95th(ms): 24.1, 99th(ms): 31.5, 99.9th(ms): 48.2, Max(ms): 226.5
[Current] STOCK_LEVEL - Takes(s): 90.0, Count: 17087, TPM: 11391.4, Sum(ms): 313816.0, Avg(ms): 18.4, 50th(ms): 17.8, 90th(ms): 25.2, 95th(ms): 27.3, 99th(ms): 33.6, 99.9th(ms): 44.0, Max(ms): 302.0
[mysql] 2023/02/16 15:57:34 statement.go:48: invalid connection
[mysql] 2023/02/16 15:57:34 packets.go:446: busy buffer
[mysql] 2023/02/16 15:57:34 statement.go:48: invalid connection
[mysql] 2023/02/16 15:57:34 statement.go:48: invalid connection
[mysql] 2023/02/16 15:57:34 statement.go:48: invalid connection
[mysql] 2023/02/16 15:57:34 packets.go:446: busy buffer
[mysql] 2023/02/16 15:57:34 statement.go:48: invalid connection
[mysql] 2023/02/16 15:57:34 packets.go:446: busy buffer
[mysql] 2023/02/16 15:57:34 packets.go:446: busy buffer
[mysql] 2023/02/16 15:57:34 statement.go:96: invalid connection
[mysql] 2023/02/16 15:57:34 statement.go:48: invalid connection
[mysql] 2023/02/16 15:57:34 statement.go:96: invalid connection
[mysql] 2023/02/16 15:57:34 statement.go:96: invalid connection
[mysql] 2023/02/16 15:57:34 statement.go:48: invalid connection
[mysql] 2023/02/16 15:57:34 packets.go:446: busy buffer
[mysql] 2023/02/16 15:57:34 statement.go:48: invalid connection
[mysql] 2023/02/16 15:57:34 statement.go:96: invalid connection
[mysql] 2023/02/16 15:57:34 statement.go:48: invalid connection
[mysql] 2023/02/16 15:57:34 statement.go:96: invalid connection
[mysql] 2023/02/16 15:57:34 statement.go:48: invalid connection
[mysql] 2023/02/16 15:57:34 statement.go:48: invalid connection
[mysql] 2023/02/16 15:57:34 statement.go:48: invalid connection
[mysql] 2023/02/16 15:57:34 statement.go:48: invalid connection
[mysql] 2023/02/16 15:57:34 statement.go:48: invalid connection
[mysql] 2023/02/16 15:57:34 statement.go:48: invalid connection
Finished
[Summary] DELIVERY - Takes(s): 899.9, Count: 171184, TPM: 11413.8, Sum(ms): 11339230.7, Avg(ms): 66.3, 50th(ms): 65.0, 90th(ms): 83.9, 95th(ms): 92.3, 99th(ms): 113.2, 99.9th(ms): 234.9, Max(ms): 1073.7
[Summary] DELIVERY_ERR - Takes(s): 899.9, Count: 12, TPM: 0.8, Sum(ms): 437.1, Avg(ms): 36.2, 50th(ms): 31.5, 90th(ms): 65.0, 95th(ms): 65.0, 99th(ms): 65.0, 99.9th(ms): 65.0, Max(ms): 65.0
[Summary] NEW_ORDER - Takes(s): 899.9, Count: 1930098, TPM: 128683.9, Sum(ms): 46853697.4, Avg(ms): 24.3, 50th(ms): 24.1, 90th(ms): 33.6, 95th(ms): 37.7, 99th(ms): 46.1, 99.9th(ms): 62.9, Max(ms): 1040.2
[Summary] NEW_ORDER_ERR - Takes(s): 899.9, Count: 42, TPM: 2.8, Sum(ms): 391.1, Avg(ms): 9.3, 50th(ms): 7.9, 90th(ms): 17.8, 95th(ms): 18.9, 99th(ms): 22.0, 99.9th(ms): 22.0, Max(ms): 22.0
[Summary] ORDER_STATUS - Takes(s): 899.9, Count: 171617, TPM: 11442.0, Sum(ms): 1855510.8, Avg(ms): 10.8, 50th(ms): 10.0, 90th(ms): 17.8, 95th(ms): 19.9, 99th(ms): 24.1, 99.9th(ms): 32.5, Max(ms): 520.1
[Summary] ORDER_STATUS_ERR - Takes(s): 899.9, Count: 3, TPM: 0.2, Sum(ms): 19.4, Avg(ms): 6.4, 50th(ms): 3.1, 90th(ms): 14.2, 95th(ms): 14.2, 99th(ms): 14.2, 99.9th(ms): 14.2, Max(ms): 14.2
[Summary] PAYMENT - Takes(s): 899.9, Count: 1843839, TPM: 122932.4, Sum(ms): 25536722.0, Avg(ms): 13.9, 50th(ms): 13.1, 90th(ms): 21.0, 95th(ms): 24.1, 99th(ms): 31.5, 99.9th(ms): 50.3, Max(ms): 570.4
[Summary] PAYMENT_ERR - Takes(s): 899.9, Count: 22, TPM: 1.5, Sum(ms): 87.8, Avg(ms): 4.1, 50th(ms): 4.2, 90th(ms): 6.3, 95th(ms): 6.8, 99th(ms): 6.8, 99.9th(ms): 6.8, Max(ms): 6.8
[Summary] STOCK_LEVEL - Takes(s): 899.9, Count: 172072, TPM: 11472.6, Sum(ms): 3047524.0, Avg(ms): 17.7, 50th(ms): 17.8, 90th(ms): 24.1, 95th(ms): 26.2, 99th(ms): 32.5, 99.9th(ms): 44.0, Max(ms): 302.0
[Summary] STOCK_LEVEL_ERR - Takes(s): 899.9, Count: 5, TPM: 0.3, Sum(ms): 47.1, Avg(ms): 9.3, 50th(ms): 4.2, 90th(ms): 21.0, 95th(ms): 21.0, 99th(ms): 21.0, 99.9th(ms): 21.0, Max(ms): 21.0
tpmC: 128683.8, tpmTotal: 285944.6, efficiency: 714.8%

2
| username: 数据小黑 | Original post link

Should the stress test be conducted on a machine outside the cluster but within the network? If you use a TiKV machine, will the traffic from the stress testing software interfere with the IO of TiKV, and can you get valuable results?

3
| username: 托马斯滑板鞋 | Original post link

Is the stress testing machine an independent machine? How many cores? Does the SLB have a maximum number of connections?

4
| username: TI表弟 | Original post link

Have you checked the CPU, memory, and disk usage of the TiDB nodes and the TiKV nodes under stress testing? If everything is normal, check the TiDB logs. If there are no errors in the TiDB logs, it must be an external cause. Check the TCP handle configuration, use telnet to check the network, and check the maximum number of connections for the SLB.

5
| username: tidb狂热爱好者 | Original post link

Your machine with 50 units can’t handle 100 threads, it seems like there’s a configuration issue. Check if the local SSD hard drive’s IO read speed is 5000MB/s; cloud SSDs might not be suitable for TiDB.

6
| username: xingzhenxiang | Original post link

How many connections should be set for TiDB?

7
| username: magic | Original post link

Machine performance issues do not affect for the time being.

8
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.