Note:
Original topic: 扩容TiKV节点后，region分布不均，store打分差异很大
Version: v5.0.3
Question:
How should this situation be handled?
However, the leader balance is normal.
Is the data disk size the same?
select store_id, address, leader_weight, region_weight from information_schema.tikv_store_status; Take a look
select store_id, address, leader_weight, region_weight from information_schema.tikv_store_status;
The settings for leader_weight and region_weight are the same.
The size of the data disks is the same.
What about the remaining space?
The remaining space is different.
Looking at the region graph, the number of regions for the two TiKVs below is slowly increasing, while the three above are decreasing.
At present, the scheduling seems to be normal, but around 12:37 PM, the cluster became unavailable for about 5 minutes. On the business side, there were connection issues (specifically, the error was “mysql gone away”), the QPS dropped significantly, and the cluster latency reached 5 minutes.
TiDB node logs:
[2022/08/19 12:37:56.735 +08:00] [WARN] [expensivequery.go:178] [expensive_query] [cost_time=60.033386751s] [conn_id=168261983] [user=srv_t1111l] [txn_start_ts=0] [mem_max=“0 Bytes (0 Bytes)”] [sql=“use testdb”]testdb”]testdb”]testdb”]testdb”]testdb”]testdb”]testdb”]testdb”]testdb”]testdb”]testdb”]testdb”]testdb”]testdb”]testdb”]testdb”]testdb”]testdb”]testdb”]testdb”]testdb”]testdb”]testdb”]testdb”]testdb”]testdb”]testdb”]testdb”]testdb”]testdb”]testdb”]testdb”]testdb”]testdb”]
[2022/08/19 12:38:35.754 +08:00] [INFO] [conn.go:812] [“command dispatched failed”] [conn=168230619] [connInfo=“id:168230619, addr:10.30.219.227:34772 status:10, collation:latin1_swedish_ci, user:srv_t1111l”] [command=Query] [status=“inTxn:0, autocommit:1”] [sql=“insert into bill_xxxxxx (user_id, hfsm_id, fsm_id, state_num, detail, c_date) values (941526237,10068300,1,3,‘github.com/pingcap/errors.AddStack github.com/pingcap/errors.Trace github.com/pingcap/tidb/server.(*packetIO).flush github.com/pingcap/tidb/server.(*clientConn).flush github.com/pingcap/tidb/server.(*clientConn).writeOkWith github.com/pingcap/tidb/server.(*clientConn).handleQuerySpecial github.com/pingcap/tidb/server.(*clientConn).handleStmt github.com/pingcap/tidb/server.(*clientConn).handleQuery github.com/pingcap/tidb/server.(*clientConn).dispatch github.com/pingcap/tidb/server.(*clientConn).Run github.com/pingcap/tidb/server.(*Server).onConn
Is there a problem with the network?
If the server configurations are inconsistent, including the network environment, it is normal for the scores to be inconsistent.
The server configuration and network environment are the same.
It’s not a network issue.
Is it still possible to check the server’s resource usage when the latency is high?
The logs suggest it might be a network issue. I recommend investigating it, such as checking for network jitter and similar problems.
It’s fine if the leaders are evenly distributed. Are there some empty regions affecting the statistics?
|
username: seiang
|
username: 
|
username: 
|
username: 
|
username: 
|
username: 
|
username: