Some(RpcFailure(RpcStatus { code: 8-RESOURCE_EXHAUSTED, message

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: Some(RpcFailure(RpcStatus { code: 8-RESOURCE_EXHAUSTED, message

| username: beebol

[TiDB Usage Environment] Testing
[TiDB Version]
[Reproduction Path] What operations were performed when the issue occurred
[Encountered Issue: Issue Phenomenon and Impact]
[Resource Configuration]
[Attachments: Screenshots/Logs/Monitoring]
TiKV log error: [2023/05/10 15:33:02.963 +08:00] [ERROR] [raft_client.rs:450] [“connection aborted”] [addr=10.200.20.175:20160] [receiver_err=“Some(RpcFailure(RpcStatus { code: 8-RESOURCE_EXHAUSTED, message: "Sent message larger than max (10610761 vs. 10485760)", details: }))”] [sink_error=Some(RemoteStopped)] [store_id=1]

| username: beebol | Original post link

The cause of the problem: When synchronizing full data load from MySQL using the DM tool, an error occurred. Later, the TiKV configuration was adjusted: raftstore.raft-entry-max-size: 16M

| username: Billmay表妹 | Original post link

What is the version number of MySQL?
Also, take a screenshot of TiDB Dashboard → Cluster Info → Hosts to check your configuration.

| username: Billmay表妹 | Original post link

The error message is Sent message larger than max, indicating that the sent message exceeds the maximum limit. This issue is usually caused by the raft-entry-max-size parameter in TiKV being set too small. The raft-entry-max-size parameter controls the maximum size of each Raft log entry, and if this limit is exceeded, the above error will occur.

In your case, it might be that when using the DM tool to synchronize full data, the size of the sent message exceeds the size set by the raft-entry-max-size parameter in TiKV. You have already set the raft-entry-max-size parameter to 16M, which should resolve this issue. However, it is important to note that setting the raft-entry-max-size parameter too large may lead to a decrease in TiKV’s performance, so it needs to be adjusted according to the actual situation.

| username: beebol | Original post link

The MySQL version is 5.6.23

| username: beebol | Original post link

Increase this parameter further?

| username: beebol | Original post link

I have already stopped DM, but this error keeps occurring. Does this mean that DM is not the cause?
[2023/05/10 16:13:52.267 +08:00] [ERROR] [raft_client.rs:450] [“connection aborted”] [addr=10.200.20.175:20160] [receiver_err=“Some(RpcFailure(RpcStatus { code: 8-RESOURCE_EXHAUSTED, message: "Sent message larger than max (10559474 vs. 10485760)", details: }))”] [sink_error=Some(RemoteStopped)] [store_id=1]

| username: TiDBer_UUTlqVvZ | Original post link

This error is caused by the message size sent by TiKV exceeding the maximum limit, resulting in the connection being interrupted. Specifically, the error message Sent message larger than max indicates that the size of the sent message exceeds the maximum limit, where the max value is 10485760, which is 10MB. The actual size of the sent message is 10610761, which exceeds this limit.

You can consider adjusting TiKV’s configuration to increase the values of raft_max_size_per_msg and raft_entry_max_size to support larger message sizes. Specifically, raft_max_size_per_msg indicates the maximum size of each message, while raft_entry_max_size indicates the maximum size of each Entry. You can adjust these two parameters’ values appropriately based on actual needs to meet business requirements.

Additionally, if this error occurs frequently, you can consider checking TiKV’s monitoring and logs to analyze possible causes, such as whether there are a large number of region merges or splits.


Take a look at the value of raft_max_size_per_msg?

| username: beebol | Original post link

I have already made the changes, but it doesn’t seem to have taken effect?
image

| username: TiDBer_UUTlqVvZ | Original post link

Did you restart it?

If the restart didn’t take effect, we might need to ask other experts to take a look…

| username: beebol | Original post link

After reloading, the query now reports: Region is unavailable, this error is reported.

| username: beebol | Original post link

I have encountered the same problem. I am using the 5.0.2 version. I have tried to set the tidb_enable_clustered_index parameter to OFF, but it still doesn’t work.

| username: db_user | Original post link

If it’s a test environment, first revert the parameter and see if the cluster can recover. Additionally, the issue might be caused by this parameter, max-grpc-send-msg-len. Before changing the parameter, check the error message. The error clearly states that the message is larger than 10M. The default values of the two parameters you changed are not 10M, so it doesn’t make sense.