Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.
Original topic: 8-RESOURCE_EXHAUSTED, message: "Received message larger than max (9612720 vs. 4194304)
[TiDB Usage Environment] Test
[TiDB Version] v5.3.0
[Reproduction Path] Running tidis
[Encountered Problem: Phenomenon and Impact]
I deployed a 3-node TiKV + 3-node PD and 3-node TiDB on 3 physical servers (each with 128G memory), and deployed Tidis based on the 3-node PD.
[Resource Configuration]
The main content of the Tidis server’s config.toml is as follows:
[server]
pd_addrs = "172.26.9.81:2379,172.26.9.82:2379,172.26.9.83:2379"
instance_id = "1"
prometheus_listen = "0.0.0.0"
prometheus_port = 8080
log_level = "info"
log_file = "tikv-service.log"
After inserting sorted set data through Tidis, the following error appeared in the tikv-service.log after a period of time:
2023/01/04 12:54:40.182 +08:00 ERRO error occurred so rollback transaction: gRPC error: RpcFailure: 8-RESOURCE_EXHAUSTED Received message larger than max (9612720 vs. 4194304)
2023/01/04 12:54:40.183 +08:00 ERRO connection error TikvClient(Grpc(RpcFailure(RpcStatus { code: 8-RESOURCE_EXHAUSTED, message: "Received message larger than max (9612720 vs. 4194304)", details: [] })))
2023/01/04 12:54:44.090 +08:00 ERRO error occurred so rollback transaction: gRPC error: RpcFailure: 8-RESOURCE_EXHAUSTED Received message larger than max (9723284 vs. 4194304)
The key information of the above error should be:
RpcFailure: 8-RESOURCE_EXHAUSTED Received message larger than max (9723284 vs. 4194304)
[Attachment: Screenshot/Log/Monitoring]
Upon investigation, it seems to be related to grpc, but I don’t know where to set or modify the configuration parameters? Thank you!
Did gRPC not handle the streaming mode? You should ask the author of tidis about this…
How can I figure it out from the code myself? What is the principle of gRPC streaming mode? Thanks.
Received message larger than max (9723284 vs. 4194304)
Try reducing the size of the data…
I guess it’s not supported yet.
Reducing the data, I don’t know how to reduce it. Theoretically, it should be possible, but wouldn’t that make it unusable?
Go to GitHub and open an issue.
Could you provide some guidance on how this is handled in TiDB? That 4194304 is exactly 4M, so I must be missing some configuration, right? It’s unlikely that TiDB can only transmit 4M. Thanks!
Also, where can I see this 4M parameter? Thanks.
By default, TiDB does not have a 4MB limit…
Refer to this document…
This is about TiDB, but the error I encountered is in the TiKV client:
2023/01/04 16:19:48.937 +08:00 ERRO connection error TikvClient(Grpc(RpcFailure(RpcStatus { code: 8-RESOURCE_EXHAUSTED, message: "Received message larger than max (72327769 vs. 4194304)", details: [] })))
The TikvClient mentioned here is from this project tikv-client.
So this question should be directed to the author… Just submit an issue on GitHub.
Sure, I have already raised the issue, but I also want to check it myself to see if I can figure out the problem quickly.
Issue location: Received message larger than max (9612720 vs. 4194304
If you want to understand it, you have to look at the source code yourself… 
Yes, at this point, you indeed need to read the source code, but I don’t have any leads yet.
PS: From my shallow understanding, I think TiDB also needs a TiKV client to interact with TiKV. TiDB uses the Go language, while the client here uses Rust. The principles should be similar. Additionally, my 4194304 is a basic size, but unfortunately, I am too ignorant about these components.
This should be a parameter in the code, which cannot be modified for the time being. However, the product team has received the issue you submitted and will update you when there is progress. Thank you for your attention!
|
username: abelard2008
|
username: 
|
username: