TiDB V6.5.5 TiDB Error

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidbV6.5.5 tidb报错

| username: 田帅萌7

Handling Suggestion: The number of TiDB threads that have crashed. An alert will be triggered when a crash occurs. The thread is usually recovered; otherwise, TiDB will restart frequently. Please collect the panic logs and locate the cause.

[2023/11/18 08:11:51.717 +08:00] [ERROR] [client_batch.go:303] [batchSendLoop] [r={}] [stack=“github.com/tikv/client-go/v2/internal/client.(*batchConn).batchSendLoop.func1\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.4-0.20230912041415-9c163cc8574b/internal/client/client_batch.go:305\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:884\nruntime.goPanicIndex\n\t/usr/local/go/src/runtime/panic.go:113\ngithub.com/pingcap/kvproto/pkg/tikvpb.encodeVarintTikvpb\n\t/go/pkg/mod/github.com/pingcap/kvproto@v0.0.0-20230726063044-73d6d7f3756b/pkg/tikvpb/tikvpb.pb.go:5438\ngithub.com/pingcap/kvproto/pkg/tikvpb.(*BatchCommandsRequest).MarshalToSizedBuffer\n\t/go/pkg/mod/github.com/pingcap/kvproto@v0.0.0-20230726063044-73d6d7f3756b/pkg/tikvpb/tikvpb.pb.go:3813\ngithub.com/pingcap/kvproto/pkg/tikvpb.(*BatchCommandsRequest).Marshal\n\t/go/pkg/mod/github.com/pingcap/kvproto@v0.0.0-20230726063044-73d6d7f3756b/pkg/tikvpb/tikvpb.pb.go:3766\ngoogle.golang.org/protobuf/internal/impl.legacyMarshal\n\t/go/pkg/mod/google.golang.org/protobuf@v1.28.1/internal/impl/legacy_message.go:402\ngoogle.golang.org/protobuf/proto.MarshalOptions.marshal\n\t/go/pkg/mod/google.golang.org/protobuf@v1.28.1/proto/encode.go:166\ngoogle.golang.org/protobuf/proto.MarshalOptions.MarshalAppend\n\t/go/pkg/mod/google.golang.org/protobuf@v1.28.1/proto/encode.go:125\ngithub.com/golang/protobuf/proto.marshalAppend\n\t/go/pkg/mod/github.com/golang/protobuf@v1.5.2/proto/wire.go:40\ngithub.com/golang/protobuf/proto.Marshal\n\t/go/pkg/mod/github.com/golang/protobuf@v1.5.2/proto/wire.go:23\ngoogle.golang.org/grpc/encoding/proto.codec.Marshal\n\t/go/pkg/mod/google.golang.org/grpc@v1.51.0/encoding/proto/proto.go:45\ngoogle.golang.org/grpc.encode\n\t/go/pkg/mod/google.golang.org/grpc@v1.51.0/rpc_util.go:595\ngoogle.golang.org/grpc.prepareMsg\n\t/go/pkg/mod/google.golang.org/grpc@v1.51.0/stream.go:1708\ngoogle.golang.org/grpc.(*clientStream).SendMsg\n\t/go/pkg/mod/google.golang.org/grpc@v1.51.0/stream.go:846\ngithub.com/pingcap/kvproto/pkg/tikvpb.(*tikvBatchCommandsClient).Send\n\t/go/pkg/mod/github.com/pingcap/kvproto@v0.0.0-20230726063044-73d6d7f3756b/pkg/tikvpb/tikvpb.pb.go:2068\ngithub.com/tikv/client-go/v2/internal/client.(*batchCommandsClient).send\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.4-0.20230912041415-9c163cc8574b/internal/client/client_batch.go:519\ngithub.com/tikv/client-go/v2/internal/client.(*batchConn).getClientAndSend\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.4-0.20230912041415-9c163cc8574b/internal/client/client_batch.go:381\ngithub.com/tikv/client-go/v2/internal/client.(*batchConn).batchSendLoop\n\t/go/pkg/mod/github.com/tikv/client-go/v2@v2.0.4-0.20230912041415-9c163cc8574b/internal/client/client_batch.go:344”]

| username: Fly-bird | Original post link

Adding TiDB nodes might improve the situation.

| username: zhanggame1 | Original post link

When an issue occurs, is it due to insufficient resources? Is the problematic TiDB instance always the same one, or could it be any of them?

| username: forever | Original post link

I feel the question is not very clear, I suggest adding more details to the description.

| username: zxgaa | Original post link

Is it a single node reporting an error?

| username: andone | Original post link

Describe in more detail.

| username: Kongdom | Original post link

Are there any other error messages?

| username: oceanzhang | Original post link

Can’t tell anything from this error?

| username: oceanzhang | Original post link

I feel like you haven’t found the original error message.

| username: 田帅萌7 | Original post link

I took a look at it, and it seems like there is an issue with grpc communication. I have deployed TiDB with three central nodes.

| username: 田帅萌7 | Original post link

Cross-data center deployment with three data centers, each with: 1 TiDB, 1 TiKV, and 1 PD. Data center latency: icmp_seq=1 ttl=57 time=0.874 ms

| username: magongyong | Original post link

It doesn’t have much impact, it’s a bug, *: update tikv/client-go to fix batch-client send loop panic issue by crazycs520 · Pull Request #47716 · pingcap/tidb · GitHub
The code is this segment


The last line go a.batchSendLoop(cfg) restarts it again
There should be an info message following this error log, which is restart batchSendLoop

| username: 田帅萌7 | Original post link

The official response is that it will be fixed in version 6.5.6.

| username: magongyong | Original post link

:+1:

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.