TiKV Continuously Reports "call GetStoreSafeTS failed"

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiKV持续报“call GetStoreSafeTS failed”

| username: TiDBer_wX9akOFm

【TiDB Usage Environment】Production Environment
【TiDB Version】v7.5.0
【Encountered Issue: Phenomenon and Impact】
The following information is continuously output in the TiKV logs. I’m not sure what the cause is or how to resolve it.
【Attachment: Screenshot/Log/Monitoring】

[2024/05/13 15:51:56.244 +08:00] [INFO] [kv.rs:955] [“kv rpc failed”] [err=RemoteStopped] [request=batch_commands] [thread_id=0x5]
[2024/05/13 15:51:59.703 +08:00] [WARN] [kv.rs:1065] [“call GetStoreSafeTS failed”] [err=Grpc(RemoteStopped)] [thread_id=0x5]
[2024/05/13 15:52:01.780 +08:00] [WARN] [kv.rs:1065] [“call GetStoreSafeTS failed”] [err=Grpc(RemoteStopped)] [thread_id=0x5]
[2024/05/13 15:52:07.941 +08:00] [INFO] [compaction_filter.rs:625] [“Compaction filter reports”] [filtered=36074] [total=347280] [thread_id=0x5]
[2024/05/13 15:52:18.348 +08:00] [INFO] [trend.rs:291] [“history window flipping: enter”] [increasing_rate=-78333.77012258547] [flip_margin_error=4083.6295946947325] [delta=4190.166666666657] [name=L2] [thread_id=0x5]
[2024/05/13 15:52:23.256 +08:00] [INFO] [kv.rs:955] [“kv rpc failed”] [err=RemoteStopped] [request=batch_commands] [thread_id=0x5]
[2024/05/13 15:52:42.867 +08:00] [WARN] [kv.rs:1065] [“call GetStoreSafeTS failed”] [err=Grpc(RemoteStopped)] [thread_id=0x5]
[2024/05/13 15:52:48.351 +08:00] [INFO] [trend.rs:272] [“history window flipping: end”] [flipping_duration=30] [increasing_rate=0] [time_based_multiple=0.00849625007211562] [flip_margin_error=2182.864407340349] [delta=2693.333333333343] [name=L2] [thread_id=0x5]
[2024/05/13 15:52:48.546 +08:00] [WARN] [kv.rs:1065] [“call GetStoreSafeTS failed”] [err=Grpc(RemoteStopped)] [thread_id=0x5]
[2024/05/13 15:53:05.765 +08:00] [INFO] [kv.rs:955] [“kv rpc failed”] [err=RemoteStopped] [request=batch_commands] [thread_id=0x5]
[2024/05/13 15:53:05.765 +08:00] [INFO] [kv.rs:955] [“kv rpc failed”] [err=RemoteStopped] [request=batch_commands] [thread_id=0x5]
[2024/05/13 15:53:05.829 +08:00] [WARN] [kv.rs:1065] [“call GetStoreSafeTS failed”] [err=Grpc(RemoteStopped)] [thread_id=0x5]
[2024/05/13 15:53:11.270 +08:00] [INFO] [kv.rs:955] [“kv rpc failed”] [err=RemoteStopped] [request=batch_commands] [thread_id=0x5]
[2024/05/13 15:53:11.270 +08:00] [INFO] [kv.rs:955] [“kv rpc failed”] [err=RemoteStopped] [request=batch_commands] [thread_id=0x5]

2
| username: 像风一样的男子 | Original post link

Info logs can be ignored.

3
| username: 友利奈绪 | Original post link

It doesn’t seem to be an error message.

4
| username: TiDBer_QYr0vohO | Original post link

These are all info-level logs, nothing problematic.

5
| username: ojbk | Original post link

Adjusting the gRPC timeout might reduce the frequency. Is there any accompanying memory growth?

6
| username: shigp_TIDBER | Original post link

Warning level, not a big issue.

7
| username: TiDBer_wX9akOFm | Original post link

There doesn’t seem to be any growth trend in memory usage.

8
| username: TiDBer_wX9akOFm | Original post link

Although it is at the info level, it keeps flooding the screen and contains the word “fail,” so I am still worried about potential risks. If you have any ideas, it might be better to address it.

9
| username: TiDBer_wX9akOFm | Original post link

There are still many WARN level logs, continuously flooding the screen.
[2024/05/13 20:58:31.961 +08:00] [WARN] [scanner.rs:137] [“resolved_ts scan get snapshot failed”] [err=“Other("[components/resolved_ts/src/scanner.rs:193]: scan task cancelled")”] [thread_id=0x5]
[2024/05/13 20:58:32.633 +08:00] [WARN] [scanner.rs:137] [“resolved_ts scan get snapshot failed”] [err=“Other("[components/resolved_ts/src/scanner.rs:193]: scan task cancelled")”] [thread_id=0x5]
[2024/05/13 20:58:33.311 +08:00] [WARN] [scanner.rs:137] [“resolved_ts scan get snapshot failed”] [err=“Other("[components/resolved_ts/src/scanner.rs:193]: scan task cancelled")”] [thread_id=0x5]
[2024/05/13 20:58:33.999 +08:00] [WARN] [scanner.rs:137] [“resolved_ts scan get snapshot failed”] [err=“Other("[components/resolved_ts/src/scanner.rs:193]: scan task cancelled")”] [thread_id=0x5]
[2024/05/13 20:58:33.999 +08:00] [WARN] [scanner.rs:137] [“resolved_ts scan get snapshot failed”] [err=“Other("[components/resolved_ts/src/scanner.rs:193]: scan task cancelled")”] [thread_id=0x5]
[2024/05/13 20:58:34.677 +08:00] [WARN] [scanner.rs:137] [“resolved_ts scan get snapshot failed”] [err=“Other("[components/resolved_ts/src/scanner.rs:193]: scan task cancelled")”] [thread_id=0x5]
[2024/05/13 20:58:34.677 +08:00] [WARN] [scanner.rs:137] [“resolved_ts scan get snapshot failed”] [err=“Other("[components/resolved_ts/src/scanner.rs:193]: scan task cancelled")”] [thread_id=0x5]
[2024/05/13 20:58:35.348 +08:00] [WARN] [scanner.rs:137] [“resolved_ts scan get snapshot failed”] [err=“Other("[components/resolved_ts/src/scanner.rs:193]: scan task cancelled")”] [thread_id=0x5]
[2024/05/13 20:58:35.349 +08:00] [WARN] [scanner.rs:137] [“resolved_ts scan get snapshot failed”] [err=“Other("[components/resolved_ts/src/scanner.rs:193]: scan task cancelled")”] [thread_id=0x5]

10
| username: Jack-li | Original post link

There is no important log information.

11
| username: zhh_912 | Original post link

You don’t need to worry about logs at the warn and info levels, mainly focus on the error level logs.