Err="other error: Coprocessor task terminated due to exceeding the deadline

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: err="other error: Coprocessor task terminated due to exceeding the deadline

| username: xie123

【TiDB Usage Environment】Production, Testing, Research
Production
【TiDB Version】
V5.4.0
【Encountered Problem】
【Reproduction Path】What operations were performed that led to the issue
sqoop data import to TiDB failed, certain tables became unreadable
【Problem Phenomenon and Impact】
Tables are unreadable
select count(*) query reports error err="other error: Coprocessor task terminated due to exceeding the deadline
tikv keeps reporting errors:
[2022/08/10 10:20:25.116 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]
[2022/08/10 10:20:25.126 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]
[2022/08/10 10:20:25.136 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]
[2022/08/10 10:20:25.137 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]
[2022/08/10 10:20:25.138 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]
[2022/08/10 10:20:25.139 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]
[2022/08/10 10:20:25.139 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]
[2022/08/10 10:20:25.140 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]
[2022/08/10 10:20:25.141 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]
[2022/08/10 10:20:25.162 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]
[2022/08/10 10:20:25.170 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]
[2022/08/10 10:20:25.173 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]
[2022/08/10 10:20:25.283 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]
[2022/08/10 10:20:25.289 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]
[2022/08/10 10:20:25.290 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]
[2022/08/10 10:20:25.291 +08:00] [WARN] [endpoint.rs:606] [error-response] [err=“Coprocessor task terminated due to exceeding the deadline”]

【Attachments】

Please provide the version information of each component, such as cdc/tikv, which can be obtained by executing cdc version/tikv-server --version.

2
| username: xie123 | Original post link

It is now normal, very strange.
Steps to resolve:

  1. Migrate the leader of a certain table’s region. After migration, the query is still abnormal (should be ineffective).
  2. Restart TiDB and see in the logs that a certain TiKV keeps appearing [wait_max_addr=xxx:20162] (query is normal after restart).
    A bit strange, so TiKV keeps spamming “Coprocessor task terminated due to exceeding the deadline”.
3
| username: xie123 | Original post link

After restarting one, none of them refresh.

4
| username: jansu-dev | Original post link
  1. Coprocessor task terminated due to exceeding the deadline is an error returned by TiKV to TiDB;
  2. When selecting data in TiDB, a stream is built between them. If an error occurs in the middle, this kind of log will be flushed in TiDB.

The expected behavior after a restart is normal because after the restart, the gRPC stream no longer exists, and the query request also no longer exists.

It is still necessary to check why there was a timeout. Combine monitoring and logs to see if it exceeded the maximum transaction execution time, or if the coprocessor was fully utilized, etc.

5
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.