Information schema is out of date: schema failed to update in 1lease, please make sure TiDB can connect to TiKV

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: Information schema is out of date: schema failed to update in 1lease, please make sure TiDB can connect to TikV

| username: AeolusX

[TiDB Usage Environment] Production Environment
[TiDB Version] V5.2.1
[Reproduction Path] Updated daily
[Encountered Problem: Phenomenon and Impact] 9 o’clock is the critical point, summarizing from 9 o’clock the previous day to the current time. The data volume is very large in the morning, and writing to TiDB is very slow, causing it to get stuck. The data volume is 4 million records inserted in 10 minutes.
[Resource Configuration] 6 PD, 6 TiDB, 96 TiKV, 48 TiFlash; hardware configuration is 64 cores, 512G memory; 6 SSD solid-state drives, each TiKV instance corresponds to one SSD.


The cluster has a total of 24 servers.
[Attachments:







Could the experts please advise if there are any optimization solutions for this issue?

| username: tidb菜鸟一只 | Original post link

Are PD and TiDB on the same machine? Is it necessary to have 6 PD nodes? Wouldn’t 3 be enough… Also, are multiple TiKV instances on the same node isolated by NUMA for resource allocation?

| username: AeolusX | Original post link

PD and TiDB are on the same machine, and the TiKV on the same machine does not have NUMA resource isolation.

| username: Hacker007 | Original post link

Check if the IO is full.

| username: 像风一样的男子 | Original post link

It seems that the QPS for insert operations is not very high. Are the disk I/O, server CPU, and memory all normal?

| username: AeolusX | Original post link

The maximum disk can reach 4k IOPS.

| username: 像风一样的男子 | Original post link

IOPS is not very high either, it feels like a hotspot write issue.

| username: 大飞哥online | Original post link

  1. Monitor and identify which TiKV’s disk I/O is high.
  2. Check if there is an I/O bottleneck and if there is a hotspot write issue.
| username: cassblanca | Original post link

Found another user with the same issue. The analysis indicates that TiKV is under too much pressure. Cause: java.sql.SQLException: Information schema is out of date: schema failed to update in 1 lease, please make sure TiDB can connect to TiKV - :ringer_planet: TiDB Technical Issues - TiDB Q&A Community (asktug.com) Based on your description, it could also be due to concentrated write pressure at the same time, causing hotspot issues.

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.