TiDB v4 Version Read Hotspot Peer Transfer Leader Causes TiDB OOM

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tidb v4 版本读热点 peer transfer leader 导致 TiDB oom

| username: TiDBer_yyy

【TiDB Usage Environment】Production Environment
【TiDB Version】4.0.16
【Reproduction Path】Unable to reproduce
【Encountered Issues: Phenomenon and Impact】

  1. Read hotspot causing peer transfer leader
    Time: 2023-03-10 04-29:58

  2. Check store load on 8C machines, multiple machines are fully loaded, with store-4609319 having the highest CPU usage

  3. Check hot read information and find that store-4609319 has many hot regions, but not the most.
    Time frame: 4:30 → 4:48

Check scheduling information

  1. 04:51 TiDB OOM, check for sudden memory increase

  2. Before OOM, no SQL with excessive memory usage was found, the maximum single memory usage was 300M, machine memory limit: single 8G, tidb-server memory 64G (single node deployment)

  3. During the execution of the hotspot table, the memory usage was around 1M.
    Machine configuration:
    tidb 8C 64G;
    tikv 8C

During the OOM period, the tidb-server log showed a large number of “TIME_COP_PROCESS” errors:

[root@ tidb-01 log]# grep -E "10 04:31" tidb_bak.log |grep TIME_COP_PROCESS |wc -l
3738
[root@ tidb-01 log]# grep -E "10 04:32" tidb_bak.log |grep TIME_COP_PROCESS |wc -l
4090
[root@ tidb-01 log]# grep -E "10 04:33" tidb_bak.log |grep TIME_COP_PROCESS |wc -l
3144
[root@ tidb-01 log]# grep -E "10 04:34" tidb_bak.log |grep TIME_COP_PROCESS |wc -l
3235
[root@ tidb-01 log]# grep -E "10 04:35" tidb_bak.log |grep TIME_COP_PROCESS |wc -l
3036
[root@ tidb-01 log]# grep -E "10 04:36" tidb_bak.log |grep TIME_COP_PROCESS |wc -l
1959
[root@ tidb-01 log]# grep -E "10 04:37" tidb_bak.log |grep TIME_COP_PROCESS |wc -l
2426
[root@ tidb-01 log]# grep -E "10 04:38" tidb_bak.log |grep TIME_COP_PROCESS |wc -l
1869
[root@ tidb-01 log]# grep -E "10 04:39" tidb_bak.log |grep TIME_COP_PROCESS |wc -l
2010
[root@ tidb-01 log]# grep -E "10 04:40" tidb_bak.log |grep TIME_COP_PROCESS |wc -l
2067
[root@ tidb-01 log]# grep -E "10 04:41" tidb_bak.log |grep TIME_COP_PROCESS |wc -l
2524
[root@ tidb-01 log]# grep -E "10 04:42" tidb_bak.log |grep TIME_COP_PROCESS |wc -l
2582
[root@ tidb-01 log]# grep -E "10 04:43" tidb_bak.log |grep TIME_COP_PROCESS |wc -l
2624
[root@ tidb-01 log]# grep -E "10 04:44" tidb_bak.log |grep TIME_COP_PROCESS |wc -l
2175
[root@ tidb-01 log]# grep -E "10 04:45" tidb_bak.log |grep TIME_COP_PROCESS |wc -l
963
[root@ tidb-01 log]# grep -E "10 04:46" tidb_bak.log |grep TIME_COP_PROCESS |wc -l
781
[root@ tidb-01 log]# grep -E "10 04:47" tidb_bak.log |grep TIME_COP_PROCESS |wc -l
1465
[root@ tidb-01 log]# grep -E "10 04:48" tidb_bak.log |grep TIME_COP_PROCESS |wc -l
669
[root@ tidb-01 log]# grep -E "10 04:49" tidb_bak.log |grep TIME_COP_PROCESS |wc -l
763
[root@ tidb-01 log]# grep -E "10 04:50" tidb_bak.log |grep TIME_COP_PROCESS |wc -l
2055
[root@ tidb-01 log]# grep -E "10 04:51" tidb_bak.log |grep TIME_COP_PROCESS |wc -l
1476

Question:
When a large number of TIME_COP_PROCESS errors occur, does it lead to excessive memory usage?

| username: 裤衩儿飞上天 | Original post link

If a single query did not exceed 300M, was there high concurrency? Check the slow SQL, many executions of the same statement.

Can you provide detailed error messages for this part?

| username: TiDBer_yyy | Original post link

  1. The task is a manually triggered task, and the business feedback concurrency is in single digits. In the Top 10 mem SQL records of the OOM, it is the same query (the 300M SQL mentioned in the post).
  • PS: The issue occurred on March 10th, and the slow query concurrency was not recorded in time. The tool recorded the slow query:

Filtering expensive SQLs by memory shows that with a concurrency of 20, the total memory consumption is 5.66G (at 60s, the SQL execution completes with mem_max around 320M).
The execution plan is as follows:

+--------------------------------+-------------+----------+-----------+---------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------
| id                             | estRows     | actRows  | task      | access object | execution info                                                                                                                                                                                                                                   |  memory   | disk |
+--------------------------------+-------------+----------+-----------+---------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------
| Projection_10                  | 1.00        | 1        | root      |               | time:1m28.8s, loops:2, Concurrency:OFF                                                                                                                                                                                                           | | 26.6 KB  | N/A  |
| └─StreamAgg_12                 | 1.00        | 1        | root      |               | time:1m28.8s, loops:2                                                                                                                                                                                                                            | | 869.5 KB | N/A  |
|   └─Projection_31              | 81926051.37 | 80763365 | root      |               | time:850.6ms, loops:81858, Concurrency:4                                                                                                                                                                                                         | 4.04 MB  | N/A  |
|     └─TableReader_24           | 81926051.37 | 80763365 | root      |               | time:920.6ms, loops:81858, cop_task: {num: 532, max: 679.6ms, min: 163.7ms, avg: 394.6ms, p95: 502.7ms, max_proc_keys: 226313, p95_proc_keys: 167951, tot_proc: 2m50.5s, tot_wait: 383ms, rpc_num: 532, rpc_time: 3m29.9s, copr_cache: disabled} || 303.5 MB | N/A  |
|       └─Selection_23           | 81926051.37 | 80763365 | cop[tikv] |               | tikv_task:{proc max:462ms, min:111ms, p80:305ms, p95:333ms, iters:85457, tasks:532}, scan_detail: {total_process_keys: 85043446, total_keys: 85137833}                                                                                           | 
|         └─TableFullScan_22     | 85043877.00 | 85043446 | cop[tikv] | table:d       | tikv_task:{proc max:457ms, min:110ms, p80:302ms, p95:331ms, iters:85457, tasks:532}                                                                                                                                                              | keep order:false     
+--------------------------------+-------------+----------+-----------+---------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------

SQL

SELECT COUNT(b.I_ID) AS id,
 IFNULL(SUM(b.c1),0) AS c1,
 IFNULL(SUM(b.c2),0) AS c2,
 IFNULL(SUM(b.c3),0) AS c3,
 IFNULL(SUM(c4),0) AS c4,
 IFNULL(SUM(b.c5),0) AS c5,
 IFNULL(SUM(b.c6),0) AS c6,
 IFNULL(SUM(CASE WHEN b.I_STATUS=1 THEN b.c7 ELSE 0 END),0) AS c7,
 COUNT(CASE WHEN b.I_STATUS=1 THEN b.c8 END) c8 ,
 IFNULL(SUM(CASE WHEN b.I_STATUS=1 THEN b.c9 ELSE 0 END),0) AS c9,
 IFNULL(SUM(CASE WHEN b.I_STATUS=1 THEN b.c10 ELSE 0 END),0) AS c10,
 IFNULL(SUM(CASE WHEN b.I_STATUS=11 AND b.c11=111 THEN b.c12 ELSE 0 END), 0) AS c11,
    IFNULL(SUM(CASE WHEN b.I_STATUS=11 AND b.c12=112 THEN b.c22 ELSE 0 END), 0) AS c12,
 IFNULL(SUM(CASE WHEN b.I_STATUS=11 AND b.c13=111 THEN b.c22 ELSE 0 END), 0) AS c12,
    IFNULL(SUM(CASE WHEN b.I_STATUS=11 AND b.c14=112 THEN b.c14 ELSE 0 END), 0) AS c14,
 IFNULL(SUM(CASE WHEN b.I_STATUS=2 OR b.I_STATUS=a THEN b.c1 ELSE 0 END),0) AS c15,
 IFNULL(SUM(CASE WHEN b.I_STATUS=2 OR b.I_STATUS=a THEN b.c16 ELSE 0 END),0) AS c16,
 IFNULL(SUM(CASE WHEN b.I_STATUS=2 OR b.I_STATUS=a THEN b.c17 ELSE 0 END),0) AS c17,
 COUNT(CASE WHEN b.I_STATUS in (a,b,c,d,e,f,g) THEN 1 END) AS c18,
 IFNULL(SUM(CASE WHEN b.I_STATUS in (a,b,c,d,e,f,g) THEN b.c22 ELSE 0 END),0) AS c22,
 COUNT(DISTINCT(CASE WHEN b.I_STATUS in (a,b,c,d,e,f,g) THEN b.c22 END)) c22,
 IFNULL(SUM(CASE WHEN b.I_STATUS in (a,b,c,d,e,f,g1) THEN b.I_TAX END),0) c22 FROM (SELECT b.c1,b.c2,b.c3,b.c3,b.c5,b.c6,b.c7,b.c8,b.c9,b.c10,b.c11 FROM table1 AS b LEFT JOIN table2 AS bro ON b.I_REF = bro.I_REF LEFT JOIN table3 AS e ON b.I_REF = e.I_REF WHERE 1=1 AND b.I_STATUS != -1) AS b
  1. The file is as follows, desensitized
    tidb_bak.log.zip (1.9 MB)
| username: 裤衩儿飞上天 | Original post link

Try to optimize this 300M memory SQL first.
Post the execution plan.

Basically, optimizing this type of SQL can solve subsequent chain issues.

| username: TiDBer_yyy | Original post link

Supplemented, sorry, I can only post the desensitized SQL (PS: the SQL is an old query from a few years ago, and it will be optimized later).

Question: I don’t know where the memory consumption went, how to avoid this problem in the future.

OOM heap.file
untitled.html (122.6 KB)

| username: 裤衩儿飞上天 | Original post link

For each query, there’s a full table scan on an 800 million row table. Solving this would be enough.

  1. Check if any query conditions are missing.
  2. Use appropriate indexes.
| username: TiDBer_yyy | Original post link

Okay, SQL issue. Let’s optimize it. But the root cause of the memory surge is still not found~

| username: 裤衩儿飞上天 | Original post link

Upload this heap profile.

| username: TiDBer_yyy | Original post link

Before the OOM, the heap profile was recorded 5 times consecutively:

| username: 裤衩儿飞上天 | Original post link

  1. Located three functions with high memory consumption

  2. After TiDB version 6.5.0, the OOM issue has been greatly improved. The previous versions indeed had serious OOM problems.

  3. You can try upgrading to the new version.

| username: TiDBer_yyy | Original post link

Thank you, expert.

  1. Previously, I also saw these functions consuming a lot of memory when looking at the heap profile. I asked the devops colleagues to check, but they couldn’t figure out why the consumption was high and how to avoid it.

  2. We plan to upgrade to a new version this year. The 4.0 TLS version is no longer supported; also, the memory statistics of versions before 6.5 are not very accurate. Could it be due to incorrect memory statistics (the 4.0 version cluster does not have TiFlash, so I don’t know if mem_max records the data_buffer memory block)?

  3. Testing the 5.0 version cluster using TiFlash for queries only used 26KB, so temporarily let the development team switch to the 5.0 version cluster.

| username: TiDBer_yyy | Original post link

The reason for excessive memory usage in the recvMsg function provided by the community members:

  • The SQL part occupies a large amount of memory when receiving data:
    COUNT(DISTINCT(CASE WHEN b.I_STATUS in (a,b,c,d,e,f,g) THEN b.c22 END)) c22
    In actual tests, removing this part of the execution plan indeed significantly reduces memory consumption.

  • There are also issues with:
    set.StringSet.Insert and stringUtil.Copy. Where do these parts usually occupy memory?

| username: 人如其名 | Original post link

This is likely due to a memory surge caused by too many unique strings when you perform a count distinct operation on strings. You might want to check if the value for count(distinct xxx) is relatively large.

| username: TiDBer_yyy | Original post link

Yes

Field type: varchar(160), this field does not have many duplicates.
+------------------------+
| COUNT                  |
+------------------------+
| 11778040               |
+------------------------+
| username: 人如其名 | Original post link

When performing count distinct on strings, you can refer to this: tidb/executor/aggfuncs/func_count_distinct.go at ee7c089e55e6d95425c745c8a9073632aa5e760e · pingcap/tidb · GitHub

The p.valSet.Insert(input) here should correspond to your set.StringSet.Insert. If the distinct values are too large, it will always follow the logic of emDelta += p.valSet.Insert(input), causing memory to keep growing. This does not support disk spilling.

The stringUtil.Copy in the code should be the logic of input = stringutil.Copy(input) here. I don’t understand why a deep copy operation is done here. The high memory usage here might be due to the input’s GC not being timely.

| username: TiDBer_yyy | Original post link

Sure, thank you.

To eliminate this issue, you can reduce the count(distinct col) aggregation calculation on a large amount of different data.

This query has been online for several years and has not been optimized. During operation, it is manually clicked by the operations team, and the R&D team has identified the concurrency to be in the single digits (PS: the query’s mem_max = 300M+). Therefore, this time the OOM feels very strange. There was no OOM before, but this time with slightly higher concurrency, it crashed.

| username: 裤衩儿飞上天 | Original post link

  1. You can optimize this SQL from the business logic level. Scanning over 80 million records and only returning one result indicates there is room for optimization.
  2. Check if there are other slow SQL queries, especially those that scan large tables entirely.
  3. Another approach is to consider upgrading.
| username: tidb菜鸟一只 | Original post link

My suggestion is to try using TiFlash. Your SQL is clearly an OLAP requirement.

| username: TiDBer_yyy | Original post link

Got it!!!
2. Querying large tables with scans, but the memory usage is not significant, so it was initially overlooked. This query caused hotspot issues, and the three tables in this query match the three tables in the heatmap.


Partial execution plan, the complete SQL is too long to fit here. The hotspot caused by this query has a SQL mem-max of 147.1 MiB.

Current situation:
After multiple TiKV nodes in the cluster experienced hotspots, regions started to schedule, and then TiDB encountered an OOM.

id                                                       	task     	estRows     	           memory   	disk
	Projection_116                                           	root     	20                     30.3 KB  	N/A
	└─TopN_119                                               	root     	20                     30.8 KB  	N/A
	  └─Union_123                                            	root     	120                    N/A      	N/A
	    ├─Projection_124                                     	root     	20                     31.5 KB  	N/A
	    │ └─Projection_622                                   	root     	20                     31.9 KB  	N/A
	    │   └─TopN_127                                       	root     	20                     32.1 KB  	N/A
	    │     └─Projection_623                               	root     	20                     31.5 KB  	N/A
	    │       └─Projection_128                             	root     	20                     29.1 KB  	N/A
	    │         └─TopN_131                                 	root     	20                     29.3 KB  	N/A
	    │           └─IndexJoin_138                          	root     	20                     549.1 KB 	N/A
	    │             ├─TopN_145                             	root     	20                     28.8 KB  	N/A
	    │             │ └─IndexJoin_152                      	root     	20.03                  524.8 KB 	N/A
	    │             │   ├─TopN_159                         	root     	20                     27.8 KB  	N/A
	    │             │   │ └─IndexJoin_166                  	root     	20.31                  459.0 KB 	N/A
	    │             │   │   ├─TopN_173                     	root     	20                     24.9 KB  	N/A
	    │             │   │   │ └─IndexJoin_181              	root     	20.99                  450.7 KB 	N/A
	    │             │   │   │   ├─TopN_186                 	root     	20                     273.2 KB 	N/A
	    │             │   │   │   │ └─TableReader_195        	root     	20                     28.6 KB  	N/A
	    │             │   │   │   │   └─TopN_194             	cop[tikv]	20                     N/A      	N/A
	    │             │   │   │   │     └─Selection_193      	cop[tikv]	81088501.84            N/A      	N/A
	    │             │   │   │   │       └─TableFullScan_192	cop[tikv]	84872190               N/A      	N/A
	    │             │   │   │   └─IndexLookUp_180          	root     	1.05                   241 Bytes	N/A
	    │             │   │   │     ├─IndexRangeScan_177     	cop[tikv]	1.09                   N/A      	N/A
	    │             │   │   │     └─Selection_179          	cop[tikv]	1.05                   	N/A      	N/A
	    │             │   │   │       └─TableRowIDScan_178   	cop[tikv]	1.09                   N/A      	N/A
	    │             │   │   └─IndexLookUp_165              	root     	1                      249 Bytes	N/A
	    │             │   │     ├─IndexRangeScan_163         	cop[tikv]	1                      N/A      	N/A
	    │             │   │     └─TableRowIDScan_164         	cop[tikv]	1                      N/A      	N/A
	    │             │   └─IndexLookUp_151                  	root     	1                      237 Bytes	N/A
	    │             │     ├─IndexRangeScan_149             	cop[tikv]	1                      N/A      	N/A
	    │             │     └─TableRowIDScan_150             	cop[tikv]	1                      N/A      	N/A
	    │             └─IndexLookUp_137                      	root     	1                      0 Bytes  	N/A
	    │               ├─IndexRangeScan_135                 	cop[tikv]	1                      N/A      	N/A
	    │               └─TableRowIDScan_136                 	cop[tikv]	1                      N/A      	N/A
	    ├─Projection_223                                     	root     	20                     31.0 KB  	N/A
	    │ └─Projection_624                                   	root     	20                     31.4 KB  	N/A
	    │   └─TopN_226                                       	root     	20                     31.6 KB  	N/A
	    │     └─Projection_625                               	root     	20                     31.0 KB  	N/A
	    │       └─Projection_227                             	root     	20                     25.8 KB  	N/A
	    │         └─TopN_230                                 	root     	20                     26.0 KB  	N/A
	    │           └─IndexJoin_237                          	root     	20                     479.8 KB 	N/A
	    │             ├─TopN_244                             	root     	20                     25.5 KB  	N/A
	    │             │ └─IndexJoin_251                      	root     	20.03                  455.4 KB 	N/A
	    │             │   ├─TopN_258                         	root     	20                     24.5 KB  	N/A
	    │             │   │ └─IndexJoin_265                  	root     	20.31                  389.6 KB 	N/A
	    │             │   │   ├─TopN_272                     	root     	20                     21.6 KB  	N/A
	    │             │   │   │ └─IndexJoin_280              	root     	20.99                  381.3 KB 	N/A
	    │             │   │   │   ├─TopN_285                 	root     	20                     214.9 KB 	N/A
	    │             │   │   │   │ └─TableReader_294        	root     	20                     26.1 KB  	N/A
	    │             │   │   │   │   └─TopN_293             	cop[tikv]	20                     N/A      	N/A
	    │             │   │   │   │     └─Selection_292      	cop[tikv]	130770487.           , N/A      	N/A
	    │             │   │   │   │       └─TableFullScan_291	cop[tikv]	249297318.           , N/A      	N/A
	    │             │   │   │   └─IndexLookUp_279          	root     	1.05                   241 Bytes	N/A
	    │             │   │   │     ├─IndexRangeScan_276     	cop[tikv]	1.09                   N/A      	N/A
	    │             │   │   │     └─Selection_278          	cop[tikv]	1.05                   0        	                                                                                                                                                                                                                                                                                                          	N/A      	N/A
	    │             │   │   │       └─TableRowIDScan_277   	cop[tikv]	1.09                   N/A      	N/A
	    │             │   │   └─IndexLookUp_264              	root     	1                      241 Bytes	N/A
	    │             │   │     ├─IndexRangeScan_262         	cop[tikv]	1                      N/A      	N/A
	    │             │   │     └─TableRowIDScan_263         	cop[tikv]	1                      N/A      	N/A
	    │             │   └─IndexLookUp_250                  	root     	1                      241 Bytes	N/A
	    │             │     ├─IndexRangeScan_248             	cop[tikv]	1                      N/A      	N/A
	    │             │     └─TableRowIDScan_249             	cop[tikv]	1                      N/A      	N/A
	    │             └─IndexLookUp_236                      	root     	1                      0 Bytes  	N/A
	    │               ├─IndexRangeScan_234                 	cop[tikv]	1                      N/A      	N/A
	    │               └─TableRowIDScan_235                 	cop[tikv]	1                      0        	                                                                                                                                                                                                                                                                                                          	N/A      	N/A
	    ├─Projection_322                                     	root     	20                     30.8 KB  	N/A
	    │ └─Projection_626                                   	root     	20                     31.2 KB  	N/A
	    │   └─TopN_325                                       	root     	20                     31.3 KB  	N/A
	    │     └─Projection_627                               	root     	20                     30.8 KB  	N/A
	    │       └─Projection_326                             	root     	20                     25.6 KB  	N/A
	    │         └─TopN_329                                 	root     	20                     25.7 KB  	N/A
	    │           └─IndexJoin_336                          	root     	20                     479.8 KB 	N/A
	    │             ├─TopN_343                             	root     	20                     25.3 KB  	N/A
	    │             │ └─IndexJoin_350                      	root     	20.03                  455.4 KB 	N/A
	    │             │   ├─TopN_357                         	root     	20                     24.3 KB  	N/A
	    │             │   │ └─IndexJoin_364                  	root     	20.31                  389.6 KB 	N/A
	    │             │   │   ├─TopN_371                     	root     	20                     21.4 KB  	N/A
	    │             │   │   │ └─IndexJoin_379              	root     	20.99                  406.0 KB 	N/A
	    │             │   │   │   ├─TopN_384                 	root     	20                     152.4 KB 	N/A
	    │             │   │   │   │ └─TableReader_393        	root     	20                     25.6 KB  	N/A
	    │             │   │   │   │   └─TopN_392             	cop[tikv]	20                     N/A      	N/A
	    │             │   │   │   │     └─Selection_391      	cop[tikv]	772532964              N/A      	N/A
	    │             │   │   │   │       └─TableFullScan_390	cop[tikv]	772532964              N/A      	N/A
	    │             │   │   │   └─IndexLookUp_378          	root     	1.05                   9.18 KB  	N/A
	    │             │   │   │     ├─IndexRangeScan_375     	cop[tikv]	1.09                   N/A      	N/A
	    │             │   │   │     └─Selection_377          	cop[tikv]	1.05                   N/A      	N/A
	    │             │   │   │       └─TableRowIDScan_376   	cop[tikv]	1.09                   N/A      	N/A
	    │             │   │   └─IndexLookUp_363              	root     	1                      243 Bytes	N/A
	    │             │   │     ├─IndexRangeScan_361         	cop[tikv]	1                      N/A      	N/A
	    │             │   │     └─TableRowIDScan_362         	cop[tikv]	1                      N/A      	N/A
	    │             │   └─IndexLookUp_349                  	root     	1                      237 Bytes	N/A
	    │             │     ├─IndexRangeScan_347             	cop[tikv]	1                      N/A      	N/A
	    │             │     └─TableRowIDScan_348             	cop[tikv]	1                      N/A      	N/A
	    │             └─IndexLookUp_335                      	root     	1                      0 Bytes  	N/A
	    │               ├─IndexRangeScan_333                 	cop[tikv]	1                      N/A      	N/A
	    │               └─TableRowIDScan_334                 	cop[tikv]	1                      0        	                                                                                                                                                                                                                                                                                                          	N/A      	N/A
	    ├─Projection_421                                     	root     	20                     30.0 KB  	N/A
	    │ └─Projection_628                                   	root     	20                     30.4 KB  	N/A
	    │   └─TopN_424                                       	root     	20                     30.6 KB  	N/A
	    │     └─Projection_629                               	root     	20                     30.0 KB  	N/A
	    │       └─Projection_425                             	root     	20                     18.7 KB  	N/A
	    │         └─TopN_428                                 	root     	20                     18.8 KB  	N/A
	    │           └─IndexJoin_435                          	root     	20                     332.3 KB 	N/A
	    │             ├─TopN_442                             	root     	20                     18.3 KB  	N/A
	    │             │ └─IndexJoin_449                      	root     	20.03                  308.0 KB 	N/A
	    │             │   ├─TopN_456                         	root     	20                     17.4 KB  	N/A
	    │             │   │ └─IndexJoin_463                  	root     	20.31                  242.1 KB 	N/A
	    │             │   │   ├─TopN_468                     	root     	20                     1.06 MB  	N/A
	    │             │   │   │ └─TableReader_477            	root     	20                     21.3 KB  	N/A
	    │             │   │   │   └─TopN_476                 	cop[tikv]	20                     N/A      	N/A
	    │             │   │   │     └─Selection_475          	cop[tikv]	172629344.           , N/A      	N/A
	    │             │   │   │       └─TableFullScan_474    	cop[tikv]	172629344              N/A      	N/A
	    │             │   │   └─IndexLookUp_462              	root     	1                      241 Bytes	N/A
	    │             │   │     ├─IndexRangeScan_460         	cop[tikv]	1                      N/A      	N/A
	    │             │   │     └─TableRowIDScan_461         	cop[tikv]	1                      N/A      	N/A
	    │             │   └─IndexLookUp_448                  	root     	1                      241 Bytes	N/A
	    │             │     ├─IndexRangeScan_446             	cop[tikv]	1                      N/A      	N/A
	    │             │     └─TableRowIDScan_447             	cop[tikv]	1                      N/A      	N/A
	    │             └─IndexLookUp_434                      	root     	1                      0 Bytes  	N/A
	    │               ├─IndexRangeScan_432                 	cop[tikv]	1                      N/A      	N/A
	    │               └─TableRowIDScan_433                 	cop[tikv]	1                      N/A      	N/A
	    ├─Projection_494                                     	root     	20                     30.5 KB  	N/A
	    │ └─Projection_630                                   	root     	20                     30.9 KB  	N/A
	    │   └─TopN_497                                       	root     	20                     31.1 KB  	N/A
	    │     └─Projection_631                               	root     	20                     30.5 KB  	N/A
	    │       └─Projection_498                             	root     	20                     19.5 KB  	N/A
	    │         └─TopN_501                                 	root     	20                     19.7 KB  	N/A
	    │           └─IndexJoin_508                          	root     	20                     344.2 KB 	N/A
	    │             ├─TopN_515                             	root     	20                     19.2 KB  	N/A
	    │             │ └─IndexJoin_522                      	root     	20.03                  319.9 KB 	N/A
	    │             │   ├─TopN_529                         	root     	20                     18.2 KB  	N/A
	    │             │   │ └─IndexJoin_536                  	root     	20.31                  254.1 KB 	N/A
	    │             │   │   ├─TopN_541                     	root     	20                     10.5 KB  	N/A
	    │             │   │   │ └─TableReader_550            	root     	20                     9.26 KB  	N/A
	    │             │   │   │   └─TopN_549                 	cop[tikv]	20                     N/A      	N/A
	    │             │   │   │     └─Selection_548          	cop[tikv]	1165                   N/A      	N/A
	    │             │   │   │       └─TableRangeScan_547   	cop[tikv]	1165                   N/A      	N/A
	    │             │   │   └─IndexLookUp_
| username: TiDBer_yyy | Original post link

Got it.

Come up with a strategy to collect business AP queries and migrate the AP queries to TiFlash.