The same query produces inconsistent results when executed multiple times!

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 同一个查询语句,多次执行结果不一致!!!

| username: TiDBer_rrt4FOef

[TiDB Usage Environment]
Production Environment

[TiDB Version]
V6.5.0

[Resource Configuration]
tidb: 3 nodes, pd: 3 nodes, tikv: 5 nodes

[Problem Description]
The same query statement produces inconsistent results each time it is executed!
The SQL statement is as follows:

select count(t.card_nbr) as balance from t_org t2,
(
select c.ab_user, a.card_nbr, a.active_day,
row_number() OVER(PARTITION BY a.xaccount ORDER by a.cdindex DESC) last_card
from f_ccs_card a, F_CCS_ACCT b, f_ccs_apma c
where a.xaccount = b.xaccount and a.xaccount = c.account
and b.close_code != 'W' and a.active_day <= '20230112' and a.active_day > 0
)
t where
trim(t.ab_user) = trim(t2.code) AND
t.last_card = 1;

The result fluctuates between 12504 and 12507 each time it is executed.
During testing, it was found that the result is consistent when only one condition is used, as follows:

where trim(t.ab_user) = trim(t2.code)
or
where t.last_card = 1

However, when both conditions are used together, the result fluctuates between 12504 and 12507.
Is this caused by the distributed multi-node environment? Is there any way to avoid this?

[Update]
The desensitized test data and the test SQL are as follows; the issue can be reproduced in my environment.
test.sql (1.9 MB)

Test SQL:

select count(t.xaccount) as balance from t_org t2,
(
select c.ab_user, a.xaccount,
row_number() OVER(PARTITION BY a.xaccount ORDER by a.cdindex DESC) last_card
from f_ccs_card a, f_ccs_apma c
where a.xaccount = c.account
)
t where
t.ab_user = t2.code and
t.last_card = 1;

Can anyone reproduce this issue in their environment?
Please help analyze what the problem might be?

| username: tidb菜鸟一只 | Original post link

Have you used TiFlash? Please post the execution plan.

| username: TiDBer_rrt4FOef | Original post link

No TiFlash deployed
Execution plan:

mysql>  explain select count(t.card_nbr) as balance from  t_org t2
,
(
select c.ab_user,a.card_nbr,a.active_day,
row_number() OVER(PARTITION BY a.xaccount ORDER by a.cdindex DESC) last_card
 from f_ccs_card a,F_CCS_ACCT b,f_ccs_apma c
  where a.xaccount=b.xaccount and a.xaccount = c.account
   and b.close_code !='W ' and a.active_day<='20230112' and a.active_day>0
)
 t where  
trim(t.ab_user)=trim(t2.code)  AND 
t.last_card = 1;
+----------------------------------------------+----------+-----------+----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id                                           | estRows  | task      | access object                    | operator info                                                                                                                                                                               |
+----------------------------------------------+----------+-----------+----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| HashAgg_29                                   | 1.00     | root      |                                  | funcs:count(busidata.f_ccs_card.card_nbr)->Column#600                                                                                                                                       |
| └─HashJoin_33                                | 12633.19 | root      |                                  | inner join, equal:[eq(Column#601, Column#602)]                                                                                                                                              |
|   ├─Projection_34(Build)                     | 71.00    | root      |                                  | trim(dgb-datamid.t_org.code)->Column#601                                                                                                                                                    |
|   │ └─TableReader_36                         | 71.00    | root      |                                  | data:TableFullScan_35                                                                                                                                                                       |
|   │   └─TableFullScan_35                     | 71.00    | cop[tikv] | table:t_org                      | keep order:false, stats:pseudo                                                                                                                                                              |
|   └─Projection_37(Probe)                     | 10106.55 | root      |                                  | busidata.f_ccs_card.card_nbr, trim(busidata.f_ccs_apma.ab_user)->Column#602                                                                                                                 |
|     └─Selection_38                           | 10106.55 | root      |                                  | eq(Column#595, 1)                                                                                                                                                                           |
|       └─Shuffle_69                           | 12633.19 | root      |                                  | execution info: concurrency:5, data sources:[HashJoin_41]                                                                                                                                   |
|         └─Window_39                          | 12633.19 | root      |                                  | row_number()->Column#595 over(partition by busidata.f_ccs_card.xaccount order by busidata.f_ccs_card.cdindex desc rows between current row and current row)                                 |
|           └─Sort_68                          | 12633.19 | root      |                                  | busidata.f_ccs_card.xaccount, busidata.f_ccs_card.cdindex:desc                                                                                                                              |
|             └─HashJoin_41                    | 12633.19 | root      |                                  | inner join, equal:[eq(busidata.f_ccs_card.xaccount, busidata.f_ccs_apma.account)]                                                                                                           |
|               ├─IndexJoin_46(Build)          | 10349.89 | root      |                                  | inner join, inner:IndexLookUp_45, outer key:busidata.f_ccs_card.xaccount, inner key:busidata.f_ccs_acct.xaccount, equal cond:eq(busidata.f_ccs_card.xaccount, busidata.f_ccs_acct.xaccount) |
|               │ ├─TableReader_60(Build)      | 10310.32 | root      |                                  | data:Selection_59                                                                                                                                                                           |
|               │ │ └─Selection_59              | 10310.32 | cop[tikv] |                                  | gt(busidata.f_ccs_card.active_day, 0), le(busidata.f_ccs_card.active_day, 20230112)                                                                                                         |
|               │ │   └─TableFullScan_58        | 16546.00 | cop[tikv] | table:a                          | keep order:false                                                                                                                                                                            |
|               │ └─IndexLookUp_45(Probe)      | 10310.32 | root      |                                  |                                                                                                                                                                                             |
|               │   ├─IndexRangeScan_42(Build) | 10310.32 | cop[tikv] | table:b, index:PRIMARY(XACCOUNT) | range: decided by [eq(busidata.f_ccs_acct.xaccount, busidata.f_ccs_card.xaccount)], keep order:false                                                                                        |
|               │   └─Selection_44(Probe)      | 10310.32 | cop[tikv] |                                  | ne(busidata.f_ccs_acct.close_code, "W ")                                                                                                                                                    |
|               │     └─TableRowIDScan_43      | 10310.32 | cop[tikv] | table:b                          | keep order:false                                                                                                                                                                            |
|               └─TableReader_66(Probe)        | 20040.00 | root      |                                  | data:Selection_65                                                                                                                                                                           |
|                 └─Selection_65               | 20040.00 | cop[tikv] |                                  | not(isnull(busidata.f_ccs_apma.account))                                                                                                                                                    |
|                   └─TableFullScan_64         | 20040.00 | cop[tikv] | table:c                          | keep order:false                                                                                                                                                                            |
+----------------------------------------------+----------+-----------+----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
22 rows in set

| username: onlyacat | Original post link

Could you provide the table structure and sample data?

Let’s see if it can be reproduced.

| username: 裤衩儿飞上天 | Original post link

Is it a partitioned table?

| username: forever | Original post link

Does Cdindex have duplicate data?

| username: TiDBer_rrt4FOef | Original post link

None of the tables are partitioned tables. t_org is a view that points to a table in another database.

| username: TiDBer_rrt4FOef | Original post link

The cdindex field does have duplicate data.

| username: TiDBer_rrt4FOef | Original post link

Because there is sensitive data, I can’t provide the table structure and sample data.
I am trying to create some similar tables to see if I can reproduce it.

| username: TiDBer_rrt4FOef | Original post link

The desensitized test data is as follows:
test.sql (1.9 MB)

Test SQL:

select count(t.xaccount) as balance from t_org t2
,
(
select c.ab_user,a.xaccount,
row_number() OVER(PARTITION BY a.xaccount ORDER by a.cdindex DESC) last_card
 from f_ccs_card a,f_ccs_apma c
   where a.xaccount = c.account
)
 t where  
  t.ab_user=t2.code  and
  t.last_card = 1
;

I executed it on the test database and was able to reproduce the issue.
The result fluctuates between 19892 and 19894.

| username: forever | Original post link

The order is based on cdindex sorting. The varying speeds of data returned by different nodes in a distributed database may result in different orders each time. Therefore, the position of the same data sorted by cdindex each time may differ, and the first piece of data retrieved each time might not be the same. When only the first condition is used, the result is always the full dataset, so it remains consistent each time. When only the second condition is used, it always retrieves all the data with 1 from t2, and the Cartesian product ensures consistency each time. You can use a unique condition to sort, and the results will be consistent.

| username: TiDBer_rrt4FOef | Original post link

I tested it again, and even after changing the sorting field in the window function to a unique condition, the issue persists, and the results still vary :face_exhaling:

What else could be the problem?

| username: forever | Original post link

Could you please share the test results and the data statistics of the sorted columns?

| username: windtalker | Original post link

Based on this test data, the minimal reproducible case shows that the result of this SQL is unstable:

select
a.xaccount, 
a.cdindex,  
row_number() OVER(PARTITION BY a.xaccount ORDER by a.cdindex DESC) last_card,
c.ab_user
from f_ccs_card a, f_ccs_apma c
where a.xaccount = c.account
order by xaccount, cdindex desc, last_card, ab_user;

However, this instability is expected because the combination of <a.xaccount, a.cdindex> is not unique in this join result set, which causes the correspondence between ab_user and last_card to be unstable. The original SQL has filtering/join conditions on ab_user and last_card, which leads to the overall SQL result being unstable.

The non-uniqueness of the combination <a.xaccount, a.cdindex> in this join result set is observed through the following comparison:

mysql> select count(*) from (select distinct a.xaccount, a.cdindex from f_ccs_card a, f_ccs_apma c where a.xaccount = c.account) x;
+----------+
| count(*) |
+----------+
|    16489 |
+----------+
1 row in set (0.02 sec)

mysql> select count(*) from (select a.xaccount, a.cdindex from f_ccs_card a, f_ccs_apma c where a.xaccount = c.account) x;
+----------+
| count(*) |
+----------+
|    16575 |
+----------+
1 row in set (0.02 sec)

It can be seen that the result after deduplication of a.xaccount and a.cdindex is different from the result without deduplication.

| username: TiDBer_rrt4FOef | Original post link

This SQL is stable in my environment, and the number of rows returned is the same each time.

Additionally, I don’t quite understand this explanation. I think this instability is unexpected.
The test statement used:

select count(t.xaccount) as balance from t_org t2
,
(
select c.ab_user,a.xaccount,
row_number() OVER(PARTITION BY a.xaccount ORDER by a.cdindex DESC) last_card
 from f_ccs_card a,f_ccs_apma c
   where a.xaccount = c.account
)
 t where  
  t.ab_user=t2.code  and
  t.last_card = 1
;
  1. The combination of <a.xaccount, a.cdindex> is indeed not unique, but the window function has already specified order by, so
select c.ab_user,a.xaccount,
row_number() OVER(PARTITION BY a.xaccount ORDER by a.cdindex DESC) last_card
 from f_ccs_card a,f_ccs_apma c
   where a.xaccount = c.account

The last_card of each record in this subquery must be fixed, and the result set is expected to be stable. Executing this subquery alone also results in a fixed result set.

  1. The result of the join query
    t2 is fixed, and the result set of t is also fixed. When two fixed data tables are joined, the returned result set is expected to be fixed.

  2. Comparison with execution results in other databases
    The result set is stable when executed in Oracle, and also stable in Java’s H2.
    Window functions are only supported in MySQL 8.0 and later. I don’t have an environment to test this, but if anyone does, please test it and see the results.
    The execution result differs from traditional databases and is unexpected.

Regarding the unstable result set, the official documentation
Unstable Result Set | PingCAP Documentation Center

This test scenario does not match any of the situations described in the documentation.

I think the result set should be stable in this scenario to be expected. If it is a bug, I hope it can be fixed.
If it is not a bug, I hope the root cause of the instability can be found and updated in the official documentation.

| username: windtalker | Original post link

The subquery outputs four columns: xaccount, cdindex, ab_user, and last_card. The window function essentially just sorts by xaccount and cdindex, so the correspondence of ab_user and last_card is unstable. For example, given the following set of data:

xaccount, cdindex, ab_user
1 1 1
1 1 2

After the window function, the result might be:

xaccount, cdindex, ab_user, last_card
1 1 1 1
1 1 1 2

Or it might be:

1 1 2 1
1 1 1 2

If you add an additional order by ab_user within the window function besides ORDER by a.cdindex DESC, the result should be stable.

| username: windtalker | Original post link

Additionally, when I mentioned instability, I wasn’t referring to the number of rows in the result set being different. The number of rows is the same, but the results themselves are different. Specifically, the last_card corresponding to the same ab_user is different (unstable).

| username: TiDBer_rrt4FOef | Original post link

Indeed, I tested with the following statement, and the results themselves are indeed different:

select ab_user, last_card, count(*) from 
(
select c.ab_user, a.xaccount,
row_number() OVER(PARTITION BY a.xaccount ORDER by a.cdindex desc, c.ab_user DESC) last_card
from f_ccs_card a, f_ccs_apma c
where a.xaccount = c.account
) t 
group by ab_user, last_card
order by ab_user, last_card;

T1

T2

The results executed on Oracle remain unchanged:
O1

After testing, adding the ab_user sorting field in the window function makes the result set fixed and unchanged.

Does this mean that when using window functions in the future, all query fields need to be included after the order by clause? Similar to MySQL’s full group by?

| username: windtalker | Original post link

I think this is just like the order by statement. If you want to get stable results, the result set cannot have duplicates in the order by column. If there is no primary key or unique key to ensure uniqueness, you need to put all columns after the order by.

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.