TiDB did not clean up residual placement rules after executing DDL

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiDB在执行DDL后未清理遗留的placement-rules

| username: dba-kit

When using tiup ctl:v6.5.3 pd config placement-rules show to observe placement rules, I found many TiDB_DDL_xxxxx rules, which seem to be left uncleared after executing DDL operations.

PS: After counting, I found that there are already 1494 groups, each containing one or more placement rules. If they are not cleaned up continuously, it will cause the PD memory to keep growing, right?

root@tipd6-f001:~# tiup ctl:v6.5.3 pd config placement-rules show | head -n 50
Starting component `ctl`: /root/.tiup/components/ctl/v6.5.3/ctl pd config placement-rules show
[
  {
    "group_id": "TiDB_DDL_11300",
    "id": "table_rule_11300_0",
    "index": 40,
    "start_key": "748000000000002cff2400000000000000f8",
    "end_key": "748000000000002cff2500000000000000f8",
    "role": "leader",
    "is_witness": false,
    "count": 1,
    "label_constraints": [
      {
        "key": "disk",
        "op": "in",
        "values": [
          "sata-new"
        ]
      },
      {
        "key": "engine",
        "op": "notIn",
        "values": [
          "tiflash"
        ]
      }
    ]
  },
root@tipd6-f001:~# tiup ctl:v6.5.3 pd config placement-rules rule-group show | jq -r '.[] | .id | match("TiDB_DDL.*") | .string' | wc -l
Starting component `ctl`: /root/.tiup/components/ctl/v6.5.3/ctl pd config placement-rules rule-group show
1494
| username: 芮芮是产品 | Original post link

It is indeed a bug.

| username: dba-kit | Original post link

Uh, it doesn’t seem to be a BUG. I found a rule and followed it. This is a PLACEMENT POLICY set for the table. According to this rule, the partition with partition_id 61818 must meet the rule disk in ["sata-new"].

» config placement-rules rule-bundle get TiDB_DDL_61818
{
  "group_id": "TiDB_DDL_61818",
  "group_index": 80,
  "group_override": true,
  "rules": [
    {
      "group_id": "TiDB_DDL_61818",
      "id": "partition_rule_61818_0",
      "index": 40,
      "start_key": "74800000000000f1ff7a00000000000000f8",
      "end_key": "74800000000000f1ff7b00000000000000f8",
      "role": "leader",
      "is_witness": false,
      "count": 1,
      "label_constraints": [
        {
          "key": "disk",
          "op": "in",
          "values": [
            "sata-new"
          ]
        },
        {
          "key": "engine",
          "op": "notIn",
          "values": [
            "tiflash"
          ]
        }
      ]
    },
    {
      "group_id": "TiDB_DDL_61818",
      "id": "partition_rule_61818_1",
      "index": 40,
      "start_key": "74800000000000f1ff7a00000000000000f8",
      "end_key": "74800000000000f1ff7b00000000000000f8",
      "role": "voter",
      "is_witness": false,
      "count": 2,
      "label_constraints": [
        {
          "key": "disk",
          "op": "in",
          "values": [
            "sata-new"
          ]
        },
        {
          "key": "engine",
          "op": "notIn",
          "values": [
            "tiflash"
          ]
        }
      ]
    }
  ]
}

After querying the metadata, it is indeed actively placing this partition on the HDD disk, which is expected.

mysql> select * from information_schema.partitions where TIDB_PARTITION_ID='61818' limit 1\G
*************************** 1. row ***************************
                TABLE_CATALOG: def
                 TABLE_SCHEMA: XXXXXX
                   TABLE_NAME: XXXXXX
               PARTITION_NAME: p202307
            SUBPARTITION_NAME: NULL
   PARTITION_ORDINAL_POSITION: 1
SUBPARTITION_ORDINAL_POSITION: NULL
             PARTITION_METHOD: RANGE COLUMNS
          SUBPARTITION_METHOD: NULL
         PARTITION_EXPRESSION: cal_date
      SUBPARTITION_EXPRESSION: NULL
        PARTITION_DESCRIPTION: '2023-08-01'
                   TABLE_ROWS: 0
               AVG_ROW_LENGTH: 0
                  DATA_LENGTH: 0
              MAX_DATA_LENGTH: 0
                 INDEX_LENGTH: 0
                    DATA_FREE: 0
                  CREATE_TIME: 2023-08-28 15:52:16
                  UPDATE_TIME: NULL
                   CHECK_TIME: NULL
                     CHECKSUM: NULL
            PARTITION_COMMENT:
                    NODEGROUP: NULL
              TABLESPACE_NAME: NULL
            TIDB_PARTITION_ID: 61818
   TIDB_PLACEMENT_POLICY_NAME: on_sata_new
1 row in set (0.31 sec)
| username: Fly-bird | Original post link

Not cleaning it won’t cause memory growth, right? Has this conclusion been verified?

| username: dba-kit | Original post link

At first, I thought that each DDL would create a record on the PD, so I was worried that the memory would keep growing. Later, after some investigation, I found that this is a normal manifestation of the placement rule and is only related to the number of Placement Rules set in TiDB. It is expected to occupy a certain amount of memory.

| username: heiwandou | Original post link

Is the memory growth severe?

| username: dba-kit | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.