After using Placement Policy, location_labels of non-default policies no longer take effect

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 使用Placement Policy后,非default策略的location_labels不再生效

| username: dba-kit

When replacing the TiKV machine in Zone D, it was found that after using the Placement policy, during scaling, the Regions of all nodes would increase/decrease, and it was not limited to Zone D. The Region change curves are as follows:

  • In the default PLACEMENT POLICY, the curve of the number of Regions when replacing TiKV in Zone D is:
  • After using other PLACEMENT POLICY, the curve of the number of Regions when replacing TiKV in Zone D is:

Directly using pd-ctl to check the Placement Rule in PD, the contents are as follows:

    1. The default global strategy, which includes location_labels content
{
    "group_id": "pd",
    "group_index": 0,
    "group_override": false,
    "rules": [
      {
        "group_id": "pd",
        "id": "default",
        "start_key": "",
        "end_key": "",
        "role": "voter",
        "is_witness": false,
        "count": 3,
        "label_constraints": [
          {
            "key": "disk",
            "op": "notIn",
            "values": [
              "sata",
              "sata-new",
              "dw-ssd"
            ]
          }
        ],
        "location_labels": [
          "zone",
          "dc",
          "host"
        ]
      }
    ]
  },
    1. The one generated by using Placement rule in SQL is as follows, lacking location_labels content
      {
        "group_id": "TiDB_DDL_89632",
        "id": "partition_rule_89632_1",
        "index": 40,
        "start_key": "748000000000015eff2000000000000000f8",
        "end_key": "748000000000015eff2100000000000000f8",
        "role": "voter",
        "is_witness": false,
        "count": 2,
        "label_constraints": [
          {
            "key": "disk",
            "op": "in",
            "values": [
              "dw-ssd"
            ]
          },
          {
            "key": "engine",
            "op": "notIn",
            "values": [
              "tiflash"
            ]
          }
        ],
        "create_timestamp": 1706864872
      }
| username: dba-kit | Original post link

When organizing, it seemed to make sense that it was caused by adding label_constraints to my default policy group. This setting caused the default policy to be ineffective for TiKV nodes with disk in ["sata", "sata-new", "dw-ssd"]. And since Placement rule in SQL itself does not set location-labels, it indeed causes the data allocation of TiKV with these three Labels to fail, meaning multiple replicas of the same region might be in the same data center. If a data center failure occurs, data might be lost.

        "label_constraints": [
          {
            "key": "disk",
            "op": "notIn",
            "values": [
              "sata",
              "sata-new",
              "dw-ssd"
            ]
          }
        ],

The official documentation also mentions this:

| username: dba-kit | Original post link

I found the official documentation at Placement Rules in SQL | PingCAP 文档中心. I will try setting SURVIVAL_PREFERENCES later.

| username: WalterWj | Original post link

Pay attention to the TiDB version, I remember this parameter requires a relatively high version.

| username: dba-kit | Original post link

After thinking about it, I’ll wait a few days to upgrade to 7.5 and use alter range to fix it directly.

| username: Jellybean | Original post link

You guys have used placement rules quite extensively. Could you write a series of articles to share your experience?

| username: dba-kit | Original post link

Indeed, version 6.5 doesn’t have this parameter :joy:, so there’s no way to fix it. We’ll have to rely on upgrading to version 7.5 to resolve it.

| username: dba-kit | Original post link

It should be said that there were many pitfalls. In fact, after providing feedback on many experiences/bugs, they were directly supported/fixed by the official team in later versions of TiDB.

| username: WalterWj | Original post link

I shared this at an event in Shanghai before, hahaha :dotted_line_face:

| username: WinterLiu | Original post link

Very complicated :crying_cat_face:

| username: lemonade010 | Original post link

Thanks for sharing, marked.

| username: dba-kit | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.