Application environment:
PoC
TiDB version:
v8.1.0
Reproduction method:
Configured Encryption at Rest with self-hosted local key (for both TiKV and PD nodes):
security.encryption.data-encryption-method: aes128-ctr
security.encryption.data-key-rotation-period: 6h
security.encryption.master-key.path: /dev/shm/tidb-master-key
security.encryption.master-key.type: file
This worked. Then we wanted to change to AWS KMS for the key:
security.encryption.data-encryption-method: aes128-ctr
security.encryption.data-key-rotation-period: 168h
security.encryption.master-key.endpoint: https://kms.us-west-1.amazonaws.com
security.encryption.master-key.key-id: 31REDACTEDba
security.encryption.master-key.region: us-west-1
security.encryption.master-key.type: kms
security.encryption.previous-master-key.path: /dev/shm/tidb-master-key
security.encryption.previous-master-key.type: file
Problem:
The cluster restarted successfully, is functional, and shows that everything is up, but we keep seeing these identical (other than timestamp) lines in the logs over and over for the last several hours:
[2024/09/06 17:01:20.678 -04:00] [INFO] [mod.rs:422] ["Try to rotate data key, current method:Aes128Ctr"] [thread_id=20]
[2024/09/06 17:01:20.696 -04:00] [INFO] [util.rs:639] ["connecting to PD endpoint"] [endpoints=http://192.168.29.39:2379] [thread_id=12]
It makes me think the server is functioning on the previous-master-key and isn’t successfully using the AWS KMS key yet.
Resource allocation:
The cluster is almost entirely idle and is barely using ample RAM, CPU, and disk.
Attachment:
See log snippet above