Resource limits and kernel behavior in /etc/security/limits.d and /etc/sysctl.d directories cannot be read by the tiup tool

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: /etc/security/limits.d 及 /etc/sysctl.d/ 目录中的资源限制与内核行为均无法被 tiup 工具读取

| username: 沈阳盛京征信有限公司

Bug Report

Specify resource limits for tidb in the /etc/security/limits.d/ directory. Cannot be read by the tiup tool.

Specify kernel behavior for tidb in the /etc/sysctl.d/ directory. Cannot be read by the tiup tool.

【TiDB Version】TiDB v7.1.1
【Impact of the Bug】

【Possible Steps to Reproduce the Issue】

[root@h200 ~]# more /etc/security/limits.d/tidb.conf 
tidb  soft nofile  1000000
tidb  hard nofile  1000000
tidb  soft stack   32768
tidb  hard stack   32768

[root@h200 ~]# cat /etc/sysctl.d/tidb.conf 
vm.swappiness = 0
fs.file-max = 1000000
net.core.somaxconn = 32768
net.ipv4.tcp_syncookies = 0
vm.overcommit_memory = 1

【Observed Unexpected Behavior】

192.168.3.201  limits        Fail    soft limit of 'stack' for user 'tidb' is not set or too low
192.168.3.201  limits        Fail    soft limit of 'nofile' for user 'tidb' is not set or too low
192.168.3.201  limits        Fail    hard limit of 'nofile' for user 'tidb' is not set or too low

192.168.3.200  sysctl        Fail    net.ipv4.tcp_syncookies = 1, should be 0

【Additional Background Information or Screenshots】

| username: redgame | Original post link

This should not be readable, right?

| username: YuchongXU | Original post link

You can try setting it globally.

| username: zhanggame1 | Original post link

What is the operating system? Check the result of ulimit -a tidb.

| username: tidb菜鸟一只 | Original post link

What does it show when you execute ulimit -a with the tidb user?

| username: ffeenn | Original post link

Try writing it into limits.conf.

| username: cassblanca | Original post link

Manually modify the system configuration file /etc/security/limits.conf

| username: ljluestc | Original post link

To solve this problem, you need to ensure that the TiUP tool can correctly read these configuration files and apply the configurations to the TiDB instance. Here are some possible solutions:

Ensure that the TiUP tool has sufficient permissions to read the files under the /etc/security/limits.d/ and /etc/sysctl.d/ directories. You can use the ls -l command to check the permissions and ownership of the files. Make sure the TiUP tool has enough permissions when it runs.

Ensure that the TiUP tool can access the /etc/security/limits.d/ and /etc/sysctl.d/ directories. Sometimes, the TiUP tool runs in a restricted environment and may not be able to access certain directories. You can try manually accessing these directories in the context where the TiUP tool runs to see if there are any permission issues.

Ensure that the format of the configuration files is correct, especially the file names and contents. The TiUP tool may only be able to read configuration files in a specific format. You can check whether the syntax and format of the files are correct.

Ensure that the configuration items in the configuration files are applicable to the TiDB process. Sometimes, configuration items in the configuration files may be ignored or overridden by other processes. Make sure the configuration items take effect for the TiDB process.

You can try writing these resource limits and kernel behavior configurations into the TiDB startup script to ensure that the TiDB process can correctly apply these configurations when it starts.

| username: 沈阳盛京征信有限公司 | Original post link

Setting resource limits in /etc/security/limits.d/tidb.conf is effective for the tidb user. However, it cannot be read by the tiup deployment tool. It must be set globally (/etc/security/limits.conf) to be read by tiup. However, when deploying Oracle, the settings can be read by the Oracle installer whether they are set globally or in the /etc/security/limits.d/ directory.

[tidb@h200 ~]$ id
uid=1001(tidb) gid=1001(tidb) groups=1001(tidb),10(wheel)
[tidb@h200 ~]$ cat /etc/security/limits.d/tidb.conf 
tidb  soft nofile  1000000
tidb  hard nofile  1000000
tidb  soft stack   32768
tidb  hard stack   32768
[tidb@h200 ~]$ ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) unlimited
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 13499
max locked memory           (kbytes, -l) 65536
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1000000
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 32768
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 13499
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

| username: ffeenn | Original post link

Setting the restrictions for that user, normally, the user should be able to read their own limits upon logging in. There’s no such thing as cross-user reading. If you are using tiup with the tidb user, it should be normal. How did you operate it? Can you post it here so I can try to reproduce it on my end?

| username: ffeenn | Original post link

From the source code, fixing potential shared blocks in the cluster is configured in /etc/security/limits.conf.

| username: dockerfile | Original post link

It means the configuration does not meet the requirements.

| username: jansu-dev | Original post link

+1

TiDB Environment and System Configuration Check | PingCAP Documentation Center

However, when deploying Oracle, whether it is set globally or in the /etc/security/limits.d/ directory, it can be read by the Oracle installer.

  • --> I think we should refer to the TiDB official guide. Just because Oracle can recognize it doesn't mean TiDB should be able to (personal opinion).
    

However, it should be considered a feature: Suggestion: adding function which multi-files can be aware of soft/hard limit · Issue #2245 · pingcap/tiup (github.com)
These are just my personal opinions. I mentioned it on Github, but whether the R&D team thinks the product should be designed this way is unknown. You can follow this issue.

I will conclude this topic with the reply from ffeennThatBoy.

| username: 沈阳盛京征信有限公司 | Original post link

Setting user resource limits independently in the /etc/security/limits.d/ directory is a standard practice in Linux systems. The /etc/security/limits.conf file describes it as follows:

[root@idc1-sjcredit-tiup ~]# cat /etc/security/limits.conf

# /etc/security/limits.conf

#

#This file sets the resource limits for the users logged in via PAM.

#It does not affect resource limits of the system services.

#

#Also note that configuration files in /etc/security/limits.d directory,

#which are read in alphabetical order, override the settings in this

#file in case the domain is the same or more specific.

#That means, for example, that setting a limit for wildcard domain here

#can be overridden with a wildcard setting in a config file in the

#subdirectory, but a user specific setting here can be overridden only

#with a user specific setting in the subdirectory.

#

#Each line describes a limit for a user in the form:
| username: 沈阳盛京征信有限公司 | Original post link

Hardcoded? Only reading from /etc/security/limits.conf?

| username: ffeenn | Original post link

Currently, the official recommendation is this. We’ll see if we will adopt the suggestion proposed by jansu-dev later.

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.