TiKV CPU Fully Utilized, kswapd0 CPU at 1000%

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tikv CPU打满,kswapd0 cpu 1000%

| username: Lawrence

[TiDB Usage Environment] Production Environment / Testing / PoC
[TiDB Version] V6.1.0
[Reproduction Path] None
[Encountered Problem: Phenomenon and Impact] TiKV CPU is fully utilized, kswapd0 process CPU 1000%
[Resource Configuration] Enter TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page

[Attachments: Screenshots/Logs/Monitoring]


Memory of a certain TiKV (screenshot after killing kswapd0)

| username: xfworld | Original post link

What’s going on? When deploying TiDB, it requires turning off swap. How did it come up?

| username: zhaokede | Original post link

Is it during peak business hours or a hot topic?

| username: DBAER | Original post link

This process swap shouldn’t normally happen. Check the system logs to see if there are any records.

| username: Lawrence | Original post link

Swap has been turned off.

| username: Lawrence | Original post link

It’s not a hotspot, there were no signs, it suddenly became like this at 6 in the morning.

| username: Lawrence | Original post link

How to operate it approximately? :blush:

| username: songxuecheng | Original post link

Mixed deployment. Check each log to see.

| username: DBAER | Original post link

Is the corresponding time point in /var/log/message normal?

| username: TiDBer_JUi6UvZm | Original post link

Check the audit and logs to confirm the operations that occurred.

| username: TiDBer_JUi6UvZm | Original post link

Additionally, if there is an external network connection, the possibility of a mining virus cannot be ruled out.

| username: xfworld | Original post link

Oh, mining viruses are so vicious~ :see_no_evil:

| username: Lawrence | Original post link

It is definitely not a mining virus. I have a large number of clusters, and only the TiDB cluster has this problem. Moreover, the kswapd0 process also belongs to the TiDB user.

| username: xfworld | Original post link

Check this strange process.

| username: 有猫万事足 | Original post link

This is a virus. It most likely infiltrated through TiDB user permissions.

According to Tencent Cloud’s report, this attack is from the “Outlaw” botnet, which was first discovered in 2018. Its main characteristic is using SSH brute force to attack target systems, while spreading Shellbot based on Peril and Monero mining malware. So, everyone must set a strong root password. There are more detailed analysis reports of this virus available online, you can search for them if you’re interested.

The TiDB user’s password was set too simply and got brute-forced.

| username: Lawrence | Original post link

Awesome, it really is… I thought it was an issue with tikv using too much memory. Thanks, expert.

| username: DBAER | Original post link

Damn, it really is mining.

| username: hacker_77powerful | Original post link

We also encountered kswapd0 occupying a super high CPU usage. Is it really a mining virus? The machine froze yesterday.

| username: xiaoqiao | Original post link

:sweat_smile: So brutal

| username: zhaokede | Original post link

If the database is on the external network, you indeed need to be cautious and take preventive measures.