The slowlog disappears every time the TiDB cluster deployed with k8s is restarted

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 用k8s部署的tidb集群,每次重启slowlog就没了

| username: China-Dwan

[TiDB Usage Environment] Production Environment
[TiDB Version] 5.4.3

[Reproduction Path] What operations were performed when the issue occurred

[Encountered Issue: Issue Phenomenon and Impact]
Because it is deployed in k8s, the slowlog is lost after the tidb node restarts. Is there any way to preserve it?

[Resource Configuration] Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page

[Attachment: Screenshot/Log/Monitoring]

| username: China-Dwan | Original post link

I feel that with the TiDB Operator deployed on a k8s cluster, I don’t know where to look for the related logs.

| username: 像风一样的男子 | Original post link

You can check the previous discussion on deploying TiDB with k8s

| username: 小龙虾爱大龙虾 | Original post link

I don’t quite understand k8s, the slow log should be a separate container.

Containers:
  slowlog:
    Container ID:  containerd://7a94c4bce915d0ca1aa44ab85085d915767aa80ec44bfb54d31e160d4cebde09
    Image:         alpine:3.16.0
    Image ID:      docker.io/library/alpine@sha256:686d8c9dfa6f3ccfc8230bc3178d23f84eeaf7e457f36f271ab1acc53015037c
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
      touch /var/log/tidb/slowlog; tail -n0 -F /var/log/tidb/slowlog;
    State:          Running
      Started:      Mon, 15 Jan 2024 14:59:09 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/log/tidb from slowlog (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gmns5 (ro)

A separate volume should be configured to persist it.

Volumes:
  annotations:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.annotations -> annotations
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      tidb1-tidb-6662316
    Optional:  false
  startup-script:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      tidb1-tidb-6662316
    Optional:  false
  slowlog:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-gmns5:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
| username: China-Dwan | Original post link

Through the tool, scaling down TiFlash, in the dashboard, the status has been incorrect all the time, and the actual container has already been terminated.

| username: China-Dwan | Original post link

Yes, what you mentioned is one aspect. For example, some logs of TiDB are printed to stdout and stderr. Can these be saved through mounting? Or do they need to be collected separately, such as using tools like filebeat?

| username: changpeng75 | Original post link

Did the TiDB node restart mean the container restarted? The slow log is not stored persistently, right?

| username: 小龙虾爱大龙虾 | Original post link

Sure, just treat TiDB as a regular container.

| username: WalterWj | Original post link

Logs can be persisted.

| username: China-Dwan | Original post link

Yes, how can this be added to an already deployed cluster?

| username: changpeng75 | Original post link

I’m not very familiar with Kubernetes, so I’ll need to look up the specific documentation.

| username: tidb菜鸟一只 | Original post link

Mount the PVC separately, and then bind it to the specified slowlog directory of the corresponding TiDB.

| username: srstack | Original post link

You can also deploy k8s log collection components, such as GitHub - vectordotdev/vector: A high-performance observability data pipeline.. This method is more flexible and does not require additional operational burdens like binding PVC for slow logs.

| username: Miracle | Original post link

The logs are lost upon reboot because they are not persisted. If you need to retain them, you can mount the logs locally and adjust the volumes and volumeMounts.

| username: yiduoyunQ | Original post link

For slow log persistence, refer to 配置 TiDB 集群 | PingCAP 文档中心

storageVolumes considerations:

additionalVolumes considerations: Since all TiDB pods will mount this disk, the example uses an NFS shared disk.

Recommendation: Use a logging tool or service to collect logs in a timely manner. Slow logs represent one SQL statement with multiple lines, and the format is basically fixed, so you need to piece it together yourself.

| username: 哈喽沃德 | Original post link

Check the log.slow-query-file configuration parameter.

| username: dba远航 | Original post link

The logs are persisted, how could they be gone after a restart?

| username: 哈喽沃德 | Original post link

Persistent storage is not configured.

| username: 双开门变频冰箱 | Original post link

Configure volume for separate persistence

| username: TiDBer_jYQINSnf | Original post link

By default, k8s logs are output to stdout and will be lost if not collected.
If there is a failure and restart, check the previous logs.
kubectl logs xxx -n xxx -c slow-log --previous