K8s Monitoring Prometheus Reports Permission Error

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: k8s监控 Promethus 报没有权限错误

| username: h5n1

The TiDB monitor pod cannot start due to Prometheus reporting a permission issue. I looked up some solutions online, such as using an init container or service account context. I want to edit the pod to add a chmod command, but it doesn’t allow me to save after editing. How should I handle this?

# kubectl logs  tidb-monitor-monitor-0 -c prometheus
ts=2023-01-11T12:57:31.443Z caller=main.go:556 level=info msg="Starting Prometheus Server" mode=server version="(version=2.41.0, branch=HEAD, revision=c0d8a56c69014279464c0e15d8bfb0e153af0dab)"
ts=2023-01-11T12:57:31.443Z caller=main.go:561 level=info build_context="(go=go1.19.4, platform=linux/arm64, user=root@d20a03e77067, date=20221220-10:48:06)"
ts=2023-01-11T12:57:31.443Z caller=main.go:562 level=info host_details="(Linux 4.19.90-17.ky10.aarch64 #1 SMP Sun Jun 28 14:27:40 CST 2020 aarch64 tidb-monitor-monitor-0 (none))"
ts=2023-01-11T12:57:31.443Z caller=main.go:563 level=info fd_limits="(soft=1073741816, hard=1073741816)"
ts=2023-01-11T12:57:31.443Z caller=main.go:564 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2023-01-11T12:57:31.443Z caller=query_logger.go:113 level=error component=activeQueryTracker msg="Failed to create directory for logging active queries"
ts=2023-01-11T12:57:31.443Z caller=query_logger.go:91 level=error component=activeQueryTracker msg="Error opening query log file" file=/data/prometheus/queries.active err="open /data/prometheus/queries.active: permission denied"
panic: Unable to create mmap-ed active query log

goroutine 1 [running]:
github.com/prometheus/prometheus/promql.NewActiveQueryTracker({0xffffceeef181, 0x10}, 0x14, {0x36e4040, 0x40001a8550})
        /app/promql/query_logger.go:121 +0x2ec
main.main()
        /app/cmd/prometheus/main.go:618 +0x638c
apiVersion: pingcap.com/v1alpha1
kind: TidbMonitor
metadata:
  name: tidb-monitor
spec:
  clusters:
  - name: tidb-test-cluster
  prometheus:
    baseImage: 10.172.49.246/zongbu-sre/prometheus-linux-arm64
    version: v2.41.0
    imagePullPolicy: Always
    logLevel: info
    reserveDays: 14
    service:
      type: NodePort
      portName: http-prometheus
  grafana:
    baseImage: 10.172.49.246/zongbu-sre/grafana-arm64
    version: 6.3.2
    imagePullPolicy: Always
    logLevel: info
    username: admin
    password: admin
    envs:
      GF_AUTH_ANONYMOUS_ENABLED: "true"
      GF_AUTH_ANONYMOUS_ORG_NAME: "Main Org."
      GF_AUTH_ANONYMOUS_ORG_ROLE: "Viewer"
    service:
      type: NodePort
      portName: http-grafana
  initializer:
    baseImage: 10.172.49.246/zongbu-sre/tidb-monitor-initializer-arm64
    version: v6.1.2
    imagePullPolicy: Always
  reloader:
    baseImage: 10.172.49.246/zongbu-sre/tidb-monitor-reloader-arm64
    version: v1.0.1
    imagePullPolicy: Always
    service:
      type: NodePort
      portName: tcp-reloader
  prometheusReloader:
    baseImage: 10.172.49.246/zongbu-sre/prometheus-config-reloader
    imagePullPolicy: Always
    version: v0.55.1
  persistent: true
  storageClassName: monitor-storage
  storage: 100Gi
  nodeSelector: {}
  annotations: {}
  tolerations: []
  kubePrometheusURL: http://prometheus-k8s.monitoring.svc:9090
  alertmanagerURL: ""
| username: h5n1 | Original post link

The PV directory used was set to 777 as a temporary solution.

| username: TiDBer_jYQINSnf | Original post link

It should be that the directory allocated by your PV does not have write permissions, right? I haven’t manually allocated PV before, so I’m not familiar with the specific situation.

| username: h5n1 | Original post link

The default permissions for the pv directory are created by root. The error message found online says that Prometheus is started with nobody.

| username: TiDBer_jYQINSnf | Original post link

Prometheus runs inside a pod, which means it runs inside a Docker container, and it does not share the same set of users as the host machine.

| username: h5n1 | Original post link

Below are the permissions for the pv directory:
image

| username: ffeenn | Original post link

It seems that you can only manually modify the directory permissions. When creating the PV mount point, you need to set the permissions to 755 or higher. To modify the startup command in the POD container to set permissions, you need to add the securityContext field. It would be more straightforward to directly modify it when creating the PV.

| username: h5n1 | Original post link

The official documentation mentions running in non-root mode and setting the context, stating that it supports tidbmonitor. However, I tried it, and if the monitor’s YAML file includes it, it directly reports an error: unknown field “podSecurityContext”.

| username: ffeenn | Original post link

Well, no need to fuss over it, just directly change the mount point permissions :joy:

| username: h5n1 | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.