Ingress Proxy TiDB Dashboard Abnormalities

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: ingress代理tidb dashboard异常

| username: liyuntang

[Test Environment for TiDB]
TiDB Version:
TiDB Version: Enterprise Edition 7.1.0
K8s Version: 1.19
TiDB-Operator Version: v1.4.6

Test Content:
Expose the TiDB dashboard service through ingress for external access

Issue:
Returning 404 when accessing the dashboard via ingress

Reproduction Steps:
The test cluster ID is bf45fbdb-1f2a-4562-900e-7cdae8b2945b, and $ns in the following text refers to this cluster ID

  1. The dashboard service (svc) has been configured in the cluster

  2. Configure ingress proxy


    Forward all requests sent to http://www.dashboard.com/dashboard/bf45fbdb-1f2a-4562-900e-7cdae8b2945b/ to the basic-dashboard service

  3. Initiate a request in the browser

Response:


Request failed

  1. Ingress log information:
    10.0.2.11 - - [24/Aug/2023:09:49:54 +0000] “GET /dashboard/bf45fbdb-1f2a-4562-900e-7cdae8b2945b/ HTTP/1.1” 302 34 “-” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36” 485 0.001 [bf45fbdb-1f2a-4562-900e-7cdae8b2945b-basic-dashboard-12333] 10.2.8.251:12333 34 0.000 302 0cc224c9a6dbc3efa30b8f08e4f1c3e6
    10.0.2.11 - - [24/Aug/2023:09:49:54 +0000] “GET /dashboard/ HTTP/1.1” 404 548 “-” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36” 448 0.001 [upstream-default-backend] 127.0.0.1:8181 548 0.000 404 8eca64452a76f47cb840ae8cf5e72b1a
    10.0.2.11 - - [24/Aug/2023:09:49:54 +0000] “GET /favicon.ico HTTP/1.1” 404 548 “http://www.dashboard.com/dashboard/” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36” 394 0.001 [upstream-default-backend] 127.0.0.1:8181 548 0.000 404 323ffbd86199d7f04020901b2b10eb75

From the logs, it is found that “GET /dashboard/bf45fbdb-1f2a-4562-900e-7cdae8b2945b/ HTTP/1.1” was changed to “GET /dashboard/ HTTP/1.1” during forwarding, resulting in a 404

  1. Suspecting that there might be an issue with the ingress configuration, let’s try forwarding the request to Grafana using the same configuration

  2. Initiate the request again


Request successful

  1. Ingress logs
    10.0.2.186 - - [24/Aug/2023:10:05:07 +0000] “GET /timonitor/bf45fbdb-1f2a-4562-900e-7cdae8b2945b/ HTTP/1.1” 302 76 “-” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36” 561 0.001 [bf45fbdb-1f2a-4562-900e-7cdae8b2945b-basic-grafana-3000] 10.2.9.185:3000 76 0.001 302 124d29c000583146a7a078579b3a34e3
    10.0.2.186 - - [24/Aug/2023:10:05:07 +0000] “GET /timonitor/bf45fbdb-1f2a-4562-900e-7cdae8b2945b/login HTTP/1.1” 200 30453 “-” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36” 566 0.003 [bf45fbdb-1f2a-4562-900e-7cdae8b2945b-basic-grafana-3000] 10.2.9.185:3000 30399 0.004 200 757dd724b6dc7483ad86848669ab309e

At this point, the ingress configuration should be fine. So the question arises, why can’t ingress proxy the dashboard service?

| username: TiDBer_oHSwKxOH | Original post link

Use nginx to proxy the path /dashboard

| username: liyuntang | Original post link

Currently, it’s using nginx as a proxy.

It can’t be changed to /dashboard because there are multiple TiDB clusters in the k8s cluster, and it needs to be forwarded according to the namespace.

| username: TiDBer_PyN8ad50 | Original post link

The full address of the dashboard is incorrect? http://www.dashboard.com/dashboard/bf45fbdb-1f2a-4562-900e-7cdae8b2945b/dashboard

| username: liyuntang | Original post link

I’ve tried all these addresses, none of them work.

| username: TiDBer_PyN8ad50 | Original post link

  1. Access this service separately to see if it is reachable, and check the pod logs.
  2. Enter the ingress pod and check the final generated nginx configuration.
| username: liyuntang | Original post link

The dashboard service is fine and can be accessed normally.

| username: redgame | Original post link

It may be caused by the path matching rules of the Ingress configuration. You should try modifying the Ingress configuration.