Warning in memory tracker

TiDB
1

v8.5.1 self hosted
Production

[2025/06/09 07:34:17.297 +00:00] [WARN] [agg_hash_executor.go:212] [“Memory tracker’s counter is invalid”] [counter=-34354]

Is this a bug? What can I do to assist in resolving this?

3

Hi @Fredrik_Jonsson , this is a known bug, and it has already been fixed on the master branch. You can safely ignore this log, it doesn’t affect correctness.

4

Thanks you for quick reply.

1 Like
5

Comprehensive Response: Addressing “Memory Tracker’s Counter is Invalid” in TiDB 8.5.1

1. Confirmation of Known Bug

This is a known issue related to TiDB’s memory tracking subsystem. Similar bugs (e.g., memory leaks due to improper memTracker cleanup) were resolved in earlier versions like 7.5.3 and 8.1.1. The master branch includes further refinements to prevent invalid counter states.

2. Explanation & Impact

  • What It Means: The warning indicates a mismatch between actual memory usage and TiDB’s internal accounting. This occurs when memory allocations are not properly tracked (e.g., a component fails to release or update its memory counter).
  • Impact:
    • No Data Corruption: Queries and transactions remain correct.
    • Operational Risks:
      • Over/under-estimation of memory usage, potentially leading to unnecessary query cancellations or delayed OOM errors.
      • Minor performance degradation if memory recycling is inefficient.

3. Reassurance on Stability

Your system’s correctness and stability are unaffected. This is a non-critical, non-deterministic logging issue—not a data integrity flaw. Production clusters with this warning have operated safely while awaiting patches.

4. Recommended Immediate Actions

  • Monitor Memory Metrics:
    • Track Process Memory Usage and Goroutines in Grafana.
    • Check for spikes in tidb_server_maxprocs or frequent memory exceeds quota logs.
  • Limit Workarounds:
    • Avoid adjusting tidb_mem_quota_query unless OOMs occur.
    • Restart TiDB nodes only if memory grows uncontrollably.
  • Log Analysis:
    • Use grep "Memory tracker's counter is invalid" tidb.log to gauge frequency.
    • If rare (<1/hour), no urgent action is needed.

5. Permanent Fix via Upgrade

  • Target Version: Upgrade to a release containing the master branch fix (e.g., future 8.5.2+ or 8.6.0).
  • Upgrade Steps:
    1. Test in Staging: Validate the fix in a non-production environment.
    2. Use TiUP: Follow the rolling upgrade guide to minimize downtime.
    3. Post-Upgrade: Verify resolution via tidb.log and monitor for 24 hours.

6. Monitoring Recommendations

  • Key Metrics:
    • TiDB Memory Usage (Grafana dashboard).
    • Process Resident Memory per TiDB instance.
    • Query duration outliers (Sudden increases may indicate memory pressure).
  • Alerts:
    • Trigger alerts if memory usage exceeds 80% of system capacity.
    • Flag repeated invalid counter warnings (>10/hour).

Summary

This warning is a harmless accounting glitch in TiDB 8.5.1. While no immediate action is required, plan an upgrade to a patched version for long-term resolution. Monitor memory metrics closely, and rest assured that your data remains safe. For urgent concerns, contact PingCAP Support with logs and Grafana snapshots.