How to Calculate Log Generation Volume (Including Raft Log and Binlog)

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 如何统计日志生成量(包括raft log和binlog)

| username: TiDBer_S4eEz9Qb

How can I calculate the amount of logs generated by the database over a certain period of time if I can’t find the corresponding system view? This includes raft logs and binlogs.

| username: xfworld | Original post link

Why are you collecting these statistics?

| username: redgame | Original post link

I remember there is a method, but it needs to be tested. In Prometheus, you can query the metric of raft log generation in TiDB using the following PromQL statement: sum(increase(tikv_raft_log_gc_process_seconds_count[1d])).

| username: redgame | Original post link

sum(increase(binlog_written_bytes[1d]))

| username: TiDBer_S4eEz9Qb | Original post link

For example, the raft log needs to be transmitted to other nodes, and the amount of logs directly determines the required network bandwidth. I want to conduct a survey to check.

| username: xfworld | Original post link

The official documentation provides basic recommended parameters for network bandwidth. Isn’t it more straightforward to refer to that?

| username: TiDBer_S4eEz9Qb | Original post link

Yes, I also want to compare the transaction log write volume of several databases. :joy:

| username: xfworld | Original post link

WAL? That probably can’t be obtained… :upside_down_face:

You can only get it through POC scenario stress testing and using Prometheus to help you obtain it.

| username: TiDBer_S4eEz9Qb | Original post link

Why can’t I find the metrics tikv_raft_log_gc_process_seconds_count and binlog_written_bytes? :sob: