Does TiDB have an official WAL export tool?

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiDB 是否有官方的 WAL 导出工具

| username: Tsunaou

[TiDB Usage Environment] Test cluster built with TiUP
[TiDB Version] v6.1.0
[Encountered Issue: Phenomenon and Impact]
Previously, I used YugabyteDB, which provides a log-dump tool to directly view the WAL logs of each table in the system.
For example, I created a table like this:

CREATE TABLE txn1 (k INT PRIMARY KEY, v INT);

It would print out content like this to indicate table properties.

Header:
major_version: 1
minor_version: 0
unused_tablet_id: "a4d569a98bf3489eab7c822ddf29e580"
sequence_number: 1
schema {
  columns {
    id: 0
    name: "k"
    type {
      main: INT32
    }
    is_key: true
    is_hash_key: true
    is_nullable: false
    is_static: false
    is_counter: false
    sorting_type: 0
    order: 1
    pg_type_oid: 23
  }
  columns {
    id: 1
    name: "v"
    type {
      main: INT32
    }
    is_key: false
    is_nullable: true
    is_static: false
    is_counter: false
    sorting_type: 0
    order: 2
    pg_type_oid: 23
  }
  table_properties {
    contain_counters: false
    is_transactional: true
    consistency_level: STRONG
    use_mangled_column_name: false
    is_ysql_catalog_table: false
    retain_delete_markers: false
    partitioning_version: 1
  }
  colocated_table_id {
  }
  pgschema_name: "public"
}
schema_version: 0

And if I execute a transaction

BEGIN;
UPDATE txn1 SET v=400 WHERE k=3;
COMMIT;

There will be corresponding content, including

  • Transaction ID
  • Transaction write operation key-value pairs information
  • Transaction start timestamp
  • Transaction commit timestamp
Entry:
type: REPLICATE
replicate {
  id {
    term: 1
    index: 6
  }
  hybrid_time: 6854731322153586688
  op_type: WRITE_OP
  write {
    unused_tablet_id: ""
    write_batch {
      write_pairs {
        key: "G\374\240H\200\000\000\003!!K\201"
        value: "H\000\000\001\220"
      }
      transaction {
        transaction_id: "0:Q\220\231q@\246\222\247Z\370\'\221;\347"
        isolation: SNAPSHOT_ISOLATION
        status_tablet: "d8d35f601a36483ba4e6746c25e94e90"
        priority: 4621407376379158149
        start_hybrid_time: 6854731233537978368
        locality: GLOBAL
      }
      DEPRECATED_may_have_metadata: true
      table_schema_version {
        schema_version: 0
      }
    }
    client_id1: 13782638009204925153
    client_id2: 992683026222397848
    request_id: 11
    min_running_request_id: 11
    batch_idx: 0
  }
  committed_op_id {
    term: 1
    index: 5
  }
  monotonic_counter: 0
}
Entry:
type: REPLICATE
replicate {
  id {
    term: 1
    index: 7
  }
  hybrid_time: 6854731334443716608
  op_type: UPDATE_TRANSACTION_OP
  committed_op_id {
    term: 1
    index: 6
  }
  monotonic_counter: 0
  transaction_state {
    transaction_id: "0:Q\220\231q@\246\222\247Z\370\'\221;\347"
    status: APPLYING
    tablets: "d8d35f601a36483ba4e6746c25e94e90"
    commit_hybrid_time: 6854731334443175936
    sealed: false
    aborted {
    }
  }
}

Both YugabyteDB and TiDB are based on RocksDB for storage and support distributed transactions. I would like to ask if there is a separate, explicit tool in the TiDB ecosystem for analyzing WAL logs? Recently, TiCDC has been promoted in the community, and I have checked some related materials. TiCDC uses kv change log for upstream and downstream, is this part of the WAL log, raft log, or a third entity?
Although TiCDC seems to be an answer, it also requires deployment, which is not convenient for direct log analysis.
I am new to TiDB, so if there are any shallow misunderstandings, please forgive me. Thank you~

2
| username: TiDBer_jYQINSnf | Original post link

TiCDC does not use WAL; it captures the changelog when writing data from TiKV (details are unclear). As far as I know, TiDB does not have a WAL analysis tool.

3
| username: neilshen | Original post link

The kv change log source of TiCDC is the raft log, which includes key value additions and deletions in RocksDB.

The raft log in TiKV can be approximately understood as the RocksDB WAL, and the information contained in both is basically the same.

4
| username: Tsunaou | Original post link

Thank you, so it’s actually about manually parsing the underlying RocksDB data in TiKV, right? Thank you very much!

5
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.