Question about tikv-importer

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 请问tikv-importer的问题

| username: 我家有个臭皮崽

I have some questions about tikv-importer:

  1. Detailed parameters of the tikv-importer configuration file;
  2. If implementing a tikv-importer client in C++, which kvproto file should be used;
  3. What are the inputs and outputs of the tikv-importer module? Is the input the key-value written by the client and the output SST files? Or something else;

The above questions are specifically about tikv-importer, not the TiDB Lightning tool.

| username: Billmay表妹 | Original post link

  1. The configuration file for tikv-importer includes the following parameters:

    [tikv-importer]
    # Backend mode, options are local or importer
    backend = "local"
    # Local storage path, effective only when backend is local
    sorted-kv-dir = "/path/to/sorted-kv-dir"
    # Listening address of tikv-importer
    listen-addr = "127.0.0.1:8287"
    # Status check address of tikv-importer
    status-addr = "127.0.0.1:8288"
    # Log level of tikv-importer, options are debug, info, warn, error, fatal
    log-level = "info"
    # Log file path of tikv-importer
    log-file = "/path/to/tikv-importer.log"
    # Maximum concurrency of tikv-importer
    concurrency = 16
    # Maximum memory usage of tikv-importer, in bytes
    memory-quota = 1073741824
    # Maximum write rate of tikv-importer, in bytes/second
    rate-limit = 0
    # Burst size of the maximum write rate of tikv-importer, in bytes
    rate-limit-burst = 0
    # Maximum batch size for writes
    batch-size = 1024
    # Maximum wait time for batch writes, in milliseconds
    batch-wait-time = 100
    # Maximum wait queue size for batch writes
    batch-wait-size = 1024
    # Handling strategy when the maximum wait queue size for batch writes is reached, options are drop or block
    batch-wait-strategy = "drop"
    # Maximum wait time when the handling strategy is block, in milliseconds
    batch-wait-max-time = 0
    # Maximum wait count when the handling strategy is block
    batch-wait-max-count = 0
    # Multiplier for the maximum wait time when the handling strategy is block
    batch-wait-timeout-multiplier = 1
    # Maximum value for the multiplier of the maximum wait time when the handling strategy is block
    batch-wait-timeout-max-multiplier = 0
    # Minimum value for the multiplier of the maximum wait time when the handling strategy is block
    batch-wait-timeout-min-multiplier = 0
    # Growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-multiplier = 2
    # Maximum value for the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-max-multiplier = 0
    # Minimum value for the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-min-multiplier = 0
    # Maximum count for the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-max-count = 0
    # Maximum time for the growth rate of the wait time when the handling strategy is block, in milliseconds
    batch-wait-backoff-max-time = 0
    # Multiplier for the maximum time of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-multiplier = 1
    # Maximum value for the multiplier of the maximum time of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-multiplier = 0
    # Minimum value for the multiplier of the maximum time of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-min-multiplier = 0
    # Maximum count for the multiplier of the maximum time of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-count = 0
    # Maximum time for the multiplier of the maximum time of the growth rate of the wait time when the handling strategy is block, in milliseconds
    batch-wait-backoff-timeout-max-time = 0
    # Multiplier for the maximum time of the multiplier of the maximum time of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-multiplier = 1
    # Maximum value for the multiplier of the maximum time of the multiplier of the maximum time of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-multiplier = 0
    # Minimum value for the multiplier of the maximum time of the multiplier of the maximum time of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-min-multiplier = 0
    # Maximum count for the multiplier of the maximum time of the multiplier of the maximum time of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-count = 0
    # Maximum time for the multiplier of the maximum time of the multiplier of the maximum time of the growth rate of the wait time when the handling strategy is block, in milliseconds
    batch-wait-backoff-timeout-max-time-max-time = 0
    # Multiplier for the maximum time of the multiplier of the maximum time of the multiplier of the maximum time of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-multiplier = 1
    # Maximum value for the multiplier of the maximum time of the multiplier of the maximum time of the multiplier of the maximum time of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-multiplier = 0
    # Minimum value for the multiplier of the maximum time of the multiplier of the maximum time of the multiplier of the maximum time of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-min-multiplier = 0
    # Maximum count for the multiplier of the maximum time of the multiplier of the maximum time of the multiplier of the maximum time of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-count = 0
    # Maximum time for the multiplier of the maximum time of the multiplier of the maximum time of the growth rate of the wait time when the handling strategy is block, in milliseconds
    batch-wait-backoff-timeout-max-time-max-time-max-time = 0
    # Multiplier for the maximum time of the multiplier of the maximum time of the multiplier of the maximum time of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-multiplier = 1
    # Maximum value for the multiplier of the maximum time of the multiplier of the maximum time of the multiplier of the maximum time of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-multiplier = 0
    # Minimum value for the multiplier of the maximum time of the multiplier of the maximum time of the multiplier of the maximum time of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-min-multiplier = 0
    # Maximum count for the multiplier of the maximum time of the multiplier of the maximum time of the multiplier of the maximum time of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-count = 0
    # Maximum time for the multiplier of the maximum time of the multiplier of the maximum time of the growth rate of the wait time when the handling strategy is block, in milliseconds
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time = 0
    # Multiplier for the maximum time of the multiplier of the maximum time of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-multiplier = 1
    # Maximum value for the multiplier of the maximum time of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-multiplier = 0
    # Minimum value for the multiplier of the maximum time of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-min-multiplier = 0
    # Maximum count for the multiplier of the maximum time of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-count = 0
    # Maximum time for the multiplier of the maximum time of the growth rate of the wait time when the handling strategy is block, in milliseconds
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time = 0
    # Multiplier for the maximum time of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-multiplier = 1
    # Maximum value for the multiplier of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-multiplier = 0
    # Minimum value for the multiplier of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-min-multiplier = 0
    # Maximum count for the multiplier of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-count = 0
    # Maximum time for the multiplier of the growth rate of the wait time when the handling strategy is block, in milliseconds
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time = 0
    # Multiplier for the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-multiplier = 1
    # Maximum value for the multiplier of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-multiplier = 0
    # Minimum value for the multiplier of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-min-multiplier = 0
    # Maximum count for the multiplier of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-count = 0
    # Maximum time for the multiplier of the growth rate of the wait time when the handling strategy is block, in milliseconds
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-time = 0
    # Multiplier for the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-time-multiplier = 1
    # Maximum value for the multiplier of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-multiplier = 0
    # Minimum value for the multiplier of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-time-min-multiplier = 0
    # Maximum count for the multiplier of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-count = 0
    # Maximum time for the multiplier of the growth rate of the wait time when the handling strategy is block, in milliseconds
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time = 0
    # Multiplier for the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-multiplier = 1
    # Maximum value for the multiplier of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-multiplier = 0
    # Minimum value for the multiplier of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-min-multiplier = 0
    # Maximum count for the multiplier of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-count = 0
    # Maximum time for the multiplier of the growth rate of the wait time when the handling strategy is block, in milliseconds
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time = 0
    # Multiplier for the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-multiplier = 1
    # Maximum value for the multiplier of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-multiplier = 0
    # Minimum value for the multiplier of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-min-multiplier = 0
    # Maximum count for the multiplier of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-count = 0
    # Maximum time for the multiplier of the growth rate of the wait time when the handling strategy is block, in milliseconds
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time = 0
    # Multiplier for the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-multiplier = 1
    # Maximum value for the multiplier of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-multiplier = 0
    # Minimum value for the multiplier of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-min-multiplier = 0
    # Maximum count for the multiplier of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-count = 0
    # Maximum time for the multiplier of the growth rate of the wait time when the handling strategy is block, in milliseconds
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time = 0
    # Multiplier for the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-multiplier = 1
    # Maximum value for the multiplier of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-multiplier = 0
    # Minimum value for the multiplier of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-min-multiplier = 0
    # Maximum count for the multiplier of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-count = 0
    # Maximum time for the multiplier of the growth rate of the wait time when the handling strategy is block, in milliseconds
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time = 0
    # Multiplier for the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-multiplier = 1
    # Maximum value for the multiplier of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-multiplier = 0
    # Minimum value for the multiplier of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-min-multiplier = 0
    # Maximum count for the multiplier of the growth rate of the wait time when the handling strategy is block
    batch-wait-backoff-timeout-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-time-max-count = 0
    # Maximum time for the multiplier of the growth rate of the wait time when the handling strategy is block, in milliseconds
    batch-wait-backoff-timeout-max-time-max-time
| username: RenlySir | Original post link

It is more appropriate to ask the following two questions directly on GitHub, as the developers usually check there.

| username: redgame | Original post link

  1. In kvproto’s import_sstpb.proto, as mentioned by developer WX
| username: 我家有个臭皮崽 | Original post link

Okay, thank you!

| username: 我家有个臭皮崽 | Original post link

I tested that import_kvpb.proto can interact with tikv-importer. I understand that the client for the import_sstpb.proto file is used by TiDB Lightning’s local mode to interact directly with TiKV.