Sync_diff Error: panic: runtime error: invalid memory address or nil pointer dereference

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: sync_diff报错panic: runtime error: invalid memory address or nil pointer dereference

| username: h5n1

sync_diff_inspector ARM version
sync_diff_inspector --version
App Name: sync_diff_inspector v2.0
Release Version: v7.1.5-4-ga005a94
Git Commit Hash: a005a9426f5bd82664907b97344f4255fc8608b1
Git Branch: HEAD
UTC Build Time: 2024-05-11 05:17:22
Go Version: go1.21.4

Error encountered during comparison:
$ sync_diff_inspector --config=sync_diff.yaml
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x526798]

goroutine 1 [running]:
database/sql.(*DB).conn(0x313cf7d?, {0x394eaa0?, 0x6057e80?}, 0x7b?)
/usr/local/go/src/database/sql/sql.go:1282 +0x28
database/sql.(*DB).query(0x0?, {0x394eaa0, 0x6057e80}, {0x30f4868, 0x10}, {0x0, 0x0, 0x0}, 0x0?)
/usr/local/go/src/database/sql/sql.go:1721 +0x40
database/sql.(*DB).QueryContext.func1(0xf7?)
/usr/local/go/src/database/sql/sql.go:1704 +0x54
database/sql.(*DB).retry(0x0?, 0x40017a77a0)
/usr/local/go/src/database/sql/sql.go:1538 +0x4c
database/sql.(*DB).QueryContext(0x20?, {0x394eaa0?, 0x6057e80?}, {0x30f4868?, 0x46?}, {0x0?, 0x42fd53b?, 0x4e?})
/usr/local/go/src/database/sql/sql.go:1703 +0x94
github.com/pingcap/tidb-tools/pkg/dbutil.GetDBVersion({0x394eaa0?, 0x6057e80?}, {0x3931688?, 0x0?})
/workspace/source/tidb-tools/pkg/dbutil/common.go:690 +0x68
github.com/pingcap/tidb-tools/pkg/dbutil.IsTiDB({0x394eaa0?, 0x6057e80?}, {0x3931688?, 0x0?})
/workspace/source/tidb-tools/pkg/dbutil/common.go:760 +0x30
main.(*Diff).pickSource(0x4000ecf8c0, {0x394eaa0, 0x6057e80})
/workspace/source/tidb-tools/sync_diff_inspector/diff.go:357 +0x60
main.(*Diff).init(0x4000ecf8c0, {0x394eaa0, 0x6057e80}, 0x40012c04e0)
/workspace/source/tidb-tools/sync_diff_inspector/diff.go:142 +0xa0
main.NewDiff({0x394eaa0, 0x6057e80}, 0x40012c04e0)
/workspace/source/tidb-tools/sync_diff_inspector/diff.go:95 +0x13c
main.checkSyncState({0x394eaa0, 0x6057e80}, 0x40012c04e0)
/workspace/source/tidb-tools/sync_diff_inspector/main.go:117 +0x80
main.main()
/workspace/source/tidb-tools/sync_diff_inspector/main.go:104 +0x4b0

Configuration file:
check-thread-count = 4
export-fix-sql = true
check-struct-only = false
skip-non-existing-table = true

[data-sources]
[data-sources.i_actpara1]
host = “10.1xxxx”
port = 22010
user = “i_actpara1”
password = “xxxxx”
[data-sources.i_actpara2]
host = “10.1xxxx”
port = 22011
user = “i_actpara2”
password = “xxxxx”

[task]
output-dir = “./output”
source-instances = [“i_actpara1”]
target-instance = “i_actpara2”
target-check-tables = [“i_actpara2.tf*”]

Both source and target are Alibaba RDS
image

Logs
[2024/06/21 12:30:11.957 +08:00] [INFO] [printer.go:46] [“Welcome to sync_diff_inspector”] [“Release Version”=v7.1.5-4-ga005a94] [“Git Commit Hash”=a005a9426f5bd82664907b97344f4255fc8608b1] [“Git Branch”=HEAD] [“UTC Build Time”=“2024-05-11 05:17:22”] [“Go Version”=go1.21.4]
[2024/06/21 12:30:11.959 +08:00] [INFO] [main.go:101] [config=“{"check-thread-count":4,"split-thread-count":5,"export-fix-sql":true,"check-struct-only":false,"dm-addr":"","dm-task":"","data-sources":{"i_actpara1":{"host":"10.xxxxx","port":22010,"user":"i_actpara1","password":"","sql-mode":"","snapshot":"","security":null,"route-rules":null,"Router":{"Selector":{}},"Conn":null},"i_actpara2":{"host":"10.12xxxxx","port":22011,"user":"i_actpara2","password":"","sql-mode":"","snapshot":"","security":null,"route-rules":null,"Router":{"Selector":{}},"Conn":null}},"routes":null,"table-configs":null,"task":{"source-instances":["i_actpara1"],"source-routes":null,"target-instance":"i_actpara2","target-check-tables":["i_actpara2.tf*"],"target-configs":null,"output-dir":"./output","SourceInstances":[{"host":"10.12xxxx","port":22010,"user":"i_actpara1","password":"","sql-mode":"","snapshot":"","security":null,"route-rules":null,"Router":{"Selector":{}},"Conn":null}],"TargetInstance":{"host":"10.1xxxxx","port":22011,"user":"i_actpara2","password":"","sql-mode":"","snapshot":"","security":null,"route-rules":null,"Router":{"Selector":{}},"Conn":null},"TargetTableConfigs":null,"TargetCheckTables":[{}],"FixDir":"output/fix-on-i_actpara2","CheckpointDir":"output/checkpoint","HashFile":""},"ConfigFile":"sync_diff.yaml","PrintVersion":false}”]
[2024/06/21 12:30:12.588 +08:00] [INFO] [mysql_shard.go:372] [“will increase connection configurations for DB of instance”] [“connection limit”=1]
[2024/06/21 12:30:12.588 +08:00] [INFO] [source.go:392] [“the source has no table to be compared”] [target-table=i_actpara2.tf_f_project]
[2024/06/21 12:30:12.588 +08:00] [INFO] [source.go:392] [“the source has no table to be compared”] [target-table=i_actpara2.tf_chl]
[2024/06/21 12:30:12.588 +08:00] [INFO] [source.go:392] [“the source has no table to be compared”] [target-table=i_actpara2.tf_b]
[2024/06/21 12:30:12.588 +08:00] [INFO] [source.go:392] [“the source has no table to be compared”] [target-table=i_actpara2.tf_b_a]
[2024/06/21 12:30:12.588 +08:00] [INFO] [source.go:412] [“table match check finished”]
[2024/06/21 12:30:12.663 +08:00] [INFO] [mysql_shard.go:372] [“will increase connection configurations for DB of instance”] [“connection limit”=10]
[2024/06/21 12:30:12.663 +08:00] [INFO] [source.go:412] [“table match check finished”]
[2024/06/21 12:30:12.663 +08:00] [WARN] [mysql_shard.go:233] [“the source has no DB connection.”]
[2024/06/21 12:30:12.663 +08:00] [INFO] [main.go:114] [“check data finished”] [cost=702.366699ms]

2
| username: Kamner | Original post link

MySQL can use pt-table-checksum from the Percona Toolkit to check if the data is consistent.

3
| username: dba-kit | Original post link

You can try adding this target-configs configuration. I remember encountering a similar issue before, which seemed to be related to the new character set system. Later, the production and research team asked to add this configuration.

######################### Task config #########################
[task]
    output-dir = "./output"
    source-instances = ["i_actpara1"]
    target-instance = ["i_actpara2"]
    target-check-tables = ["i_actpara2.tf*"]

    target-configs = ["config1"]

[table-configs.config1]
    target-tables = ["*.*"]
    range = "true and true"
4
| username: h5n1 | Original post link

Still reporting this error.

5
| username: dba-kit | Original post link

It’s indeed not the same issue. At that time, it was an error caused by the new character set, which is different from yours.