Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.Original topic: TiDBv6.5.0 执行创建副本命令后,tiflash进程不断重启,是什么原因呢?
【TiDB Usage Environment】Test
【TiDB Version】
TiDB v6.5.0
【Reproduction Path】
ALTER DATABASE test SET TIFLASH REPLICA 1;
【Encountered Problem: Symptoms and Impact】
Tiflash_error.log
$ tail -f tiflash_error.log
[2023/01/04 17:15:29.929 +08:00] [WARN] [StorageConfigParser.cpp:241] [“The configuration "path" is deprecated. Check [storage] section for new style.”] [thread_id=1]
[2023/01/04 17:18:39.194 +08:00] [ERROR] [BaseDaemon.cpp:377] [########################################] [source=BaseDaemon] [thread_id=27]
[2023/01/04 17:18:39.194 +08:00] [ERROR] [BaseDaemon.cpp:378] [“(from thread 26) Received signal Illegal instruction(4).”] [source=BaseDaemon] [thread_id=27]
[2023/01/04 17:18:39.194 +08:00] [ERROR] [BaseDaemon.cpp:473] [“Illegal operand.”] [source=BaseDaemon] [thread_id=27]
[2023/01/04 17:18:45.690 +08:00] [ERROR] [BaseDaemon.cpp:570] [“\n 0x61214f1\tfaultSignalHandler(int, siginfo_t*, void*) [tiflash+101848305]\n \tlibs/libdaemon/src/BaseDaemon.cpp:221\n 0x7fbaed382070\t [libc.so.6+266352]\n 0x682303b\tDB::countBytesInFilter(DB::PODArray<unsigned char, 4096ul, Allocator, 15ul, 16ul> const&) [tiflash+109195323]\n \tdbms/src/Columns/ColumnsCommon.cpp:66\n 0x16d4d10\tDB::DM::DMVersionFilterBlockInputStream<1>::read(DB::PODArray<unsigned char, 4096ul, Allocator, 15ul, 16ul>*&, bool) [tiflash+23940368]\n \tdbms/src/Storages/DeltaMerge/DMVersionFilterBlockInputStream.cpp:373\n 0x6a93168\tDB::DM::BoundedSSTFilesToBlockInputStream::read() [tiflash+111751528]\n \tdbms/src/Storages/DeltaMerge/SSTFilesToBlockInputStream.cpp:307\n 0x16d8a44\tDB::DM::SSTFilesToDTFilesOutputStream<std::__1::shared_ptrDB::DM::BoundedSSTFilesToBlockInputStream >::write() [tiflash+23956036]\n \tdbms/src/Storages/DeltaMerge/SSTFilesToDTFilesOutputStream.cpp:200\n 0x6a89c8f\tDB::KVStore::preHandleSSTsToDTFiles(std::__1::shared_ptrDB::Region, DB::SSTViewVec, unsigned long, unsigned long, DB::DM::FileConvertJobType, DB::TMTContext&) [tiflash+111713423]\n \tdbms/src/Storages/Transaction/ApplySnapshot.cpp:360\n 0x6a89364\tDB::KVStore::preHandleSnapshotToFiles(std::__1::shared_ptrDB::Region, DB::SSTViewVec, unsigned long, unsigned long, DB::TMTContext&) [tiflash+111711076]\n \tdbms/src/Storages/Transaction/ApplySnapshot.cpp:275\n 0x6ae4666\tPreHandleSnapshot [tiflash+112084582]\n \tdbms/src/Storages/Transaction/ProxyFFI.cpp:388\n 0x7fbaee7abe28\tengine_store_ffi::$LT$impl$u20$engine_store_ffi…interfaces…root…DB…EngineStoreServerHelper$GT$::pre_handle_snapshot::h14b79dee0dc709dc [libtiflash_proxy.so+17415720]\n 0x7fbaee7a3899\tengine_store_ffi::observer::pre_handle_snapshot_impl::ha6024895a135c813 [libtiflash_proxy.so+17381529]\n 0x7fbaee797236\tyatp::task::future::RawTask$LT$F$GT$::poll::haa9f3b11a91f3b25 [libtiflash_proxy.so+17330742]\n 0x7fbaf05a59b3\t$LT$yatp…task…future…Runner$u20$as$u20$yatp…pool…runner…Runner$GT$::handle::h3ec77eaab16f0540 [libtiflash_proxy.so+48847283]\n 0x7fbaf059820c\tstd::sys_common::backtrace::__rust_begin_short_backtrace::h2dafe5d145aa6fe4 [libtiflash_proxy.so+48792076]\n 0x7fbaf0598ccc\tcore::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h8730f42d18399d62 [libtiflash_proxy.so+48794828]\n 0x7fbaefd13375\tstd::sys::unix::thread_start::hd2791a9cabec1fda [libtiflash_proxy.so+39859061]\n \t/rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/sys/unix/thread.rs:108\n 0x7fbaed3cc4ca\t [libc.so.6+570570]”] [source=BaseDaemon] [thread_id=27]
【Resource Configuration】
【Attachments: Screenshots/Logs/Monitoring】
Data replication process stalled
mysql> SELECT * FROM information_schema.tiflash_replica;
±-------------±---------------------±---------±--------------±----------------±----------±---------+
| TABLE_SCHEMA | TABLE_NAME | TABLE_ID | REPLICA_COUNT | LOCATION_LABELS | AVAILABLE | PROGRESS |
±-------------±---------------------±---------±--------------±----------------±----------±---------+
| test | GLOBAL_VARIABLES | 107 | 1 | | 0 | 0 |
| test | cluster_config | 109 | 1 | | 0 | 0 |
| test | warehouse | 288 | 1 | | 0 | 0 |
| test | history | 318 | 1 | | 0 | 0 |
| test | new_order | 328 | 1 | | 0 | 0 |
| test | orders | 338 | 1 | | 0 | 0 |
| test | item | 368 | 1 | | 0 | 0 |
| test | t_0008 | 523 | 1 | | 0 | 0 |
| test | t_0010 | 528 | 1 | | 0 | 0 |
| test | t_0001 | 530 | 1 | | 0 | 0 |
| test | t1 | 687 | 1 | | 0 | 0 |
| test | t2 | 692 | 1 | | 0 | 0 |
| test | t3 | 696 | 1 | | 0 | 0 |
| test | t4 | 698 | 1 | | 0 | 0 |
| test | t_test | 703 | 1 | | 0 | 0 |
| test | cluster_config_612 | 716 | 1 | | 0 | 0 |
| test | GLOBAL_VARIABLES_612 | 718 | 1 | | 0 | 0 |
| test | cluster_config_650 | 782 | 1 | | 0 | 0 |
| test | GlOBAL_VARIABLES_650 | 784 | 1 | | 0 | 0 |
| test | stock | 358 | 1 | | 0 | 0 |
| test | customer | 308 | 1 | | 0 | 0 |
| test | order_line | 348 | 1 | | 0 | 0 |
| test | district | 298 | 1 | | 0 | 0.12 |
±-------------±---------------------±---------±--------------±----------------±----------±---------+
23 rows in set (0.21 sec)
Tiflash process keeps restarting:
[tidb@olap-high-01 log]$ ps -ef|grep tiflash
tidb 474164 1 99 17:26 ? 00:00:12 bin/tiflash/tiflash server --config-file conf/tiflash.toml
tidb 475271 452527 0 17:27 pts/0 00:00:00 grep --color=auto tiflash
[tidb@olap-high-01 log]$ ps -ef|grep tiflash
tidb 474164 1 99 17:26 ? 00:00:13 bin/tiflash/tiflash server --config-file conf/tiflash.toml
tidb 475273 452527 0 17:27 pts/0 00:00:00 grep --color=auto tiflash
[tidb@olap-high-01 log]$ ps -ef|grep tiflash
tidb 474164 1 99 17:26 ? 00:00:13 bin/tiflash/tiflash server --config-file conf/tiflash.toml
tidb 475275 452527 0 17:27 pts/0 00:00:00 grep --color=auto tiflash
[tidb@olap-high-01 log]$ ps -ef|grep tiflash
tidb 475285 452527 0 17:27 pts/0 00:00:00 grep --color=auto tiflash
[tidb@olap-high-01 log]$ ps -ef|grep tiflash
tidb 475287 452527 0 17:27 pts/0 00:00:00 grep --color=auto tiflash
[tidb@olap-high-01 log]$ ps -ef|grep tiflash
tidb 475289 452527 0 17:27 pts/0 00:00:00 grep --color=auto tiflash
[tidb@olap-high-01 log]$ ps -ef|grep tiflash
tidb 475290 1 99 17:27 ? 00:00:08 bin/tiflash/tiflash server --config-file conf/tiflash.toml
tidb 476381 452527 0 17:27 pts/0 00:00:00 grep --color=auto tiflash