TiFlash suddenly crashes and fails to start

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiFlash突然宕机,启动不起来

| username: 大飞飞呀

【TiDB Usage Environment】Production Environment
【TiDB Version】7.1.0
【Reproduction Path】Operations performed that led to the issue
【Encountered Issue: Phenomenon and Impact】
[2023/10/30 22:54:02.439 +08:00] [WARN] [store.rs:1793] [“set thread priority for raftstore failed”] [error=“Os { code: 13, kind: PermissionDenied, message: "Permission denied" }”]

[2023/10/30 23:05:51.485 +08:00] [ERROR] [Exception.cpp:90] ["Code: 33, e.displayText() = DB::Exception: Cannot read all data, e.what() = DB::Exception, Stack trace:
       0x1bfe42e\tDB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+29353006]
                \tdbms/src/Common/Exception.h:46
       0x1c84ff0\tDB::ReadBuffer::readStrict(char*, unsigned long) [tiflash+29904880]
                \tdbms/src/IO/ReadBuffer.h:161
       0x73bcadf\tDB::DM::readSegmentMetaInfo(DB::ReadBuffer&, DB::DM::Segment::SegmentMetaInfo&) [tiflash+121359071]
                \tdbms/src/Storages/DeltaMerge/Segment.cpp:318
       0x73bcd48\tDB::DM::Segment::restoreSegment(std::__1::shared_ptr<DB::Logger> const&, DB::DM::DMContext&, unsigned long) [tiflash+121359688]
                \tdbms/src/Storages/DeltaMerge/Segment.cpp:349
       0x7348013\tDB::DM::DeltaMergeStore::DeltaMergeStore(DB::Context&, bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned int, long, bool, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > const&, DB::DM::ColumnDefine const&, bool, unsigned long, DB::DM::DeltaMergeStore::Settings const&, DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >*) [tiflash+120881171]
                \tdbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp:303
       0x7e5ad37\tDB::StorageDeltaMerge::getAndMaybeInitStore(DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >*) [tiflash+132492599]
                \tdbms/src/Storages/StorageDeltaMerge.cpp:1842
       0x7e66d33\tDB::StorageDeltaMerge::getSchemaSnapshotAndBlockForDecoding(DB::TableDoubleLockHolder<false> const&, bool) [tiflash+132541747]
                \tdbms/src/Storages/StorageDeltaMerge.cpp:1204
       0x7f7d614\tDB::writeRegionDataToStorage(DB::Context&, DB::RegionPtrWithBlock const&, std::__1::vector<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> >, std::__1::allocator<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> > > >&, std::__1::shared_ptr<DB::Logger> const&)::$_2::operator()(bool) const [tiflash+133682708]
                \tdbms/src/Storages/Transaction/PartitionStreams.cpp:126
       0x7f7a18a\tDB::writeRegionDataToStorage(DB::Context&, DB::RegionPtrWithBlock const&, std::__1::vector<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> >, std::__1::allocator<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> > > >&, std::__1::shared_ptr<DB::Logger> const&) [tiflash+133669258]
                \tdbms/src/Storages/Transaction/PartitionStreams.cpp:181
       0x7f79e4c\tDB::RegionTable::writeBlockByRegion(DB::Context&, DB::RegionPtrWithBlock const&, std::__1::vector<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> >, std::__1::allocator<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> > > >&, std::__1::shared_ptr<DB::Logger> const&, bool) [tiflash+133668428]
                \tdbms/src/Storages/Transaction/PartitionStreams.cpp:359
       0x7fa41ad\tDB::Region::handleWriteRaftCmd(DB::WriteCmdsView const&, unsigned long, unsigned long, DB::TMTContext&) [tiflash+133841325]
                \tdbms/src/Storages/Transaction/Region.cpp:721
       0x7f620ce\tDB::KVStore::handleWriteRaftCmd(DB::WriteCmdsView const&, unsigned long, unsigned long, unsigned long, DB::TMTContext&) const [tiflash+133570766]
                \tdbms/src/Storages/Transaction/KVStore.cpp:300
       0x7f82195\tHandleWriteRaftCmd [tiflash+133702037]
                \tdbms/src/Storages/Transaction/ProxyFFI.cpp:97
0x7f8e31e4fb98\t_$LT$engine_store_ffi..observer..TiFlashObserver$LT$T$C$ER$GT$$u20$as$u20$raftstore..coprocessor..QueryObserver$GT$::post_exec_que:

【Resource Configuration】Go to TiDB Dashboard - Cluster Info - Hosts and take a screenshot of this page
【Attachments: Screenshots/Logs/Monitoring】

| username: 芮芮是产品 | Original post link

@大飞飞jeffery Give 777 permissions to tidb under /tmp

| username: 像风一样的男子 | Original post link

The error seems to be a disk permission issue.

| username: Fly-bird | Original post link

No write permission, this permission should be the tidb user’s permission, not the root permission.

| username: Kongdom | Original post link

Based on the error message, it seems to be caused by a lack of write permissions.

| username: zhanggame1 | Original post link

Which component’s log is “Permission denied”?

| username: TiDBer_yyy | Original post link

It should not be this issue. Another post on the same topic:

| username: TiDBer_yyy | Original post link

How do you identify the root cause of the problem?

| username: TiDBer_yyy | Original post link

tiflash_tikv.log

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.