TiFlash v5.3.0 Error and Restart

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tiflash v5.3.0 报错重启

| username: lindoubled

[TiDB Usage Environment]
Production Environment
[TiDB Version]
v5.3.0
[Reproduction Path]
None
[Encountered Problem: Problem Phenomenon and Impact]

[2023/01/02 01:55:35.309 +08:00] [ERROR] [<unknown>] ["DeltaMergeStore[db_428.t_2727]: Task MergeDelta on Segment [1] failed. Error msg: DeltaMerge return wrong result, current handle[23793238]version[438459978676699146]@read[31]@pos[3102] is expected >= last_handle[23814470]last_version[438459978624532501]@read[31]@pos[3101]"] [thread_id=21670]
[2023/01/02 01:55:35.331 +08:00] [ERROR] [<unknown>] ["void DB::BackgroundProcessingPool::threadFunction(): Code: 0, e.displayText() = DB::Exception: DeltaMerge return wrong result, current handle[23793238]version[438459978676699146]@read[31]@pos[3102] is expected >= last_handle[23814470]last_version[438459978624532501]@read[31]@pos[3101], e.what() = DB::Exception, Stack trace:\n\n0. bin/tiflash/tiflash(StackTrace::StackTrace()+0x16) [0x370f6a6]\n1. bin/tiflash/tiflash(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x26) [0x37034a6]\n2. bin/tiflash/tiflash(DB::DM::DeltaMergeBlockInputStream<DB::DM::DeltaValueReader, DB::DM::DTCompactedEntries<55ul, 20ul, 3ul>::Iterator, true>::beforeReturnBlock(DB::Block const&)+0x53c) [0x75e342c]\n3. bin/tiflash/tiflash(DB::DM::DeltaMergeBlockInputStream<DB::DM::DeltaValueReader, DB::DM::DTCompactedEntries<55ul, 20ul, 3ul>::Iterator, true>::read()+0xc4) [0x75e6c64]\n4. bin/tiflash/tiflash(bool DB::DM::placeInsert<true, DB::DM::DeltaTree<DB::DM::EmptyValueSpace, 55ul, 20ul, 3ul, DB::ArenaWithFreeLists> >(std::shared_ptr<DB::DM::SkippableBlockInputStream> const&, DB::Block const&, DB::DM::RowKeyRange const&, bool, DB::DM::DeltaTree<DB::DM::EmptyValueSpace, 55ul, 20ul, 3ul, DB::ArenaWithFreeLists>&, unsigned long, DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul> const&, std::vector<DB::SortColumnDescription, std::allocator<DB::SortColumnDescription> > const&)+0x1ec) [0x75de72c]\n5. bin/tiflash/tiflash(bool DB::DM::Segment::placeUpsert<true>(DB::DM::DMContext const&, std::shared_ptr<DB::DM::StableValueSpace::Snapshot> const&, std::shared_ptr<DB::DM::DeltaValueReader> const&, unsigned long, DB::Block&&, DB::DM::DeltaTree<DB::DM::EmptyValueSpace, 55ul, 20ul, 3ul, DB::ArenaWithFreeLists>&, DB::DM::RowKeyRange const&, bool) const+0x3a1) [0x75e0aa1]\n6. bin/tiflash/tiflash(DB::DM::Segment::ensurePlace(DB::DM::DMContext const&, std::shared_ptr<DB::DM::StableValueSpace::Snapshot> const&, std::shared_ptr<DB::DM::DeltaValueReader> const&, std::vector<DB::DM::RowKeyRange, std::allocator<DB::DM::RowKeyRange> > const&, unsigned long) const+0x496) [0x75c41a6]\n7. bin/tiflash/tiflash(DB::DM::Segment::getReadInfo(DB::DM::DMContext const&, std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > const&, std::shared_ptr<DB::DM::SegmentSnapshot> const&, std::vector<DB::DM::RowKeyRange, std::allocator<DB::DM::RowKeyRange> > const&, unsigned long) const+0x237) [0x75c54c7]\n8. bin/tiflash/tiflash(DB::DM::Segment::getInputStreamForDataExport(DB::DM::DMContext const&, std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > const&, std::shared_ptr<DB::DM::SegmentSnapshot> const&, DB::DM::RowKeyRange const&, unsigned long, bool) const+0x87) [0x75c72a7]\n9. bin/tiflash/tiflash(DB::DM::Segment::prepareMergeDelta(DB::DM::DMContext&, std::shared_ptr<std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > > const&, std::shared_ptr<DB::DM::SegmentSnapshot> const&, DB::DM::WriteBatches&) const+0xa3) [0x75c75b3]\n10. bin/tiflash/tiflash(DB::DM::DeltaMergeStore::segmentMergeDelta(DB::DM::DMContext&, std::shared_ptr<DB::DM::Segment> const&, DB::DM::DeltaMergeStore::TaskRunThread, std::shared_ptr<DB::DM::SegmentSnapshot>)+0x3c0) [0x7596250]\n11. bin/tiflash/tiflash(DB::DM::DeltaMergeStore::handleBackgroundTask(bool)+0x3ae) [0x759e8fe]\n12. bin/tiflash/tiflash(DB::BackgroundProcessingPool::threadFunction()+0x55f) [0x767c45f]\n13. bin/tiflash/tiflash() [0x8b729ef]\n14. /lib64/libpthread.so.0(+0x7ea5) [0x7feb2bdd9ea5]\n15. /lib64/libc.so.6(clone+0x6d) [0x7feb2b8008dd]\n"] [thread_id=21670]
[2023/01/02 01:55:35.403 +08:00] [ERROR] [<unknown>] ["DeltaMergeStore[db_428.t_2727]: Task MergeDelta on Segment [1] failed. Error msg: DeltaMerge return wrong result, current handle[23793238]version[438459978676699146]@read[31]@pos[3102] is expected >= last_handle[23814470]last_version[438459978624532501]@read[31]@pos[3101]"] [thread_id=39624]
[2023/01/02 01:55:35.424 +08:00] [ERROR] [<unknown>] ["void DB::BackgroundProcessingPool::threadFunction(): Code: 0, e.displayText() = DB::Exception: DeltaMerge return wrong result, current handle[23793238]version[438459978676699146]@read[31]@pos[3102] is expected >= last_handle[23814470]last_version[438459978624532501]@read[31]@pos[3101], e.what() = DB::Exception, Stack trace:\n\n0. bin/tiflash/tiflash(StackTrace::StackTrace()+0x16) [0x370f6a6]\n1. bin/tiflash/tiflash(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x26) [0x37034a6]\n2. bin/tiflash/tiflash(DB::DM::DeltaMergeBlockInputStream<DB::DM::DeltaValueReader, DB::DM::DTCompactedEntries<55ul, 20ul, 3ul>::Iterator, true>::beforeReturnBlock(DB::Block const&)+0x53c) [0x75e342c]\n3. bin/tiflash/tiflash(DB::DM::DeltaMergeBlockInputStream<DB::DM::DeltaValueReader, DB::DM::DTCompactedEntries<55ul, 20ul, 3ul>::Iterator, true>::read()+0xc4) [0x75e6c64]\n4. bin/tiflash/tiflash(bool DB::DM::placeInsert<true, DB::DM::DeltaTree<DB::DM::EmptyValueSpace, 55ul, 20ul, 3ul, DB::ArenaWithFreeLists> >(std::shared_ptr<DB::DM::SkippableBlockInputStream> const&, DB::Block const&, DB::DM::RowKeyRange const&, bool, DB::DM::DeltaTree<DB::DM::EmptyValueSpace, 55ul, 20ul, 3ul, DB::ArenaWithFreeLists>&, unsigned long, DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul> const&, std::vector<DB::SortColumnDescription, std::allocator<DB::SortColumnDescription> > const&)+0x1ec) [0x75de72c]\n5. bin/tiflash/tiflash(bool DB::DM::Segment::placeUpsert<true>(DB::DM::DMContext const&, std::shared_ptr<DB::DM::StableValueSpace::Snapshot> const&, std::shared_ptr<DB::DM::DeltaValueReader> const&, unsigned long, DB::Block&&, DB::DM::DeltaTree<DB::DM::EmptyValueSpace, 55ul, 20ul, 3ul, DB::ArenaWithFreeLists>&, DB::DM::RowKeyRange const&, bool) const+0x3a1) [0x75e0aa1]\n6. bin/tiflash/tiflash(DB::DM::Segment::ensurePlace(DB::DM::DMContext const&, std::shared_ptr<DB::DM::StableValueSpace::Snapshot> const&, std::shared_ptr<DB::DM::DeltaValueReader> const&, std::vector<DB::DM::RowKeyRange, std::allocator<DB::DM::RowKeyRange> > const&, unsigned long) const+0x496) [0x75c41a6]\n7. bin/tiflash/tiflash(DB::DM::Segment::getReadInfo(DB::DM::DMContext const&, std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > const&, std::shared_ptr<DB::DM::SegmentSnapshot> const&, std::vector<DB::DM::RowKeyRange, std::allocator<DB::DM::RowKeyRange> > const&, unsigned long) const+0x237) [0x75c54c7]\n8. bin/tiflash/tiflash(DB::DM::Segment::getInputStreamForDataExport(DB::DM::DMContext const&, std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > const&, std::shared_ptr<DB::DM::SegmentSnapshot> const&, DB::DM::RowKeyRange const&, unsigned long, bool) const+0x87) [0x75c72a7]\n9. bin/tiflash/tiflash(DB::DM::Segment::prepareMergeDelta(DB::DM::DMContext&, std::shared_ptr<std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > > const&, std::shared_ptr<DB::DM::SegmentSnapshot> const&, DB::DM::WriteBatches&) const+0xa3) [0x75c75b3]\n10. bin/tiflash/tiflash(DB::DM::DeltaMergeStore::segmentMergeDelta(DB::DM::DMContext&, std::shared_ptr<DB::DM::Segment> const&, DB::DM::DeltaMergeStore::TaskRunThread, std::shared_ptr<DB::DM::SegmentSnapshot>)+0x3c0) [0x7596250]\n11. bin/tiflash/tiflash(DB::DM::DeltaMergeStore::handleBackgroundTask(bool)+0x3ae) [0x759e8fe]\n12. bin/tiflash/tiflash(DB::BackgroundProcessingPool::threadFunction()+0x55f) [0x767c45f]\n13. bin/tiflash/tiflash() [0x8b729ef]\n14. /lib64/libpthread.so.0(+0x7ea5) [0x7feb2bdd9ea5]\n15. /lib64/libc.so.6(clone+0x6d) [0x7feb2b8008dd]\n"] [thread_id=39624]
[2023/01/02 01:55:35.540 +08:00] [ERROR] [<unknown>] ["DB::EngineStoreApplyRes DB::HandleWriteRaftCmd(const DB::EngineStoreServerWrap*, DB::WriteCmdsView, DB::RaftCmdHeader): Code: 0, e.displayText() = DB::Exception: DeltaMerge return wrong result, current handle[23793238]version[438459978676699146]@read[31]@pos[3102] is expected >= last_handle[23814470]last_version[438459978624532501]@read[31]@pos[3101], e.what() = DB::Exception, Stack trace:\n\n0. bin/tiflash/tiflash(StackTrace::StackTrace()+0x16) [0x370f6a6]\n1. bin/tiflash/tiflash(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x26) [0x37034a6]\n2. bin/tiflash/tiflash(DB::DM::DeltaMergeBlockInputStream<DB::DM::DeltaValueReader, DB::DM::DTCompactedEntries<55ul, 20ul, 3ul>::Iterator, true>::beforeReturnBlock(DB::Block const&)+0x53c) [0x75e342c]\n3. bin/tiflash/tiflash(DB::DM::DeltaMergeBlockInputStream<DB::DM::DeltaValueReader, DB::DM::DTCompactedEntries<55ul, 20ul, 3ul>::Iterator, true>::read()+0xc4) [0x75e6c64]\n4. bin/tiflash/tiflash(bool DB::DM::placeInsert<true, DB::DM::DeltaTree<DB::DM::EmptyValueSpace, 55ul, 20ul, 3ul, DB::ArenaWithFreeLists> >(std::shared_ptr<DB::DM::SkippableBlockInputStream> const&, DB::Block const&, DB::DM::RowKeyRange const&, bool, DB::DM::DeltaTree<DB::DM::EmptyValueSpace, 55ul, 20ul, 3ul, DB::ArenaWithFreeLists>&, unsigned long, DB::PODArray<unsigned long, 4096ul, Allocator<false>, 15ul, 16ul> const&, std::vector<DB::SortColumnDescription, std::allocator<DB::SortColumnDescription> > const&)+0x1ec) [0x75de72c]\n5. bin/tiflash/tiflash(bool DB::DM::Segment::placeUpsert<true>(DB::DM::DMContext const&, std::shared_ptr<DB::DM::StableValueSpace::Snapshot> const&, std::shared_ptr<DB::DM::DeltaValueReader> const&, unsigned long, DB::Block&&, DB::DM::DeltaTree<DB::DM::EmptyValueSpace, 55ul, 20ul, 3ul, DB::ArenaWithFreeLists>&, DB::DM::RowKeyRange const&, bool) const+0x3a1) [0x75e0aa1]\n6. bin/tiflash/tiflash(DB::DM::Segment::ensurePlace(DB::DM::DMContext const&, std::shared_ptr<DB::DM::StableValueSpace::Snapshot> const&, std::shared_ptr<DB::DM::DeltaValueReader> const&, std::vector<DB::DM::RowKeyRange, std::allocator<DB::DM::RowKeyRange> > const&, unsigned long) const+0x496) [0x75c41a6]\n7. bin/tiflash/tiflash(DB::DM::Segment::getReadInfo(DB::DM::DMContext const&, std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > const&, std::shared_ptr<DB::DM::SegmentSnapshot> const&, std::vector<DB::DM::RowKeyRange, std::allocator<DB::DM::RowKeyRange> > const&, unsigned long) const+0x237) [0x75c54c7]\n8. bin/tiflash/tiflash(DB::DM::Segment::getInputStreamForDataExport(DB::DM::DMContext const&, std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > const&, std::shared_ptr<DB::DM::SegmentSnapshot> const&, DB::DM::RowKeyRange const&, unsigned long, bool) const+0x87) [0x75c72a7]\n9. bin/tiflash/tiflash(DB::DM::Segment::prepareMergeDelta(DB::DM::DMContext&, std::shared_ptr<std::vector<DB::DM::ColumnDefine, std::allocator<DB::DM::ColumnDefine> > > const&, std::shared_ptr<DB::DM::SegmentSnapshot> const&, DB::DM::WriteBatches&) const+0xa3) [0x75c75b3]\n10. bin/tiflash/tiflash(DB::DM::DeltaMergeStore::segmentMergeDelta(DB::DM::DMContext&, std::shared_ptr<DB::DM::Segment> const&, DB::DM::DeltaMergeStore::TaskRunThread, std::shared_ptr<DB::DM::SegmentSnapshot>
2
| username: xfworld | Original post link

It looks like the error is this, probably a known bug. TiFlash updates too quickly…

Suggestion: Cancel the original synchronization configuration, wait for the data cleanup to complete, and then re-enable synchronization…

3
| username: tidb菜鸟一只 | Original post link

TiDB seems to recommend disabling swap, right?

4
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.