Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.Original topic: 升级6.1后，TiFlash服务异常
|
username: qihuajun
【TiDB Usage Environment】Production Environment
【TiDB Version】5.7.25-TiDB-v6.1.0
【Problem Encountered】After upgrading to 6.1.0 yesterday, the TiFlash service encountered a failure early this morning. The TiFlash service ran out of memory, causing the service to restart. After the cluster restarted this morning, queries on TiFlash became very slow after running for a while. Suspecting that it was caused by partition pruning of the partition table, we changed tidb_partition_prune_mode from dynamic to static, and it is currently running normally. We are still observing.
Additional Information: Currently, TiFlash nodes and TiKV are mixed.
We are unable to determine the root cause of the problem. Please help analyze.
【Attachments】Related logs and monitoring (https://metricstool.pingcap.com/)
Partial logs during the service outage early in the morning:
[2022/06/23 03:05:35.467 +08:00] [ERROR] [DAGDriver.cpp:197] ["DAGDriver:DB Exception: Allocator: Cannot malloc 1.00 MiB., errno: 12, strerror: Cannot allocate memory
0x1d272d3 StackTrace::StackTrace() [tiflash+30569171]
dbms/src/Common/StackTrace.cpp:23
0x1d248d6 DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+30558422]
dbms/src/Common/Exception.h:41
0x1defa2e DB::SharedQueryBlockInputStream::waitThread() [tiflash+31390254]
dbms/src/DataStreams/SharedQueryBlockInputStream.h:169
0x1deed0f DB::SharedQueryBlockInputStream::readImpl() [tiflash+31386895]
dbms/src/DataStreams/SharedQueryBlockInputStream.h:124
0x6688b8a DB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator<false>, 15ul, 16ul>*&, bool) [tiflash+107514762]
dbms/src/DataStreams/IProfilingBlockInputStream.cpp:75
0x66888c4 DB::IProfilingBlockInputStream::read() [tiflash+107514052]
dbms/src/DataStreams/IProfilingBlockInputStream.cpp:43
0x738e696 DB::ParallelInputsProcessor<DB::UnionBlockInputStream<(DB::StreamUnionMode)0, false>::Handler, (DB::StreamUnionMode)0>::loop(unsigned long) [tiflash+121169558]
dbms/src/DataStreams/ParallelInputsProcessor.h:298
0x738e1f7 DB::ParallelInputsProcessor<DB::UnionBlockInputStream<(DB::StreamUnionMode)0, false>::Handler, (DB::StreamUnionMode)0>::thread(unsigned long) [tiflash+121168375]
dbms/src/DataStreams/ParallelInputsProcessor.h:236
0x1db172c auto DB::wrapInvocable<std::__1::function<void ()> >(bool, std::__1::function<void ()>&&)::'lambda'()::operator()() [tiflash+31135532]
dbms/src/Common/wrapInvocable.h:36
0x1db1895 std::__1::packaged_task<void ()>::operator()() [tiflash+31135893]
/usr/local/bin/../include/c++/v1/future:2089
0x1da0e68 DB::DynamicThreadPool::executeTask(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >&) [tiflash+31067752]
dbms/src/Common/DynamicThreadPool.cpp:101
0x1da0d46 DB::DynamicThreadPool::dynamicWork(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >) [tiflash+31067462]
dbms/src/Common/DynamicThreadPool.cpp:142
0x1da29bd auto std::__1::thread DB::ThreadFactory::newThread<void (DB::DynamicThreadPool::*)(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >), DB::DynamicThreadPool*, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> > >(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, void (DB::DynamicThreadPool::*&&)(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >), DB::DynamicThreadPool*&&, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >&&)::'lambda'(auto&&...)::operator()<DB::DynamicThreadPool*, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> > >(auto&&...) const [tiflash+31074749]
dbms/src/Common/ThreadFactory.h:47
0x1da2799 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::thread DB::ThreadFactory::newThread<void (DB::DynamicThreadPool::*)(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >), DB::DynamicThreadPool*, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> > >(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, void (DB::DynamicThreadPool::*&&)(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >), DB::DynamicThreadPool*&&, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >&&)::'lambda'(auto&&...), DB::DynamicThreadPool*, std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> > > >(void*) [tiflash+31074201]
/usr/local/bin/../include/c++/v1/thread:291
0x7f62c8ca4ea5 start_thread [libpthread.so.0+32421]
0x7f62c87b79fd __clone [libc.so.6+1042941]"] [thread_id=882]
[2022/06/23 03:05:35.474 +08:00] [ERROR] [Server.cpp:310] ["grpc:/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tics/contrib/grpc/src/cpp/thread_manager/thread_manager.cc, line number: 38, log msg : Could not create grpc_sync_server worker-thread"] [thread_id=59020]
Partial exception monitoring screenshots during slow queries in the morning:
For questions related to performance optimization and fault troubleshooting, please download the script and run it. Please select all and copy-paste the terminal output results for upload.
|
username: 
|
username: 
|
username: