After a 10-minute SQL query from Oracle, writing to TiDB using Datax causes the Datax task to crash

This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 从Oracle 经过10分钟 sql 查询后 用 Datax 写入TIDB,Datax 任务直接挂掉

| username: 卡卡其其

[TiDB Usage Environment] Production Environment
[TiDB Version] 6.5.0
[Reproduction Path] Operations performed that led to the issue
connect_timeout = 1800s
delayed_insert_timeout = 1800s
[Encountered Issue: Problem Phenomenon and Impact]
Unable to use Datax for long (10 minutes) queries to write into TiDB, shorter queries (less than 5 minutes) can be written
[Resource Configuration] 2g - 4g
[Attachments: Screenshots/Logs/Monitoring]

| username: tidb狂热爱好者 | Original post link

Use the upgraded version of CloudCanal.

| username: xingzhenxiang | Original post link

Try setting @@global.tidb_gc_life_time = “8h”.

| username: wzf0072 | Original post link

Brother Jiang, are you using CloudCanal for synchronization in your production environment?

| username: xingzhenxiang | Original post link

I am considering whether to handle one transaction at a time. The default GC is 10 minutes, so it is recommended to set a longer GC time.

| username: TiDBer_pkQ5q1l0 | Original post link

Looking at this error, it seems to be related to the JDBC connection configuration. How about trying Addax?

| username: 卡卡其其 | Original post link

What is addax? This is my JDBC parameter:

“jdbcUrl”: “jdbc:mysql://xxxxxxxxx:4000/ads_global?useUnicode=true&characterEncoding=utf-8&useSSL=false&autoReconnect=true&useServerPrepStmts=true&cachePrepStmts=true&prepStmtCacheSize=1000&prepStmtCacheSqlLimit=2048000&useConfigs=maxPerformance&rewriteBatchedStatements=true&allowMultiQueries=true&useCursorFetch=true”

| username: 卡卡其其 | Original post link

Can you provide an address? Do you know which company owns Canal and YunCanal? The DataX I have here is an offline synchronization tool.

| username: Jolyne | Original post link

CloudCanal, this Product Introduction | CloudCanal of ClouGence

| username: 卡卡其其 | Original post link

After this adjustment, the TiDB database was not restarted, and the task still reported an error after running for about 10 minutes.

| username: 卡卡其其 | Original post link

It’s paid, so I won’t consider it. I thought TiDB’s components were free. :cry:

| username: Jolyne | Original post link

There is a free version, but it has limitations. You can try it out first. TiDB also has its own components, such as DM and CDC, which are data synchronization tools.