Error Occurred When Executing Insert in TiSpark

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiSpark执行Insert时报错

| username: Zealot

TiDB version: 7.1.2
TiSpark version: [tispark-assembly-3.3_2.12-3.2.2.jar]
Spark version: 3.3.1

When executing “insert into select”, it was executed 3 times, but only succeeded once. The other two times reported the following error:

operating ExecuteStatement: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 46.0 failed 4 times, most recent failure: Lost task 1.3 in stage 46.0 (TID 2248) (hadoop-0003 executor 2): org.tikv.common.exception.TiBatchWriteException: Execution exception met.
at org.tikv.txn.TwoPhaseCommitter.doPrewriteSecondaryKeys(TwoPhaseCommitter.java:308)
at org.tikv.txn.TwoPhaseCommitter.prewriteSecondaryKeys(TwoPhaseCommitter.java:259)
at com.pingcap.tispark.utils.TwoPhaseCommitHepler.$anonfun$prewriteSecondaryKeyByExecutors$1(TwoPhaseCommitHepler.scala:102)
at com.pingcap.tispark.utils.TwoPhaseCommitHepler.$anonfun$prewriteSecondaryKeyByExecutors$1$adapted(TwoPhaseCommitHepler.scala:90)
at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2(RDD.scala:1011)
at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2$adapted(RDD.scala:1011)
at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2268)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:136)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: org.tikv.common.exception.TiBatchWriteException: > max retry number 3, oldRegion={Region[9855557] ConfVer[7425] Version[419822] Store[2853733] KeyRange[t\200\000\000\000\000\020\262\335_i\200\000\000\000\000\000\000\001\001\0009\0001\0001\0001\377\0000\0001\0000\0005\377\000M\000A\0000\0002\377\0000\0009\000G\0006\377\0004\000J\000\000\000\000\373]:[t\200\000\000\000\000\020\262\335_i\200\000\000\000\000\000\000\001\001\0009\0001\0001\0001\377\0000\0001\0000\0008\377\0007\0002\0003\0009\377\0006\0009\0003\0007\377\0006\000Y\000\000\000\000\373]}, currentRegion={Region[9857771] ConfVer[7425] Version[419823] Store[2853733] KeyRange[t\200\000\000\000\000\020\262\335_i\200\000\000\000\000\000\000\001\001\0009\0001\0001\0001\377\0000\0001\0000\0005\377\000M\000A\0000\0002\377\0000\0009\000G\0006\377\0004\000J\000\000\000\000\373]:[t\200\000\000\000\000\020\262\335_i\200\000\000\000\000\000\000\001\001\0009\0001\0001\0001\377\0000\0001\0000\0005\377\000M\000A\0000\0005\377\000D\000A\000B\0008\377\0005\000Y\000\000\000\000\373]}
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.tikv.txn.TwoPhaseCommitter.doPrewriteSecondaryKeys(TwoPhaseCommitter.java:285)
… 14 more
Caused by: org.tikv.common.exception.TiBatchWriteException: > max retry number 3, oldRegion={Region[9855557] ConfVer[7425] Version[419822] Store[2853733] KeyRange[t\200\000\000\000\000\020\262\335_i\200\000\000\000\000\000\000\001\001\0009\0001\0001\0001\377\0000\0001\0000\0005\377\000M\000A\0000\0002\377\0000\0009\000G\0006\377\0004\000J\000\000\000\000\373]:[t\200\000\000\000\000\020\262\335_i\200\000\000\000\000\000\000\001\001\0009\0001\0001\0001\377\0000\0001\0000\0008\377\0007\0002\0003\0009\377\0006\0009\0003\0007\377\0006\000Y\000\000\000\000\373]}, currentRegion={Region[9857771] ConfVer[7425] Version[419823] Store[2853733] KeyRange[t\200\000\000\000\000\020\262\335_i\200\000\000\000\000\000\000\001\001\0009\0001\0001\0001\377\0000\0001\0000\0005\377\000M\000A\0002\377\0000\0009\000G\0006\377\0004\000J\000\000\000\000\373]:[t\200\000\000\000\000\020\262\335_i\200\000\000\000\000\000\000\001\001\0009\0001\0001\0001\377\0000\0001\0000\0005\377\000M\000A\0005\377\000D\000A\000B\0008\377\0005\000Y\000\000\000\000\373]}
at org.tikv.txn.TwoPhaseCommitter.doPrewriteSecondaryKeysInBatchesWithRetry(TwoPhaseCommitter.java:361)
at org.tikv.txn.TwoPhaseCommitter.retryPrewriteBatch(TwoPhaseCommitter.java:390)
at org.tikv.txn.TwoPhaseCommitter.doPrewriteSecondaryKeysInBatchesWithRetry(TwoPhaseCommitter.java:369)
at org.tikv.txn.TwoPhaseCommitter.retryPrewriteBatch(TwoPhaseCommitter.java:390)
at org.tikv.txn.TwoPhaseCommitter.doPrewriteSecondaryKeysInBatchesWithRetry(TwoPhaseCommitter.java:369)
at org.tikv.txn.TwoPhaseCommitter.retryPrewriteBatch(TwoPhaseCommitter.java:390)
at org.tikv.txn.TwoPhaseCommitter.doPrewriteSecondaryKeysInBatchesWithRetry(TwoPhaseCommitter.java:369)
at org.tikv.txn.TwoPhaseCommitter.retryPrewriteBatch(TwoPhaseCommitter.java:390)
at org.tikv.txn.TwoPhaseCommitter.doPrewriteSecondaryKeysInBatchesWithRetry(TwoPhaseCommitter.java:369)
at org.tikv.txn.TwoPhaseCommitter.lambda$doPrewriteSecondaryKeys$0(TwoPhaseCommitter.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
… 3 more

| username: xfworld | Original post link

The documentation hasn’t been updated yet, so it temporarily doesn’t support it (please wait, please wait for the version update).

| username: Zealot | Original post link

It should be supported. I can successfully execute the insert into values statement. My issue is similar to the one below:

I read this post, and the log information only contains retry information without explaining why it is retrying. I checked my TiSpark logs and didn’t find any additional log information. I wonder if there is a place where I can find more detailed information.

| username: Zealot | Original post link

I looked at the source code, and it seems there is an INFO level message.

| username: Zealot | Original post link

It seems like this is the only useful information.

| username: dba远航 | Original post link

Check if it is caused by the table not having a primary key or unique key.

| username: Zealot | Original post link

The table has a primary key.

| username: Jellybean | Original post link

This error seems to be an exception occurring during the two-phase commit process of a distributed transaction. It is usually due to transaction conflicts with the submitted data or version check failures. Internally, it will retry with backoff, and eventually report an error.

Try reducing the batch size of a single Spark task submission.
If possible, also consider reducing the concurrency of the Spark job.

| username: Zealot | Original post link

Reducing concurrency (spark.default.parallelism) seems to be useful, I will observe for a few more days.

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.