Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.
Original topic: jdbc batch insert失败但是没有异常怎么排查
tidb v6.1
Loading into the database through Flink
Inserting data via JDBC batch insert method
The situation is that under normal circumstances, the insert statement returns an array with a size equal to the batch size, and all elements are -2.
However, sometimes it returns an array where each element is -2, but the size is smaller than the batch size. No exceptions are caught by the program, and no anomalies are found in the TiDB logs. Seeking troubleshooting ideas.
Could you please share the code snippet and table creation statement or the reproduction method?
The table is a very ordinary partitioned table, partitioned by day, with a primary key and an index on the table.
Is there a pattern to the occurrence of the error? Is there an SQL log when returning a -2 array?
No, you can see the rollback in the monitoring.
There are some messages about inserting into the delete-range table.
Please provide the reproduction steps. Without detailed information, I can only guess…
public void executeBatch(Connection connection) throws SQLException {
long start = System.currentTimeMillis();
if (rows.size() != 0) {
try {
rows.forEach(row → {
try {
JDBCUtils.setRecordToStatement(this.statement, this.fieldTypes, (Row) row);
this.statement.addBatch();
} catch (SQLException e) {
throw new RuntimeException(e);
}
});
int ints = this.statement.executeBatch();
connection.commit();
int j = ints.length;
if (rows.size() != j) {
throw new RuntimeException(String.format(“The number of records inserted is inconsistent, Rows.size():%d, number of successful executions, length of the int array returned by batch execution: %s”, rows.size(), j));
}
long s = System.currentTimeMillis() - start;
LOG.info(“Batch inserted {} records, took {}ms, actually inserted {} records”, rows.size(), s, j);
rows.clear();
} catch (Exception e) {
LOG.info(“Batch insertion failed, switching to single record insertion…{}, rows:{}”, e, rows);
connection.rollback();
connection.commit();
this.statement.clearBatch();
executeUpdate(connection);
}
}
}
public synchronized void executeUpdate(Connection connection) {
long start = System.currentTimeMillis();
AtomicInteger success = new AtomicInteger();
AtomicInteger failed = new AtomicInteger();
rows.forEach(row -> {
try {
JDBCUtils.setRecordToStatement(statement, fieldTypes, row);
int i = statement.executeUpdate();
if (i <= 0) {
throw new RuntimeException("executeUpdate failed, affected " + i + " rows");
}
connection.commit();
success.getAndAdd(i);
} catch (Exception e) {
try {
// connection.rollback();
connection.commit();
} catch (SQLException e1) {
throw new RuntimeException(e1);
}
failed.getAndIncrement();
LOG.error("Insert failed, data: {{}}", row);
LOG.error("", e);
}
});
long s = System.currentTimeMillis() - start;
LOG.info("execute update completed, total {} records, successfully inserted {} records, failed {} records, took {}ms", rows.size(), success.get(), failed.get(), s);
rows.clear();
}
I remember that the JVM will modify the returned exception for certain types of exceptions under a large number of repetitions (the exception object has no message and stack), retaining only the stack error information from the front, and the latter will be gone.
Reference: https://www.51cto.com/article/658494.html
So consider restarting, and then see what the error is.
You can also check TiDB Dashboard to see if there are any abnormal SQLs, or check the logs to see if the related insert SQLs are coming in.
This topic was automatically closed 60 days after the last reply. No new replies are allowed.
|
username: dxss-lee
|
username: 
|
username: 
|
username: 
|
username: