TiSpark, Insert Affected by spark.tispark.replica_read Parameter

This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: tispark, insert 受 spark.tispark.replica_read 参数影响

| username: ShawnYan

The TiSpark insert operation is affected by the parameter spark.tispark.replica_read.

In spark-default.conf, setting spark.tispark.replica_read follower causes an error when executing insert in spark-shell.

If this parameter line is commented out, the operation succeeds:


Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.3.1

Using Scala version 2.12.15 (OpenJDK 64-Bit Server VM, Java 1.8.0_292)

| username: Jellybean | Original post link

Setting this value should directly access the follower node. In Raft, the follower node only supports read operations. Removing this configuration will default to accessing the Raft leader node. The leader node supports both read and write operations, so you can insert.

| username: ShawnYan | Original post link

No, the replica_read parameter, as the name suggests, controls which node to select during “read” operations and should not affect “write” operations. Similar parameters in the TiDB server are also designed to address read hotspot issues. In any case, the situation where reads “block” writes is something that needs attention.

| username: Jellybean | Original post link

The feature of tidb_replica_read indeed “only” controls “read” operations. My first reaction to the parameter spark.tispark.replica_read is that the operation from TiSpark to TiKV replica only supports read, not write.

I can understand the confusion here. The meaning of this parameter might need some clarification from the official team to avoid any misunderstanding.

| username: redgame | Original post link

Setting the spark.tispark.replica_read parameter to follower may lead to data inconsistency.

| username: ShawnYan | Original post link

Well, I think it’s better to ask Yuhang for guidance @shiyuhang0

| username: shiyuhang0 | Original post link

It should be that the design was not well thought out. I think this parameter should not affect writing.
I have raised an issue, [BUG] Insert fail with follower read · Issue #2756 · pingcap/tispark · GitHub.
But this problem is easy to avoid, so fixing it is not very urgent. I will handle it when I have time, or anyone interested can also contribute.

| username: ShawnYan | Original post link

Thank you, Yuhang!

| username: ShawnYan | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.