Which version of the jar package should be used for TiSpark to be compatible with the Spark 3.2.2 cluster?

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 适配Spark3.2.2集群的TiSpark应该用哪个版本的jar包

| username: OnTheRoad

【TiDB Usage Environment】Test Environment
【TiDB Version】5.4.2
【Encountered Issues】

  1. In the TiDB 5.4 documentation, the description of TiSpark mentions that TiSpark 3.0.x is compatible with Spark 3.0.x/3.1.x/3.2.x, and the Scala version is 2.12.
  2. In the TiSpark repository, the TiSpark package compatible with Spark 3.2.x is tispark-assembly-3.2_2.12-3.0.1.jar, which means Spark is 2.2 and Scala is 2.12.
  3. However, the Spark official website has the following note:
    Note that Spark 3 is pre-built with Scala 2.12 in general and Spark 3.2+ provides additional pre-built distribution with Scala 2.13.

【Issue Phenomenon and Impact】
If I choose Spark cluster version 3.2.2, which TiSpark jar package should I choose?

2
| username: ShawnYan | Original post link

Hi, you can refer to the [TiSpark Version Description] section,
\https://tidb.net/blog/5e697bac#TiSpark%20%E5%AE%89%E8%A3%85/TiSpark%20%E7%89%88%E6%9C%AC%E8%AF%B4%E6%98%8E

Choose this jar package: tispark-assembly-3.2_2.12-3.0.1.jar

3
| username: 数据小黑 | Original post link

Currently, only TiSpark 3.0.X supports Spark 3.2.X. As mentioned above, you need to choose the jar: tispark-assembly-3.2_2.12-3.0.1.jar.

4
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.