How does TiSpark support both Spark 3.1 and Spark 3.2 in a single package?

Translated
1

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: TiSpark如何在一个包里同时支持spark3.1和spark3.2？

| username: TiDBer_PYmtzTZg

Hi, our data synchronization task uses TiSpark + Deltastreamer to import into Hudi. The company is upgrading from Spark 3.1 to Spark 3.2, but this upgrade is done in a phased manner by queue (controlled by Livy), which means TiSpark needs to support two versions of Spark simultaneously.

We tried to package using TiSpark v3.1.1, but:

  • Selecting both profiles simultaneously,
  • Setting activeByDefault to true in the pom.xml,
  • Or adding both versions in the compile option -P,
    None of these methods worked.

Screenshot 2023-09-08 at 10.34.21 AM

Error when running TiSpark profile for Spark 3.2 on Spark 3.1:

image
image1154×157 17.1 KB

Error when running TiSpark profile for Spark 3.1 on Spark 3.2:

image
image1541×162 14.9 KB

So, I want to confirm if the compatibility mentioned in the official documentation refers to separate compatibility (separately in different jars) rather than simultaneous compatibility?

Screenshot 2023-09-08 at 10.36.47 AM
Screenshot 2023-09-08 at 10.36.47 AM779×400 33.2 KB

2
| username: tidb菜鸟一只 | Original post link

The official documentation indicates that both 3.1 and 3.2 are supported, but that doesn’t mean you can use 3.1 and 3.2 simultaneously…

3
| username: TiDBer_PYmtzTZg | Original post link

Well, I realize that now too. I just wanted to confirm and struggle a bit more.

4
| username: 天蓝色的小九 | Original post link

This can’t be placed, right?

5
| username: ShawnYan | Original post link

Compatible separately.

For more information, please refer to:

6
| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.