Record: tispark + spark 3.2.2 (scala 2.13) error

Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.

Original topic: 记录一下,tispark + spark 3.2.2 (scala 2.13) 报错

| username: ShawnYan

Right, the error message is as follows:

shawnyan@centos7:~/spark-3.3.2-bin-hadoop3-scala2.13/conf$ pyspark
Python 3.6.8 (default, Nov 16 2020, 16:55:22)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
Type "help", "copyright", "credits" or "license" for more information.
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/07/18 17:18:15 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
23/07/18 17:18:16 WARN SparkSession: Cannot use org.apache.spark.sql.TiExtensions to configure session extensions.
java.lang.NoClassDefFoundError: Could not initialize class com.pingcap.tispark.TiSparkInfo$
        at org.apache.spark.sql.TiExtensions.apply(TiExtensions.scala:39)
        at org.apache.spark.sql.TiExtensions.apply(TiExtensions.scala:35)
        at org.apache.spark.sql.SparkSession$.$anonfun$applyExtensions$1(SparkSession.scala:1224)
        at org.apache.spark.sql.SparkSession$.$anonfun$applyExtensions$1$adapted(SparkSession.scala:1219)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:926)
        at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$applyExtensions(SparkSession.scala:1219)
        at org.apache.spark.sql.SparkSession.<init>(SparkSession.scala:106)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:238)
        at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
        at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
        at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
        at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
        at java.lang.Thread.run(Thread.java:748)
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 3.3.2
      /_/

Using Python version 3.6.8 (default, Nov 16 2020 16:55:22)
Spark context Web UI available at http://xxx:4040
Spark context available as 'sc' (master = spark://xxx:7077, app id = app-20230718171816-0001).
SparkSession available as 'spark'.
>>> print("pyspark "+str(sc.version))
pyspark 3.3.2
>>> import pyspark
>>> print("pyspark",pyspark.__version__)
pyspark 3.3.2
>>> from pyspark.sql import SparkSession
>>> spark = SparkSession \
...         .Builder() \
...         .appName('sql') \
...         .master('local') \
...         .getOrCreate()
23/07/18 17:19:31 WARN SparkSession: Using an existing Spark session; only runtime SQL configurations will take effect.
>>> url = "jdbc:mysql://xxx:4000/test?user=root&password=&useSSL=false&rewriteBatchedStatements=true"
>>> df=spark.read.format("jdbc").options(url=url,
...   driver="com.mysql.jdbc.Driver",
...   dbtable="t1"
...   ).load()
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "/home/shawnyan/spark-3.3.2-bin-hadoop3-scala2.13/python/pyspark/sql/readwriter.py", line 184, in load
    return self._df(self._jreader.load())
  File "/home/shawnyan/spark-3.3.2-bin-hadoop3-scala2.13/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1322, in __call__
  File "/home/shawnyan/spark-3.3.2-bin-hadoop3-scala2.13/python/pyspark/sql/utils.py", line 190, in deco
    return f(*a, **kw)
  File "/home/shawnyan/spark-3.3.2-bin-hadoop3-scala2.13/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o58.load.
: java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider com.pingcap.tispark.v2.TiDBTableProvider could not be instantiated
        at java.util.ServiceLoader.fail(ServiceLoader.java:232)
        at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
        at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
        at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
        at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
        at scala.collection.convert.JavaCollectionWrappers$JIteratorWrapper.next(JavaCollectionWrappers.scala:38)
        at scala.collection.StrictOptimizedIterableOps.filterImpl(StrictOptimizedIterableOps.scala:226)
        at scala.collection.StrictOptimizedIterableOps.filterImpl$(StrictOptimizedIterableOps.scala:222)
        at scala.collection.convert.JavaCollectionWrappers$JIterableWrapper.filterImpl(JavaCollectionWrappers.scala:58)
        at scala.collection.StrictOptimizedIterableOps.filter(StrictOptimizedIterableOps.scala:218)
        at scala.collection.StrictOptimizedIterableOps.filter$(StrictOptimizedIterableOps.scala:218)
        at scala.collection.convert.JavaCollectionWrappers$JIterableWrapper.filter(JavaCollectionWrappers.scala:58)
        at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:657)
        at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSourceV2(DataSource.scala:725)
        at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:207)
        at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:171)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:282)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
        at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: scala/Serializable
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        at java.lang.Class.getDeclaredConstructors0(Native Method)
        at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
        at java.lang.Class.getConstructor0(Class.java:3075)
        at java.lang.Class.newInstance(Class.java:412)
        at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
        ... 25 more
Caused by: java.lang.ClassNotFoundException: scala.Serializable
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        ... 42 more

>>>

| username: 有猫万事足 | Original post link

I have seen many people encounter this issue when upgrading from Scala 2.12 to 2.13.
It should be a package dependency problem.

| username: ShawnYan | Original post link

Since TiSpark does not yet support Scala 2.13, an error will occur. It is still being tested, and I am recording the error information in the meantime.

| username: redgame | Original post link

Make a point…

| username: system | Original post link

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.