Note:
This topic has been translated from a Chinese forum by GPT and might contain errors.Original topic: tispark的兼容问题
|
username: TiDBer_nVNsYWwJ
[TiDB Usage Environment] Test
[TiDB Version] 6.5.2
[Reproduction Path] Created a local TiDB cluster (6.5.2) and a Spark cluster 3.1.2. When using spark-shell, I cannot see the TiDB cluster’s database. The command to start spark-shell is /bin/spark-shell --jars …/jars/mysql-connector-java-8.0.19.jar …/jars/tispark-assembly-3.1_2.12-3.2.1.jar --master local[2]
Configuration file:
spark.eventLog.enabled true
spark.eventLog.dir hdfs://node02:8020/spark_log
spark.eventLog.compress true
spark.master 192.168.0.106:7077
spark.sql.extensions org.apache.spark.sql.TiExtensions
spark.tispark.pd.addresses 192.168.0.106:2379
spark.sql.catalog.tidb_catalog org.apache.spark.sql.catalyst.catalog.TiCatalog
spark.sql.catalog.tidb_catalog.pd.addresses 192.168.0.106:2379
spark.tispark.grpc.framesize 1564087970
spark.tispark.grpc.timeout_in_sec 100
spark.tispark.meta.reload_period_in_sec 60
spark.tispark.plan.allowaggpushdown true
spark.tispark.index.scan_batch_size 500000
spark.tispark.index.scan_concurrency 2
spark.tispark.table.scan_concurrency 256
spark.tispark.tidb.addr 192.168.0.106
spark.tispark.tidb.port 4000
spark.sql.tidb.user root
spark.sql.tidb.password 123456
spark.tispark.isolation_read_engines tikv
spark.tispark.tikv.addr 192.168.0.106
spark.tispark.tikv.port 20160
spark.sql.auth.enable true
spark.sql.tidb.addr 192.168.0.106
spark.sql.tidb.port 4000
[Encountered Problem: Problem Phenomenon and Impact] A WARN TiConfiguration: Unable to find tikv.properties is reported, and the databases in the TiDB cluster cannot be seen. The spark-shell run log is as follows:
scala> spark.sql(“show databases”).show()
Loading class com.mysql.jdbc.Driver'. This is deprecated. The new driver class is com.mysql.cj.jdbc.Driver’. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
23/05/17 11:35:53 WARN TiConfiguration: Unable to find tikv.properties
23/05/17 11:35:57 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.3.0
23/05/17 11:35:57 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore parallels@192.168.0.106
±--------+
|namespace|
±--------+
| default|
±--------+
[Resource Configuration] Enter TiDB Dashboard - Cluster Info - Hosts and screenshot this page
[Attachment: Screenshot/Log/Monitoring]
Please advise, experts
|
username: 
|
username: 
|
username: 
|
username: