新的火花并尝试了stackoverflow的其他解决方案,但没有运气
我已经安装了 spark 3.1.2 并且做了很少的配置设置(用户 spark/conf/spark-defaults.conf)将 aws rds mysql 指向一个元存储(远程)
spark.jars.packages com.amazonaws:aws-java-sdk:1.12.63,org.apache.hadoop:hadoop-aws:3.2.0
spark.jars /home/newdependencies/jtds-1.3.1.jar, /home/newdependencies/mysql-connector-java-6.0.6.jar, /home/newdependencies/postgresql-42.2.20.jar
spark.hadoop.javax.jdo.option.ConnectionURL jdbc:mysql://testhivemetastore.asdfasfar.us-west-2.rds.amazonaws.com:3306/metastore
spark.hadoop.javax.jdo.option.ConnectionUserName username
spark.hadoop.javax.jdo.option.ConnectionPassword password
spark.hadoop.javax.jdo.option.ConnectionDriverName com.mysql.jdbc.Driver
错误消息 - 尝试运行显示数据库时,
import os.path, sys
sys.path.append(os.path.join(os.path.dirname(os.path.realpath('__file__')), os.pardir))
import findspark
findspark.init()
import pyspark
sp = pyspark.sql.SparkSession.builder.enableHiveSupport().appName(f"Test spark configurations").getOrCreate()
sqlStr = 'show databases'
sp.sql(sqlStr).show()
Error: org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
仅供参考 - 我也没有安装 Hadoop 和 Hive(不知道它是否强制)