我正在寻找一种在写入操作之前更新所有表元数据缓存条目的方法。我已经通过 找到了方法spark.catalog.refreshTable(table)
,但是我不确定它是否会更新 spark.sql() 中用于制作表的所有表元数据存储。
代码示例:
val dataFrame1 :DataFramew = ...
dataFrame1.createOrReplaceTempView("table1")
val dataFrame2 :DataFramew = ...
dataFrame2.createOrReplaceTempView("table2")
val dataFrame3 = spark.sql(select * from table1 inner join table2 .....)
dataFrame3.createOrReplaceTempView("table3")
spark.catalog.refreshTable(table3)
\\ does the above line of code will update the cached table metadata for table1 and table 2 as well before the write operation? Or do we need to refresh the table separately ?
dataFrame3.write.parquet("/temp/.....)