1

我正在寻找一种在写入操作之前更新所有表元数据缓存条目的方法。我已经通过 找到了方法spark.catalog.refreshTable(table),但是我不确定它是否会更新 spark.sql() 中用于制作表的所有表元数据存储。

代码示例:

 val dataFrame1 :DataFramew = ...
 dataFrame1.createOrReplaceTempView("table1")

 val dataFrame2 :DataFramew = ...
 dataFrame2.createOrReplaceTempView("table2")

 val dataFrame3 =  spark.sql(select * from table1 inner join table2 .....)
 dataFrame3.createOrReplaceTempView("table3")

 spark.catalog.refreshTable(table3) 
\\ does the above line of code will update the cached table metadata for table1 and table 2 as well before the write operation? Or do we need to refresh the table separately ?

 dataFrame3.write.parquet("/temp/.....) 
4

0 回答 0