How to create a Spark Dataframe from Parquet file?

https://bigdataelearning.com/course/apache-spark-2-with-scala/
https://bigdataelearning.com/courses
https://bigdataelearning.com
Creating a spark dataframe from a Parquet file, is done using the spark.read.load API.

Let’s create a dataframe from a parquet file. Using spark.read.load API to load the data. Specifying the path to the Parquet file within double quotes and enclosing with round braces.

This read.load API reads the contents of the users.parquet file and creates users-Df dataframe.

To write the dataframe contents to an external Parquet file, the dataframe.write.save method is used.

Here I am applying write.save method on the users-Df dataframe. Within the braces, specifying the target path where we would like the dataframe contents to be stored.

This is how we can read and write parquet data using Spark dataframe.

Скачать видео