pyspark dataframe to csv 1

pyspark dataframe to csv

df.toPandas().to_csv('new_csv_file.csv')

Here is what the above code is Doing:
1. We are reading the csv file using the spark.read.csv() method.
2. We are using the .withColumn() method to add a new column to the dataframe.
3. We are using the .withColumnRenamed() method to rename the column.
4. We are using the .toPandas() method to convert the dataframe to a pandas dataframe.
5. We are using the .to_csv() method to write the pandas dataframe to a csv file.

Similar Posts