#Pandas drop duplicate rows how to

If we need to consider only a subset of the columns when dropping duplicates, we must first make a column selection before calling distinct(), as shown below. Println('Count of DataFrame After dropping duplicates is = '+distinct_df.count()) Println('Count of DataFrame before dropping duplicates is = '+df.count()) Using distinct() to drop Row Level duplicates If we observe, the record count of distinct_df is nine, which means the duplicate records are removed, and the distinct() is applied over all the columns in the Dataframe. Println('Dataframe Record count is '+df.count())īy default, this distinct() method is applied on all the columns of the dataframe when dropping the duplicates. Val df = data.toDF('emp_name', 'department', 'salary') Here Total record count of the DataFrame is 12, and we have three duplicate records, which are highlighted in the resulting image shown below. When distinct() applied over a DataFrame, it returns a new DataFrame containing the distinct rows in this DataFrame. Let us create a sample DataFrame that contains some duplicate rows in it. Note that the examples we’ll use to explore these methods have been constructed using the scala API. Additionally, we will discuss when to use one over the other. Here, we will understand how both of these functions work and what their main difference is. Even these methods do the same work they are scenarios in which one method serves the actual need whereas the other does not. These are distinct() and dropDuplicates(). The Spark DataFrame API comes with two functions that can be used to remove duplicates from a given DataFrame.

Databricks Community Edition click here.

#Pandas drop duplicate rows how to

Recipe Objective: How to eliminate Row Level Duplicates in Spark SQL?.