In this article we will discuss how to find NaN or missing values in a Dataframe.
Manytimes we create a DataFrame from an exsisting dataset and it might contain some missing values in any column or row.For every missing value Pandas add NaN at its place.
Lets create a dataframe with missing values i.e.
Contents of the dataframe are,
Name Age City Country 0 jack NaN Sydeny Australia 1 Riti NaN Delhi India 2 Vikas 31 NaN India 3 Neelu 32 Bangalore India 4 John 16 New York US 5 John 11 NaN NaN 6 NaN NaN NaN NaN
dataframe.isnull[]
Now lets count the number of NaN in this dataframe using dataframe.isnull[]
Advertisements
Pandas Dataframe provides a function isnull[], it returns a new dataframe of same size as calling dataframe, it contains only True & False only.With True at the place NaN in original dataframe and False at other places. Lets call this function on above dataframe dfObj i.e.
It will return a new DataFrame with True & False data i.e.
Name Age City Country 0 False True False False 1 False True False False 2 False False True False 3 False False False False 4 False False False False 5 False False True True 6 True True True True
This contains True at the place NaN in dfObj and False at other places. We are going to use this dataframe to calculate total NaN in original dataframe dfObj.
Count all NaN in a DataFrame [both columns & Rows]
dfObj.isnull[].sum[].sum[]Calling sum[] of the DataFrame returned by isnull[] will give the count of total NaN in dataframe i.e.
9
Now suppose we want to count the NaN in each column individually, lets do that.
Count total NaN at each column in DataFrame
dfObj.isnull[].sum[]Calling sum[] of the DataFrame returned by isnull[] will give a series containing data about count of NaN in each column i.e.
Name 1 Age 3 City 3 Country 2 dtype: int64
Count total NaN at each row in DataFrame
To count the total NaN in each row in dataframe, we need to iterate over each row in dataframe and call sum[] on it i.e.
Its output will be,
Nan in row 0 : 1 Nan in row 1 : 1 Nan in row 2 : 1 Nan in row 3 : 0 Nan in row 4 : 0 Nan in row 5 : 2 Nan in row 6 : 4
Complete example is as follows,
import pandas as pd import numpy as np def main[]: # List of Tuples students = [ ['jack', np.NaN, 'Sydeny' , 'Australia'] , ['Riti', np.NaN, 'Delhi' , 'India' ] , ['Vikas', 31, np.NaN , 'India' ] , ['Neelu', 32, 'Bangalore' , 'India' ] , ['John', 16, 'New York' , 'US'] , ['John' , 11, np.NaN, np.NaN ] , [np.NaN , np.NaN, np.NaN, np.NaN ] ] #Create a DataFrame object dfObj = pd.DataFrame[students, columns = ['Name' , 'Age', 'City' , 'Country']] print["Original Dataframe" , dfObj, sep='\n'] print["Check NaN in Dataframe" , dfObj.isnull[], sep='\n'] print["***Count all NaN in a DataFrame [both columns & Rows]***"] print["Total NaN in Dataframe" , dfObj.isnull[].sum[].sum[], sep='\n'] print["***Count NaN in each column of a DataFrame***"] print["Nan in each columns" , dfObj.isnull[].sum[], sep='\n'] print["***Count NaN in each row of a DataFrame***"] for i in range[len[dfObj.index]] : print["Nan in row ", i , " : " , dfObj.iloc[i].isnull[].sum[]] if __name__ == '__main__': main[]
Output:
Original Dataframe Name Age City Country 0 jack NaN Sydeny Australia 1 Riti NaN Delhi India 2 Vikas 31 NaN India 3 Neelu 32 Bangalore India 4 John 16 New York US 5 John 11 NaN NaN 6 NaN NaN NaN NaN Check NaN in Dataframe Name Age City Country 0 False True False False 1 False True False False 2 False False True False 3 False False False False 4 False False False False 5 False False True True 6 True True True True ***Count all NaN in a DataFrame [both columns & Rows]*** Total NaN in Dataframe 9 ***Count NaN in each column of a DataFrame*** Nan in each columns Name 1 Age 3 City 3 Country 2 dtype: int64 ***Count NaN in each row of a DataFrame*** Nan in row 0 : 1 Nan in row 1 : 1 Nan in row 2 : 1 Nan in row 3 : 0 Nan in row 4 : 0 Nan in row 5 : 2 Nan in row 6 : 4
Advertisements