Drop Columns with NaN Values in Pandas DataFrame

Here are 2 ways to drop columns with NaN values in Pandas DataFrame:

(1) Drop any column that contains at least one NaN:

df = df.dropna(axis='columns')

(2) Drop column/s where ALL the values are NaN:

df = df.dropna(axis='columns', how ='all')

In the next section, you’ll see how to apply each of the above approaches using a simple example.

The Example

For demonstration purposes, let’s create a DataFrame with 5 columns, where:

  • 3 columns will contain NaN values
  • 2 columns will not contain any NaN values

Here is the syntax to create the DataFrame:

import pandas as pd
import numpy as np

data = {'Column_A': [1,2,3,4,5,np.nan,6,7,np.nan],
        'Column_B': [11,22,33,44,55,66,77,88,99],
        'Column_C': ['a','b',np.nan,np.nan,'c','d','e',np.nan,'f'],
        'Column_D': ['aa','bb','cc','dd','ee','ff','gg','hh','ii'],
        'Column_E': [np.nan,np.nan,np.nan,np.nan,np.nan,np.nan,np.nan,np.nan,np.nan]
        }

df = pd.DataFrame(data,columns=['Column_A','Column_B','Column_C','Column_D','Column_E'])

print (df)

As you can see, 3 columns (‘Column_A’, ‘Column_C’ and ‘Column_E’) contain NaN values:

Drop Columns with NaN Values in Pandas DataFrame

The ultimate goal is to drop the columns with the NaN values in the above DataFrame.

(1) Drop any column that contains at least one NaN

You can use the following template to drop any column that contains at least one NaN:

df = df.dropna(axis='columns')

For our example:

import pandas as pd
import numpy as np

data = {'Column_A': [1,2,3,4,5,np.nan,6,7,np.nan],
        'Column_B': [11,22,33,44,55,66,77,88,99],
        'Column_C': ['a','b',np.nan,np.nan,'c','d','e',np.nan,'f'],
        'Column_D': ['aa','bb','cc','dd','ee','ff','gg','hh','ii'],
        'Column_E': [np.nan,np.nan,np.nan,np.nan,np.nan,np.nan,np.nan,np.nan,np.nan]
        }

df = pd.DataFrame(data,columns=['Column_A','Column_B','Column_C','Column_D','Column_E'])

df = df.dropna(axis='columns')

print (df)

Once you run the code, you’ll notice that the 3 columns, which originally contained the NaN values, were dropped. Only the other 2 columns (without the NaN values) were maintained:

How to Drop Columns with NaN Values in Pandas DataFrame

(2) Drop column/s where ALL the values are NaN

What if you’d like to drop only the column/s where ALL the values are NaN?

In that case, you can use the template below to accomplish this goal:

df = df.dropna(axis='columns', how ='all')

Note that columns which contain a mix of NaN and non-NaN values will still be maintained. Only the columns where all the values are NaN will be dropped.

In our example, the only column where all the values are NaN is ‘Column_E.’

Here is the code that you can use:

import pandas as pd
import numpy as np

data = {'Column_A': [1,2,3,4,5,np.nan,6,7,np.nan],
        'Column_B': [11,22,33,44,55,66,77,88,99],
        'Column_C': ['a','b',np.nan,np.nan,'c','d','e',np.nan,'f'],
        'Column_D': ['aa','bb','cc','dd','ee','ff','gg','hh','ii'],
        'Column_E': [np.nan,np.nan,np.nan,np.nan,np.nan,np.nan,np.nan,np.nan,np.nan]
        }

df = pd.DataFrame(data,columns=['Column_A','Column_B','Column_C','Column_D','Column_E'])

df = df.dropna(axis='columns', how ='all')

print (df)

As you may notice, ‘Column_E’ (that contained only NaN) was dropped:

Example of DataFrame with NaN values

Additional Resources

You can check the Pandas Documentation to learn more about dropna.

Optionally, you can check the following guide to learn how to drop rows with NaN values in Pandas DataFrame.