Exploratory Data Analysis with Python

In this article, I am going to analyze a data set from Kaggle. Kaggle is a platform for predictive modeling and analytics competitions in which companies and researchers post data and statisticians and data miners compete to produce the best models for predicting and describing the data.

This is my first ever EDA. I have tried to analyze the data as much as I can but there might be still some missing/wrong points.

The competition that I am going to analyze is about predicting house pricing. You can find the competition by clicking here

So, what is Exploratory Data Analysis?

Exploratory Data Analysis (EDA) is the first step in your data analysis process. Here, you make sense of the data you have and then figure out what questions you want to ask and how to frame them, as well as how best to manipulate your available data sources to get the answers you need.

In this example and for all the future examples, I am going to use Python programming language.

Challenge Description

With 79 explanatory variables describing (almost) every aspect of residential homes in Ames, Iowa, this competition challenges you to predict the final price of each home.

Let’s get to work!

When we open the link above, we can see that there are some data files. For this analysis, we only need data_description.txt and train.csv files. the data_description.txt file contains information of each attribute given in train.csv. We will load the train.csv and whenever we need desription of attributes, we will look at data_description.txt.

First see which libraries we are going to use in this project:

Let’s load the data set from train.csv:

Let’s have a quick look at our data:

Example output that includes first 7 columns:

In the output we can see that there are total of 1460 entries and 81 columns. The last column is our class variable, SalePrice, and the first 80 columns are the features that will shape our prediction. Full description of the each column is given in the data_description.txt by Kaggle.
In the output of data_train.info(), We see that next to each column, there are three attributes. In the first one it shows how many times that column is not null for the entire set. And when we check all the columns, we can see that some columns contain less entries. This shows us that there are missing values in our data set. We can also see that there are both numerical and categorical attributes. First of all, I would like to examine the missing data. Missing values in data set can affect prediction or classification of a model negatively. We have seen that there are some missing attributes. Let’s print them out:

Missing Data Handling

Let’s look at the definition of those attributes from data_description.txt and handle missing values:

Here missing value for PoolQC means there is no pool in the house, therefore it is actually meaningful. Instead of deleting it, we should put a value into those columns that will tell us it is NA. But what if the house has pool but its quality is actually missing? We can compare the PoolQC with PoolArea values to see that if we actually have missing pool quality.

It seems like whenever PoolQC is null, PoolArea is zero. Therefore we can be sure that those missing PoolQC values actually mean that the house doesn’t have a pool. Now that we are sure, we can put a meaningful data into those columns such as None

If it is NA, I will assume the house doesn’t have MiscFeature.

If it is NA, I will assume the house has no Alley access.

If it is NA, I will assume the house has no Fence.

Just like PoolQC; although NA means no fireplace, there might be some missing data. Let’s check:

For all the null FireplaceQu’s, Fireplaces is zero. We are safe to assume that those houses don’t have a fireplace

There are 259 missing data for the LotFrontage, which is 0.177397 percent. The ratio is not high enough to delete the column because we might lose some valuable information.
Firstly, I would like to check if there is a high correlation between LotFrontage and other attributes.

It seems like there are no high correlation between any attributes. And also there is no strong bond with the class variable. We can actually discard this column but instead, I want to fill it with the median. The LotFrontage may have some similarity with other houses of the same neighborhood. Since Neighborhood is categorical attribute, I will use box plot to examine it.

There is no strong correlation between Neighborhood and LotFrontage. But it seems like it is still better to take median(I am not taking mean because of outliers) of the same neighborhood instead of all set.

Let’s handle Garage features together because they are all connected. When we check the null values for the above list, we can see that they are all null at the same rows.
But there are 2 more Garage attributes which are not null. GarageArea and GarageCars.

For the null garage attributes, GarageArea and GarageCars are zero. We can fill the numerical null values with zero and categorical null values with None.

Just like garage attributes, these attributes are null at the same row(Except for the two rows where in one of them only BsmtExposure is null and in the other only BsmtFinType2 is null). When we check the other 3 Basement attributes(BsmtUnfSF, BsmtFinSF1 and BsmtFinSF2) which are not null, we can see that they are zero just like above example. We will do the same technique and fill the numerical attributes with zero and categorical attributes with None, except for those two lines where only one attribute is null. For those lines we will fill the missing value with the most frequent values.
For the missing BsmtExposure:

I will fill the missing BsmtExposure value on line 948 with No which is the most occured value for BsmtExposure

For the missing BsmtFinType2:

I will fill the missing BsmtFinType2 value on line 332 with Unf which is the most occured value for BsmtFinType2

For the rest of them we can fill the numerical null values with zero and categorical null values with None.

There are only 8 missing values for both attributes and they are from the same house. We can fill MasVnrArea’s with 0 and MasVnrType’s with None, because they are the most occured values.

There is only 1 missing data for electrical system. We can fill it with the most frequent value which is SBrkr.

I have also checked if there is any duplicate row by checking Id column, and I have seen that there is no duplicate row.

Now we don’t have any missing data in our train set. We can continue exploring the data set.

Correlations and Graphs

Let’s start with checking how the SalePrice is distributed:

Here we can see that the graph is right skewed, and there are some outliers. But I won’t remove them before understanding the whole data set.
We can check the distribution of each attribute:

When I check all categorical attributes, I can see that for some attributes a specific category is very dominant. For example;
For the attributes such as Street, Utilities, Condition2, RoofMat1 and Heating the value is same for around %99 of the rows. We can actually discard those attributes as they won’t be very effective. For the visualization, we can look at the box plot of one of them:

Here we can clearly see that AllPub category dominated the Utilities feature, therefore there is no need to take the Utilities into account.

We saw the details of each attributes but it is not enough. We should check correlations between attributes.

Here we can see correlation between numerical attributes. Let’s print out highly correlated attributes.

As we can see here, there are some highly correlated attributes. The problem here is that so called Multicollinearity increases the standard errors of the coefficients. Let’s also print attributes which are highly correlated with class attribute and compare it with above table.

Now I will analyze attributes that have more than 0.5 correlation with class attribute and those that have strong correlation with other attributes.
Attributes that has high correlation with SalePrice:
OverallQual, GrLivArea, GarageCars, GarageArea, TotalBsmtSF, 1stFlrSF, FullBath, TotRmsAbvGrd, YearBuilt, YearRemodAdd

Let’s examine them;

1) OverallQual has correlation with lots of attributes but I think those correlations are not enough to take action.

2) GrLivArea has strong correlation with TotRmsAbvGrd. It is actually pretty normal that the total room count depends on the total living area. But I won’t remove any of them as I think they should be considered separately. Because for the same total living area, some people may prefer fewer rooms with higher space while others prefer more but smaller rooms.

3) GarageArea and GarageCars are both highly correlated with SalePrice, but also they are highly correlated with each other. When we check their meaning;

It is obvious that garage car capacity depends on the are of the garage. Instead of taking both of them, we may remove one of those attributes. In this case, GarageCars is little bit more correlated with class attribute. So, we can consider taking it and removing GarageArea.

4) TotalBsmtSF and 1stFlrSF are highly correlated with SalePrice as well as each other. considering most of the time basement would be same size of the first floor. We can just take one of them and remove the other. In this case TotalBsmtSF is little bit more correlated with class attribute. Therefore I will ignore 1stFlrSF.

5) YearBuilt and YearRemodAdd are also correlated with each other. In the description of YearRemodAdd, it says; it is same as YearBuilt if there was no re-modelling. But when I check the amount of time they are same, they appeared to be same for 764 times while there are 1460 samples. Which means a lot of house have remodeling or additions. Therefore I won’t delete those attributes.

After this quick comparison, I will create my important numerical feature list that is consist of;
OverallQual, GrLivArea, GarageCars, TotalBsmtSF, FullBath, TotRmsAbvGrd, YearBuilt, YearRemodAdd

We can check their distribution with class variable:

Now let’s check distribution of categorical attributes. To check their distribution with class variable we can use box plots.

Actually the code to print box plots is very simple;

The reason behind complexity(redundance) of above code is because I wanted to plot box graph of each attribute at the same time and specifically 3 x 3 attributes on per frame. Plotting one graph per page would be a trouble as it pops up a new page for each frame. And plotting every attribute on the same page would make it unreadable, therefore I choose to go this way.
I have also wrote another code which does exactly the same thing.
Second way (Slower):

The code looks cleaner than first one but the former is slightly faster. Therefore, I am using it. The calculation time for the first one was around 3.2 seconds while for the second one it was around 3.3 seconds. The difference doesn’t seem much but it can go up if we get more attributes. There might be much more efficient way to do the same job, but I am very new to these seaborn and matplotlib libraries and I have tried to come up with this solution by myself.

Here we can see attributes such as ExterQual, KitchenQual, HeatingQC, BsmtQual and FireplaceQu are highly correlated with SalePrice. When their quality is Excellent, prices tend to go up.
Although analyzing these graphs can give some insight about the effect of categorical attributes on SalePrice, creating custom features by turning those categories into numerical values would make it much more easier for us to see the correlation. To encode categories into ordered numbers, we should have information about their order. For example;

We can easily create new feature from HeatingQC whose values would be 1, 2, 3, 4 and 5 (1 = Po, 5 = Ex)
Actually it is possible to encode all categorical attributes into numerical values even if their values can not be ordered. We can achieve this by simply taking mean SalePrice for each category of given attribute and then ordering them. It may be necessary for the feature engineering but I think for this EDA (Exploratory Data Analysis) checking box plots is enough.

Thank you for reading so far! I hope it was somehow useful.