scikit-learn : Data Preprocessing I - Missing / categorical data

Scikit-learn_logo.png

(追記) (追記ここまで)

bogotobogo.com site search:

Handling missing data

In real-world samples, it is not uncommon that there are missing one or more values such as the blank spaces in our data table.

Quite a few computational tools, however, are unable to handle such missing values and might produce unpredictable results. So, before we proceed with further analyses, it is critical that we take care of those missing values.

(追記) (追記ここまで)

Pandas DataFrame

To get a better feel for the problem, let's create a simple example using CSV file:

to get a better grasp of the problem:

CSV-Pandas-DataFrame.png

The StringIO() function allows us to read the string assigned to csv_data into a pandas DataFrame via the read_csv() function as if it was a regular CSV file on our hard drive.

Note that the two missing cells were replaced by NaN.

We can use isnull() method to check whether a cell contains a numeric value ( False ) or if data is missing ( True ):

df_isnull.png

For a larger DataFrame, we may want to use the sum() method which returns the number of missing values per column:

is_null_sum.png

Note that we can always access the underlying NumPy array of the DataFrame via the values attribute before we feed it into a scikit-learn estimator:

df_values.png

So, while scikit-learn was developed for working with NumPy arrays, it can sometimes be more convenient to preprocess data using pandas' DataFrame.

Eliminating samples/features with missing cells via pandas.DataFrame.dropna()

We can remove the corresponding features (columns) or samples (rows) from the dataset.

The rows with missing values can be dropped via the pandas.DataFrame.dropna() method:

df_dropna.png

We can drop columns that have at least one NaN in any row by setting the axis argument to 1:

df-dropna-axis-1.png

where axis : {0 or 'index', 1 or 'columns'}.

The dropna() method has several additional parameters:

df-dropna-additional-parameters.png

The removal of missing data appears to be a convenient approach, however, it also comes with certain disadvantages:

There are chances that we may end up removing too many, which will make our analysis not reliable.
By eliminating too many feature columns, we may run the risk of losing valuable information for our classifier.

In the next section, we will look into interpolation techniques which one of the most commonly used alternatives for dealing with missing data.

Estimating -missing values via interpolation

Mean imputation is a method replacing the missing values with the mean value of the entire feature column. While this method maintains the sample size and is easy to use, the variability in the data is reduced, so the standard deviations and the variance estimates tend to be underestimated.

We'll use the sklearn.preprocessing.Imputer class:

ImputedData.png

Here, we replaced each NaN value by the corresponding mean from each feature column.

We can use the row means if we change the setting axis=0 to axis=1:

ImputedDataAxis1RowMeans.png

As for the strategy parameter, there are other options such as median or most_frequent.

Scikit-learn estimator API

The Imputer class we used in the previous section belongs to the so-called transformer classes in scikit-learn that are used for data transformation.

There are two methods of estimators:

fit:
This method is used to learn the parameters from the training data.
transform:
transform() method uses those parameters to transform the data.

The diagram below shows how a transformer fitted on the training data is used to transform a training dataset as well as a new test dataset:

fit-transform-scikit-learn-estimator.png

In supervised learning tasks, we additionally provide the class labels for fitting the model, which can then be used to make predictions about new data samples via the predict() method:

scikit-learn-predict-method.png

picture source : Python machine learning by Sebastian Raschka

Dealing with categorical data

Not all data has numerical values. Here are examples of categorical data:

The blood type of a person: A, B, AB or O.
The state that a resident of the United States lives in.
T-shirt size. XL> L> M
T-shirt color.

Even among categorical data, we may want to distinguish further between nominal and ordinal which can be sorted or ordered features. So, T-shirt size can be an ordinal feature, because we can define an order XL> L> M.

Let's create a new categorical data frame:

create-categorical-data.png

As we can see from the output, the DataFrame contains a nominal feature (color), an ordinal feature (size) as well as a numerical feature (price) column. In the last column, the class labels are created.

Ordinal feature mapping

In order for out learning algorithm to interpret the ordinal features correctly, we should convert the categorical string values into integers.

However, since there is no convenient function that can automatically derive the correct order of the labels of our size feature, we have to define the mapping manually.

Let's assume that we know the difference between features such as XL = L + 1 = M + 2.

size_mapping.png

If we want to transform the integer values back to the original string representation, we can simply define a reverse-mapping dictionary "inv_size_mapping" that can then be used via the pandas' map method on the transformed feature column similar to the "size_mapping" dictionary that we used previously:

inv_size_mapping.png

Class labels encoding

Usually, class labels are required to be encoded as integer values.

While most estimators for classification in scikit-learn convert class labels to integers internally, as a good practice, we may want to provide class labels as integer arrays to avoid any issues.

To encode the class labels, we can use an approach similar to the mapping of ordinal features discussed in the previous section.

Since class labels are not ordinal, it doesn't matter which integer number we assign to a particular string-label.

So, we can simply enumerate the class labels starting at 0:

Class_mapping_1.png

Now we want mapping dictionary to transform the class labels into integers:

df-class-mapping-1.png

As we did for "size" in the previous section, we can reverse the key-value pairs in the mapping dictionary as follows to map the converted class labels back to the original string representation:

inverse_class_labels.png

scikit-learn's LabelEncoder class

Though we encoded the class labels manually, luckily, there is a convenient LabelEncoder class directly implemented in scikit-learn to achieve the same:

LabelEncoder-scikit.png

Note that the fit_transform() method is just a shortcut for calling fit and transform separately, and we can use the inverse_transform() method to transform the integer class labels back into their original string representation:

inverse-label-encoder.png

Nominal feature encoding

So far, we used a simple dictionary-mapping approach to convert the ordinal size feature into integers.

Because scikit-learn's estimators treat class labels without any order, we used the convenient LabelEncoder class to encode the string labels into integers.

We can use a similar approach to transform the nominal color column of our dataset as well:

Color-Label-Encoder.png

We may want to create a new dummy feature for each unique value in the nominal feature column. In other words, we would convert the color feature into three new features: blue, green, and red. Binary values can then be used to indicate the particular color of a sample; for example, a blue sample can be encoded as blue=1, green=0, red=0. This technique is called one-hot encoding.

In order to perform this transformation, we can use the scikit-learn.preprocessingOneHotEncoder:

One-hot-Encoding.png

Note that when we initialized the OneHotEncoder, we defined the column position of the variable that we want to transform via the categorical_features parameter which is the first column in the feature matrix X.

The OneHotEncoder, by default, returns a sparse matrix when we use the transform() method, and we converted the sparse matrix representation into a regular (dense) NumPy array for the purposes of visualization via the toarray() method. Sparse matrices are simply a more efficient way of storing large datasets, and one that is supported by many scikit-learn functions. It is especially useful if it contains a lot of zeros.

If we want to omit the toarray() step, we may initialize the encoder as OneHotEncoder(...,sparse=False) to return a regular NumPy array:

Sparse-False-One-Hot-Encoder.png

Another way which is more convenient is to create those dummy features via one-hot encoding is to use the pandas.get_dummies() method. Applied on a DataFrame, the get_dummies() method will only convert string columns and leave all other columns unchanged:

pandas-get_dummies.png

Github repository

Source is available from bogotobogo-Machine-Learning .

Next:

Data Preprocessing II - Partitioning a dataset / Feature scaling / Feature Selection / Regularization