Data Science

LDA: Linear Discriminant Analysis - How to Improve Your Models with Supervised Dimensionality...

An in-depth comparison of PCA and LDA with Python examples

Saul Dobilas
August 8, 202112 min read

LDA: Linear Discriminant Analysis - How to Improve Your Models with Supervised Dimensionality Reduction

Linear Discriminant Analysis (LDA). Image by author.
Linear Discriminant Analysis (LDA). Image by author.

Intro

Linear Discriminant Analysis (LDA) is a commonly used dimensionality reduction technique. However, despite the similarities to Principal Component Analysis (PCA), it differs in one crucial aspect.

Instead of finding new axes (dimensions) that maximize the variation in the data, it focuses on maximizing the separability among the known categories (classes) in the target variable.

In this article, I give an intuitive explanation of how LDA works while highlighting the differences to PCA. At the same time, I provide a Python example of performing Linear Discriminant Analysis on real-life data.

Contents

  • The category of Machine Learning techniques LDA belongs to

  • Intuitive explanation of how LDA works

  • Python example of performing LDA on real-life data

  • Conclusions

What category of Machine Learning techniques does Linear Discriminant Analysis (LDA) belong to?

Unlike Principal Component Analysis (PCA), LDA requires you to provide features and class labels for your target. Hence, despite being a dimensionality reduction technique similar to PCA, it sits within the supervised branch of Machine Learning.

The below graph is interactive, so please click on different categories to enlarge and reveal more👇 .

If you enjoy Data Science and Machine Learning, please subscribe to get an email whenever I publish a new story.

How does Linear Discriminant Analysis (LDA) work?

The easiest way to grasp the concepts of LDA is by working through an example. Hence, instead of focusing on the maths behind the algorithm, I have created a visual explanation for us to go through.

Data

Assume we have collected a bunch of data on apartment prices in the city. We categorized them into "expensive" and "cheap" based on some threshold, say $1M.

text
I.e., <$1M = cheap, and ≥$1M = expensive. 

We display the data on a scatterplot where the x and y axes represent the latitude and longitude, giving us the apartment's location. Meanwhile, color represents the category/class (cheap vs. expensive). Note that we have already standardized* the data in this fictitious example, so it is centered around the origin.

_*S_tandartization is a data transformation technique that rescales the data so each attribute has a mean of 0 and a standard deviation of 1. This transformation can be described with the following formula:

Standardization. Image by author.
Standardization. Image by author.

Let's take a look at the scatterplot of our example data:

Scatterplot of apartments data. Image by author.
Scatterplot of apartments data. Image by author.

Now let's compare the PCA and LDA process to understand how these algorithms work and how they differ.

Applying Principal Component Analysis (PCA)

Since PCA is an unsupervised Machine learning technique, we do not need to provide a target variable with category labels. This means that PCA does not care about whether the apartment is part of a "cheap" or "expensive" category.

Nevertheless, I have kept the color labels in the below graph to highlight the main difference from LDA.

Finding Principal Component 1 (PC1). Image by author.
Finding Principal Component 1 (PC1). Image by author.

The goal of PCA is to capture the maximum amount of variance, which is achieved by the algorithm finding a line that minimizes the distances from data points to that line. Interestingly, this is equivalent to maximizing the spread of data point projections on that same line, which is why we can capture the maximum amount of variance.

As you can see from the graph above, we have done a good job finding a new axis that preserves most of the variance, enabling us to go down from two dimensions to one. This is what it looks like after we map the data to our new single dimension (PC1):

Apartments data mapped to a single dimension (PC1). Image by author.
Apartments data mapped to a single dimension (PC1). Image by author.

You can clearly see that, in this scenario, we were able to preserve most of the variance but still lost some information that would have been useful when trying to separate the two categories (classes).

If you would like to get a more in-depth understanding of PCA, you can refer to my previous article:

PCA: Principal Component Analysis - How to Get Superior Results with Fewer Dimensions?

Applying Linear Discriminant Analysis (LDA)

Now let's go back to our original example data and apply LDA instead of PCA. Quick reminder, the goal of LDA is to maximize the separability of the known categories in our target variable ("cheap", "expensive") while at the same time reducing dimensions.

Here is what the new axis looks like when we perform LDA:

Finding Linear Discriminant 1 (LD1). Image by author.
Finding Linear Discriminant 1 (LD1). Image by author.

As you can see, the choice of the new axis, in this scenario, is very different from that of PCA. In this case, we end up losing a significant amount of variance by mapping data onto LD1. However, we achieve better separation of the two categories:

Apartments data mapped to a single dimension (LD1). Image by author.
Apartments data mapped to a single dimension (LD1). Image by author.

While the separation is imperfect, it is visibly better, with only a few overlapping observations from the two categories.

How does LDA find the right axis?

Two key criteria are used when finding the "best" line for our new axis, which are considered simultaneously.

  1. Maximizing the distance (d²). 2 categories - when you have two classes in your target variable, the distance refers to the difference between the mean (μ) of class 1 and the mean of class 2. More than 2 categories - when you have three or more classes in your target variable, the algorithm first finds a central point to all of the data and then measures the distance from each category mean (μ) to that central point.

Let's illustrate the two criteria on a graph:

LDA steps. Image by author.
LDA steps. Image by author.

In practice, the calculations are typically performed with the help of either Singular Value Decomposition (SVD) or Eigenvalue decomposition. Both of these options are available in sklearn's LDA implementation, which we will use in the next section.

Python example of performing LDA on real-life data

Finally, it's time for the fun stuff where we get to apply LDA using Python. Let's start by getting the right libraries and data for our analysis.

Setup

We will use the following data and libraries:

Let's import all the libraries:

python
# Data manipulationimport pandas as pd # for data manipulation# Visualizationimport plotly.express as px # for data visualization# Skleranfrom sklearn.preprocessing import StandardScaler # for data standardizationfrom sklearn.metrics import classification_report # for model evaluation metricsfrom sklearn.preprocessing import OrdinalEncoder # to encode categorical variablesfrom sklearn.tree import DecisionTreeClassifier # for decision tree models# Sklearn dimensionality reductionfrom sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA # for LDA analysisfrom sklearn.decomposition import PCA # for PCA analysis

Next, we download an ingest house price data from Kaggle.

python
# Read in datadf = pd.read_csv('Real estate.csv', encoding='utf-8')# Print dataframedf

Here's a snippet of what the data looks like:

Real estate data from Kaggle. Image by author.
Real estate data from Kaggle. Image by author.

Data Preparation

For this example, we want to create a multiclass target variable. Hence, we take 'Y house price of unit area' and divide it into 3 equal categories. We call them: '1.Affordable' (bottom 33%), '2.Mid-range' (middle 33%), and '3.Expensive' (top 33%).

python
# Bin unit price into 33.3% quantilesdf['Price Band'] = pd.qcut(df['Y house price of unit area'], 3, labels=['1.Affordable (bottom 33%)', '2.Mid-range (middle 33%)', '3.Expensive (top 33%)'])# Check distributiondf['Price Band'].value_counts().sort_index()
Category (class) label distribution. Image by author.
Category (class) label distribution. Image by author.

Since we have created categorical labels while the algorithm requires numerical ones, let's use the Ordinal encoder to convert them to numeric. [Note, we could have directly assigned numeric labels, but the categorical ones will make it easier for us to read the upcoming visualization].

python
# Select encoding methodenc=OrdinalEncoder() # Encode categorical valuesdf['Price Band enc']=enc.fit_transform(df[['Price Band']])# Check encoding results in a crosstabpd.crosstab(df['Price Band'], df['Price Band enc'], margins=False)
Encoded target variable class labels. Image by author.
Encoded target variable class labels. Image by author.

To keep the ability to visualize the data, we will limit ourselves to the first 3 feature variables: 'X1 transaction date', 'X2 house age', and 'X3 distance to the nearest MRT station'.

Let's see what the data looks like by plotting these three features on the 3D scatterplot:

python
# Create a 3D scatter plotfig = px.scatter_3d(df,                     x=df['X1 transaction date'], y=df['X2 house age'], z=df['X3 distance to the nearest MRT station'],                    color=df['Price Band'],                    color_discrete_sequence=['#636EFA','#EF553B','#00CC96'],                     hover_data=['X3 distance to the nearest MRT station', 'Y house price of unit area', 'Price Band enc'],                    height=900, width=900                   )# Update chart looksfig.update_layout(#title_text="Scatter 3D Plot",                  showlegend=True,                  legend=dict(orientation="h", yanchor="top", y=0, xanchor="center", x=0.5),                  scene_camera=dict(up=dict(x=0, y=0, z=1),                                         center=dict(x=0, y=0, z=-0.2),                                        eye=dict(x=-1.5, y=1.5, z=0.5)),                                        margin=dict(l=0, r=0, b=0, t=0),                  scene = dict(xaxis=dict(backgroundcolor='white',                                          color='black',                                          gridcolor='#f0f0f0',                                          title_font=dict(size=10),                                          tickfont=dict(size=10),                                         ),                               yaxis=dict(backgroundcolor='white',                                          color='black',                                          gridcolor='#f0f0f0',                                          title_font=dict(size=10),                                          tickfont=dict(size=10),                                          ),                               zaxis=dict(backgroundcolor='lightgrey',                                          color='black',                                           gridcolor='#f0f0f0',                                          title_font=dict(size=10),                                          tickfont=dict(size=10),                                         )))# Update marker sizefig.update_traces(marker=dict(size=2))fig.show()

We can see that the data is scattered significantly. Also, we note that cheaper real estate is located further away from MRT stations, while the expensive ones tend to be much closer to MRT and also newer.

One final step in data preparation is the standardization of the features.

python
# Select featuresX=df[['X1 transaction date', 'X2 house age', 'X3 distance to the nearest MRT station']]# Select targety=df['Price Band enc'].values# Get scalerscaler=StandardScaler()# Perform standard scaling on model featuresX_std=scaler.fit_transform(X)

Note that we should also split our data into train and test samples at this stage, but we'll work with the entire dataset to keep things simple for this example.

Performing Principal Component Analysis (PCA)

For interest, we'll also perform PCA so we can compare the results of PCA to the results of LDA.

python
# Select the model and its parameterspca = PCA(n_components=2) # reducing from 3 dimensions to 2# Fit the modelX_trans_pca=pca.fit_transform(X_std)# Print the resultsprint('*************** PCA Summary ***************')print('No. of features: ', pca.n_features_)print('No. of samples: ', pca.n_samples_)print('No. of components: ', pca.n_components_)print('Explained variance ratio: ', pca.explained_variance_ratio_)
PCA results. Image by author.
PCA results. Image by author.

Interestingly, explained variance ratios for PC1 and PC2 only cover ~69% of the total variance, meaning that we have lost about 31% of the total variance when going down from 3 dimensions to 2. This is largely due to little correlation between the 3 features.

Here is a scatter plot of the resulting 2D plot after applying PCA:

python
# Create a scatter plotfig = px.scatter(df, x=X_trans_pca[:,0], y=X_trans_pca[:,1],                  labels={                     "x": "PC1",                     "y": "PC2",                 },                 opacity=1, color=df['Price Band'])# Change chart background colorfig.update_layout(dict(plot_bgcolor = 'white'))# Update axes linesfig.update_xaxes(showgrid=True, gridwidth=1, gridcolor='lightgrey',                  zeroline=True, zerolinewidth=1, zerolinecolor='lightgrey',                  showline=True, linewidth=1, linecolor='black')fig.update_yaxes(showgrid=True, gridwidth=1, gridcolor='lightgrey',                  zeroline=True, zerolinewidth=1, zerolinecolor='lightgrey',                  showline=True, linewidth=1, linecolor='black')# Set figure titlefig.update_layout(title_text="PCA scatter plot")# Update marker sizefig.update_traces(marker=dict(size=3))fig.show()
PCA plot. Image by author.
PCA plot. Image by author.

While we did lose 31% of the variance, we still managed to keep a lot of the spread across the two new dimensions. However, since PCA only considers features and not the target, the three categories (classes) ended up all mixed together.

Next, let's see how LDA compares to PCA.

Performing Linear Discriminant Analysis (LDA)

We will use Eigen decomposition as our solver (sklearn implementation enables you to choose between SVD, LSQR, and Eigen) and set the components parameter (number of dimensions) to 2, leaving all the rest as default. Note that this time we also need to supply our target (y) to the algorithm.

python
# Select the model and its parameterslda = LDA(    solver='eigen', #{'svd', 'lsqr', 'eigen'}, default='svd'    n_components=2, #int, default=None    #shrinkage=None, #'auto' or float, default=None    #priors=None, #array-like of shape (n_classes,), default=None, The class prior probabilities. By default, the class proportions are inferred from the training data.    #store_covariance=False, #bool, default=False, If True, explicitely compute the weighted within-class covariance matrix when solver is 'svd'.     #tol=0.0001, #float, default=1.0e-4, Absolute threshold for a singular value of X to be considered significant, used to estimate the rank of X.)# Fit transform the dataX_trans_lda=lda.fit_transform(X_std,y)# Print the resultsprint('*************** LDA Summary ***************')print('Classes: ', lda.classes_)print('Priors: ', lda.priors_)print('Explained variance ratio: ', lda.explained_variance_ratio_)
LDA results. Image by author.
LDA results. Image by author.

As before, we also visualize the results on a 2D scatterplot.

python
# Create a scatter plotfig = px.scatter(df, x=X_trans_lda[:,0], y=X_trans_lda[:,1],                  labels={                     "x": "LD1",                     "y": "LD2",                 },                 opacity=1, color=df['Price Band'])# Change chart background colorfig.update_layout(dict(plot_bgcolor = 'white'))# Update axes linesfig.update_xaxes(showgrid=True, gridwidth=1, gridcolor='lightgrey',                  zeroline=True, zerolinewidth=1, zerolinecolor='lightgrey',                  showline=True, linewidth=1, linecolor='black')fig.update_yaxes(showgrid=True, gridwidth=1, gridcolor='lightgrey',                  zeroline=True, zerolinewidth=1, zerolinecolor='lightgrey',                  showline=True, linewidth=1, linecolor='black')# Set figure titlefig.update_layout(title_text="LDA scatter plot")# Update marker sizefig.update_traces(marker=dict(size=3))fig.show()
LDA plot. Image by author.
LDA plot. Image by author.

There is a notable difference in LDA results when compared to PCA. Since the goal of PCA is to separate target categories instead of maximizing variance, it managed to find new axes that reduced the spread of blue ('2.Mid-range') and red ('3.Expensive') points, keeping them largely within their own space. Meanwhile, despite the large variance remaining in the green ('1.Affordable) class, the separation from red and blue is pretty good.

To validate what we see in these scatter plots, let's build a couple of decision tree models and see how well we can predict 'Price Band' based on features created using PCA and LDA.

Creating prediction models with Decision Tree Classifier

Let's set up a reusable function that we can quickly call to train a model and display the results.

python
def fitting(X_in, y, criterion, splitter, mdepth, clweight, minleaf):    # Fit the model    model = DecisionTreeClassifier(criterion=criterion,                                    splitter=splitter,                                    max_depth=mdepth,                                   class_weight=clweight,                                   min_samples_leaf=minleaf,                                    random_state=0,                                   )    clf = model.fit(X_in, y)    # Predict class labels on training data    pred_labels_tr = model.predict(X_in)    # Use score method to get accuracy of the model    print('*************** Tree Summary ***************')    print('Classes: ', clf.classes_)    print('Tree Depth: ', clf.tree_.max_depth)    print('No. of leaves: ', clf.tree_.n_leaves)    print('No. of features: ', clf.n_features_)    #print('Feature Importance: ')    #print(list(zip(X.columns, clf.feature_importances_)))    print('--------------------------------------------------------')    print("")    print('*************** Evaluation on Training Data ***************')    score_tr = model.score(X_in, y)    print('Accuracy Score: ', score_tr)    # Look at classification report to evaluate the model    print(classification_report(y, pred_labels_tr))    print('--------------------------------------------------------')    # Return relevant data for chart plotting    return clf

Now let's use it to build a model with PCA-transformed features.

python
# Fit the model and display resultsclf_pca = fitting(X_trans_pca, y, 'gini', 'best', mdepth=3, clweight=None, minleaf=10)

Here are the model results:

Decision tree model results using PCA-transformed features. Image by author.
Decision tree model results using PCA-transformed features. Image by author.

As expected, the results are not great, with accuracy only at 64%. Mixing of the categories (classes) after applying PCA has definitely made it harder to predict the price label.

Let's repeat the same with LDA-transformed features.

python
# Fit the model and display resultsclf_lda = fitting(X_trans_lda, y, 'gini', 'best', mdepth=3, clweight=None, minleaf=10)
Decision tree model results using LDA-transformed features. Image by author.
Decision tree model results using LDA-transformed features. Image by author.

This time the model seems to be much better, with nearly 79% accuracy. These results support our earlier intuition that LDA has better preserved the information relevant to predicting an apartment's price when compared to PCA.

Conclusions

LDA is a great tool when we want to reduce the dimensionality of our data while keeping as much information relevant to our prediction target.

However, the direct comparison of PCA and LDA might not be completely fair because they are two different techniques meant for different purposes. For example, PCA is an unsupervised learning technique, while LDA falls under the supervised branch of ML.

Hence, please don't take the results above as proof of LDA superiority over PCA. Instead, please first assess the applicability of these algorithms to your unique case.

I sincerely hope my article has helped you better understand Linear Discriminant Analysis, enabling you to incorporate it into your own Data Science projects.

Cheers! 👏 Saul Dobilas


If you liked this story, here are couple more articles that you might enjoy:

DBSCAN Clustering Algorithm - How to Build Powerful Density-Based Models

k-Nearest Neighbors (kNN) - How To Make Quality Predictions With Supervised Learning?

Related Articles