Data Science

Logistic Regression in Python- A Helpful Guide to How It Works

A detailed explanation of the algorithm together with useful examples on how to build a model in Python

Saul Dobilas
January 3, 202117 min read

Machine Learning

Logistic Regression in Python— A Helpful Guide to How It Works

Logistic Regression. Image by author. (See how this graph was made in the Python section below)
Logistic Regression. Image by author. (See how this graph was made in the Python section below)

Preface

Just so you know what you are getting into, this is a long article that contains a visual and a mathematical explanation of logistic regression with 4 different Python examples. Please take a look at the list of topics below and feel free to jump to the sections that you are most interested in.

Intro

Machine Learning is making huge leaps forward, with an increasing number of algorithms enabling us to solve complex real-world problems.

This story is part of a deep dive series explaining the mechanics of Machine Learning algorithms. In addition to giving you an understanding of how ML algorithms work, it also provides you with Python examples to build your own ML models.

This story covers the following topics:

  • The category of algorithms logistic regression belongs to

  • An explanation of how logistic regression works

  • Python examples of how to build logistic regression models, including:

  • Binary target with 1 independent variable

  • Binary target with 2 independent variables

  • Multinomial with 3 class labels and 2 independent variables

  • Multinomial with 3 class labels and 2 independent variables + oversampling

What category of algorithms does logistic regression belong to?

Looking at the below chart's supervised learning branch, we can see that we have two main categories of problems: regression and classification.

  • Regression: we use regression algorithms when we have a continuous (numerical) target variable. For example, predicting the price of a house based on its proximity to major amenities.

  • Classification: used when the target variable is categorical. For example, predicting a win/loss of a game or customer defaulting/not-defaulting on a loan payment. Note, it does not necessarily have to be a binary outcome.

While logistic regression has a "regression" in its name, it actually belongs to the classification algorithms. However, there are some similarities between linear regression and logistic regression, which we will touch upon in the next section.

The below graph is interactive, so make sure to click on different categories to enlarge and reveal more👇 .

If you enjoy Data Science and Machine Learning, please subscribe to get an email whenever I publish a new story.

How does logistic regression work?

Let's begin the explanation by looking at the following example.

Assume we have a class of 10 pupils where each of them had to take an exam. Their preparation time, final score, and outcome (pass/fail) are displayed below. Note, the passing score is 40.

Now, let's see how we would approach this problem using linear regression vs. logistic regression.

A quick recap on linear regression

If we were to build a simple linear regression model, we could use 'hours of study' as our independent variable and 'final score' as the dependent (target) variable. This is because 'final score' is a continuous variable as required by regression. This would lead us to a result summarized by a best-fit line taking the following form:

text
where β(0) is an intercept, β(1) is a slope, and x(1) is the sole independent variable. 

Note, adding more independent variables would result in having more elements in your equation:

Logistic function

Let's now assume that we do not have a 'final score.' All we have is an outcome( pass/fail flag). We want to build a logistic regression model where we use 'hours of study' to predict a student's likelihood of passing the exam.

As you can see from the table above, there is a strong correlation between 'hours of study' and 'exam outcome,' although we cannot perfectly separate the two classes. Hence, we want to have a model that gives us a probability of passing the exam given the study hours.

This is done by using a logistic function, also known as a sigmoid function:

If we were to plot a logistic function on a chart, it would look like this:

Odds

To understand how the data is mapped to the logistic function, we first need to learn about the relationship between probability, odds, and log-odds.

  • Odds - this is simply a ratio between the number of events (in this case, exam passes) and non-events (exam failures). Say, if you had 5 pupils that spent 7 hours each studying for an exam with 3 pupils passing and 2 failing it, the odds of passing would be 3:2, which is 1.5 in decimal notation.

  • Log-odds - is just a natural logarithm of odds. So if, the odds are 3:2 = 1.5, then log(odds) = log(1.5) = 0.405...

  • Probability vs. odds - you can easily convert between probability and odds. So if, the odds are 3:2, then the probability is 3/5=0.6.You can use the following equations to convert between probability and odds:

  • The last thing to note is that S(t) in the logistic function is the probability p. Hence, using the above equations, we can derive that t=log(odds). Which makes our logistic function:

Obviously, we could simplify it further, which would lead us back to the original equation of probability expressed through odds. However, we are happy with this form because now we can go one step further to find the log-odds equation.

Log-odds equation

Let's use another example to plot the data onto a graph to understand how the log-odds equation is created.

We can plot this data onto a chart with 'study hours' on the x-axis and log-odds on the y-axis:

Now, this looks familiar. The relationship between our independent variable x (hours of study) and log-odds is linear! This means that we can draw the best fit-line through the points using the same type of line equation:

This makes our Logistic function:

A general form with multiple independent variables becomes:

Maximum Likelihood Estimation (MLE)

When you build logistic regression models, the algorithm's goal is to find the coefficients β(0), β(1), etc. Unlike linear regression, though, it is not done by minimizing squared residuals but finding the maximum likelihood instead.

Maximum likelihood is most often expressed through a log-likelihood formula:

text
where p is the probability for points with an actual outcome of event ("pass") and 1-p is the probability for points with an actual outcome of non-event ("fail").

There are multiple methods available to maximize the log-likelihood. Some of the most commonly used ones would be gradient descent and Newton–Raphson.

In general, methods used to find the coefficients for the logistic function go through an iterative process of selecting a candidate line and calculating the log-likelihood. This is continued until the convergence is achieved and the maximum likelihood is found.

Note, I will not go into the mechanics of these algorithms. Instead, let's build some logistic regression models in Python.

Logistic regression in Python

Now is the time to build some models using the knowledge that we acquired.

Setup

We will use the following libraries and data:

Let's import all the libraries:

python
import pandas as pd # for data manipulationimport numpy as np # for data manipulationfrom sklearn.linear_model import LogisticRegression # for building a logistic regression modelfrom sklearn.model_selection import train_test_split # for splitting the data into train and test samplesfrom sklearn.metrics import classification_report # for model evaluation metricsfrom imblearn.over_sampling import RandomOverSampler # for oversampling / undersamplingimport plotly.express as px  # for data visualizationimport plotly.graph_objects as go # for data visualization

We will use data on chess games from Kaggle, which you can download following this link: https://www.kaggle.com/datasnaek/chess.

Once you have saved the data on your machine, we ingest it with the following code:

python
# Read in the csvdf=pd.read_csv('games.csv', encoding='utf-8')# Print the first few columns df.iloc[:,:12]
A snippet of the data from Kaggle's chess dataset. Image by author.
A snippet of the data from Kaggle's chess dataset. Image by author.

As we will want to use the 'winner' field for our dependent (target) variable, let's check the distribution of it:

Chess match data winner distribution. Image by author.
Chess match data winner distribution. Image by author.

It is good to see that the wins between white and black are quite balanced. However, a small minority of matches ended up in a draw. Having an underrepresented class will make it harder to predict it, which we will see the multinomial examples later.

For the binary outcome model, we will try to predict whether the white pieces will win using the player rating difference. Meanwhile, for the multinomial case, we will attempt to predict all three classes (white win, draw, black win).

First, let's derive a few new fields for usage in model predictions.

python
# Difference between white rating and black rating - independent variabledf['rating_difference']=df['white_rating']-df['black_rating']# White wins flag (1=win vs. 0=not-win) - dependent (target) variabledf['white_win']=df['winner'].apply(lambda x: 1 if x=='white' else 0)# Match outcome (1=white wins, 0=draw, -1=black wins) - dependent (target) variable for multinomial regressiondf['match_outcome']=df['winner'].apply(lambda x: 1 if x=='white' else                                        0 if x=='draw' else -1)# Check by printing last few cols in a dataframedf.iloc[:,13:]
Three newly derived columns were added to a DataFrame. Image by author.
Three newly derived columns were added to a DataFrame. Image by author.

Logistic regression for a binary outcome - 1 independent variable

Let's start building! We will use the difference between white and black ratings as the independent variable and the 'white_win' flag as the target.

After splitting the data into train and test samples, we fit the model. We chose sag (stochastic average gradient) solver for finding beta parameters of the log-odds equation this time. As listed in the comments below, there are other solvers, which we will try in the next few examples.

python
# Select data for modelingX=df['rating_difference'].values.reshape(-1,1)y=df['white_win'].values# Create training and testing samplesX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)# Fit the model# Note, solvers available: {'newton-cg', 'lbfgs', 'liblinear', 'sag', 'saga'}, default='lbfgs'model = LogisticRegression(solver='sag')clf = model.fit(X_train, y_train)# Predict class labels on a test dataLR1_pred_labels = model.predict(X_test)# Print slope and interceptprint('Intercept (Beta 0): ', clf.intercept_)print('Slope (Beta 1): ', clf.coef_)
Intercept and slope for the model equation. Image by author.
Intercept and slope for the model equation. Image by author.

This gives us the following log-odds and logistic equations:

Let's check our model performance metrics on the test sample:

python
# Use score method to get accuracy of modelscore = model.score(X_test, y_test)print('Accuracy Score: ', score)print('--------------------------------------------------------')# Look at classification report to evaluate the modelprint(classification_report(y_test, LR1_pred_labels))
Model performance metrics. Image by author.
Model performance metrics. Image by author.

A quick recap on the performance metrics:

  • Accuracy = Correct predictions / Total predictions

  • Precision = True Positives / (True Positives + False Positives); lower precision means higher number of False Positives

  • Recall = True Positives / (True Positives + False Negatives); low recall means that the model contains many False Negatives, i.e., it could not correctly identify a large proportion of the class members.

  • F1-score = Average between Precision and Recall (weights can be applied if one metric is more important than the other for a specific use case)

  • Support = Number of actual observations in that class

We can see that while the model is not great, it still helps us to identify the white win in 64% of the cases, which is better than a random guess (a 50% chance of getting it right).

Next, let's plot a Logistic function with each class mapped onto it. We will do some data preparation first:

python
# Predict probabilities for each class label on test dataprobs=model.predict_proba(X_test)#probs[:,1]# This is for plottling logistic function# Create 100 evenly spaced points from smallest X to largest XX_range = np.linspace(X.min(), X.max(), 100)# Predict probabilities for each class labely_range = model.predict_proba(X_range.reshape(-1, 1))# Create a boolean array for masking data (needed for graph)mask_0 = y_test < 0.5   mask_1 = y_test > 0.5 print(y_test)#print(mask_0)print(mask_1)
Boolean array to be used for masking. Image by author.
Boolean array to be used for masking. Image by author.

We will use masking in the graph to create two separate traces, one with events (white won) and the other with non-events (white did not win). As you can see, it is simply a boolean array contain True for 1 and False for 0.

python
# Create a scatter plotfig = px.scatter(df, x=X_test.ravel(), y=y_test,                  opacity=0.8, color_discrete_sequence=['black'],                 labels=dict(x="Rating Points Difference Between White and Black", y="Predicted Probability for White Win",))# Add additional tracesfig.add_traces(go.Scatter(x=X_test.ravel()[mask_1], y=probs[:,1][mask_1]+0.01, name='White won', mode = 'markers', opacity=0.9, marker=dict(color='limegreen')))fig.add_traces(go.Scatter(x=X_test.ravel()[mask_0], y=probs[:,1][mask_0]-0.01, name='White did not win', mode = 'markers', opacity=0.9, marker=dict(color='red')))fig.add_traces(go.Scatter(x=X_range, y=y_range[:,1], name='Logistic Function', line=dict(color='black')))# Change chart background colorfig.update_layout(dict(plot_bgcolor = 'white'))# Update axes linesfig.update_xaxes(showgrid=True, gridwidth=1, gridcolor='lightgrey',                  zeroline=True, zerolinewidth=1, zerolinecolor='lightgrey',                  showline=True, linewidth=1, linecolor='black')fig.update_yaxes(showgrid=True, gridwidth=1, gridcolor='lightgrey',                  zeroline=True, zerolinewidth=1, zerolinecolor='lightgrey',                  showline=True, linewidth=1, linecolor='black')# Set figure titlefig.update_layout(title=dict(text="Binary Logistic Regression (1 Independent Variable) Model Results",                              font=dict(color='black')))# Update marker sizefig.update_traces(marker=dict(size=7))fig.show()
Logistic regression model results. Image by author.
Logistic regression model results. Image by author.

Let's take a look at what is displayed here.

  • The black dots at the top are the test dataset observations with the actual class of 1 (white won). In comparison, the black dots at the bottom are observations with the actual class of 0 (white did not win).

  • The black line is the logistic function which is based on the equation we derived with our model giving us the following parameters:intercept = -0.00289864 and slope = 0.00361573.

  • Green dots are black dots with class=1 mapped onto the logistic function using the probabilities from the model.

  • Red dots are black dots with class=0 mapped onto the logistic function using the probabilities from the model.

Quick note, I had to offset green and red dots by a small amount (0.01) to avoid overlapping for easier reading.

In summary, while the model can correctly predict a white win in 64% of the cases {p(white win)>0.5}, there are also lots of cases (36%) where it did not predict the outcome successfully. This suggests that having a higher rating in chess does not guarantee success in a match.

Logistic regression for a binary outcome - 2 independent variables

Let's add an additional independent variable to the next model. We will use a field called 'turns,' which tells us the total number of moves made in a match.

Note that we are somewhat cheating here as the number of total moves would only be known after the match. Hence, this data point would not be available to us if we were to make a prediction before the match starts. Nevertheless, this is for illustration purposes only, so we will go ahead and use it anyway.

python
# Select data for modelingX=df[['rating_difference', 'turns']]y=df['white_win'].values# Create training and testing samplesX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)# Fit the model# Note, solvers available: {'newton-cg', 'lbfgs', 'liblinear', 'sag', 'saga'}, default='lbfgs'model = LogisticRegression(solver='lbfgs')clf = model.fit(X_train, y_train)# Predict class labels on a test dataLR2_pred_labels = model.predict(X_test)# Print slope and interceptprint('Intercept (Beta 0): ', clf.intercept_)print('Slopes (Beta 1 and Beta 2): ', clf.coef_)
Intercept and slope for the model equation. Image by author.
Intercept and slope for the model equation. Image by author.

Note that we have two slope parameters this time, one for each independent variable. β(2) is slightly negative, suggesting that a higher number of 'turns' indicates a lower chance of white winning. This makes sense as the white not winning also includes 'draws,' and they are more likely to occur after a long match (after many moves).

Let's take a look at model performance metrics on a test sample:

python
# Use score method to get accuracy of modelscore = model.score(X_test, y_test)print('Accuracy Score: ', score)print('--------------------------------------------------------')# Look at classification report to evaluate the modelprint(classification_report(y_test, LR2_pred_labels))
Model performance metrics. Image by author.
Model performance metrics. Image by author.

We can see that all classification metrics have improved for this model with 66% correct predictions. Not a surprise, given we used the 'turns' field, which gives us information about how the match has evolved.

Let's now do some data prep and plot a logistic function again, although this time, it will be a surface on a 3D graph instead of a line. It is because we used 2 independent variables in our model.

python
# Predict probabilities for each class label on test dataprobs=model.predict_proba(X_test)#probs[:,1]# --------- This is for plottling logistic function plane --------# Increments between points in a meshgridmesh_size = 10# How much to extend beyond min and max values (optional)margin = 0# Identify min and max values for input variablesx_min, x_max = X_test['rating_difference'].min() - margin, X_test['rating_difference'].max() + marginy_min, y_max = X_test['turns'].min() - margin, X_test['turns'].max() + margin# Return evenly spaced values based on a range between min and maxxrange = np.arange(x_min, x_max, mesh_size)yrange = np.arange(y_min, y_max, mesh_size)# Create a meshgridxx, yy = np.meshgrid(xrange, yrange)# Use models to create a prediciton plane - Logistic Regressionpred_LR = model.predict_proba(np.c_[xx.ravel(), yy.ravel()])pred_LR = pred_LR[:,1].reshape(xx.shape)# Note, .ravel() flattens the array to a 1D array,# then np.c_ takes elements from flattened xx and yy arrays and puts them together,# this creates the right shape required for model input# prediction array that is created by the model output is a 1D array,# Hence, we need to reshape it to be the same shape as xx or yy to be able to display it on a graph# --------- Create a boolean array for masking data (needed for graph) -------- mask_0 = y_test < 0.5   mask_1 = y_test > 0.5 #print(y_test)#print(mask_0)#print(mask_1)

Plot the graph:

python
# Create a 3D scatter plot with predictionsfig = px.scatter_3d(df, x=X_test['rating_difference'], y=X_test['turns'], z=y_test,                  opacity=0.8, color_discrete_sequence=['black'],                 labels=dict(x="Rating Points Difference Between White and Black",                              y="Number of Turns",                             z="Predicted Probability for White Win",))# Set figure title and colorsfig.update_layout(title_text="Scatter 3D Plot with LR Prediction Surface",                  scene = dict(xaxis=dict(backgroundcolor='white',                                          color='black',                                          gridcolor='lightgrey'),                               yaxis=dict(backgroundcolor='white',                                          color='black',                                          gridcolor='lightgrey'                                          ),                               zaxis=dict(backgroundcolor='white',                                          color='black',                                           gridcolor='lightgrey')))# Update marker sizefig.update_traces(marker=dict(size=3))# Add prediction planefig.add_traces(go.Surface(x=xrange, y=yrange, z=pred_LR, name='LR Prediction',                          colorscale=px.colors.sequential.Sunsetdark))fig.add_traces(go.Scatter3d(x=X_test['rating_difference'][mask_1], y=X_test['turns'][mask_1], z=probs[:,1][mask_1],                             name='White won', showlegend=False, mode = 'markers', opacity=0.9, marker=dict(color='limegreen', size=3)))fig.add_traces(go.Scatter3d(x=X_test['rating_difference'][mask_0], y=X_test['turns'][mask_0], z=probs[:,1][mask_0],                             name='White did not win', showlegend=False, mode = 'markers', opacity=0.9, marker=dict(color='blue', size=3)))fig.show()
Logistic regression with 2 independent variables - model result. Image by author.
Logistic regression with 2 independent variables - model result. Image by author.

This graph shows how the black dots at the top (class=1) and the bottom (class=0) have been mapped onto the logistic function prediction surface. In this case, green dots show probabilities for class=1 and blue ones for class=0.

Multinomial logistic regression - 2 independent variables

Let's now build a model that has 3 class labels:

  • -1: black wins

  • 0: draw

  • 1: white wins

python
# Select data for modelingX=df[['rating_difference', 'turns']]# Match outcome (-1=black wins, 0=draw, 1=white wins)y=df['match_outcome'].values# Create training and testing samplesX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)# Fit the model# Note, solvers available: {'newton-cg', 'lbfgs', 'liblinear', 'sag', 'saga'}, default='lbfgs'model = LogisticRegression(solver='newton-cg', max_iter=1000)clf = model.fit(X_train, y_train)# Predict class labels on a test dataLR3_pred_labels = model.predict(X_test)# Print slope and intercept - note each class will have its own equation# Order: (-1=black wins, 0=draw, 1=white wins)print('Intercepts (Beta 0s): ', clf.intercept_)print('Slopes (Beta 1s and Beta 2s): ')print(clf.coef_)
Intercept and slope for the model equation. Image by author.
Intercept and slope for the model equation. Image by author.

Note that for a multinomial case, we have three intercepts and 3 pairs of slopes. This is because the model creates a separate equation for predicting each class.

Let's look at the model performance:

python
# Use score method to get accuracy of modelscore = model.score(X_test, y_test)print('Accuracy Score: ', score)print('--------------------------------------------------------')# Look at classification report to evaluate the modelprint(classification_report(y_test, LR3_pred_labels))
Model performance metrics. Image by author.
Model performance metrics. Image by author.

As expected, the model had some difficulty predicting class=0 (draw) due to the unbalanced data. You can see a lot fewer draw outcomes (175 in the test sample) than wins by either white or black.

Based on precision, we can see that the model got 43% of its 'draw' predictions right. However, the recall is only 0.02, meaning that there were very few cases where the model predicted a 'draw' with most of the 'draw' outcomes being unidentified.

There are multiple ways of dealing with unbalanced data, with one approach being to oversample the minority class (in this case, class=0).

Multinomial logistic regression with oversampling - 2 independent variables

We will use the "random oversampler" from the imbalanced-learn package to help with our quest.

python
# Select data for modelingX=df[['rating_difference', 'turns']]# Match outcome (-1=black wins, 0=draw, 1=white wins)y=df['match_outcome'].values# Create training and testing samplesX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)# This will oversample the minority class (draw) to match majority classes in the training dataoversample = RandomOverSampler(sampling_strategy='minority')X_over, y_over = oversample.fit_resample(X_train, y_train)# Fit the model# Note, solvers available: {'newton-cg', 'lbfgs', 'liblinear', 'sag', 'saga'}, default='lbfgs'model = LogisticRegression(solver='newton-cg', max_iter=1000)clf = model.fit(X_over, y_over)# Predict class labels on a test dataLR4_pred_labels = model.predict(X_test)# Print slope and intercept - note each class will have its own equation# Order: (-1=black wins, 0=draw, 1=white wins)print('Intercepts (Beta 0s): ', clf.intercept_)print('Slopes (Beta 1s and Beta 2s): ')print(clf.coef_)
Intercept and slope for the model equation. Image by author.
Intercept and slope for the model equation. Image by author.
python
# Use score method to get accuracy of modelscore = model.score(X_test, y_test)print('Accuracy Score: ', score)print('--------------------------------------------------------')# Look at classification report to evaluate the modelprint(classification_report(y_test, LR4_pred_labels))
Model performance metrics. Image by author.
Model performance metrics. Image by author.

These are the final results. We can see that the model accuracy has gone down due to a reduction in precision for class=0. This is expected with oversampling as the model expects the class to be much more common than it actually is, leading to more frequent predictions of a 'draw.'

While this harmed precision, it has helped with recall as the model was able to identify more of the 'draw' outcomes.

Clearly, this model is far from ideal and more work is needed to improve it. This can be done by adding more independent variables and employing additional techniques such as undersampling majority classes.

However, the purpose of these examples was to show you how you can build different types of logistic regression models rather than finding the best model for this specific set of data. I believe I have given you plenty of examples to work with. Hence, I will stop the story here.

Conclusions

This has been one of the longer stories I have written. If you managed to get all the way to the end, then kudos to you! 👏

I hope you now have a good understanding of what logistic regression is and that I have inspired you to open your notebook and to start building logistic regression models yourself.

Cheers! 👏 Saul Dobilas


Related stories you may like:

SVM Classifier and RBF Kernel - How to Make Better Models in Python

Naive Bayes Classifier - How to Successfully Use It in Python?

Related Articles