Data Science

Building a Sentiment Analysis model using Yelp Reviews and ML Ensemble Methods

What is actually the difference between a three-star review and a four-star review? Natural Language Processing has the answer!

Ben Chamblee
October 3, 202111 min read

Building a Sentiment Analysis Model using Yelp Reviews and ML Ensemble Methods

Image by abillion on Unsplash
Image by abillion on Unsplash

If you've ever read through Yelp reviews you might have noticed that there is a lot of inconsistency from reviewer to reviewer. Some reviews say the food is "life-changing" while only leaving three stars, while others say the food is "just okay" and leave five stars. The five-star review system is flawed because there's no definitive way to define a five-star establishment versus a four-star one. However, if we ignore the number of stars and instead read through the actual reviews we should get a much better idea of what Yelp reviewers are actually saying. This sounds like much more work, but by using Python's Natural Language Toolkit we can make our code do all the hard work.

Part 1: Getting Yelp Reviews

You can go about this in a few different ways.

  • There are few Kaggle datasets like this one that have plenty of content but might not have the most up-to-date reviews.

  • There are also some other independent datasets like this one from Hugging Face. People have been using Yelp reviews for analysis for years now and there are tons of resources like this to get your started.

  • The best option, in my opinion, is to get reviews straight from the source. Yelp has a public dataset containing over 8 Million reviews all stored in a JSON file. You should be able to find a great variety of reviews, old and new all in a simple format.

Part 2: Cleaning and Prepping for Sentiment Analysis

Image by Author
Image by Author

When working with text-based data, it's beneficial to reduce the amount of words and characters so that your code doesn't take hours to run. We want to do this while still retaining the important words though, words with meaning like 'good' or 'bad' that give us an idea of the reviewer's feelings. Words like 'the', 'or', and 'is', also known as stop words, aren't very important. There's no reason to waste time and processing power going through these stops words, so it's best to just remove them. Python's Natural Language Toolkit makes this process simple:

text
from nltk.corpus import stopwordsfrom nltk.tokenize import word_tokenizeexample_sent = """This is a sample sentence,                  showing off the stop words filtration."""stop_words = set(stopwords.words('english'))word_tokens = word_tokenize(example_sent)filtered_sentence = [w for w in word_tokens if not w.lower() in stop_words]filtered_sentence = []for w in word_tokens:    if w not in stop_words:        filtered_sentence.append(w)print(word_tokens)print(filtered_sentence)
Image by Author
Image by Author

You'll also notice I'm using the word 'token' in some of my variables. That just means I'm taking a sentence and breaking it down into a list of words, that process is known as tokenization.

Image bt Author
Image bt Author

You can see here that Yelp reviews have are about 40โ€“45% stop words on average. Getting rid of them will save us a lot of time!

Now that we've filtered out the stop words, let's try and simplify the dataset using a process called stemming. Stemming is the process of reducing all forms of words to their base form. If you have words like 'greatly', 'greatest', and 'greater', those will all be reduced down to 'great' for easier grouping and processing. Once again, Python's Natural Language Toolkit makes this process very easy:

text
import nltkfrom nltk.stem.snowball import SnowballStemmersnowBallStemmer = SnowballStemmer("english")sentence = yelp_data_unpacked[40]wordList = nltk.word_tokenize(sentence)stemWords = [snowBallStemmer.stem(word) for word in wordList]stemmed = ' '.join(stemWords)print(sentence)print('')print(stemmed)
Image by Author
Image by Author

It doesn't look like that much of a change, but if we tokenize these sentences and group them together there will be fewer unique key-value pairs.

At this point in the project, we've got our data cleaned and ready for processing. Before we get to building our classification model we have to decide how we're going to classify it.

Part 3: Categorization and Hyperparameters

At the start of this article, I said that I wanted to improve Yelp's five-star rating system by analyzing the actual content in each review. So, instead of sorting the reviews into five groups based on the number of stars, I'm sorting them into four groups based on sentiment. These groups are 'Negative', 'Slightly Negative', 'Slightly Positive', and 'Positive'. To sort out my reviews for a training dataset, I used Textblob's Polarity score as a baseline.

text
from textblob import TextBlob
text
yelp_data_s['Polarity'] =yelp_data_s['Stemmed'].apply(lambda x: TextBlob(x).sentiment[0])
text
sentiment = []for i in range(len(yelp_data_s)):    if yelp_data_s['Polarity'][i] >= 0.4:        sentiment.append('Positive')    if yelp_data_s['Polarity'][i] > 0.2 and yelp_data_s['Polarity'][i] < 0.4:        sentiment.append('Slightly Positive')    if yelp_data_s['Polarity'][i] <= 0.2 and yelp_data_s['Polarity'][i] > 0:        sentiment.append('Slightly Negative')    if yelp_data_s['Polarity'][i] < 0:        sentiment.append('Negative')
text
yelp_data_s['sentiment'] = sentiment
Image by Author
Image by Author

I wanted to use more than just this score to build our model though, so I decided to look at some other features.

Image by Author
Image by Author

For the second graph, I used this list of positive sentiment words to determine what a 'positive word' is.

Now I had a good idea of what features to look for while building a model. The last step before actually selecting a model was to look at the content in each review by using a Term Frequency Inverse Document Frequency Matrix (TFIDF)

Simply put, this is a way to find the most unique and impactful words in a review. Term Frequency just tells you how often the word appears, but once you add the inverse document frequency parameter you can also analyze how infrequently that word appears throughout the whole dataset. It's important to analyze these unique words because they can tell you much more than the common words that every reviewer is using.

To create the training dataset, I used these as my columns:

  • Name: just the name of the restaurant

  • Review: the full review written on Yelp

  • Polarity: The TextBlob Polarity Score

  • Positive_Words_P: The percentage of positive words

  • Sentiment: The Assigned Sentiment score

  • TFIDF: Each unique word is a column with the frequency listed as it's value, there will be a lot of columns in this dataset (which is why we removed stop words and stemmed our reviews at first)

text
from sklearn.feature_extraction.text import TfidfVectorizer
text
text = yelp_data_s['Stemmed:Sentence']Tvectorizer = TfidfVectorizer()Tvectorizer.fit(text)Tvector = Tvectorizer.transform(text)
text
voc = Tvectorizer.vocabulary_df = pd.DataFrame(Tvector.toarray(),columns=voc)col_name1 ='Name'col_name2 = 'Review'col_name3 = 'Polarity'col_name4 = 'Sentiment'col_name5 = 'Positive_Words_P'col1 = yelp_data_s['Name']col2 = yelp_data_s['Review']col3 = yelp_data_s['Polarity']col4 = yelp_data_s['sentiment']col5 = yelp_data_s['Positive_Words_P']df.insert(0,col_name1,col1)df.insert(1,col_name2,col2)df.insert(2,col_name3,col3)df.insert(3,col_name4,col4)df.insert(4,col_name5,col5)

Using this training dataset and a simple logistic regression model to test it, I got a baseline score of 67% accuracy. I knew I could improve this by tuning some hyperparameters though, so I looked at the TFIDF matrix again.

  • Minumum Word Frequency: While it is important to look at the really unique words, it might help with overfitting to raise the threshold of words that we're processing. I tested this same model but added the hyperparameter of minimum word frequency to see how it impacted accuracy.

Image by Author
Image by Author
Image by Author
Image by Author
  • Total Number of ngrams: This basically tells us how the data is processed word by word. If you have ngrams set to 1, the words will process one by one like 'this', 'is', 'not', 'lame' for example. If you have ngrams set to 2, you can process word pairs like 'this is', 'is not', 'not lame' for example. You can probably see how this would impact accuracy, with ngrams set to 1 the model will only see 'lame' and process it as negative while the model with ngrams set to 2 will see 'not lame' and process it as positive. This is reflected on this accuracy graph:

And with these two hyperparameters, we've improved our baseline accuracy to 71%! This might not seem like much, but when we start working with other models it's important to know that we have the most optimal hyperparameters.

Part 4: Model Selection

Finally, it's model selection time. For this project I wanted to try many different classification models, the whole process looked like this:

  • Choose a model type

  • Define a pipeline and use Gridsearch to find the best parameters

  • Save the model so we don't have to run it multiple times

  • Load the model and run accuracy and f1 scores

  • Repeat the process with other models

I also wanted to split up my model selection into two parts, the first four will analyze the numerical component (% of positive words, polarity score, etc.) and the last two will analyze the TFIDF matrices. Here's how each model performed:

  • Logistic Regression: F1 score of 71%, lr__C: 0.1

text
from sklearn.linear_model import LogisticRegressionfrom sklearn.model_selection import GridSearchCVfrom sklearn.metrics import make_scorer, accuracy_score, f1_scorefrom sklearn.model_selection import train_test_splitfrom sklearn.pipeline import Pipelinefrom sklearn.preprocessing import StandardScaler
text
steps = [('scaler', StandardScaler()), ('lr', LogisticRegression(solver = 'lbfgs'))] pipeline = Pipeline(steps)parameters = {'lr__C':[0.01, 0.1, 1, 10, 100]}clf = GridSearchCV(pipeline, parameters, cv = 10, scoring="accuracy") clf.fit(X_train, y_train)clf.best_params_
  • Random Forest: F1 score of 66%, max_features: 'sqrt', n_estimators: 50

text
from sklearn.ensemble import RandomForestClassifier
text
steps = [('scaler', StandardScaler()), ('rf', RandomForestClassifier())] pipeline = Pipeline(steps)
text
parameters = {'rf__n_estimators':[10 , 20, 30, 40, 50], 'rf__max_features':['auto','sqrt']}clf = GridSearchCV(pipeline, parameters, cv = 10, scoring="accuracy") clf.fit(X_train, y_train)clf.best_params_
  • Support Vector Classification (SVC): F1 score of 70%, svc__C: 0.01

text
from sklearn.svm import SVCsteps = [('scaler', StandardScaler()), ('svc', SVC(probability=False,kernel='linear',gamma='auto'))] pipeline = Pipeline(steps) 
text
parameters = {'svc__C':[0.01, 0.1, 1]}clf = GridSearchCV(pipeline, parameters, cv = 3, scoring="accuracy") clf.fit(X_train, y_train)clf.best_params_
  • Gradient Boosted Classifier: F1 score of 72%, learning_rate: 0.15, n_estimators: 500

text
from sklearn.ensemble import GradientBoostingClassifiersteps = [('scaler', StandardScaler()), ('gbc', GradientBoostingClassifier(max_features='sqrt'))] pipeline = Pipeline(steps) 
text
parameters = {'gbc__n_estimators':[10, 50, 100, 200, 500], 'gbc__learning_rate': [0.05, 0.1, 0.15, 0.2, 0.25]}clf = GridSearchCV(pipeline, parameters, cv = 10, scoring="accuracy") clf.fit(X_train, y_train)steps = [('scaler', StandardScaler()), ('gbc', GradientBoostingClassifier(learning_rate = 0.15, max_features = 'sqrt', n_estimators = 500))]  clf = Pipeline(steps)  clf.fit(X_train, y_train)
  • Naive Bayes with TFIDF: F1 Score of 64%, alpha: 1, min_df: 10

text
from sklearn.feature_extraction.text import TfidfVectorizerfrom sklearn.naive_bayes import MultinomialNBsteps = [('vec', TfidfVectorizer(stop_words = 'english', ngram_range = (1, 2))), ('nb', MultinomialNB())] pipeline = Pipeline(steps) 
text
parameters = {'vec__min_df':[0.01, 0.1, 1, 10, 100], 'nb__alpha':[0.01, 0.1, 1, 10, 100]}clf = GridSearchCV(pipeline, parameters, cv = 10, scoring="accuracy") clf.fit(X_train, y_train)clf.best_params_
  • Gradient Boosted Classifier with TFIDF: F1 score of 57%, learning_rate: 0.25

text
steps = [('vec', TfidfVectorizer(stop_words = 'english', ngram_range = (1, 2))),          ('gbc', GradientBoostingClassifier(max_features='sqrt',n_estimators=500))] pipeline = Pipeline(steps) 
text
parameters = {'gbc__learning_rate': [0.05, 0.1, 0.15, 0.2, 0.25]}clf = GridSearchCV(pipeline, parameters, cv = 3, scoring="accuracy") clf.fit(X_train, y_train)clf.best_params_

Unfortunately, these models' scores are all around the 60โ€“70% range which isn't the best. There is a way to take the performances from the best models and combine them to improve results, it's known as Ensemble Methods. For this project, I took the best performing text-based model (Naive Bayes with TFIDF) and its performance as a feature in the best performing numerical model (Gradient Boosted Classifier). This part has a lot of steps but the code looks something like this:

text
# define train/test split 
text
X = yelp_data.Review # word tokensy = yelp_data.Sentiment # assigned sentimentindices = yelp_data.indexX_train, X_test, y_train, y_test, i_train, i_test = train_test_split(X, y, indices, train_size = 0.8, random_state = 7)
text
# define Naive Bayes TFIDF model
text
steps = [('vec', TfidfVectorizer(ngram_range = (1, 2))), ('nb', MultinomialNB())] pipeline = Pipeline(steps)parameters = {'vec__min_df':10, 'nb__alpha':0.1}clf = GridSearchCV(pipeline, parameters, cv = 10, scoring="accuracy") clf.fit(X_train, y_train)
text
# turn model scores into feature in dataframe (prediction probability)
text
Xtrain_proba = pd.DataFrame(clf.predict_proba(X_train), index = i_train) Xtest_proba = pd.DataFrame(clf.predict_proba(X_test), index = i_test)
text
# define new train/test split
text
X = yelp_data.iloc[0:,4:] # first four columns are text, review, positive word percentage, and polarityy = yelp_data.Sentiment # assigned sentimentindices = yelp_data.indexX_train, X_test, y_train, y_test, itrain, itest = train_test_split(X,y,indices,train_size=0.8,random_state=7)
text
# Adding prediction probability
text
Xtrain_combined = pd.merge(X_train, Xtrain_proba, left_index=True, right_index=True) Xtest_combined = pd.merge(X_test, Xtest_proba, left_index=True, right_index=True)
text
# define Gradient Boosted Classifier
text
steps = [('scaler', StandardScaler()), ('gbc', GradientBoostingClassifier(max_features='sqrt'))] pipeline = Pipeline(steps) 
text
parameters = {'gbc__n_estimators':[10, 50, 100, 200, 500], 'gbc__learning_rate': [0.05, 0.1, 0.15, 0.2, 0.25]}clf = GridSearchCV(pipeline, parameters, cv = 10, scoring="accuracy") clf.fit(X_train, y_train)
text
# with best features
text
steps = [('scaler', StandardScaler()), ('gbc', GradientBoostingClassifier(learning_rate = 0.2, max_features = 'sqrt', n_estimators = 500))] clf = Pipeline(steps) clf.fit(X_train, y_train)

And finally, with this stacked model we have a very improved F1 score of 92%! To show you how much more accurate this is, let's take a look at the confusion matrix of the original logistic regression (accuracy of 71%):

Image by Author
Image by Author

Versus the confusion matrix of our stacked model (accuracy of 92%):

Image by Author
Image by Author

As you can see, Ensemble Methods has a huge impact on our accuracy. It may have taken some extra time to define all those models and combine the two together, but for a 21% accuracy boost, I think it was well worth it. I would highly recommend reading more about Ensemble Methods, it will be very helpful for almost any machine learning project you work on!

Part 5: Results

Now let's put our model to the test. I went to Yelp and found a random three-star review for an Italian restaurant. A three-star review doesn't really tell you how the reviewer felt and I didn't feel like reading the entire review, so I let the model do all the work.

Image by Author
Image by Author

From this simple output, I can get a much better idea of how the reviewer felt about this Italian restaurant. The words and numbers below the sentiment show the 15 words that had the largest impact on the score, these words are usually unique and close to sentiment words like 'good' or 'bad'. From this result, I know that this reviewer had a slightly positive experience and that the scallops are probably really good. I also know that this place is probably a little humid and that this reviewer was horrified by something in the restaurant. If I were the owner of this restaurant, I would really benefit from this because I can see which words are contributing to positive reviews and which ones are contributing to negative reviews.

Natural Language Processing has a wide variety of uses, Yelp Reviews are just one of the many forms of text-based data we can use to build Sentiment Analysis Models. I learned so much from building this model, something like this would be a great introduction to NLP with Python. If you're interested in learning more I would highly suggest reading through the book Natural Language Processing in Action by Cole Howard, Hannes Hapke, and Hobson Lane.

I hope that this project walkthrough inspires you to create your own sentiment analysis model or to look into Ensemble Methods! Thank you for reading!

Full project: https://github.com/Bench-amblee/yelp_sentiment_analysis

Related Articles