Data Science

Machine learning & data science for beginners and experts alike.
The Expert Exam is now live online! Read about the specifics and what it took to bring it to life in the blog by our very own Elizabeth Bonnell!
Alteryx Certified Partner

“Confusion Matrix” is not just a term for how we all felt in 1999 after watching the Wachowski’s masterpiece for the first time. A confusion matrix is also a way of looking at the effectiveness of a model’s predictive power. So jack in as I use this blog as an excuse to write about my favourite film. 

 

Hang on, what is a Confusion Matrix? 

A confusion matrix is simply a table that shows the number of members of the actual categories in a data set, compared to the number of members in the categories predicted by a model. A generic example of a binary category would look like this: 

 

 

 

Actual True 

Actual False 

Predicted True 

True Positives 

False Positives 

Predicted False 

False Negatives 

True Negatives 

 

 

That’s all very nice, but why should I care? 

Why should you use a confusion matrix? Let’s imagine Morpheus is trying to find The One. He’s given his spiel to tons people, but most of them end up choosing the blue pill. The one thing he knows about The One is that they will definitely take the red pill, so based on some other information he’s gathered on potential candidates he wants to predict which pill they will take and limit his time talking to unviable candidates.  

 

With a dataset that has 92 Blue Pill takers and 8 Red Pill takers Morpheus’ first model for predicting who will take the Blue Pill looks like this: 

 

CristonS_0-1602540954708.png

 

 

Morpheus looks at some stats for this model and sees that the model is 92% accurate (<Correct Predictions> / <Total Predictions>) but if we look at the confusion matrix we see that this stat doesn’t tell the whole story.  

 

 

 

Actual Blue 

Actual Red 

Total 

Predicted Blue 

92 

8 

100 

Predicted Red 

0 

0 

0 

Total 

92 

8 

100 

 

 

This ‘model’ successfully predicted all of the Blue Pill takers, but none of the Red Pill takers. Other stats will also miss out on this model’s shortfall:  

 

  • Precision (<True Blues> / <Total Predicted Blues>) = 100%  
  • Recall (<True Blues> / <Total Actual Blues>) = 92% 
  • F1 Score (a harmonic mean of Precision and Recall) = 95.83% 

 

Based on these stats one might think that this is a great model, but actually it’s just reflecting the imbalance of the data set. From looking at the confusion matrix however, Morpheus can immediately see the problem with the ‘model’ and goes in search of a better one.  

 

Nice, so how does it work in practice? 

This time Morpheus is going to use his full dataset of 1000 candidates he’s previously met. He’s got some data about them (Age, Sex, Salary, Job Type) and he’s also logged which pill they took.  

 

 

CristonS_1-1602540954708.png

 

 

Like a good data analyst, Morpheus has cleaned and normalised his variables, and split this data set into a 70/30 split to train a number of models. He’s opted for: 

 

  • Decision Tree 
  • Forest Model 
  • Logistic Regression 
  • Boosted Model

And he’s using the Model Comparison Community Macro to evaluate which model to use: 

 

CristonS_2-1602540954709.png

 

 

 

The accuracy of the 4 models is as follows: 

 

 

Model 

Decision Tree 

Forest Model 

Logistic 

Boosted 

Accuracy 

77.33% 

79.67% 

77.67% 

78.33% 

 

 

However, as we’ve seen this stat isn’t the be-all-and-end-all. If we look at the confusion matrices for the 4 models, we get a more nuanced picture: 

 

 

CristonS_0-1602541278882.png

 

 

From these we can see that, despite the Forest Model having the highest overall accuracy, the Logistic Regression was better at predicting who took the Red Pill. As that is what Morpheus is actually interested in, he decides to take his Logistic Regression and use it on his dataset of potential candidates.  

 

CristonS_1-1602541368343.png

 

 

Now he can filter his list of potential candidates only to those who were more likely to take the Red Pill than the Blue Pill. This means that he can spare the majority of his list from having their entire world view destroyed. 

 

 

CristonS_3-1602540954709.gif

 

 

If you want to know more about how predictive analytics can change lives, check out the Data Science Learning Path. And if you want to try the predictive tools out, and you aren’t already addicted to the Weekly Challenges like me, then have a go at one of the predictive challenges (like #18). If you want to talk about The Matrix then hit me up on twitter @olliehclarke.