Challenge #157: An Expert Challenge

Last week's solution can be found here.

Have you ever wanted to take a sneak peek at one of the questions on the Alteryx Designer Expert exam? We thought you might, so for the first time ever we’re releasing one of our retired Expert exam questions as a Weekly Challenge! The amazing @CristonS created this question and it made its debut at the first Expert exam in Anaheim last summer. This question gave everyone a hard time, and most people avoided it all together, so if it seems intimidating you’re not alone! We wanted to keep this in the same format as the actual exam question so you won’t see an output file, just an input. We’ll post the answer and our solution next week.

You are provided a dataset (Q2_variables.yxdb) that contains multiple variables. Select the ten (10) numeric variables with the highest Mean Decrease Gini coefficient from the variable importance plot. Use these variables to build a model to predict the target variable, [H0]. Compare two models: one based on all of the selected variables, and another that includes the selected variables except [F_38]. What is the effect of removing this variable [F_38] from the model? Provide the Chi-Sq effect as your answer.

challenge_157_start_file.yxzp

Challenge_157_solution.yxzp

Predictive Analysis

Data Preparation

Predictive

Expert

Difficult

Accepted answers

All comments

patrick_digan

I would have been skipping this problem as well on the exam! Here is my best guess

Spoiler

I found via this unanswered post that perhaps the forest model would give us the variance importance plat with the mean decrease of the Gini. Thus I took the top 10 from this output of the forest model:

To get the Chi squared, I just used a simple formula and copied the data from the confusion matrix. Here is the 10 variable confusion matrix:

challenge_157_start_file.yxmd

pjdit

Well, this one was way over my head. :-) but took my best shot at it..

Spoiler

challenge_157_PD_Solution.yxmd

RolandSchubert

Really a hard one - the main problem was to find that "Variable Importance Plot Mean Decrease in Gini", but Alteryx Community has been very helpful on that.

Spoiler

The post Help Mean Decrease in Gini for Dummies was the first approach.

I decided to use Logistic Regression because field H_0 seems to be binary.

The Nested Test tool does exactly what is needed - it compare two models with one using only a subset of the variables.

challenge_157_solution_rsc.yxzp

danilang

Also in way over my head. Google was definitely my friend today!

Spoiler

Like @RolandSchubert I first searched for "Mean Decrease in Gini" which brought up the same community article,
resulting in the

Using the top 10 variables in a logistic regression and the same minus F_38 in another gave me the outputs. Then another search for "alteryx compare two models with subset of predictor variables" which brought me to the Nested Test help page.

Which gave me the Chi-Sq score

Given how much I didn't know about this entire domain, I don't think I'll be writing my expert exam any time soon.

Dan

challenge_157_solution_dl.yxzp

kat

Expert is tough! So much respect for anyone who passed!

Here is my best guess for now..

Spoiler

I did a forest (in the container) to get the top 10 variables. I then did two logistic regressions - one with and one without F38 - used this as the target is categorical. Then scored my model, made the result categorical again and let the categorical tool calculate the Chi.

This is about where I got with googling and guessing - maybe time to understand what everything means?

Challenge #157.yxzp

cplewis90

I am with most and would have skipped this one. This was a fun challenge though that I did two different ways.

Spoiler

I started by using the forest model tool to find the information needed. Definitely had to do some research on the tool mastery pages for this: Tool Mastery Index
Challenge 157-1.PNG

Once I had found my variables I put them into decision tress again, just with the 10 variables: S_3, S_6, S_11, S_12, F_15, F_17, F_21, F_23, F_29, and F_38.
I then scored the models and calculated the chi-squared statistic for each model. I finally compared the two.
Challenge 157-3.PNG

In the second method I used logistic regression as the target variable is binary. Using Logistic Regression tools, I then compared the models with the nested test tool.
Challenge 157-4.PNG

Interestingly enough, both methods produce very different answers, so I end my quest with confusion, but two really good guesses.
Challenge 157-6.PNG

challenge_157_CL.yxmd

RichoBsJ

Hi! Here my solution

Spoiler

challenge_157_richobsj.yxzp

pasccout

Here is my solution... hopefully I've found the correct CHI calculator...

Guess I need to understand more on what predictive really does

Spoiler

challenge_157_start_file.yxmd

edwin_isensee

Here is my solution.

Spoiler

challenge_157_solution_file.yxmd

T_Willins

Wow. So not ready for the Expert Exam...yet. I would have spent my entire exam time researching the tools for this. The difficult part is knowing which predictive tools to use. Went down a rabbit hole to combine results and calculate Chi-Squared until I revisited the tool documentation for predictive tools and found the Nested Test tool. I wish I had Alteryx with these tools in college (soooo long ago).

Spoiler

Chi Squared.GIF Variable Importance Plot.GIF Workflow 157.GIF

challenge_157_solution.yxmd

32bit

Pardon me if I'm confused, but what does a question like this have to do with a designer certification? Find the answer to that, and you'll know why most people avoided it.

estherb47

Cool challenge! I have no idea if my solution is anywhere near correct, but enjoyed getting there nonetheless!

Spoiler

The first approach would normally be to use a Field Summary tool to explore the data. But, since we already know what we want to target, a Summarize tool on H0 is much faster. With just 2 answers, this is probably binary, so I'll model with a Logistic Regression.

Determining the Mean Decrease Gini Coefficient means a Forest model (see the same article that many of us found here)
Variables outlined in the red box below

2 Logistic regressions, one with F_38 and one without. And a Nested Test to compare the two models. Chi-squared is in the nested test

challenge_157_EHB_completed.yxzp

kelly_gilbert

Whew, had to spend some quality time with the tool documentation on this one. Statistically, this was out of my depth, however, I was totally surprised to find that I was actually close to the right path.

Spoiler

I had no idea how (or why) to use the model outputs in a chisq test. TIL about the Nested Test Tool!

challenge_157_solution_klg.yxzp

Ladarthure

Nice challenge, way too hard without the community's help

challenge_157_start_file.yxmd

Garrett

Glad this one's been retired.

challenge_157_garrett.yxmd

ggruccio

Facing my old nemesis from the expert exam! I am one of the people who attempted this one. LOL @CristonS you won't believe what I did...

Spoiler

So...I knew immediately how to get to the variables with the highest Mean Decrease Gini by using a Random Forest Model. However when it said compare two models and provide the difference in Chi-Sq I assumed this was also using Random Forest Model for just the 10 variables then removing F38. Saw no obvious Chi-Sq difference in the output so I googled and believe I found a way to calculate it manually from the confusion matrix. I would have never figured out that I had to use Logistic Regression and had to peek at the solution to understand it....

challenge_157_gg_finish.yxzp

NicoleJohnson

Community resources to the rescue!!!

Spoiler

Part of me is extremely pleased when the solution to a question has so few tools... the other part of me, which does not entirely understand what I did to get to this answer, is mildly terrified of the fact that this answer required so few tools.

Clearly, I now know what I should be doing for the next two months until the Expert exam at Inspire Nashville...

Cheers!

challenge_157_NicoleJohnson.yxzp

JosephSerpis

Challenge Completed

challenge_157_Joe_Serpis.yxmd

philipmannering

Thanks to @patrick_digan for showing me where this variable importance plot was hiding.

My solution attached

Spoiler

workflow

Challenge 157 - An Expert Challenge.yxmd

Verakso

If am not mistaken, the Expert Exam is also only two hours.

After have spending more than double the time on this one question, I either need to really step up my Alteryx skills, or seriously downgrade my expectations on parsing the Expert exam.

Spoiler

So the first part on identifying the 10 numeric variables was not that difficult - but are you even allowed to use Google at the exam?

But after consulted Google I used the Forest Model to find the 10 numeric variables with the highest mean decrease Gini coefficient.

Mean Decrease in Gini
And the I got stocked, apparently there was no automatic way forward from here, so the 10 variables identified, was manually entered into two Logistic Regression models

And then I got stucked again 😕
I blankly admit that I had no idea on how to find the Chi-Square nor what that means for that matter. but I ended up the same place as @danilang at the Nested Test Tool page.
That did not help me much, and normally I look for a better explanation on the tool on the Tool Mastery Index, but there was no Nested Test Tool here.
Then I normally go to the Live Training, to see if there is any videos on the subject, but again I was out of luck

Workflow
So I did what I'll assume what many others do when the give up - look what others have done, so I did take a deeper look at @danilang's solution, since it seemed so close to mine.

And by peer coincidence, we end up with the same result. 😁

Chi-Sq result
Personally I did not find the Help file on the Nested Tool page that helpful, and that is where I normally turn to the Tool Mastery for a further explanation, I did manges to get the two models compared, but was in doubt how they should be connected, and how/why to use the full data set as well.
I would have hoped for some better explanation on what this tool actually does and how it works, because right now - I am not any wiser.

I did take a peek into the macro, which can provides some additional help (at least on what to connect where), but what it actually does is still a mystery to me since it uses R and other sub-macros.

So in the end, all this seems pretty simple and straight forward, but only it you know this stuff deeply and is a statistician (which I am definitely not).
I have tried to widen my horizon here, and as I mentioned - spent a lot of hours on this, mostly reading up on all the links I could find on this, but it is sure heavy stuff.

Take the decrease Gini coefficient for instance. What I have read then this is way of derimine how good the model is in the decision tree, and the lower the value is, the better (pure) you model is. If that is correct understood, then I think it is odd, that we should look at the ones with the highest value 😕

Still Climbing
/Verakso

challenge_157_solution_verakso.yxzp

Reesetrain2

Chi SQ Image.png

This was an interesting question.

-The H values were all Binary categorical variable whereas the other variables were linear.

-I used the Logistic Regression and then stepwise to eliminate the variables and then further used a Spline Model in order to see which of the linesr variables were of greatest importance as well as using a model without the categorical and examining the importance AND using stepwise to see which were eliminated.

-Very interesting Challenge.

Matt

bingqian_gao

Hi Verakso,

I will attempt to share some explanation to this one, hoping that if any of it doesn't make sense or is incorrect then someone more knowledgeable will come to rescue..

Spoiler

The way I approach to understand the Gini Importance is to go "backwards", starting from the surface level of interpreting the result:

First layer, "Mean Decrease in Gini" is a measure of variable importance for predicting a target variable. Since it's an "importance" measure, the higher it is, the more important it is.
Second layer, go one step back, Mean Decrease in Gini is the weighted average (mean) of a variable’s total decrease in node impurity, weighted by the proportion of samples reaching that node in each individual decision tree in the random forest. Since it's "impurity", it is bad, so the more it decreases (i.e. the higher the "mean decrease in Gini"), the better.
Third layer, go one step further back, Gini Impurity measures how often a randomly chosen record from the data set used to train the model will be incorrectly labelled (hence it's "bad") if it was randomly labelled according to the distribution of labels in the subset. This is essentially measuring the probability of a new record being incorrectly classified at a given node in a Decision Tree, based on the training data.

Regarding the Nested Test, it's essentially using a likelihood-ratio test (LR test) to compare the goodness of fit of two models. (https://en.wikipedia.org/wiki/Likelihood-ratio_test). The null hypothesis is that the full model is not better than the reduced model. When chi sq value is large enough (compare to the threshold), which also means the p-value is small enough, we can reject the null hypothesis, meaning we can conclude that one model (the full model) is better than the other (the reduced). In this case, since the chi sq is quite large, and the p-value is very small (typical threshold is 0.05), we can say that the effect of removing F_38 from the full model is SIGNIFICANT. The below table shows how to get to the p-value (range) from the chi sq (but Alteryx / R already gives you the p value so you don't need to worry about the conversion).

chi sq table_example.jpg

I hope this helps, and please let me know if anyone sees anything that I said was faulty...

Bingqian

Kenda

Spoiler

This would have been disastrous without help of community resources! Luckily I am familiar with the mathematics behind the tools, but I have never used 2 of the 3 key tools before. Uf! This is pretty cool that Alteryx can test models like this, though!

challenge_157_start_file.yxmd

jasperlch

Solution attached.

challenge_157_JL.yxmd

DanHare

Good fun reading around all this, and this wouldn't have been my immediate choice for Friday !

Spoiler

jamielaird

Here's my solution

Spoiler

challenge_157_solution_JL.yxmd

David-Carnes

Wish I had known then what I know now. I skipped over this in my first attempt at the Expert exam. Took me 9 minutes, start to finish, for the Challenge, and that included some reading of documentation.

Spoiler

David-Carnes_157.yxmd

LordNeilLord

Spoiler

challenge_157_LNL.yxmd

RWvanLeeuwen

Now this I like

Spoiler

Short answer: 15

Long Answer:

Spoiler

Is Gain Ratio the same as Gini increase? I think it is impossible that any rankings change with this different interpretation. Please do correct me if I'm wrong on that.

challenge_157_start_file - RWvL.yxmd

SeanAdams

This was a great one - thank you for the learning!

Thank you also to @SydneyF for the great explanatory articles that really helped.

Spoiler

First: Used forest model to get the variable importance plot. Much to learn here about what this plot is saying (thanks again @SydneyF

Then you need to compare two models that predict H0. Given that H0 is a binary variable, this looks like logistic regression, so this means that we can use the Nested Test tool to spot the difference between the two models

Which then gives the Chi-SQ difference caused by removal of F_38

challenge_157_SeanAdams_Solution.yxmd

TimothyManning

Spoiler

Definitely hard! I figured out with Google to use the forest model for the mean decrease Gini coefficient, but then wasn't sure which model to use and how to find the chi square comparison. After getting several R errors when using various tools and connecting things in the wrong way, I used @pjdit's spoiler to see it was the logistic regression and nested test tools, which I don't think I've used before.

157. Predictive.yxzp

TonyA

Solution below.

Spoiler

The first part was pretty simple since only a couple of tools have variable importance plots. But I went down a rat hole trying to do the model because I went with random forest and the results came up exactly the same for the 9 variable or 10 variable case. That's when I went back to my standby, logistic regression and saw some differences in the Model comparison tool. The chi-squared thing had me thrown for a bit. I did find the calculations and was ready to go that way, but stumbled across nested test, which made it a lot easier. My solution winds up being similar to the one provided, although I didn't verify the independence of the variables.

challenge_157_TonyA.yxmd

Quick Links

This months top contributors

hhaigh1 280

FláviaB 262

JORGE4900 235

caltang 177

DanM 167