Comparing output from Nested Test tool

[WonderHog]

I ran the nested test tool on a logistic regression based on a 60% training sample using the output from the 2 LR models.  I concurrently ran the nested test tool against a complimentary 40% validation sample using the same LR model outputs (from the training sample).  I expected the outputs from the 2 nested test tools to be slightly different, but they were exactly the same.  I have done the same with several different combinations of input variables and always get identical outputs from the nested test tools.  Is there something I am doing wrong? 

The workflow is shown in the attached Word document.

Thanks.

[SydneyF · Accepted solution]

Hi @WonderHog!

The outputs your are seeing are as expected. The Nested Test Tool is used to compare if a model using more predictor variables is significantly better (p <=0.05) than a more parsimonious version of the same model (built using the same training data). The tool is directly comparing the two logistic regression models with a likelihood-ratio test. The training data included as an input in the Nested Test Tool is not used for that calculation. The reason that the tool asks for the training data as an input is to check that all the variables used in the models are in the training data set. Because you are comparing the same two models in both of your Nested Test Tools, it makes sense that the outputs are identical.

If you would like to compare how each of your models performs on a validation data set, I would recommend using the Score Tool to generate predicted values for the validation data set and then calculating a metric like Mean Square Error to compare the results. 

Here are some additional resources on the Nested Test Tool:

https://community.alteryx.com/t5/Advanced-Analytics/Nested-Test-Output/td-p/37268

http://www.public.iastate.edu/~alicia/stat328/Multiple%20regression%20-%20nested%20models.pdf