Time Series Model Score MAPE vs Forecast MAPE very different

[Billigans]

Hello Alteryx Community, 

I need help understanding why the MAPE for my ARIMA forecast is so much higher than the MAPE in the TS Compare model tool using the same ARIMA tool. 

I am trying to forecast revenue(y) by day using units sold(cv) as a covariate.  In both the TS Model and the TS Forecast I use the same training data and covariate data but the MAPE jumps from 22% to 50% and I cannot understand why for the life of me.  

I have attached the workflow and data as a packaged workbook. Any help is greatly appreciated. Thank you so much. 

[joshuaburkhow]

Hey @Billigans 

Take a look at some of the modifications I made here for you.

a) I did change a bit of the layout so you don't have to do any of the MAPE calcs as the tools can do those for you 😉

b) I also took a look at your data and found that you will benefit from tuning your model to use the box-cox transformation. 

c) overall the message is much the same (your covariates have a decent impact) but with a better model.

Let me know if this helps! 

Joshua

[Billigans]

@joshuaburkhow,

Thank you for taking the time to look at my model. I really appreciate it. 

As a point of clarification, I was using the manual calculations to calculate the MAPE of the output of the forecast because I needed to use the forecasted values of units(cv) as the covariate, not the actual unit sales themselves. I won't have the future period unit sales when I'm doing the actual forecast.

I am trying to understand why the ARIMA tool, which doesn't use the future periods, when fed into the TS Model Compare tool creates such a good forecast, but when I feed the information to a TS Forecast tool the forecast flatlines. 

I have attached a simplified workflow to help show the problem if you look at the forecast in the TS Model vs the Forecast from the TS Forecast tool you can see a major difference. 

Thanks again for your help.

[RolandSchubert]

Hi @Billigans ,

the difference seems to be the data for the covariate. The same model is used for both the TS Covariate Forecast tool and the TS Compare tool, but the data for the covariate is different. Covariate data for the TS Covariate Forecast tool is generated using a TS Arima + TS Forecast tool resulting in a widely flat line ("O" anchor of TS Forecast tool) thereby leading to a flatline for the forecast. Covariate data for the TS Compare tool is the "original" test data (records 1540-1600) - these data vary widely and the result is a varying forecast for the target value.

I've modified the workflow and added a second TS Covariate Forecast tool, using the same model as the first one, but the covariate data used for the TS Compare tool. In my opinion, forecast should be closer to what you expect. Hope this is helpful.

Best,

Roland

[Billigans]

@RolandSchubert 

Thank you for taking a look at my workflow. 

I believe I understand what is happening now. The future period's cv data was passed to the model in the TS Model tool. 

My question then becomes, how can I predict y using cv as a covariate without knowing the future periods?

[RolandSchubert · Accepted solution]

You have to create a prediction for cv first (as you did) and use the predicted value in forecast calculate (you also did that). But it sees, ARIMA (or at least ARIMA in the configuration you used) does not produce a reliable prediction for cv. I see basically two options now. You can use ARIMA and customize the model manually (e.g. modify parameters for seasonal and non-seasonal components) or you try ETS for cv and feed the results to your ARIMA model. What do you think?