Macro for pulling data from SQL Server

Question

I am pulling large amount of data (41M rows) for an ML project. It currently takes over 41 mins for it to finish running. To optimize it, I converted in to a macro and only pull this data once. The workflow is simple and it works as expected. However when I pull this in to another workflow it is requires mapping  to other fields and I am not able to see good solution anywhere that explains what steps do I need to take to map these fields. I am just not following the logic because I dont have input that will have all these fields. Essentially, I need help to find a way to be able to pull 41M rows one time and hold it as a macro so it can be utilized in other workflows. If there is a better approach than Macros I would love to learn more about it. I would appreciate if someone can point me to solution thread that may have same issue addressed in the past. Thanks,

Macro converted workflow

Workflow where I would like to use the macro

barrymoe · Accepted Answer

Hi @TrivediH123,

I'm assuming you aren't asking for us to help you optimize the query run-time but you're asking about how to hold it in your workflow so you can troubleshoot.

Have you tried to just hold it in cache?

If you right click the input tool and run and cache, after the first run it will hold it for you in the background and it wont re-run the query every time you want to test something.

Let me know if that's what you mean, or you are asking for something esle.

Thanks!