Cached Data prior to running workflow

Hi all,

I run a workflow that contains two very large datasets. One of which I receive a few weeks before the other. However, when I receive the second dataset (which is about 1/8th the size of the first dataset) I need to complete the workflow pretty quickly. I takes quite a bit of time to run the workflow when I have to read all of the data into the workflow. I was wondering, the day before, can I cache the first source of data up to the point right before it interacts with the second source of data? Then, once I get the second source I can run the workflow. Since the first dataset is already cached I have essentially saved myself that amount of time. Does it work like this? Please let me know if you have any questions.

EDIT: It takes a few hours to read in the large dataset so my intuition tells me that if I cache this data and the manipulations I need to perform beforehand then I will be saved all that time when I actually run the full workflow. I just am not sure if I leave Alteryx open for a day with all that info cached will that cause any unexpected issues.

Input

Tips and Tricks

Accepted answers

All comments

TomWelgemoed

Hi,

I don't think what you're asking is possible. I would rather look at a few things to speed up your process. Example areas are:

Can you store the data set (even better, an aggregated version of the dataset) to .yxdb format
Can you use In-database tools?
Can you store less data - only what you need - in a database/.yxdb after the extraction?
Look at why the extract is taking so long? If it's from a database, have you done all joins & filters in the extract itself?

Those are some of the immediate points where I would start to look.

Regards,

Tom

AlteryxTrev

Can I not just do as I have done in the picture below? I have cached my workflow in three spots (circled in black). As you can see these are just additional data sources that do not interact with my initial data source immediately. Then once I get my last input file (The top input file that is not circled) I place it in there and run. The black portions were already ran and cached previously so it doesn't need to perform those steps again, it only needed to bring in the cached data.

To answer your questions:

Can you store the data set (even better, an aggregated version of the dataset) to .yxdb format
- I am not exactly sure what you mean by this.
Can you use In-database tools?
- I am not sure what you mean but this either
Can you store less data - only what you need - in a database/.yxdb after the extraction?
- I have to bring in all the data because I am performing matches at each stage and at the end I need to return the full main file with appended fields. I am basically doing a vlookup to find a match and bring over a field from the circled inputs into the not-circled input.
Look at why the extract is taking so long? If it's from a database, have you done all joins & filters in the extract itself?
- I am not sure what you mean by 'extract'. But the reason why this is taking so long is due to sheer size of the data files. Loading the files take a few hours and then the details within each macro takes the majority of the remainder of time (joins and fuzzy matches).

Quick Links

This months top contributors

atcodedog05 19458

Qiu 15866

binu_acs 15708

MarqueeCrew 13708

apathetichell 13703