Incremental Load concept for Alteryx workflows

What type of files are they? This kind of process is daily stuff with databases where you can specify where clauses, but with files the only way that comes to mind would be using Calgary files. Calgary files with an index on the data values would allow you to only return the records with the incremental date values you're interested in.

Pankhudri20

Hi @CharlieS ,

Thank you for your response.

I have daily excel files added to the shared folder.

These files are read in the workflows and takes a lot of time to load since the data volume is huge.

What are calgary files?

Regards,

Pankhudri

CharlieS

The daily file is that large, or the entire directory is and you only want to open the last day? If you only want to open a subset of the Excel files in the directory, that can be achieved using a Directory and Dynamic Input tool (or batch macro).

Kaustubh17

Hi @Pankhudri20

Welcome to alteryx, i will suggest you use only those files which are latest i mean dynamic files that will reduce the workload as @CharlieS suggest he is right we can achieve this task by using dynamic tool and directory tool. for your reference i am attaching one link that might help you.

https://community.alteryx.com/t5/Alteryx-Designer-Knowledge-Base/How-to-dynamically-run-the-most-recent-file-in-a-file-folder/ta-p/24204

I hope this article resolve your issue.

Thanks,

Kaustubh

Pankhudri20

Hi @CharlieS ,

Yes, I am reading the files using directory.

But if I read only a subset of files, will the whole output file be over written or new data will be appended?

Thank you

Regards,

Pankhudri

Kaustubh17

No @Pankhudri20 append will not happen try at your end and check is it helpfull for you. In case you have further questions you can get back to me

Thanks,

Kaustubh

CharlieS

@Pankhudri20 wrote:
Hi @CharlieS ,
Yes, I am reading the files using directory.
But if I read only a subset of files, will the whole output file be over written or new data will be appended?

Sounds like you've got the marginal/partial input part down

Outputting only the new records depends on what your output target is. Some support appending like a database, and others (.yxdb) do not support appending data and would have to be entirely re-written.

Pankhudri20

Hi, @Kaustubh17 ,

Thank you for your response.

I am unable to find a solution for this.

jrgo

@Pankhudri20

A setup like the below is probably how you need to set it up. The target file is rewritten every time, but not before it's first read in and union'd with the new records.

Pankhudri20

Hi @jrgo ,

Thank you so much for your quick and detailed response.

I have a doubt here:

Can I read past 2 weeks of data every time and append and replace data for dates which are already in output file?

So essentially the new data will be 1 file and the rest 13 days will be re processed. This is because on some days the data file could be missed or uploaded late or the person is on vacation who uploads. So, I want to avoid errors due to missing data.

Can you please tell me how can I handle this situation ?

Looking forward to your response soon!

Regards,

Pankhudri

Quick Links

This months top contributors

FláviaB 262

JORGE4900 235

caltang 177

DanM 167

IraWatt 157