Incremental Load concept for Alteryx workflows

[Pankhudri20]

Hello,

I have workflows which will be run daily.

It essentially will load data upto 4 months in past.

However, it takes too long for the workflows to load since data files are huge.

Is there a way that to perform incremental load in alteryx?

Such that every day the workflows append the results to existing output files?

Thank you

Looking forward to a response soon!

Regards,

Pankhudri

[CharlieS]

Hi @Pankhudri20 

What type of files are they? This kind of process is daily stuff with databases where you can specify where clauses, but with files the only way that comes to mind would be using Calgary files. Calgary files with an index on the data values would allow you to only return the records with the incremental date values you're interested in. 

[Pankhudri20]

Hi @CharlieS ,

Thank you for your response.

I have daily excel files added to the shared folder.

These files are read in the workflows and takes a lot of time to load since the data volume is huge.

What are calgary files?

Regards,

Pankhudri

[CharlieS]

The daily file is that large, or the entire directory is and you only want to open the last day? If you only want to open a subset of the Excel files in the directory, that can be achieved using a Directory and Dynamic Input tool (or batch macro). 

[Kaustubh17]

Hi @Pankhudri20 

Welcome to alteryx, i will suggest you use only those files which are latest i mean dynamic files that will reduce the workload as @CharlieS suggest he is right we can achieve this task by using dynamic tool and directory tool. for your reference i am attaching one link that might help you. 

https://community.alteryx.com/t5/Alteryx-Designer-Knowledge-Base/How-to-dynamically-run-the-most-rec...

I hope this article resolve your issue.

Thanks,

Kaustubh

[Pankhudri20]

Hi @CharlieS ,

Yes, I am reading the files using directory.

But if I read only a subset of files, will the whole output file be over written or new data will be appended?

Thank you

Regards,

Pankhudri

[Kaustubh17]

No @Pankhudri20  append will not happen try at your end and check is it helpfull for you. In case you have further questions you can get back to me 

Thanks,

Kaustubh

[CharlieS]

---

@Pankhudri20 wrote:

Hi @CharlieS ,

Yes, I am reading the files using directory.

But if I read only a subset of files, will the whole output file be over written or new data will be appended?

---

Sounds like you've got the marginal/partial input part down

Outputting only the new records depends on what your output target is. Some support appending like a database, and others (.yxdb) do not support appending data and would have to be entirely re-written.

[Pankhudri20]

Hi, @Kaustubh17 ,

Thank you for your response.

I am unable to find a solution for this.

[Pankhudri20]

Hi @CharlieS ,

Yes, I want to append the data in the output files.

Or is there a way that everyday I copy the data in another file and append the latest output to that file?

That could be an indirect solution for this ?

But I dont know if I can store the output data in another file and append new output everyday?

Looking forward to your response !

Thank you

Regards,

Pankhudri