Alteryx Designer Discussions

Find answers, ask questions, and share expertise about Alteryx Designer.

Append to YXDB

pcatterson
11 - Bolide

So I have a file that is delivered by someone on another team creates.  It always has the same name and is always delivered to the same directory location.  It is typically updated every workday but this time of the update is always changing. 

 

I want to pull it in whenever it is updated and aggregated together to become a historical file of all the data delivered each day.  I can run a workflow to pull it in and I'm able to schedule this to run.  Ideally I want the output to be a .YXDB file.

 

The problem I'm having is that although I can schedule it to pull hourly, this causing tons of duplicates to be pulled in from the file which doesn't change daily.  If instead I made the scheduled workflow run hourly, then it could miss a file if wasn't updated until after the time of the scheduled run.

 

Is there a good way to append files based on the LastWriteTime attribute of the file?  I have thought about joining against all the fields in the file, but I'd imagine that join would get very slow as the data got larger.

5 REPLIES 5
patrick_digan
16 - Nebula
16 - Nebula

@pcatterson What about if you used the directory tool which would give you info last last modified time?

danilang
18 - Pollux
18 - Pollux

Hi @pcatterson 

 

You could use a technique similar to this one, based on keeping track(externally, .yxdb, SQL, etc)  of the LastWriteTime of the file at the end of your workflow

 

1.  Read in the LastWriteTime from the external source

2.  Start with a Directory tool followed by a Filter to only pass files newer than the time in step one.

3.  Send this into a dynamic input and process normally

4.  Output the last write time. 

 

You can run this every hour and the file will only be processed when it changes.

 

Dan

pcatterson
11 - Bolide

Yes, that is exactly what I'm trying to do.  I'm trying to use the Directory and the Dynamic Input tools together to bring in only new records.  My trouble is that the records don't have their meta-information besides the FullPath after they've passed through the Dynamic Input tool.

pcatterson
11 - Bolide

I would like to avoid having to create a second file to keep track of the updates.

danilang
18 - Pollux
18 - Pollux

Hi @pcatterson 

 

Append the Last run time as a column in the main output data .yxdb and use this as the basis for LastWriteTime comparison

 

Dan

Labels