Use directory tool to pull files based on creation date?

Hello,

I would like to have the directory tool pull just the file data for the files created today (dynamically). I know I can read the folder and pass through a filter, but the folder has a truly obscene number of files in it, so pulling all that data takes a very long time. I could have the WF run in a fraction of the time if I could specify the creation date.

And no, I can't use file name because the file name does not contain the date.

Alternatively, I COULD look for files based on a dynamic list of a portion of the file name.... if there's a way to do that.

But creation date would be best.

Input

Accepted answers

All comments

alexnajm

How many files are there? It should be only pulling metadata from those files, so I'd be surprised if it was taking more than several seconds... because you described exactly what I'd do - use the creation date in a filter after the directory tool. You also already noted the file specification wouldn't work.

Have you turned on Performance Profiling? That might help determine your workflow's bottlenecks - it's right under the Runtime tab!

TimN

I see this:

https://community.alteryx.com/t5/Alteryx-Designer-Desktop-Discussions/How-to-get-results-of-DIR-command-into-AlterYX/td-p/16332

I don't think it's exactly what you need but you might be able to use this approach from @jdunkerley79

DaveSchloat

There's a few hundred thousand files. I'll see if I can find the performance profile, but I really think it's just because there are so many files.

alexnajm

Sure, go ahead and try the Performance Profile! I just tried a Directory tool on one of our large folders - 97,000 records in 4.6 seconds. This is why I am surprised it's taking so long on your end - I wouldn't imagine it taking longer than 20-30 seconds.

So then my guess is that it's likely a memory or network issue - is that possible? I agree that my second thought was that it'd have to be some sort of run command, which it looks like @TimN provided a possible solution. I don't have the Run Command knowledge to help form something proper, but hopefully it helps.

DaveSchloat

Thanks Alex - I'm trying that run command. It's definitely possible that it's a network issue. The folder is a network folder and "lives" on the other side of the country from me. But the directory command pulled down the file names in 90 seconds. That's very not bad!

alexnajm

Great news!

TimN

Hi,

If the Run command solution works, please mark as the solution.

Thank You.

DaveSchloat

If I can figure out how to use the file name to go back to the folder and get the file creation date, then I certainly will.

DaveSchloat

Interesting. It seems the Run command solution only can function with /B (file name only) in the output. I tried to be slick and change it to /T:C to get the creation timestamp, too, and the command tells me there are too many fields on row 9. Trying to figure that part out....but I'm not a DOS guy.

DaveSchloat

Yes. I'm really hopeful that I get granted read access to a table that has all this stuff, AND a code I can filter off of. But I think I have to put up my first born as collateral for that. Eh, might be worth it. LOL! Have a great holiday, and thanks to everyone for their input!

Quick Links

This months top contributors

mceleavey 383

mbarone 337

Hollingsworth 335

LanisC 335

JeffF 335