Split large input files

[TripleJJJ]

Hi there,

I'm trying to solve for generating a flow that a user can browse to a file, and split it out to multiple files based on a volume size decided by the user.  Example, input file has 100 records, so they browse to a file, and let's say they want to split it out into 5 files that are 20 records each.

You can see in my screenshot, I can easily hard code this by using a Record ID but how do I make it dynamic so the user picks the size of the file outputs.

Much appreciated!

[apathetichell]

numeric up/down

action tool

formula tool

create_new_field - bin

ceil([RecordID]/10)

configure the action tool to replace a specific string - replace 10.

next formula tool. - create your filename as [engineworkflowdirectory]+"myfile":+tostring([bin])+(your extension) add a select tool to drop bin.

add an output data tool - scroll down to the bottom - change entire path to your filename.

note - you can also do this using multi-field binning in a batch macro. The only reason why that is appealing is because I have never used multi-field binning in a workflow before and have previously stated that it is the single most useless tool in the tool pallette.

[TripleJJJ]

Thanks for above, I'm not familiar with using the numeric up/down function but it looks like it max's out at 100.  I'll need something much larger as the files may be 3 million records for example so I would want to chunk those into 1million buckets.

If it's not to much trouble can you upload a mock graph with some dummy data showing me your suggestions to get me going?

Appreciate the help!!

[apathetichell]

That's controlled by the configuration of the numeric updown. Here's a version with a 1,000,000 row default amount.

[TripleJJJ]

Thank you!  in your example there is only 1 input row so I would expect it to only have one output file.  However it just keeps outputting file after file until I cancel the app.  Regardless if I change the number up/down to 1 when I run it, it just keeps outputting files.

If I add let's say 20 rows to your input, and run it and change the # to 5 I would expect 4 .csv files on output.

I really appreciate the help on this one!!

[apathetichell · Accepted solution]

Run this as an app using the wand. If you do not run this as an app - the default number will not change.

this is set up with 10 as the bin number. I have generate rows set up for 3,000,000 rows. You will create 300,000 files. My sample data is one line and a generate rows. Disconnect those tools and use your real data.

[TripleJJJ]

Thank you!  I had to replace your number up/down with just the autoID function and that worked like a charm.  Otherwise it just kept writing additional files, and hanging.  You got me 99% of the way there.

Again, thanks kindly for the help.  

Attached is the updated flow that is working.