nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

Split large input files

TripleJJJ

Hi there,

I'm trying to solve for generating a flow that a user can browse to a file, and split it out to multiple files based on a volume size decided by the user. Example, input file has 100 records, so they browse to a file, and let's say they want to split it out into 5 files that are 20 records each.

You can see in my screenshot, I can easily hard code this by using a Record ID but how do I make it dynamic so the user picks the size of the file outputs.

Much appreciated!

App Builder

Accepted answers

apathetichell

Run this as an app using the wand. If you do not run this as an app - the default number will not change.

this is set up with 10 as the bin number. I have generate rows set up for 3,000,000 rows. You will create 300,000 files. My sample data is one line and a generate rows. Disconnect those tools and use your real data.

All comments

apathetichell

numeric up/down

action tool

formula tool

create_new_field - bin

ceil([RecordID]/10)

configure the action tool to replace a specific string - replace 10.

next formula tool. - create your filename as [engineworkflowdirectory]+"myfile":+tostring([bin])+(your extension) add a select tool to drop bin.

add an output data tool - scroll down to the bottom - change entire path to your filename.

note - you can also do this using multi-field binning in a batch macro. The only reason why that is appealing is because I have never used multi-field binning in a workflow before and have previously stated that it is the single most useless tool in the tool pallette.

TripleJJJ

Thanks for above, I'm not familiar with using the numeric up/down function but it looks like it max's out at 100. I'll need something much larger as the files may be 3 million records for example so I would want to chunk those into 1million buckets.

If it's not to much trouble can you upload a mock graph with some dummy data showing me your suggestions to get me going?

Appreciate the help!!

apathetichell

That's controlled by the configuration of the numeric updown. Here's a version with a 1,000,000 row default amount.

divide rows test.yxwz

TripleJJJ

Thank you! in your example there is only 1 input row so I would expect it to only have one output file. However it just keeps outputting file after file until I cancel the app. Regardless if I change the number up/down to 1 when I run it, it just keeps outputting files.

If I add let's say 20 rows to your input, and run it and change the # to 5 I would expect 4 .csv files on output.

I really appreciate the help on this one!!

apathetichell

Run this as an app using the wand. If you do not run this as an app - the default number will not change.

TripleJJJ

Thank you! I had to replace your number up/down with just the autoID function and that worked like a charm. Otherwise it just kept writing additional files, and hanging. You got me 99% of the way there.

Again, thanks kindly for the help.

Attached is the updated flow that is working.

divide rows test.yxwz

Quick Links

Popular Tags

This months top contributors

atcodedog05 19598

Qiu 15910

binu_acs 15783

MarqueeCrew 13710

apathetichell 13703