Output multiple CSVs if data exceeds what CSV can handle (~1,000,000) rows

[skatofia]

Hi All,

I have a job that inputs Amazon Redshift data, which contains about 36,000,000+ rows and I'm trying to create an excel output that will create multiple workbooks in order to fit this data. Ex: Since ~36,000,000 rows of input data, this would require about 36 CSV workbooks for this data in Excel. How would I do this? I cannot post workflow since sensitive data.

Thanks!

[JosephSerpis]

Hi @skatofia in the Output Data Tool you can edit the max records in the configuration window. This will add a number to the file when it creates a new file when the max records are exceeded.

[Emil_Kos]

Hi @skatofia,

You should try multi-field binning tool.Actually it is a first time that I used it but I belive it will meet your needs. 

you can pick how many tiles you want to create:

and then you need use it in order to create a path destination:

Which you can later use in the output tool:

If this was helpful please mark my post as a solution!

Good luck!

[Emil_Kos]

Hi @JosephSerpis,

I wasn't aware of this functionality. Thank you for sharing.

It is a shame I spend so much time to prepare my post as your solution is much better 😀

[rayJ]

@Emil_Kos , agreed, @JosephSerpis's answer is the way to go.

Trust me, I should be ashamed

I was preparing my solution too and didn't even get the chance to post.

[rayJ]

Here's my solution anyway:

1. Use two multi-row function to figure out how many record should be in one workbook and the workbook number

2. Then create the path just like @Emil_Kos mentioned

3. Finally, change the setting of output data function: