Output multiple CSVs if data exceeds what CSV can handle (~1,000,000) rows

Question

Hi All,

I have a job that inputs Amazon Redshift data, which contains about 36,000,000+ rows and I'm trying to create an excel output that will create multiple workbooks in order to fit this data. Ex: Since ~36,000,000 rows of input data, this would require about 36 CSV workbooks for this data in Excel. How would I do this? I cannot post workflow since sensitive data.

Thanks!

rayJ · Answer

Here's my solution anyway:

1. Use two multi-row function to figure out how many record should be in one workbook and the workbook number

2. Then create the path just like @Emil_Kos mentioned

3. Finally, change the setting of output data function:

Output seperate files.yxmd

rayJ · Answer

@Emil_Kos , agreed, @JosephSerpis's answer is the way to go.

Trust me, I should be ashamed

I was preparing my solution too and didn't even get the chance to post.

Emil_Kos · Answer

Hi @JosephSerpis,

I wasn't aware of this functionality. Thank you for sharing.

It is a shame I spend so much time to prepare my post as your solution is much better 😀