nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

I have an extremely large .csv file , how do I change it to a a .yxmd file

ALexAn

Hello ,

I have an extremely large .csv file-750 Mb . I want to turn it into a .yxdb file so that the workflow does not crash and then work on it .

issues I face -

1 . my data gets truncated , if I give the normal file limit

2. the workflow crashes with an error - too many records in field 2

3. 1048575 exceeded , excel couldn't get the data in from the csv , when I attempted to pull it into an excel then into a .yxdb file

Can I get some suggestions on how to handle such large data that comes in as a csv

Input

Accepted answers

MarqueeCrew

@ALexAn ,

Bragging about your csv again? Shame on you.

Did you know that when you open a csv, there are configuration options that can cause issues for you once it starts running?

#7: Field Length

The default is 254. If you have a field that is longer than 254, you might be in trouble. The default needs to be increased to make sure that it doesn't truncate data or cause the closing quotes to be deleted.

#9: Ignore Delimiters in Quotes/Single Quotes/Auto/None

As for your excel issue, that surprises me. I don't know why the excel driver is being invoked. You might want to test by changing the name to .txt ?

I have before read the file in as a single field. I use \0 as the delimiter and uncheck #6 about first row being data.

Now you can use PARSE tools to get to the right columns and AUTOFIELD for the data types. Using a DYNAMIC RENAME I can use the first row of data to become the field names.

I'm happy to help further if you need it.

Cheers,

Mark

All comments

MarqueeCrew

@ALexAn ,

Bragging about your csv again? Shame on you.

Did you know that when you open a csv, there are configuration options that can cause issues for you once it starts running?

#7: Field Length

#9: Ignore Delimiters in Quotes/Single Quotes/Auto/None

As for your excel issue, that surprises me. I don't know why the excel driver is being invoked. You might want to test by changing the name to .txt ?

I have before read the file in as a single field. I use \0 as the delimiter and uncheck #6 about first row being data.

Now you can use PARSE tools to get to the right columns and AUTOFIELD for the data types. Using a DYNAMIC RENAME I can use the first row of data to become the field names.

I'm happy to help further if you need it.

Cheers,

Mark

JWarmbrodt

That was very helpful! Thanks for sharing your question. I've had something like this happen recently too.

ALexAn

😄 Thank you so much @MarqueeCrew .

Yes , the delimiter was the issue and I could not open the file to figure out what I should use.

But trial and error and the refresh in the configuration of the input tool saved me, I used \t as delimiter.

About why the excel didn't take the load , I am not sure , but the csv to .yxmd worked like magic and starting at line 2 gave me all the details I needed without much delimiting .Also I didn't have to change the size from 254.

And I could use the wildcard to read in 11 to 14 files of the same size and make it into one .yxmd. Saved me a lot of time .

thank you again.

alteryx.png

Quick Links

Popular Tags

This months top contributors

atcodedog05 19598

Qiu 15911

binu_acs 15783

MarqueeCrew 13710

apathetichell 13703