I have an extremely large .csv file , how do I change it to a a .yxmd file

[ALexAn]

Hello ,

I have an extremely large .csv file-750 Mb . I want to turn it into a .yxdb file so that the workflow does not crash and then work on it .

issues I face -

1 . my data gets truncated , if I give the normal file limit 

2. the workflow crashes with an error - too many records in field 2

3. 1048575 exceeded , excel couldn't get the data in from the csv , when I attempted to pull it into an excel then into a .yxdb file 

Can I get some suggestions on how to handle such large data that comes in as a csv

[MarqueeCrew · Accepted solution]

@ALexAn ,

Bragging about your csv again?  Shame on you.

Did you know that when you open a csv, there are configuration options that can cause issues for you once it starts running? 

#7:  Field Length

The default is 254.  If you have a field that is longer than 254, you might be in trouble.  The default needs to be increased to make sure that it doesn't truncate data or cause the closing quotes to be deleted.

#9:  Ignore Delimiters in Quotes/Single Quotes/Auto/None

As for your excel issue, that surprises me.  I don't know why the excel driver is being invoked.  You might want to test by changing the name to .txt ?

I have before read the file in as a single field.  I use \0 as the delimiter and uncheck #6 about first row being data.

Now you can use PARSE tools to get to the right columns and AUTOFIELD for the data types.  Using a DYNAMIC RENAME I can use the first row of data to become the field names.

I'm happy to help further if you need it.

Cheers,

Mark

[JWarmbrodt]

That was very helpful! Thanks for sharing your question. I've had something like this happen recently too. 

[ALexAn]

😄 Thank you so much @MarqueeCrew  .

Yes , the delimiter was the issue and I could not open the file to figure out what I should use.

But trial and error and the refresh in the configuration of the input tool saved me, I used \t as delimiter.  

About why the excel didn't take the load , I am not sure , but the csv to .yxmd worked like magic and starting at line 2 gave me all the details I needed without much delimiting .Also I didn't have to change the size from 254.

And I could use the wildcard to read in 11 to 14 files of the same size and make it into one .yxmd. Saved me a lot of time .

thank you again.