How can you double check that a CSV input file is a UTF8 CSV file?

Question

Does anyone know how to check in the workflow that a CSV input file is a CSV UTF8 file format?

I need to detect this and raise a warning that special characters may have been lost from one or more input files generated by various parties.

Many thanks

F_Latulipe · Answer

Thank you both, the environment we use is Microsoft based, so no native Unix commands for me.  But I will experiment with the Python chardet module.

ArnaldoSandoval · Answer

Hi @F_Latulipe

I searched for a command line enconding checker finding these options:

* file.exe is a native unix console command; are you using unix?
* I found Encoding Checker at GitHub, I downloaded and tested on my laptop, below its output:

* This utility is a promising option.
* Unfortunatelly, it does not allow to write its output to disk, so we could use it in Alteryx
* The good news is the source code is available in GitHub.

Questions:

Do you have access to developers with CSharp (C#) knowledge? having the source code, this utility could be enhanced in different ways, like creating a console version or adding an ouput option.

Let me know

hth

Arnaldo

philipmannering · Answer

I don't think there's a way you can know for sure. You chould try using Python's chardet module. See example attached,

Check File Encoding.yxzp