Encoding Error in R tool

[Toshi92]

I want to use R-tools for files in Japanese, encode is Shift-JIS.

I think I have to use encoding='cp932' parameter in R script.

But it doesn't work by my trials...

I attached my flow.

Would someone please check my workflow?

Thanks,

Toshi

[Toshi92]

Hi, @SydneyF

Awesome!

I can see correct data after changing encode.

How can I change the encoding of output file from UTF-8 into Shift-JIS?

Thanks,

Toshi,

[SydneyF]

Hi @Toshi92

I am so glad you are able to see the correct data now! 

From my research, it appears that there is not an option to write out in Shift-JIS from R. In the documentation on Encodings and R, Shift-JIS is included as an encoding that causes problems with R.

Encodings which are likely to cause problems include:

• Vietnamese (VISCII). This uses 186 characters including the control characters 0x02, 0x05, 0x06, 0x14, 0x19, 0x1e: the Windows GUI makes use of these as control characters.

• Big5, GBK, Shift-JIS. These are all 1- or 2-byte encodings including ASCII as 1-byte chars (except Shift-JIS replaces backspace by ¥) but whose second byte overlaps the ASCII range.

Please let me know if you have any additional questions I might be able to answer for you!

Thanks!

[Toshi92]

Really thanks @SydneyF

I got it.

I newly read outputtest.csv from Input tool, then I can change the encoding in Output tool.

Can I make above processing on the same flow?

I mean, I want to read outputtest.csv after completion of processing of R tool.

Should I use some special tools, such as Runner Macro? 

[SydneyF · Accepted solution]

Hi @Toshi92

You could develop a batch macro that reads in an input, and place the batch macro after a block until done tool, attached to the R tool. You would also need to add a Formula Tool to update the filepath to where the UTF-8 file has been written out

The batch macro would be very simple, just a control parameter attached to an Input Tool, configured to update the file path.

Read Input Batch Macro

Action Tool Configuration

Your workflow would look like this:

Workflow with added input/output functionality

The blank blue tool is the new batch macro, and the formula tool is configured to create a new field with your filepath, which is then used as the Control Parameter for your macro.

Formula Tool ConfigurationBatch macro configuration

In order for this to work, you will also need to add the following code at the end of the script in your R tool:

write.Alteryx("output", 1)

All this code does is creates a dummy output for the R Tool, which then allows the downstream tools to run once the R code is finished executing. 

Does this all make sense? Please let me know!

Thank you!

[Toshi92]

Thanks @SydneyF

It works!

I really appreciate your help, you are my superhero!

I attached final workflow for other Alteryx users in Japan.

Regards,

Toshi

[Toshi92]

This is batch macro for above workflow.

[SydneyF]

Hi @Toshi92

Just so you know, you should be able to use the same strategy to modify almost any R-based Tool in Alteryx to handle Japanese characters,  I would just make sure to save your modifications to a different location than the original tools, as they will no longer be supported by Alteryx Support.

You will just need to change the read.Alteryx argument in the code of the R-Tool inside the Tool workflow to read_csv, and the output to write_csv. The library(readr) command needs to be included to load the package, and setwd() should be set to define the location the inputs and outputs are read from and written to.

# ADD TO START OF CODE
# Set your working directory at the start of the code
# Use forward slashes (/) or double back slashes (\\) in your filepath
setwd("C:/YOURWorkingDirectory/UseFrontSlashes/OrDoubleBackSlashes")

# if package is already installed on machine, all you need to do is load the package in is the library function
library("readr")

# replace read.Alteryx code with read_csv, pointing to your written out input data

#X <- as.matrix(read.Alteryx.First("#1", %Question.chunk.size%))
X <- as.matrix(read_csv("yourdata.csv"), locale = locale(encoding = "UTF-8")) #you can change UTF-8 to Shift-JIS to read-in Shift-JIS encoding

--------------------------------------------------------------------------------------------------------------------------------------------------------------------
R TOOL CODE
--------------------------------------------------------------------------------------------------------------------------------------------------------------------

# ADD TO END OF CODE
# replace write.Alteryx with write_csv, the output will be written with UTF-8 encoding

#write.Alteryx(as.data.frame(the_matrix), 1, TRUE)
write_csv(as.data.frame(the_matrix), "output.csv")

# add dummy output for downstream processing
write.Alteryx("test", 1)

For larger data sets, a more elegant solution than the Block Until Done Tool before the input will likely need to be developed, to ensure the output finishes writing before the R-Tool attempts to read it in.

To install readr, you can either use the install R packages analytic application, or with the following code

# helper function to install R packages. source_location optional
install_package <- function(package_name, source_location = FALSE) {
	# grab the alteryx repo
	altx.repo <- getOption("repos")
	# set your primary repo if you haven't already
	altx.repo["CRAN"] <- "http://cran.rstudio.com"  
	options(repos = altx.repo)
	#check to see if the package we want is already installed
	# if it is, we will just load it, otherwise we will install it
	if(package_name %in% rownames(installed.packages())==FALSE){ 
		if(source_location==FALSE) {
			install.packages(package_name)
		} else {
			install.packages(source_location, repos = NULL, type = "source")
		}
	}
	require(package_name, character.only=TRUE)
}

##install readr present
install_package("readr")

I hope this has all been helpful! I've had a lot of fun working on it :)

[Toshi92]

Thanks @SydneyF

I will try to adjust into other R tools!

Regards,

Toshi

[voovovo]

@SydneyF

I think it would be better to use "Dynamic Input" with using R tool output for file path or name in R tool(only 1 file path for each output line) instead of dummy output, so you can control the timing of input.

[Toshi92]

Hi, @voovovo

It works!

Your method doesn't need Batch Macro, and I can use "relative path".

Really thanks!

Toshi