Alteryx workflow not running due to big size

Question

Hello everyone.

I am joining 2 databases: 3.7 mill rows and 800k rows, respectively.

I am using the Fuzzy Match tool to join them due to different data in the joining field, and that takes a loong time. It's stuck at 51% and it doesn't seem like its going anywhere even after 2 hours. I have 16gb RAM only so that might be the issue.

Any ideas on how I can do this? Is there a way to run this workflow, a workaround maybe?

Thank you!

Amalainic · Answer

Thank you both for your input. I have cleaned the data before using the fuzzy match and also tried the AMP engine.

@NiclasH, it seems now that the Fuzzy Match tool throws the error "Invalid char in UTF-8 String" when the AMP engine is on. Any idea how I can solve that? I can't seem to find anything online.

NiclasH · Answer

Hey Amalainic,

you can also try turning on the Alteryx AMP Engine, which is designed to work with larger volumes of data at higher velocity and typically executes workflows more quickly, with efficient usage of your machine resources.

Just click onto the blank canvas, go into the "runtime" tab and tick "Use AMP Engine".

But as TheOC suggested, probably try to clean up the data as much as you can beforehand.

Good luck!

TheOC · Answer

hey @Amalainic

I would recommend bringing both datasets in, and doing as much cleaning up of the data as possible, potentially even dropping all columns you dont need prior to the fuzzy match, and then adding these back in afterwards, as i believe fuzzy match to be a very ram intensive process.

Also, i would possibly recommend changing this into a batch/iterative macro, and having it do this process in batches, to reduce the amount of ram necessary. In this case, it would only be matching say, 1000 rows at a time. You could also put a message tool within the macro, to say (for example) "Currently processing records 20000-21000), to help keep track of the progress.

I hope this helps!
TheOC