data migration using alteryx

[rahul1695]

can i use alteryx to copy data from big query to teradata, i have been data proc jobs to do this but many issues are coming for the same

is there a way i can do that usign alteryx or any method (any suggestions is appreciated) the problems is the volume of data is very high around 200 millions records i have decided to divide the data by source

[OTrieger]

@rahul1695 

have you checked this?
https://help.alteryx.com/current/en/designer/data-sources/teradata.html 

[apathetichell]

what's your timeframe? is it a one off 200mm records - or is this daily/hourly? one off. Sure. I'd batch this by 10-20 mm records and then run. Depending upon your memory and network connections - 200mm is a ton. Also - to confirm - you are going from BQ? If so can you migrate it first to a GCS bucket? probably faster.

[rahul1695]

its one table which has 200 million records
yes its from BQ to teradata
could you please explain the gcs thing u mentioned a bit in detail

[rahul1695]

its one table which has 200 million records
yes its from BQ to teradata
could you please explain the gcs thing u mentioned a bit in detail

[rahul1695]

@apathetichell 
its one table which has 200 million records
yes its from BQ to teradata
could you please explain the gcs thing u mentioned a bit in detail

[apathetichell · Accepted solution]

GCS is the object storage for GCP. You can export your BQ table in raw files (say CSV) into GCS with a partition (say every 15mm rows) you can then parse these chunks in Alteryx. I would do this because:

1) you are dividing up your table via partition in BQ

2) you are transferring objects (files) vs query data -> this is optimized.

3) you are able to run your Alteryx workflow piecemeal by files (ie one part breaks -> that's fine - just rerun).

Not saying I'm convinced that Alteryx is the best tool here - but assuming you have your Terradata connection already - and assuming you have the licenses you definitely CAN do what you are looking to do via Alteryx.