Upload csv to Bigquery with download tool

[paul_houghton]

Im trying to upload some csv files into google bigquery with the download tool (I couldn't get the output tool to upload with the simba drivers). 

I have successfully got the authentication sorted, but when i try to either upload a multipart POST option i get a '413 query too large error' while when I try to set up a resumable upload (that would probably be more useful) the connection times out when getting the initial resumable session.

Any ideas for what I'm doing wrong?

[JCR]

@GarthM & team. Any thoughts?

[NeilR]

Could you share your workflow (removing the data and auth credentials)?

[paul_houghton · Accepted solution]

So I managed to work out how to do this and the key issue I had was realising I needed to use the blob family of tools. (I went with the blob input and take the file from an temporary csv file).

I have published the tool on the gallery and would love more feedback on any errors that people might find so I can make it more robust.

https://gallery.alteryx.com/#!app/Big%2BQuery%2BTable%2BUpload/576ba6a9aa690a12b4e1279c

Thanks

[shreyanshrathod]

Hi @paul_houghton @NeilR 

Warm Greetings!!

Can you guide me into analysing best practices to follow while uploading data to BigQuery from Alteryx Designer ?

I am employing Google BigQuery Output tool for the same, but have little clue about best adjusting the configurations of it.

The problem is because of huge data size. My team member ran a job to upload data to BQ with a default batch size of 500 and after running for 13 hours, the job failed!!

There is very less documentation for Alteryx connectivity with BQ!

Any insights from your end will be really helpful.

Thanks in advance!!

Best Regards,

Shreyansh Rathod

[NeilR]

@shreyanshrathod I've never used the BigQuery Output tool but the documentation indicates it has an append mode so perhaps instead of writing out the full table you can break it up into parts and append each part.