Designer taking over 20 hours to retrieve 3rd party data via ODBC

[ThornhillTaylor]

Everyone, I have an ODBC connector that is pulling roughly 4 million lines of data there is no workflow yet, just the data pull.

This pull is taking upwards of 21 hours... what steps can I take to speed this up?

Computer RAM?

Computer memory? currently at 50% remaining after data pull is Started.

increased Server capability?  -- Do not have an Alteryx Server

Any help would be appreciated.

[Prometheus]

@ThornhillTaylor Have you tried in-database tools?

[ThornhillTaylor]

@Prometheus,  thanks for your comment. I tried the other day.. to no avail.. I am trying now, will update once the data has been pulled... currently at 3 minutes 

[ThornhillTaylor]

that seems to have worked it took 4 minutes for 250k rows

[CharlieS]

also try specifying "with (nolock"/uncommitted read in case it's waiting on other processes. 

[caltang]

Is this a recurring thing or a one off thing? The In-DB route would be a good pick as suggested by @Prometheus 

[ThornhillTaylor]

This is a recurring situation... sorry my delay but the In-DB tools were taking excessive amounts of time to run... in fact 3 connections were over 18 hours with the largest of the 3 being 350,000 rows...

Im considering other options now as the ODBC connector my be my issue....

Any idea on writing python scripts to retrieve the data??

[ThornhillTaylor]

this is coming from Shopify... just no experience trying to do this