Data Optimization | Data import Speed

[kw]

Hello Alteryx Team,

I am importing data using the OBDC connector. I have selected to only bring in columns needed, however it take 3 hrs to import due to the data size. I do not need all the data rows and would like to cross reference with an excel file.

Is there a way to speed up the import by only bringing in the ROWS that match my excel document? How can I speed up the import process?

Thanks

[Joe_Mako]

Can you use the In-Database tools?

If so, you may be able to use the Data Stream In tool to create a temp table in your database and then the Join In-DB tool.

[kw]

yes, to a certain extent. Crazy question: but is there a way to use the in data base tools without having "create table access" for the database?I want to stream in my excel data to do a join without having to create a table to the database.

[HenrietteH · Accepted solution]

Hi @kw

The short answer is "no", Alteryx can't load the data to the database without writing it somewhere. However, you can select the "temporary table" option in the data stream in tool and the temporary table will be removed as soon as Alteryx is done with it. 

If that doesn't work, another option might be the dynamic input tool. You can use your excel sheet to update the sql query in the dynamic input tool as explained here. 

The only drawback would be that this would execute 3000 queries against your database.

[kw]

Hello Alteryx team,

I used the solution above to stream in data. I have a data set of 62K rows that I streamed in-DB to join with other records. I did a left outer join, therefore when I stream the data out, it should only contain 62K records I brought in with the new fileds merged. However, it is streaming over 13M rows of data out.

I'm not sure why and was hoping you could assist me in things to look for to limit the data I want to stream out.

Thanks,

KW

[HenrietteH]

Hi @kw, that sounds like your join is creating duplicates. I would suggest looking for duplicates in the join field(s) by uniquing the data.