This site uses different types of cookies, including analytics and functional cookies (its own and from other sites). To change your cookie settings or find out more, click here. If you continue browsing our website, you accept these cookies.
I have to read some values from an SQL table in a server so therefore I use an In-Database tool to read from it. There are millions of records in this table so takes at least 1 or 2 minutes just to load from the server.
Now I want to do many kinds of configurations to this data for e.g. calculations for new fields, creating new fields, joining, etc. It's very intensive and I am wondering if I should use the In-Database tools or Standard Workflow tools? Which will be faster in terms of processing it all?
the standard workflow tools will require the data to be extracted from the SQL Server and travel across the network to your machine, for you to then work in-memory with it, locally.
In general and without looking at your use case specifically, I'd say the In-DB tools will be way faster for you in this case because the data never actually leaves the data base and you are using the performance of the machine hosting the SQL Server, which presumably are higher than your local machine's ones.
If there are millions of rows of data your workflow will run faster if you can do as much data transformation before you stream out using the In-Database tools. An example I had it took 2 hours to stream out 3 million rows of data and do various data transformations compared to 15 minutes when I transformed the workflow to use all In-Database tools.