Is Data Stored whilst using In-DB Tools on Server?

caltang

Hi all, this is my current process:

Current Flow:

Step 1: Call data from Google Big Query.

Step 2: Stream data out so that we can use the standard Alteryx tools.

Step 3: ETL with standard Alteryx tools.

Step 4: Once ETL complete, we will stream data back to Google Big Query.

Step 5: Write to a table that stores our outputs.

From there, our PowerBI will connect to the outputs generated by BigQuery to be visualized. 

Questions:

• Question 1

Refer to the following statement:

"If a user uses in-db tools or connects to DBs directly, the data will not pass through the Server location instance as it’ll be processed in-situ without data leaving the database. Even if data is streamed out in the workflow, it will deal direct between the source DB and the Worker."

Does this mean that if we use the In-DB tools for our workflows and when uploaded to the Server, data will not be stored wheresover on the MongoDB/Server- is this correct?

• Question 2

If we stream the data out from In-DB tools as from Step 2 onwards, will any data from the ETL process be stored on the Server or MongoDB?

• General Question

My ask is this: Does the Server or MongoDB store any data whatsoever when we use In-DB tools for all the steps listed above? 

If it creates temp files that are purged after a workflow is done - that is fine. If the temp files or input/outputs are stored for a day or more, then that's not what I want. Additionally, if data streams in fully, does that mean Alteryx + Server/MongoDB downloads the data from the DB? That's a big no-no.

FAQs:

• Why?

Our reasons for doing so is that the ETL process is much easier on Alteryx, easier to maintain, easier to visualize, and easier to provide documentation and handover work. 

• Where will this be used?

We will put this method on the Alteryx Server.

Thank you!

jNunez

Question 1: No, the data would only be saved to MongoDB if you have an output file that's writing to a non-network location. Ex: result_1.yxdb vs something like \\server\filepath\output.yxdb. The first would save data to mongoDB and the results would be viewable if you select workflow results when viewing it in the gallery. The second would just save a file to the location.

Question 2: Data will only be saved in some persistent manner, if you have an output tool of some sort. Otherwise, the only thing being logged to MongoDB are the info messages you normally get from a workflow running.