Is Data Stored whilst using In-DB Tools on Server?

[caltang]

Hi all, this is my current process:

Current Flow:

Step 1: Call data from Google Big Query.

Step 2: Stream data out so that we can use the standard Alteryx tools.

Step 3: ETL with standard Alteryx tools.

Step 4: Once ETL complete, we will stream data back to Google Big Query.

Step 5: Write to a table that stores our outputs.

From there, our PowerBI will connect to the outputs generated by BigQuery to be visualized. 

Questions:

• Question 1

Refer to the following statement:

"If a user uses in-db tools or connects to DBs directly, the data will not pass through the Server location instance as it’ll be processed in-situ without data leaving the database. Even if data is streamed out in the workflow, it will deal direct between the source DB and the Worker."

Does this mean that if we use the In-DB tools for our workflows and when uploaded to the Server, data will not be stored wheresover on the MongoDB/Server- is this correct?

• Question 2

If we stream the data out from In-DB tools as from Step 2 onwards, will any data from the ETL process be stored on the Server or MongoDB?

• General Question

My ask is this: Does the Server or MongoDB store any data whatsoever when we use In-DB tools for all the steps listed above? 

If it creates temp files that are purged after a workflow is done - that is fine. If the temp files or input/outputs are stored for a day or more, then that's not what I want. Additionally, if data streams in fully, does that mean Alteryx + Server/MongoDB downloads the data from the DB? That's a big no-no.

FAQs:

• Why?

Our reasons for doing so is that the ETL process is much easier on Alteryx, easier to maintain, easier to visualize, and easier to provide documentation and handover work. 

• Where will this be used?

We will put this method on the Alteryx Server.

Thank you!

[caltang]

@fmvizcaino @jrgo 

Your inputs are much appreciated here!

[jNunez · Accepted solution]

Question 1: No, the data would only be saved to MongoDB if you have an output file that's writing to a non-network location. Ex: result_1.yxdb vs something like \\server\filepath\output.yxdb. The first would save data to mongoDB and the results would be viewable if you select workflow results when viewing it in the gallery. The second would just save a file to the location.

Question 2: Data will only be saved in some persistent manner, if you have an output tool of some sort. Otherwise, the only thing being logged to MongoDB are the info messages you normally get from a workflow running.

[caltang]

Thank you @jNunez ! Very helpful. Just to expand further on your Question 1 answer, if I save back to DB, does MongoDB or the Server store anything or does it just save the data back to DB without copying/saving on MongoDB?

[jNunez · Accepted solution]

Correct, the data is only being saved to the database.

[caltang]

Awesome! Thanks for clearing my doubts. My concerns were on the Data Stream Out and In parts (Step 2 - 4), but based on what you explained, it is consistent with my own understanding and the answer provided by Alteryx / other Server users. 

Before I mark both your replies as accepted solutions, do you mind waiitng for a bit until I hear more opinions/feedback from others? Then people will respond to it. Thank you @jNunez !

[jrgo · Accepted solution]

@caltang in general, externally sourced data read in from a normal Input tool or In-DB Stream Out tool are not persisted in MongoDB. Depending on the size, the data will stay in memory or temporary store on disk in the temp workspace the job is using, which is automatically pruned when the job is finished.

As @jNunez mentioned, only time data will be persisted in Mongo is if you design your workflow/app to produce a downloadable file in the workflow result page.

[caltang]

Since I'm writing back to DB, there isn't really any export of data for end users. I think that should be fine then right?

[jrgo · Accepted solution]

Correct.

[caltang]

Thank you very much @jNunez @jrgo !!