Alteryx designer Discussions

Find answers, ask questions, and share expertise about Alteryx Designer.

Tde Append/Replace

Highlighted
Alteryx Partner
Alteryx Partner

Has anyone figured out a way to do a partial .tde replacement? The scenario I have is 5 years of total data that need to be reported on and the latest 6 months can be restated at any time. I would like to replace a portion of the .tde and keep the rest the same. The only way I can think of to accomplish this is to use two .tdes with version 10.x of Tableau where I can append data sources to each other which would allow me to append to the 4.5 year .tde every week and replace the 6 month tde. This can work when we upgrade our server to version 10.x, but until then I was hoping Alteryx could help.

 

Where Alteryx comes in:

I noticed the portion that takes the longest within Alteryx is the compression of the .tde not the transmission. I could replace the entire .tde if I didnt have to wait for the compression time. Is there a way to reduce the compression time by saving off the 4.5 years in a compressed format (preferrably .tde) and appending the latest 6 months to the file, without having to decompress the 4.5 years, and then sending? The other catch is this needs to be done on in Alteryx server.

Highlighted
Quasar
Quasar

The fact that you have data in the TDE for which values can change is going to be the issue that will force you to rebuild the TDE each time.  Tableau's 'append' to a TDE feature is really only intnded to handle new data, not updates.

 

I would simple store your contents in a yxdb or actual database table, then use Alteryx to determine if new records mus be written, then publish the TDE directly to Tableau server using the app from the public gallery.

Highlighted
Alteryx Partner
Alteryx Partner

Hi @dataMack,

 

I don't think I did a very good job explaining the possible solution. I currently use the app from the Alteryx public gallery and it takes a considerable amount of time to compress larger data sets (they can range from 25GB to 100GB) and they need to be sent remotely. We currently store the data in a star schema database and conduct mild transformations on the data prior to sending it to Tableau. Currently all of the data in the .tde is replaced every week, but we are reliant of several departments reporting their reconcilation to us, dating back up to 6 months (out of a total date range of 5 calendar years, YTD). The reliance on the other teams prevents us from processing the data whenever we would like because some of the reconcilation is done manually. My goal is to reduce the reporting runtime when the other teams provide us with data by only processing the data they can alter. I would like to create 2 separate .tdes as follows:

 

Ex: Current Date is 1/3/17

TDE 1- Contains date range 1/1/2013 through 6/30/2016, which will be the historical data that should not be restated. This information should be solid and has no reason to be reproccessed. 

 

TDE 2- Contains date range 7/1/2016 through 1/1/2017, which is subject to repocessing and will need to be placed.

 

Append the two and you have 5 years worth of data to report on.

 

Following Week

 

TDE 1- Contains date range 1/1/2013 through 7/7/2016. The only action performed on this .tde would be the appending of the week 7/1/2016 through 7/8/2016. This would only require 1 week to be processed and compressed as a tde. instead of 4.5 years. 

 

TDE 2- Contains date range 7/8/2016 through 1/8/2017. This would be a full replacement of the .tde.

 

End of Year

TDE 1- Contains date range 1/1/2013 through 6/30/2017.  6 months worth of additional data will have been appended to this file 1 week at a time.

 

TDE 2-Contains date range 7/1/2017 through 1/1/2018. This would be a full replacement of the .tde.

 

Rolling Over to the Next Year-

 

TDE 1- Replace TDE to reduce back down to the reporting time period of 5 years YTD by doing date range 1/1/2014 through 7/7/2016.

 

TDE 2-Contains date range 7/8/2017 through 1/8/2018. This would be a full replacement of the .tde.

 

Years 2014-2018 places it back to 5 years (YTD) for reporting, when the .tdes are appended.

 

By creating this type of structure my hope would be to reduce the amount of reprocessing and recompressing time for a majority of the year. In this use case the full record set would only need to be processed 1 per year. It could be done more frequently if it were a rolling reporting period and you wanted to remove the oldest data for vizualizaiton efficiencies.

 

Have you ever seen anything like this @dataMack?

Highlighted
Moderator
Moderator

Sorry @AndrewDataKim, but @dataMack is correct. Not sure if this type of functionality is even possible within a TDE, but you could always submit the idea to the Product Ideas page.

Dan Chapman
Program Manager, Customer Support
New to the community? Get started here.
Labels