Conditional Tableau Publish

[Pradeep66]

Hi team, 

I publish data from alteryx to tableau.

Here is my scenario.

dataset1: every morning, I create( overwrite the history yxdb file) this contains data from previous day to history.

dataset2 : this is something runs on the day every 20 minutes. (today data at 20 mins snapshot) my workflow runs every 20 mins.

I am overwriting this data to using publish to tableau server. ( because this is not append type data but snapshot level at dataset2.)

Requirement:

Every time I publish, is there a way to delete last 20 mins data deletion and append only latest 20 mins data to tableau server?

simply to put, delete old snapshot of today data from whole data of published data source and append latest via tableau publish to server.

this would save rewrite(overwrite of old data) time?

[ArtApa]

Hi @Pradeep66 - You would need to add a column with a time stamp to filter out 20 min. There are two ways to do that:

1) Date Time Now tool - In this case the tool will generate a Time Stamp at the beginning of your workflow run and you will need to Append Field to your dataset. All records will have the same time stamp.

2)  DateTimeNow() function in a Formula tool - In this case time stamps will be generated for every record separately. 

[Pradeep66]

@ArtApa , thanks for you reply. Unfortunately this is not so straight forward. 

Once first publish is done ( that is, history + first 20 mins) is written to tableau server.

For second publish this should be ( history -first 20 mins+ second 20 mins) to tableau server.

for third publish his should be ( history - second 20min+ third 20 mins data)

and the list goes on... to latest iteration.

and every run is RAM type run, I mean we do not store any where in files but in tableau server via publish of data source.

the exact scenario is deletion of subset data ( last published 20 min data and append latest 20 mins) from whole published data

[Christina_H]

Rather than removing the previous 20 mins from the published data source, can't you just overwrite the whole thing with history + latest 20 mins?

[Pradeep66]

@Christina_H  I am doing the same as you have mentioned. 

Challenge over here is, My workflow is scheduled every 20 mins to publish. With the increase of data data day by day data publish is taking longer time. for 15 million records ( 1.5 month data) we are intended to keep last 6 months data for datasource publish. 

for 1.5 months its is taking 4-5 minutes, I worry by the time I reach end of 6th month data will be huge and overwrite takes longer times. 

I have time limit to complete with in 20 mins, as per business.

[Christina_H]

In that case, could you just append the latest 20 minutes and filter to the correct data in Tableau?  Based on either a timestamp or some other identifier for each 20min set of data.