This site uses different types of cookies, including analytics and functional cookies (its own and from other sites). To change your cookie settings or find out more, click here. If you continue browsing our website, you accept these cookies.
I have to stitch certain files from SFTP. SFTP contains daily files.
Each file has 90 days of data in it, so date ranges that we wish to pull outside of 90 days require multiple files. The issue with this is that the files are most accurate with days further in the past (i.e. we could be missing data points if we were to simple stitch files every 90 days). To fix this issue, we should be stitching the files together in a way that prioritizes the last 30 days of each file. Below is an example.
Date Range Pull: 1/1/2021 – 6/1/2021 = 151 days
o Most recent date for the range pull should dictate which file we start with
o 30 days prior to 6/1/2021
o 31 days prior to 5/2/2021
o The reason this one is not 30 days is because the number of days between 1/1/2021 and 6/1/2021 is not divisible by 30, so instead of using another file to take the other day, we would be fine with using 4/1/2021 to use the 31st day in the file.
o The last file can leverage the last 30-60 days in the file, but if the number of days exceeds 60, we must pull a new file.
Seems to me that you need a dynamic solution that allows you to specify the file as well as the range of dates you want to from each file and that stitching the data together after the range/file has been correctly set is not an issue.
You can try building an app with the following inputs:
1) File browse x 3 (since you need a max of 3 files only)
2) 6 drop down tools, 2 for each file, to specify the start/end dates.
3) create 3 groups, each group has 1 file browse and 2 drop down tools
For each of the files you select, add a filter tool which filters in the desired date range based on the inputs from the drop down tools.