Why does the Download Tool download the data twice?

[abragg0103]

I have a large file (500mb) that my workflow has to download and then process.

I noticed that the Download Tool downloads the file once, then overwrites the file and downloads it a second time. I watched the folder that the file is downloaded to and confirmed that this happens.

Is there a way to avoid this? Am I setting up my Download Tool incorrectly? Unfortunately I cannot share the link, but it is being imported from the URL field of a connected text input tool.

[HomesickSurfer]

Hi @abragg0103 

Could it be caching the file until completely downloaded, appearing as twice?

[abragg0103]

Possibly? That's kind of my question. The problem is it takes about 8 minutes to download the first time, then it spends another 8 minutes overwriting it and downloading it the second time. Ultimately it's not a massive problem but I was curious if I'm doing something wrong in my settings or not.

[AZuc · Accepted solution]

@abragg0103 

How many rows your download tool is receiving. Could you check if you're not sending the link twice?

[mceleavey]

Hi @abragg0103 ,

as @AZuc suggested you're probably sending the link multiple times to the download tool. If you are dynamically building your URL based on logic, or if you are scraping the sub-URLs from the parent page, then you will need to add a summary tool and group by the URL field. This will then produce the unique URLs that you are downloading.

M.