Prevent HUGE amount of data when scraping HTML

Good morning

We are collecting HTML data from this this web page using a Download tool. When we run the workflow the Download tool tells us the data we are downloaded is a huge 1.1TB - which is obviously not good 😧 - but when we look at the raw data there are only about 50,000 records.

Has anyone had to overcome this kind of thing before? Perhaps there are some tricks to avoid creating such a data volume in the download or possibly a way to avoid it happening in the first place.

Here's hoping 🤞

Thanks

ianjonna

Download

Accepted answers

ianjonnaCAA

thanks Re: Salesforce Input Tool

I will close this off now until we get the newer version rolled out!!!!

Many thanks

Cheers🖐

@danilang

@ChrisTX

@Ben_H

All comments

Ben_H

Hi @ianjonnaCAA,

I have absolutely no idea how you've got to 1.1TB!

Do you have an example of your workflow to share?

When I try it I get about 1.3MB!

Regards,

Ben

Scrape.yxmd

ianjonnaCAA

Hi Ben

Most grateful for you getting back to me. Having seen your workflow it looks like the way I configured the tool matches but ................ looks like my issue actually originates in a following xml parse tool instead. A copy of my workflow is attached if you have any ideas?

Thanks again

Cheers

Scrape workflow.yxmd

Ben_H

Hi @ianjonnaCAA

Ah I see!, It's due to the "Include in Output" checkbox.

With it turned on you've ended up with the full download data duplicated 100,000+ times, hence the massive size.

Regards,

Ben

ianjonnaCAA

thanks @Ben_H , will look now 😀

danilang

Hi @ianjonnaCAA

If you need more examples check out, Weekly Challenge-116 A Symphony of Parsing Tools. It's input is a large, complex XML file. The file is large enough that using a straight linear expansion, causes your memory use to explode to the point where the workflow would take days to run. The submitted solutions demonstrate how to parse the required elements and also remove the resulting xxx_Outer_XML fields to keep your memory use in check.