Is there a way to limit data downloaded from the Download Tool?

Question

Context, I’m scraping a bunch of websites as practice. I realised that the Download Tool slows down tremendously if the data gets exponential.

Is there a way to limit the amount of Data that goes into the Download Tool?

For example, if I parsed that website into rows, i would have about 60k rows per website. I just need the first 7k as that’s where the data is.

Any and all advice / help appreciated! Thanks.

- Cal

caltang · Answer

That is sad… well guess it works well for less than 100 unique sites. Thanks @apathetichell!

apathetichell · Answer

Are you scraping a web page or are you using an API - most APIs support this using a syntax of limit= - but the specifics of how you do this in the API vary greatly. If you are scraping a webpage (ie non-API). It would really depend on the html structure of the page you are scraping... Potentially no.

caltang · Answer

That’s more on API limits though. I don’t see such an option for URL specific…