Web scraping to download csv files

Question

Good morning from merry old England,

I am hoping that someone a lot more savvy with Alteryx than I can help with my issue. I am attempting to download the entire collection of csv files from a website using Alteryx. My issue is that in order to download them, you need to manually click buttons to apply filters and I am wanting EVERY possible table going on this webpage.

I have spoken to a colleague and he has suggested web scraping to solve my issue. I have no clue about web scraping unfortunately and he is busy on another project so I am at a loss.

The website I am looking at is the following: https://www.hesa.ac.uk/data-and-analysis/performance-indicators/widening-participation

If you look at the website, you may see what I mean about all the different iterations of tables with all their filters. Ideally, I am looking to only download the "England" tables where you can apply this filter at the top.

I appreciate that this is a large ask but even the smallest amount of help down this road would be outstanding!

Thank you in advance,

Dane.

danilang · Accepted Answer

Hi @danespoors

Check out the link under each table called Download source data (csv).

I believe, though I haven't verified for all the tables, that the Source data table contains all the values necessary to display all the variations for the given table.   Download all 8 of these and you should have all the data to build all variants.

Dan

danespoors · Accepted Answer

I have found a way to solve this now.

I have analysed the exact url of the source table csv files. If I perform a download and then follow basic web scraping procedures (that I've hastily read about) I can collect all of the data from the website. I do need to reconstruct the tables since it piles it all together into an insanely large cell but now it's just a matter of time.

Thanks to @danilang  for your help, you definitely helped a lot.

I'll mark this and your answer as a solution since this has solved my issue of sourcing the data from this website.

Many thanks,

Dane.

danilang · Accepted Answer

You're welcome @danespoors

I'll just leave this here since I already had it built.

This downloads the HTML for the site, finds the tables and associated URLs and downloads them to the current directory as separate files.  This should save you the trouble of parsing the giant single cell.

Dan