Download Tool to scrape multiple pages where URL doesn't change

Team - I'm using the Download tool to scrape information off of this website: https://www.aacsb.edu/accreditation/accredited-schools

I'm nearly there, I've managed to parse all of the information I'm looking for and join everything together for the first page. I'm now ready to expand my solution to all the pages of schools. The only problem I'm running into is that when I select "View All" at the bottom of the page the URL doesn't update, so I'm not sure how to go about Downloading the entire list of available schools.

Workflow attached (TEST). What am I missing here? I feel like it's something simple but I'm a bit of a newbie at this

_____________________________________________

EDIT: 2021-04-27

Thanks to @TheOC and @mceleavey for helping out. I did have to redesign the parsing sections a bit, but was able to make quick work of it. Final workflow attached below (v2).

Output looks like this:

Web_scraping_business_schools_TEST.yxmd

Web_scraping_business_schools_v2.yxmd

Download

Accepted answers

Hi @AcevedoYo ,

This requires some wizardry in the Payload/Header section.

I've attached a workflow which will return all results, courtesy of @TheOC . He named the workflow, but I thought it was accurate.

Hope this helps,

M.

TheOC Is a wizard.yxmd

All comments

@AcevedoYo
Now I am getting this by clicking your URL

Oh, really? That's odd... Link is working for me. Any difference when you try the home page? https://www.aacsb.edu/

Hi @AcevedoYo ,

This requires some wizardry in the Payload/Header section.

I've attached a workflow which will return all results, courtesy of @TheOC . He named the workflow, but I thought it was accurate.

Hope this helps,

M.

TheOC Is a wizard.yxmd

I could cry. Thank you so much @mceleavey and @TheOC !!! The people on the internet never fail me.

Just goes to show there is always so much to learn!

I am trying to attempt this same thing and i cannot figure out how you did this.

I am trying to pull Google Scholar Profile information.

This is the website i am attempting to pull Profiles (google.com)

Google Scholar.yxmd

Quick Links

Tags

This months top contributors

atcodedog05 19458

Qiu 15866

MarqueeCrew 13708

apathetichell 13703