Scraping universities webs and description maybe.

Hello, I need help in scraping universities from this website which is really complicated for me: https://whed.net/home.php

since I need names and some of their field of studies or at least websites and it only show if i click on the map or select description anyway this can be done efficiently please.

Thanks in advance.

Workflow

API

Accepted answers

All comments

IraWatt

Hey @MZ900605,

Is it this information your interested in?

or this:

Can you screenshot and highlight what information specifically your interested in?

Thanks,

Ira

smoskowitz

Hi @MZ900605 --

I took a quick look at the site and I think its do-able. Personally, I think it would be somewhat easier using Python and BeautifulSoup than Alteryx, but the Python script can be put into the Python tool for any downstream processing.

I don't have the time to try to code this out (as my coding skills are weak.) The biggest challenge I see is figuring out the URL for each country or state web address. Once you crack that, then you can:

Write each country URL to a list.
Loop through each country and get all school URL's and write all of those to a list.
The loop through each school URL and collect the relevsant data and maybe write that to a pandas dataframe.

Hopefully that provides some guidance.

Thanks,

Seth

MZ900605

the actual names and websites if we can.

MZ900605

would love python also..

MZ900605

@IraWatt I hope this helps Thanks.
@smoskowitz That's really interesting will give it a try if Alteryx did not help well.

IraWatt

Hey @MZ900605,

just looking at your Canada example on the map when you click Canada then

This is the request which generates the page:

(this is the view source of that page Search Results – WHED – IAU's World Higher Education Database):

The links to each box is stored on the page here. Eg the popup for "Canada - Northwest Territories" has the address: https://whed.net/detail_system.php?JTo2MF0tIzRgCmAK which you can request from Alteryx with the download tool.

I think you would need to replicate these requests in Alteryx or Python to download all the countries information.

MZ900605

@IraWatt so what i need is to copy each link and connect it to download tool ?

Screenshot (59).png

IraWatt

@MZ900605 Here is an example workflow to get this page:

I also updated my initial look at the problem above^

Workflow.yxmd

MZ900605

@IraWatt this will take ages

I hope to find a way to collect universities names and the courses they have but this was the only website I found

Quick Links

This months top contributors

atcodedog05 19598

Qiu 15875

binu_acs 15708

MarqueeCrew 13708

apathetichell 13703