Getting Info from Website

[KamenRider]

Hi,

Is there a way we can get the info from website and save it in the excel file? The website is "https://www.mba.org/news-and-research" and then from there we locate the percentage either increase or decrease in this topic "Mortgage Applications Increase in Latest MBA Weekly Survey" or "Mortgage Applications Decrease in Latest MBA Weekly Survey" There are only two figures (in red font) we are going to capture:

1. Washington, D.C. (May 8, 2024) - Mortgage applications increases 2.6 percent. (note that it is taken from May 8 and will be posted as May 10 in the excel file (every Friday).

2. The refinance share of mortgage activity to 30.6 percent

Data is being appended weekly and should look like this:

 

Hoping to share your expertise and assistance. Thanks,

Kamen

 

[Brando]

@KamenRider 

Do you have a better link? This one presents a 404-page-not found error.

[KamenRider]

hi @Brando 

try again

https://www.mba.org/news-and-research

kamen

[Hsandness]

Hi Kamen,

I don't have a solution, but I would recommend looking into using the Beautiful Soup library within python. It allows you to scrape data from the web and should allow you to pull the information you need by locating it via HTML tags. The site here walks you through simple web scraping examples and should get you started.

Beautiful Soup: Build a Web Scraper With Python – Real Python

If you look in the attached screenshot you'll see that the data you're after is nested inside a <div> tag.

[cmcclellan]

Hi @KamenRider 

This should work for you, but you might have to tweak it to fit your exact needs.  I built it pretty fast, so there's a lot of extra data floating around that you can remove as well.

Check the Browse tool at the end and let me know what you think

[KamenRider]

Hi @cmcclellan 

Thank you for the response. Can't understand but I am having this error. Please help me on this.

Thanks,

Kamen

 

[cmcclellan]

hmm .... interesting.

Can you run again?

Maybe you're on a VPN or something that needs authentication ?

[KamenRider]

Hi @cmcclellan 

I'm not quite sure. I've run it multiple times. I have also tried to remove the "https://" and same error occurs. Would you know any solutions?

Thanks,

Kamen

[cmcclellan]

It's working fine for me, I just did this:

Can you run the workflow at home, or on a different network or wifi connection to test?

[apathetichell]

turn off amp.