Getting Info from Website

Hi,

Is there a way we can get the info from website and save it in the excel file? The website is "https://www.mba.org/news-and-research" and then from there we locate the percentage either increase or decrease in this topic "Mortgage Applications Increase in Latest MBA Weekly Survey" or "Mortgage Applications Decrease in Latest MBA Weekly Survey" There are only two figures (in red font) we are going to capture:

1. Washington, D.C. (May 8, 2024) - Mortgage applications increases 2.6 percent. (note that it is taken from May 8 and will be posted as May 10 in the excel file (every Friday).

2. The refinance share of mortgage activity to 30.6 percent

Data is being appended weekly and should look like this:

excel scrape.JPG

Hoping to share your expertise and assistance. Thanks,

Kamen

Workflow

Reporting

Documentation

Accepted answers

All comments

Brando

@KamenRider

Do you have a better link? This one presents a 404-page-not found error.

KamenRider

hi @Brando

try again

https://www.mba.org/news-and-research

kamen

cmcclellan

Hi @KamenRider

This should work for you, but you might have to tweak it to fit your exact needs. I built it pretty fast, so there's a lot of extra data floating around that you can remove as well.

Check the Browse tool at the end and let me know what you think

mortgage applications scraper.yxmd

KamenRider

Hi @cmcclellan

Thank you for the response. Can't understand but I am having this error. Please help me on this.

site error.PNG

Thanks,

Kamen

cmcclellan

hmm .... interesting.

Can you run again?

Maybe you're on a VPN or something that needs authentication ?

KamenRider

Hi @cmcclellan

I'm not quite sure. I've run it multiple times. I have also tried to remove the "https://" and same error occurs. Would you know any solutions?

Thanks,

Kamen

cmcclellan

It's working fine for me, I just did this:

2024-05-10 10_52_22-Alteryx Designer x64 - mortgage applications scraper.yxmd.png

Can you run the workflow at home, or on a different network or wifi connection to test?

apathetichell

turn off amp.

cmcclellan

No need to turn AMP off, I've had it on the whole time. I don't know how AMP can affect connectivity to a website though.

KamenRider

HI @cmcclellan ; @apathetichell and ALL

I can't understand why the workflow is not working. May it be because it is looking for API?

api mnissing.PNG

Why is this happening?

Kamen

cmcclellan

It doesn't need an API, it's just Alteryx reading the URL.

Did you try on a different internet connection?

Hsandness

CharltonFranklin had the same issue with the download tool back in 2022 Download Data Tool - Error Transferring Data: Fail... - Alteryx Community. He was able to pull the URL data when he removed the https://www from the URL or changed https to http.

apathetichell

https://knowledge.alteryx.com/index/s/article/How-To-Capture-a-Fiddler-Trace

get a fiddler trace. I think this is a your machine/internet issue.

KamenRider

Hi @apathetichell @Hsandness

Thanks for your response. Sad to say I can't just download and install because it is being block. And I think the reason why I am not able to connect using Alteryx is that the connection might be blocking external sites although I am able to view the site when I open it.

I am not quite really sure about this but some says this is the reason.

Thanks All for your help

Kamen

cmcclellan

Have you tried it on a different connection though ?

Can you try on your home computer, not hooked up to the work intranet, or hotspot to your phone or something.

I'm 99.5% sure it's your work connection that is the issue here, not Alteryx

apathetichell

@cmcclellan re: AMP - the reason why the 'turn off amp' advice often works is because if there is another issue in the workflow - AMP can stall, misdirect and prevent diagnosising the overall issue. Turning it off allows for better error identification.

Quick Links

This months top contributors

atcodedog05 19598

Qiu 15867

binu_acs 15708

MarqueeCrew 13708

apathetichell 13703