Web Scraping from a Password-Protected Site

Hi Alteryx Community,

I am attempting to design a workflow that can extract credit ratings from a website such as Standard&Poors. These websites however require the user to login to the site before displaying any data.

I have been able to execute a POST request to login to the S&P website according to the instructions found on the webpage linked -> https://www.thedataschool.co.uk/joe-carr/webscraping-through-alteryx-as-if-you-are-logged-in/, however, I cannot figure out how to remain logged in while executing a GET request to initiate my web scraping (even after retaining the cookie returned from the POST request).

I have already tried discussing this with an Alteryx Solutions Engineer but we could not resolve this query. I am hoping somebody in the community can help point me in the right direction. I am not very savvy with web programming so I may be structuring my requests incorrectly as well.

Download

Accepted answers

PhilippK

Hi @RogerWyllie ,

I solved a similar challenge via the Alteryx Python tool and the package Selenium (with this you can automate/remote control your browser).

@DavidM wrote a greate article about that:

https://community.alteryx.com/t5/Alteryx-Designer-Discussions/Python-Code-Tool-Web-Scraping-Dynamic-Websites-Using-Selenium/td-p/300335

Best regards

Phil

All comments

BrandonB

Do you have access to their API? This would be a much more direct route: https://www.spglobal.com/marketintelligence/en/documents/spciq_api_v2.pdf

This will likely depend on the licensing that your company has arranged with theirs.

RogerWyllie

Hi Brandon,

I am trying to avoid the use of any APIs as this will complicate the approval process for the tool I am trying to create. If I cannot find any solution then I will be sure to use the link you provided though, thank you!

Quick Links

This months top contributors

AkimasaKajitani 387

mceleavey 383

mbarone 337

Hollingsworth 335

LanisC 335