Automation and scraping a website

I have an Excel file with a column filled with +4000 URLs each one in a different cell. I need to use Alteryx to open with Chrome and scrape some of the data from a website. past them in excel.

And then do the same step for the next URL. Could you please help me with that?

Thanks,

Simon

Macros

Best Practices

Batch Macro

Dynamic Processing

Accepted answers

All comments

mceleavey

Hi @Simon1187 ,

Yes, you need to build a batch macro to loop through each URL. This will return the HTML per row for every URL. You then simply need to parse the information from the URL, but with 4000 URLS, assuming those URLs do not contain the exact same HTML, you will need to build it individually.

However, if they are the same, then you only need to build it once for all.

If you care to share the URLs and let me know what you need I can get you started.

Simon1187

Hi @mceleavey

Thanks for your prompt response. I think I should clarify a little bit. When I said 4000+ URLs, I meant the first parts of the URLs are the same. I have attached an Excel file for you to give you some idea about that. Unfortunately, as the website is internal, it is not accessible from the outside. However, I have put a screenshot of the page.

The page has fixed parts like (Profile views, phone calls, etc) but the numbers beside them are dynamic.

The goal is I could read those parts and save them in an Excel file. The data needs to be read once per day.

I appreciate it if you could give me further assistance on this case, please.

Thank you!

Simon

Sample URLs.xlsx

danilang

Hi @Simon1187

If possible, it's always better to go for source data as opposed to scraping data from a web page. Since your websites are internal, I would suggest reaching out to the people who maintain them and working with them to get access to the data that's used to build the pages.

Another option is to try to call the webpages will different options. Your current URLs look like this

https://X.COM/112949/profile-analytics?from=2018-03-01&graph=bar&sort=monthly&to=2021-11-03

The part in bold is specifying that you want to to receive a bar graph. Maybe there's an option that you can specify to return the data in table form or csv. Here again, you have to reach out to the people that maintain the data to find out what's available.

Dan

Quick Links

This months top contributors

AkimasaKajitani 389

mceleavey 388

mbarone 337

michael_weaver 335

Hollingsworth 335