Web Scraping, selecting alternative url links based on a condition

Question

Hi,

Pretty new to web scraping so I'll try and make myself as clear as I can but hopefully someone will be able to point me in the right direction.

I currently have a workflow which will pull the url links from a website which has documents set out in different languages, most of these urls have 3 documents (3 languages) per page, what I would like to do is perhaps set a formula up in the workflow which recognises key words in the url relating to the language selection e.g. 'English', 'Arabic', 'French', 'Dutch' and select an alternative link if any of the non required languages appear in the files.

The problem with the links is they don't all have the same languages, so a page with Arabic text may only have links for French, German or Hindi whilst a page with English text may only have links for Dutch, French, Arabic etc. so doing a switch of the key word won't always work as the language options I want may not necessarily be in those links.

I basically have a list of languages I don't require so what I want to achieve is have the workflow recognise the key word e.g. if I don't want Arabic, I want it to automatically go to the second or even third link.

Can anyone advise if this is possible?

Thanks!

dougperez · Answer

@Lili7891  can you provide some mock data to me?

DawnDuong · Answer

Hi @Lili7891

Thanks for clarifying. Unless there is an easy way to limit how far need to scrape before detecting the language then it can mean need to scrape the entire page...

I think there is an R package to guess the language (auto detect language), but i'm not as familiar with it. Very interesting topic - i'll watch this space to hear advice from the community gurus

Cheers

Dawn.

Lili7891 · Answer

Hi @DawnDuong Appreciate the help, I am familiar with the check box in an app format, however I want the workflow to be able to automatically scrape based on the condition as opposed to an end user being able to select the language, hope that makes sense.