Scraping issue dealing with a multi-language website
I am trying to build a tool that scrapes a webpage to extract some particular data.
For example: 1 - 20 of 200 results and I want to get that "200" number.
Now this tool is to be used by people from different countries, and each country have their own version of that website. (.com, .co.uk, .ca, .ru, .de , .co.jp etc), what would be the best way for me to extract that value? Taking into consideration that different countries will have different ways of displaying this string and can have different character sets.
Thanks!
It really depends on your tool. I am also building a webscraper, and I got xpath and css paths as ways to pick out information. So I can do stuff like:
Which basically means, find links (
a) inside alielement, below theulwithclasspagination.With that result I get links to the other pages that might be in the pagination.
If your target website uses the same system for all languages, you might be able to just write one code, that parses that type of site. Or you might not be that lucky. It really depends on a lot of stuff.
Hey Eddy.
The site you're using will probably have the "same" basic structure for all (most) languages.
You're going to have to examine the site to see if the site is:
Static/Dynamic
Static site is pretty much no javascript. It can be parsed/handled via python/php/regex/etc..
Dynamic, a bit harder. Need to have capacity/tech to walk through site to see what the javascript does. Worse case, will need headless server tech to handle.
Site layout
Is the site consistent between the different languages. Most sites are consistent regarding the html/javascript/layout elements, with the user/text changing.
Once you've gotten an understanding of the above, should be fairly straightforward to extract the data.
*** All of this assumes the site hasn't gone "crazy" and uses recaptcha/google tech to make scraping difficult!
hope this helps
Thanks!