What is your experience of web-scraping?
Hi. I'm doing a bit of research. I used to write scrapers for a living and wonder if there's much demand for bespoke scrapers.
Who here employs web-scraping in their business? Do you write scrapers yourself? Use SaaS scrapers? Hire other developers to write scrapers?
What sort of sites do you scrape? What problems have you faced? Was there anything you couldn't scrape? Have you ever considered web-scraping but didn't know where to begin?
I would love to hear people's thoughts.
I usually hire data-scrapers on a freelance platform that either automates it or does it by hand, depending on the project size and difficulty. If I need info from one specific website, I usually automate it, which is not worth doing when sources are different, as you have to build scraping automation for each source separately.
What I noticed is that people don't realize how great info you can scrape for your project which can help in decision-making. As a bespoke scraper, your job should be not just to scrape but to actually make people see how they can use scraping for their business needs.