How to use User Agent strings to prevent blocking while web scraping ?

What is a user agent string ? The User-Agent string of a web browser helps servers (websites) to identify the browser (Chrome, Edge, FireFox, IE etc.), its version and also the operating system (Windows, Mac, Android, iOS etc.) on which it is running. This mainly helps the websites to serve different pages for various platforms …

Scraping Instagram Images using WebHarvy

WebHarvy can be used to scrape text as well as images from websites. In this article we will see how WebHarvy can be used to scrape Instagram Images. How to scrape images from Instagram? The following video shows how WebHarvy can be configured to scrape Instagram images (download images) by searching Instagram for a tag …

Scraping Twitter using WebHarvy

WebHarvy can be used to scrape data from social media websites like Twitter, LinkedIn, Facebook etc. In the following video you can see how easy it is to scrape tweets from Twitter searches using WebHarvy. Similar technique can be used to scrape tweets from a Twitter profile page. In this video, pagination via JavaScript code …

WebHarvy 6.1 – Internal Proxies, Database/File Update, New Capture window options

The following are the main changes in this version. Option to leave a blank row when data is unavailable for a keyword/category/URL In WebHarvy’s Keyword/Category settings page a new option has been added to leave a blank row filled with corresponding keyword/category/URL when data is unavailable for that item. This option is available only when …