WebHarvy can be used to scrape data from social media websites like Twitter, LinkedIn, Facebook etc. In the following video you can see how easy it is to scrape tweets from Twitter searches using WebHarvy. Similar technique can be used to scrape tweets from a Twitter profile page.
In this video, pagination via JavaScript code is used to scrape multiple pages of Twitter search results.
The JavaScript code used in the above video is copied below.
groupEl = document.getElementsByTagName(‘article’)[0].parentElement.parentElement.parentElement.parentElement;
groupEl.children[groupEl.childElementCount-1].scrollIntoView();
Normally, pages which load more data as we scroll down can be configured by following the method explained at https://www.webharvy.com/tour3.html#ScrollToLoad. But in the case of Twitter, the page also deletes tweets from the top as we scroll down. Hence, JavaScript has to be used for pagination.
Try WebHarvy
In case you are interested, we recommend that you download and try using the free evaluation version of WebHarvy available in our website. To get started, please follow the link given below.