Web Scraping Using AI

Part 1 : Extracting Data Using LLMs

In this mode, you configure WebHarvy as usual using point-and-click, and for any field that needs more than a simple copy of the page text, you ask an LLM to process it. The AI generated output is captured as a data column, just like any other selected field.

1.1 Supported AI Providers

WebHarvy can connect to the following AI providers:

• Local LLMs - via Ollama or LM Studio (see Scraping with Local LLMs)
• Cloud LLMs - OpenAI and Anthropic (+ all providers which support the OpenAI API Protocol)

1.2 What can you do with AI in WebHarvy?

Once configured, AI can be used during configuration and mining to:

• 1. Generate summaries from blocks of text
• 2. Analyze the sentiment of scraped content
• 3. Extract complex insights from unstructured data
• 4. Transform or clean data before it is captured
• 5. Extract data that is difficult to select using regular point-and-click methods
• etc.

1.3 Configuring AI Settings

Before you can use AI while selecting data, you need to connect WebHarvy to your preferred AI provider.

1. 1. Open WebHarvy Settings from the Home menu.
2. 2. Switch to the AI tab.

3. 3. Select your AI provider (Ollama (local), OpenAI or Anthropic) and provide the required connection details, such as the local server address for Ollama/LM Studio, or the API key for OpenAI/Anthropic.
4. 4. Select the model you would like WebHarvy to use, then save the settings.

Note: Click the 'Test' button to verify that the connection parameters you provided are correct.

1.4 Extracting Data Using AI

Once an AI provider has been configured, you can use it while selecting data during configuration.

1. 1. Click on the area of the webpage from which you want to extract data using AI.
2. 2. From the More Options menu, select Extract with AI.

3. 3. In the window that appears, specify:
   • The extraction area - either the currently selected region, or the entire page.
   • The source to use for extraction - the page's displayed (rendered) text, or its underlying HTML code.
4. 4. Describe what you want the AI to do - for example, summarize the selected text, determine its sentiment, or pull out specific values from it - and WebHarvy will capture the AI generated output as a data column, just like any other selected field.

Note: Using cloud based AI providers (OpenAI, Anthropic) sends the selected page content to the respective provider for processing, and may incur usage costs as per their pricing. Local LLMs run via Ollama or LM Studio process data entirely on your own computer and do not incur charges.

Part 2 : Scraping Using AI Assistants via MCP

Here the roles are reversed. Instead of WebHarvy calling an AI model, a desktop AI assistant such as Claude Desktop, Codex (ChatGPT) or LM Studio takes control of WebHarvy. You tell the assistant what data you need, in plain English, and it uses WebHarvy's browser to open the website, navigate and click through pages, and extract the data for you.

Everything runs on your own PC. The AI assistant uses the same browser you see in the WebHarvy window, so it works with pages you have already opened or logged in to, and you can step in at any time, for example to solve a CAPTCHA.

2.1 What is MCP?

MCP (Model Context Protocol) is an open standard which allows AI applications to connect to external tools. When WebHarvy's MCP server is enabled, WebHarvy exposes its browser as a tool which MCP clients like Claude Desktop, Codex and LM Studio can use. This combines the reasoning ability of the AI assistant with WebHarvy's real browser, which can load and interact with JavaScript heavy websites just like a human user.

This is useful when you want to:

• 1. Quickly grab data from a website without building a configuration first
• 2. Describe a scraping task conversationally and let the AI work out the steps
• 3. Have the AI analyze, compare or summarize the extracted data within the same conversation

2.2 Video : Claude Desktop scraping data using WebHarvy

2.3 Supported AI Clients

WebHarvy can connect to these desktop AI clients:

You can connect more than one client at the same time. They all share WebHarvy's single browser window, and their actions run one at a time.

Note: Other MCP clients which can start a local (stdio) MCP server should also work using the same command (see Manual Setup below), but they haven't been tested.

2.4 Enabling the MCP Server & Connecting Your AI Client

1. 1. Open WebHarvy Settings from the Home menu.
2. 2. Switch to the MCP tab.

3. 3. Check the Enable MCP Server option. The server starts right away, and again every time WebHarvy starts, until you turn it off.
4. 4. Under Connect to an MCP Client, choose your AI client (Claude Desktop, Codex or LM Studio) from the Client list.
5. 5. Click Connect Automatically. WebHarvy adds itself to that client's configuration file and leaves any other entries in the file unchanged.
6. 6. Click OK to save the settings.
7. 7. Restart the AI client, so that it loads the new configuration.
8. 8. Keep WebHarvy open. The AI client can use WebHarvy only while it is running, with the MCP server enabled.

Note: If the client's configuration folder doesn't exist, WebHarvy reports that the client isn't installed. Install the client (and run it once) before connecting.

2.5 Manual Setup

If you'd rather edit the configuration file yourself, or automatic setup can't update it, use the Copy and Open Config Folder buttons in the MCP tab. Copy the entry shown and add it to the client's configuration file.

Claude Desktop and LM Studio (JSON):

{
  "mcpServers": {
    "webharvy": {
      "command": "C:\\Program Files\\SysNucleus\\WebHarvy\\WebHarvy.exe",
      "args": ["--mcp-stdio-bridge"]
    }
  }
}

Codex (TOML):

[mcp_servers.webharvy]
command = "C:\\Program Files\\SysNucleus\\WebHarvy\\WebHarvy.exe"
args = ["--mcp-stdio-bridge"]

Note: The MCP tab fills in the actual path of your WebHarvy installation. The path above is only an example. If the file already contains other MCP servers, add the webharvy entry next to them, instead of replacing the file contents.

2.6 Scraping Data Using Your AI Assistant

Once connected, WebHarvy will be listed among the available tools / MCP servers in your AI client. Make sure that WebHarvy is open, then start a new chat and describe the data you need. For example:

• "Using WebHarvy, go to https://www.example.com/products and extract the name, price and rating of all products listed on the page."
• "Go through the next 5 pages of results and collect the same details."
• "Open the first 5 product links and get their descriptions as well. Show the results in a table."
• "Save the extracted data as an Excel file."

The assistant will use WebHarvy's browser to load the pages, click and navigate as required, and extract the data. You can watch it working in WebHarvy's browser. It can then display the extracted data in a table, save it as a CSV, Excel, JSON, XML or any other file format, or analyze it further as you continue the conversation.

Note: When using MCP, the content of the pages loaded by WebHarvy is sent to the AI assistant (and its AI provider) for processing, and its usage limits and pricing apply. With a local model in LM Studio, everything stays on your computer.

2.7 What the AI Assistant Can Do

WebHarvy gives the AI assistant 24 browser actions (tools), in four groups.

Navigation

Reading the page

Interacting with the page

Scrolling and loading more content

2.8 Good to Know

• CAPTCHAs and bot checks - When WebHarvy detects a visible reCAPTCHA, hCaptcha, Cloudflare challenge or "Access Denied" page, it tells the AI assistant. The assistant then asks you to solve it in the WebHarvy window and tries again once you confirm.
• Your normal work comes first - While you're setting up a configuration by hand in WebHarvy, the AI assistant can't use the browser. It's asked to wait until you've finished.
• Keep WebHarvy restored, not minimized - A minimized window stops lazy-loaded content and screenshots from working, and the assistant tells you if that happens. The window can stay behind other windows.
• One browser, one action at a time - All connected clients share the single visible browser. Their requests are carried out one after another, never at the same time.
• No network port - The MCP server doesn't listen on any network or localhost port. The AI client starts a small helper (WebHarvy.exe --mcp-stdio-bridge), which talks to the running WebHarvy through a local Windows named pipe. Other computers can't reach it.
• Off by default - The server only runs when Enable MCP Server is checked. When MCP isn't in use, configuring and mining work exactly as before.