Java uses WebHarvest to crawl web page data
WebHarvest is an open source Java framework for obtaining structured data from the World Wide Web. It supports defining crawl rules in an explicit manner, using XPaths and regular expressions for data extraction, thereby transforming data from web pages into structured data, such as XML or Java objects.
The advantages of WebHarvest include:
1. Easy to use: WebHarvest provides a concise crawling rule definition language, making it easy to write and maintain crawlers.
2. Powerful data extraction function: WebHarvest supports XPaths and regular expressions, making data extraction flexible and powerful.
3. Scalability: WebHarvest supports custom plugins, which can easily extend its functionality.
4. Multi threading support: WebHarvest can run multiple crawling tasks simultaneously, improving the efficiency of crawling.
5. Support for multiple data formats: WebHarvest can convert the captured data into various formats such as XML and JSON.
The drawbacks of WebHarvest include:
1. Slow update: WebHarvest is a relatively old framework, although powerful, its latest version has been released less frequently and updates are slow.
2. Complicated configuration: For some complex web page structures, it is necessary to have a deep understanding of XPaths and regular expressions, and the configuration may be slightly complex.
To use WebHarvest, the following dependencies need to be added to Maven:
<dependency>
<groupId>net.webharvest</groupId>
<artifactId>webharvest-core</artifactId>
<version>2.1</version>
</dependency>
The following is a simple web crawler Java code example implemented using WebHarvest:
import net.webharvest.definition.ScraperConfiguration;
import net.webharvest.runtime.Scraper;
import net.webharvest.runtime.variables.Variable;
import org.apache.commons.io.FileUtils;
import java.io.File;
import java.io.IOException;
public class WebHarvestExample {
public static void main(String[] args) throws IOException {
//Load WebHarvest configuration file
File configFile = new File("config.xml");
String configXml = FileUtils.readFileToString(configFile, "UTF-8");
//Create a WebHarvest crawler and execute it
ScraperConfiguration scraperConfiguration = new ScraperConfiguration(configXml);
Scraper scraper = new Scraper(scraperConfiguration);
scraper.execute();
//Obtaining Grab Results
Variable variable = scraper.getContext().getVar("result");
if (variable != null) {
System.out.println(variable.toString());
}
}
}
The above code assumes the existence of a configuration file called "config. xml", in which WebHarvest's crawling rules can be defined.
Summary: WebHarvest is a powerful and easy-to-use Java crawler framework that helps developers easily extract structured data from web pages. Although WebHarvest updates slowly and may be cumbersome to configure, it is a mature and stable framework suitable for most simple to medium complexity crawling tasks.
