Crawling a site with Java

I wanted to crawl a site using Java.


I didn't want to invent the wheel so I wanted to find a good java library which crawls sites.

On top of my head I know of several libraries, so I thought of checking them first.
I began with Solr & Lucene and continued with Nutch.

After doing some reading I understood that the above libraries are out of my scope and although they will do the job, they are an absolute overkill.

Why ?


Well, I guess I need to begin with my actual requirements
Before the requirement I suggest reading the following post, just to understand the basics (I won't use that actual code, but it does help to understand the concept and to sharpen the requirements):
How to build a crawler in Java



Requirements

  1. Crawls a full site
  2. Minimum amount of logic so it won't go to the same URL more than once etc
  3. Multithreading
  4. Good API so it will be easy to work with, I need a simple API to define the number of threads, site root, filter which pages to pull according to their URL, not going to external links, easy way to parse every page and do something with it.
  5. Unbloated code, so no need for any GUI on top of it etc - just a minimal library
  6. Open source
  7. As always I prefer a current library (from the last 3 years), and prefer one which is regularly updated
  8. Not search engine library, so no "client / server" code, no need to save all of the website to a nosql DB, no need for scaling, multiple servers etc - just a simple library


Requirement #7 removed the libraries I knew of from the list (Solr / Lucene / Nutch / Bixo / Elstic search)

I googled for a list of crawling libraries and found this good list:

I checked all of those libraries but almost all of them fell on requirement #6
Most of those libraries are ancient.
Archnid (2002), Weblech (2004), WebEater (2003), Jspider (2003), WebSphinx (2002)

Some libraries were closed source (Requirement #5): Infant, Heritrix (Used by web-archive!  which stores all of the internet in all times)
Some were bloated: Web-harvest (and last release was at 2010)


Which left with just two options:
Crawler4j (2013)
Niocchi (2011)

Both of these seem good choices.


Summary

In my search for a library which will crawl a full site, I found that the crawling libraries are separated into two main groups:
Libraries which are intended to be part of a full fledged search engine, which needs to contain all of the crawled data and to index it for fast retrieval, where the better ones give support for several servers holding the content as a grid, in this group you will find big names like Solr / Lucene / Nutch / Bixo / Elstic search.

Second group consists of the atomic task of crawling a site, and providing a small and efficient library for doing so.
After eliminating most of the libraries due to my requirements, I boiled them down to the following two libraries:
Crawler4j
Niocchi

I picked Crawler4j.




As a side note: if I was pressed to use a closed source library and had a big project I would definitely check Heritrix.




Comments

Post a Comment

Popular posts from this blog

Profiling Java @ 2019

A good profiler can really help you in your work while coding Java (I assume while coding any coding language but I am referring to Java here) It can find memory leaks (Which object and how many of it takes the most of your limited memory?), CPU bottlenecks and other surprising data like the time consumed by each network call / DB query etc. Thus it is not so surprising to find many different java profilers around. I will try to map here the Java profiling tools lying around My short list of requirements will include the following: Profiler Requirements Constantly updated profiler (how a profiler built to profile Java6 will behave with Java11?) Clear GUI - As I will want to dive into the profiling data - it will be much simpler using UI Free (With a big advantage to the open sourced ones) (Optional) Remote profiling List of Profilers Dead Profilers (At least 5 years without an update) JIP - Java Interactive Profiler  - Last update 2010 Profiler4J - Last U...
Read more

Ant Explorer graphical Ant build tool

Every serious java developer needs to create or edit some ant building script. Eclipse offers nice ant handling features which make the task much easier, eclipse's plugin (which is embeded into eclipse since version 3.1 or so) understands the ant scripting language thus enables not only syntax coloring but also nice mouse hovering which reveals values of properties, variable names which enables finding the source of each variable and also some nice debugging features. But, there is one field in which eclipse's ant handling lacks perfection, its in viewing the whole build file in one gaze graphically feature, that feature isn't supported by eclipse. Luckily enough, it is supported by YWorks, which created a nice and free util which loads an ant script file and shows it in a beautiful and so very useful way. I think that this util (which was released as a standalone as well as an eclipse and idea) is a must for any serious java developer. Unluckily, YWorks have remo...
Read more

Start Working with AutoIT

I have decided I needed a quick GUI creator which can create a small exe file displaying the GUI, with no system dependencies like a .NET framework or Java runtime. What options do I have for a Quick GUI Creator? Maybe there are many options I am not aware of, but after extensive searching I found these two: Auto Hot keys AutoIT (Saw some recommendations to try Delphi or Pascal for this task but I passed) Both tools began from the same code source, autohotkeys branched off and continued to be open source while AutoIT is more commercial. I have compared the two (well, not actually compared, just google'd out other's comparisons) and found out that the AutoHotKeys is slightly better and is more open sourced (which makes it the better solution) but lacks in one field which is crucial for me - the script syntax is awful!  That tipped the balance (as a programmer i couldn't stand AHK's syntax) so I chose AutoIT. How to get Started with AutoI...
Read more