Crawling a site with Java

I wanted to crawl a site using Java.


I didn't want to invent the wheel so I wanted to find a good java library which crawls sites.

On top of my head I know of several libraries, so I thought of checking them first.
I began with Solr & Lucene and continued with Nutch.

After doing some reading I understood that the above libraries are out of my scope and although they will do the job, they are an absolute overkill.

Why ?


Well, I guess I need to begin with my actual requirements
Before the requirement I suggest reading the following post, just to understand the basics (I won't use that actual code, but it does help to understand the concept and to sharpen the requirements):
How to build a crawler in Java



Requirements

  1. Crawls a full site
  2. Minimum amount of logic so it won't go to the same URL more than once etc
  3. Multithreading
  4. Good API so it will be easy to work with, I need a simple API to define the number of threads, site root, filter which pages to pull according to their URL, not going to external links, easy way to parse every page and do something with it.
  5. Unbloated code, so no need for any GUI on top of it etc - just a minimal library
  6. Open source
  7. As always I prefer a current library (from the last 3 years), and prefer one which is regularly updated
  8. Not search engine library, so no "client / server" code, no need to save all of the website to a nosql DB, no need for scaling, multiple servers etc - just a simple library


Requirement #7 removed the libraries I knew of from the list (Solr / Lucene / Nutch / Bixo / Elstic search)

I googled for a list of crawling libraries and found this good list:

I checked all of those libraries but almost all of them fell on requirement #6
Most of those libraries are ancient.
Archnid (2002), Weblech (2004), WebEater (2003), Jspider (2003), WebSphinx (2002)

Some libraries were closed source (Requirement #5): Infant, Heritrix (Used by web-archive!  which stores all of the internet in all times)
Some were bloated: Web-harvest (and last release was at 2010)


Which left with just two options:
Crawler4j (2013)
Niocchi (2011)

Both of these seem good choices.


Summary

In my search for a library which will crawl a full site, I found that the crawling libraries are separated into two main groups:
Libraries which are intended to be part of a full fledged search engine, which needs to contain all of the crawled data and to index it for fast retrieval, where the better ones give support for several servers holding the content as a grid, in this group you will find big names like Solr / Lucene / Nutch / Bixo / Elstic search.

Second group consists of the atomic task of crawling a site, and providing a small and efficient library for doing so.
After eliminating most of the libraries due to my requirements, I boiled them down to the following two libraries:
Crawler4j
Niocchi

I picked Crawler4j.




As a side note: if I was pressed to use a closed source library and had a big project I would definitely check Heritrix.




Comments

Post a Comment

Popular posts from this blog

Profiling Java @ 2019

A good profiler can really help you in your work while coding Java (I assume while coding any coding language but I am referring to Java here) It can find memory leaks (Which object and how many of it takes the most of your limited memory?), CPU bottlenecks and other surprising data like the time consumed by each network call / DB query etc. Thus it is not so surprising to find many different java profilers around. I will try to map here the Java profiling tools lying around My short list of requirements will include the following: Profiler Requirements Constantly updated profiler (how a profiler built to profile Java6 will behave with Java11?) Clear GUI - As I will want to dive into the profiling data - it will be much simpler using UI Free (With a big advantage to the open sourced ones) (Optional) Remote profiling List of Profilers Dead Profilers (At least 5 years without an update) JIP - Java Interactive Profiler  - Last update 2010 Profiler4J - Last Updat
Read more

Shared/Pro/Managed Hosting for your site - which to choose ? (NO AFFILIATE LINKS!)

 You buy a domain and want to start your site running, time to find your perfect hosting. This is one of our first steps, and might be your most crucial one. I won't tell you which hosting company to choose, I will attempt just to clarify the picture. What are your Hosting options ? Shared Hosting You can choose shared hosting - obviously this one will be the cheapest option. In this category you might even find free hosting options, with or without ads on your site. Although this option sounds great for beginners as it is cheap and is comes with many promises from the hosting company (unlimited bandwidth, unlimited disk space etc.), there are no free meals, and as you are not alone on this server ( shared hosting), you don't really know who your neighbors are. So nothing is guaranteed, you might have a neighbor which takes lots of bandwidth, so you will actually have less to use etc. You might have a malicious neighbor which will attempt to hack your site using the common host
Read more

ZVR converter

Yesterday, a friend asked me to find him a ZVR sound file converter. What's the problem? I asked... Give me 5 minutes and i'll google it out. Well, I was wrong, finding a decent converter isn't an easy task, it took me at least 7 minutes (kidding). What's a ZVR file? It's a dictation recorder output file, used by the korean SAFA company, their recording machine is used mainly to capture sound files using its microphone during class lessons or whenever one wants. I just don't get it, why each company comes up with its own sound format?  grab a good and free existing audio format and stick to it! The ZVR files are very small hence their low quality... Anyway, a nice ZVR to WAV file converter was found. It's an incredibly small file (140kb). Download ZVR sound file converter  Enjoy, Avi.
Read more