Crawling a site with Java
I wanted to crawl a site using Java. I didn't want to invent the wheel so I wanted to find a good java library which crawls sites. On top of my head I know of several libraries, so I thought of checking them first. I began with Solr & Lucene and continued with Nutch. After doing some reading I understood that the above libraries are out of my scope and although they will do the job, they are an absolute overkill. Why ? Well, I guess I need to begin with my actual requirements Before the requirement I suggest reading the following post, just to understand the basics (I won't use that actual code, but it does help to understand the concept and to sharpen the requirements): How to build a crawler in Java Requirements Crawls a full site Minimum amount of logic so it won't go to the same URL more than once etc Multithreading Good API so it will be easy to work with, I need a simple API to define the number of threads, site root, fil...