ChaiPuter

Let's build a web spider / crawler Features Multithreaded (else network will slow it to a crawl) Work with proxies Manage Robots .txt (disable or change agent) Resumeable (in case of a crash it should continue where it stopped) Politeness (wait at least X milliseconds before connecting the same domain) Binary crawling (be able to parse PDFs etc) Configurable stop conditions like: MaxDepth, MaxNumOfPages, MaxSize? Be able to crawl https Evade spider traps Heed redirects (status code 3xx) Normalize URLs (so it won't crawl same page twice) Configurable shouldVisit (Page page, URL url) which will enable the user to focus the crawler on specific pages (regexp of suffix) specific domain etc SIMPLE API (disable all advanced features) Use third party libraries for: URL Normalization, Robot management, PageFetcher, Multithreaded framework, Frontier DB, Page Parser (Tika for binary) Enable the user to grab and whatever he wants with the visit (Page page)  (Op...

I wanted to crawl a site using Java. I didn't want to invent the wheel so I wanted to find a good java library which crawls sites. On top of my head I know of several libraries, so I thought of checking them first. I began with Solr & Lucene and continued with Nutch. After doing some reading I understood that the above libraries are out of my scope and although they will do the job, they are an absolute overkill. Why ? Well, I guess I need to begin with my actual requirements Before the requirement I suggest reading the following post, just to understand the basics (I won't use that actual code, but it does help to understand the concept and to sharpen the requirements): How to build a crawler in Java Requirements Crawls a full site Minimum amount of logic so it won't go to the same URL more than once etc Multithreading Good API so it will be easy to work with, I need a simple API to define the number of threads, site root, fil...