Web Crawler / Spider Features

Let's build a web spider / crawler



Features

  • Multithreaded (else network will slow it to a crawl)
  • Work with proxies
  • Manage Robots.txt (disable or change agent)
  • Resumeable (in case of a crash it should continue where it stopped)
  • Politeness (wait at least X milliseconds before connecting the same domain)
  • Binary crawling (be able to parse PDFs etc)
  • Configurable stop conditions like: MaxDepth, MaxNumOfPages, MaxSize?
  • Be able to crawl https
  • Evade spider traps
  • Heed redirects (status code 3xx)
  • Normalize URLs (so it won't crawl same page twice)
  • Configurable shouldVisit(Page page, URL url) which will enable the user to focus the crawler on specific pages (regexp of suffix) specific domain etc
  • SIMPLE API (disable all advanced features)
  • Use third party libraries for: URL Normalization, Robot management, PageFetcher, Multithreaded framework, Frontier DB, Page Parser (Tika for binary)
  • Enable the user to grab and whatever he wants with the visit(Page page) 
  • (Optional) pluggable Jsoup for the output?
  • (Optional) Pluggable big data for the output ?
  • (Optional) Pluggable internal DB for frontier ?


The Crawler's Flow

  • Initial configuration (properties file as the spider's only argument)
  • Initial Seed
  • URL passes through filters1 to see if we should proceed (design pattern?), filters will be:
    • User's configured regexp shouldVisit()
    • Depth
    • MaxPages
    • Non crawlable suffix like: jpg, mp3 etc
    • Checks internal archived frontier DB (if already parsed then don't do it again), this will be done using the Normalized url's Hash as key
  • Robots is checked (if not disabled)
  • PageFetcher (CrawlerCommon)
  • Page passes through filters2
    • Status Code (4xx ?  Redirect ?)
    • Size (if size was limited, should it be limited?)
    • Binary content (if user configured not to fetch binary content)
  • Page is parsed
  • URLs extracted and adder to frontier
  • Parsed page sent to visit(Page page) for the user's enjoyment
  • Pluggable JSoup? Pluggable BigData ?


Internal DB

  • Should be a queue where all new URLs are stored and then popped when the crawler will consume them
  • Should be an archive of crawled URLs, so the same URLs won't be crawled twice
  • The URLs in those DBs should be normalized
  • The URLs in those dbs should be hashed for quick find
  • The internal DB technology should be a pluggable implementation so it could be changed easily
  • The DB schema should be something as follows:
    • Hash | normalized URL | Timestamp | ParentID | Depth | Rank | Counter
    • Where Rank can be calculated for an advanced spider
    • Counter is a sort of a quick rank to see how many times a URL was called
  • Famous implementatios for the internal DB will be:
    • BerklyDB
    • LevelDB
    • ...

Comments

Post a Comment

Popular posts from this blog

Profiling Java @ 2019

A good profiler can really help you in your work while coding Java (I assume while coding any coding language but I am referring to Java here) It can find memory leaks (Which object and how many of it takes the most of your limited memory?), CPU bottlenecks and other surprising data like the time consumed by each network call / DB query etc. Thus it is not so surprising to find many different java profilers around. I will try to map here the Java profiling tools lying around My short list of requirements will include the following: Profiler Requirements Constantly updated profiler (how a profiler built to profile Java6 will behave with Java11?) Clear GUI - As I will want to dive into the profiling data - it will be much simpler using UI Free (With a big advantage to the open sourced ones) (Optional) Remote profiling List of Profilers Dead Profilers (At least 5 years without an update) JIP - Java Interactive Profiler  - Last update 2010 Profiler4J - Last Updat
Read more

Ant Explorer graphical Ant build tool

Every serious java developer needs to create or edit some ant building script. Eclipse offers nice ant handling features which make the task much easier, eclipse's plugin (which is embeded into eclipse since version 3.1 or so) understands the ant scripting language thus enables not only syntax coloring but also nice mouse hovering which reveals values of properties, variable names which enables finding the source of each variable and also some nice debugging features. But, there is one field in which eclipse's ant handling lacks perfection, its in viewing the whole build file in one gaze graphically feature, that feature isn't supported by eclipse. Luckily enough, it is supported by YWorks, which created a nice and free util which loads an ant script file and shows it in a beautiful and so very useful way. I think that this util (which was released as a standalone as well as an eclipse and idea) is a must for any serious java developer. Unluckily, YWorks have remo
Read more

Start Working with AutoIT

I have decided I needed a quick GUI creator which can create a small exe file displaying the GUI, with no system dependencies like a .NET framework or Java runtime. What options do I have for a Quick GUI Creator? Maybe there are many options I am not aware of, but after extensive searching I found these two: Auto Hot keys AutoIT (Saw some recommendations to try Delphi or Pascal for this task but I passed) Both tools began from the same code source, autohotkeys branched off and continued to be open source while AutoIT is more commercial. I have compared the two (well, not actually compared, just google'd out other's comparisons) and found out that the AutoHotKeys is slightly better and is more open sourced (which makes it the better solution) but lacks in one field which is crucial for me - the script syntax is awful!  That tipped the balance (as a programmer i couldn't stand AHK's syntax) so I chose AutoIT. How to get Started with AutoI
Read more