Web Crawler / Spider Features

Let's build a web spider / crawler



Features

  • Multithreaded (else network will slow it to a crawl)
  • Work with proxies
  • Manage Robots.txt (disable or change agent)
  • Resumeable (in case of a crash it should continue where it stopped)
  • Politeness (wait at least X milliseconds before connecting the same domain)
  • Binary crawling (be able to parse PDFs etc)
  • Configurable stop conditions like: MaxDepth, MaxNumOfPages, MaxSize?
  • Be able to crawl https
  • Evade spider traps
  • Heed redirects (status code 3xx)
  • Normalize URLs (so it won't crawl same page twice)
  • Configurable shouldVisit(Page page, URL url) which will enable the user to focus the crawler on specific pages (regexp of suffix) specific domain etc
  • SIMPLE API (disable all advanced features)
  • Use third party libraries for: URL Normalization, Robot management, PageFetcher, Multithreaded framework, Frontier DB, Page Parser (Tika for binary)
  • Enable the user to grab and whatever he wants with the visit(Page page) 
  • (Optional) pluggable Jsoup for the output?
  • (Optional) Pluggable big data for the output ?
  • (Optional) Pluggable internal DB for frontier ?


The Crawler's Flow

  • Initial configuration (properties file as the spider's only argument)
  • Initial Seed
  • URL passes through filters1 to see if we should proceed (design pattern?), filters will be:
    • User's configured regexp shouldVisit()
    • Depth
    • MaxPages
    • Non crawlable suffix like: jpg, mp3 etc
    • Checks internal archived frontier DB (if already parsed then don't do it again), this will be done using the Normalized url's Hash as key
  • Robots is checked (if not disabled)
  • PageFetcher (CrawlerCommon)
  • Page passes through filters2
    • Status Code (4xx ?  Redirect ?)
    • Size (if size was limited, should it be limited?)
    • Binary content (if user configured not to fetch binary content)
  • Page is parsed
  • URLs extracted and adder to frontier
  • Parsed page sent to visit(Page page) for the user's enjoyment
  • Pluggable JSoup? Pluggable BigData ?


Internal DB

  • Should be a queue where all new URLs are stored and then popped when the crawler will consume them
  • Should be an archive of crawled URLs, so the same URLs won't be crawled twice
  • The URLs in those DBs should be normalized
  • The URLs in those dbs should be hashed for quick find
  • The internal DB technology should be a pluggable implementation so it could be changed easily
  • The DB schema should be something as follows:
    • Hash | normalized URL | Timestamp | ParentID | Depth | Rank | Counter
    • Where Rank can be calculated for an advanced spider
    • Counter is a sort of a quick rank to see how many times a URL was called
  • Famous implementatios for the internal DB will be:
    • BerklyDB
    • LevelDB
    • ...

Comments

Post a Comment

Popular posts from this blog

Profiling Java @ 2019

A good profiler can really help you in your work while coding Java (I assume while coding any coding language but I am referring to Java here) It can find memory leaks (Which object and how many of it takes the most of your limited memory?), CPU bottlenecks and other surprising data like the time consumed by each network call / DB query etc. Thus it is not so surprising to find many different java profilers around. I will try to map here the Java profiling tools lying around My short list of requirements will include the following: Profiler Requirements Constantly updated profiler (how a profiler built to profile Java6 will behave with Java11?) Clear GUI - As I will want to dive into the profiling data - it will be much simpler using UI Free (With a big advantage to the open sourced ones) (Optional) Remote profiling List of Profilers Dead Profilers (At least 5 years without an update) JIP - Java Interactive Profiler  - Last update 2010 Profiler4J - Last Updat
Read more

Shared/Pro/Managed Hosting for your site - which to choose ? (NO AFFILIATE LINKS!)

 You buy a domain and want to start your site running, time to find your perfect hosting. This is one of our first steps, and might be your most crucial one. I won't tell you which hosting company to choose, I will attempt just to clarify the picture. What are your Hosting options ? Shared Hosting You can choose shared hosting - obviously this one will be the cheapest option. In this category you might even find free hosting options, with or without ads on your site. Although this option sounds great for beginners as it is cheap and is comes with many promises from the hosting company (unlimited bandwidth, unlimited disk space etc.), there are no free meals, and as you are not alone on this server ( shared hosting), you don't really know who your neighbors are. So nothing is guaranteed, you might have a neighbor which takes lots of bandwidth, so you will actually have less to use etc. You might have a malicious neighbor which will attempt to hack your site using the common host
Read more

ZVR converter

Yesterday, a friend asked me to find him a ZVR sound file converter. What's the problem? I asked... Give me 5 minutes and i'll google it out. Well, I was wrong, finding a decent converter isn't an easy task, it took me at least 7 minutes (kidding). What's a ZVR file? It's a dictation recorder output file, used by the korean SAFA company, their recording machine is used mainly to capture sound files using its microphone during class lessons or whenever one wants. I just don't get it, why each company comes up with its own sound format?  grab a good and free existing audio format and stick to it! The ZVR files are very small hence their low quality... Anyway, a nice ZVR to WAV file converter was found. It's an incredibly small file (140kb). Download ZVR sound file converter  Enjoy, Avi.
Read more