How to analyze text in Java

I have a site with a form.

Users use that form and send me requests, so I naively thought to automate the process of reviewing those requests by analyzing the text - makes sense no ?


Well, apparently I opened a Pandora box called NLP.

It seems that text analyzing is a very vast subject with different algorithms of doing so.

In order to have some order out of the chaos I want to separate the subject to several sub-subjects:
  • Sentence isolation - breaking the paragraph to sentences [not everyone is using "period"]
  • Naming - identifying names, places, dates, currency etc.
  • POS-TAGging - finding the type of each word in the sentence (Noun, Verb etc)
  • Parsing - Identifying sentence parts like subject, direct object etc
There are many more parts and sub parts but the above are those I decided to focus on
Please note that in order to be accurate the tools need a big "dictionary" of the parsed language, thus these tools might be very heavy on megabytes (some are several hundred MB).



I will take the following sentence as an example sentence:
"I live in California and I want to buy a horse"


How will I analyze it ?
(more on java specific implementation in the next paragraph)

First I will use the sentence isolation tool to isolate this sentence
I will then use the Naming tool to identify that the user lives at California
I will use the POS-TAGging tool to identify the noun (NN) as "horse"
Then make sure it is the sentence's "direct object" by using the Parsing tool.



What are the Java tools of the trade ?


Requirements

  1. SIMPLE (many of those tools are very complicated, but I have a small project, so I don't want to delve in tiny language details)
  2. I prefer one tool which has all of the functionality
  3. Up to date (from the last 3 years) and preferably updated regularly
  4. Prefer a tool which has support for several languages
  5. FAST (some of these tools can be very slow in doing their thingy)
  6. Prefer Open source

I didn't review all of the Java libraries out there but here are my results:
Gate - Fell on requirement #1 - although it has tons of documentation, it is too complicated to begin with
OpenNLP - Has all components, updated, seems easy to use, open source
Stanford - Has it all, might be slow, not sure how simple it is - Looks very professional though (for parser consider it's shift reduce parser)
Berkly Parser - Seems like a great and fast Parser, unfortunately I can't find any documentation, and I don't think it contains the rest of the needed tools.  BUT it supports 6 languages, (falls on requirement #2)
Mate-Tools - POS-Tag + Parser for 4 languages, no documentation, seem to have good algorithms, but I don't think it is too user friendly and they lack the full stack of tools I need. (falls on requirement #2)
Dk-Pro-asl - One tool that supposedly has it all, not sure about the number of languages supported



So what will I use?

For a big project I think it is worth it to investigate Gate.
for my regular small projects I will try OpenNLP which is under apache, and have Dk-Pro-asl as a fallback.



Side note: For identifying a site's language I will use: Language-detector

Comments

Post a Comment

Popular posts from this blog

Profiling Java @ 2019

A good profiler can really help you in your work while coding Java (I assume while coding any coding language but I am referring to Java here) It can find memory leaks (Which object and how many of it takes the most of your limited memory?), CPU bottlenecks and other surprising data like the time consumed by each network call / DB query etc. Thus it is not so surprising to find many different java profilers around. I will try to map here the Java profiling tools lying around My short list of requirements will include the following: Profiler Requirements Constantly updated profiler (how a profiler built to profile Java6 will behave with Java11?) Clear GUI - As I will want to dive into the profiling data - it will be much simpler using UI Free (With a big advantage to the open sourced ones) (Optional) Remote profiling List of Profilers Dead Profilers (At least 5 years without an update) JIP - Java Interactive Profiler  - Last update 2010 Profiler4J - Last U...
Read more

Ant Explorer graphical Ant build tool

Every serious java developer needs to create or edit some ant building script. Eclipse offers nice ant handling features which make the task much easier, eclipse's plugin (which is embeded into eclipse since version 3.1 or so) understands the ant scripting language thus enables not only syntax coloring but also nice mouse hovering which reveals values of properties, variable names which enables finding the source of each variable and also some nice debugging features. But, there is one field in which eclipse's ant handling lacks perfection, its in viewing the whole build file in one gaze graphically feature, that feature isn't supported by eclipse. Luckily enough, it is supported by YWorks, which created a nice and free util which loads an ant script file and shows it in a beautiful and so very useful way. I think that this util (which was released as a standalone as well as an eclipse and idea) is a must for any serious java developer. Unluckily, YWorks have remo...
Read more

Start Working with AutoIT

I have decided I needed a quick GUI creator which can create a small exe file displaying the GUI, with no system dependencies like a .NET framework or Java runtime. What options do I have for a Quick GUI Creator? Maybe there are many options I am not aware of, but after extensive searching I found these two: Auto Hot keys AutoIT (Saw some recommendations to try Delphi or Pascal for this task but I passed) Both tools began from the same code source, autohotkeys branched off and continued to be open source while AutoIT is more commercial. I have compared the two (well, not actually compared, just google'd out other's comparisons) and found out that the AutoHotKeys is slightly better and is more open sourced (which makes it the better solution) but lacks in one field which is crucial for me - the script syntax is awful!  That tipped the balance (as a programmer i couldn't stand AHK's syntax) so I chose AutoIT. How to get Started with AutoI...
Read more