How to analyze text in Java

I have a site with a form.

Users use that form and send me requests, so I naively thought to automate the process of reviewing those requests by analyzing the text - makes sense no ?


Well, apparently I opened a Pandora box called NLP.

It seems that text analyzing is a very vast subject with different algorithms of doing so.

In order to have some order out of the chaos I want to separate the subject to several sub-subjects:
  • Sentence isolation - breaking the paragraph to sentences [not everyone is using "period"]
  • Naming - identifying names, places, dates, currency etc.
  • POS-TAGging - finding the type of each word in the sentence (Noun, Verb etc)
  • Parsing - Identifying sentence parts like subject, direct object etc
There are many more parts and sub parts but the above are those I decided to focus on
Please note that in order to be accurate the tools need a big "dictionary" of the parsed language, thus these tools might be very heavy on megabytes (some are several hundred MB).



I will take the following sentence as an example sentence:
"I live in California and I want to buy a horse"


How will I analyze it ?
(more on java specific implementation in the next paragraph)

First I will use the sentence isolation tool to isolate this sentence
I will then use the Naming tool to identify that the user lives at California
I will use the POS-TAGging tool to identify the noun (NN) as "horse"
Then make sure it is the sentence's "direct object" by using the Parsing tool.



What are the Java tools of the trade ?


Requirements

  1. SIMPLE (many of those tools are very complicated, but I have a small project, so I don't want to delve in tiny language details)
  2. I prefer one tool which has all of the functionality
  3. Up to date (from the last 3 years) and preferably updated regularly
  4. Prefer a tool which has support for several languages
  5. FAST (some of these tools can be very slow in doing their thingy)
  6. Prefer Open source

I didn't review all of the Java libraries out there but here are my results:
Gate - Fell on requirement #1 - although it has tons of documentation, it is too complicated to begin with
OpenNLP - Has all components, updated, seems easy to use, open source
Stanford - Has it all, might be slow, not sure how simple it is - Looks very professional though (for parser consider it's shift reduce parser)
Berkly Parser - Seems like a great and fast Parser, unfortunately I can't find any documentation, and I don't think it contains the rest of the needed tools.  BUT it supports 6 languages, (falls on requirement #2)
Mate-Tools - POS-Tag + Parser for 4 languages, no documentation, seem to have good algorithms, but I don't think it is too user friendly and they lack the full stack of tools I need. (falls on requirement #2)
Dk-Pro-asl - One tool that supposedly has it all, not sure about the number of languages supported



So what will I use?

For a big project I think it is worth it to investigate Gate.
for my regular small projects I will try OpenNLP which is under apache, and have Dk-Pro-asl as a fallback.



Side note: For identifying a site's language I will use: Language-detector

Comments

Post a Comment

Popular posts from this blog

Profiling Java @ 2019

A good profiler can really help you in your work while coding Java (I assume while coding any coding language but I am referring to Java here) It can find memory leaks (Which object and how many of it takes the most of your limited memory?), CPU bottlenecks and other surprising data like the time consumed by each network call / DB query etc. Thus it is not so surprising to find many different java profilers around. I will try to map here the Java profiling tools lying around My short list of requirements will include the following: Profiler Requirements Constantly updated profiler (how a profiler built to profile Java6 will behave with Java11?) Clear GUI - As I will want to dive into the profiling data - it will be much simpler using UI Free (With a big advantage to the open sourced ones) (Optional) Remote profiling List of Profilers Dead Profilers (At least 5 years without an update) JIP - Java Interactive Profiler  - Last update 2010 Profiler4J - Last Updat
Read more

Shared/Pro/Managed Hosting for your site - which to choose ? (NO AFFILIATE LINKS!)

 You buy a domain and want to start your site running, time to find your perfect hosting. This is one of our first steps, and might be your most crucial one. I won't tell you which hosting company to choose, I will attempt just to clarify the picture. What are your Hosting options ? Shared Hosting You can choose shared hosting - obviously this one will be the cheapest option. In this category you might even find free hosting options, with or without ads on your site. Although this option sounds great for beginners as it is cheap and is comes with many promises from the hosting company (unlimited bandwidth, unlimited disk space etc.), there are no free meals, and as you are not alone on this server ( shared hosting), you don't really know who your neighbors are. So nothing is guaranteed, you might have a neighbor which takes lots of bandwidth, so you will actually have less to use etc. You might have a malicious neighbor which will attempt to hack your site using the common host
Read more

ZVR converter

Yesterday, a friend asked me to find him a ZVR sound file converter. What's the problem? I asked... Give me 5 minutes and i'll google it out. Well, I was wrong, finding a decent converter isn't an easy task, it took me at least 7 minutes (kidding). What's a ZVR file? It's a dictation recorder output file, used by the korean SAFA company, their recording machine is used mainly to capture sound files using its microphone during class lessons or whenever one wants. I just don't get it, why each company comes up with its own sound format?  grab a good and free existing audio format and stick to it! The ZVR files are very small hence their low quality... Anyway, a nice ZVR to WAV file converter was found. It's an incredibly small file (140kb). Download ZVR sound file converter  Enjoy, Avi.
Read more