This document is a white
paper for our PhD project “Multilingual Text Mining”. It explains in
layman terms different concepts used in Text Mining. Then application of these
concepts and techniques to our project are discussed.
What is Text Mining ?
“Text
mining, also known as intelligent text analysis, text data mining or
knowledge-discovery in text (KDT), refers generally to the process of extracting
interesting and non-trivial information and knowledge from unstructured text.
Text mining is a young interdisciplinary field which draws on information
retrieval, data mining, machine learning, statistics and computational
linguistics. As most information (over 80%) is stored as text, text mining is
believed to have a high potential”
Early Developments
For approximately 4000 years, man has organized information for later retrieval and usage. A typical example is the table of contents of a book. Since the volume of information eventually grew beyond a few books, it became necessary to build specialized data structures to ensure faster access to the stored information. An old and popular data structure for faster information retrieval is a collection of selected words or concepts with which are associated pointers to the related information (or documents) -- the index. In one form or another, indexes are at the core of every modern information retrieval system. They provide faster access to the data and allow the query processing task to be speeded up.
For centuries, indexes were created manually as categorization hierarchies. In fact, most libraries still use some form of categorical hierarchy to classify their volumes (or documents). Such hierarchies have usually been conceived by human subjects from the library sciences field. More recently, the advent of modern computers has made possible the construction of large indexes automatically. Automatic indexes provide a view of the retrieval problem which is much more related to the system itself than to the user need. In this respect, it is important to distinguish between two different views of the IR problem: a computer-centered one and a human-centered one.
In the computer-centered view, the IR problem consists mainly of building up efficient indexes, processing user queries with high performance, and developing ranking algorithms which improve the `quality' of the answer set. In the human-centered view, the IR problem consists mainly of studying the behavior of the user, of understanding his main needs, and of determining how such understanding affects the organization and operation of the retrieval system. According to this view, keyword based query processing might be seen as a strategy which is unlikely to yield a good solution to the information retrieval problem in the long run.
©2005 Jatit