Information retrieval and text mining using distributed latent semantic indexing

DETAILED DESCRIPTION Referring to FIG.
1 the inventive method of the document collection processing of the present invention is set forth.
At step 110 the method of the present invention generates a frequency count for each term in each document in the collection (or set) of documents.
The term "data objects" in this context refers to information such as documents, files, records etc.
Data objects may also be referred to herein as documents.
In an optional preprocessing step 100 the terms in each document are reduced to their canonical forms and a predetermined set of "stop" words are ignored.
Stop words are typically those words that are used as concept connectors but provide no actual content such as "a" "are" "do" "for" etc.
The list of common stop words is well known in the art.
Suffix strippers that reduce a set of similar words to their canonical forms are also well known in the art.
Such a stripper or parser will reduce a set of words such as computed, computing and computer to a stem word "comput" thereby combining the frequency counts for such words and reducing the overall size of the set of terms.
At step 120 the heterogeneous collection of data objects is partitioned by concept domain into sub-collections of like concept.
If it is known that one or more separate sub-collections within a larger collection of data are homogenous in nature, the initial partitioning need not be done for those known homogenous data collections.
For initial sorting of data objects into more conceptually homogeneous sub collections, the bisecting k-means algorithm in a recursive form with k=2 at each stage to obtain k clusters is preferably used.
Clustering techniques have been explored in "A Comparison of Document Clustering Techniques" by M.
Steinbach et al.
Technical Report 00-034, Department of Computer Science and Engineering, University of Minnesota.
Although the bisecting k-means algorithm is preferred, the "standard" k-means algorithm or other types of spatial clustering algorithms may be utilized