System and method for partitioned distributed scanning of a large dataset for viruses and other malware

DETAILED DESCRIPTION The present invention provides a system and method for concurrently scanning a large dataset for computer viruses and other forms of malware.
The dataset is organized into a set of distributed databases each containing a plurality of groups storing individual data items.
The data items are each uniquely identified by an identifier and can be included in a plurality of the groups.
A plurality of malware servers cooperatively scan the groups for viruses and malware by using a commonly shared centralized database for tracking and synchronization.
Scanned data items are tracked using a message identifier table and a last read table both maintained within the centralized database.
The scanning of multiple part data items is synchronized using a threads table with the centralized database.
Thus, the concurrent malware scanners can divide up the groups for concurrent processing in a highly scalable manner.
An embodiment of the present invention is a system and a method for performing partitioned scanning of a dataset for malware in a distributed computing environment.
A dataset is maintained in a plurality of structured databases in the distributed computing environment.
Each database stores a plurality of data item groups which each include a plurality of individual data items.
Each such data item is uniquely identified within the dataset by a data item identifier.
A set of indices is stored in a centralized database.
The set of indices includes a list of scanned data item identifiers for each data item within the dataset scanned for malware and a list of last entry numbers for each data item group stored in each database.
Each last entry number corresponds to one such data item within the data item group last scanned for malware.
A plurality of malware scanners are executed in substantial concurrency.
For each malware scanner, one such database and each such data item group within the selected database having data items not appearing in the list of last entry numbers are selected