Efficient Analysis of Live and Historical Streaming Data and itsApplication to Cybersecurity

Description

Applications that query data streams in order to identifytrends, patterns, or anomalies can often benefit from comparing the livestream data with archived historical stream data. However, searching thishistorical data in real time has been considered so far to beprohibitively expensive. One of the main bottlenecks is the update costsof the indices over the archived data. In this paper, we address thisproblem by using our highly-efficient bitmap indexing technology (calledFastBit) and demonstrate that the index update operations aresufficiently efficient for this bottleneck to be removed. We describe ourprototype system based on the TelegraphCQ streaming query processor andthe FastBit bitmap index. We ...
continued below

Publisher

Provided By

UNT Libraries Government Documents Department

Serving as both a federal and a state depository library, the UNT Libraries Government Documents Department maintains millions of items in a variety of formats. The department is a member of the FDLP Content Partnerships Program and an Affiliated Archive of the National Archives.

Contact Us

What

Descriptive information to help identify this article.
Follow the links below to find similar items on the Digital Library.

Description

Applications that query data streams in order to identifytrends, patterns, or anomalies can often benefit from comparing the livestream data with archived historical stream data. However, searching thishistorical data in real time has been considered so far to beprohibitively expensive. One of the main bottlenecks is the update costsof the indices over the archived data. In this paper, we address thisproblem by using our highly-efficient bitmap indexing technology (calledFastBit) and demonstrate that the index update operations aresufficiently efficient for this bottleneck to be removed. We describe ourprototype system based on the TelegraphCQ streaming query processor andthe FastBit bitmap index. We present a detailed performance evaluation ofour system using a complex query workload for analyzing real networktraffic data. The combined system uses TelegraphCQ to analyze streams oftraffic information and FastBit to correlate current behaviors withhistorical trends. We demonstrate that our system can simultaneouslyanalyze (1) live streams with high data rates and (2) a large repositoryof historical stream data.

Collections

This article is part of the following collection of related materials.

Office of Scientific & Technical Information Technical Reports

Reports, articles and other documents harvested from the Office of Scientific and Technical Information.

Office of Scientific and Technical Information (OSTI) is the Department of Energy (DOE) office that collects, preserves, and disseminates DOE-sponsored research and development (R&D) results that are the outcomes of R&D projects or other funded activities at DOE labs and facilities nationwide and grantees at universities and other institutions.