Efficient Tracking of Statistical Properties of Data Streams with Rapid Changes

Abstract

Many real-life dynamical systems change rapidly followed by almost stationary periods. In this paper, we consider streams of data with such rapidly changing behavior and investigate the problem of tracking their statistical properties in an online manner. The streaming estimator is accompanied with a second estimator, suitable to adjust to rapid changes in the data stream. When a statistically significant difference is observed between the two estimators, the current estimate jumps to a more suitable value. Such a tracking procedure have previously been suggested in the literature. However, our contribution lies in building the estimation procedure based on the difference between the stationary estimator and a Stochastic Learning Weak Estimator (SLWE). The SLWE estimator is known to be the state-of-the art approach to tracking properties of nonstationary environments and thus should be a better choice to detect changes in rapidly changing environments than the far more common sliding window based approaches. Extensive simulation results demonstrate that our estimation procedure is easy to tune and performs very well. Further, the suggested estimator outperforms the popular and state-of-the-art estimator ADWIM2 with a clear margin.