Fundamental breakthroughs are needed in mathematical, computational and decision science methods geared towards detecting and predicting rare events or unusual patterns, especially from heterogeneous and multi-source datasets, particularly within complex systems. Complex systems as defined in this context refer to complexity in the unobserved data generation processes as well as the observed data. The data generation processes may be potentially nonlinear, even chaotic, non-stationary, and typically multi-dimensional (e.g., space and time). Recurrence patterns of extremes, rare events or anomalies may be hypothesized to have deterministic or stochastic patterns. The data from such systems are likely to exhibit complex dependence structures, including proximity-based correlations (e.g., temporal or spatial autocorrelations), long-range dependence or long memory processes, nonlinear synchronicity, as well as noise or variability that may be colored, possibly even â1/fâ. Furthermore, the data may be massive and streaming, disparate enough to be occasionally difficult to compare directly with each other, but the patterns of the unusual or rare behavior as well as their precursors may be difficult to extract from data. In fact, the data useful for rare events may even be relatively short and few, given the nature and type of noise and the complexity of the generation processes. The processes may be inherently interconnected at multiple scales.
Our research in these areas has focused on national and homeland security as well as climate change, among other multidisciplinary areas such as evaluation of social science theories and models. The aspiration in security applications is on the possibility of extracting actionable predictive insights from elusive indicators of threat which may be present in large data sets. Our focus on climate change is on understanding or projecting extreme events and abrupt change, or low probability but high impact events, from massive volume data generated from large-scale computational models as well as remote or in-situ sensors across the world at multiple scales in space and time. Our methodological approaches have been interdisciplinary, blending concepts and leveraging methods from statistics and econometrics, data mining and machine learning in computer science, complex networks and graphical models, nonlinear dynamics and signal processing in physics and engineering, as well as translational insights developed across multiple domains in the sciences and engineering.