Multi-GPU Accelerated Processing of Time-Series Data of Huge Academic Backbone Network in ELK Stack

We report our operational experience in deploying multi-GPU accelerated monitoring system of huge academic backbone network in ELK stack. Science Information Network (SINET) is a Japanese academic backbone network for more than 800 research institutions and universities. Since 2016, our SOC team has been running the monitoring system in the SINET's gateway for handling hundreds of millions of session data generated by PaloAlto-7080 per day. For providing the deep insights with SOC operators, Multi-GPU server (DGX-1) is running on the workflow between Elastic Stack and Splunk. We qualitatively introduce the past bottlenecks (2016–2018) in coping with PA-7080’s traffic stream stored in ELK stack. To name a few, we illustrate some techniques such as multi-process invocation of scroll API, parallel CUDA Thrust API invocation and massively parallel access to highly concurrent container. We also report the performance measurements in processing randomly generated 729 GB session data in about 910 minutes.