Multi-GPU Accelerated Processing of Time-Series Data of Huge Academic Backbone Network in ELK Stack

Monday, October 28, 2019 - 2:00 pm2:45 pm

Ruo Ando, Center for Cybersecurity Research and Development, National Institute of Informatics

Abstract: 

We report our operational experience in deploying multi-GPU accelerated monitoring system of huge academic backbone network in ELK stack. Science Information Network (SINET) is a Japanese academic backbone network for more than 800 research institutions and universities. Since 2016, our SOC team has been running the monitoring system in the SINET's gateway for handling hundreds of millions of session data generated by PaloAlto-7080 per day. For providing the deep insights with SOC operators, Multi-GPU server (DGX-1) is running on the workflow between Elastic Stack and Splunk. We qualitatively introduce the past bottlenecks (2016–2018) in coping with PA-7080’s traffic stream stored in ELK stack. To name a few, we illustrate some techniques such as multi-process invocation of scroll API, parallel CUDA Thrust API invocation and massively parallel access to highly concurrent container. We also report the performance measurements in processing randomly generated 729 GB session data in about 910 minutes.

Ruo Ando, Center for Cybersecurity Research and Development, National Institute of Informatics

Ruo Ando is an associate professor of NII (National Institute of Informatics) by special appointment in Japan. He has a Ph.D. in computer science. Before joining NII, he was engaged in a research project supported by US AFOSR in 2003 (Grant Number AOARD 03-4049). He has presented his research at PacSec2011 (BitTorrent crawler) and DEFCON 26 (packet dump analyzer). He was co-author at LISA 2006 (hypervisor security). His current research interest is massively parallel computing.

BibTeX
@conference {240798,
author = {Ruo Ando},
title = {Multi-GPU Accelerated Processing of Time-Series Data of Huge Academic Backbone Network in {ELK} Stack},
year = {2019},
address = {Portland, OR},
publisher = {{USENIX} Association},
month = oct,
}