2017 Technology Exchange

Use Internet2 SiteID

Already have an Internet2 SiteID?
Sign in here.

Internet2 SiteID

Your organization not listed? Create a local account to use Internet2 services.

Create SiteID

Flexible Network Analytics in the Cloud

Time 10/18/17 11:20AM-11:40AM

Room Grand Ballroom A

Session Abstract

There are many commercial and open source products that can be used to analyze network measurements. At ESnet we have some use cases that don't fit very well in the capabilities of existing systems and have been investigating the use of cloud services to analyze our data. We store our data in a very raw format and then produce views of that data which provide greater insights. These views represent design decisions but since the raw data is retained we can make new views as we discover new use cases.

By using Apache Beam (https://beam.apache.org/) in the cloud we are able to focus on the analytics results we want to see from our data and not worry about managing infrastructure to perform the calculations. This lets us focus on the 'what' instead of the 'how'. "What insight do we want to extract from this data" instead of "How do we analyze all this data?" We plan to handle both SNMP and Flow data but the approach is general enough that many other kinds of data could be analyzed in a similar way.

This talk will describe the overall system architecture, describe what has gone well, what has been challenging and give some details about our pre-production system. There will also be a discussion of the general principles behind this system that can be used regardless of what tools or frameworks are used.


Speaker Jon Dugan ESnet (DOE Office of Science - Energy Sciences Network)

Speaker Sowmya Balasubramanian ESnet (DOE Office of Science - Energy Sciences Network)

Presentation Media

media item thumbnail Flexible Network Analytics in the Cloud

Speaker Jon Dugan ESnet (DOE Office of Science - Energy Sciences Network)

Primary track Advanced Networking

Secondary tracks Web-Scale Computing

gold Sponsors

bronze Sponsors