2015 Technology Exchange

Use Internet2 SiteID

Already have an Internet2 SiteID?
Sign in here.

Internet2 SiteID

Coexisting Production and Experimental Testbeds at AmLight: A Secure Approach

Time 10/06/15 09:00AM-09:20AM

Room Grand Ballroom A

Session Abstract

Currently, AmLight network connects Latin American R&E networks to all other R&E networks in the world. This connectivity is provided by a set of four 10G international long-haul circuits using four different undersea cables. In 2014, AmLight has enabled OpenFlow to create its Software-defined Networking infrastructure, focusing on provisioning optimization and network virtualization support. The virtualization feature has been deployed to support experimental testbeds, providing a platform for innovation for researchers in the U.S. and Latin America. Even though AmLight is a production infrastructure, experimentation has been widely supported and researchers encouraged to use for prototyping their applications. Since 2014 more than eight testbeds were created in parallel with production applications. Each experimental testbed installed so far is completely different from every other, not just in its usage, but also the deployment and testing methodology available. To avoid any unexpected downtime caused by new testbeds, a testing environment was created with the same network devices and software from the production environment. The differences between each testbed create a complex testing process, involving AmLight engineers and researchers. After agreed that the testbed’s application is ready to be added to the production SDN network, AmLight engineers and researchers will manage and operate the application in a joint effort.

Running experimental testbeds on a production network involves risks and complex operation and troubleshooting processes. The risks involved in supporting testbeds are due to code instabilities on the OpenFlow agents deployed in the network devices and to the complexity of testing all functionalities supported by OpenFlow controllers and applications. Furthermore, code failures might crash not just the OpenFlow agent, but the entire network device, causing unexpected network outages that affect all services in place. These crashes might be caused by malformed or unsupported OpenFlow messages, or even buffer/synchronization flaws embedded in the network device's operating system code. Along the last months, AmLight has observed network outages due to a wide range of reasons, mostly due to network device's software failures. This instability has been forcing the AmLight engineers to be careful about new applications and testbeds to be added to the production environment, what justifies the testing environment and all procedures associated to it. Even all SDN applications going through the testing methodology, eventually outages happen. In most part of the outage situations, troubleshooting has proven dificult due to the complex architecture involved, where SDN applications, network hypervisor and network devices could cause different kinds of problems. Not all of them have good support for troubleshooting. At every situation, event logs and packet inspecion are used. Unfortunately sometimes these tools don't provide enough information which makes impossible to reproduce the event.

Based in the experience acquired along the last months operating testbeds at AmLight, RNP - responsible for the FIBRE testbed running at the AmLight - has been invited for a joint effort to design and develop a solution that would be responsible for (a) filtering all undesired OpenFlow messages per network device, per network device's line card, per software and/or per software version; and (b) logging flows based on profiles, which could be per match, per action, per switch and/or using multiple combinations. This solution, called Testbed Sanitizer, would be placed between the Hypervisor and the SDN application/testbed. With this solution, AmLight engineers will be able to better monitor, understand and troubleshoot the SDN environment, improving the network resilience.

The Testbed Sanitizer solution will be discussed and results presented. Use cases will be described to show how today's challenges in operating both a production and experimental environment supporting SDN applications can coexist productively.


Speaker Jeronimo Bezerra Florida International University

Speaker Marcos Schwarz RNP (Rede Nacional De Ensino E Pesquisa)

Presentation Media

media item thumbnail Coexisting Production and Experimental Testbeds at AmLight: A Secure Approach

Speaker Jeronimo Bezerra Florida International University

Speaker Marcos Schwarz RNP (Rede Nacional De Ensino E Pesquisa)

Primary track Advanced Networking/Joint Techs

Secondary tracks Security

platinum Sponsors

gold Sponsors

silver Sponsors

supporter Sponsors