100G R&D at Fermilab
Time 01/15/13 01:20PM-01:35PM
Fermilab is at the leading edge of Big Data. For a sense of scale, one if its largest facilities, the LHC CMS Tier-1 center, stores ~40 PetaBytes of data, routinely reaches aggregate LAN traffic levels of 140 Gb/s in moving that data between archive and local processing farms, and frequently sustains WAN traffic levels in excess of 30 Gb/s.
In this context, the High Throughput Data Program (HTDP) at Fermilab was created to investigate the impact of and provide solutions for the scientific computing challenges in Big Data faced by the Fermilab stakeholders. HTDP has been actively using the ESNet 100G Advanced Network Initiative (ANI) infrastructure since 2011 to identify gaps in commonly-deployed data management middleware when used over 100 Gb/s networks. The program has tested middleware such as GridFTP, GlobusOnline, the Storage Resource Broker, XRootD, and Squid, providing feedback to the developers on uncovered shortcomings. The creation of a Fermilab R&D Testbed Facility connected to the 100 Gb/s link at Fermilab opens up new opportunities for several new programs and the continuation of the High Throughput Data.
This talk will describe the results of the middleware investigation on 100G, present the R&D testbed facility, and introduce our work program.
Speaker Gabriele Garzoglio Fermi National Accelerator Lab