The Fast Data Transfer Tool Tutorial: Overcoming Limitations to High Performance Transfers Over the Wide Area Network (Separate registration required)
Time 10/15/17 01:00PM-05:00PM
Room Pacific K
SEPARATE REGISTRATION REQUIRED: Register Here
One important challenge of performing data transfers is being as fast and efficient as possible, while, at the same time, keeping the usage of system resources as low as possible. Ideally, the software that manages these data transfers should be able to organize them in such a way that one is able to have them run up to the hardware limits.
In this tutorial we will describe Fast Data Transfers (FDT), an open source tool developed at Caltech for performing fast data transfers over the Wide Area Network (WAN).
FDT is an Application for Efficient Data Transfers which is capable of reading and writing at disk speed over wide area networks (with standard TCP). It is written in Java, runs an all major platforms and it is easy to use.
FDT is based on an asynchronous, flexible multithreaded system and uses the capabilities of the Java NIO libraries. Its main features are:
- Streams a dataset (list of files) continuously, using a managed pool of buffers through one or more TCP sockets.
- Uses independent threads to read and write on each physical device
- Transfers data in parallel on multiple TCP streams, when necessary
- Uses appropriate-sized buffers for disk I/O and for the network
- Restores the files from buffers asynchronously
- Resumes a file transfer session without loss, when needed
FDT can be used to stream a large set of files across the network, so that a large dataset composed of thousands of files can be sent or received at full speed, without the network transfer restarting between files.
Caltech recently made FDT open source under a Apache 2.0 license.
We will present an in-depth hands-on look at Fast Data Transfer (FDT), a simple open-source TCP application developed by Caltech and its partner teams serving high energy physics, genetics, biochemistry and many other fields of science. The tutorial will cover:
- FDT's origins and science. Areas of application of FDT.
- Illustrations of FDT's capability to achieve high throughput up to the full capacity of current networks; Limiting throughput at any level up to wire speed with FDT.
- A brief summary of affordable platforms for data transfers in the 1, 10, 40, 100 and 100+ Gbps throughput ranges
- How to obtain, install and run the code hands-on during the session
- How to join the open source community developing FDT
- Areas of code development - both ongoing and opportunities for new developers
Exercises during the tutorial:
- FDT Installation and use
- Available transfer options
- Writing Custom User extensions
- Running 3rd party transfer
- Secure transfers
- Transfer node optimization
Organizers of the tutorial: Prof. Harvey Newman, Dorian Kcira, Justas Balcas, Wayne Hendrics
California Institute of Technology
Speaker Harvey Newman Caltech (California Institute of Technology)
Speaker Justas Balcas Caltech (California Institute of Technology)
Speaker Dorian Kcira Caltech (California Institute of Technology)
Primary track Advanced Networking
Secondary tracks Applications for Research