Session Abstract

Syndicate is a global filesystem optimized to deliver large files (e.g,. scientific data sets, library collections, videos) to a scaleable number of clients distributed throughout the world. From the user's perspective, Syndicate offers an intuitive drag-and-drop interface (similar to Dropbox) through which data producers can make large files (and directories of files) available to a scalable set of clients, and data consumers can retrieve large files generated by one or more remote sites. Underlying this familiar interface is a scalable content delivery network, based on CoBlitz, a widely used CDN. Files written (published) to Syndicate are broken into chunks and distributed to a global set of high-performance caches. Files read from Syndicate are then pulled, in parallel, from these distributed caches, reassembled into their original form, and made available for local processing. Syndicate also maintains and distributes a file directory hierarchy, thereby supporting a full file system interface.

Syndicate (and the underlying CDN that it exploits) has application to both the scientific community that that has large data sets to distribute, and to community anchors that wish to both access and publish a wide assortment of content (including, but not limited to, scientific data sets) over modestly-provisioned access links.