Logistical Networking technologies offer a highly scalable means to manage distributed content using shared network storage. Our software tools allow users to deploy local IBP storage "depot(s)" or utilize shared IBP storage deployed worldwide to easily accomplish long haul data transfers, temporary storage of large data sets (on the order of terabytes), prepositioning of data for fast on-demand delivery, and high performance content distribution such as streaming video. Of immediate interest to SciDAC collaborators is the ease with which Logistical Networking facilitates the transfer of massive data sets.
Logistical Networking software currently in use by the SciDAC community include:
1.) Internet Backplane Protocol (IBP): IBP is a low-level mechanism for managing remote storage as a sharable network resource through deployment and shared use of lightweight, time limited storage allocations called storage "depots." IBP v1.3 depot software offers improved reliability and IPv6 compatibility.
2.) exNode: The exNode is a generalized data structure which holds the metadata necessary to manage distributed content stored on IBP depots and allow file-like structuring of stored data. The exNode records information such as which IBP depots house replicas of data content.
3.) Logistical Backbone (L-Bone): The L-Bone provides a directory service for registered IBP storage depots, cataloguing an ever growing international deployment of 200+ depots that serve 15 TB of storage as a shared resource for the scientific community. Public depots near the national backbone include a 4.7 TB server connected at StarLight in Chicago. A second, private directory is being established to serve the SciDAC community exclusively (see below).
4.) Logistical Runtime System (LoRS Tools): The LoRS Tools software provides high-level file management capabilities, high-performance access, and end-to-end services such as data compression, checksums, and encryption. SciDAC collaborators use the LoRS Tools to store, manage, and retrieve data via the Logistical Network. LoRS Tools v0.81 includes an improved graphical user interface, as well as the LoRS View visualization tool for viewing data manipulations in real time.
5.) Data Movers: DataMovers are auxiliary IBP depot modules that support customized or special purpose depot-to depot-transfers. Two new DataMover plug-in features are included with IBP v1.3 to enable UDP point-to-point and multicast transfer mechanisms.
A primary research drive has been interactions with the Terascale Supernova Initiative (TSI) group. TSI uses Logistical Networking to share massive data sets between distant collaboration sites. Using the LoRS tools, they can now transfer data at speeds up to 220 Mbps between key research sites at ORNL and NCSU. See Figure 1.
A new, private Logistical Networking infrastructure will be used specifically for the advancement of TSI and other SciDAC research endeavors. The first of five high-performance, 1.9 TB dedicated IBP servers has already been deployed at ORNL. Deployment of four more high powered machines at other principle TSI sites (SDSC, SUNY-Stony Brook, LBL, NCSU) will provide an additional 8 TB of IBP storage.
Two new Logistical file systems provide a familiar file system interface for our novel network storage infrastructure.
The Read Only Logistical File System v1.0 (ROLFS) provides a content directory service with active exNode management, to allow effortless data sharing. ROLFS performs scheduled renewals of time-limited IBP storage allocations, as well as maintaining preferred fault-tolerance and performance levels by automatically creating redundant copies of data fragments when necessary. A private ROLFS directory for TSI will be in place by early fall.
The Logistical File System (LFS) is a full-featured file system for use on workstations or file servers. LFS provides a more scalable and flexible foundation for distributed data management than a traditional file system. LFS stores data on IBP depots via the Logistical Network, instead of a local disk or file server on the local network. LFS transparently handles the tasks of finding an IBP depot, allocating storage, and storing data on the Logistical Network.
A beta version of LFS ported with NCSA's Hierarchical Data Format (HDF) v4.1 scientific data management library allows TSI and other SciDAC researchers to release output from complex modeling applications directly to the Logistical Network. Using Logistical Networking functionality to enhance the widely used HDF library is an important segue for reaching the broad SciDAC community.