Very large databases often require storage over multiple locations. This can make accessing the data difficult, without knowing the physical location of the data. The solution to this problem is the Storage Resource Broker (SRB).

What is the SRB?

The SRB organises data logically into a single virtual file system in a similar way to a filesystem found on a stand-alone computer rather than physically. The use of stored metadata about each file facilitates the querying of these distributed data.

Metadata – data about the data – allow users to quickly find a dataset, what it contains, its format, when or where the data were collected or created, etc. Metadata querying and browsing enables users to have transparent access to each other’s data collections. Users can store or replicate their data collections across several servers to facilitate data preservation, while not losing local access control. Access permissions on individual files can be set so that other users can access the files. Users wishing to access these collections will view a single collection through the SRB middleware and do not need to know the physical location of the data.

How does it work?

Figure 1 illustrates how the SRB works in a typical scenario (the arrows show the typical flow of data). A user needs a file stored in ‘Data Resource 2’ which could be a tape archive or a hard disk etc. However the user does not have this information. The only thing that the user has to do is login and request the file by making a query via an SRB client. What follows happens behind the scene. The client contacts the local SRB Server A and requests the file (1). Server A contacts the MCAT-enabled SRB Server B in order to find the physical location of the file (2). Server B looks up the information in the MCAT and passes this information back to Server A (3). As the file is located on a resource maintained by SRB Server C, Server A redirects the request to SRB Server C (4). Server C retrieves the file from Data Resource 2 (5 and 6) and services the client directly (7). The files held on these devices appear to the user as part of a single file system. The full process is transparent to the user who does not need to know in which data resource the requested file is located.

Figure 1.

Usage and deployment of SRB

QCIF is currently supporting the deployment and support of SRB for a number of different scientific communities. In 2004 the UQ HPC unit upgraded its data storage capacity to 120TB of tape storage. The storage capacity was further increased in 2006, with an upgrade of another 100TB of tape. SRB is currently being deployed on the data storage infrastructure.

A number of projects are using the SRB, including the SensorNet project.

For more information visit the VisLab Storage Resource Broker project page.

Participants

David Gwynne, Terry Simmich, Nicole Bordes

Vislab, University of Queensland

Publications

Bordes, N., Ulm, S., Pettersen, O. et al., 2006. "Data Grid for the Management, Reconstruction, Analysis and Visualisation of Archaeological Data". Accepted for publication: In print. (2006)

Bordes, N., Ulm, S. et al., 2006.  "Towards an Australian Archaeological Data Grid". Submitted to Computer Applications and Quantitative Methods in Archaeology (2006)